Simplification is the problem here, arguably. Even a simple-sounding objective (say, a bicycle wheel that holds load the best) has at least one implicit assumption - it will be handled and used in the real world. Which means it'll be subject of sloppy handling and thermal spikes and weather and abuse and all kinds of things that are not just meeting the goal. Any of those cheesy AI designs, if you were to 3D-print/replicate them, they'd fall apart as you picked them up. So the problem seems to be, ML algorithm is getting too simple goal function - one lacking the "used in the real world" part.
I feel that a good first step would be to introduce some kind of random jitter into the simulation. Like, in case of the wheels, introduce road bumps, and perhaps start each run by simulating dropping the wheel from a short distance. This should quickly weed out "too clever" solutions - as long as the jitter is random enough, so RL won't pick up on it and start to exploit its non-randomness.
Speaking of road bumps: there is no such thing in reality as a perfectly flat road; if the wheel simulator is just rolling wheels on mathematically perfect roads, that's a big deviation from reality - precisely the kind that allows for "hacky" solutions that are not possible in the real world.
You would have to introduce jitter to every possible dimension, when the dimensions themselves are continually expanding (as illuminated by the bike wheel example).. the combination of jitter x dimensions leads to an undefined problem (AKA theory of everything) in exponential fashion
Humans don't simplify problems by reducing them to objective functions: we simplify them by reducing them to specific instances of abstract concepts. Human thought is fundamentally different to the alien processes of naïve optimising agents.
We do understand the "real objectives", and our inability to communicate this understanding to hill-climbing algorithms is a sign of the depth of our understanding. There's no reason to believe that anything we yet call "AI" is capable of translating our understanding into a form that, magically, makes the hill-climbing algorithm output the correct answer.
My point was that instead of blaming ML - or optimisation tools really - for gaming objective functions and coming up with non-solutions that do maximise reward, AI could instead be used to measure the reward/fitness of the solution.
So to the OP's example "optimise a bike wheel", technically an AI should be able to understand whether a proposed wheel is good or not, in a similar way to a human.
I see a lot of comments wondering how AI could be useful as a software engineer so I'll give my take on it:
I envision it being able to fully replace a junior engineer, and in some use-cases senior engineers as well.
In the case of junior engineers: the AI should have access to an internal knowledge base (i.e. Confluence) and the task/ticketing system (Jira), and ideally the internal chat (Slack). I would assign tickets to it, and I'd expect the AI to reply with comments asking for clarification when there is something unclear, or proposing an implementation before starting if it's not a very simple task (this could even be defined in the task, using story points for instance.
Once cleared, the AI submits a PR linked to the task - so far just like any engineer in the team would. The PR gets reviewed as usual, with suggestions/requests for changes made by the (human) team, which then get addressed by the AI. With the big difference that all this process may happen in less than 1h from ticket creation to PR review.
I wouldn't expect it to be able to implement complex features, onboard new libraries, or rearchitect the system in major ways to accommodate future features - just like I wouldn't expect that from junior team members.
It would obviously be amazing if it could incorporate previous PR comments into it's "context" for future work, so it could learn and improve.
Separately I mentioned it could also do part of the job of senior team members - in the form of PR reviews. If it has access to every previous PR review and learns from them it might be able to give insightful suggestions, and for very large codebases it could have an advantage over humans as it could find patterns or existing code that may be overlooked (i.e. "It looks like this util function you added is the same as this one that was already there", or "This code looks different to similar areas, but follows a different pattern, you might want to rewrite it this way //..."
Is GPT-4 there? Definitely not, perhaps an LLM is not even the way to achieve this, but I absolutely see this becoming an incredible asset to a tech team.
Best thing you could do, spend some time learning how it works properly (query keys, invalidations... ) and how you'd set up things like pagination.
If at any point anyone goes like "oh it doesn't seem to support this, we'll have to write some wrappers/custom stuff", take a step back and read the docs again or ask around - the library in general takes care of everything it should and gets out of the way really well otherwise.
Assuming you have a table where the identifiers are stored you'd have the internal one (UUIDv7) and the encrypted version from it (external id).
You could rotate encryption keys whenever you want for new external id calculation, so that older external ids won't change (as they are external, they need to stay immutable).
If you're storing the identifiers then you don't need encryption, you just generate a UUIDv4 and use that as the external identifier. And then we're back at where the blog post started.
We recently migrated to Poetry [1] for dependency management and so far it's been a breath of fresh air - it feels like what Python deps should've always been!
You can even have dependency groups, to separate main/dev dependencies for instance. It also brings env management, and plays very nicely with Docker if you use containers.
Poetry doesn't really work well if you try to ssh in. It tries to setup a keyring for some reason whenever you use it, but even if just to Runa project. I feel like it shouldn't be that hard to get going, but after trying to hack it for four hour I used a workaround of telling the keyrings to use a null provider & then was able to download deps & run the project I'd downloaded.
Admittedly only a single nit but it was still one of thr saddest most frustrating python experiences I've ever had.
Not a week goes by without a new issue popping up. A dependency manager is a critical piece of infrastructure, it should not be the main developer experience bottleneck when building an application. A good package manager is out of sight, out of mind. You don't hear people complaining so much about Cargo every day.
I don't really feel like defending poetry, I don't even use it. But in this particular case, I think the wheel format and the way people try to host it on their servers and the entitled researchers who act like 80 year olds and "just want things done" instead of taking a little bit of time to learn to work with their tools are to blame. None of this exists with the cargo infra.
This is something I've been thinking for a while for my long term future!
A dream of mine would be to setup a hostel of sorts, with an area for camping, a cafe, maybe food trucks in the summer?, surf/climbing options nearby and perhaps a partnership with schools/guides.
And the main thing: a big hangar to work on campervan conversions, with a ton of tools available and some staff supervision. The idea being that you bring your van, pay some sort of membership/one-time fee and work on it, with our help.
Perhaps do some in-house conversions for clients, but that would be just a side business to help keep things going.
Just an FYI, but if it’s at an airfield, the FAA can come after you if you don’t also use it for some kind of aviation. I forget the exact wording, but it’s basically to help protect hangar prices for people who fly.
Well, I think there's value to both points - how much of the maps code would you need to be able to get something working, and how much work would it take to implement all the custom internal dependencies that would probably be missing?
I'm sure there are very complex valuable pieces of code within the maps codebase (or any other), but it would be a fairly massive task finding those pieces, extracting them, and getting them to work properly in a different codebase...
If anything, AI could help by "understanding" the real objective, so we don't have to code these simplified goals that ML models end up gaming no?