It's more useful to think in terms of what forces are needed to be applied to the environment to accomplish a task. Then you can build a solver to develop innovative physics based solutions that are unconstrained by the semantics.
"Move egg insides into oiled pan on heated surface, move egg shell into trash" it can be purely physics based.
You'll have to find the egg- it's inside the carton inside the fridge behind the milk.
The fridge door, the milk, and the carton are all blocking the desired trajectory of shell and interior.
From a physics perspective you can minimise the energy expenditure required to apply the forces and motions of all objects in the immediate environment to accomplish the objective.
Such as: The door needs to rotate about the hinge, not be ripped out of the way. The milk could be pushed gently to the side rather than removed and set on the counter to expose the carton.
We still haven't programmed proper gripping technique for an egg, but if the chef robot only has a single spatula for a manipulator, a physics based robot would just have to generate teppanyaki techniques and balance the egg on the spatula rather than fail.
That said, I don't relish the idea of my oven or toaster being internet-connected (https://tech.slashdot.org/story/20/11/25/1910244/aws-outage-...).
I also think it's good to also think long term beyond what is immediately feasible or commercially viable. Personally, as someone who enjoys the sci-fi movie Blade Runner, I would much prefer to see a robotic android being "more human than human", than having to compromise in the design of homes.
Software wise it'd be both individually programmable and have access to some sort of thingiverse equivalent with actions that others have dreamed up. Otherwise it would not really need an internet connection at all, except perhaps to the local wifi to talk to other more-or-less automated systems in the same house.
For applications, some of the ideas I had while typing the rest of this comment:
- If there is a package outside the door, open the door, pull the package inside, close the door again.
- If the mail has been delivered (for houses that have a front door with a mail slot in them), use the vacuum attachment to pick up the mail and deliver it to a central location (desk? dinner table? kitchen?)
- If the dishwasher is done, use universal gripper to pull it open, open the closet and put the dishes away. (Or into a drying rack, whatever).
- For most people I know the washing machine for dirty laundry is someplace away from the bedroom. I don't think it'd be reasonable with current tech to expect robot folding of clothing but at least it could pick up dirty laundry from various places where it's collected and bring it to the washing machine so people only have to come in and turn the machine on.
- Depending on how good the gripper is and how well adapted my coffee machine is to it, perhaps it could pick up a coffee capsule and prepare it while I'm in the shower.
- Once per day it would make a round around all the bathrooms and refill the spare toilet paper roll holder if there is only one roll left.
- Maybe it could wipe down sinks and stuff regularly too.
All in all, it'd be useful several times per day I think. Not doing anything I couldn't do myself but definitely taking care of various simple chores throughout the day. It would need pretty good sensor coverage as well of course. Finally, I realize that with current tech this would be prohibitively expensive.
Thanks for enumerating the use cases. They are fun to think about. I wanted to dive into the first application you mentioned. As mentioned in my post, even the most basic manipulation tasks on human-centric objects and spaces are actually full of little details.
I hope people think more on that low-level dextrous manipulation when designing robot hardware rather than the fairly high level "open door, pull package inside, close the door", which might make robotics people receptive to the idea that a humanoid is truly the only viable solution.
What if your door has a step down to the porch where the package is delivered? The robot can't "just" pull a large box inside, it would get stuck. You'd need to lift it up, and at this point you'd require two arms or a gantry that can extend outside of the home above the porch. Obviously the home can be re-designed around this, but my point is that there are really two kinds of robotics - ones that try to solve a human problem, and ones that try to do everything a human can.
There are plenty of kitchen tasks which are way easier for robot, for exampling making a bowl of cereal. Which brings me to the second point. You say:
> The only viable hardware for a robot meant to do any task in human spaces is an adult-sized humanoid, with two-arms, two-legs, and five fingers on each hand.
but I think this is only true if robot is constrained to using tools designed for humans. In the strawberry example, the knife is designed for multiple fingers, and cutting board has no elevated border to work well with the human knifes.
I think a properly designed "kitchen arm" (with maybe compliant/under-actuated fingers, a few sharp blades, a vacuum grip, etc..) as well as robot utensils (like uneven cutting boards that the food does not roll off from) would allow robots to do a large fraction of the kitchen tasks.
And maybe your bag of dates would be cut open instead of opened nicely -- but you'd still get to eat them.
The biggest problem by far is the software - in particular the AI. The biggest companies in the world have thrown billions of dollars at AI, Universities have had some of the most brilliant minds among their ranks, and we essentially got some (impressive) slightly better search algorithms.
There are absolute fundamental questions (+) that need to tackled in order to have the kind of generalized AI such environments require. Most AI (that I'm aware of) currently lacks the ability to do anything other than optimize itself for strictly specified scenarios/environments.
Personally I quite like the information theoretic approaches to self-motivated agents, there are some nice mechanisms out there such as empowerment . It's not the full picture, but it's a step in the right direction. I don't think this is something we can throw larger neural networks and computation resources at and hope it solves itself.
(+) This is the subject of a paper I am currently writing.
Also, what would a good interface between a Software 1.0 program and a Software 2.0 program look like in robot software? I mean, what would the boundary between (3) and (4), and (4) and (5) look like in this imaginary stack?:
(5) Autonomous controller (software 2.0)
(4) A high level interface for giving instructions to (3), and finding out what (5) is doing
(3) Motor manipulation controller (software 2.0)
(2) A daemon for converting NN outputs into safe hardware control outputs (software 1.0)
(1) OS kernel (software 1.0)
Could I really have a pick_up_object() start up? There’s actually demand for that?
One you go to a large, heavy arm you need more powerful and precise motors, and the cost of mistakes dramatically increases (monetarily and physical damage).
and probably others.
This type of research makes me optimistic about possibilities for generalization in the future.
But due to the depth of the problem and not having resources or much knowledge, it was maybe a little silly to create the company. But I like the idea of reserving that name, just in case I ever get anywhere.
From the hardware side, I feel like some of the robotics issues can be resolved by "just" copying people more closely. For example, it seems like the way real arms and muscles work should provide more leverage and force than the typical servo setup. And having five fingers provides the potential that manipulations could be copied from people.
There is also a very promising new type of artificial muscles called HASEL.
Of course, in order to efficiently build these human-like limbs, "all" we need is a way to 3D print with several materials at once, including a new type of conductive ink that can handle high voltages for the HASEL muscles.
But the starting point to me is a robot that can actually understand what it's looking at. In that it sees with depth, and understands the composition of objects and their orientation, etc.
Capsule networks seem interesting but also maybe are a bit computationally expensive and unproven? Also he seems to be focused on just the transformation matrix, but it seems like there are more aspects of the state that could be relevant and maybe are unique to different object types. But I am slowly trying to understand capsules anyway.
I have seen a few ideas about more general neural network-based systems that suggest it is necessarily to have multiple neural networks, or networks of networks, or neural modules, etc.
To me it seems like the ideal thing would be to have some standard shapes for networks or modules and also be able to reuse and adapt them for different tasks.
So my vague ideas now are something like: standard-shaped modules, trained on core modeling tasks such as finding 3d surfaces in 2d images. But at the same time somehow segmenting into different objects. And the potential high-level objects should be able to feed into the potential low-level understanding and visa-versa.
My intuition is that ideally there is a sort of 3d wireframe overlayed on the 2d image, identifying each object and sub-object with its exact dimensions, shape and orientation. Kind of like I've seen in one or two science fiction movies. So if I can somehow generate all of that, I know I have properly decoded the image.
Today I was looking at a GAN tutorial. But I have never made a CNN before, so decided that must be first.
Usually I think about this stuff for awhile and then just decide I don't really know what to do and then go back to Coursera. I finish Ng's first class and am looking at the hyperparameters one. I feel like I need to make some actual neural networks on my own though, because mainly Ng is teaching me how to convert from math notation to vectorized Python as much as anything.
If nothing else, this is really motivating me to learn about existing AI techniques. Which I feel like, to be a good programmer, I actually should be able to use things like Tensorflow etc. for narrow AI tasks.
Feel free to get in touch if you want to chat and trade ideas. (Email in profile)