Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Put this touch sensor on a robot and learn super precise tasks (any-skin.github.io)
390 points by raunaqmb 81 days ago | hide | past | favorite | 66 comments
We just released a very excited touch sensor that finally simplifies touch sensing for robotics.

Our most exciting result: Learned visuotactile policies for precise tasks like inserting USBs and credit card swiping, that work out-of-the-box when you replace skins! To the best of our knowledge, this has never been shown before with any existing tactile sensor.

Why is this important? For the first time, you could now collect data and train models on one sensor and expect them to generalize to new copies of the sensor -- opening the door to the kind of large foundation models that have revolutionized vision and language reasoning.

Would love to hear the community's questions, thoughts and comments!




So you embed magnetic particles in silicon rubber and magnetize them, then use magnometers to detect how the magnetic field is changing from a few different points of reference in order to detect the deformation of the rubber and from that you can analyze the "pressure points" on the surface. the innovation here is that you dont have a lengthy re-calibration of your "input signal" to the particular magnet-infused silicone interface because the manufacturing makes them consistent enough to be replaceable parts?

this makes advanced touch sensors more like machine-cut screws than bespoke hand-forged nails.


I'll bet you could open the grippers fully and recalibrate on power up.


Great idea, you could do the same with a capacitive xy sensor.


Exactly! You need little to no re-calibration.

With capacitative sensors, it is unclear from existing literature if it is possible to detect shear. Additionally, they generally operate at significantly lower frequencies.


I don't know anything about this space, but damn, this looks impressive!

Could it be used to sort trash and recycling? Could it recalibrate if gunk got on it, or as it aged? (I guess silicon is probably pretty resistant to aging.) Can it wash and de-stem a tomato?

I think I want a trackpad made out of this. How much resolution could it get? I suppose I wouldn't want to sacrifice a lot of resolution for the pressure, tilt, etc. that I am assuming this would provide.

(I said "think", because I might find out that it feels like running my finger over skin, and I'm wondering how creepy that might feel. I don't really want my laptop to have a fleshy part.)


I've worked in this space in automation with industrial grade robots and more bespoke end effectors that don't look like mainstream robots, but fulfil specific needs. Responding to some of your questions with how I could see the above touch sensor helping:

Trash sort and recycling: Not many robots here, majority of sorting takes advantage of object material properties. Some companies tried to add delta robots to keep up with the high rates required to even approach profitability, but they weren't good enough. Maybe some municipalities or universities that have lots of funding could justify adding robots, but it's just hard to financially justify.

Recalibration: I'm curious what the developers have for handling reduced magnetic fields over time along with gunk. Silicone is washdown rated, but anything soft at high throughput with parts will start to wear out and change pickup characteristics.

Washing and destemming a tomato is more of a problem to solve now that will need another 10+ years of price reductions in robot+end effector costs and increased efficiency before it beats bulk washing and hand-destemming (or crude machine work). Maybe it'll be a grad-student's project for a theoretical future home-bot

The Lenovo TrackPoint is likely already 95% of what you'd need from a trackpad, but this touch sensor is likely not even focused at that market.

Things I see useful for this robot touch sensor:

* Simpler version that detects part presence, is just a Boolean feedback of "part detected" which can stick on existing end effectors. This is often handled by load calculations of the robot to detect if it has a part, but could also detect if a part has substantially "moved" while it's been gripped, sending a signal to the robot to pause

* Harder to suggest items for food as soft grippers (inflatable fingers) will grip at the precise pressure that they're inflated, reducing the need for sensitive feedback. The application for this touch sensor would be food that needs a combination of different pressures to properly secure something, can't think of a great example

* Hard to also suggest places where this sensor would help with fine alignment, as major manufacturers have motor and arm feedback with WAY more sensitivity than the average person would realize, google Fanuc " Touch Sensing". But, this could help when the end effector is longer and it's harder for the joints to detect position

* Fabric manipulation. Fabric is just a hard problem for robots, adding in more information about the "part" should be helpful. Unlocking more automations for shoe manufacturing at reasonable prices is a big wall


This is a very insightful summary, thank you! A few things to add about AnySkin that might be relevant:

- AnySkin expressly handles wear and gunk by being replaceable. So if it wears out, and you have a heuristic or learned model for the old skin, it will work pretty well on the new skin! We verify this through an analysis of the raw signal consistency across skins, as well as through visuotactile policies learned using behavior cloning. We found swapping skins to work for some pretty precise tasks like inserting USBs and swiping credit cards.

- Could definitely be used for part motion detection

- Soft, inflatable grippers are effective, but often passive. AnySkin is not just soft, but also offers contact information from the interaction to actively ensure that blueberry doesn't get squished!

- This sensor would be key for robots that seek to use learned ML policies in cluttered environments. Robots are very likely to encounter scenarios where they see an object they must interact with, but the object is occluded either by their own end-effector(s) or by other objects. Touch, and an understanding of touch in relation to vision becomes critical to manipulate objects in these settings.

- Industrial robots do have very sensitive motor and arm feedback. However, these systems are bulky and unsafe to integrate into household robotic technologies. Sensors like AnySkin could be used as a powerful, lightweight solution in these scenarios, potentially by integrating with some exciting recent household robotics models like Robot Utility Models.

- ReSkin, the predecessor to AnySkin, has previously been used quite effectively for fabric manipulation! (see work from David Held's group at CMU). AnySkin is more reliable as well as more consistent and could potentially improve the performance seen in prior work.


> - Industrial robots do have very sensitive motor and arm feedback. However, these systems are bulky and unsafe to integrate into household robotic technologies. Sensors like AnySkin could be used as a powerful, lightweight solution in these scenarios, potentially by integrating with some exciting recent household robotics models like Robot Utility Models.

I bet having good touch sense would let you get away with much cheaper mechanical systems for the robots.


Industrial robots are mainly bulky because they need to be very robust and precise at almost incredible speeds. (I work with those). Its not uncommon to have a 500+ kg robot on 500kg rail (totaling 7 axes) to actuate a 1mm wide, 5cm long nozzle, and moving it at speeds of 1+ meter/second, while navigating it in a gap where it has 0.5mm space on each side of the nozzle. Consistently, all day every day.

Lots of industrial robots arent even meant to touch their work piece, yet the robustness is the only way to make the whole assembly rigid enough.

I can imagine a touch-sense equiped arm could be made way smaller (less rigidity being compensated by quick enough feedback loop), but the speeds would probably have to decrease quite a bit. Not a problem for home robots tho.


> Washing and destemming a tomato is more of a problem to solve now that will need another 10+ years of price reductions in robot+end effector costs and increased efficiency before it beats bulk washing and hand-destemming (or crude machine work). Maybe it'll be a grad-student's project for a theoretical future home-bot

Heh, fair. I wasn't thinking of this as a practical usage, it was just the first thing to come to mind when imagining a task requiring a lot of pressure sensitivity and a range of forces.

Then again, now that I've said it, I believe the current approach to this is to breed really hard, tasteless tomatoes and then agitate them in a vat. Perhaps we can eventually get tastier produce if robots can handle more fragile things!

Hm... or you could invert things and make a glove, then use it as a controller. (VR, or just a richer set of control dimensions for eg photo editing or something.) I guess that needs to generalize across hand shapes and sizes, not just swapping out the glove, but I'd be up for a calibration/training phase.

> * Harder to suggest items for food as soft grippers (inflatable fingers) will grip at the precise pressure that they're inflated, reducing the need for sensitive feedback. The application for this touch sensor would be food that needs a combination of different pressures to properly secure something, can't think of a great example

How do you know the right pressure without feedback? A lot of foods vary in firmness over time and ripeness. Lemons, for example. I guess most don't, as long as you're sticking to a single type of food.


At the heart of it is a nice 3 axis magnetometer chip[1] in an array. The magnetic particles embedded in the replaceable skin get oriented in parallel at the magnetization stage of manufacture. This is a really interesting mix of stuff towards the leading edge of stuff we can all use in the home shop.

[1] https://www.digikey.com/en/products/detail/melexis-technolog...


Yeah, they frustratingly leave out the design of their circuit and the part from the paper but reference their older work.

ReSkin: versatile, replaceable, lasting tactile skins https://arxiv.org/abs/2111.00071

> Z- coordinate system [36]. For an overall sensing area of 20mm x 20mm (Figure 3), we measure magnetic flux changes using 5 magnetometers. Four magnetometers (MLX90393; Melexis) are spaced 7mm apart around a central magnetometer. All 3D-printed molds, circuit board files, bill of materials, and libraries used have been publicly released and opensourced on the website

https://reskin.dev/

a breakboard is available here https://www.adafruit.com/product/4022


We only leave the circuit design out because it is identical to Reskin! https://ReSkin.dev

More than happy to answer questions about it either here or on my email as the corresponding author on the paper!


This seems like it would be really useful for electronic musical instruments. E.g. Linnstrument (https://www.rogerlinndesign.com/linnstrument) which uses a grid of force sensing resistor strips. Do these sensors interfere with each other if they're sitting side by side?


Agreed, if these can be put into an array without interfering with each other it seems like it'd make a really cool expressive instrument. Cost would be a concern, though many of the existing instruments are exactly low-cost as is.


This was my first thought too! Other stuff in this category I know of are the Roli instruments (Seaboard, Blocks) and the Haken Continuum. All of these are pretty darn expensive for the larger models and I wonder if this new tech would be a cheaper way to make these work.


Why? You could just hardwire the control. Simulating a finger adds nothing.

This might be interesting for musical instruments with more tactile feedback, like hand drums or violins. But an electronic control surface like that only exists because human musicians aren’t already robots.


I think you've misunderstood - there are no robots in my proposal. You use this sensor as a key, then touch it with your finger, and it can detect force and directionality. Put together a grid of these sensors and you've got an instrument with really expressive potential.


Ohhhh - yes that’s awesome, love that. It’s like 3d aftertouch.


Maybe you could just make a big one.


Yeah I doubt you couldn't make one long strip and a few dozen magnetometers in a grid below it. Might be tricky to implement close multitouch reliably though.


For inserting USBs and similar tasks, is it sensing the angular change (and/or pressure differences between the two 'fingers') as the robot aligns into the hole? (as if the robot is 'feeling' it's way to aligning the usb plug).

Other questions: Is the primary skin material a molded silicone or possibly TPU (can be 3d printed)?


Looks like it's a cured silicon, and you can do whatever the heck you want with it.

https://www.smooth-on.com/products/dragon-skin-10-slow/

So I don't think you could 3d print it, but you could 3d print a mold.


Yes, you can 3D print a mold and we release this design tool: https://cad.onshape.com/documents/f3ec62110b01a3ad0fcb6d85/w... You can make whatever 2D shape you want in shape_sketch, as long as it is within the bounding square, and we automatically generate molds with the requisite inlet and outlet channels! It is still in prototype mode and we are working to make it robust, but it generally works and was used to make all the different shapes you see on the website and in the paper.


As for what it is sensing, we learn end to end policies in this case, and allow the neural network can pick up on whatever it needs for the particular task! but we have run experiments with a predecessor of AnySkin, ReSkin: https://reskin.dev that indicate you can localize contact at sub-mm scale as well as sense normal and shear forces!


https://formlabs.com/blog/inside-production-robot-hand/ is from almost a decade ago, but yeah - print a mold, stick electronics in slots in the mold, pump silicone around it. Back then it was sort of novel that commercial-grade resin printers could produce smooth enough surfaces for this; I expect that today, hobbyist-grade ones probably suffice.


there are a lot of magnetic-particle FDM materials out there, I guess a project like this is waiting for a filament house that wants to start experimenting with magnetic powders + TPU.


Very nice, and much easier to manufacture than the old Takktile sensors https://biorobotics.harvard.edu/takktile.html - it also looks like you could use the skins to destructive levels of force, without damaging the circuit boards at all, with a stiff enough layer between the chips and the skin (the Takktile system put the epoxy directly in contact with the pressure sensors, so while you could use protective layers over that, it would necessarily reduce the sensitivity.)

How tech-independent is the policy learning part? Do the models end up relying on how the board is giving you direction vectors, rather than contact location? (Nothing wrong with that, I'm just wondering if the directional aspect "factors out" certain kinds of change, and thus simplifies the learning process.)


While the sensor gives us direction vectors, they serve as good proxies for contact location, as we showed with ReSkin, https://reskin.dev.

That being said, the exact quantities the policy depends on are hard to interpret, given the use of deep learning. This could potentially be modality agnostic, but there has been no sensor so far that has shown (1) the ability to detect intuitively relevant quantities like contact location and 3-axis forces, and (2) sufficient signal consistency for deep learning models to generalize across instances. This was a key motivating factor for AnySkin, and we found a relatively straightforward fabrication procedure that enables this for magnetic sensing.


Curious, could you not calibrate using a force sensor, then include the output as a learning parameter. This seams a naive approach, which likely means it has been tried early on with other low hanging fruit, but I'm curious what the analysis of that approach is. Is there a fundamental reason this wouldn't work?


You could, and this is what we did with ReSkin, https://ReSkin.dev

The reason we don't want to do this is that it is difficult to cover all possible characteristics. Say we do single point contact localization, and 3-axis forces prediction. What happens when we have multi-point contact? The calibration has only been used to calibrate/align in a lower dimensional space. This is primarily why not needing calibration and baking this into the hardware is a lot more appealing. The user/designer no longer needs to think about the task and the dimensions of alignment required for that task.


I did some robotics tactile research, it was super fun! We used "biotac" sensors, which are very capable, but are 1) crazy expensive and 2) crazy hard to replace the skins, which do wear out.

One advantage biotacs have over these is that I can send a guy a (very large) check and buy them. Most academically-sourced things like this cannot be gotten for any price. These look cool, I'd love to have a few.


Seems like you could make the skin pretty straightforwardly in a home-shop. You'd just need to 3d print TPU and embed some high quality magnets (you can remagnetize your own pretty easily probably, not cheaply though? https://www.magnet-physik.de/en/magnetizing-technology/magne...)

And the board underneath is just a grid of these https://www.adafruit.com/product/4022 ?


I love the "Fabrication process" graphic. You can't make it simpler than that.


I get the application being a touch sensor, but the real breakthrough is the embedding of magnetic particles with parallel orientations in a flexible medium. A few immediate questions:

-The company that makes Magnequench presumably knows their particles can be embedded in other materials. What's the most common use case for these types of particles? Are they typically embedded in fixed (non-flexible) solid medium or liquids?

-Is it necessary to use Dragon Skin? The idea of mixing magnetic particles in a semi-solid medium is intriguing to me. Like putting particles into Jello or Silly Putty. Could I then apply an external magnetic field and have the particles deform/shape the medium they are in?


> Learned visuotactile policies for precise tasks like inserting USBs and credit card swiping...

> opening the door to the kind of large foundation...

Sounds like this enables robots to literally open a door, using a door handle or door knob. Exciting!


Yes, and importantly we find that visuotactile policies work even when replacing skins. This hasn't been shown before, to the best of our knowledge, and opens the door to a number of exciting large-scale applications of this sensor.


Nice packaging.

Sensors like that have been around for decades, but this is a nice way to package them. The replaceable cover is a big win. "Skin" type sensors have been built many times, but the part that wears out contained the sensors, so they were not suitable for production use.[1]

You have to have a Google account to order one. Even though this was funded by Meta.

[1] https://www.youtube.com/watch?v=HLdVkaF9ZR4


We are just collecting emails on the Google form as contact information to get more details when shipping samples. I am sorry that the form is asking for a google account - we will fix that as soon as possible.


I'm imagining a small, flat robot with a skinny gripper using this sensor that can roam a floor, going under furniture, and grab anything it can find and bring it to a collection point. So it would have to deal with picking up coins laying flat, empty soda cans, full soda cans, toys, dead bugs, dust balls, pens, a very wide variety of objects. Different objects may have different force feedback requirements, picking up an empty aluminum can would be different than picking up a full can. Picking up a coin might require good sensing at the very edge of the sensor.

It might be useful to create a map of the sensors response across its entire surface. Is the edge response weaker? That might suggest some improvements like working on creating less dead space around the edges of the sensor.


The fun thing about using microparticles is that there's no dead zone! In fact, the edge response is even stronger (as you can see on the video on our website) because despite the distance from the chips, the skin is much more deformable at the edges.


I did watch the video but that's not the same as a precise repeatable experiment. As you say the edge response does seem stronger which means the sensor response is not linear across its surface. I guess I'm thinking of precision manipulations of objects in predictable ways, which is probably not your original intent, but it seems likely you could improve the sensing at the edge. An experimental measurement of the response might show some nonlinearities across the surface which you might consider correcting by using a microparticle cap that varies in thickness or correcting it in software, to produce a more accurate sensor surface. While it seems quite useful as it is, adding precision may expand the possible uses, such as finer manipulation of more fragile objects. It would also be interesting to see how the response varies for different kinds of contacts with objects, such as gripping a cube by the corners and by the sides, by the sides of a sphere, soft objects in various orientations, maybe others. Another possibility is being able to infer the mass of an object when the sensors are used to lift an object. The deformation at that time may directly correspond to weight. Together it may be possible to do some rough object identication, such as "pointy contact surfaces with mass of 20g". Combined with steroscopic cameras to ID the object, this could give a machine learning algorithm more to work with when figuring out how to manipulate objects. You might be able to use the ability to measure slip and the known distance between the grippers to tell how soft an object is, and together with camera input, decide how fragile an object is and whether the gripper is crushing it. The force changes during a crushing motion might indicate if the object is just soft or if it is semirigid and might break. Besides gripping, you could explore pushing and rubbing objects as well. Rubbing could tell you something about surface texture, which is also related to what the object is made of. Maybe there are uses in rolling objects between the grippers also? To reorient the objects along an axis of rotation while simultaneously characterizing the nature of the object?


Seems like the tough part of this is access to the magnetizing machine. I wonder if the same effect could be achieved by placing tiny off the shelf magnets[0] in the molded part instead?

[0]https://www.kjmagnetics.com/proddetail.asp?prod=D101-N52


While this is possible, it would create stress concentrations within the elastomer and could significantly affect its durability. We saw this effect even when using larger magnetic particles as with ReSkin, https://reskin.dev. If instead we make the elastomer more rigid, it would worsen grasp stability.


I think you could 3d print TPU half shells w/ some reasonable infill (i'm sure there's one with good force transmission characteristics for this?), and then seal the magnet array inside of the two halves.


That’s really cool.

Skin is our largest organ. It’s like another vision system with high bandwidth signal but we don’t have anything close in digital world.

Even putting snow gloves heavily impairs dexterity since we can’t feel as much.


Very cool! This kind of material is potentially very interesting for biotech/lab automation tasks. Any info on surviving sterilization techniques (autoclave, ethylene oxide, clydox, etc)?


The exact soft polymer doesn't seem to matter, and the magnetic particles are protected as well.

So rather than this specific study, if there is any soft polymer that would survive sterilization techniques, you could make the skin out of that specifically for this purpose (or give the robot a glove).

So, that's a question of commercialization and product range rather than the technique itself.


One thing that I noticed from watching the first ~minute of the video: rather than simply pressing on the skin, it looks like the finger is mostly pulling the skin down over the edge. My intuition is that this is because there isn't all that much deformation from simply pressing, and so this pulling action triggers a stronger response. But I might be overinterpreting from a few seconds of video.


Please make a laundry folding robot. #aithatmatters


Shut up and take my money!!

Would that really need AI though? I guess recognizing the particular item and then deciding how to fold/hang it would be some sort of ML. Again though, that's just a bunch of if statements being labeled as AI.


There are lots of laundry-folding "graduate student robotics project" systems. Most of the videos of them are sped up 8x to make them "merely slow", and involve a rough surface to drag the material against - since they typically use one gripper; two-coordinated-grippers is still a mostly demo/research thing.


If it takes 3 days to fold a load of washing, it's still quicker than it takes me now. For commercial use it wouldn't work, but for consumers speed is not an issue.


Mine tends to go from the drier to laying out flat on top of the drier. So the fact it gets folded would be great. Every now and then I actually fold and put away and I feel like an adult. We try not to do that much around here


Very cool! Seems to me like slip detection would work better with fingerprint-like ridges molded into the surface. Maybe also combined with an accelerometer or mems microphone to sense vibration.


Yes! The sleek form factor leaves a lot of room to integrate other sensors and modalities!


How is AnySkin different from ReSkin? And what is the IP situation?


ReSkin is the predecessor to AnySkin. The novelty of replaceable skin with automatic recalibration and the skin manufacturing process seems to stand out.


Very interesting approach! Question: How do you deal with environmental noise and unexpected flux from nearby devices?


Magnetic fields drop off quickly with distance, so I would presume this noise is quite low unless you are picking up magnets or working close to large powered coils.


can make a self massaging robot. where you pressure is where the robotic arm will massage, to the strength u need.


I put in a request for one, would love to get my hands on it to integrate with our framework.


What sensor chip are they using?


they say "AnySkin uses the same 5-magnetometer circuitry as ReSkin,". and ReSkin uses Melexis MLX90393.

(Interesting fact: in MLX90393, you can customize 2 lower address bits using config pins, and there are multiple part numbers which only differ in higher address bits - so a single bus can have dozens of magnetometers attached. It's a very helpful feature, a lot of magnetometers are designed for compass use only, and thus have no support for address customization at all)


We use the MLX90393


This reminds me of velostat!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: