
AI-powered Lego sorting machine built with Lego bricks - Gys
https://www.brothers-brick.com/2019/12/06/this-fan-created-the-ultimate-lego-brick-sorting-machine-powered-by-ai-exclusive-feature/
======
9nGQluzmnq3M
First AI-powered Lego-sorting machine, eh? jacquesm@ would like to have a word
with you.

[http://jacquesmattheij.com/sorting-two-metric-tons-of-
lego/](http://jacquesmattheij.com/sorting-two-metric-tons-of-lego/)

~~~
sambe
He mentions this in the video which is titled "World's first UNIVERSAL lego
sorting machine" \- emphasis mine. It seems the emphasis was shifted to AI by
someone else.

~~~
jacquesm
And mine is just as universal, and likely also not the first but the first one
that got wide visibility.

This guy did a great job of addressing some of the shortcomings of my machine,
specifically in the number of output bins (18 instead of 12), but it is a
regression in other ways (built out of Lego so it will break frequently,
mechanical part movement from the belts, which will lead to frequent jams and
will wear as well as that it is slow and more timing sensitive).

All in all a nice effort and very nice video production but not nearly as much
of an improvement that would make it reproducible/mass marketable.

To _properly_ build a machine like this will take a significant budget and a
lot of hours behind cad software to make it out of industrial parts rather
than Lego. I you want to make something mass marketable then you have a very
long road ahead of you.

~~~
whichfawkes
I believe what he means by "universal" is the ability for the machine to
recognize parts it has never physically encountered before (since it's trained
on 3d models).

It seems like increasing the number output bins shouldn't really matter much,
since with either machine you could just reprogram it and run parts through
multiple times, right?

~~~
jacquesm
You can train it on three D models but that's just a stand-in for the parts as
though you have them. But - and this is quite important - if you want to
achieve some efficiency with the 38000 or so parts that Lego has made over the
years then you're going to have to cut a couple of corners and one of those is
to give some more weight to the parts that you do encounter and to train your
model to the point that it will come up with the right category even if it has
never seen the part _at all_ , virtually or IRL.

That's roughly what I got it to with showing it samples of wheels but not all
wheels and it detected wheels that it had never seen before just fine and so
on. This is a lot easier than it might seem, you'll get a good match on a
class but no good hit for any of the subclasses: that's a new part right
there.

And yes, that's exactly how I did it, multiple passes. But more bins is
certainly useful, I just ran out of room, the machine was already getting
quite large, and the belt and frame I used was fixed in size and dictated a
lot of the dimensions of the machine (it was a re-purposed running trainer).

~~~
JustASquid
Hey, Daniel West here, the creator.

First of all, as I mention in the video, this project is inspired by the
machines built by akiyuky and jacquesm@. So I'm certainly not claiming this is
the world's first LEGO sorting machine!

In the video I try and focus on the unique aspect of this machine, which is
the _Universal_ part. Because the CNN is trained on synthetic data, the
machine is able to recognize any LEGO part that has ever been produced. This
is with the notable exception of ignoring printed parts, mold variations, and
oversized parts, as well as parts that are not present in the LDraw part
library. These restrictions bring the tens of thousands of part categories
down to just under 3000. Accurately classifying 3000 categories is
prohibitively difficult using manual labelling alone, simply because it's very
hard to get physical access to all these different parts (not to mention the
labelling process itself is quite laborious).

Is it overkill? Totally. But one goal of this project is to be a testbed for
the use of synthetic data in a real-world classification system, and I'm quite
happy with how well it ended up working.

I have produced another video explaining the AI system in detail:
[https://youtu.be/-UGl0ZOCgwQ](https://youtu.be/-UGl0ZOCgwQ)

It has also been noted that the machine would not be mass marketable in its
current state. That is totally accurate! This machine should be considered a
proof-of-concept/prototype and I have no plans to try and create a mass-
produced version.

~~~
jacquesm
Very neat project, congratulations on seeing it through. Note that being able
to sort down to single parts is less useful than it may seem simply because of
the number of passes required and that you'll likely reach your 'universal'
stage a lot earlier simply because neural nets are good at classifying stuff
they've never seen before as long as enough of the bits and pieces match.

How long does it run between fixes? How many kg of lego have you put through
it?

Beware of _lots_ people that will bug you to give them all your work, to this
day not a week goes by that someone contacts me with a thinly veiled request
to get a copy of the hard parts 'just for their personal curiosity' and then
when you search for a bit on Google you'll find them associated with large
parts resellers.

Highly annoying.

Best of luck with your project, curious about what you'll do next with it.

~~~
JustASquid
Thank you Jacques! Indeed, for practical sorting purposes going down to
individual parts is less useful. But there are intriguing possibilities, like
for example using it to keep track of your collection, or extracting sets from
a collection.

The machine is quite robust to jams so long as you're careful not to insert
oversized parts. That being said, I'd say there's a problem every 30-60
minutes, so certainly not something you could leave overnight. Running for
hours and hours means that it will eventually grind itself into dust anyway
(I've already burned out a motor!)

I'll probably end up sharing everything one way or another. I'm sure you'll
agree that any real commercial application of something like this is pretty
far-fetched so I'm more than happy to just treat this as a contribution to the
body of domain randomization application research.

------
Zobat
Is it just me that would like this machine to be used to find parts for a
specific model/box. We've tried to (re)build my sons police station a number
of times and it would be perfect to have a machine that goes through the box
of lego parts and spits out just the parts we need.

------
zackmorris
I would love to work on something similar for sorting plastics for recycling.
I wish there was an organization like Uber for organizing groups to do this in
every city, along with finding funding to pay developers (>
$100/developer_day). I would devote considerable effort to this full time if
there are any angels out there or people who know anyone that could raise
funds.

Also, does anyone know of a group working to improve the recycling symbol?
It's too small/shallow to even be read by humans, much less machines, on many
plastic containers. Short of that, we could break down the task into 1)
isolating the symbol on the container and 2) reading the symbol maybe with
extra wavelengths like infrared/ultraviolet or their combination with visible
light.

Here in Boise, we have regular recycling that only accepts types 1 and 2, then
they make participants sort 4, 5, 6, and 7 into an orange bag that gets sent
to Nevada to be turned into diesel fuel. I see a ton of recycling in the trash
because people aren't motivated to sort it manually:

EDIT: Added JPG: [https://imgur.com/a/puSmAzO](https://imgur.com/a/puSmAzO)

PDF (old):
[https://www.cityofboise.org/media/1632/detailedplasticsguide...](https://www.cityofboise.org/media/1632/detailedplasticsguide-7-2018.pdf)

~~~
bargl
I have thought about this a bit. It would be awesome if there was a camera
next to the recycling bin that could recognize the container and give advice
on what to do with the material and any rules on it. Such as cardboard can be
recycled if it's clean but composted if it has any grease. Some places don't
have compost so there it'd say to put it in the trash.

It would be nice just to _know_ what to do with specific materials rather than
guessing half the time.

~~~
zackmorris
Oh that's a good idea. I was thinking that there should be an open source
object recognizer that can sort any material (kind of like a 3D printer). But
having cameras on recycle bins would be great. Used cell phones could do it.
There are probably unlimited amounts available for $5-10 each:

[https://www.ebay.com/sch/i.html?_nkw=lot+cell+phone](https://www.ebay.com/sch/i.html?_nkw=lot+cell+phone)

Also image recognition has been a solved problem since about 2010, so any
roughly 1 GHz processor can do it in under a second for something like a
plastics recycling symbol.

I just can't believe that there is no money available for projects like these.
I was thinking about it just now at the gym and I'd like to see society move
away from donating time at a personal level, and towards organizational
donating where the workers involved are paid the going rate on their labor. So
instead of working a full-time job and being expected to put in a couple
nights and weekends per month on volunteering, we'd instead donate money to
foundations that could go deep on problems with real backing. It seems to me
that this would also be a good fit with open source in general. These problems
are so much easier than the yak shaving we do at work that we could solve most
of them in a matter of a year or two.

------
altcognito
Not bad! I'm curious what the accuracy is for parts. I'll look further than
this short article. (edit: on follow up, not quite the level of detail I was
hoping for, I'll keep an eye out)

I took a VERY similar approach for generalized universal lego sorter and got
to about 120 classes of parts recognizing acceptably well for about 70 of the
classes. I used Mathias' articles as a guide.

I was picking and choosing physically varied lego to find difficult pieces to
sort. Beveled pieces tended to be the worst. Small pieces hid important
information maybe? I suspect any solution will require mirrors to get better
information.

I also got discouraged because I wanted to sort into about 2-300 different
"buckets" and found the mechanical piece of that intimidating and got
distracted by some other fun project. Maybe I'll return to it.

 _edit_ : I followed up by reading the longer linked article. I can't believe
how similar the software the two of us wrote was. There are some labeling
enhancements that I were probably the next step that I appreciate he took.

~~~
danimal88
Just a though on the 200-300 buckets - perhaps you could create buckets in a
grid and then just have the grid or dropper move around - once you know which
bin is most appropriate for the part, the process of placing in the grid at
that point seems relatively simple with just X and Y movement :)

~~~
altcognito
Yeah, this type of setup is fairly reasonable but slow. I've considered
droppers into banks of containers, but I haven't really gotten that far into
it yet.

------
viggity
When he was talking about the sim-to-real problem it reminded me of a talk I
went to on neural networks. The presenter's company identifies retinopathy
(diabetes damages the retina). When they first were training the CNN, they
only jpg images. However a clinic used slightly different equipment and
uploaded pngs and it caused it to not work at all (false positives and false
negatives).

------
esquire_900
Very cool project from a technical point of view, but I can't resist thinking
how unnecessary this is in the real world. I remember a blog post of one of
those first sorters, which ended with the guy owning a huge stack of sorted
legos and no buyers.

Well done nonetheless, seems to work pretty smooth (though I see a lot of
mixed materials in the buckets?).

~~~
turtlebits
IME, the ecosystem for buying Lego is not very mature.

I tried to buy a bunch of 44 specific lego pieces (the parts were listed from
a book and fairly common) through Brickseek and couldn't do it.

Brickseek has a "Easy Buy" where you can add items to your list and have it
find sellers, but it didn't work at all for me. Manually trying to find each
piece is not worth the effort.

~~~
danimal88
It would seem then that the real business opportunity here is probably in
reselling used legos on an as needed basis.

You buy or just get donated legos from people that no longer use them and then
you can sell custom sets for people by running through the semi sorted bins
and creating the order. So you don't want to sell the machine but rather, sell
Legos. Perhaps the main issue is whether resale rights are allowed but seems
solvable.

------
thomasfl
It must feel empowering to be able to turn of the shelf toys for children into
state of the art sorting machine. Hardware is still much harder to do than
software projects. Lego lowers the barrier for many people.

------
moonbug
just quit it with the "AI" adjective already.

~~~
jacquesm
Well, for once an ML label would be appropriate.

~~~
DonHopkins
It's not real human-level AI until the machine can truly experience what it is
to step on a lego with a bare foot.

~~~
jacquesm
Fancy meeting you here ;)

It's been a subject of some research and here is the most damning piece of
evidence:

[https://9gag.com/gag/aBWA6LP/this-is-what-legos-do-when-
were...](https://9gag.com/gag/aBWA6LP/this-is-what-legos-do-when-were-not-
looking)

~~~
DonHopkins
For safety, an AI Lego sorter should have a whole fleet of these:

[https://robots-blog.com/2018/03/21/finally-never-again-
step-...](https://robots-blog.com/2018/03/21/finally-never-again-step-
barefoot-on-a-lego-brick-in-the-night-leguino-is-now-on-kickstarter/)

