
Using AI and AR to improve ArcellorMittal's production process at a Hackathon - netb
https://lab.onebonsai.com/arcelormittals-hack4steel-hackaton-43d9a0338a4
======
peterbonney
This is a really cool piece of engineering, but I'm bothered by the idea that
the going rate for a huge multinational corporation to get a prototype of a
potentially extremely valuable technology is less than 10k of prize money [1],
plus a "mini-drone race" and "snacks and food".

Maybe it's just me, but I find this type of corporate-sponsored hackathon
exploitative and distastefull. We're not talking about an event that is
producing a social good in an area that is otherwise underinvested. This is
purely profit-motivated, and ArcelorMittal can afford to pay a lot more for a
prototype like this.

1\. [https://www.eventbrite.nl/e/the-arcelormittal-challenge-
hack...](https://www.eventbrite.nl/e/the-arcelormittal-challenge-hack4steel-
tickets-34228761146)

~~~
onebonsai
Thanks for your view. However, it is important to see this event also for what
it is. It is a way for Arcelor to think out-of-the-box, and get input from
people that are not directly linked to their business. Secondly, it is a way
for students to get a glimps of what the corporate projects and challenges
are.

Additionally, for us, as professionals, it is a way to show, although in a
very rough fashion, what is possible, and where we can help. We saw this event
more as a team building than work.

The result we got at the hackathon is a very rough prototype, not a finished
product. In no way, this can directly be implemented on the floor. As such,
input from experts is still required, either us, or other experts in the
field.

Therefore, these hackathons are useful, in my opinion, not only to the
company, but also the participants :)

~~~
corford
Really agree with this. Not everything is about money, if it was we wouldn't
have squid, postfix, apache, nodejs, python and the millions of other open
source software that I and everyone else here relies on daily. Ask the
maintainer of GnuPG if he was ever given $10K by a company (prior to the Linux
Foundation's Core Infrastructure Initiative grant).

~~~
peterbonney
This is the distinction I was getting at - something like GnuPG provides
tremendous value to _society as a whole_. Saving ArcelorMittal some money in
its manufacturing process doesn't benefit anyone except ArcelorMittal's
executives and stockholders.

So I stand by my point. There's no doubt the participants got a lot out of
this Hackathon, but I'd wager that ArcelorMittal got significantly more.

~~~
corford
I see your point but I assume the producers of the hack own the rights and IP
to their work i.e. there's nothing stopping them releasing what they did as
open source and building a community around it (at least that's been the case
with every hackathon I've ever participated in).

If the above assumption is true then I don't see the problem. ArcelorMittal
get some buzz and maybe some ideas to pursue, the hackers get a fully paid fun
weekend and a little cash for their efforts, society potentially benefits from
the seeds that were planted.

------
onebonsai
Hey!

Thanks guys for your comments! I'm the writer of the article (Evarest).

Just a bit of an update in the mean while :) Our AI team is now busy writing a
bit more detail on the actual AI part of the project. The project used:

Random Forested Height Data Deep Learning Models

We trained on the entire dataset of 60.000+ elements, but did pre-processing
on the data. I am not placed to answer detailed questions on the AI part, but
we will update the article soon with more details.

Note that this is not a perfect result, nor that we claim to be all-knowing or
have all the answers. This is a result from a hackathon, which involved
working in suboptimal conditions under heavy time constraints.

As for the AR part, we did not have access to AR helmets. We are aware that
not all the data that we showed would work on an AR helmet (currently).
However, this tech is totally feasible in the short to medium term. Also, it
is completely feasible on tablet or similar.

The concept of the hackathon was to show the potential of these technologies,
and to integrate the AR and AI, which I find we succeeded in.

~~~
corford
Great hack. Really enjoyed reading this! Noob question: the weld as shown in
the AR video looks really detailed (fissures, deformations etc). Was that
rendered by you using the heightmap data alone or did you have access to top
and bottom images of the welds as part of the training data?

~~~
loa_in_
I think it was data from the laser scanner as it was monochromatic (w/
shading)

~~~
corford
Thanks!

------
evarest
Thanks all for your great comments! For some reason, I cannot answer all your
questions as the system blocks me from posting, probably because I did not
post the original link here... Therefore, I am going to answer as much as
possible in one post :)

I totally understand the feelings of some about "corporate hackathons". While
it is valid that companies get probably more from these hackathons than us, I
want to stress the following:

\- We had the opportunity to work on a challenge with a huge amount of data
that I would not otherwise be able to get

\- We were able to work with great guys I would not have met otherwise

\- We stretched ourselves to see if we could combine AI and AR, which would
probably not have happened otherwise

\- The implementation of anything built during the weekend would takes months
for anyone, even with the code we produced

\- This is a learning experience for all parties

The data we used was based on input from a laser scanner. The scanner provides
a height map, which would be shown as a colored PNG. We also had point clouds,
but only for a few months, so not much use in AI... We used the PNG to
generate a bump and normal map to show in AR.

Thanks again and hope that you had fun reading it :)

------
onebonsai
UPDATE: Thank you for your comments and remarks !

Based on the inputs of the community, we rewrote the AI part of the article
(see section 'Building the AI Assistant'). More details are available now.

\--> [https://lab.onebonsai.com/arcelormittals-hack4steel-
hackaton...](https://lab.onebonsai.com/arcelormittals-hack4steel-
hackaton-43d9a0338a4)

~~~
specialist
I suggest changing "conjest" to "condense".

------
ambicapter
Damn, how many years of experience do you need to be able to come up with
workable solutions as fast as this?

~~~
MariuszGalus
6 months, here's why: They random forested height data, everyone random
forests these days. It's quick and dirty and yields good accuracy. Not only
that, everyone seems to support a built in Random Forest training model,
programs from pandas/numpy/scikit to R has it built in. Height data seems the
easiest to go by, to short and it has a kink that may cause cracking in the
future as the metal oxidizes. What were their 3 models? Who knows? But their
idea of combining 3 models, that 3 different teams have made is standard in
all DataScience competitions. This is called ensembling. ArToolkit for Unreal
Engine, someone else made that and they probably just connected a bunch of
blocks together with Unreal's script engine. Honestly, 6 months max and you
can do this in a day yourself.

~~~
Drdrdrq
Multiple years, here's why: you need to decide what will work and what will
not, as you don't have time to go the wrong route. And that takes experience
which it would be difficult to gain in 6 months.

------
FLUX-YOU
I just got into Unreal Engine with the VR template project, which includes
grabbing/teleportation functionality by default. It's been really easy to get
into and if you have a headset, I'd really recommend trying it out if you've
had a VR development itch.

~~~
onebonsai
Indeed, thanks for the comment! Unreal Engine is a great engine to quickly
build prototypes in. We are used to work with the included VR Template of
Unreal, and it most of the times works great.

Also for more serious projects, Unreal did not let us down.

------
kerberos84
How much money will Arcelor earn from this piece of Software? You just
implemented it for free!

~~~
rmorm
Hey ! I am one of the AI guy of the team. I agree that if the solution gets
implemented on the lines, they will probably make more net money out of it
than what was spent for the challenge (which is more than 8.5k€ and some
pizzas as said in other comments BTW, the challenge was probably more
expensive to organize). However, let's be realistic, they will have to put a
lot more money into this before they get a viable product to insert in their
production lines. The solution we have built is way too simple regarding
actual operational constraints (e.g.: operator safety).

~~~
wenc
This. A lot of the other commenters have no idea what it takes to get software
productionized in an industrial company.

This is more of a proof-of-concept (and a very impressive one,
congratulations), but it will take much more development and resources to get
it working on the production line.

~~~
kerberos84
I think nobody here thinks that this piece of code from weekend will be used
on Monday on the band. The hardest part is coming up with a solution and that
company got it almost for free. I am pretty sure that this company will even
aplly for a patent for this solution! The team should at least get a
remuneration if this "proof-of-concept" implemented and will actually be used.

------
Odjansse
Can you elaborate on: "We received feedback from the AI subteam, and
incorporated methods to show parts of the weld that were prone to breaking"?
What techniques did they use to extract this information from the network?

~~~
Simoncarbo
Hi! I'm from the AI subteam. We actually didn't have time to implement this
part of the solution. Our idea was to use this library:
[https://github.com/raghakot/keras-vis](https://github.com/raghakot/keras-vis)

------
nartam11
Awesome work! I understand its a proof of concept for a hackathon but what's
the value of AR here? If AI can indicate a fault with a high success rate how
does AR help this situation?

~~~
evarest
Thanks! Good question, the AI part is clear, it will indicate if there is a
big chance of break. The AR will help the operator decide without going to the
actual break whether the AI is correct.

The AI only suggests, does not decide on its own. As such, an operator will be
able to see the break in full detail (the top and bottom and up close) to
better be able to decide whether to continue production, or do a reweld.

Also, sometimes the AI will not be able to provide a good input, as new
materials might be welded together and no data is available for them. In that
case, again it's up to the operator to actually give the go or no-go.

The AI will inform the operator after about 2 seconds, the operator then still
has about 8 seconds to finalize his decision.

The current prototype is not really usable, as it has some big usability
issues. We might work further on improving this prototype with the organizer,
depending on their interest.

------
option_greek
I didn't understand where the AR was overlaid. Was it on the glasses on which
the operator was wearing or on a LED monitor ?

~~~
mring33621
They did not have VR glasses, so they simulated them using a helmet-mounted
webcam and a monitor. The AR weld risk profile (my words) is overlaid on the
captured video images. See the article's video with caption 'Screen-capture of
the AR experience...'

------
webbrahmin
Can you please elaborate on the deep learning part. What kind of network dis
you use?

~~~
Simoncarbo
We used two convolutional neural networks, one for each side of the weld.
Features extracted by these nets are then concatenated, as well as the meta
data. Two fully connected layers are then applied on this vector, giving the
final prediction.

Everything was trained on the raw images resized to 20x150 pixels.

------
tshanmu
extremely inspiring!!

