
Playstation's secret weapon: a nearly all-automated factory - gumby
https://asia.nikkei.com/Business/Companies/PlayStation-s-secret-weapon-a-nearly-all-automated-factory
======
zbrozek
Lots of folks here are talking about the latency and throughput, but nobody is
amazed at what the robots are doing. Plugging in cables and putting down tape
is fiddly and hard. I'm _super impressed_ that they elected to automate those
things based on cost merit. Massive kudos to the manufacturing engineers who
managed to pull that off.

~~~
vmception
I had invested in a semiconductor factory early last decade, and it had
started making bitcoin miners, it had reminding me of Zion from the Matrix
because it seemed like the first time that a machine could make another
machine which could immediately start earning a negotiable asset and then
transacting for resources.

Very surreal to think about.

~~~
AceJohnny2
There was this little game I played on Linux years ago where you played an AI
trying to reach singularity status while evading detection by humans.

[https://en.wikipedia.org/wiki/Endgame:_Singularity](https://en.wikipedia.org/wiki/Endgame:_Singularity)

Early in the game, you'd hack machines to run mechanical-turk style jobs to
earn money. Later you'd build a factory to manufacture more processors... ;)

~~~
vmception
Haha just a few years ahead of its time!

Some of the first bitcoin faucets had captchas to “reduce spam”, as in prevent
humans from getting too much free bitcoin, but really the captcha was a real
captcha that a bot was stuck on, and was just paying humans to solve them to
access some greater bounty

~~~
wallacoloo
Morally ambiguous, but I actually implemented exactly what you’re describing
back when bitcoin was at $50-100. I found some service that would pay me $1
for every 1000 captchas I solved, and then I scraped that and built a BTC
faucet site (I think I called it captchacoin.com, or captchabit.com) that
would essentially pay the user $0.50 in BTC for each captcha they solved.

It went from 0 captchas per day to 10000 captchas/day over the course of about
a month with just word-of-mouth. Then the upstream service just quit paying me
with no explanation. My accuracy rates were good. Maybe latency was high? I
don’t know why they terminated it. But I struggled to keep the users while
finding a replacement, and in the end I shut it down.

On the one hand, I’m bummed, because that thing could have mostly run itself
and made a tidy profit for someone still pursuing a degree. On the other
hand... maybe it was for the best: 90% of captcha solving work is paid for by
spammers who make the internet worse.

~~~
AceJohnny2
Have an upvote for the story, but don't pretend it was "morally ambiguous".
That was plain evil with one level of detachment, and helped enable spammers
and scammers.

~~~
vmception
cryptocurrency needed faucets and it helped adoption

~~~
voltagex_
Cryptocurrency is the biggest backwards step for the climate since CFCs.

~~~
vmception
Thats a huge misconception actually, the information you have is the amount of
energy being used, but you don't have information on the source of the energy.

Would you be willing to accept or at least corroborate that cryptocurrency
mining is one of the cleanest sectors and a boon for sustainability?

70-80% of that energy use is renewable energy or reducing pollution -
specifically hydrocarbons. So existing energy is not being wasted or
reallocated, and additional unclean energy is not being ramped up to
facilitate mining, and previously wasted energy that _was_ going into the
atmosphere is now being used.

The educated discussion is to make sure it stays this way. As nation states
are the only actors that could mine at a loss with inefficient ways.

------
hobofan
If you enjoy that you will probably also enjoy the fully-automated Valve Steam
Controller assembly line (including video)[0].

[0]: [https://www.theverge.com/2015/12/11/9890914/valves-steam-
con...](https://www.theverge.com/2015/12/11/9890914/valves-steam-controller-
assembly-line-video)

~~~
bluejellybean
This is a fantastic video, extremely well done in my opinion. I remember
fanatically watching "How It's Made", in that show they focused on more run of
the mill assembly lines for generic plastic things and usually there was a ton
of labor involved. The video linked here was really the first time I saw this
level of automation and my jaw nearly hit the floor, the technology industry
has created some incredible feats of engineering.

~~~
phjesusthatguy3
I watched How It's Made with my kids for a good number of years, and I was
just amazed that the parts that were automated worked as well as they did. It
made manufacturing look _really complicated_ and that human hands were the
most important part of the assembly line.

------
latch
In case anyone misses the link in the article, they have a graphic-rich
version:

[https://vdata.nikkei.com/en/newsgraphics/sony-
playstation/](https://vdata.nikkei.com/en/newsgraphics/sony-playstation/)

(The content is different, though both articles cover the same topic. A bit
weird)

~~~
whywhywhywhy
This article is crying out for GIFs, great pictures but you can't hype me up
about how these robots move and not show me.

------
baybal2
Fully automated assembly lines rarely make sense economically even in Europe,
and Japan.

A "lights out" factory needs a really expensive, complex widget to make, but
one which doesn't vary, or change over time much.

Game consoles are exactly like: long product life times, and can be engineered
specifically with automated lines in mind, without compromises.

Robotic factories are by no means a final solution for manufacturing.

~~~
arcticfox
> Robotic factories are by no means a final solution for manufacturing.

Robotic factories are by no means a final solution for manufacturing...Yet

~~~
zanny
Definitionally they kind of are inevitably the _final_ solution. We _will_
either totally automate all production, go extinct, or get trapped in a loop
of destroying civilization putting the survivors back to pre-industry and
having to rebuild in perpetuity. Its only a question of time scales.

------
actuator
This looks so amazing and I wasn't aware that Playstations are still
manufactured in Sony's home country itself. I guess automation helps in
keeping the manufacturing cost low.

Now I am intrigued about what high selling electronic devices are manufactured
in their home country itself and not by the ODMs/CMs in countries with cheap
labour like China.

~~~
mytailorisrich
I have a feeling that automation slowed down in many sectors when China came
"online". I think this cycle is coming to an end, and there is no comparable
pool of labour anywhere in the world so that fully automated factories are
going to be the next step (that wasn't technically possible in the early 90s
but I think we now have the technology or very close to).

~~~
PeterisP
" there is no comparable pool of labour anywhere in the world " -> this is not
fully true, as when wages in China started to rise, lots of labor offshoring
moved to SE Asia (e.g. Vietnam) and there's still a lot of potential for the
same to happen in various African countries.

~~~
mytailorisrich
There are pools of cheap-ish labour but none comparable to China. They are
much smaller and many, like Africa, are not able to take this on now.

~~~
imtringued
You made a prediction that these places will never be comparable to China.
What do you base this on? What prevents India and African countries from
establishing special economic zones that are attractive to foreign investors?

------
enginoor
If anyone is interested in factory automation I'm working on a site to catalog
different processes. A good design can often be recycled into a similar
machine.

[https://automate.engineer/](https://automate.engineer/)

I'm mostly building it for myself but I thought other people may enjoy.

~~~
strider12
wonderful website, thanks for sharing!

------
smoe
> Just a few humans were present to deal with a handful of tasks -- two to
> feed bare motherboards to the line, and two to package the finished
> consoles.

Maybe I'm just ignorant, but, especially the first task, seems rather trivial
compared to what they achieved with their robots? Even the second one, If I
remember correctly Amazon does/did this manually because every single package
is different and no robot flexible enough, but I assume in Sonys case packages
are all exactly the same. Doesn't sound a whole lot harder than plugging in
the cables.

So why wouldn't they automatize those steps as well?

~~~
aidenn0
Loading into the sort of packaging used for electronics is definitely hard to
automate. For low margin products it used to be universal that boxed items
were manufactured where labor was cheap, and that's for very simple to load
items.

Automation is improving at a rapid rate though. I've definitely seen automated
boxers for some simple to load items start to show up.

------
impalallama
I wonder if this speaker to its nature of being a Japanese article but the
tone is largely awe and pride of the accomplishment lacking the usual fear of
automation and jobs lost I see in other such articles.

>“If you keep watching them long enough, they will start to look like humans,”
said one engineer. The look in his eye is gentle like a father watching over
his children. Unlike large robots that lift heavy items, the delicate
movements of compact robots resemble those of human arms and have surprising
warmth.

~~~
smoe
I don't know if it is just cliche, but I have heard that Japanese people have
a very different relationship to robots/automatization than western countries.

Were we immediatly think Skynet, they see something that helps them.

~~~
fomine3
I'm doraemon

------
blakesterz
"The 31.4-meter line, completed in 2018, has the ability to churn out a new
console every 30 seconds."

That seems.... slow? I mean, I have no idea what I expected, but 2 a minute,
is that really fast for this type of manufacturing? How does that compare to
Xbox or maybe iPhone output? I'm sure 2 a minute is really impressive, but I
started reading that thinking something like hundreds or thousands an hour, I
guess I was naive.

~~~
LeifCarrotson
Controls engineer here - no, that's pretty normal. I'm accustomed to
automotive work, but I believe the electronics assembly lines are similar. The
flexible materials handling stuff (connecting flat flex cables? Applying
Kapton tape?) is indeed super impressive, but the timing is ordinary.

A sizeable 6-axis robot like those used here has a minimum cycle time of a
second or two for the simplest possible "move in, place the thing, move out"
operation. They're not ultralight delta or SCARA bots, they're about as fast
as a human can control their arm. Also, there's a second or two lost while the
pallets moves along and are parked at the multiple stations along the line,
you want to design the system to maximize the ratio of useful on-station time
to transition time.

The line is bottlenecked by the slowest process. You can make a slow station
run in parallel - as in the photo at [1] with the 16 stations arrayed around
the robot - but that adds a lot of complexity. I'd assume that their
requirements were to accommodate up to ~8 minutes for burn-in testing at that
station, and that by the empty stations, the software guys ended up only using
4 minutes. You can't make the whole line wait that long, so you spend the
extra effort and introduce extra complexity by buffering parts there, but now
your process has forked and needs to be merged again - really a pain for
tracking suspect parts or tools. All that matters is that you unload each
pallet before it needs to move on and replace it with a completed part.

Some stations may be doing nothing but running a 5 second electrical test. If
there's some station that has to pick up a cable, photograph it, analyze it,
contort it by rotating robot joints through large angles, plug it in, and get
out of the way, that could easily take 10 seconds. Others may have a human who
needs to keep up. Others might be heat-staking a plastic component and need
time to blow on the stake until it cools enough to be unclamped. Others might
have multi-headed tooling and can connect several different parts all at once.
30 seconds is a nice middling part-to-part value if you've got a few complex
operations, a few human operations, and don't want heavy robots slinging your
expensive parts around at high speed. I wouldn't have been surprised at a
10-second cycle time, but 30 seconds is still reasonable.

A final consideration is that you don't want an entire factory dependent on a
single line. If you want to produce thousands of parts an hour, it's much,
much better to have 10 identical lines that work at a 30-second cycle time
than a single line frenetically thrashing to put out out parts in 3 seconds.
If one line goes down, you still have 90% capacity, and can probably catch
back up. If your single line goes down, you're in big trouble.

[1]: [https://s3-ap-northeast-1.amazonaws.com/psh-ex-
ftnikkei-3937...](https://s3-ap-northeast-1.amazonaws.com/psh-ex-
ftnikkei-3937bb4/images/2/1/7/9/28039712-10-eng-
GB/Cropped-1593718166%E9%9B%BB%E5%AD%90%E7%89%88%E8%BF%BD%E5%8A%A0%EF%BC%89%E3%83%97%E3%83%AC%E3%82%B9%E3%83%86%E3%81%AE%E8%A3%8F%E5%81%B4%E3%80%81%E9%9B%BB%E5%AD%90%E7%89%88%E7%94%A8%E3%81%AE%E8%BF%BD%E5%8A%A0%E5%86%99%E7%9C%9F%E2%91%A320200703034521471_Data.jpg)

~~~
blackrock
Can you, or someone else, share your experience in becoming a Controls
Engineer?

Is your educational background in Mechanical Engineering or Electrical
Engineering?

I’ve always found programming robots to be fascinating. And especially when
building a robot to build other robots. I’m sure others will be interested in
learning of this as well.

~~~
contingencies
Most of the available training for this stuff is vendor-specific, vendor-
provided and really expensive to undertake. Indeed, many vendors have solution
provider networks who will prefer to do this programming for you, because it
is so unintuitive/dangerous/error-prone.

You can, however, learn the control systems by grabbing and reading the
manuals. Look for manufacturers then seek out manuals. Many are online though
they are often not the latest edition as some manufacturers now seek to keep
this internal to paying customers.

All the systems I have seen use G-Code variants with manufacturer specific
extensions. Many now offer GUIs to help with spatial planning, multi-machine
integration, common code generation, etc. However, at the end of the day it is
still nearly all about generating/writing/modifying/replaying stored G-Code.

------
velox_io
Electronics manufacturing has been moving to more and more automation for
years. Humans don't place components on PCBs anymore (well, most components).
it's done by machines that look more like a gatling gun than human arm. It's
quite stunning how small the components are these days.

Gamers Nexus: How Motherboards Are Made (2019) | Taiwan Automated Factory
Tour, ft. Gigabyte
[https://www.youtube.com/watch?v=cnAFTMaS5R0](https://www.youtube.com/watch?v=cnAFTMaS5R0)

------
Iv
I am happy that China reached the point where their average wage is high
enough that we can finally resume the manufacturing automation effort that
kinda stalled since the 90.

------
chinathrow
More graphics directly at [https://vdata.nikkei.com/en/newsgraphics/sony-
playstation/](https://vdata.nikkei.com/en/newsgraphics/sony-playstation/)

~~~
vmception
That's a great sub-site. This is such a good year for exclusives! 2020 was
always going to be epic for that, and I haven't been disappointed.

------
jariel
Powerful example of operational innovation, and that there is so much untapped
opportunity we are not leveraging in so many areas.

A company like Apple may not have the wherewithal or impetus to do something
like this. When they need manufacturing, they are resigned to the traditional
approach: China. Since labour is cheap there, and they just may not be
thinking in these terms and may not have the wherewithal either ... the cost-
effective solution invariable involves high labour intensity.

It may very well take factories like this to inspire other organisations to
have the necessary 'Eureka!' moment to grasp that they too, might be better
off 'fully automated' as well.

Note the eye-watering scale of the business however, quoted at nearly $100
Billion in sales, which is rather a lot of money indicating that it may take a
product with a very long life cycle wherein there are considerable profits for
this to be feasible.

Looking at this, one has to wonder why Western nations are not more keen to
duplicate.

~~~
owenmarshall
> Note the eye-watering scale of the business however, quoted at nearly $100
> Billion in sales, which is rather a lot of money indicating that it may take
> a product with a very long life cycle wherein there are considerable profits
> for this to be feasible.

You mention Apple. The iPhone SE 2 is basically "iPhone 11 guts in an iPhone 8
shell" – and as I understand it Apple went to software to handle things like
the camera so they could keep the older generation optics and not change the
body. That type of tick/tock cadence would support this nicely.

------
rusk
God I love my PS4 - it has to be the peak of this class of machine. I can't
imagine how it could be improved upon without going beyond the "box under the
telly format". Incredible piece of machinery. I can't imagine the PS5 will be
much more than an upgrade.

In my opinion the second best console is the PS2. I love Nintendo but they're
more like toys than hard core gaming and entertainment machines. PS2 up until
PS4 was pretty much the pinnacle, but I think the way PS4 improved upon it is
the development platform, and the online experience (which even so does
require some improvement). I still play games on the PS2 from time to time and
they still feel viable as offline single-player experiences and I could
imagine development for the PS4 could continue for 20 years or so and still be
producing great and relevant games.

Where to from here? VR is one obvious route but may I suggest a HN-friendly
alternative? Hows about selling a more open system that allows end-users to
tinker and contribute themselves? I think in order for this to be viable in
terms of protecting the profitability and stability of the ecosystem there are
some business and technical challenges but I would love to see them take this
direction. They've dabbled before with the Yaroze system and Playstation Linux
… both had their business operational issues but hardly insurmountable?

EDIT - can't respond below any more so I just have "one more thing" to say:

> PS4 is famous for being the cheapest sony home console relatively speaking

and what exactly is the problem with that? This is a feature not a bug. The
PS3 was a novel system with exotic hardware but it suffered from some very
serious flaws. It's reliance on proprietary technologies meant it was harder
for developers to work with. It never took off as a result and is little more
than a footnote in the lineage's history. A learning experience if you will.

Surely an extension of making something easier for professional developers to
get to grips with is making something that hobby developers (and other classes
of creative) can get to grips with?

~~~
easytiger
> it has to be the peak of this class of machine.

I'm sure phones had better GPUs 3-4 years ago than the PS4, and more RAM

When i enter the notification quick tab it takes 30-40 seconds to show the
message I've received, then another 10-15 to accept it.

Currently one single game, COD:MW consumes the entirety of the space on the
device. From launch it wasn't ever going to be enough. You need to keep <x>
space free because the update->patch->relink process needs to duplicate the
game.

The menu system is also insane and makes no sense.

The PS4 Store App has to be one of the most poor implementations of something
so simple i have ever seen. And despite enabling every level of security
possible i still get DM spam that could be detected by a python 1 liner.

On the COD:MW lobby screen alone the console appears to turn into a 747 on
take off.

Games are frame locked at 60fps - awful by PC standards and in a world of
crossplay, a disadvantage. Performance limitations also limit FOV settings etc
etc.

The controller battery life is terrible and it doesn't have bluetooth audio.

> may I suggest a HN-friendly alternative? Hows about selling a more open
> system that allows end-users to tinker and contribute themselves?

Could call it a PC or something.

~~~
whywhywhywhy
> I'm sure phones had better GPUs 3-4 years ago than the PS4, and more RAM

Means nothing though, where is the software? These supposedly dated and under-
powered machines are still where the production of some of humanities not only
most ambitious art of the last 100 years but most profitable art is being
produced.

------
duxup
I've always wondered if you could automate a sort of standard assembly system
with a certain amount of adjustments within some parameters (device
dimensions, how things are connected)... and pump out any number of different
electronics all on the same assembly line / factory using the same robots?

Granted that probably already is a thing, or isn't for good reason.

~~~
mkroman
As you might imagine, this is already the case in places like Shenzhen -
there's off the shelf “building blocks” for machines - not necessarily to any
written standard, though. In this[1] Strange Parts video he mentions it. If
you watch some of his other videos you'll start to notice that many of the
machines actually look similar - I highly recommend watching them, they're
great.

[1]:
[https://youtu.be/PZBQzLfCKpw?t=1019](https://youtu.be/PZBQzLfCKpw?t=1019)

~~~
blackrock
That video was incredible. The machines that makes the product, the lithium
ion battery for your iPhone, is this super intricate process, that outputs the
final product. And everything is made from the raw materials.

Now, this is what I call a true fabrication process.

~~~
6nf
'Industrial lego' as some call it. Get all right bits and pieces, put them
together and you have a production line. Every piece comes with detailed
specifications, performance curves, maintenance instructions etc - which means
an industrial part is more expensive than consumer grade stuff doing the same
task. But when you're building a big machine you don't want unplanned
downtime.

------
everseason
So why do most of the PS4s I find on ebay all say "Made in China"? Are the
parts made in China and then assembled in Japan at the Kisarazu plant? Or are
they only making PS4 units in Japan for the domestic Japanese market...and
making the rest of the world's units in China?

~~~
cbmuser
Sony uses multiple OEMs, including Sony EMCS in Japan and Foxconn in China.

Usually, the first batches of a Sony product are made in Japan or the ones
sold domestically.

If you buy an early PlayStation 1, 2, 3 or 4 in Japan, they're usually all
Made in Japan.

------
partingshots
A good baseline to measure how fast these are would be to look at comparative
Xbox production times.

------
LockAndLol
Taking out the human element. China knows it's coming and it's a big reason
they're "investing" in poor countries. They won't have the upper hand on cheap
labor soon.

------
mips_avatar
I'm so fascinated by lights out "autofac" manufacturing. Instead of amazon
warehouses along I-90 let's just have "autofac" clusters that can make most
anything.

------
hinkley
Also the origin story for the Portal games.

------
KaoruAoiShiho
This is so cool, but how long till they automate the packaging too?

~~~
ashtonkem
At only 2 workers per line to package them, would it be worth it to automate
that role? Seems like they might've min/maxed the automation already.

~~~
blaser-waffle
There is also some (if limited) utility in having someone on the line doing
stuff, watching stuff, and serving as a form of QC. You can hang in the
control room watching the latest, greatest SCADA software run things, but it's
useful to have some eyes-on as the line runs.

------
shmerl
When will Sony start supporting Vulkan on PS?

~~~
bigcheesegs
I seriously doubt it. The PS4 has a custom graphics API which directly creates
command buffers in the GPUs native format. There's not really much of a
graphics driver at all other than what's necessary to isolate the game from
the OS.

~~~
shmerl
Why not though? It makes it easier for developers, instead of the need to
address yet another proprietary API. Simplifying things has value.

That API was created by Sony when something like Vulkan didn't exist yet, but
today there is no good reason _not to_ support Vulkan.

------
dschuetz
That indeed is impressive.

------
taytus
Nah. They are killing it because of amazing exclusives. That's all.

~~~
KorfmannArno
Aha, color me confused. Which would be those?

~~~
taytus
The Last of Us, Uncharted, God of War, Horizon: Zero Dawn

~~~
KorfmannArno
It's all senseless shooting or close combat...

------
6510
nice video

------
throwaway888abc
Impressive

