
OpenAI's blog post about "solving the Rubik's cube" and what they actually did - bratao
https://twitter.com/GaryMarcus/status/1185679169360809984
======
amayne
How is OpenAI misleading? The entire post on OpenAI is about the physics of
the problem (different sized cubes, materials, etc.)

When I saw the press release I understood the demonstration was about hand
dexterity and not trying to use AI to solve a Rubik’s Cube pattern. That would
be overkill IMHO. You don’t need a neural net to solve it and I never thought
OpenAI was trying to mislead.

Side note: One of the commenters on the Twitter thread referred to Marcus as
the James Randi of AI in jest. I worked for Randi for several years handling
the Million-Dollar Paranormal Challenge and investigating unusual claims. I
can tell you a lot about misleading claims...

~~~
mindgam3
> How is OpenAI misleading? ...When I saw the press release I understood the
> demonstration was about hand dexterity and not trying to use AI to solve a
> Rubik’s Cube pattern.

Uh... the headline of the article was literally "Solving Rubik’s Cube with a
Robot Hand".

Kudos to you for apparently reading past the headline and understanding that
the demo was actually about hand dexterity. But come on. The average layman
reading the headline is going to believe that what's newsworthy is that OpenAI
solved a rubik's cube.

If OpenAI didn't intend for that to be the case, they should have used
different words. For example, "Using a neural net to achieve a breakthrough in
robot hand dexterity" would actually describe what the demo is about.

Unfortunately that is about 1000x less interesting than "AI robot solves rubik
cube". Which is why OpenAI didn't choose that headline, and why people like
Gary Marcus are criticizing them for being misleading.

~~~
paulsutter
More like 1/1000x, there are a bunch of rubik’s cube solvers on github.

Dexterity to manipulate a Rubik’s cube is really incredible, especially though
the entire sequence of solving it. It’s a very well-chosen dexterity
challenge.

This whole criticism is bizarre

~~~
thaumasiotes
The claim "I solved a Rubik's cube using a computer" is totally uninteresting.

The claim "I solved a Rubik's cube using a neural network" is different, and
much more interesting than the first claim.

~~~
solveit
It really isn't. It's quite obviously possible and not worth the effort if you
happen to know anything about Rubik's cubes and neural networks.

In fact, I would be much more interested in "I solved a Rubik's cube using a
computer" because you can then talk about the mathematics of a Rubik's cube
(presumably the algorithm used is a human-comprehensible one), while for "I
solved a Rubik's cube using a neural network" the only sensible question is
"and how badly did you have to overfit to do that?"

~~~
thaumasiotes
You can assume the "computer" solution is overfitted, too. There's no reason
to, because general methods are well-known, but it's even easier to just
hardcode a cube and the list of moves that solves it than it is to implement
one of those general methods.

Why assume that "I solved a Rubik's cube using a neural network" guarantees
that I cheated?

------
minimaxir
Relevant thread on /r/MachineLearning:
[https://www.reddit.com/r/MachineLearning/comments/dkd4vz/d_g...](https://www.reddit.com/r/MachineLearning/comments/dkd4vz/d_gary_marcus_tweet_on_openai_still_has_not/)

Greg Brockman commented there:

> We ping journalists to ask them to correct factual errors in reporting when
> we see them (though they may not always agree with our corrections). For
> example, the Washington Post article
> ([https://www.washingtonpost.com/technology/2019/10/18/this-
> ro...](https://www.washingtonpost.com/technology/2019/10/18/this-robotic-
> hand-learned-solve-rubiks-cube-its-own-just-like-human/)) feels misleading,
> so we've emailed them and linked them to the relevant sections in our blog
> post (namely, that we use Kociemba's algorithm as you mention).

------
Animats
If they had to instrument the cube internally, that makes the result much less
interesting. And if it's only succeeding 20% of the time, that's not good
either.

Robot manipulation is hard. There are lots of systems that work some of the
time. Few work well enough in an uncontrolled environment to be useful. Amazon
is still looking for a robot picking system, and nothing works well enough
yet.

~~~
gdb
(I work at OpenAI.)

We also have results with an uninstrumented cube (as described in section 7 in
the paper, or "Behind the scenes: Rubik's Cube prototypes" in the blog post),
which are slightly weaker (see Table 6 in the paper). The 20% number is an
example of critics cherry-picking their facts — the success rate is 60% under
normal conditions, but 20% with a maximally-scrambled cube (which would happen
randomly with probability less than 10^-20).

Also note: success here means that the robot was able to perfectly unscramble
the cube — which requires perfectly performing up to 100 moves — without
dropping it once. What it means, in practical terms, is that you need to wait
for a long time in order to witness even a single failure. If you pick up the
cube and place it back in the hand, it'll get right back to solving.

Note that like with OpenAI Five, the success rate is more of a function of how
far we've had time to push the system than something fundamental. We're not
building commercial robots; we're in the business of making new AI
breakthroughs. So the next step for our robotics team will be finding a new
task that feels impossible today, and see what it takes to make it no longer
feel impossible.

~~~
gus_massa
From the blog post:

> _Our method currently solves the Rubik’s Cube 20% of the time when applying
> a maximally difficult scramble that requires 26 face rotations. For simpler
> scrambles that require 15 rotations to undo, the success rate is 60%._

And looking at the data in [http://cube20.org/qtm/](http://cube20.org/qtm/) ,
with a random cube, the probability to have a maximal-scrambled cube that
needs 26 quarter-turns is 10^-20, but most (~75%) of the random cubes need 20
or 21 quarter-turns. Most of the algorithms don't use the most efficient path
to solve the cube, so if the best path has 20 steps, the actual path will have
a hundred or more steps.

To solve the cube in 15 steps, it must start as almost solved. It's not what
people usually call "normal conditions".

------
cabalamat
> What was learned was object manipulation, not cube solving

Which is almost certainly a harder problem

~~~
uoaei
The point is it wasn't ad hoc object manipulation + cube solving, the hand had
its hand held regarding which macro-action it was supposed to take next.

The PR I've seen about it doesn't make this clear and is happy to leave the
ambiguity because it means more publicity, rather than demonstrating a better
understanding of editorial ethics.

~~~
OnlineGladiator
As much as I'd love to give OpenAI the benefit of the doubt, I have to agree
that they are purposefully being misleading to generate more interest rather
than be as straight-forward as possible about their accomplishment. After the
confusion over whether or not they're really "open" after taking MS
investment, and what I personally consider terrible mismanagement of the GPT-2
announcement and release, I've just lost faith in the organization to be
honest. They never directly lie, they always ambiguously lie, so I rarely even
bother to explain to people why I don't like them and just recommend my
friends look for work elsewhere.

~~~
gdb
> I've just lost faith in the organization to be honest

:( sorry to hear that.

Note that we have a publicly-available Charter which spells out how we
operate: [http://openai.com/charter](http://openai.com/charter). We all use
that to guide our actions, and we have not changed a single word in it since
publication. I hope that as time goes on, you'll increasingly see the
consistency between our actions and the words in the Charter, we'll be able to
win back your support.

~~~
OnlineGladiator
FWIW, just watching you interact with your critics on here has done a lot for
you to earn my respect. I hope I've been too quick to judge and that OpenAI is
able to make me reconsider my stance in the future.

------
duaoebg
I think the blog title use of 'solving' is misleading. As OpenAI didn't make
the solving algorithm. It's a neat but limited use of transfer learning in an
environment which requires heavy instrumentation. I could tell this
immediately from the photo. What I didn't know was that the Rubix cube also
had to be modified and that it had a high failure rate. While I'm sure this is
state of the art in something it does seem overblown.

------
zestyping
This tweet by Woj Zaremba is bewildering. If we all agree it's about physics
and manipulation, how do you get to claim that a failure in manipulation
doesn't matter?

[https://twitter.com/woj_zaremba/status/1185799143664443392](https://twitter.com/woj_zaremba/status/1185799143664443392)

> It’s 20% success rate to solve the most difficult configuration of Rubik’s
> cube. However, on average the success rate is 60%. Moreover, the failure is
> by dropping the cube. The hand always solves the Rubik’s cube if you put it
> back the cube after the drop.

~~~
scotty79
I think it just points out that hardest part is not dropping the cube. The
hard problem they solved completely is orienting and manipulating the cube in
an intended way, provided they managed not to drop it.

Determining how to manipulate cube to get it solved state wasn't part of the
problem as it is trivially easy for a machine or any human that learned how to
solve rubiks cube.

------
nojvek
If anyone reads Gary’s new book Rebooting AI, his whole MO is that to create
really good generalized intelligence current level of neural nets are too
narrow.

And here OpenAI made a claim about solving Rubik’s cube with a robot hand. One
would assume they found a generic algorithm from the headline that does Rubik
solving From both physical and algorithmic perspective. In actuality OpenAI
made a demo of a very specific Rubik’s cube that gave Bluetooth info about its
state (not pure vision like humans do). The Rubik’s algorithm was a pre-
programmed one, not something that was learnt. Only hand manipulation of that
specific cube was learnt.

And we don’t know how general the hand manipulation was. Does it work with
different sized Rubik’s cube? What about a non-bluetooth one? Can the same
system also fold clothes? Assemble lego blocks into some fixture?

Basically OpenAI has taken a fuck ton of VC funding. So the headlines are
hyperbolic when reported. Whether intentional or not, I don’t know. To the
layman it’s sending the wrong message and creating unnecessary fear.

OpenAI, DeepMind, AAMFG need to always explicitly say how narrow their AI is
when they make claims. I.e Here’s 10 things it’s good at and these are the
boundaries. If you change things slightly in the following ways it will fail.

------
MrQuincle
Solving it analytically seems to use deep nets and Monte Carlo tree search
[1]. Some additional structure to deep learning and you're done. Manipulation
is the harder problem.

It's like complaining that a juggler can't count to three.

[1] [https://arxiv.org/abs/1805.07470](https://arxiv.org/abs/1805.07470)

------
jgalt212
> instrumented cube as vision has not been adequately solved.

this speaks to Elon's stubbornness in avoiding the use of lidar.

[https://twitter.com/GaryMarcus/status/1185908530013818881](https://twitter.com/GaryMarcus/status/1185908530013818881)

------
curiousgal
I mean sure, what better way to present your arguments than a picture on
Twitter and asking for people to zoom.

That is it, I'm adding a ublock filter that blocks Twitter thread posts here.

------
lonelappde
OP violates HN guidelines about charitable interpretations and middlebrow
dismissals.

