
Image unshredding using a TSP solver - robinhouston
https://github.com/robinhouston/image-unshredding
======
preygel
A bit tangentially, this is also a great display of _just how good_ readily
available approximate solvers have gotten for a wide range of combinatorial
optimization problems (like TSP).

The LKH solver used here has rather impressive performance. From
[http://webhotel4.ruc.dk/~keld/research/LKH/](http://webhotel4.ruc.dk/~keld/research/LKH/):
"LKH has produced optimal solutions for all solved problems we have been able
to obtain," and the studies linked there show that it really does find optimal
solutions "with an impressively high frequency."

While the point here is to use an off-the-shelf solver, it can be nice to have
visibility into what's actually happening:

\- This is a reasonable example for explaining some of the local search TSP
heuristics. For instance a "2-opt" move corresponds to picking a contiguous
range of columns and flipping them.

\- The previous HN post used simulated annealing, which would not be an
outright terrible approach to TSP itself -- were it not for the better Lin-
Kernighan-based approaches (like LKH).

\- If we want, we can tell LKH to start from Sangaline's "nearest-neighbor"
approach ("INITIAL_TOUR_ALGORITHM = NEAREST-NEIGHBOR"). This does not make a
difference here though.

~~~
robinhouston
Not so long ago, we’d say “That’s an NP-complete problem” with the implication
that it was hopeless to expect an efficient solution. Now (to exaggerate a
little) it’s more like a sign that there probably _is_ an efficient solution
using some powerful solver.

LKH is a lovely piece of work. I used it a couple of years ago to find a
counterexample to an unimportant but fairly long-standing conjecture in
combinatorics[0], and the list of scientific applications[1] is impressively
diverse.

The best SMT solvers are also hugely impressive. Z3 has an excellent
interactive tutorial and online solver.[2]

[0] [https://arxiv.org/abs/1408.5108](https://arxiv.org/abs/1408.5108) [1]
[http://webhotel4.ruc.dk/~keld/research/LKH/ScientificApplica...](http://webhotel4.ruc.dk/~keld/research/LKH/ScientificApplications.html)
[2] [http://rise4fun.com/z3/tutorial](http://rise4fun.com/z3/tutorial)

~~~
botw
I always thought "NP-complete/NP-hard" metaphor only meaningful in theoretical
work. In practice, it does not make much sense in guiding what/how the real
world problem is to be solved.

~~~
robinhouston
I don't know. I find it useful in practice. Apart from anything else it means
there's (probably) no point looking for an exact polynomial-time algorithm,
which can save a lot of wasted effort. But it also gives useful clues about
what sort of algorithmic approaches might be useful, and allows you to relate
your problem to a huge amount of existing research.

It's really basic taxonomy: what sort of problem are we dealing with? If you
don't know that, how could you even begin to design a reasonable algorithm for
it?

~~~
botw
Maybe you are heavily on huge algorithmic work. In practice, most algorithms
are polynomial-time, and we are looking for under-polynomial-time alternative
- log(n). For exponential-time, we are just looking for an approximate
alternative.

------
alpb
I'm really surprised by double-shuffling can actually be solved.
[https://github.com/robinhouston/image-unshredding/#double-
sh...](https://github.com/robinhouston/image-unshredding/#double-shuffling) It
looks very unintuitive.

~~~
jcoffland
Isn't the definition of insanity something about doing the same thing and
expecting different results.

~~~
jcoffland
Funny, I thought more people would have gotten what I was saying here.

~~~
touch_o_goof
You're forgetting HN is serious discussion for serious people or something
like that. Humour in a substantive post seems fine, but usually any humour on
its own doesn't fare well here.

------
barrkel
The 2D shuffling result is a bit surprising at first glance, but less so when
you think about it - row comparison doesn't care about the order of the pixels
in each row as long as the order is the same in both, and the shuffling
mechanism guarantees that.

~~~
dastbe
the important bit is you have to unscramble in reverse order of the
scrambling. Unscrambling the columns and then unscrambling the rows would lead
to smooth gibberish.

~~~
barrkel
No. Column scrambling doesn't affect row ordering and row scrambling doesn't
affect column ordering. They are literally orthogonal. You can't tell, from a
scrambled matrix, whether the rows or the columns were scrambled first.

Here's a more explicit breakdown:

    
    
        (0,0) (1,0) (2,0)
        (0,1) (1,1) (2,1)
        (0,2) (1,2) (2,2)
    

Column scrambling: swap 1&2:

    
    
        (0,0) (2,0) (1,0)
        (0,1) (2,1) (1,1)
        (0,2) (2,2) (1,2)
    

Now row scrambling: swap 0&1:

    
    
        (0,1) (2,1) (1,1)
        (0,0) (2,0) (1,0)
        (0,2) (2,2) (1,2)
    

Notice how the x coordinates still all match along the vertical, and y
coordinates still all match along the horizontal.

Let's try it in the other order, row first:

    
    
        (0,1) (1,1) (2,1)
        (0,0) (1,0) (2,0)
        (0,2) (1,2) (2,2)
    

And columns:

    
    
        (0,1) (2,1) (1,1)
        (0,0) (2,0) (1,0)
        (0,2) (2,2) (1,2)
    

It's the same result irrespective of the order.

~~~
kmill
Another way to think of it is that permuting rows corresponds to multiplying a
matrix on the left by a permutation matrix, and permuting columns, on the
right. Since matrix multiplication is associative, they commute!

~~~
evincarofautumn
Yet another way to think of it is that a matrix is an encoding of a linear (or
affine) function, matrix multiplication is composition of such functions, and
since composition is associative, they commute. :)

------
xchip
Shuffling lines was the scrambling algorithm used in nagravision a while
back...
[https://www.youtube.com/watch?v=gFAinn6sddQ](https://www.youtube.com/watch?v=gFAinn6sddQ)
there was a software that could descramble the video using a similar technique

------
jcoffland
Now I'd like to see this run on a more realisticly shredded image. A real
paper shredder creates strips that are more than one pixel thick are not
straight on (I.e. the pixels don't necessarily align with the cuts) and
possibly have cutting defects such as ragged edges or nicks.

~~~
foota
Seems like with a high resolution enough image, some of these effects may
actually make the problem easier.

~~~
jcoffland
Perhaps, but you would have to look at more than just neighboring pixel color
to take advantage.

------
bluesign
Reminds me nagravision decoders (if i recall correctly), one of the first pay
tv protection algorithms

Edit:
[https://en.m.wikipedia.org/wiki/Nagravision](https://en.m.wikipedia.org/wiki/Nagravision)

------
syphilis2
Could be used to compress images? Perhaps instead of shuffling the columns and
rows randomly they could be ordered in ways that are better suited for
compression.

~~~
gpderetta
That's what the Burrows–Wheeler transform does for text. A similar approach
could work for images.

~~~
OskarS
It's a good point. PNG uses DEFLATE internally for the compression, it would
be interesting to see what would happen if you used bzip2 instead, which uses
the Burrows-Wheeler transform.

------
Retr0spectrum
Roughly how long does this algorithm take to run?

~~~
robinhouston
On my laptop (real times for a single run):

1.6 seconds to make the TSPLIB2 file representing the dissimilarity graph.
(time make tsp/instances/blue-hour-paris.tsp)

0.3 seconds to run the solver and produce the optimal tour. (time make
tsp/tours/blue-hour-paris.tour)

0.4 seconds to reconstruct the image from the shuffled version using the tour.
(time make images/reconstructed/blue-hour-paris.png)

------
aratno
I wonder how well this would work with some image pre-processing. Wouldn't
something like supersampling improve the speed per-pixel? Of course, it would
require more pixels to be sorted, so I doubt it's any more optimal.

------
LolWolf
This is super interesting; how does it compare to other sampling algorithms?
While the problem is indeed reducible; it also has a _lot_ more structure than
TSP, especially if we note that natural images are forced to come from some
nice, underlying manifold which is probably very well-behaved.

I'd be inclined to believe that with some assumptions on the problem itself
(e.g. the images satisfy some smoothness assumptions and of course,
assumptions on the objective, itself), we can always come up with a PTAS for
this particular instance for any accuracy, ε>0.

------
emmelaich
I guess this could be used to reassemble images corrupted in other ways (other
than shuffling).

I have a disk image of jpegs and other data I'd like recovered. It was Windows
formatted a few different ways but any blocks that format didn't touch are
still there.

There's other data there of course as well, so there would be lot of blocks
that aren't part of images so would need to be ignored when reassembling the
images.

~~~
taeric
I'd guess not, actually. In this a lot of the information is there. Just in a
different order. Corrupting data is a different problem.

That said, I'm sure I could be surprised by what is possible.

------
botw
I guess the difference with the simulated annealing approach would be that
this approach(and the simple picking-the-most-similar-column approach) assumes
you know the particular shredding method in advance, whereas the simulated
annealing does not have this requirement. For example, does this still work
out for shredding diagonally in some unknown degree?

------
askee
How good does it work with non-natural images where rows/columns are not
similar to each other? White noise for example should prove rather difficult.

------
rixed
Do you know of any image compression technique that would drop information
that we expect to be redeemable in a reasonable time using a solver of this
sort?

~~~
inimino
Since the shuffling adds information, it seems unlikely.

~~~
rixed
Don't you think that for any image there exist a shuffled version that
compress better?

~~~
inimino
I don't see any reason why there necessarily would be. Generally speaking,
compression algorithms do better when adjacent pixels have similar colors,
which is the unshuffled state.

In cases where there is a version that compresses better, unshuffling might
not work. If you had an "pinstripe" image consisting of alternating black and
white columns of pixels, obviously it could compress better after being
"shuffled" with all the similarly colored columns together. But then this
algorithm would be unable to get the original image back.

------
microcolonel
Looks like it's time to go back to burning.

