
Parallel Seam Carving - ngzhian
https://shwestrick.github.io/2020/07/29/seam-carve.html
======
nwallin
Barrier synchronization is always going to wreck parallel performance. When
possible, it's good to break up your work so that data dependencies are
explicit and fine grained. So instead of breaking up work like this:

    
    
        IIIIJJJJKKKKLLLL
    
        <barrier>
    
        EEEEFFFFGGGGHHHH
    
        <barrier>
    
        AAAABBBBCCCCDDDD
    

... where each row is dependent on the entire previous row, break it up like
this:

    
    
        HHHHIIIIJJJJKKKK
    
        EEEEEEFFFFGGGGGG
    
        AAAABBBBCCCCDDDD
    

Where E is dependent on A and B, (and begins when they are done) H is
dependent only on E, F is dependent on B and C, etc. This way you don't have
to wait for the entire strip to finish, you can start work on E after A+B, but
while D is still working. This also gives you weird block sizes for "free",
which is a huge no-no with barrier synchronization. For instance, Es and Gs
are size 6; if the first example had most elements of size 4, but one element
of size 6, it would be catastrophic. All your threads would be done when the
final thread is only 66% complete, and it's single threaded from there on out
for that strip.

~~~
lpage
This pencil and paper approach to breaking up work is good at parallelizing
things that don't appear parallelizable at first glance, e.g. prefix sum [1].
Compilers are good at identifying opportunities to reorder instructions in
ways that break low level data dependencies. They tend to miss the big higher
level opportunities like this.

[1]
[https://en.wikipedia.org/wiki/Prefix_sum#:~:text=A%20work%2D...](https://en.wikipedia.org/wiki/Prefix_sum#:~:text=A%20work%2Defficient%20parallel%20prefix,%2C%20z2%2C%20..).

------
gabeiscoding
I created a desktop GUI tool [0] that used multiple cores for seam carving
images of your choice. Including masking parts of image to keep or remove. The
backend code parallelizing the work is from the CAIRE project.

It was ages ago, and has since been archived on google code :)

[0] [https://code.google.com/archive/p/seam-carving-
gui/](https://code.google.com/archive/p/seam-carving-gui/) [1]
[https://github.com/esimov/caire](https://github.com/esimov/caire)

------
kanobo
This is so cool, it reminds me of this series of responsive pixel art:
[https://essenmitsosse.de/pixel/?showcase=true&slide=5](https://essenmitsosse.de/pixel/?showcase=true&slide=5)

------
thamer
If folks are interested in a simple implementation to read or reuse, here's a
JS version I wrote back when HTML5 and <canvas> were new and exciting (over 10
years ago now): [https://nicolasff.github.io/canvas-seam-
carving/](https://nicolasff.github.io/canvas-seam-carving/)

The button says "warning: very slow" but JavaScript performance has
significantly improved since then :-)

Still a very cool technique, and pretty easy to implement.

------
derefr
A related idea would be “image summarization”, i.e. an (as-yet hypothetical)
technique for generating an image that’s an “icon” of a larger image. In the
same sense of “icon” as the one visual designers are trained to create: one
where both large-scale structure (shape+color) and fine-scale structure
(texture) are preserved, while “redundant” detail is sacrificed, in order of
least to most important for visual recognition by a human or image-classifier
model. You could also call this a type of _psycho-visual_ compression, in the
same vein as chroma subsampling.

Simply resampling an image smaller keeps the large-scale structure (the shape
and color), but loses the fine-scale structure (texture.) That’s why we don’t
just resample photos down to 32x32 and use them as icons :)

Seam Carving (whether parallel or not) does the opposite: it preserves the
fine-scale structure (texture) but distorts the large-scale structure (shape.)

But not always! The amount of shape distortion in Seam Carving is worst case
O(N) with the number of seams deleted, but best-case o(1). It depends on the
algorithm’s choices of seam.

Regular _iterative_ Seam Carving, as described by its original paper, isn’t
stateful-enough to be able to minimize any whole-image quality (like loss of
large structure.) So it’s no surprise it hasn’t been used for something like
this.

But _parallel_ Seam Carving might just be the ticket for this problem. A model
could be trained to select an optimal set of seams that work together to
minimize the large-structure deformation of the image (i.e. the image’s lower-
band difference in frequency-space), while also individually being low-entropy
seams from the Seam Carving algorithm’s perspective.

Anyone want to take a crack at this?

------
solstice
That's really neat. What are some practical applications for Seam Carving?
Maybe in video games with dynamic resizing of landscapes?

~~~
jetrink
I have a photo of my friend, his ex-wife and a few others standing together on
a camping trip. I wanted to share that photo with the group since it was the
tenth anniversary of that trip, but didn't want to remind my friend of his
divorce, so I used seam carving to elide his ex-wife. (The tool I used allows
you to paint areas of low or high energy.) The result was almost seamless (pun
intended) and you wouldn't notice anything amiss if you weren't looking for
it.

~~~
macca321
This is a SAAS product if ever there was one

------
anonymoushn
You can divide the work into twice as many independent chunks of the same size
like so:

If you were going to divide the image into K horizontal strips of triangles,
instead divide the top half into K/2 horizontal strips of triangles and divide
the bottom half into K/2 horizontal strips of triangles. To make things easier
at the end, please include a 1px tall overlap between the top and bottom
parts. Work downward from the top and upward from the bottom until you reach
the middle. At the middle, for each column, you have the cost of the top half
of the minimum seam that includes that pixel and the cost of the bottom half
of the minimum seam that includes that pixel. So you can add those together
then take the min as before.

------
repsilat
Does repeated single-pixel seam-carving minimise energy for the "remove n
seams" problem? If not, is the global algorithm in P?

Also, I guess this is beside the point, but surely almost all of the DP
structure can be reused if you're repeatedly removing seams...

~~~
boothby
> Also, I guess this is beside the point, but surely almost all of the DP
> structure can be reused if you're repeatedly removing seams...

It's not besides the point at all. If you can reduce this linear-time
algorithm to, say, amortized constant, then you'll handily beat parallelism.

After you remove a seam, you need to recompute the cones below every removed
pixel -- which ends up being the cone below the topmost removed pixel (sadly,
you can't just zip down the neighbors of the seam, but you _can_ avoid
expensive data moves with a 2d linked list). If the image is super wide, that
gives you a speedup by approximately the aspect ratio. If it's square, you
should be able to cut the runtime in half. If it's tall and skinny, neither
this nor the author's approach are terribly helpful.

~~~
goldenkey
Can't the image just be rotated if it is tall and skinny? Then unrotated after
the transformation?

~~~
almostarockstar
Rotating as you suggest would be equivalent of swapping between vertical and
horizontal seams. A tall and skinny image with vertical seams has the same
problems that a long and wide image has with horizontal seams.

------
etaioinshrdlu
This is probably also a very interesting technique for data augmentation for
computer vision ML. (That is, seam carving. Not much to do with the parallel
version...)

Unlike other typical data augmentation operations, spatial relationships are
altered in nonlinear but still realistic ways.

~~~
dguest
Cases where data augmentation works always pose a fun challenge: clearly if
there's some way to manipulate the data which shouldn't alter the algorithm
output, there's also some way to make the ML invariant to that manipulation.
If you can create an invariant algorithm then the augmentation should be
unnecessary.

There is some interesting work on making ML invariant to rotations and such,
but I'd be curious if there's an algorithm which is invariant to this. I could
imagine that convolutions and pooling might be relatively invariant to this
technique, for example, as long as the algorithm isn't doing a lot with the
large scale structure of the image.

------
EE84M3i
This is confusing to me because I thought the right way to do it was push-
relabel.

------
amelius
How does seam carving preserve important symmetries (e.g. in the human face)?

~~~
chias
In most actual implementations, you can mark areas to avoid (e.g faces) and
areas to prioritize.

