
Content-aware image resize library - boyter
https://github.com/esimov/caire
======
rudi-c
Happy to see this here! Seam carving is one of those neat little algorithms
that hit a sweet spot, being a sum of a few moderately complex algorithms,
comes from a readable paper, and gives very satisfying result. I had a lot of
fun implementing it in undergrad.

One thing that I always wondered is how Photoshop managed to make it so fast
that you can resize in real-time. If n is the width or height of the image,
then the dynamic programming part is O(n^2) and needs to be recomputed after
every seam removed. Since every seam is a single pixel wide, resizing the
image by a non-trivial amount (say half) is O(n^3). There are other papers
that remove multiple seams at a time but the quality isn't as good. GPU
acceleration perhaps?

Another thing I learned while testing seam carving extensively is that it
works nicely in certain scenes/situations, but tends to break down most of the
time. The two most common scenarios are: 1) lines that are off by the
horizontal by more than a few degrees get cutoff 2) objects loose their
proportions and even when the manipulation is not directly obvious, it tends
to feel off (in the uncanny valley sense).

I expect some interesting work to use deep learning for content-aware
resizing, since neural nets could theoretically be more semantically and
holistically aware of objects in the image.

~~~
vanderZwan
Well, there is this paper called _" Real-time content-aware image resizing"_,
maybe that describes the method you are curious about?

> _In this paper, we present a more efficient algorithm for seam based
> content-aware image resizing, which searches seams through establishing the
> matching relation between adjacent rows or columns. We give a linear
> algorithm to find the optimal matches within a weighted bipartite graph
> composed of the pixels in adjacent rows or columns._

[https://web.archive.org/web/20110707030836/http://vmcl.xjtu....](https://web.archive.org/web/20110707030836/http://vmcl.xjtu.edu.cn/Real-
Time%20Content-
Aware%20Image%20Resizing.files/real_time_content_aware_image_resizing.pdf)

~~~
rudi-c
That's the one I implemented -- with a bit of low-level hackery, it could run
fast enough to resize at ~5-8 fps on mobile back in 2013. But the result
didn't look great on most images if you resized by more than ~15%. The
technique works by calculating the "importance" field of the image once, and
all the seams upfront. Then you can just number the seams from 1-n and remove
the first m seams when you resize the image by m. The runtime is then O(n^2)
instead of O(n^3). In contrast, with regular seam carving, you'd often update
the "importance" field every time you remove a seam. The result is that the
seams that get removed are less likely to introduce artifacts. I think it's a
similar problem as the forward energy v.s. backward energy problem discussed
elsewhere in the thread, but worse.

~~~
vanderZwan
Hmm, then I don't know. Still, they mustn't have stood still since then and
come up with new tricks.

I wonder if you could do a kind of quad-tree version where you hone in on the
precise seam and minimise recalculations to bubbling new energy up the tree.
That might also give a way to weigh global-to-local structure.

------
Grue3
Just saying, but GIMP had this for years with Liquid Rescale plugin (I think
before even Photoshop):
[http://liquidrescale.wikidot.com/](http://liquidrescale.wikidot.com/)

~~~
saurik
Neither GIMP nor Photoshop are libraries ("just sayin'"). It sounds like there
were other previous libraries: that would be more relevant of a callback.

[http://liblqr.wikidot.com/](http://liblqr.wikidot.com/)

------
mstade
I remember first seeing content aware image scaling at Adobe MAX in Barcelona
I think it was. We were completely dumbfounded by what was surely magic
happening on stage. When they showed removing objects we just lost it, jaws on
the floor and all. That was a fun conference.

~~~
esimov
I'm the author of this library. The algorithm permits to remove an object from
the image, only it needs somehow to be localized. After localization it can be
applied to that area a higher energy map, which means that part will be
avoided by the seam carver. I will implement a face detection algorithm to
automatically exclude faces to get not altered by the carver.

~~~
rcarmo
There are a few interesting things to explore in that field, with different
twists:

[https://algorithmia.com/algorithms/opencv/SmartThumbnail](https://algorithmia.com/algorithms/opencv/SmartThumbnail)

[https://blog.twitter.com/engineering/en_us/topics/infrastruc...](https://blog.twitter.com/engineering/en_us/topics/infrastructure/2018/Smart-
Auto-Cropping-of-Images.html)

------
gabeiscoding
I wrote a GUI for another seam carving library back in 2009[1], and it looks
like although in archive mode, you can still access the source as well as the
windows / mac binaries. Just tested it and it still works!

Not as fancy as photoshop I'm sure, but does have the ability to paint a mask
of regions to keep / remove to aid the algorithm and get the desired result.
Multi-threaded too!

[1] [https://code.google.com/archive/p/seam-carving-
gui/](https://code.google.com/archive/p/seam-carving-gui/)

~~~
oh_sigh
Thanks for that. I used it way back when(mid 2010) to make it look like a a
30' cliff jump was a 60' one. Really impressed my friends with it.

~~~
MaxBarraclough
High praise indeed - you know the software is good when you can use it to lie
to your friends.

~~~
oh_sigh
Now that this thread is essentially tially dead, I can let out my little
secret - I'm not the bad boy you think I am. I told them thw truth eventually
-about 30 seconds after showing them the doctored one. I hope this doesn't
change your opinion of me.

------
Terretta
Came for the “seam carving”... okay.

Didn’t expect image expanding... was blown away.

Mix in temporal energy plus a ML objects DB, put that in a chip, and turn
decades of 4:3 TV into widescreen!

~~~
serpix
I'm already quite interested in resizing wide images to square images for
Convolutional Neural Networks training

------
mfbx9da4
I actually also implemented seam carving in JS a while ago with a live demo in
the browser [http://davidalbertoadler.com/projects/seam-carving-
js/](http://davidalbertoadler.com/projects/seam-carving-js/)
[https://github.com/mfbx9da4/seam-carving-
js](https://github.com/mfbx9da4/seam-carving-js)

------
AgalmicVentures
Would this still be covered under the original patent until 2029 [1]?

[1]
[https://www.google.com/patents/US8213745](https://www.google.com/patents/US8213745)

~~~
jimrandomh
That patent was filed in 2009, and appears to be a copy of a 2007 paper
[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.570...](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.570.6321&rep=rep1&type=pdf)
with no novel contributions, and no overlap in authorship.

~~~
aaronblohowiak
This one has one of the original authors and was filed in 2008:
[https://www.google.com/patents/US8280191](https://www.google.com/patents/US8280191)
not sure about differences in claims. IANAPA

~~~
vanderZwan

         2008: Original Assignee	Abode Systems Incorporated
    
         2009: Original Assignee	Eastman Kodak Company
    

Perhaps the Kodak patent covers a hardware-implementation for their cameras?

~~~
dotancohen
_Eastman_ Kodak manufactures and distributes cinema film. The camera division
is the _other_ Kodak.

------
vanderZwan
Seam carving is fascinating, and especially the first version of the algorithm
really really simple to implement. However, that algorithm introduces certain
artefacts - the boat in this picture[0] makes it pretty obvious.

The cause of these artefacts are mainly that the original algorithm did not
look at energy that was _introduced_ by removing a seam: each time this
happens, the pixels adjacent to this seam become neighbours, which creates a
new energy gradient. So sometimes removing the least-energy seam would produce
a net _increase_ in energy.

The original authors of the seam carving paper realised this[1], which lead to
the obvious fix for it: instead of only looking at the current energy of a
picture, and removing the least-energy seam, look how much of a net energy
difference the removal of a seam would make, aka "forward energy".

A few years later another paper came out that used "forward gradient
difference maps" which supposedly work even better, but to be honest the
formulas described in that paper are too complex for me to understand[2].
Conceptually though, I think they just extended the original energy function
(a simple Sobel operator[3]) with a few other ones that include orientation
and others:

> _The energy function measures the curvature inconsistency between the pixels
> that become adjacent after seam removal, and involves the difference of
> gradient orientation and magnitude of the pixels. Our objective is to
> minimize the differences induced by the removed seam, and the optimization
> is performed by dynamic programming based on multiple cumulative energy
> maps, each of which corresponds to the seam pattern associated with a pixel.
> The proposed technique preserves straight lines and regular shapes better
> than the original and improved seam carving, and can be easily combined with
> other types of energy functions within the seam carving framework_

With that in mind, it shouldn't be that complicated to come up with more
improvements to seam-carving: just stack on different energy functions (either
forward or current energy), and compute the ideal seam based on the (weighted)
sum of them.

[0] [https://user-
images.githubusercontent.com/883386/35498498-3c...](https://user-
images.githubusercontent.com/883386/35498498-3c0f182c-04d7-11e8-9af8-695bc071e0f1.jpg)

[1]
[http://www.eng.tau.ac.il/~avidan/papers/vidret.pdf](http://www.eng.tau.ac.il/~avidan/papers/vidret.pdf)

[2]
[http://cvlab.postech.ac.kr/~hyeonwoonoh/acmmm2012.pdf](http://cvlab.postech.ac.kr/~hyeonwoonoh/acmmm2012.pdf)

[3]
[https://en.wikipedia.org/wiki/Sobel_operator](https://en.wikipedia.org/wiki/Sobel_operator)

~~~
bitL
Seam carving is also slow and doesn't work well for stretching images beyond
around 50%. Moreover, computing optimal sequence of carving steps using
dynamic programming for large images is not practical.

------
rhema
Looks really interesting. I'd love to see an animated version that shrinks
from large to small.

~~~
asoto
I wrote a version of this in javascript with a live demo you can play with at
[https://alexander.soto.io/seam-carving](https://alexander.soto.io/seam-
carving)

~~~
peterburkimsher
Your JavaScript version is very easy to use, and I'm so glad I can upload my
own images instead of just using examples!

The algorithm itself looks like it needs face (and body) detection to be
really useful, though.

Example: My girlfriend and I on holiday in Australia.

Original:

[http://peterburk.free.fr/seamCarving/lighthouse.jpg](http://peterburk.free.fr/seamCarving/lighthouse.jpg)

Shrunk:

[http://peterburk.free.fr/seamCarving/lighthouse_seamCarving....](http://peterburk.free.fr/seamCarving/lighthouse_seamCarving.jpg)

~~~
lysium
In all fairness, the read me says “Todo: face detection”

~~~
peterburkimsher
That's true, and I agree with the author!

It will be more complicated than it seems, though, because I just tried the
GUI version from gabeiscoding above, and got the following result:

[http://peterburk.free.fr/seamCarving/lighthouse_seamCarvingG...](http://peterburk.free.fr/seamCarving/lighthouse_seamCarvingGUI.jpg)

------
michaelthiessen
I am always blown away by this stuff, but it still applies some distortion to
the image in a way that feels slightly unnatural.

It's similar to the effect that changing focal length on a camera has. eg.
Wider angle lenses seem to squish far away objects and enlarge closer objects.
However, this affects the entire image in a consistent way whereas changing
focal length does not.

I wonder if this could be used to simulate focal length changes?

------
barrystaes
Nice implementation. Like face detection also text-detection would be nice, to
avoid distorting / aliasing text (like in advertisements).

------
visarga
How well can it remove objects from images? Can it remove an object in a
cluttered environment? I'm thinking of image augmentation for deep CNN's.
Normally, augmentation can help with invariability to small rotations,
translations and flips, but if we could remove objects from images, it might
be useful for creating many examples from one image, for image based
reasoning.

~~~
esimov
The algorithm has the ability to remove image parts based on the energy level
of each object calculated by a simple sobel operator. So in order to remove an
object you have to either localize the object and give a higher energy value
or to train a CNN to localize some specific object type, then again increase
the energy value. Right now i'm working to implement the face detection part.
So if anyone is interested in extending the library even further and to
integrate a CNN, it's more than welcome.

------
utkarshsinha
Surprisingly, imagemagick already supports this.

~~~
cooper12
Link: [http://www.imagemagick.org/Usage/resize/#liquid-
rescale](http://www.imagemagick.org/Usage/resize/#liquid-rescale). Seems like
it was added as an experimental feature in v6.3.8-4. It requires compiling
with the option `--with-liblqr`.

------
microcolonel
Not to downplay the cool factor of a new implementation, Liquid Rescale (long-
time GIMP plugin) also has protective masks, which prevent or delay the laying
of seams in some parts of the image. Caire seems to intend to do a more
automagical subset of that, with the "Face detection" todo, but the better
short term solution is to accept masks directly.

[http://liquidrescale.wikidot.com/en:examples](http://liquidrescale.wikidot.com/en:examples)

------
anandkunwar
Looks cool, will check out soon. I remember I created an image resize web
application using seam carving during college for my software engineering
project as I was fascinated by the algorithm. Not sure if in working state
currently.
[https://github.com/pixation/pixation](https://github.com/pixation/pixation)

------
justkez
Loosely tangental; can anyone recommend an image library management system
they've had success with? Looking for uploading, thumb-nailing, search
metadata, access levels and ideally some on-the-fly transformations like
Cloudinary offer

~~~
igordebatur
I would suggest having a look at Uploadcare,
[https://stackshare.io/uploadcare/how-uploadcare-built-a-
stac...](https://stackshare.io/uploadcare/how-uploadcare-built-a-stack-that-
handles-350m-file-api-requests-per-day) (though I am biased as a CEO)

------
aaronblohowiak
Is anyone aware of applying a similar technique to computer models ie: fake
parametric design?

------
ttoinou
Could we use any direction for pixel removal, rather than only vertical or
horizontal ?

~~~
jontro
Just run the algorithm twice?

~~~
ianjsikes
Better yet, scale the image proportionally down to the largest of the two
target sides, then run the algorithm once in the direction that needs to
shrink.

------
pcurve
It looks promising, but the current result isn't great. Use cases for
something like this is very specific... perhaps mass conversion of thousands
of photos from 4:3 into 16:9...

------
Jack000
is this still the best approach to this problem? I would think that a GAN or
similar would be better for keeping the correct features for things like faces
etc.

------
airstrike
Time to write an iOS app and make millions?

------
justinator
I think they had this in Photoshop circa 1999, or I def. remember seeing a
demo of it by Adobe around that time.

Anyways, neat! Nice contribution.

~~~
bringtheaction
It was much more recent than that. Content Aware Fill was introduced in
Photoshop CS5 which was released in 2010. Perhaps you were thinking of the
Spot Healing Brush of Photoshop which is somewhat similar but much more manual
and limited.

~~~
justinator
I distinctly remember the tech being demo'd in 1999-2000. Huge amounts of
"wow's" being echo'd in the auditorium. This was around when PS 4.5/5 was out.
I guess it was a feature to be slated for PS 6?

------
mholt
Seam carving is cool and has a lot of potential, especially for certain kinds
of images. I can't wait until we see something like it in pro photo editors
like Affinity Photo.

~~~
bitL
Photoshop has it for ages; even bought the original research.

