
Elasticsearch Based Image Search Using RGB Signatures - elasticdog
http://sujitpal.blogspot.com/2016/05/elasticsearch-based-image-search-using.html
======
GrantS
For anyone interested in the computer vision side of this topic, the author
here is using a variant of color histograms, which was state of the art around
1990 [1][2]. Since 2003, bag of visual words approaches have usually meant
extracting SIFT-like features from a database of images, quantizing the
features down to a list of thousands or millions of "words", and then treating
the images like documents containing those "visual words" [3][4]. (Nothing
wrong with the approach he's using [simple and fast], but the bag of words
terminology in the article usually suggests a different class of approaches.)

[1]
[https://staff.fnwi.uva.nl/r.vandenboomgaard/IPCV/_downloads/...](https://staff.fnwi.uva.nl/r.vandenboomgaard/IPCV/_downloads/swainballard.pdf)

[2]
[https://www.cs.utexas.edu/users/dana/Swain1.pdf](https://www.cs.utexas.edu/users/dana/Swain1.pdf)

[3]
[http://www.robots.ox.ac.uk/~vgg/publications/papers/sivic03....](http://www.robots.ox.ac.uk/~vgg/publications/papers/sivic03.pdf)

[4] [http://www-
inst.eecs.berkeley.edu/~cs294-6/fa06/papers/niste...](http://www-
inst.eecs.berkeley.edu/~cs294-6/fa06/papers/nister_stewenius_cvpr2006.pdf)

~~~
infinitone
Forgive my ignorance, but is there something like word2vec but for images-
like an image2vec? In terms of text processing, word2vec is one of the best
approaches. So can't you describe the features of images using neural networks
and then vectorize them using word2vec?

Its been acouple years since my computer vision course (was my favorite course
in university) but isn't SIFT a bit '99? Aren't there better methods now such
as neural networks for feature description?

~~~
apstls
Recent research in deep manifold traversal may interest you [1]. This method
uses deep neural networks to approximate the manifold of natural images,
learns transformations that traverse the manifold from one image class to
another (i.e. from images of young people to images of old people), and then
allows you to transform any source image into a new, automatically-generated
image by mapping it to the manifold and performing the transformation. So, for
example, the paper's authors learn the transformation from images of young
people to old people, and then are able to generate realistic-looking images
of the celebrities with older facial features (as well as darken hair colors,
change skin tones, and even colorize black and white images). This is somewhat
analogous to the way that word2vec vectors allow you to do things like queen =
king + (man - woman), with an image's projection onto the approximated
manifold being analogous to a word's word2vec vector.

[1]
[http://arxiv.org/pdf/1511.06421v3.pdf](http://arxiv.org/pdf/1511.06421v3.pdf)

------
softwaredoug
My "Ghost in The Search Machine" talk builds a really naive image search demo
(which Sujit uses for his starting point). You might enjoy that:

[https://www.elastic.co/elasticon/conf/2016/sf/opensource-
con...](https://www.elastic.co/elasticon/conf/2016/sf/opensource-connections-
the-ghost-in-the-search-machine)

------
sandeepc
If you're interested in searching photos with ES - I took a some what simpler
approach focusing just on major colors in the image.

But with some of the machine vision API google cloud etc. you could extend to
other "features"

Details: [http://blog.sandeepchivukula.com/posts/2016/03/06/photo-
sear...](http://blog.sandeepchivukula.com/posts/2016/03/06/photo-search/)

------
jerluc
In my [limited] experience with CBIR and image based search, I found that
using a color space with perceptual spatial qualities (such as one of the CIE
LaB variants) to be more effective than a purely normalized geometric color
space (such as RGB or HSV), as color similarities in the latter may not make
much sense to a human.

------
UncleChis
I have put quite a bit of similar effort to image retrieval using
Elasticsearch before. While it is nice and convenient, what I found was that
Elastic Search is too slow for larger scale (million of Images with dictionary
of millions visual words in BoW model). May be there're some steps that I did
not do right, but I gave up.

------
infinitone
Really cool. But maybe its just me, aren't butterflies kind of hard to
distinguish between each other. I feel as though his search result page- i
couldn't really tell if it was good or not because they all looked kind of
similar shape-wise. Only difference is color and even then its fairly little
color.

~~~
deckar01
The strategy does seem to assume a highly normalized set of images that only
differ in color and contour.

------
willcodeforfoo
Another open-source implementation using a similar concept and Elasticsearch
is image-match: [https://github.com/ascribe/image-
match](https://github.com/ascribe/image-match)

------
chrischen
Does anyone have a solution to this where the input (search) is also a vector
as opposed to a single color, but would still allow for exact color matches?

Bucketing means that you can't get the granularity of a specific shade of a
color.

------
rcarmo
I wonder if pHash wouldn't make this a lot more effective. Anyone tried
building an ES-usable distance function for pHash?

~~~
sujitpal
Hello, author of the post here. Thanks for all your valuable comments. I
noticed that there was some talk about neural network driven models. In my
follow up post to this one:

[http://sujitpal.blogspot.com/2016/06/comparison-of-image-
sea...](http://sujitpal.blogspot.com/2016/06/comparison-of-image-search-
performance.html)

I describe how I use transfer learning to generate image vectors for my
butterfly images from the Caffe reference model trained on ImageNet. There are
some other approaches too, but probably not as interesting to this audience.

@GrantS - thanks for the intro and the links, and for pointing out that I was
using BoVW incorrectly. Not an image person, trying to get into it from a
search/NLP background.

@infinitone and @deckar01 - I measured the goodness of search by checking how
many times the query butterfly was the #1 result. I agree its pretty hard to
distinguish butterflies from one another, but I needed something to work with,
and that was as good as any. I did not want one that crosses subject areas,
such as flowers and cars, so a search for a red flower might bring back a red
car.

@chrischen - if you got rid of the bucketing portion in my pipeline, I think
you might achieve what you are after. But if you are only looking for exact
match on color, then hashing might be cheaper.

@rcarmo - thanks for the reference to pHash, I think it might work better in
near-duplicate detection kind of cases. Will check it out.

