
Using k-d trees to efficiently calculate nearest neighbors in 3D vector space - nkurz
http://blog.krum.io/k-d-trees/
======
georgecmu
Extract from Andrew Moore's PhD Thesis: Ecient Memory-based Learning for
Robot Control PhD. Thesis; Technical Report No. 209, Computer Laboratory,
University of Cambridge. 1991.

[1991]
[https://www.ri.cmu.edu/pub_files/pub1/moore_andrew_1991_1/mo...](https://www.ri.cmu.edu/pub_files/pub1/moore_andrew_1991_1/moore_andrew_1991_1.pdf)

 _Table 6.4 describes my actual implementation of the nearest neighbour
algorithm. It is called with four parameters: the kd-tree, the target domain
vector, a representation of a hyperrectangle in Domain, and a value indicating
the maximum distance from the target which is worth searching._

~~~
amerine
It’s been a really long time since I’ve read that paper. Thanks for the link
and the call-out out 6.4. That “target distance worth searching” seems
interesting when thinking about color-replacement.

------
faragon
A faster alternative, using a LUT (lookup table):

\- At initialization time, from the given palette (e.g. the CSS palette),
build a 16-bit (RGB565) to 8-bit LUT, and find for every 2^16 entry the
nearest palette color. Time complexity: O(n^2), but done once, and can be
avoided if having the LUT pre-calculated as a constant.

\- At the image conversion, with per-pixel O(1) time complexity: for every
RGB888 pixel, pack the pixel to RGB565, and use the LUT to find de palette
index i.e. pal_index = lut[RGB888to565(pixel)]. With e.g. a 3 GHz modern 4-way
OooE CPU you can convert pixels at > 1 Gpixel per second (> 10^9 pixels per
second), because of the LUT high cache hit ratio.

In case of requiring more precision, you could use a 24-bit LUT (uint8_t[2²⁴]
-> 16 MiB of memory, instead of the uint8_t[2¹⁶] -> 64 KiB of memory), or a
intermediate case of willing to avoid running out of the L2/L3 cache e.g. a
21-bit LUT (RGB685, uint8_t[2^²¹] -> 2MiB). The 16-bit table would allow you
run in the L1 data cache, being very fast (you could use even a smaller table,
e.g. 15-bit RGB555 (32 KiB), or 14-bit RGB554 (16KiB), with similar results
regarding color conversion, but with even lower L1 data cache requirements.

------
agibsonccc
For those interested in this space, folks might like some of the work that
went in to visualizing T Distributed Stochastic Neighbor Embedding and the
successor to that LargeVis which use multiple forms of KNN trees each with
their own trade offs:

[https://arxiv.org/abs/1301.3342](https://arxiv.org/abs/1301.3342) (Barnes
Hut)

[https://arxiv.org/abs/1602.00370](https://arxiv.org/abs/1602.00370)
(LargeVis)

One uses Vantage Point Trees which sudivides the space in to quadrants, and
the other uses Random Projection Trees.

Both are interesting exercises in using KNN like techniques to visualize high
dimensional spaces.

~~~
joshvm
There's also FLANN which is an approximate nearest neighbour algorithm which
is routinely used for things like feature/keypoint matching (e.g. SIFT
keypoints are 128 dimensional).

[https://www.cs.ubc.ca/research/flann/](https://www.cs.ubc.ca/research/flann/)

Barnes Hut is a fun one to implement for N-body simulations.

~~~
_wmd
I was toying with compqring similarity of texts just last night, is there
anything that could handle 10k+ sized sparse vectors? Obviously there are
tools like lucene, but I wanted to mess around in the interactive interpreter

~~~
agibsonccc
Check out LargeVis then. The original github repo by the authors has a sparse
mode:
[https://github.com/lferry007/LargeVis/](https://github.com/lferry007/LargeVis/)

TSNE didn't scale well no matter what you did to it (hence the newer
technique) - in general though you can't beat cosine similarity of TFIDF
vectors as a baseline.

------
IvanK_net
Fun Fact: I am an author of
[https://www.Photopea.com](https://www.Photopea.com) \- advanced image editor,
that has 50 000 lines of code.

In my code, I have three independent implementations of KD-trees, with
different dimensionality for different purposes ... I did not realize it until
now :D

------
enhray
This is a very good intro to K-D trees, big thank you to the author. However,
I think that a three-dimensional lookup table would be more performant in this
case. 16M (256^3) of entries are needed to cover RGB color space with 8 bits
per channel, that's a lot, but lookup time is O(1) with a very small constant.

~~~
contravariant
I think the problem there is that if you're only looking at a few images then
generating the 16M lookup table takes more time than using the K-D tree
itself.

Some kind of memoisation scheme might work though.

~~~
rndgermandude
You can actually use the k-d table to compute the lookup table, as the lookup
table is nothing more really than an image containing a "pixel" with each
possible color exactly once. The author already ran the algorithm on a
12285x14550 image, which took 10x more time than running on a 256x256x256
"image".

Of course, if you want to compute the lookup table dynamically on each run,
you might use memoization and hope the image is either small (and therefore
uses comparatively few colors) or large with few actual colors in use.

------
fho
Which reminds me that some time ago I build a simple [kd-tree library in
haskell]([https://github.com/fhaust/kdtree](https://github.com/fhaust/kdtree)).

The interesting part is that it's easy to construct a (lazy) list of all
points ordered based on the distance to the query point. Other queries
(n-nearest neighbours, points in radius) are just derived from that function
and traverse only the first some elements of the list.

------
geophile
Simpler:

\- Compute z-value for each vector, store in binary tree (or sorted array,
etc.), keyed by z-value.

\- Compute z-value of search vector, search the data structure, find the k
adjacent items in each direction (k is small, e.g. 1-5).

\- Of the vectors found, compute minimum distance from the search vector, r.

\- Do another search of your data structure, using a sphere of radius r, (easy
z-order algorithm).

\- The nearest neighbor is the vector resulting from this search whose
distance from the search vector is minimum.

~~~
klodolph
That's only viable when you have a small data set, which is when you don't
care much about indexing to begin with.

~~~
geophile
Huh? I've used z-order inside database systems, with many millions of records.
Works incredibly well.

------
photon-torpedo
Semi-related: Anybody know what's the current state of the art for calculating
nearest neighbors in high-dimensional spaces (at least 10D, maybe 100s of D)?

~~~
Radim
Various approaches based on LSH (locality sensitive hashing) and random
projections.

But "the best" really depends on your data profile, as the generic problem
formulation, for completely generic vectors, is too… generic. See also _the
curse of dimensionality_ [0].

In practical terms, if you're looking for an actual implementation to use,
here's a benchmark of some state-of-the-art implementations for high-
dimensional vector search: [http://radimrehurek.com/2013/12/performance-
shootout-of-near...](http://radimrehurek.com/2013/12/performance-shootout-of-
nearest-neighbours-contestants/) and also the updated
[https://github.com/erikbern/ann-benchmarks](https://github.com/erikbern/ann-
benchmarks).

In particular, for something good AND open source:

\- Leonid Boytsov's NMSLIB [1]

\- Erik Bernhardsson's Annoy [2]

\- Facebook's FAISS [3]

For a commercial engine focused more on practical use (index management,
transactions, versioning, support), see ScaleText [4] (disclaimer: our
product).

[0]
[https://en.wikipedia.org/wiki/Curse_of_dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality)
[1]
[https://github.com/searchivarius/nmslib](https://github.com/searchivarius/nmslib)
[2] [https://github.com/spotify/annoy](https://github.com/spotify/annoy) [3]
[https://github.com/facebookresearch/faiss](https://github.com/facebookresearch/faiss)
[4] [https://scaletext.com/](https://scaletext.com/)

~~~
photon-torpedo
Very helpful, many thanks!

------
Ono-Sendai
Discussion on reddit:
[https://www.reddit.com/r/programming/comments/7wg91g/using_k...](https://www.reddit.com/r/programming/comments/7wg91g/using_kd_trees_to_calculate_nearest_neighbors_in/)

------
electricslpnsld
How easy is it to extend k-d trees to deal with objects with finite size (vs.
points)? Are there any libraries that do this? I might have to do something
similar to this in the future, but I need nearest neighbors for 'spheres' in
higher dimensions.

~~~
rwbt
AABB Trees (Axis Aligned Bounding Box) might be an option. Most realtime
collision detection systems utilize them.

~~~
yoklov
Or more generally bounding volume hierarchies, of which AABB trees are a
subset.

They’re great. Good perf, while being one of the simpler ways of accelerating
spatial queries (compared to other types of trees, at least)

------
malingo
This is color quantization, right?
[https://en.wikipedia.org/wiki/Color_quantization](https://en.wikipedia.org/wiki/Color_quantization)

------
im3w1l
Which leads us to dithering

~~~
agumonkey
I can't find it right now but there was a lovely article on old ways and new
ways to dither.

other articles:

\-
[https://news.ycombinator.com/item?id=11886318](https://news.ycombinator.com/item?id=11886318)

\-
[https://www.cs.princeton.edu/courses/archive/fall00/cs426/le...](https://www.cs.princeton.edu/courses/archive/fall00/cs426/lectures/dither/dither.pdf)

\-
[https://hbfs.wordpress.com/2013/12/31/dithering/#more-4980](https://hbfs.wordpress.com/2013/12/31/dithering/#more-4980)

------
petters
There is a relatively large distance between the native approach and kd-trees.
I'm not sure that can be explained with equidistance alone. There might be a
subtle bug.

------
Jordanpomeroy
If I remember correctly these things don’t do well with dynamic updates,
right? Don’t you have to rebuild the whole tree for every update?

