I've had enough of headlines that overpromise and underdeliver. It's essentially false advertising. It's not like the word "subset" would put it over the length limit.
I'm aware of this to an extent. Do you know of any list of what degree of parallelization to expect out of various components? I know this whole napkin-math thing is mostly futile and the answer should mostly be "go test it", but just curious.
I was interviewing recently and was asked about implementing a web crawler and then were discussing bottlenecks (network fetching the pages, writing the content to disk, CPU usage for stuff like parsing the responses) and parallelism, and I wanted to just say "well, i'd test it to figure out what I was bottlenecked on and then iterate on my solution".
Napkin math is how you avoid spending several weeks of your life going down ultimately futile rabbit holes. Yes, it's approximations, often very coarse ones, but done right they do work.
Your question about what degree of parallelization is unfortunately too vague to really answer. SSDs offer some internal parallelism. Need more parallelism / IOPS? You can stick a lot more SSDs on your machine. Need many machines worth of SSDs? Disaggregate them, but now you need to think about your network bandwidth, NICs, cross-machine latency, and fault-tolerance.
The best engineers I've seen are usually excellent at napkin math.
Not everyone will appreciate your authentic self, that's the inherent risk which requires bravery to overcome. It might also be simply not worth the risk, but either way you should act in a way that you can respect. There are things I wouldn't do because I consider them too risky, but I can still respect myself because I'm the one making the judgement call. It's when I act inauthentically because of the fear, not the risk, that I lose respect for myself and I do my best to avoid it.
The legal fiction that is the corporation is the antithesis of authenticity, and the brand they invest so heavily in, its mask. Having an authentic employee risks removing the mask.
What happens when something is put on the scale while it's sampling? Does the curve depend on properties of the scale, or just properties of the object and the manner in which it was put on the scale?
It's the latter. The scale is meant for real-time monitoring of rapidly varying force (the primary application is about monitoring the force derivative and repeatable max force logging). It uses an aluminum load cell if you're familiar with those, there's a tad of a multi-kHz resonance that is typically overshadowed by the object properties.
I guess everyone has their own preferences, I just find this opinion surprising given the wealth of other hikes in the country.
As someone who grew up hiking in the White Mountains before moving to Washington, the mountains in Washington (and many places in the West) are just on a whole different level.
Franconia ridge is stellar, but so is angels landing in Zion or the Half Done cables or the Kaibob trail in the Grand Canyon or many others that are hard to compare and all worth the trip.
It's higher than average/median in the US, but certainly not exceptional. Pretty normal for certain groups of people. There's a huge gap in miles driven between urban and rural area. US average is something like 13-15k miles per year (for all driving, not just commute).
20,000 miles solely for commuting would be about 43 miles each way (if you work 235 days per year), which is obviously more unusual than 20k total miles driven from all sources.
I've sometimes been confused by the term "inverted index". The example in this post feels like what I would just call an "index"... i.e documents indexed by the words they contain. Feels about the same as the index in the back of a physical book.
Is the distinction that an index on a multi-valued attribute is called an inverted index?
Inverted indexes are what databases call indexes. It's used in the IR field to differentiate from forward indexes, which are less common, so you're right that we could just say "index's.
But when we talk about inverted indexes, they are almost always term -> posting list, and most index data structures lay these out so that posting lists are sorted and compressed together. Traditional database indexes like B-trees are optimized for rapid insertion and deletion, while inverted indexes tend to be optimized for batch processing, because you typically deconstruct text into words for a large batch and then lazily integrate this batch into the main index.
Part of this is about scale; a row in a database typically has a single column or maybe 2-3 columns in a composite index; but a document text may tokenize into thousands, hundreds of thousands, or millions of words. At this scale, the fine-grained nature of words mean B-trees aren't as a good a fit.
Another part of it is that inverted indexes aren't for point queries, which is what B-trees are optimized for; you typically search for many words at a time in order to rank your search results by some function like cosine similarity. You rarely want a single posting; you want the union or intersection of many posting sorted by score.
NIT: That's not quite correct if your first statement is meant to imply an equality rather than a subset relation.
The idea of an index is more general, as an index can be built for many different domains. For example, B-trees can index monoidal data and inverted indexes are just an instance of such a monoid that a B-tree can efficiently index.
Furthermore, metric spaces (e.g., levenshtein distance) can also be efficiently indexed using other trees: metric trees. So calling inverted indexes just indexes would be really confusing since string data is not the only kind of data that a database might want to support having efficient indexes for.
My point is that all indexes are "inverted" in the sense that they map some searchable value to occurrences of said value. That is true even if method of comparison is not strict equality.
Most indexes people hear about are like that. However, there are indexes that work the other way around, like Postgres' Block Range Indexes (BRIN). They are mostly useful as skip indexes - for a given block, they have a summary that tells whether some given data may be there.
The trade-off this kind of index makes is that it is more optimized for (batch) writes than the more popular B-Tree indexes, but it is less optimized for reads on the other hand. If the write throughout of a given table is very high, you might want to remove all B-Tree indexes that are not strongly correlated to the insert order and have BRIN indexes instead. Combine it with table partitioning, and you can add B-Tree indexes in the cold partitions, or even migrate them to columnar storage if available (with the Citus extension).
By the way, a few years ago a Bloom BRIN variant was added, not to be confused with Postgres' Bloom indexes which are something else.
I wouldn't say BRIN indexes are "the other way around"; index structure is still one where data values are looked up to find the area where occurrences exist.
"Coarse" indexes like BRIN and ClickHouse's data-skipping indexes are still indexes in a broad sense of serving to narrow down a search.
A document can be viewed as an object with a set of pointers to the words it contains.
The inverse of that, was a word object, with a list of pointers to the documents it is found it. This was referred to an an inverted DOCUMENT index. This is what people would normally just call an index.
At some point, people dropped the "DOCUMENT" part, and started just calling it an "inverted index". This makes no sense, grammatically, as it's the document that is inverted, not the index, but it is what it is.
Wow, thanks for the explanation! That was driving me nuts too, as I was waiting for the point where they would invert the thing they built and what that would look like, though that point never came. But now I don't need to put in the time that you did!
In summary, they are not "inverted index" in the sense of "the inversion of what you'd normally think of as an index" but instead in the sense of "a map which provides the inversion of the map from documents to words in them, in other words, an index".
No it's the same thing. With any book you have built-in mechanism to go to a page number see what words are there. An inverted index lets you do the inverse (words -> page numbers).
People (non-tech) don't tend to refer to "go to page 106" as using an index. The pages at the back of the book providing the word->page numbers lookup are commonly known as the book's "index"
It's pointless in the sense that the word "inverse" in the term is pointless, a mild way of saying that it's confusing or even unnecessary to the point of being incorrect.
The discussion about it is not pointless since it clears up confusion. It might not have been for you, but it's clearly for many others, so if you think that's pointless then allowing yourself to appreciate other perspectives could go a long way.
An inverted map is still a map, but if you are typically thinking of the map A->B and then suddenly somebody talks about an inverted map, then it's understandable that people start to assume that this is now about B->A and get confused if it somehow actually isn't really.
If the documents where themselves stored in a database they have and id and the contents. The clustering key (an index) would be on the id. It's inverted because the contents are deconstructed into tokens with a list of ids that contain that token. Now the contents (tokens) server as the indexed value.
Is it a difference between server hardware managed by knowledgeable people and random hardware thrown together by home PC builders?
reply