>I want to have just indexes to do the queries I need to do with insane speed
couchdb seems to fit this requirement really well. You explicitly create indexes on the fields that you want to query. The index creation part is slow but once it's done your queries are really fast. All data is compressed with snappy[1].
couchdb also encourages you to split your data across multiple databases. Effectively you can have thousands of databases all managed by a single couchdb server instance. You can move your data in temporary databases and "purge" them when they are no longer required. It's all really cool once you get a hang of how to use this feature. Although if you query across databases you'll have to "join" the result set within your application.
You should give it a try, you'll really like it :)
Indexes are cached on disk. As the new data comes in that satisfies the query function,that you have written,it will automatically be appended to the index that has already been built up to that point.
The query results are retuned with Etags that you can cache like you do any http resource.
couchdb seems to fit this requirement really well. You explicitly create indexes on the fields that you want to query. The index creation part is slow but once it's done your queries are really fast. All data is compressed with snappy[1].
couchdb also encourages you to split your data across multiple databases. Effectively you can have thousands of databases all managed by a single couchdb server instance. You can move your data in temporary databases and "purge" them when they are no longer required. It's all really cool once you get a hang of how to use this feature. Although if you query across databases you'll have to "join" the result set within your application.
You should give it a try, you'll really like it :)
[1]https://github.com/google/snappy