
FishStore: A new storage layer for fast ingestion and indexing - skyprophet
https://github.com/microsoft/FishStore
======
skyprophet
FishStore is a new ingestion and storage layer for flexible- and fixed-schema
datasets. It allows you to dynamically register complex predicates over the
data, to define interesting subsets of the data. Such predicates are called
PSFs (for predicated subset functions).

FishStore performs partial parsing of the ingested data (based on active PSFs)
in a fast, parallel, and micro-batched manner, and hash indexes records for
subsequent fast PSF-based retrieval. To accomplish its goals, FishStore
leverages and extends the FASTER hash key-value store, and uses an unmodified
parser interface for fast parsing (we use simdjson in many of our examples).

FishStore is being open-sourced as a research prototype, by researchers from
Microsoft Research and the University of Utah. You can read more about the
concepts behind FishStore in the SIGMOD 2019 research paper. Note that the
research paper uses Mison as its parser, whereas this open-source release of
FishStore provides a generic parser interface model, with simdjson as an out-
of-the-box example.

You are more than welcome and try it and give us some feedback!

