- I peeked at the VVM implementation, since I've been looking for an implementation of data frames for my shell Oil [1]. I've looked at Hadley Wickham's dplyr code, R's data.table library, Pandas (which is somewhat awkwardly based on NumPy), and a little bit at Apache Arrow. I also remember R has a "zoo" library though I haven't used it much.
Was your designed influenced by any system in particular? I've had a hard time finding any descriptions of data frames other than the code. I have less experience with time series, but I believe the main issue on top of data frames is having joins by time columns (e.g. your "asof" operator).
But otherwise, could VVM could be used for dplyr-style analysis? dplyr has a very rich set of operators.
Hadley does a good job of describing the high level philosophy, but I've been looking for low level advice, like how to do vectorized math quickly with overflow checks and so forth. (Do you have ints or is everything a float?) Maybe it's not a big deal, but it's not something I have experience with. I'd like to read about someone's implementation, especially in portable C / C++. I think a lot of earlier systems were in Fortran/assembly.
I guess your implementation is fairly different because the language is statically typed. I peeked and it looks like DataFrame is std::vector<void star>, which makes sense for static typing.
Have you looked at how Julia does things? It has macros and fast code generation. I imagine you started this project before Julia 1.0, where they added NA for data frame support. I'm more of an R and Python user, but I find Julia pretty interesting, e.g. something like this approach is impossible in R or Python AFAIK:
- I watched your video, which is a nice demo. My feedback: if you want to maximize the number of people that get through it, I would make the font bigger and also raise the bottom of the window so the typed code is more readable. The code is nearly clipped off which causes some friction for viewers. Hope that's helpful.
Anyway I hope to have time to play with this a bit more. I don't have that many time series use cases but I'm definitely interested in data frames!
[1] The slogan for why a shell could use data frames is: "the output of ls and ps is a table". For those unfamiliar with data frames, here's my intro: What Is a Data Frame? (In Python, R, and SQL)http://www.oilshell.org/blog/2018/11/30.html
VVM is column-oriented, which is how pretty much every Dataframe implementation works. Each column is a vector of whatever the user's type represents; Int64 in Empirical is i64 in VVM and int64_t in C++.
VVM has its own statically typed assembly language. You can see examples of it in the regression tests; here's one that sorts a table:
Since it's a virtual machine, VVM is pretty low level and really only meant as a compilation target. While it does some of the heavy lifting to match keys or determine the order of indices in a vector, Empirical is needed to coordinate the moving pieces.
Empirical takes a very different approach from Julia. Empirical is statically typed (not "gradually" typed), is focused around Dataframes, and compiles to a VM that is then interpreted.
As I mention elsewhere, I haven't done much on the performance side of things. I eventually want SIMD and JIT, but my priority for now is getting the Empirical language right.
- I peeked at the VVM implementation, since I've been looking for an implementation of data frames for my shell Oil [1]. I've looked at Hadley Wickham's dplyr code, R's data.table library, Pandas (which is somewhat awkwardly based on NumPy), and a little bit at Apache Arrow. I also remember R has a "zoo" library though I haven't used it much.
Was your designed influenced by any system in particular? I've had a hard time finding any descriptions of data frames other than the code. I have less experience with time series, but I believe the main issue on top of data frames is having joins by time columns (e.g. your "asof" operator).
But otherwise, could VVM could be used for dplyr-style analysis? dplyr has a very rich set of operators.
https://www.rstudio.com/wp-content/uploads/2015/02/data-wran...
Hadley does a good job of describing the high level philosophy, but I've been looking for low level advice, like how to do vectorized math quickly with overflow checks and so forth. (Do you have ints or is everything a float?) Maybe it's not a big deal, but it's not something I have experience with. I'd like to read about someone's implementation, especially in portable C / C++. I think a lot of earlier systems were in Fortran/assembly.
I guess your implementation is fairly different because the language is statically typed. I peeked and it looks like DataFrame is std::vector<void star>, which makes sense for static typing.
Have you looked at how Julia does things? It has macros and fast code generation. I imagine you started this project before Julia 1.0, where they added NA for data frame support. I'm more of an R and Python user, but I find Julia pretty interesting, e.g. something like this approach is impossible in R or Python AFAIK:
http://scattered-thoughts.net/blog/2016/10/11/a-practical-re...
http://scattered-thoughts.net/blog/2018/08/16/julia-as-a-pla...
- I watched your video, which is a nice demo. My feedback: if you want to maximize the number of people that get through it, I would make the font bigger and also raise the bottom of the window so the typed code is more readable. The code is nearly clipped off which causes some friction for viewers. Hope that's helpful.
- Nice to see someone else using Zephyr ASDL! I have linked these blog posts a few times here: http://www.oilshell.org/blog/tags.html?tag=ASDL#ASDL
Anyway I hope to have time to play with this a bit more. I don't have that many time series use cases but I'm definitely interested in data frames!
[1] The slogan for why a shell could use data frames is: "the output of ls and ps is a table". For those unfamiliar with data frames, here's my intro: What Is a Data Frame? (In Python, R, and SQL) http://www.oilshell.org/blog/2018/11/30.html