
2017 Outlook: Pandas, Arrow, Feather, Parquet, Spark, Ibis - TheAlchemist
http://wesmckinney.com/blog/outlook-for-2017/
======
aorth
_My goal is to deliver the same quality pandas user experience on 10x as much
data. pandas works well on 1GB of data, but less well on 10GB. This has to
change for pandas to remain a relevant tool in the future._

Awesome! Great to see such honest commitment. Pandas is already a great
library, and I can't wait to see the progress!

------
pixelmonkey
Thank you for all you do for the Python and data community, Wes! Arrow and
"native" parquet support for Python are very exciting developments.

~~~
lintiness
i've migrated back to r for various reasons, but wes really is a beast in the
python finance world.

~~~
peatmoss
I'd expect that improvements to Arrow, Feather, Parquet etc. will help lift
lots of data communities including those of us who also do R. I'm really
looking forward to reusable in-memory dataframe structures like Arrow to be
the foundations for R/Pandas-like data munging in lots of languages that
aren't R or Python

------
ttub
Very cool, but does anyone know more about the plan to offer a "lib pandas"
mentioned here?[https://mail.python.org/pipermail/pandas-
dev/2015-December/0...](https://mail.python.org/pipermail/pandas-
dev/2015-December/000395.html)

~~~
peatmoss
Hoping Wes will chime in here if I'm wrong, but I'm guessing that the
technologies he discusses here will more or less take the place of a lib
pandas.

~~~
wesm
No, it's all closely inter-related. pandas as a project is responsible for
data manipulation and in-memory analytics. These other projects are all
complementary technologies.

------
mojoe
Feather is awesome, and I really hope that Dask
([http://dask.pydata.org/en/latest/](http://dask.pydata.org/en/latest/)) can
eventually integrate it so that we have a lightweight way to write to/from
disk in parallel.

------
Nexxxeh
I was expecting a post about Microsoft's email client.

An annoying side effect of using commonly used words as product names. Signal
being the most egregious offender recently.

The capital O of Outlook didn't help.

I know nothing of Python, R and big data, but things are looking up by the
sound of it?

------
SolarNet
At first I thought this was a list of animals going extinct or something.

------
xapata
I hope the API becomes more Pythonic and less NumPy-ic. A little more purity
would be practical in this case.

------
ipstone2014
Nice road-map and Kudos to Wes for the great work/tools for the data
community!

------
filereaper
Found out about Apache Kudu, written in C++...

Are these systems moving away from the JVM now?

