
Spark 2.0.0 Released - Gimpei
https://spark.apache.org/news/spark-2-0-0-released.html
======
minimaxir
Spark 2.0 was released last week and submitted about 10 times, but the only
time it hit the front page
([https://news.ycombinator.com/item?id=12171026](https://news.ycombinator.com/item?id=12171026))
it received _zero_ discussion. The behavior of data science HN submissions is
always weird. :p

I spent the weekend playing around with Spark/PySpark 2.0 for a blog post
which will be released in a couple days. PySpark is almost on the parity of
scikit-learn in terms of available data structures and algorithms. (Although
some Spark design decisions are more enterprise-friendly than personal user-
friendly. For example,Spark 2.0 is the first update with the ability to
natively _read CSV_ s. And let's not get started on _exporting_ CSVs:
[https://twitter.com/minimaxir/status/759830773893509125](https://twitter.com/minimaxir/status/759830773893509125))

~~~
drewda
Speaking of Python wrappers for Spark, do you know if this release allows use
of GraphX from Python? Or is that still only available through the Scala
libraries?

~~~
minimaxir
You may be thinking of GraphFrames:
[https://databricks.com/blog/2016/03/03/introducing-
graphfram...](https://databricks.com/blog/2016/03/03/introducing-
graphframes.html) (apparently it is not ready for 2.0)

There is no native support otherwise.

~~~
drewda
Thanks for the pointer to GraphFrames. I was unaware that effectively
supersedes GraphX.

I had been thinking back to
[https://issues.apache.org/jira/browse/SPARK-3789](https://issues.apache.org/jira/browse/SPARK-3789)
and now that I've doubled checked that issue, I see that Python-bindings for
GraphX are moot, given GraphFrames.

------
rm999
For anyone interested in Spark, I created
[http://www.reddit.com/r/apachespark](http://www.reddit.com/r/apachespark) a
couple years ago. It's a relatively small subreddit given the popularity of
Spark, and could use more users :)

~~~
redsymbol
Thanks for creating it. It's one of only 8 subreddits I subscribe to.

