
Show HN: Pfs – A Git-like distributed file system for the Docker ecosystem - jdoliner
http://pachyderm-io.github.io/pfs/
======
tiles
> Pachyderm will eventually be a complete replacement for Hadoop, built on top
> of a modern toolchain instead of the JVM.

Is the JVM legitimately not considered a modern toolchain? I understand the
benefits of leveraging Docker, but this comparison seems short-sighted.

~~~
alrs
The bummer about the JVM is that the first piece of advice you get when things
are going wrong is "you're using Oracle Java, not OpenJDK, right?"

There are situations now where the JVM is unavoidable. It's an increasingly
poor choice for greenfield development.

~~~
necubi
Since Java 7, Oracle Java and OpenJDK are nearly identical [0]. There's little
reason to use the former instead of the latter. Previous to 7 OpenJDK had a
much worse performance than the Sun/Oracle JDK, but that hasn't been true for
a while now.

It remains that the JVM is incredibly mature and performant, and almost any
library you could want has been implemented on it. It also remains a fruitful
ground for new language development, although LLVM is an increasingly popular
alternative.

[0] [http://stackoverflow.com/questions/22358071/differences-
betw...](http://stackoverflow.com/questions/22358071/differences-between-
oracle-jdk-and-open-jdk-and-garbage-collection)

------
doublerebel
Very creative way to combine the pros of recent fault-tolerant, distributed
systems. Looks like you are the same jdoliner behind RethinkDB?

The link "git-like semantics" points to an article about ZFS -- not btrfs. I
was already thinking of ZFS because Pfs reminds me of Joyent's Manta. Manta
also provides virtualized containers for data where compute/MapReduce jobs can
be run directly. I love ZFS and have been relying on it for years. Which makes
me wonder:

* Why Btrfs and not ZFS? Licensing? Features? Lack of availability of ZFS hosts? [1]

* Joyent just open-sourced Manta, is the timing a coincidence?

* Manta provides "SnapLinks" for version control but otherwise does not really leverage ZFS to provide a history. What other differences and advantages does Pfs aim to provide?

Looking forward to how this project evolves.

[1] I own zfshosting.com since I see a need for affordable, semi-managed,
online ZFS storage. If you share my vision please contact me.

~~~
jdoliner
> Looks like you are the same jdoliner behind RethinkDB There are many great
> people behind RethinkDB, 2 of them are involved in pfs actually. I am
> however the only one named jdoliner :)

> Why Btrfs and not ZFS? Licensing? Features? Lack of availability of ZFS
> hosts? [1] This came down to minimizing dependencies, Docker uses btrfs so
> we know it's going to be installed anywhere we deploy. I actually think that
> ZFS is better and am interested in using it in the future.

> Joyent just open-sourced Manta, is the timing a coincidence? Yeah total
> coincidence, I haven't gotten a chance to read a ton about it but it seems
> interesting. One big difference is that we intend to fill out a full set of
> git-like with things such as branching and merging. Another is that our
> model of MapReduce is very deeply intertwined with this set of git-like
> semantics.

Thanks for the kind words and zfshosting looks awesome. I'd love to chat about
ZFS with you sometime.

~~~
doublerebel
Very neat. Btrfs is a practical choice. Looks like the Go http server listens
for both changes and jobs?

Could you tell me about the advantages git semantics add to MapReduce?

Last, how does Pfs choose to distribute data across containers?

You've inspired me to put up an Unbounce page. Let's talk ZFS soon!

~~~
jdoliner
> Very neat. Btrfs is a practical choice. Looks like the Go http server
> listens for both changes and jobs?

Right now pfs is just a file system so servers only listen for changes.
However longterm this is going to be the case because submitting a job is
actually going to be implemented as making a commit in the file system that
tells pfs which data to run the job over and the container to use to do so.

> Could you tell me about the advantages git semantics add to MapReduce?

One is that it gives you an audit log which is very useful for debugging. I
often find myself with data in HDFS that doesn't look like what I expect it to
and wish that I know what process changed it.

Another killer feature is that it removes the distinction between stream
processing and static MapReduce. Since our file system is commit based we
think of MapReduce as an operation over a stream of commits rather than a
static set of files. When more commits come in you can recompute your job for
only the changes.

> Last, how does Pfs choose to distribute data across containers?

Right now pfs uses a simple static hashing algorithm based on filename. This
probably isn't going to work for production deployments. We're planning to
improve it soon.

> You've inspired me to put up an Unbounce page. Let's talk ZFS soon!

Yes please.

------
ComNik
Is there a distributed, ZFS-like filesystem usable in clustered production
environments today?

I'm intrigued by the idea of Git-like filesystems, but am not aware of any
great clustered filesystems at all.

~~~
jdoliner
I don't know of one and I did a decent amount of looking while developing pfs.
Hopefully we can eventually meet your needs!

------
Yacoby
I am not sure I see the point in implementing MapReduce when most of the
current work seems to be in generalising MapReduce - i.e. Apache Spark and
YARN. Is there any reasoning behind this?

~~~
coffeemug
Not the author of the project but I can think of two reasons.

Firstly, you can think of map/reduce as the infrastructure for higher level
operations (sort of like the assembly language of large scale data processing
that higher-level data processing systems compile to). A breakthrough in the
quality of the operational engine significantly impacts the experience of
doing higher-level work, so if someone finds a better way to run map/reduce
jobs, it's a win for everyone. Shipping jars instead of docker containers, and
not having snapshots are serious drawbacks in the existing map/reduce
infrastructure that significantly impact users in negative ways.

Secondly, an easier way to specify map/reduce jobs (via a simple web server
that exposes API endpoints to do data grouping, mapping, and reduction) is a
dramatically simpler, more composable way to expose map/reduce jobs. Building
higher level infrastructure on top of this abstraction is an order of
magnitude easier than doing it on top of Hadoop, so it could be a better
underlying platform for the generalization work being done in the community.

~~~
jdoliner
This says what I was trying to say a lot better than I did. +1 coffeemug.

------
rectang
What's the license? I don't see anything in any of the source files, nor the
README, nor any files such as LICENSE/COPYING/etc at the top of the repo.

~~~
jdoliner
We actually haven't spent time thinking about this yet. Sorry about that we'll
have one up soon.

~~~
Hz8NSD
I tend not to read tutorial/documentation until license is clear. Hope to see
LICENSE file soon.

Q: It looks like could do `curl` on pfs. Is that mean pfs act as basic http
server like AWS S3?

~~~
jdoliner
Hi,

Just added a LICENSE file (Apache 2.0).

> Q: It looks like could do `curl` on pfs. Is that mean pfs act as basic http
> server like AWS S3?

Yes!! Pfs has a microservice architecture which means it's made entirely of
http servers which communicate with each other and users over RESTful APIs. To
your point, this means pfs can be used as a simple webserver for static
content via the `/pfs` route.

One of the features we'll be building in the near term is a web interface that
users can point their browsers at to check on the status of the cluster and
browse data stored in it. This is most likely going to just be implemented by
injecting the .html, .css, .js etc files directly in to the file system.

------
tveita
Maybe I'm misunderstanding how these pieces will fit together, but from what I
can see, the file system part is a fragile implementation of master-slave
replication, which isn't what I would describe as a "distributed file system"
like HDFS.

------
jrcii
So I could potentially run pf and pfs on the same server?

