
Amazon Elastic File System – Production-Ready in Three Regions - jeffbarr
https://aws.amazon.com/blogs/aws/amazon-elastic-file-system-production-ready-in-three-regions/
======
KaiserPro
YES

Finally we can start using standard posix semantics for things.

Yes people might say its primitive, but currently all I can see is people
trying to re-create shared files systems over protocols not designed for it
_cough cough_ HTTP

Finally I can have a shared home, with a shared environment.

A single readonly binary source (great for docker by the way.) also great for
ensuring one version of scripts, without having to ansible everything.

~~~
merpnderp
At 10x the price for S3.

~~~
KaiserPro
You are correct. However its not for the same things.

S3 is high latency, HTTP semantics, no locking mechanism.

EFS is low latency, high throughput shared file system with standard posix
interface. _everything_ can write to standard files. Not many things can write
using S3 effectively.

------
jedberg
This is really cool but also kind of a shame. I feel like it will encourage
bad behavior.

I can see people using this to deploy code by deploying onto the shared volume
so that all their instances get an "instant update". Which is great when it
works, but lord help you when EFS goes down and every app server you have is
hung on a broken NFS connection.

~~~
Corrado
I think that deploying onto a shared volume is exactly the workload that EFS
encourages. That things will stop functioning can be said about any (AWS)
technology. "Lord help you when (S3|RDS|SQS|etc.) goes down and every (thing)
you have is hung up on a broken (thing) connection."

The rational argument here is that if uptime is important to you, the solution
is to utilize multiple regions, of which EFS is now in 4. You are using AWS
because of the SLA and the assumption that when something goes wrong there are
legions of technical folks trying to fix it as soon as possible.

~~~
jedberg
The difference is that all of those services are accessed via an API, which
means your client can do clever things when it fails, like timeout, give a
fallback, find an equivalent resource in another zone or region, etc.

With EFS, it's exposed as NFS, which hooks in much deeper. If it goes down,
there isn't anything you can do to work around the problem, unless you start
hacking your own file system kernel modules.

~~~
zeisss
Can't you measure and timeout on the file-descriptor writes too?

~~~
pyre
Do you want to start wrapping your system calls to read files in code that
forks to periodically check for a hung system call?

~~~
jabl
It's worse that that, actually. With hard mounts (the default), the only way
to interrupt is with SIGKILL, which if my memory serves me correctly, is a
process-wide signal. So you'd have to do all NFS I/O in a separate child
process.

With soft mounts (does EFS support this?) you can exchange SIGKILL for
religiously checking return values for all I/O syscalls, retrying partial
writes and whatnot.

------
sheraz
I think it is worth mentioning that Azure offer something similar with Samba /
CIFS/ SMB protocol. [1]

This is the underlying tech that powered their Docker volume plugin as well
[2]

I'm using in production for serving up small images to some web servers, and
I'm currently in playground with docker stuff.

So far, so good. I'm impressed.

[1] [https://azure.microsoft.com/en-
us/documentation/articles/sto...](https://azure.microsoft.com/en-
us/documentation/articles/storage-dotnet-how-to-use-files/)

[2] [https://azure.microsoft.com/en-us/blog/persistent-docker-
vol...](https://azure.microsoft.com/en-us/blog/persistent-docker-volumes-with-
azure-file-storage/)

edit: footnotes

~~~
aluskuiuc
A notable difference is that Azure file shares have a 5 TB limit for the whole
share, and a 1 TB limit on the size of any given file [1], while EFS has no
limit on the size of the file system, and a 52 TB limit on the size of any
given file. [2]

Disclosure, work for AWS.

[1] [https://azure.microsoft.com/en-
us/documentation/articles/sto...](https://azure.microsoft.com/en-
us/documentation/articles/storage-scalability-targets/#scalability-targets-
for-blobs-queues-tables-and-files)

[2]
[http://docs.aws.amazon.com/efs/latest/ug/limits.html](http://docs.aws.amazon.com/efs/latest/ug/limits.html)

------
buremba
We use a columnar database and we insert data to it periodically (30s to 90s)
using Kinesis as middleware. Files are immutable and each insert creates
[count of columns] * [count of tables] files. The database also periodically
merges the batch files in background.

Since EFS was not available, I was trying to develop a FUSE filesystem that
writes data to both S3 and local filesystem (EBS) for durability and read only
from local filesystem for performance. Even though it's cheaper than EFS, it's
slow since each file needs to be sent to S3 individually. Our requirements are
not that much because the files are immutable, there are not many files (I
heard that EFS doesn't work well with many small files.), no need for
concurrent access. I think I will give it a shot since it also fits our use-
case.

~~~
takeda
If files are immutable, isn't s3 much better fit for it? s3 main weaknesses is
that you can't modify part of a file, which makes it unsuitable for a
filesystem, but looks like it satisfies your use case.

~~~
buremba
Maybe but unfortunately the database doesn't support "backup store" that we
can use directly S3, it just writes the data to specified directory in a
filesystem. We should either fork the database or implement "S3 backed"
filesystem and we picked the latter one. Unfortunately the performance FUSE is
not that good for read and writing data to S3 is somehow expensive.

~~~
ak4g
If you don't mind me asking, what DB is this?

~~~
buremba
We're implementing Clickhouse
([http://clickhouse.yandex](http://clickhouse.yandex)) to our open-source
analytics platform Rakam. ([https://github.com/rakam-
io/rakam](https://github.com/rakam-io/rakam))

------
Corrado
I am excited by EFS but during the beta it didn't perform as well as I would
have liked. I had a small Wordpress install that I wanted to make elastic in
that I could have multiple web servers and all of the data was stored apart
from the AMI. After testing my setup I found that latency was way too high and
the performance of my site was terrible. I'm guessing it's because the amount
of data was really low (100MB) and the requests were infrequent.

I ended up going with a shared volume, which works fine and performance is
great, but I can only attach the volume to a single instance at a time which
prevents me from running multiple instances. That said, it looks like there
has been some performance tweaks to EFS so maybe it might be better this time.

~~~
jeffbarr
We addressed this use case during the preview. Could you try it again now?

------
jakozaur
Price is a bit high $0.3 GB / month vs. $0.1 GB / month for SSD EBS and even
less for local storage. S3 is even cheaper $0.03 GB / month. So EFS is 10
times more expensive than S3.

On I/O front also EBS (gp2) and (st1) are several times cheaper.

So most of EFS advantages are in flexibility and sharing same file system
across many instances. This may be great e.g. for sharing readonly binaries,
but large scale data pipeline likely can be done much cheaper using different
technologies.

~~~
KaiserPro
You are right for small scale things.

I'll have to double check my working out, but you're only paying for what you
use, not provision, which is one big things.

Also, if you have > 3 machines working from the same dataset (ie docker
image/video/large binaryblob) there is an instant saving, without factoring in
provisioned vs useable space with EBS

but, your original point, as a 1-1 replacement of a properly sized EBS mount,
is correct, its more expensive.

~~~
pmalynin
Nitpick

"large binaryblob" is literally "large binary binary large object"

~~~
KaiserPro
Haha, point taken.

------
kldavenport
Pricing looks competitive
[https://aws.amazon.com/efs/pricing/](https://aws.amazon.com/efs/pricing/) I
wonder if this helps those who perceive S3 as vendor lock-in since it
replicates a traditional file system.

~~~
LeoPanthera
Does it? It looks significantly more expensive than, say, rsync.net.

------
Cidan
We've been in the EFS preview for quite a while now, and we are incredibly
pleased with it. We do a few different things using EFS, including using it as
an output for some of our Spark jobs.

It's really a top notch product; couldn't be happier.

------
objectivefs
Cool! If you are looking at shared filesystems on EC2, you might also want to
take a look at ObjectiveFS[1]. Works in all regions and on GCS, and you can
mount your filesystem securely (end-to-end encryption) between regions.

[1]: [https://objectivefs.com](https://objectivefs.com)

~~~
arqq
ObjectiveFS looks awesome and I've been wanting to use it, but using non-S3
storage is locked to "enterprise, I can't afford it". :(

But I also can't afford S3's bandwidth pricing.. I could, however afford to
run an S3-API compatible service myself with bandwidth pricing I could afford.

But then you wouldn't let me use it because then I'd be "enterprise".

~~~
objectivefs
Great that you have been wanting to use ObjectiveFS. We are happy to talk with
you about your non-S3 storage use case and see if we can find a plan that
works for you.

------
atrilumen
Can EFS be mounted from the Lambda environment?

~~~
jeffbarr
Not at present. I would be interested in learning about use cases for this.
Post here or find me online, as you wish.

~~~
jedberg
One use case would be one that came up a while back here on HN -- The person
had a large binary blob that they needed to load up every time their Lambda
function ran. If they could mount EFS from Lambda, they could put the large
binary blob there.

Depending on how they wrote their code, if the Lambda could seek to the right
part of the file then this would work, but if it has to load the entire file
into RAM anyway, then it wouldn't gain them much except perhaps a slightly
easier way to get the file into RAM.

~~~
boucher
Couldn't you just download the portion of the one file you want from S3?
Unless EFS is significantly faster...

From a caching perspective, you wouldn't seem to gain much in lamda. You can
cache in memory/tmpfs until your container eventually dies, and then you
likely aren't on the same machine so there's no EFS level caching to take
advantage of.

~~~
jedberg
Right that was my point. You most likely have to load the whole thing into
memory, but in the rare case that you can seek directly to the right spot in
the file, ECS might have an advantage for you in Lambda.

~~~
boucher
Yeah, I just meant that S3 has partial downloads, so you can still seek.

EFS/NFS makes a lot of sense I think when you have a lot of
random/unpredictable access across large numbers of files. You could do the
same thing with FUSE/S3 but across lots of smaller files/accesses I would
think the overhead adds up faster.

------
ngrilly
To store small files (10 to 1000 KB) uploaded by users in a web app, what is
the best tool, S3 or EFS?

~~~
RossM
Object storage (i.e. s3+cloudfront) is still the correct way to store
immutable files. that said, I know this will be used when the options are to
rewrite filesystem access to an abstract (i.e. fs/s3/azure/etc) or just use
nfs. People already use s3fs-fuse[1] which is dubious at best.

[1]: [https://github.com/s3fs-fuse/s3fs-fuse](https://github.com/s3fs-
fuse/s3fs-fuse)

~~~
ngrilly
Would you say that the main choice criteria is immutability?

Immutable files => Object storage => S3

Mutable files => File storage with POSIX semantics => EFS/NFS

------
fulafel
Any ideas what they are using for NFS servers software?

------
deskamess
Can you expose the EFS file systems to CloudFront? For a future migration
(local to cloud) I would like a backend application running on EC2 to use
standard path names, but public access would be through CloudFront. It would
be easier to migrate existing apps to the cloud with that approach, and then
slowly work on changing the apps to use S3 (for cost reasons).

------
mike503
Why hasn't anyone mentioned ObjectiveFS? When EFS was in preview just a month
or so ago it was absolutely abysmal with performance for basic smaller web
stuff.

OFS feels nearly like an attached disk, and uses S3. You only pay per mount.
Their support has been awesome and very personal.

It provides a fuse-based full POSIX filesystem backed by S3 and can be mounted
from multiple places in multiple regions. Some basic things like snapshotting
they said are coming soon too.

I gave up on EFS and even though it's finally out of preview, I think I still
am going to prefer OFS based on what I've seen so far...

------
gtirloni
I don't have a lot of experience with pNFS but it's supposed to be an
extension to NFS 4.1, which Amazon EFS is supporting. However, pNFS is not
mentioned in this blog post, their website or the official docs.

That would be an interesting feature.

------
s17tnet
Finally! My client was unable to use AWS for lacking of real POSIX fs.

------
merb
I wonder if some can run a postgresql database on top of that. This would make
it relativly easy to have a master without downtime and be way cheaper than
RDS.

~~~
Sanddancer
It claims to present itself as a full filesystem so I don't see why it
wouldn't, however, you're probably going to run into pretty hardcore
performance issues, because the bandwidth amazon provides in general is on the
stingy side, but under a terabyte, they're downright miserly, with a 10 gig
filesystem only able to provide 0.5 MB/s sustained. This product is
problematic at best.

~~~
Someone
Also, it is a full-fledged _NFS-mounted_ file system.

I would be very wary of running a database from that.

~~~
MrMullen
It sounds like a disaster waiting to happen. The latency of NFS is just too
high (Even if it is very fast for NFS) for it to be safe for a database.

~~~
SteveNuts
We're running Oracle on NFS volumes, and some MySQL databases also with no
problems whatsoever.

It'll never be as fast as local disk but the flexibility it provides is very
nice.

~~~
Sanddancer
That's normal NFS. Amazon's offerings, if you have a 10 gigabyte database, you
can only get 500KB/sec bandwidth --

[http://docs.aws.amazon.com/efs/latest/ug/performance.html](http://docs.aws.amazon.com/efs/latest/ug/performance.html)

~~~
SteveNuts
Huh. well that's kinda shitty.

Our NetApp can saturate 10GBe without even trying.

------
bhouston
Are there any third party performance metrics yet? Read/write ops per second,
and read/write bytes per second from one or many machines?

Would love to see this in comparison to a 10Gbps (or 1Gpbs) attached spinning
rust or SSD NSF drive. That would really help me understand what tradeoffs are
happening here.

------
spullara
I really want to be able to access EFS file systems from Lambda. Has anyone
figured out a way to access NFS?

------
madbiz
Can EFS be used as a volume for ECS tasks?

~~~
bbgm
Here's a post we wrote earlier in the year about EFS and ECS:
[https://aws.amazon.com/blogs/compute/using-amazon-efs-to-
per...](https://aws.amazon.com/blogs/compute/using-amazon-efs-to-persist-data-
from-amazon-ecs-containers/)

~~~
madbiz
In the blog post the EFS is mounted by the EC2 instances from the start but i
expected to be dynamic depending on the scheduled tasks on the instance. I
dont want to couple EFS volumes which might be used by the containers if they
might get scheduled on an instance. I would like to have a general purpose
cluster and let the scheduler instrument the EC2 instance to mount an EFS
volume if its needed by a scheduled task dynamically.

------
agperson
Glad to finally see this in production, wish it was available across VPC
peering, VPNs, etc.

------
api
This is pretty awesome.

Even though we don't use AWS much due to lack of IPv6 support, we've wanted to
have a giant bottomless FS for various purposes. It'd be easy to set that up
in here and export it over a virtual network.

------
TheGuyWhoCodes
Does EFS provide better throughput and latency than S3?

------
boraturan
Seems like no Windows support?

~~~
Corrado
That's a good question. Originally it was Linux only as the EFS client was
locked to NFS 4.0 and the Windows Server NFS client was 4.1 (why 4.1 is not
backwards compatible, I don't know). However, the announcement mentioned using
the PowerShell cmdlets to attach EFS volumes, and in fact the PowerShell help
pages have specific commands for working with EFS. So, my guess is that it is
now available for Windows and Jeff didn't do a very good job of bringing that
to light.

~~~
rincebrain
The EC2 example shows mounting with nfsvers=4.1, so I'm guessing it got
resolved by bumping the version supported.

------
rymohr
Still no support for incremental snapshots though right?

------
Sanddancer
This feels rather...primitive for a network filesystem introduced in 2016.
Where are the snapshots? Where are the subvolumes? Where's the ability to
send/receive volumes/subvolumes to filesystems in other availability zones?
There are also I/O limits if you have a smaller filesystem with frequently
accessed data, an inability to mount on machines through vpn gateways, and a
lot of other seriously rough edges.

It's an interesting start to a project, but the public features feel very much
lacking.

~~~
rincebrain
It seems like this is more of a glue layer between a more interesting backend
service (EFS) and clients, but I don't know that I'd expect Amazon to do
something like write a native client, versus just probably improving the NFS
client behavior on Windows/Linux/...

The neat trick would be if NFS 4.X grew support for communicating interesting
operations like snapshotting.

