

S3Proxy allows applications using the S3 API to access other object stores - gaul
https://github.com/andrewgaul/s3proxy

======
kawsper
I was considering something like this to have a local (or local networked)
cache store for S3, I am doing 99,9% of readings, so it could be nice to put
the cache in the S3-layer.

Does anyone knows if that exists?

~~~
peedy
I think you can setup nginx proxy with cache. Unfortunately, that's easy only
for public files and authenticated requests won't save you much (for that you
can also write a small script to authenticate yourself if there's a cache
hit).

~~~
kawsper
Oh, yes, thanks for the suggestion. I will see if I can use NGINX as a proxy,
all my files are public anyway.

------
rajbala
How does this work when the object stores don't support all the features of
S3?

For instance Atmos doesn't have public buckets, DNS names for buckets, named
keys implemented like S3, regions, etc.

But it does have features that S3 doesn't have like byte range updates,
erasure coding, etc.

~~~
gaul
S3Proxy cannot provide more that the S3 API allows, such as byte range
updates, although some features can be enabled by bucket-level policies.
S3Proxy can emulate features that other providers lack like public buckets and
could provide other features like bucket-in-hostname. Apache jclouds provides
the underlying object store compatibility so S3Proxy will benefit from
improvements to it.

------
bdunbar
Little help, please.

We have a process that uses s3express (aka using the S3 API) to copy files
from local filesystem to S3.

S3Proxy would allow the process to copy files to [another_server, atmos,
azure] ?

~~~
ihsw
Yes.

------
feintruled
This is great timing for me, currently working on trying to iron out some bugs
in a project with S3, and this helps me eliminate one line of inquiry. Nice
project!

------
xentronium
Shouldn't this be solved on the client level via proper adapter support?

I like how it can be attached to a black box though.

~~~
khc
[the author of s3proxy is my friend]

It seems silly for every client to implement support for every single object
store out there. Libraries like jcloud can handle it, but then it's tricky if
your client is not written in Java.

Also, sometimes it's not trivial to add support to existing projects. For
example, you can use s3fs with s3proxy to store your files on non-s3 object
stores.

------
gamegoblin
Very cool project. What sort of latency does the interface add? I imagine not
much, but I didn't see any stats in the readme.

~~~
goddamnsteve
Agreed. Even I was looking for the statistics in the Readme. May be, they will
add after seeing your comment.

~~~
mattmanser
Surely the latency cost would be trivial compared to the inherent latency of
using an off-site storage anyway?

I'm not sure why you're both asking for it, bit strange given how tiny the
cost of using a generic interface is compared to the cost of sending a file
over the internet.

------
jsmeaton
Does this work with the s3cmd CLI?

~~~
gaul
Unfortunately s3cmd requires bucket-in-hostname support which S3Proxy lacks:

[https://github.com/s3tools/s3cmd/issues/149](https://github.com/s3tools/s3cmd/issues/149)

[https://github.com/andrewgaul/s3proxy/issues/8](https://github.com/andrewgaul/s3proxy/issues/8)

I also added a compatibility list wiki:

[https://github.com/andrewgaul/s3proxy/wiki/S3Proxy-
compatibi...](https://github.com/andrewgaul/s3proxy/wiki/S3Proxy-
compatibility-list)

