
Rclone banned from Amazon Drive for having the encrypted secrets in code - msravi
https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314
======
teraflop
For context: Amazon Cloud Drive is a Google Drive/Dropbox analog, except that
it provides "unlimited" storage space and bandwidth for $5/month. The built-in
user interface requires you to manually upload/download files through a web
browser, but the service also supports an API for programmatic access. So
tools like rclone and acd_cli were developed to let you do bulk transfers
and/or mount your storage as a network filesystem, optionally with encryption
(which defeats any attempts Amazon might try to use for deduplication).

Now Amazon is suffering the obvious consequence of offering unlimited storage:
people are using it to store tens of terabytes of media and/or backups at very
low cost. In an attempt to kill off heavy users, they shut down registration
of new API keys several months ago, and now they're systematically revoking
the API keys used by popular open-source tools.

~~~
brandur
Yeah, this is a hard one. There are users on /r/DataHoarder/ that claim to
have uploaded literally 100s of (encrypted) TBs to Cloud Drive, which is
plainly unsustainable and abusive.

On the other hand, they've also killed the product for a lot of more
legitimate users as well. The Amazon web interface and apps for Cloud Drive
are obnoxiously terrible, and Rclone really is just a better way to use it.
I've been using it to sync 10s of GBs of photos between all my different
computers, but with Rclone unavailable, I'll have to fall back to Google
Drive, S3, or some other option (the unlimitedness of Cloud Drive was good
peace of mind).

I'll be keeping an eye on it over the next few weeks to see whether shipping a
binary with OAuth secrets was actually the reason for the ban, or just a
pretext for getting the Rclone users off the service (personally, I suspect
the latter).

~~~
Oletros
If a service claims that they provide UNLIMITED storage why using it is
abusing?

~~~
burkaman
abuse: use (something) to bad effect or for a bad purpose; misuse.

Abuse doesn't mean breaking the rules. It's like going to an all-you-can-eat
buffet and staying for a week. Or taking a job with unlimited vacation time
and coming to work once a month.

~~~
milcron
They should declare limits, then.

~~~
nkrisc
I bet in the ToS there are likely limits spelled out, or barring that, a
clause that grants Amazon to right to shut you down if they unilaterally
decide you're "abusing" the system.

~~~
Oletros
I have looked at the TOS in Spain and you won the bet :)

They can shut down your account for the reason you stated

~~~
Dylan16807
Yeah, never make a bet that a company _doesn 't_ have a vaguely worded way to
kick off whoever they want at any time.

But it would be nicer for everyone if they could just honestly state a number
of TB.

------
geofft
From the discussion at

[https://www.reddit.com/r/crypto/comments/61zupq/amazon_drive...](https://www.reddit.com/r/crypto/comments/61zupq/amazon_drive_developer_guide_dont_build_apps_that/)

I learned that Amazon Drive is a service that provides _unlimited_ storage for
a fixed cost, and that various applications, including rclone, use it for an
application that I would expect (as a developer) to use S3 for, and therefore
to charge my users per amount of data stored. (Amazon Drive is not part of
AWS, I believe.)

I am not extremely surprised that the terms of service of Amazon Drive make it
hard to build a compliant application like what rclone is trying to build. I'd
expect they can do this with S3 straightforwardly -- at the expense of users
needing to spend O(n) instead of O(1) on their data.

~~~
bad_user
I paid for Amazon Drive in order to do backups, some of them with rclone. I am
not using Amazon Drive for commercial purposes.

They said Unlimited storage, they should provide Unlimited storage, otherwise
it's false advertising.

In other words, I hate apologists.

~~~
chaosfox
they are providing unlimited storage sure, but they also say they can
terminate your account at any moment for any reason.

If you stop to think about this part for a second you will realize this
service is completely inadequate for backups.

~~~
scott00
The difference here is they splash "Unlimited" on their website in 32 point
font in the 1 sentence summary, and put "at our discretion" buried in their
2500 word terms of service. If they want to operate the service under those
terms, fine, but they should have to advertise "as little as 0 bytes of
storage", not "unlimited".

~~~
xiphias
This is easy to solve: make font size legally binding. If 2 statements in an
advertisement don't agree on something, the more visible statement should
overwrite the other

------
niftich
In a comment of mine earlier this week [1], I (along with others) speculated
whether the shipping of OAuth2 secrets was the reason for the ban, and whether
they're other contributing, strategic factors. I wrote:

 _Amazon has revoked rclone 's OAuth2 API key. However, consider that rclone's
default OAuth2 client id and secret are compiled into the rclone executable,
and thus effectively public; aka. anyone can extract them and pretend to be
rclone, and fool users into obtaining access and abuse them for unrelated
purposes._

 _A far better option is for the cloud provider to let users generate their
own OAuth2 clients, such as Google does (and supposedly Microsoft, although
for me it 's always errored out). Unfortunately, Amazon has a "call us" style
of Developer access, which effectively translates to no new API access being
granted to these types of users._

 _The speculation around the web is that Amazon also wanted to shut this down
because they offered "unlimited" storage, and people were using it to store
very large amounts of hard-to-compress, hard-to-dedup data. Breaking a popular
tool used to accomplish this (e.g. it supported on-the-fly encryption,
producing the exact style of difficult data) will cause some portion of less
profitable users to migrate elsewhere. This may or may not be true, but it's
certainly an intriguing point._

Some companies like Google allow easy registration of OAuth2 clients, so any
rclone user can make their own API keys at console.developers.google.com and
feed those to rclone, instead of having to use the built-in credential.
Rclone's documentation refers to this ability, but positions it as a
performance improvement for advanced users to sidestep a shared rclone-wide
quota, rather than for any other purpose.

However, it, unlike many, many other applications, at least lets you plug in
your own credential -- this shouldn't be an exotic flow, but rather the normal
way to grant access to a third-party application, instead of distributing a
hardcoded secret. Sadly, the app developer community has been slow to realize
and demand this, and providers have been slow to implement it. Google lets you
register as many applications as you want; GitHub even lets you make your own
OAuth2 scopes. But there's few others who come close.

[1]
[https://news.ycombinator.com/item?id=14380106](https://news.ycombinator.com/item?id=14380106)

~~~
eridius
> _Some companies like Google allow easy registration of OAuth2 clients, so
> any rclone user can make their own API keys at console.developers.google.com
> and feed those to rclone, instead of having to use the built-in credential_

How does making new Google API keys help with an Amazon service?

~~~
dekhn
It doesn't, but the point stands that users should be able to create their own
auth keys and use them to authenticate a third party application to the target
service, rather than having the app author bake some arbitrary key into the
code.

~~~
eridius
Well yes, that would be nice, but it sounds like Amazon Drive doesn't operate
that way, so rclone has no choice but to use a single shared API key.

------
firloop
Link to the infringing source, for the curious:

[https://github.com/ncw/rclone/blob/a9d29c22645c9b5d2ab938ebb...](https://github.com/ncw/rclone/blob/a9d29c22645c9b5d2ab938ebbbed0809a2044087/amazonclouddrive/amazonclouddrive.go#L37-L38)

~~~
bm1362
So is this basically sharing the same API key? Wouldn't that make using this
insecure, as you'd be seeing the same data as other rclone users?

~~~
koolba
Not if the source data is encrypted using a local key. What may be possible is
deleting other user's data as I believe it'd all be authorized with the same
API key.

~~~
tgsovlerkhgsel
In OAuth, you have basically two tokens: One for the "software"/"third party
service provider", one for the account itself.

So for example, if you want to use SomeMagicTweetTool to send tweets, they use
their token to prove it's them, and you grant them a special token for your
account.

Someone who has their token can pretend to be them, use their quota if there
is per-software/per-provider quota, but cannot access your account unless you
gave them a token for your account.

------
jedberg
This is a perfect example of the tragedy of the commons. No one person feels
the impact of abuse, but ends up ruining it for everyone.

Clearly they didn't intend for this service to be a general backup system.

~~~
SeanDav
This is a perfect example of why companies should never offer unlimited
services, if they don't expect users to actually take them up on the offer...

While I can understand and agree that fair-play should be a part of the
service, if the offer on hand is _Unlimited_ , then it is not abuse to upload
tera-bytes, peta-bytes, exa-bytes or even zetta-bytes.

Amazon is hardly a neophyte as far as understanding big data, they are very
well placed to know that offering unlimited storage could actually attract
really big chunks of data.

~~~
huebnerob
I guarantee that every company that has ever marketed an unlimited service has
internally raised this question. And the answer always is, "we'll just boot
the most demanding 0.01% of customers when necessary." This is a tried and
true tactic, no one bats an eye.

~~~
harshreality
Arbitrary and capricious.

    
    
        while (1) {  for user in userlist {
            # Top .01% of users are abusive and we boot them
            if usage_distribution_top(user, .0001) {
                disable_account(user);
                send_acctdisabled_notice_to_user(user, "Excessive usage of unlimited storage");
            }
        } sleep 1; }
    

Hours later: "How have we lost half our customers? We only ever removed the
top .01% for their 'abusive' usage of our unlimited service!"

If the concept isn't clear to some people, the problem is this: You can't
repeatedly cull the outliers — the top x% of the distribution — because by
culling you shift the distribution to create more outliers.

I'm sure businesses recognize that, intuitively if not explicitly in their
planning meetings, and therefore cull rarely enough that it doesn't impact the
bulk of users (in part due to influx of new users). So you cull the top x%...
who decides what x is, and why, if you state in your promotional material that
the service is unlimited? "If we set x to .001% and cull monthly, it'll help
our bottom line. Those users are scum data hoarders anyway and we won't serve
their kind with our 'unlimited' storage service." That should be lawsuit-
worthy.

Companies in this situation should be forced to specify a fixed usage limit
(exactly what these companies _don 't_ want to do), or live with the outliers.

~~~
bigdatababy
They create a kind-of-fixed limit.

They don't create a running list of the top X% of usage and ban those users.
They study their customers usage for some period and find the top X%, then
translate that to a fixed upper limit. Then they punish anyone who goes over
that fixed limit. If their customers or technology changes, they can re-
calculate that number.

As far as "unlimited": marketing is full of bullshit words that don't mean
what you think they mean. Courts (no wait, its all company-paid arbitrators
now) unsurprisingly favor the companies.

When the Supreme Court sided with AT&T on the binding arbitation clause, no
joke I got an email from Charter Communications the very same day the ruling
was released that said they updated their terms of service to include binding
arbitration.

~~~
harshreality
Then _WHY NOT STATE THAT SEMI-FIXED LIMIT TO BEGIN WITH AND SAVE EVERYONE THE
HASSLE OF TRYING TO GUESS WHETHER THEIR USAGE WILL BE ALLOWED OR NOT_?

If the limit is dynamic and periodically re-evaluated, then you run into the
"there's always an outlier" problem.

If the determination isn't dynamic, and Snake Inc. advertises its service as
'unlimited' instead of _stating the limit_ , and then starts discriminating
against users or tools associated with higher usage, and get a reputation of
being a snake in addition to actually having an undisclosed cap, what then?
They'll not only lose their high-end customers to other services (which
apparently they don't want or they could have disclosed the limits _to begin
with_ ), but as already mentioned they also hurt their reputation, because
whatever the courts say, people are not happy when an unlimited service
actually discriminates against high-usage customers.

------
markwillis82
Notice our backups failing over the weekend. Set up a new account with
backblaze and re-ran the backups.

It sounded to good to be true, fortunately we were still on the trial.

~~~
rsync
Backblaze is also unlimited, flat rate storage and is therefore _also_ too
good to be true.

"paying a flat rate for unlimited storage, or transfer, pits you against your
provider in an antagonistic relationship. This is not the kind of relationship
you want to have with someone providing critical functions."[1]

[1] [http://blog.kozubik.com/john_kozubik/2009/11/flat-rate-
stora...](http://blog.kozubik.com/john_kozubik/2009/11/flat-rate-storage-
services-vs-rsyncnet.html)

