
Introducing Google Cloud Storage Nearline - paukiatwee
http://googlecloudplatform.blogspot.com/2015/03/introducing-Google-Cloud-Storage-Nearline-near-online-data-at-an-offline-price.html
======
omh
The detailed pricing is at [https://cloud.google.com/storage/pricing#nearline-
pricing](https://cloud.google.com/storage/pricing#nearline-pricing)

In short, if I'm understanding correctly:

    
    
      $0.01 per GB per month storage
      $0.01 per GB retrieval
      Normal egress fees on top, so additional ~$0.10 per GB if you want to retrieve outside of Google Cloud.
      Early deletion fee. Effectively just a minimum storage charge of 1 month.
    

It seems this is cheaper than Glacier and quite a bit simpler. The speed
restrictions are interesting though.

~~~
fuzzythinker
Unless Glacier just changed their pricing, how is it cheaper and simpler?
Glacier is also $.01/GB storage, but only $.09/GB retrieval, which is cheaper
than google's $.12/GB. Glacier also comes with 1st GB retrieval free/mo.

~~~
panarky
Glacier pricing is surprisingly complicated, and the actual cost can be much
higher than $0.01 per GB-month if you don't read the fine print.

The biggest gotcha is that you can only access 0.17% of the data you've stored
in any given day without extra charges. So if you've stored 1000 GB, you can
only access 1.7 GB per day for free.

The cost for going over your daily "retrieval allowance" can be large, because
cost is driven by "peak retrieval rate". They find the highest hourly
retrieval rate over the entire month, and charge the whole month's retrieval
at the peak rate.

This can get expensive fast. Again, with 1000 GB stored, if you retrieve 200
GB over the course of 4 hours, your hourly retrieval rate is 200 GB / 4 hours
= 50 GB per hour. Your free retrieval allowance is 1.7 GB per day / 24 hours =
0.07 GB / hour. So your excess retrieval is 49.93 GB per hour.

Amazingly, that 49.93 GB per hour for 4 hours is charged for all 720 hours in
the month, so that's 49.93 * 720 * 0.01 = $359.49.

That's an astonishing $1.79 per GB just to retrieve one-fifth of the data from
Glacier storage.

So Glacier only makes sense for true cold storage, where you are very unlikely
to touch more than 0.00007 of it in any given hour.

Google's Nearline is much simpler and faster for the same storage cost -- and
far cheaper if you need to access more than a tiny fraction of the data.

Source:
[http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retr...](http://aws.amazon.com/glacier/faqs/#How_much_data_can_I_retrieve_for_free)

~~~
AceJohnny2
Thanks for the detailed breakdown, that's really interesting! Sounds like you
were bitten by this? :)

~~~
ak217
Don't know about the OP, but we were bitten by this - badly - after our logic
to throttle Glacier retrieval failed.

AWS has now added the ability to set a spending limit to avoid runaway
retrievals. Nonetheless, from my point of view the ultra-complicated pricing
scheme is nothing short of a disaster for Glacier as a product. I think it
will continue to seriously impact its uptake, and Google is smart to exploit
the opportunity to offer a simple, straightforward pricing scheme.

~~~
NeutronBoy
The use case is people who have a requirement to store data and _probably_
never need to access it, either for reg requirements, or legal requirements,
etc. In which case paying a couple of hundred bucks for retrieval is fine,
especially if they can access at a high level of granularity

edit: Also as others have pointed out, Nearline is limited in retrieval time
as well, so the cost difference isn't nearly as large.

------
weitzj
Quote: "This is a Beta release of Nearline Storage. This feature is not
covered by any SLA or deprecation policy and may be subject to backward-
incompatible changes."

So should I believe in Google's good-will? I would be fine trying out some
services, which are in Google Beta. But my valuable data? They should have a
SLA right from the start to gain the user's trust.

~~~
libria
If we place too many restrictions on how companies should offer preview
releases, they'll just stop entirely. Are you suggesting that even those
comfortable taking on risk to get a sneak peak should be forced to wait for
general availability?

~~~
texthompson
Nobody is placing restrictions, but it's understandable if people are
skeptical.

~~~
pbreit
Praise be the skeptical! It allows the courageous easy advantages. SLAs are
almost worthless. And, really, has Google ever retired something like this
(i.e., not an acquisition, not Google Reader, etc)?

~~~
pbreit
ReplyTo aros: The two main criteria I would have are: 1) not an acquisition
and 2) important development/developer service. I can't think of any and
didn't see any after a cursory look at the list.

I think SLAs are literally worthless since I don't think they encourage even
slightly more effort in minimizing whatever issues the customer is concerned
about. No one wants servers to go down, no one wants to shut down a service.
That a few Google bucks might be on the line would have zero impact.

------
gaul
I added GCS nearline to my object storage comparison:

[http://gaul.org/object-store-comparison/](http://gaul.org/object-store-
comparison/)

~~~
ddorian43
you are missing runabove edit: also missing constant.com

~~~
pi-rat
Didn't know they offered object storage, thanks. $0,01/GB/month - for online
content - replicated 3 times.. Wow

Any experience with them? how is their stability and speed?

~~~
ddorian43
No I haven't used them. But their parent-company(ohv) has the ~lowest prices
you can find.

------
crb
1c/GB/mo for data stored, ~3 seconds response times (cf. tape/Glacier at
multiple hours), 11 9s durability, same API as Google Cloud Storage online.

(edit: add /month)

~~~
keysheath
There's no mention of durability numbers anywhere in the docs. S3 and Glacier
mention 11 9s durability in their FAQ though.

~~~
BrandonY
Google Cloud Storage Nearline offers the same durability as the Standard-
priced storage offering. See the first paragraph of
[https://cloud.google.com/storage/docs/nearline-
storage](https://cloud.google.com/storage/docs/nearline-storage)

------
sp332
Isn't the price a little high? $10/TB/mo is the same as Dropbox, and this has
a lot fewer features than Dropbox.

~~~
nda
It's a bit apples to oranges comparison. Dropbox is a consumer product, like
Google Drive. Nearline is targeted to app developers and enterprises

~~~
sp332
Exactly, so why would the lower-level service aimed at big slow transfers be
more expensive than a full-featured product?

~~~
manacit
There's a few reasons:

* Dropbox doesn't guarantee 11 9s like this service does - when I'm backing critical data up, I want to make sure it's _there_. * Dropbox likely wouldn't take kindly to me storing 10PB, whereas that is what this service is designed for. * I've got SLAs and guaranteed speeds with this, Dropbox isn't designed for me to suddenly download 10PB very quickly.

~~~
thrownaway2424
Where are you seeing these durability claims? I can't find them. And what
would 12 9s durability even mean? You lose one byte out of every TB?

~~~
manacit
It's 11, not 12, I misspoke.

Those claims are from glacier:
[http://aws.amazon.com/glacier/faqs/](http://aws.amazon.com/glacier/faqs/)
which is priced the same per GB of data.

In this case, the durability refers to the loss of data/objects stored per
year - if you're sending multiple PBs of data off to Glacier, you want to be
able to retrieve them many years later. Even 5 9s would mean that 1 object out
of 100,000 is lost every year, which is quite poor.

------
moe
This is a nail in the coffin of the SaaS backup industry (Backblaze,
RescuePlan, etc.).

With storage becoming essentially free the last excuse for _not_ using self-
hosted, secure backup tools like Arq[1] disappears.

[1]
[http://www.haystacksoftware.com/arq/](http://www.haystacksoftware.com/arq/)

~~~
sreitshamer
I'm adding support for creating nearline buckets in Arq right now.

~~~
StavrosK
This is exactly what I want for Linux (headless), but I haven't been able to
find anything like it. The closest thing I've found is attic, which, while
amazing, requires a daemon to be running on the remote end (which is what
makes checking, deduplication and deltas fast).

Does anyone know an Arq alternative for linux?

~~~
rsync
duplicity ...

[https://en.wikipedia.org/wiki/Duplicity_(software)](https://en.wikipedia.org/wiki/Duplicity_\(software\))

~~~
StavrosK
Unfortunately duplicity is very inefficient, it needs to reupload the full set
every so often, which takes up a _lot_ of bandwidth. I guess attic is my best
bet in the foreseeable future.

~~~
insaneirish
Haven't used it, but Duplicati is another one:
[http://www.duplicati.com/](http://www.duplicati.com/)

~~~
StavrosK
Looks very interesting, thanks! I'm investigating it now.

------
SEJeff
So is this a blatant dig at Amazon's Glacier? "Fast performance... unlike
competitors"

As they generally do tit for tat on price wars, it will be interesting to see
what Amazon responds with.

~~~
dragonwriter
> So is this a blatant dig at Amazon's Glacier?

Glacier is the other big offering from a comparable cloud vendor in this
space, so, yeah. The performance claim is directed largely at Glacier, as is
the consistent access one.

------
pjc50
So how is this implemented? Spun-down disk drives?

~~~
bbrazil
My guess would be disk drives that are otherwise used for I/O intensive
purposes, and where there's tons of unused/wasted space.

4MB/s is only around 4% of the bandwidth of a modern 1TB hard disk.

~~~
nda
You can store 1 Pb and get 4000 MB/s

~~~
keysheath
And at 3MB/s per TB it will always take over 3 days to retrieve all of your
data. The 2-5 second retrieval latency is irrelevant when you're going to be
waiting for 3 days...

~~~
jlebar
FWIW it's nominally 4MB/s/TB:

> You should expect 4 MB/s of throughput per TB of data stored as Nearline
> Storage.

But still, you are correct in that it will take about 1TB / 4MB/s = 2.9d to
retrieve all your data.

If you need the ability to do a restore faster than this, then you need to pay
more for storage, is all. For many of us, waiting 3d to recover from a
catastrophic failure isn't a big deal.

------
bkruse
I own a big wholesale telco that does tons of data center business and
bandwidth. Of course, margins are super thin so we need great pricing. We are
no Google, but we can achieve the same pricing, including the cost of
bandwidth (that is 9-9s of durability and no spin-up time, which is where the
3-seconds come from)

Maybe I should start my own service to harness all this infrastructure with
something like swift!

------
nda
from the documentation: "You should expect 4 MB/s of throughput per TB of data
stored as Nearline Storage. This throughput scales linearly with increased
storage consumption."

~~~
keysheath
That would mean that if you had about 1TB stored it would take more than 3
days to retrieve it (with an initial 3 second delay before it starts).

~~~
wongarsu
And since bandwidth scales linearly with used storage it would presumably
always take more than 3 days to retrieve all of your data.

------
nda
The same egress and data transfer charges as for Google Cloud Storage Standard
/ DRA. Can transfer the data from Standard to Nearline bucket for free until
Jun, 11 - [https://cloud.google.com/storage/pricing#network-
regions](https://cloud.google.com/storage/pricing#network-regions)

~~~
teraflop
That promotion is talking about transferring data between geographical
regions, not storage classes.

------
tiernano
slightly off topic here, but when i tried to setup an account, which should
get me $300 free, it auto selects my country (Ireland) and then tells me:

Tax status Business

This service can only be used for business or commercial reasons. You are
responsible for assessing and reporting VAT.

and then makes me enter a VAT number... well, I will stick with AWS and Azure,
since neither of those "REQUIRE" a vat number or business status.

PS: i know a business name costs about EUR20 to setup, but the VAT requirement
is the pain in the left one...

~~~
deepsun
Where did you read about $300 for free?

------
AndyJack1979
Our story with Amazon Glacier (50TB):, we are a company with 50TB of data on
multiple Servers and NAS, and some External drives, most of the data we needed
to save/Archive for 10 years at least, it’s starting to become very hard to
manage locally, so we saw glacier a great solution for us! Well, not exactly

We tried to manually using it, like APIs, Scripts, etc… it’s impossible, it
was very had

So we tried cloudberry to help us upload, it will upload but also It won’t
work for huge data, with glacier you won’t be able to search, list, find any
file you want to download easily, also its not practical to manage all the
backup and millions of files manually, also we got millions of photos we
needed a way to find them easily like thumbs So Glacier had so many
restrictions like 3-5 hours restore time, 5% restore quota hard to use even
with utilizes like cloudberry, cant list, search, it’s not useable as its! But
the price was attractive

We considered Seagate Evault , but it was expensive and so many hidden fees
and complicated for our case Then we tired another solution called Zoolz (
[http://www.zoolz.com](http://www.zoolz.com) ), Zoolz does not use your own
AWS Account, but utilize their own account and from what I heard from them
they got 5 Petabyte, so they got massive restore quota, around 5% of the 5
Petabyte, also its simple to use, they offer Zero restore cost, it’s like Mozy
or Crashplan but for business and on glacier , like they internally create
thumbs for your photos and store it on S3, so you have instant preview, also
you will be able to search and browse your data easily and instantly, they got
Servers and Polices, and got a reasonable price, all we wanted , we got the
50TB for $12,000 / year, its more expensive than using Glacier by itself,
using Glacier will cost around $6000, but its not practical for a company to
use it as a standalone storage

The only disadvantage is that when/if we needed a file we have to wait 4 hours
to get it, which is fair, it was faster than when we used to use Tapes :), awe
tried to restore 1 TB with 1.2 million files, it took us around 10 hours to
complete, which was okay

------
g8oz
Can you do rsync style differential backups with this?

~~~
crb
gsutil (the Google Cloud Storage command-line tool) has an rsync option:
[https://cloud.google.com/storage/docs/gsutil/commands/rsync](https://cloud.google.com/storage/docs/gsutil/commands/rsync)

------
bernadus_edwin
Is azure has the product like this? Usually cloud war start from amazon, then
google, and then azure is the last one

------
Jolielov
I still rather use a Cloud Service provider like Zoolz as it would be more
efficient to benefit from other plans. Amazing idea though!

------
cornellwright
Great to finally see a cloud service competing against Glacier. Hopefully this
will lead to some sort of response from AWS.

------
joelthelion
I wonder what the underlying hardware is?

------
knocte
Does anyone know of any online-storage offering that allows paying in bitcoin
and allows API access? Thanks

~~~
wongarsu
I'm sure there are various FTP storage providers that are accepting Bitcoin.

------
zobzu
"nearline, still online, but we could make a graph that is supposed to make
you believe its not"

------
wyldfire
The transport for this service is HTTP? I imagine most competitors use that as
well, right? How does encoding factor in here? I have to transcode my data to
base64 in order to put it in or take it out, I assume? I "know" I'd only get
billed for the data stored as the original octet/binary encoding. But what
about the egress fees? Encoded or decoded data is the input for the billing?

~~~
Guvante
HTTP can transfer arbitrary binary data just fine. There is no need to encode
it.

I don't know which they use but HTTP traditionally uses a `Content Length` or
an empty chunk to represent the end of data.

------
fivedogit
Some of these Google/Amazon/Apple "Introducing Product X..." stories are
starting to strike me as intellectually devoid, creeping into "free
advertising" territory.

This particular story is about a Google service that is an extremely late
entrant into the market (Amazon Glacier, etc) and offers almost nothing new.
I'm not saying it's a bad product or not useful to folks, but it's not, by any
means, groundbreaking. I'd much rather see the top spot of HN occupied by some
startup's new idea or a researcher's new findings.

I also don't mean to imply that "big corporate" == bad. Certain products --
self driving cars, Space X automated landings, etc -- are absolutely worthy of
our attention and discussion. I just hope that people would think twice before
upvoting a story merely because it's from Google.

~~~
vgt
I suggest you read, for example, this article from TC:

[http://techcrunch.com/2015/03/11/google-launches-cloud-
stora...](http://techcrunch.com/2015/03/11/google-launches-cloud-storage-
nearline-a-low-cost-storage-service-for-cold-data/)

This information should challenge your argument that Nearline is not
groundbreaking, at least in the "cloud archival storage" space.

