
Cloud Platform at Google I/O – new Big Data, Mobile and Monitoring products - Goranek
http://googledevelopers.blogspot.com/2014/06/cloud-platform-at-google-io-new-big.html
======
nostromo
This is one space where Google really excels.

We're in the AWS ecosystem, and the database offerings are really subpar.
DynamoDB, which I originally expected to be somewhat comparable to MongoDB, is
an incredibly frustrating (and expensive) product to use. AWS Data Pipeline is
extremely confusing and very expensive as well.

AWS's offerings really lag behind Google's offerings (like BigQuery) in this
space. Hopefully AWS can catch up because I'd rather not have requests
bouncing between data centers.

~~~
tedsumme
What do you think is the analogous product to Cloud Dataflow in the AWS
ecosystem? SWF? [http://aws.amazon.com/swf/](http://aws.amazon.com/swf/)

~~~
persona
I believe it's Kinesis:
[http://aws.amazon.com/kinesis/](http://aws.amazon.com/kinesis/)

~~~
hatred
+1. It's Kinesis.

~~~
eweise
DataFlow is not Kinesis. It's more like Kinesis plus Esper plus BigQuery and
you still wouldn't have one set of queries to run against streaming and batch
data like you do with DataFlow.

------
persona
Google Dataflow seems to be the big one here specially if it works well for
stream processing. Fault-tolerant stream processing with huge scalability?
Perfect for the IoT!

------
isbadawi
Judging from the code samples they showed during the keynote, I'd guess that
Google Cloud Dataflow is based on (or an extension of, or a public version
of...) FlumeJava, described in this PLDI 2010 paper:
[http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/F...](http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf)

------
davecap1
Anybody have experience moving from AWS to Google Cloud? If so, did you have
any surprises in terms of difficulty or cost?

------
IanCal
The streaming data stuff looks _extremely_ interesting. My main concern is
around cost, unfortunately many of these things are great if you've got a
massive data problem but not particularly worth it if you've got much smaller
data.

I'm in a rather awkward phase of having small enough data that I don't need
"Scale to 1000 machines!", I want just one or a few machines occasionally but
managed for me (turn on, run code, shut off). Tutum works very well for this,
but I'd like to use more of the ecosystem available at Google or AWS (pay-per-
usage datastorage, for example). GCE is pretty decent, but a bit awkward,
although the new docker support helps (but I've had problems getting it even
working).

Maybe this is my magic bullet :)

~~~
smoe
I'm using MITs StarCluster to quickly spin up a bunch of AWS Spot Instances,
run some calculations and shut them down again.

[http://star.mit.edu/cluster/](http://star.mit.edu/cluster/)
[http://www.youtube.com/watch?v=2Ym7epCYnSk](http://www.youtube.com/watch?v=2Ym7epCYnSk)

~~~
IanCal
Thanks, I'll have a look around at that!

------
nwfzp
It looks like an attempt to respond to AWS kinesis which was released last
year. The monitoring stuff seems to be about the software they got when they
bought stackdriver.

------
opendais
Google adding cloud monitoring has me sorely tempted to abandon a side project
of mine. I'm sure Google could do it better. Bah. :P

~~~
samstave
What type of monitoring? We use Stackdriver, which Google just recently
bought.

~~~
opendais
I probably will end up building the bare minimum to meet my needs and moving
on tbh.

It was basically a monitoring/metrics system to merge how I handle the
monitoring of crons, work queue, system metrics, analytics, etc. into a single
service. Right now, I'm stuck using 3.

Sure, I could just build something to merge it together ... but at that point,
I'm halfway to building my own.

~~~
cmelbye
I was about to do the same thing. App Engine sorely lacks those features
currently, so I'm very excited for this (assuming it has good support for App
Engine in addition to Compute Engine which I saw in the keynote).

------
curiousDog
Anyone know if they'll be offering Spanner as a publicly available service?

~~~
Goranek
Probable in the future.. They never offer the latest technology...

