I played from 1995~2001/2ish. Probably to this day the most rich online gaming experience I've had; the book version of a succession of inferior films.
97th to 104th between 2nd & 3rd is basically all public housing, and there's public housing along the east river that will also have closer, less crowded stations.
A lot of the tunnels for Phase 2 (extending the Q to 125th) were built or partially built in the 1970s, and this phase already has funding commitments.
What? He also predicted how and why it would fail. Sun was a big enough player then that it could have survived plenty of other ways. Apple of today looks nothing like the company in 2000, but Sun got caught out more or less exactly as described and never adapted.
A few comments on some things I'm seeing in these.. comments.
- What about Cassandra/PostgreSQL/Redis?
One of the implications of "99% of data is never read" is that it's incredibly wasteful to keep it all in memory. You might be assuming that you are letting the data expire eventually, but I'm actually not; expiry is a secondary concern to storage.
Once you start to involve the disk (PostgreSQL and Cassandra), you start to get into locality issues that these databases weren't really designed for.
For a more concrete description, lets say I have 2000 machines. Our app is Python, so they run 16 dockerized processes each, with each container reporting 10 simple system metrics every 10 seconds. These metrics are locally aggregated (like statsd) into separate mean, max, min, median, and 90pctile series. I've not instrumented my app yet and that's already 160k writes/sec on average; if our containers all thunder at us it's 1.6m, and frankly this is the "simple" case as:
* we've made some concessions on the resolution
* we only have dense series
Anyone who has used graphite at scale knows that this is a really painful scenario, but these numbers are not particularly big; anywhere you could take an order of magnitude there are a few other places you can add one.
I'm also assuming we are materializing each of these points into their own timeseries, but that's more or less a necessity. It gets back to the locality issues; If we wanted to see the "top 10 system load per node", it's actually quite imperative that we aren't wasting IO cycles on series that _aren't_ system load; we need all we've got to handle the reads.
(As a side point, this is why people in the cloud are adopting so much Go so quickly; it's proven to be easy to write AND read, and also to reduce orders of magnitude in several of the dimensions above, eg. "we can use 1 process per box not 16 containers, and we can get by with 1000 machines not 2000." Having to write your own linked list or whatever doesn't register in the calculus.)
- 1s resolution isn't dense:
No, not always. It's hard to please everyone with these things. In my world, 1s is good (but not great), but 10s seems to more or less be accepted universally, and much sparser (5m, 30m, 6h) is not actually uncommon. At the other end of the spectrum, you can be instrumenting at huge cardinality but very sparsely (think per-user or per-session), perhaps only a few points per timeseries, and the whole of what I've described above kinda gets flipped on its head and a new reality emerges. For what I've described, I quite like the Prometheus approach, but for my very specific use case 1-file-per-metric only beats the filesize block overhead often enough for very long timeframes; to long.
- Why are all TSDB over-engineered?
I hope some of the above has made explicit some of the difficulties in collecting and actually making this data readable. I've only actually thusfar discussed the problem of "associate this {timestamp,value} pair with the correct series in storage"; there are also the following problems:
* you can't query 3 months of 1s resolution data in a reasonable amount of time, so you need to do rollups, but the aggregations we want aren't all associative so you have to do a bunch of them or else you lose accuracy in a huge way (eg, if you do an avg rollup across min data, you flatten out your mins.. which you don't want); this means adding ANOTHER few dimensions to your storage system (time interval, aggregator)
* eventually you have to expire some of this junk, or move it to cold storage; this is a process, and processes require testing, vigilance, monitoring, development, etc.
* you need an actual query system that takes something useful and readable by a human ("AVG system.cpu.usage WHERE env=production AND role=db-master") and determines what series' actually fall into those categories for the time interval you're querying. Anything holistic system that _doesn't_ do this is an evolutionary dead end; eventually, something like Prometheus or Influx will replace them.
These are minimum requirements once you "solve" storage, which is always a very tricky thing to have claimed. If you get here, you've reached what decent SaaS did 4 years ago and what very expensive proprietary systems handled 10 years ago.
- What about Prometheus/InfluxDB/Kdb+/Et al.
Kdb+ is very expensive, its open source documentation is difficult, and its source is unintelligible. It is basically from a different planet that I'm from. Even recently, when I encounter people from, say, the C# world and tell them I work with Python and Go, they ignore Go and say "Wow, there are like no jobs for Python", which I find utterly bewildering. Of course, I never encounter any jobs using C#, either. This is how little some of these spheres overlap sometimes. Someone from the finance world is going to have to come in and reproduce the genius of K and Q for us mortals in a language we understand.
As for Prometheus and InfluxDB, I follow these more closely and have a better understanding of how they operate. I think that they are both doing really valuable work in this space.
From a storage aspect, I think the Prometheus approach is closer to the one that I need for my particular challenges than the InfluxDB one is, and in fact it looks a bit like things we've already had (but also Catena, Parquet, et al..) For most people, storage actually isn't important so long as it's fast enough.
And this is kind of the point of my article. There's starting to be a bit of a convergence among a few Open Source TSDB, and I've taken I've tried to highlight some issues in those approaches and suggest that there's room for improvement. I have my own ideas about what these improvements might look like based on my work at Datadog, and once they're proven (or even disproven) they'll be published somehow.
I have actually heard of Kx and Kdb+ and Q and have looked into those products. You're right that I should have mentioned them, because they are important in that space, and in many ways they are miles ahead. The state of time series query languages is quite poor (outside of Influx's effort), so there is a lot to learn from Kdb+ as well.
I was however focusing on recent Open Source efforts and on the general approaches. Hence, I didn't really discuss in detail my own tsdb.
I also find virtually everything to do with K and Kdb to be be simultaneously impressive and utterly unfathomable:
> The belief that ORMs are evil is precisely the belief that this sort of code should be repeated everywhere database access is performed. If you have generalized routines for interacting with the database with more comfortable abstractions then string concatenation, you are using an ORM
This is incorrect.
The sqlx library, included in OP, has generalized routines for interacting with the database, but is not an ORM. The squirrel[1] library for Go lets you produce queries without "string concatenation", but is also very much not an ORM.
An ORM is a specific style of library that attempts to map object oriented data patterns to relational concepts. That's why it's called an Object-Relational Mapping. There are good reasons[2] why people find this approach problematic, which aren't down to cargo culting them as "evil" or believing that everyone should repeat data access code in all projects.
The simple/typical use case of Django's ORM, sqlalchemy, Rails ActiveRecord, etc. is not much more than a native API for composing SQL queries, and the automated mapping of table rows from the RDBMS into native data structures in the application (which happen to be objects because of the language).
When you claim that you are not using an ORM, a reasonable person would take it to mean that you are forgoing the use of query builders and automated mapping from SQL results to application data structures.
So, it may be true that classifying these libraries as ORMs is incorrect, but the "user experience" as a developer between these libraries and typical ORMs appears to be pretty much the same. Or is that unfair?
I have a lot of experience with Django's ORM and I would say that the essential character of these heavier ORMs is missing from the afore mentioned libraries. They can be used in the way you describe, but that's not their typical usage.
AR/Django encourage you to describe your entire schema, _including_ the relationships between tables, as attributes on your model objects. Upon doing this, you get simple and reliable programmatic access to some basic access and storage patterns.
Using this knowledge of your data model, the ORM can now provide you with more advanced tools: it can automatically join across tables (`select_related`), lazily load dependent data[1], generate SQL schema for you, transparently cache queries[2], automatically provide HTML form validators, and even automate database schema migration for you. The more completely you model the system, the more it can do for you.
In these systems, the database is subservient to the model. This is a problem, because the database is reality and the model is a model.
Dropping to "opaque SQL strings" is discouraged, both because it's considered error prone ("You should leave SQL to the professionals! There are lots of eyes on this!") and because there is often no graceful way to integrate custom query code with your model layer; instead, you investigate how to do so within the confines of the ORM. For every case where you can eventually find what you need (`select_related("user__friends__email")`), there are a dozen where you can't.
People start writing things in application code that could be handled easily and more efficiently by the database: aggregations are a classic, since ORMs support is either missing or incredibly complex. As soon as the application becomes non-trivial[3], the problems magnify. They don't do this because they are stupid, or bad developers, they do this because it's what the tools encourage: they encourage simplistic CRUD access and mistrust/suspicion/fear of SQL.
SQLAlchemy is quite different from this, because its primary focus is to model databases, not to provide some kind of declarative object language with its own set of semantics that do not exist in SQL.
Because Go can't do things like meta classes or creating/modifying types at runtime, a lot of this simply isn't there, even though people want it.
Sqlx does like 2 things; it adds named query parameter support, and it marshals rows into structs. Squirrel is just a query builder. The whole philosophy that the database must somehow be modeled and that access to the database is done via that model is absent.
[1] This is especially durable tarmac with which to pave your road to hell.
When I wrote the same kind of article in Nov 2011 [1], I came to similar conculsions; ujson was blowing everyone away.
However, after swapping a fairly large and json-intensive production spider over to ujson, we noticed a large increase in memory use.
When I investigated, I discovered that simplejson reused allocated string objects, so when parsing/loading you basically got string compression for repeated string keys.
The effects were pretty large for our dataset, which was all API results from various popular websites and featured lots of lists of things with repeating keys; on a lot of large documents, the loaded mem object was sometimes 100M for ujson and 50M for simplejson. We ended up switching back because of this.
Hey Jason. Thats pretty interesting. I have also noticed similar things but for my case, we needed faster loading/unloading for some cases, hence ujson.
I'm pretty sure symbols are not meant to be created from "user" input where user is untrusted, can't this lead to ddos atacks? Same thing for interning. De-Duping doesn't have that risk.
Lua has an interesting approach here. In Lua, all strings are interned. If you have "two" strings that consist of the same bytes, you are guaranteed that they have the same address and are the same object. Basically, every time a string is created from some operation, it's looked up in a hash table of the existing strings and if an identical one is found, that gets reused.
However, that hash table stores weak references to those strings. If nothing else refers to a string, the GC can and will remove it from the string table.
This gives you great memory use for strings and optimally fast string comparisons. The cost is that creating a string is probably a bit slower because you have to check the string table for the existing one first.
It's an interesting set of trade-offs. I think it makes a lot of sense for Lua which uses hash tables for everything, including method dispatch and where string comparison must be fast. I'm not sure how much sense it would make for other languages.
You can discover what internal strings are held in a web application via a timing attack.
Better hope you never hold onto a reference to internal credentials inside the application! (Say... DB username / password? Passwords before they're hashed? Etc.)
Depends on symbol implementations and intended usage.
For example Erlang symbols are deeply ingrained into language, and vm doesn't even garbage collects them, so creating symbols from user data is basically giving user 'crush vm' button.
On the other hand, if symbols are treated as another data type, as string with some optimizations - no such problems shall arise
I think most JSON structures are unlikely to have user input be used as keys. This is also likely where there would be the most benefit from interning since keys are often repeated many times.
Being a long time gevent user, I found goroutines pretty familiar and even unspectacular (no pool.map?!). What really converted me to Go was all of the other ways I found writing Go programs to be more pleasant and less error prone than Python programs:
* consistent formatting for all code
* superior distribution story
* compiler magnitudes faster and more effective than pylint
* programs run faster and use far less memory
* far simpler semantics make it easy to read
* higher quality standard components (net/http, eg)
What I traded for this was a 10-20% drop in productivity, which I was fine with. I use Python for all sorts of quick & dirty tasks still, including some batch processing where there's a big discovery phase, but I write all my software in Go.
