If it weren’t for such a vibrant open source world then they’d really be stuck if they tried to apply their knowledge outside of their companies.
Thanks for sharing the repo!
Can someone please share even a high level summary?
Is the replacement a high cardinality event store like Facebook’s Scuba ?
* All metrics have an associated type. Eg. Response time (milliseconds). That's great because units for derived metrics can be dynamically computed. Eg. Bytes/second.
* The query language can fairly efficiently compute metrics at query time rather than needing everything precomputed (eg. 95 percent latency across 1000 tasks can be calculated in real-time).
* The config system is a mess and nobody likes it. Borgmon uses a DSL which is obscure, but almost identical to Prometheus. Monarch has various different config frontends (mostly around the idea of running code to produce an expanded protobuffer config) which all suck. Luckily because there isn't a strong requirement for rules to aggregate data, you don't need much config for most services - just say "scrape everything and keep it for a year".
* There are "levels" of storage at different speeds. In memory, on disk, etc. You have to configure where to put what data. You can also downsample (eg. Change scrape interval to 5 mins after a week).
* Metric names follow a directory-like heirachy. Since tasks can easily have 10k exported metrics, that's pretty important. No need to scrape the ones that aren't relevant.
* It has a shinier UI.
* It has support for exemplars. So to answer the question "Give me an example of a request which saw this high request latency". With not much added code to the monitored service, a small number of exemplars are captured and aggregated in a way that median and outlier exemplars are available. They're super useful for finding out the cause of random slow performance.
* It is run as a service. Rather than code that every team has to run, the new thing is a single instance for all teams in Google. That in turn means it can be more complex, have more dependencies, etc, without being a burden on the user.
I'm having hard time figuring out how query language might affect exectuion time. Granted, I've never seen borgmon in action, so the first question should probably be: how different its query language is from what Prometheus offers? If it's more on imperative side than declarative one then your point might actually make sense to me as it is.
> You can also downsample (eg. Change scrape interval to 5 mins after a week).
Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does?
> Does it allow different aggregation methods (avg/sum/max/whatever) for different metrics, like Carbon does?
All of these methods are available. Stackdriver Monitioring is the public face of Monarch.
I don't remember the latter being much more sophisticated than a bunch of files in GFS with the timeseries and some metadata, either. Monarch, I think, is based on Bigtable. It has a richer API. It might even use BT coprocessors to perform computations closer to the where the raw data is stored, rather than in the Monarch frontends. (I haven't watched the public talk, so I'm just going from vague recollection.)
Compare to borgmon where use of the DSL is obligatory.
I’m sure there are some people that have lived only within this ecosystem, but that’s the possibly the height of privilege.
Besides it’s a moot point since no company that competes with Google for talent asks exclusively domain specific questions in their interview loops.
Pros: Get's checked in like code, gets reviewed, and leaves a nice paper trial of edits.
Cons: Doesn't invite comments/discussions as well as Docs does.
I can imagine that can quickly become a fulltime job though. Still, a formal role of documenter / archivist in a bigger software endeavour should be considered.
What are your favorite/must-have Critique features?
(Also would love to jump on a video chat if you or anyone else is interested, and we could live-stream/share it on YouTube for others interested. Email me at firstname.lastname@example.org if interested.)
- the ability to batch all comments in a single send. GitHub has the problem where replies are immediate and don’t necessarily represent the PR state.
- built-in fixes and lints that show up as comments
- the ability to diff between snapshots instead of losing all previous context every force push to a Github PR.
- Github’s diff is really bad for code moving or changing indentation.
- auto submit if tests pass and you have an lgtm. It’s frustrating having to wait for tests that will probably pass.
GitHub actually does some of these things already. Do these solutions help/suffice?
- Batching all comments in a single send: This is a core GitHub code review feature. You can send immediately by pressing the "Add single comment" button, but if you "Start a review", your comments are batched.
- Built-in fixes and lints on the PR "Files changed" tab: GitHub Checks offer this, although not many teams have adopted it (that I'm aware of).
- Changing indentation: The "Diff settings" menu has a "Hide whitespace changes" option, although it's not (afaik) possible to persist this.
- Auto-submit if tests pass: The Refined GitHub browser extension offers this (https://github.com/sindresorhus/refined-github), kind of. You need to keep the page open in your browser while it finishes.
Also, have you seen https://reviewable.io/ (made by an Xoogler) and does that appeal to you?
Google -> Real World
free food -> :(
free food -> free time
If your income as a Google engineer is so tight that you have to make sure you get the most out of breakfast and dinner, go for it (but also maybe have a long hard think). The culture may well vary between sites, but this idea that Google (and others) exclusively employ fresh college grads and try to embrace then 24x7 just isn't true. We have families just like anyone else, and they matter to us every bit as much as everyone else's families matter to them.
I for my part am happy that I get a healthy, hot meal for lunch so dinner can be a quick snack when I get home, so I have more time with the kids. The alternative is more time spent on chores.
Please stop spreading these tired tropes.
I absolutely hate the "Whats for lunch today?" shuffle.
If you can eat faster, you get back to work faster. An early breakfast or tasty late dinner at work is a temptation to spend more time there.
I had a quick google and I’m not alone in my opinion. I’m not criticising the practice, it makes business sense and employees can choose not to use it.
(The other major driver behind food was supposedly the fact that the founders were PhD students and had found that some of their best ideas had arisen from informal discussions with other researchers over lunch, so they wanted to create an environment where employees had that same opportunity)
Because well, kids usually don't tend to cook for themselves so the parents end up cooking for them and eating with them.
That might be my bubble though, my friends / family have jobs that allow them to leave work around 5-ish and with school ending somewhere between 4-5, it's not that much after the kids are home. (or they pick up the kids after school).
Seriously, if you're working somewhere where people care what time you go home, you can really do better!
Now that Amazon took cloud9 (c9.io) out of the market, it seems like there is a sizeable void.
I want web-based Spacemacs or Evil! Or even just intelli-j.
IDE is based on Theia (https://theia-ide.org), i.e. VS Code with cloud-first mindset.
The other options are just IDEs in a browser where you still have to hammer together your own setup + build manually every time.
We couldn't address it in time for the launch, so basically all the website's pricing options currently point directly to where you can buy them in the app, which implies a GitHub sign-in (where only a valid email is requested, NB).
We're planning to add un-authed pages with more info about each plan, but we also prefer spending our time on improving the developer experience rather than on payment options, so it may take a bit of time.
I don't enjoy being the insect who claimed a tall blade of grass only to be mowed down first. If you value your time and value stability, don't use closed-core tools.
In services there should be vitess since it's both a YouTube and an open source project.
It seem to have a new transactional messaging feature with message sending and Ack: https://vitess.io/docs/advanced/messaging/, which seem to fill a slightly different niche than rabbitmq/pubsub/etc.
When I needed something like RecordIO to store market data, I couldn't find anything. So I implemented https://github.com/romkatv/ChunkIO. I later learned of https://github.com/google/riegeli (work in progress), which could've saved me a lot of time if only I found it earlier. I think my ChunkIO is a better though.
Edit: it even includes an open source implementation of Cord, which they've renamed to Chain for some reason.
I suppose you mean "exactly" in a figurative way. Riegeli is definitely inspired by RecordIO and is meant as a successor to it but it's not RecordIO.
> Is there a reason that doesn't meet your requirements?
I need to store timeseries with fast lookup by timestamp. Riegeli doesn't support this out of the box. If I had discovered it before I built ChunkIO, I probably would've pulled the low-level code out of it and added timeseries support on top. Or maybe not. Reliability is very important to me and it's risky to use work-in-progress software that may or may not have any production footprint (I'm no longer with Google so I don't know if they use it internally.)
What you are describing sounds like SSTable. Perhaps you could benefit from LevelDB.
For example, I see Pipelines could be replaced by CodeDeploy. I hear nix is similar to the brazil build system. Any other similar mapping by others?
Hopefully that answers your question ;-) Same as with a discussion about patents the other day. It’s fascinating to learn / read / understand how big businesses work as someone who’s never worked at one nor aspires to work at one, but does want to build one ;-)
Are people really still jerking about a company that's been around for like a 1/4 of a century?
And the linked article, while somewhat related has an entirely different message/purpose as this one.