What we're doing at Cloudflare (including some of what the author works on) samples adaptively. Each log batch is bucketed based on a few fields, and in each bucket if there's lots of logs in each bucket we only keep the sqrt or log of the number of input logs. It works really well... but part of why it works well is we always have blistering rates of logs, so can cope with spikes in event rates without the sampling system itself getting overwhelmed.
The status quo isn't an application. "Digital ID" is a misnomer, it's a government backed application akin to what you'd see in an authoritarian crony state (China, Vietnam, etc). Unbelievable that a western country is signing off on this.
> if they can't explain what they want to do clearly and why, it probably shouldn't be done. Quality goes up by slowing down.
I kind of agree. Without what you describe, teams often get lost. But I’ve also seen that approach keep teams stuck in comfortable local minima. Sometimes you’ve got to take risks.
People need agency, and feel empowered to go forward. I'm not arguing against that interpretation of the post.
But having agency doesn't equal a lack of accountability, or being able to explain what you're doing.
In the end, it's a tricky balance between moving fast and slow, which is why no one has found the perfect, long-term viable solution yet.
I agree, it's an interesting view. I found it quite positive about the culture in that part of Splunk. The writer parachuted in, threw their weight around, bruised a bunch of people, but lost out to another manager who openly acknowledged the contributions of their people.
The effect is really cool. I like the idea of visualizing data flowing through work systems like this. I think I'll go for 3D-printed or laser-cut panels instead though - would work out cheaper. Thanks :)
Ordering satellite imagery and counting cars is just a weekend project. The last time I looked at ordering imagery, the main obstacle was the minimum order size, so it'd actually scale better for monitoring every store car park than for looking at a single car park.
So if Macy's parking lots have 11% more cars than the same time last year, is that a buy or a sell? Are people actually buying more, or are they more cash strapped and spending more time looking for value?
How busy are the car parks by their dispatch center? are the cars staying longer because people are working overtime? how many UPS trucks are visiting?
you buy data flow data from ISPs at all tiers, so even though they're encrypted, knowing how much traffic is going to Macy's.com vs JCPenney.com gives you information you can act on.
We know this is being done, because of reports that say Netflix is X% of Internet traffic. The undredacted reports from those same data sources have much more detail. It's also why some apps that don't appear to have any business model are actually quite valuable.
reply