I literally spent the last week speccing out a system just like this and you are completely correct. You’ve touched on almost every single thing we ran into.
Oh, I absolutely agree that the complexity is in these topics. I'm just sceptic that they're enough to turn a task that could run on a laptop into one that requires an entire cluster of machines.
The third party API is the part that has the potential to turn this straightforward task into a byzantine mess, though, so I suspect that's the missing piece of information.
I'm comparing this to my own experience with IRC, where handling the same or larger streams of messages is common. And that's with receiving this in real time, storing the messages, matching and potentially reacting to them, and doing all that while running on a raspberry pi.
> In the 1950s and ‘60s, as Hong Kong grew as an industrial hub, many people migrated to the rapidly expanding urban centers for better working opportunities. “It’s hard farming and fishing out there in these remote areas, so a lot of people moved to the city to work in the factories,”
That explained why they were abandoned, but not why they haven't been reoccupied, since HK's population has exploded since the 1950s.
The short answer seems to be corruption: a cabal of property developers colludes with the government to ensure prices are sky-high and nobody outside the system (like, say, whoever owns the land in these abandoned villages) gets building permits.
A missing part of the story is that so many moved abroad. So the land rights now belong to first or second generation migrants and there just isn't enough pull to get them back. Yet there are sentimental and family reasons to hold onto the land--it costs nothing. And the land plots can't really be sold to developers at a high price unless the whole village, or a section of it, all get together and agree.
I added “really” to imply it’s impractical. What developer would pay a high premium? They can’t build a big tower unless they have a ton of adjoining plots. The village is abandoned and the owners are all abroad or just disinterested. The developer would have to do a lot of leg work just to find these people. And if one of them objects then the thing stalls.
It’s a lot of effort and high risk. Much easier to buy out active villages where you can knock on a door, show them a check and do it that way.
Edit: actually it may be even more difficult than that as the inheritance right appears limited to the heir building one house. I’m not sure the plots even can be sold. https://en.wikipedia.org/wiki/Small_House_Policy
At least around me in CA I see folks every day looking at their phones while driving. I believe license requirements and overall driving safety is much higher in Europe vs the US. I have friends from Europe who are shocked at how easy it is to get a license in the US.
These are great tips for load testing any web service. Don't forget that you can also saturate memory, disk or network too, so watch those graphs too. I've seen load tests unable to saturate CPU because another resource was limited.
Also don't forget that you are load testing all the dependencies of your service. Database, caching tier, external services, etc. Make sure other teams are aware!
Also nothing beats real world traffic. Users' connections will stay open longer than a synthetic tool may hold them open due to bandwidth, they make very random, sporadic requests too. Your service will behave very differently under large amounts of real world traffic vs synthetic.
Other options if you are running multiple web servers is to shift traffic around to increase traffic to 1 host and see where it fails. That is usually a very reliable signal for peak load.
And don't forget to do this on a schedule as your codebase (and your dependencies codebases) changes!
Good on the author, but using a Virtual Private Server skirts very close to not self-hosting. When I read 'self-hosting', I imagined buying/building a physical server and either putting it into a datacenter or running it in your home.
Lately I've been thinking of creating a bare-bones HTML website of my own and maybe I'll run it on a Raspberry Pi at home. I think that would qualify as 'self-hosting'.
It makes little difference who owns the hardware in the data center. The important thing is who selects, controls, and manages the software that runs on it. Therefore that’s the main dividing line.
Moving from a PaaS to a VPS is 95% the same amount of effort and energy as spinning you own rust under your desk. Semantics matter but holding a definition to “need to suffer a power supply failure to count” isnt really necessary.
Between bots and the Reddit/HN hug of death, if you ever do this don't ever advertise your website (to avoid getting DoS'd) and put some firewall in front.
This reminds me of the book Flatland which is about what a 2 dimensional universe would be like and the experience of a 3 dimensional entity visiting it. Great read! https://en.wikipedia.org/wiki/Flatland
I wonder whether anyone has done a Pac-Man variant inspired by Flatland.
Say, a powerup temporarily turns the screen to 3D, with the ghosts only perceiving the 2D slice. Pac-Man might be a cone that can 'shrink' to squeeze around ghosts, can jump through 3rd dimension to 'teleport', can move back and forth between different 'levels' floors in a building, etc.
It's not exactly Flatland or everything you describe, but there is an old arcade game named Pac-Mania where Pac can jump over the ghosts in 3D but they only move in 2D.
The real learnings of this incident is how to handle incidents effectively. The author cites ones that I use every time I manage one:
* Centralize - 1 tracking doc that describes the issue, timeline, what's been tested, who owns the incident. Have 1 group chat, 1 'team' (virtual or in person). Get an incident commander to drive the group.
* Create a list of hypotheses and work through them one at a time.
* Use data, not assumptions to prove or disprove your hypotheses.
* Gather as much data as you can, but don't let a particular suspicious graph lead you into a rabbit hole. Keep gathering data.
If you don't do the above you are guaranteed to have a mess, have to repeat yourself over and over and waste time.
Yeah tbh after working with Ruby for nearly 2 decades now and with a handful of other languages I think Ruby could use some parentheses and other common symbols. It really is too easy to create what are DSLs which requires learning another language and layer of abstraction.
Social media org like Linkedin is hardly comparable with SO style KB the R/W workloads are much higher with Social media, than a KB where vast majority of the transactions can be cached, Wikipedia would be a better comparison to SO.
Also Linkedin is 100x the size in users : 1B vs just around ~15M for StackOverflow.
All of a sudden the scale makes much more sense although I will have to fact check this because the number seems absurdly high for a professional platform.
DAU/MAU is the more accurate metric to track by. Not as many people would log in to linkedin everyday also people create a profile and never log in again or only login when they are looking actively for a new job.
Facebook itself has been having significant loss of engagement in recent years. Instagram and WhatsApp are far more likely to have higher DAU than FB main app.
I highly recommend anything by Adrian Tchaikovsky. His themes and ideas are deep and fascinating. https://www.amazon.com/stores/Adrian-Tchaikovsky/author/B002... (Children of Time Series, Dogs of War and City of Last Chances are great). Him and Dan Simmons are my favorite new(ish) authors. If you haven't read the Hyperion quadrilogy, it's my favorite sci-fi series ever.
I hate to be that guy, but I find Andy Weir's books to be very simple and 'light' sci-fi. Fun reads but usually suffers from the protagonist being a genius and smart aleck. No deep themes or characters. I never understood the crazy appeal after reading all of Asimov and Arthur C Clarke. Personally I want giant themes, huge space operas, complex characters and long storylines!
Thanks for the suggestions. I read the first book in the Children of Time series and really enjoyed it. I don't think the next two books had been released yet when I read it, so I definitely will go and follow up on those. I will give the Hyperion quadrilogy a look, too.
I agree with your take that Andy Weir's writing can feel a bit lacking - at least in the Martian. But I was willing to give him a bit of a pass based on how the story was originally self-published in serial form. My hope is that Project Hail Mary will be a bit more refined - but even if it reads just like the Martian, I am definitely interested to give it a read.
I enjoyed Children of Time, but I also was frustrated as the characters are a bit flat, they don't have any type of arc or change or learn in any way. The main character was interesting but the rest just kinda there. I loved the world though.
* Reading/fetching the data - usernames, phone number, message, etc.
* Generating the content for each message - it might be custom per person
* This is using a 3rd party API that might take anywhere from 100ms to 2s to respond, and you need to leave a connection open.
* Retries on errors, rescheduling, backoffs
* At least once or at most once sends? Each has tradeoffs
* Stopping/starting that many messages at any time
* Rate limits on some services you might be using alongside your service (network gateway, database, etc)
* Recordkeeping - did the message send? When?