The problem is that on-call is something in between work and not work. Depending on how often you are actually paged, it is closer to one or the other. As such, the policy I usually implement is simple:
- Being on-call is compensated, although at a much lower rate. The compensation is for the inconvenience of having to carry you computer with you, not being able to get drunk, not sleeping as well (if that applies to you), etc.
- When you get paged, you can generously compensate that time as TOIL. E.g. if you get paged and you realise it's a false positive and go back to bed, you can still compensate 2h as you need to fall back asleep and likely will not be as rested. If you have to do something for 4 hours, take off a day because you likely had to cancel some personal plans or have not had any meaningful sleep at night.
If I can't drop everything, fly abroad, and disappear into the wild because my boss needs me to do something, then I'm at work. Everything else is just an excuse to pay me less whilst still taking a tole on me mentally.
It actually depends on the country (and I assume even state and city). Where I grew up, most firemen actually sleep at home and not at the station. They have "normal" work hours during which they do non-emergency work, such as train, maintain equipment, raise awareness for fire prevention, etc. and they get paid for a target work week of ~40 hours. Being called out "outside" of hours is compensated, AFAIK in TOIL. Point being, there is no one size fits all model and it has to work for your environment.
Asking for being on-call for 7 days to be equivalent of 168h of work, i.e. after you have been on-call for a week you go on vacation for 3 weeks is just as unreasonable in most situations as companies asking their staff to be on-call without additional compensation.
I know the article is intentionally provocative and over the top to provoke a reaction... whether it's a healthy reflection on your attitude to success or crippling self-loathing probably depends on the reader.
Also, there are undeniably some hard truths hidden in there. However, my experience in almost 20 years in the tech/software/product industry often paints a different picture. Yes, our brains keep us from changing and evolving and yes, obviously you need skills to be successful in life and your career. But in my industry in particular, hard and soft skills are not the dominant factor that keeps individuals from succeeding or progressing. I'm lucky enough to work with an abundance of talent and skill, and yet, one of the major factors of dissatisfaction is lack of "progression". One of the main factors is self-confidence and in the more severe cases even mental health issues. Some of the most skilled and knowledgeable engineers I worked with struggled to realise their potential because of it. If the leaders in your organisation think they can just shout at them to "learn self-confidence as a skill" and get over it you're going to have a bad time. It will attract a certain type and employee that thrives in that environment and disengage everyone else. Wasting talent, wasting skills and ultimately a lot of money.
Creating an environment and learning how to tease the potential out of skilled and talented individuals is not a "hippie/hipster" thing to do, it is good for business.
Not necessarily. Granted, you obviously need to make a network call at some point but you can host the entire experiment configuration in a file that is served through a CDN and the mobile client only needs to download it once (and it can be cached however appropriate for your use-case).
The client SDK can then use the file to derive the experiment configuration by passing the required parameters to determine experiment allocation into the SDK which can then compute without further network calls.
It depends on your definition of fully embraced. If you mean that there is no synchronous communication between services, then no, and neither does it make sense in the real-world scenarios I am aware of.
However, I am an advocate of the pattern and have seen it used successfully repeatedly. The largest scale as the data lead for a product maintained by 100-200 developers and several thousand transactions per second.
To answer your specific questions
>handling breaking schema changes or failures in an elegant way, and keeping engineers and other data consumers happy enough?
We did not allow for breaking schema changes. If there is a breaking change, it's a new event/topic. We used Kafka and every topic needed to have a compatibility scheme defined (see https://docs.confluent.io/platform/current/schema-registry/a...) to clarify what constitutes a breaking change. Even though some claim that producers and consumers can be fully decoupled, you will need to have a good idea who your consumers are and the time horizon of the data they consume. Application engineers are usually easier to keep happy than machine learning practitioners and other data consumers that want to consume events emitted over a long time period, potentially years.
> As a trivial example, everybody talks about dead-letter queues but nobody really explains how to handle messages that end up in one.
Dead letter queues are a tool you can use when the context demands it, applying it wholesale is likely creating too much overhead. But to provide you with a specific example. Some emitted events will be revenue impacting and depending on your setup, you actually want to use the events for financial reporting (careful! some more info later). In this specific use-case, if you can't process a record, the last thing you want to do is throw the message away. Somebody will need to have a look at these records, fix the cause and then either re-emit the records based on what you know about them from the header or fix the records in the DLQ.
So think about the guarantees you need to provide and decide whether a DLQ makes sense for your use-case.
Some other thoughts and considerations.
- Topics more or less directly become analytics tables. Almost creating a unified view on your application's data otherwise difficult to create.
- How are the messages emitted. Are the messages emitted from the application logic? If so, what guarantees do you need? What happens if the app crashes (e.g. after a DB transaction happens and before the event was emitted). Depending on what you need, have a look at the transaction outbox pattern.
Which would become an one-way route, because once a nation has become a criminal-refugee paradise, why would third countries accept them travelling to their shores?
I second that. Once a vendor becomes important enough I even stop thinking of it as a vendor and rather as a partner that you interact with as if it was part of your own organisation. I am not amused when members of my team are rude to a vendor and jeopardise a relationship that took a long time to build. Even when you pay six figures a month, you can still get preferential treatment compared to other customers in the same tier when it comes to influence over the roadmap if you actively foster the relationship.
> When Google acquired Alooma, they slowed down the development and dropped the support for other destinations such as Redshift and Hive.
Do you have a source for that? As far as I can see on their website [1] they still support it
I haven’t personally played with scala yet, but do you need to add any additional runtime components for it to work? Or perhaps scalaz? Or is it just making a .jar and you are good to go? :)
I can second this. I always take 3-6 months off between jobs for travel and personal development (given I stayed there ~2 years). I'd say I technically progressed about the same during these periods as I did during the preceding employment (minus the real world experience) because I actually really enjoy it.
I have never once gotten a negative response in an interview because of it.
- Being on-call is compensated, although at a much lower rate. The compensation is for the inconvenience of having to carry you computer with you, not being able to get drunk, not sleeping as well (if that applies to you), etc.
- When you get paged, you can generously compensate that time as TOIL. E.g. if you get paged and you realise it's a false positive and go back to bed, you can still compensate 2h as you need to fall back asleep and likely will not be as rested. If you have to do something for 4 hours, take off a day because you likely had to cancel some personal plans or have not had any meaningful sleep at night.