There isn't a one-size-fits-all solution to this issue.
Before joining the department, I was part of a start-up where, with a considerably smaller team, we were able to process even larger amounts of data. We could introduce a new feature in a single day - a feat unthinkable in the department I just described. This speed was not achieved at the expense of stability but was a direct result of our choice of tools.
Addressing the aspect of "questionable design and lack of readily available support", I believe only a handful of individuals are capable of developing something on the scale of Docker or Kubernetes, let alone doing it effectively. Assuming you're fortunate enough to hire such individuals, would you genuinely want them dedicating their cognitive resources to these types of problems? Even if they are highly skilled, can their solution truly compete with industry-standard frameworks developed by a team of equally competent individuals? And even if such an individual could design a superior solution, they are now responsible for maintaining it and training juniors on a system they likely have little motivation to understand.
Moreover, the 'Not Invented Here' syndrome can lead to a cascade of more of the same. For instance, we needed to orchestrate several processes. Had we adopted Kubernetes, I would have suggested utilizing Airflow, which could have been implemented in just a few days. However, we chose to develop a custom pipeline runner, which took several months to complete.
I understand that the no-frameworks sentiment has its roots – perhaps from Java's insistence on using frameworks. However, it seems that the industry swings from one extreme to the other. Golang emerged as a reaction to a world overrun with frameworks, but it appears to have veered too far in the other direction.
Dare I suggest that it is possible that the nimble startup was moving quickly not necessarily because of tool choice, but because of the small team size and because the appropriate amount of complexity was introduced? And conversely the inflated team was slow exactly because it was inflated and complexity was introduced? This effect happens with or without bringing third-party or in-house tooling/frameworks into the equation, and is well known and well covered since Mythical Man Month.
And no, I don't see the point of rebuilding Kubernetes and Docker in-house and most critics also don't. What I'm saying that teams will be better of if they work towards not needing Docker or Kubernetes (or any NIH replacement) at all. But that requires challenging the assumption that software needs "something" like Docker or Kubernetes.
Sure, if there's a tool for X then by all means use it, but my point is more that not doing anything in-house can often lead to the same issues.
The speed difference boils down to flexibility and tool usage. In the startup, I could swiftly create a new service, harness Kubernetes for deployment, and seamlessly integrate it into the existing architecture. Data computation used tools like Airflow/BigQuery, storing results in production PostgreSQL.
In contrast, with a Golang monolith, service integration often involved complex, time-consuming conversations due to rigid NIH abstractions. And given the tendency of senior engineers to resist changes, progressing became challenging. A developer like me would be caught between resistant seniors and a product team demanding immediate feature release. Additionally, data processing often required extensive custom coding, slowing down operations. And if more than one machine was needed, it would mean a total overhaul and a long wait.
In comparison, a lean startup allowed me to use apt tools (like BigQuery), containerize it with Docker, and then let it be - without involving anyone senior. Moreover, in case of a new feature, we could simply replace the entire component instead of struggling with additions.
The agile startup emphasized less on seeking senior developers' approval for every change. In the monolithic Golang environment, senior devs often fixated on code quality without acknowledging that, in a microservices context, it's frequently more effective, faster, and simpler to reboot and rewrite parts instead of altering the existing solution. Hence, the intense focus on solution maintainability becomes less critical.
I think there's too many red flags there to chalk it up to a single problem: time-consuming conversations, rigid abstractions, resistance to changes, demanding PMs, total overhauls needed, more bureaucracy on change approval, you also complained about fixation on code quality.
Saying that it's all up to the one single difference even though there's so many ancillary issues screams of "I've found a silver bullet syndrome" to me.
Before joining the department, I was part of a start-up where, with a considerably smaller team, we were able to process even larger amounts of data. We could introduce a new feature in a single day - a feat unthinkable in the department I just described. This speed was not achieved at the expense of stability but was a direct result of our choice of tools.
Addressing the aspect of "questionable design and lack of readily available support", I believe only a handful of individuals are capable of developing something on the scale of Docker or Kubernetes, let alone doing it effectively. Assuming you're fortunate enough to hire such individuals, would you genuinely want them dedicating their cognitive resources to these types of problems? Even if they are highly skilled, can their solution truly compete with industry-standard frameworks developed by a team of equally competent individuals? And even if such an individual could design a superior solution, they are now responsible for maintaining it and training juniors on a system they likely have little motivation to understand.
Moreover, the 'Not Invented Here' syndrome can lead to a cascade of more of the same. For instance, we needed to orchestrate several processes. Had we adopted Kubernetes, I would have suggested utilizing Airflow, which could have been implemented in just a few days. However, we chose to develop a custom pipeline runner, which took several months to complete.
I understand that the no-frameworks sentiment has its roots – perhaps from Java's insistence on using frameworks. However, it seems that the industry swings from one extreme to the other. Golang emerged as a reaction to a world overrun with frameworks, but it appears to have veered too far in the other direction.