> "The company has 685 employees of which approximately a third are engineers."
Given that they've produced some impressive things, Etsy always seemed to be the example of a shop that was overdoing it - but I had actually assumed they were overcomplicating it with 20 engineers or so. Looking at their github now, and hearing those recent numbers, it feels much worse. That level of work is great, but for a company that does what Etsy does, much of it seems like overkill. (Managing 1000 instances should be the same as managing 100, this is where automation kicks in. Architecture enables that.)
Ultimately, that's a tremendously large amount of engineers, to where I can't really mentally assign what all but about 20 of them might do. Let's be generous and say 30 - or even 40 :) All of that eats into the bottom line (and also makes adding more staff harder, as your tech becomes more and more custom).
Really, at the basic level, it's a CRUD-based storefront website. Yes, it has scaling needs. But it's not really novel in that regard.
Yet, they've built more automation around their automation (lots of things in the Chef ecosystem) and probably made things a lot more custom.
I can understand some of this for say, multiplayer gaming or video streaming infrastructure maybe, but I would suspect Etsy could have been run a lot easier with less wheel reinvention.
Startups should really strive, IMHO, to write as little of software as possible, that's outside of their line of business. When wheels are square, build better wheels sure, but what I see from that number looks like MASSIVE levels of maintaince costs for being that custom.
So anyway, thanks for producing things that other people use and that are useful - YES - but from the startup accounting / efficiency perspective, I wouldn't want to see my company run that way.
Appreciate your insight on automation, and fully agree with the need for startups to use commodity software as much as possible and focus on business needs. However, I suspect most of the 200+ engineers would be needed to keep the business running in either case.
They've written StatsD (Node), a custom deployment framework, an A/B testing framework and a bunch of other technology software (https://github.com/etsy) that could need dedicated engineers to build and maintain. But all of this would pale compared to the needs of the business itself.
At sufficiently large scale, any CRUD-based storefront website becomes an entire supply chain management system. I suspect there are many ways to slice up an SCM, but from a functional level, you'll need subsystems for each part of the domain: managing customer orders, warehouses, logistics, payments, offers & promotions, developer facing APIs, accounting & book-keeping, ..
These systems can be built using a service-oriented architecture, but Etsy is famous for promoting the large monolithic app architecture (http://highscalability.com/blog/2014/7/28/the-great-microser...), which would remove the need for a lot of SOA plumbing work and communication overhead.
Apart from the core business logic, they'll also need a whole bunch of QAs, Android and iOS developers, designers, DevOps folks, database admins, security specialists and so on. All told, they could probably run a leaner ship than the 200-odd engineers they now have, but I wonder how much more leaner.
My main point was that writing all the custom stuff was why the staffing numbers were so high.
One small correction - In this case they also do not have warehouses since items are shipped from craftspeople. I also would expect mobile apps could be done by a team of a handful of people.
As for SOA vs monolithic apps (highscalability link currently is a 404), it's largely horizontal scaling and a load balancer. That in itself shouldn't change architectual choices.
I think this article addresses what they do; they work their way down the stack and broaden what they provide to the business vs outsourcing to some SaaS company.
Sure, you could run this in the cloud and not worry about physical machines, but that will probably be more expensive. As companies grow, it becomes cheaper to bring lots of things in house, rather than outsourcing them.
As companies grow, small % differences can result in large total impact, at which point having engineers eek out small gains may make a lot of sense, e.g. think of Google's results around latency, or small improvements in recommendation engines that drive higher sales.
> you could run this in the cloud and not worry about physical machines, but that will probably be more expensive.
I question what the current cost structure looks like. As a public company, my guess is that investors will do the same as well.
In a world where Instagram deals with orders of magnitude more customers with a whopping five engineers, it's a pretty high bar to show why so many resources are necessary for this particular operation.
I'm sure the Etsy guys are sensitive to that as well, but from the outside it's hard to believe there isn't a better way to use resources more efficiently.
Very good point. A transaction involving a user's money carries a great deal more weight in terms of reliability, accuracy, etc.
That said, the resource difference between these two companies is considerably large, and the tech in question (e-commerce) is not exactly an area of unknown innovation. From a technical standpoint, that's sufficiently a solved problem.
It is very difficult to imagine what kind of scaling needs are, if running in the cloud is more expensive than having 200+ engineers. Even salary of one engineer buys quite a gigantic amount of hosting.
Sounds like they really drank the commodity developer Kool-Aid. What I mean by this is heavy process/standards with tightly controlled tech stacks which attracts salaryman types. They don't try to attract talent, they just want enough bodies that enough work gets done to achieve their goals. It eliminates the need for recruitment by throwing money at the problem. I've mostly seen this at very larger companies but it is not restricted to them.
Completely different domains and polar opposites on levels of technical requirement. Etsy is an end user web application (don't get me wrong, it's a great service). AWS is pretty much the pioneer in IaaS. I'd say Amazon (AWS) built the system so folks like Etsy wouldn't have to build as much as a system.
That's getting the causality backwards. AWS exists because Amazon had so many engineers working on these problems it made sense to generalize it as a separate product. It wasn't like Amazon didn't have a lot of engineers but decided to build AWS so then hired a bunch of engineers.
Well technically, Amazon built AWS so they could make money from other companies who can't, or don't want to, run their own infrastructure.
At the scale of Etsy, they don't get any benefit from AWS - they're certainly big enough to justify the cost of an operations team, at which point AWS is just an expensive exercise in vendor-lockin.
It really surprises me how many people think that being more self-reliant and understanding/controlling more of your dependencies, is a bad thing. It's not always possible, but why wouldn't you, if you had the resources to do so?
The only time I got close to an answer on stuff like this was when I asked why the company behind a large tv/broadcast network insisted on using "vendors" for every little piece of technology - they would pay for hosted forums and battle through the issues with integration (of which there were many) rather than picking up an existing forum product and customising it as necessary.
The only answer I was ever given: they want someone to sue when it goes to shit.
I will give you another answer: I cannot hire developer talent with the company name & sexiness / vision with budget to attract and build a team that would be needed to build the software in-house. This is typical for most enterprises that really aren't in the software business as a legit line of business. So it's pretty much only possible to just pay a vendor and deal with all the BS of having to learn another party's software while having the downsides of procurement and politics in big companies.
The throat to choke argument is not sufficient anymore for a lot of enterprises though because they are sick of their vendors giving poor results or ones not aligned to their own goals in the product lines. I haven't exactly heard of anyone getting actually sued, but you can make a little money back sometimes if a vendor doesn't meet their SLA.
I started advocating mostly open source solutions partly because using vendors is like taking drugs and eventually they'll start dictating your architecture rather than you pick your architecture, which is likely to conflict with a competing vendor's architecture. All of this talking and demoing is also costly, but these type of evaluators are fairly easy to find compared to talented / competent developers.
But really, companies that already have solid lines of business that can only grow a bit are far more concerned about loss of revenue than trying to grow n% faster in most parts of the company.
There are significant benefits to using AWS or a similar service, even when handling traffics loads large enough to justify the investment in bare metal. For starters, you can scale your infrastructure up and down as needed.
At the scale where you run your own dc you can save just as much by owning the hardware and using smart virtualisation to put unused physical resources into standby/low power mode.
Amazon didn't invent virtual infrastructure that adjusts dynamically based on load.
When I read callahad's comment, I thought it was referring to Amazon as a "CRUD-based storefront website". What would you say about the retail Amazon store compared to Etsy?
Well, Amazon also had to define what it meant to be a web store at scale.
I think that mpdehaan2's point was that (from the outside looking in, at least) almost everything Etsy's doing seems to be a minor variation on a solved problem. That was definitely not the case for Amazon.
> One way Etsy drives this kind of commitment to deep understanding that seems to underlie its technology choices, is limiting the number of tools it uses as a company. It would rather be the master of few tools than have a broad knowledge across many different ones.
Every large(-ish) company I know has pretty strict rules about the languages, frameworks, and tools used for the main products.
The more flexible of these do allow prototyping using a different toolsets. The ones that have a developer audience always experiment with new environments.
New tools and new processed get adopted over time, but in many cases the cycle they go through emulated pretty closely the adoption cycle seen outside. Even if the new shinny toy syndrome is very strong in this field, this makes a lot of sense, as nobody really wants to throw away everything. (_While writing this I got to think about the numerous rewrites eBay has seen over years_).
> The company owns and operates its hardware and networks in its own datacenter. Part of the reason for avoiding the cloud is control, but not necessarily just from a security perspective as you might think. The real reason is Etsy personnel want to get under the hood and understand every aspect of every piece of technology the company controls.
I continue to be very curious about how polarizing the cloud remains.
It's totally reasonable to say you wouldn't want to work in a company that prioritized unsurprising, proven technology over novelty and in-house solutions. That's a far cry from it being unilaterally a "terribly place to work" headed by a "moron".
Some programmers are more motivated by solving business problems than they are by inventing and adopting shiny new technologies. At this point in my career I'm definitely more interested in applying proven languages and systems to novel product problems than I am inventing new frameworks and debugging cutting-edge runtimes.
So: different strokes for different folks. Insulting people who disagree on issues for which there's more than one answer is just unnecessarily snarky IMHO.
I'd add that some programmers wish to build their exciting new technology upon a strong foundation rather than sand. Sometimes bleeding edge technology requires a non-trivial amount of work to keep up with breaking changes in return for something that can be obtained another way (isolation via docker vs traditional virtualization).
Sometimes boring technology choices are a sign that the programmers are innovating in their actual technology rather than blindly adopting whatever they see in blog posts.
> Some programmers are more motivated by solving business problems than they are by inventing and adopting shiny new technologies.
My stance isn't really about shiny new tech. I find flavor of the week stuff about as annoying as anyone.
Rather, my reaction is this is basically a puff piece. Fair enough, but as a professional reading about the supposed virtues of this guys "philosophy", my reaction is that I wouldn't want to work in an environment where I would be treated with so little esteem as etsy seems to regard their engineers. Maybe that's harsh, but that's my impression.
Here's an example of what I mean: "Shadow IT, that idea that employees can and will provision their own software (because they can), is everywhere because IT is powerless to stop it. Elliott-McCrea, however, is adamant that his company has no patience for freelancing when it comes to software and he claims it’s not happening." As a programmer, this strikes me as ridiculous -- I use non-company-sanctioned software to do my job all the time and nobody bats an eye (ie, text editors, graphics software, etc). If their trust of developers is so low that they won't even let them choose their own software, what does it say about how much respect they have for them?
If this were an objective analysis I wouldn't be so mean, but given that it's a puff piece, I don't see any reason not to be blunt about how toxic I think that CTO's ideas are.
At one place I worked, they let developers use whatever tools they wanted.
The result was that their website was dependent on 50 Javascript packages that took forever to load. They had parts of their system written in Javascript, PHP, Ruby, Python, and .NET.
That's how you get job ads that say "Candidate must have 2+ years experience in Python, PHP, Ruby, C#, and Javascript."
Imagine putting a big effort into Angular 1.0, only to discover a year later that it's no longer going to be supported. Is it so bad for Etsy to be avoiding problems like that?
The problem in this scenario is not letting the developers use whatever they wanted.
The problem is hiring the wrong developers, ones that use that freedom irresponsibly.
Keeping developers on a short leash doesn't solve that problem, it just covers it up. And it becomes a self-fulfilling prophecy, because any developer capable of making their own decisions won't work there, so what you end up with is people that need that level of micromanagement. (Yes, this is a form of micromanagement.)
And that's what I read in the whole article: an inexperienced CTO who has shaped his policy based on his own limited experience, not acknowledging that that vision is heavily influenced by his own weaknesses.
He is projecting his own attitude as a developer on all developers, and extrapolates from there.
I've met far more developers who would add some extra tech to the stack without a lot of deep thought about the actual consequences of doing so, than those who would either not do that, or discuss it with peers/senior devs/mentors etc.
If you give developers carte blanche permission to add whatever they feel is necessary, you're using your business success/failure to teach them. That is wrong.
As the CTO, the author's primary responsibility is to make sure the right tech decisions/solutions are made. It sounds to me like he's doing exactly that.
If an experienced, "responsible" developer thinks they can make good use of tech X, surely he/she will understand that there is a standard process to have this new thing approved for use, and then it can benefit the entire company.
If you can't identify to someone else why you think tech X should be used, what problem it solves, what process it makes more efficient, etc, then you're probably the wrong person to be making a decision about adding new software to the stack.
Any "engineer" (I really bet actual engineers love how software developers have co-opted that term) who is "responsible" as you put it, and wants to add software to the stack, will surely discuss that plan with at least someone, right?
Or maybe not. Maybe you think every developer should be allowed to make grand architectural changes to a platform worked on by 200+ other staff at their own discretion? If so I suspect somehow, that your reply is just a knee-jerk reaction to a place that expressly prevents cowboy-developer antics before they begin.
>If you give developers carte blanche permission to add whatever they feel is necessary, you're using your business success/failure to teach them. That is wrong.
Also, many developers will use whatever tools they feel like padding their resume with, instead of what is best for your project.
Yes, I've also met far more irresponsible, unprofessional developers than responsible, professional ones.
As the CTO, my first responsibility is to hire the people that make the "right" decisions/solutions. Of course they discuss those decisions with their coworkers and me. That's just common sense.
Sure, I'll make the odd hiring mistake, but that's no excuse to create and entire culture around the assumption that every developer I hire has the maturity and professionality of a teenage sociopath.
My reaction is a knee-jerk reaction to a culture of distrust, bureaucracy and micromanagement.
The alternative to letting the inmates running the asylum is not to lobotomise the inmates, it's to not turn the place into in asylum in the first place.
Oh I believe it - I've worked on projects where a single developer adds a dozen JS libraries/jquery plugins/whatever because there is no concept of how maintainable it is long term.
I don't even really like to call this "conservative". To me, "conservative" would be no XHR, no custom fonts, etc - the sort of stuff you know will work on IE5. I would consider the approach Etsy uses simply as "sane".
Because they place little value on the expertise of their engineering staff, as evidenced by the comical restrictions they inflict on intelligent professionals.
It says quite clearly, they have a process for adding new tech to the stack, and that developers have more freedom for their own environment than for things going into prod.
Just because you have to convince someone to agree with and approve your use of shiny-new-thing v0.0.0.1alpha doesn't mean they have "comical restrictions".
I should clarify: my opinion doesn't have anything to do with the use of shiny new tech. I think using tried-and-true tech stacks is perfectly sensible. My commentary has to do with the impression that engineers there are not trusted to make those decisions.
If 230 "engineers" all make choices about tech in use, you end up with a fucking nightmare mess of every imaginable tech stack mixed and matched.
It says very clearly they will use new things when the benefit can be shown.
So it still just sounds like you don't like that you would need to prove why you're changing something on a whim, because "oh this is better" doesn't cut it
Well, trust is a very important part of a working relationship. I don't change things willy nilly for fun because I wouldn't want to betray my employers trust in me. I've been in high-freedom situations in companies with lots of engineers, and you know what? Chaos did not ensue. Professional pride, interest in being able to collaborate with others, etc tended to reign things in just fine.
The article, to my perspective, suggests a situation in which there is very little trust towards their employees. Having seen both ends of the spectrum, I wouldn't want to work in an environment like that.
It turns out that engineers love working at Etsy more than almost any other company. You should try talking to some of them and wonder what made you come to a conclusion so divorced from reality.
Given that they've produced some impressive things, Etsy always seemed to be the example of a shop that was overdoing it - but I had actually assumed they were overcomplicating it with 20 engineers or so. Looking at their github now, and hearing those recent numbers, it feels much worse. That level of work is great, but for a company that does what Etsy does, much of it seems like overkill. (Managing 1000 instances should be the same as managing 100, this is where automation kicks in. Architecture enables that.)
Ultimately, that's a tremendously large amount of engineers, to where I can't really mentally assign what all but about 20 of them might do. Let's be generous and say 30 - or even 40 :) All of that eats into the bottom line (and also makes adding more staff harder, as your tech becomes more and more custom).
Really, at the basic level, it's a CRUD-based storefront website. Yes, it has scaling needs. But it's not really novel in that regard.
Yet, they've built more automation around their automation (lots of things in the Chef ecosystem) and probably made things a lot more custom.
I can understand some of this for say, multiplayer gaming or video streaming infrastructure maybe, but I would suspect Etsy could have been run a lot easier with less wheel reinvention.
Startups should really strive, IMHO, to write as little of software as possible, that's outside of their line of business. When wheels are square, build better wheels sure, but what I see from that number looks like MASSIVE levels of maintaince costs for being that custom.
So anyway, thanks for producing things that other people use and that are useful - YES - but from the startup accounting / efficiency perspective, I wouldn't want to see my company run that way.