Hacker News new | past | comments | ask | show | jobs | submit login
I push, therefore I am: two days at Etsy (2013) (mhawthorne.net)
173 points by luu on April 23, 2020 | hide | past | favorite | 75 comments



> I think it is important that engineering organizations identify a primary thing that they want to excel at. It affects the day to day mindset of the engineers, gives context to help with decisions, and provides guidance for what types of engineers we want to hire.

Aligning expectations is extremely important, especially when hiring - the culture of an engineering organization has an outsized impact on the final product. I have seen a lot of contention when people are brought into an environment where they expect to be moving fast, only to realize soon into the job that stability is much higher valued than frequent releases.


I agree with your general point about engineering culture (there's also more broadly the company culture) and it's always frustrating when management can't decide what the engineering values actually are to any real specificity, leaving it in the hands of each team doing interviews.

I disagree though that stability and frequent releases need to be traded off -- it seems only at infrequently releasing orgs do people think that. As for new people coming in and being surprised by the slowness, I see it most happen with acqui-hires rather than new hires, since new hires who want to avoid being surprised by such things tend to ask enough questions (like "how are releases done?") in the interview stage.


Bryan's amazing talk on this nails it for me https://vimeo.com/230142234


It was really interesting to see Etsy experiencing the same issues as Netflix in serving multiple devices and UI styles with the same API, and attempting to solve the issue with a similar conceptual approach but with a totally different technology stack. I think we will see a lot of new ideas in this space over the next few years as more companies feel the need to adjust their API designs due to device coverage, bandwidth constraints, UI innovation, and so on.

Less than two years before GraphQL was publicly announced


> 6. Maybe PHP Doesn’t Suck

Being a PHP programmer, I can concur: https://stitcher.io/blog/php-in-2020


> While many of the arguments against PHP still stand today, there's also a bright side: you can write clean and maintainable, fast and reliable applications in PHP.

You can, but in my experience, most people don't. Especially in Wordpress land. I've paid for plugins with egregious problems that "work okay" because error reporting is silenced.

> if you want to, you can write modern and clean PHP code, and leave behind much of the mess it was 10 years ago.

But most people haven't, and I'm not really interested in spending my career and hobby time playing Russian Roulette with coworkers and collaborators.


> You can, but in my experience, most people don't.

The same argument can be said of all languages.

It's true with PHP because the language is popular.

It's true with Java, because the language is pop^Wthe holy grail of entreprise software language.

Having seen both worlds in small and large corporate settings (and others, in Ruby, Python, C, Perl), I somehow prefer PHP, because at least, it does not pretend to be "the right thing" for enterprise software.


>The same argument can be said of all languages

At least languages like Java and C# require static typing (though I'm sure there's some lazy sociopath out there writing C# with every single variable as a dynamic), PHP doesn't even have that minimum bar


Neither do Python, JS, or Ruby. Personally I’d say a good dynamic language like Python is better than a rough dynamic language, but a good static language is better than a good dynamic language.


And that is a solid argument against Python, JS, and Ruby.


Yeah I'm pretty sure the language that has birthed conventions like HasThisTypePatternTriedToSneakInSomeGenericOrParameterizedTypePatternMatchingStuffAnywhereVisitor and RefreshAuthorizationPolicyProtocolServerSideTranslatorPB are rife with great practices that are in no way obscured by layers upon layers upon layers of half-assed abstraction.

The one with System.ServiceModel.Install.Configuration.ServiceModelConfigurationSectionGroupCollection in it surely musts be awesome too; I've heard great things about it.

The footguns in a given language being more ornate and towering than in another doesn't make that language better, but the ability to produce resilient, maintainable solutions for a given problem space in it.


Right, but a type system is as good as it is used.

And Java having a good one is no protection against the development cultures of large (consulting or non-tech) companies where the main metric is not the performance of the software neither the frugality of the hardware it runs on.


Can it be helped with a good linter? JavaScript had similar problems as there were more bad things about it than good, but with ESLint and a good config (Airbnb's or other) community evolved and most of the new stuff adheres to best practices by default.


But the stack in JS is a terrible burden in itself, and people only accept it because we have no realistic alternative to JS on the most popular plateform in the world: the web browser.

PHP doesn't have this accidental monopoly: you can use a language that doesn't require discipline and tooling to make something decent.

Bottom line, PHP is better than it used to be. But the technical debt remains, and it's easy to avoid if you can, so we do.

I started with PHP, but I have no reason to go back to it. It has no benefit and is worth it only if you have already a massive PHP code base, or a pool of PHP devs you need to employ.


If you know a magical language that can result in excellent code without discipline, I'm all ears.


I don't know of a magical language that can result in excellent code without discipline. But languages do vary significantly in how hard it is to take non-excellent code written without discipline and safely make it better code. The dynamic scripting languages are on the bottom of that list; all the freedom they enable also simultaneously make that harder.

This is probably the best way to understand the increasing push towards static types lately. It isn't so much necessarily the experience of writing the code the first time, where static types may at least sometimes indeed be a bit harder to work with. It's that it's a relatively easy to work with code that takes a list of OrderIDs and CustomerIDs and returns a map of CustomerID -> Order, and calls a function that takes an OrderID and yields an Order, a method that takes an Order and yields a Customer structure, and a method that takes a Customer and yields a CustomerID, and finally constructs exactly the promised structure. You can change it and feel pretty good about what's going on and what's going to happen both to the code it calls and the code that call it.

On the other hand, working with code that takes a something and a something and returns a something, calling a function that takes as something and yields a something, another function that yields a something and a something, and a method on one of those somethings that takes a something and yields and something, and then returns something, is harder to work with, before we even talk about the fact that these somethings may have no literal source code existence because there were dynamically assembled, and after the somethings were dynamically assembled, other somethings may have come along and turned those somethings into something else.

Obviously, the first case still isn't heavenly and the second case isn't as bad as I make it sound in this brief summary, but there's still a real difference.


I think the spirit of their argument is that there are plenty of languages with more benefits from an engineering standpoint and fewer downsides. Static typing, cleanser language construction, etc.


> Can it be helped with a good linter?

No. But it can be helped with a tech lead/team lead who can kindly but ruthlessly enforce good coding practices. I was lucky to have one of those when I was learning Java, and a lot of the habits they've taught me ended up being pretty universal, and I owe them a great deal of gratitude.


I would not classify Airbnb's linting rules as good in ANY metric for server side code. It has rules designed for frontend, and then people go and use those settings for NodeJS...


Can you expand on this? Why would linting rules, which are almost always strictly about formatting, impact server vs. frontend performance?


One example is they disallow for of loops because they are hard to transpile to es5, but that makes no sense for NodeJS.


people can laugh at php, all they want. I personally, don't use php, but admire the community. because they, python n ruby dev's are good at shipping software frequently. look at laravel ecosystem. people getting shit done. n contributing i.e tailwind css | ui | alpine.js


There's no linter for hiring candidates. When you announce a PHP position, you are acknowledging that you'll get someone that doesn't care about maintainability, security, and whatever. Candidates will react accordingly.

And yes, Javascript has similar problems, but much less severe.


Every job where I've done PHP has been sold to me as a job programming in something else. :-)

I do think that one of the big problems for PHP is that people who voluntarily do PHP instead of something else are immediately suspect. Why aren't they interested in learning something else? Do they not really care about code quality? Do they just say 'good enough' and only give a half-assed effort?

Modern PHP is fine, comparable to other dynamic languages but with some ugly weird bits (`empty`). However, most PHP I run into is not modern. It's leftover code from the 00s that has huge security holes that no one has noticed yet. If someone wants to make a new project with Laravel or Craft CMS, I wouldn't stand in their way, but I can't really imagine doing it myself just because of all the flashbacks to code that parses query parameters and calls the database in the middle of a templated for loop…


"I do think that one of the big problems for PHP is that people who voluntarily do PHP instead of something else are immediately suspect"

I write lots of PHP for work and for most of my personal projects. Learning other languages is just syntax, when you know the core concepts.

I once converted a personal project over to Python 3.X in a couple of weeks and it was much slower and took more memory than the PHP counterpart, so I scrapped it.

"However, most PHP I run into is not modern"

This may be the case for you, but I've worked with lots of companies that built their apps using PHP and most are modern.

"but I can't really imagine doing it myself just because of all the flashbacks to code that parses query parameters and calls the database in the middle of a templated for loop"

I'm sure there were many poor practices with other languages 20+ years ago.

I think many here on HN are living in a bubble where most companies use cool new frameworks and are looked down upon for their stack choices.

I haven't been out of work in the past two decades, and I work almost exclusively with PHP code bases. I suppose I should be happy for all of this language snobbery, because I make a very good living and haven't spent more than a few days between gigs.


> I'm sure there were many poor practices with other languages 20+ years ago.

I agree with most of what you wrote, but not this. Code written in PHP from the 5 and before era was bad in a unique way. Before it came bad VB and now there is a deluge of bad React. That’s what happens when a language is popular with inexperienced programmers. But bad PHP was bad and insecure by default, which led to a ton of security bugs. The only thing comparable is C, which has had tons of buffer overflow security bugs—but those were written by pros. ;-) I can’t think of anything else web facing with so many problems with SQL injection and string escaping. The problem was that making a webpage by naive string interpolation is a bad idea, but it seemed like a good idea and it’s what PHP did best and why it was so popular. PHP was lucky to be the wrong paradigm at the right time to take off like a rocket.

Modern PHP is MVC-ish like everything else, but without the simplicity of stupid string interpolation, it can’t attract users like bad PHP used to.


I think the parent comment is just describing some of the perceptions they've seen about PHP. I've done PHP work, but I've noticed the same perception. Also in my area, there's a lot of PHP work, but it's generally much lower paid. I know a lot of tech people here, and some of them are writing really modern PHP - for much lower pay than a .NET developer at a comparable job. It's definitely possible to do great work in PHP, and it's also definitely perceived (rightly or wrongly) at least in some circles to not be awesome.


> Every job where I've done PHP has been sold to me as a job programming in something else. :-)

"Oh and we also have a couple [dozen] Wordpress sites but I doubt you'll be touching those much."


Facebook has almost certainly the world's largest codebase in any PHP-family language (Hack, which is essentially PHP + gradual typing + a better standard library, but still very much PHP-flavored). The code is fine.

Part of the reason is the improvements of Hack over PHP, part of the reason is very aggressive linting (to, e.g., enforce the use of types), and part of it is probably that Facebook doesn't hire for "PHP positions", but for generalist software engineers.


Facebook indeed does not have the same hiring bar as "$company php (sweat)shop".

However, all the other companies align themselves to the level of the average candidate they can hire.

The quality of the library ecosystem in both js and php is a good measure of that level - sadly.


Anecdotal, but I was involved in technical interviews for a full stack developer position within a team last year.

Our stack at the time was transitioning to .NET Core, but the HR posting was some generic one (I believe for a "full-stack dev" with PHP experience).

Got a bizarre range of candidates who had got past the inital filtering, all except one whose PHP amounted to wordpress only experience (some who were not able to clearly differentiate between serverside and clientside state or logic).

Don't want to tar folks with the same brush, but that experience led me to believe the folks who got filtered through for possessing PHP experience, in this case, were extremely bottom of the barrel.


Honestly, if I'm ever in charge of hiring, I will probably be heavily biased against anyone who mentions having PHP experience without a deep, regretful sigh as an accompaniment.


> When you announce a PHP position, you are acknowledging that you'll get someone that doesn't care about maintainability, security, and whatever.

It's also announcing your pay will be way on the low end. At least that's what it means around here.


That section really should have been retitled:

6. Maybe Monoliths Don't Suck

There really wasn't anything PHP-specific there. My monolithic Java backend restarts much faster than webpack can rebuild my frontend. Complex microservice systems (like Netflix's) make for complicated developer setups in any language.


Are you using Spring? I've seen medium-large Spring Java apps take 10-15s to start. Not to build - to start. They're usually very fast then though.


as someone that knows vanishingly little about PHP, is there anything in PHP I can't do in say, TypeScript (or at least not as expressively?)


It’s less about the language and its expressiveness. PHP has a shared nothing concurrency model. For the typical Web 1.0 requires response processing it’s quite convenient.


> the biggest problem we face is maintaining an understanding of what is happening in our system

This really resonated with me. Distributed systems are _hard_ and tests (while valuable) are no substitute for being able to reason about what your application is doing and what might be affecting it.


I was stuck by this part too. It made me think of advice I read recently that went something like this:

"The best way to see the results of a change to a complex system is to make it and see what happens"


From the article:

5. Designing a Platform is Designing an Organization

An ex­-colleague of mine taught me the phrase “you ship your org chart”, meaning that the structure of your system in Production usually matches the structure of your organization. In my experience this is true.

Also known as Conway's law, this was the single biggest lesson at my last job (just before it ended). I've come to believe that nearly all code problems are actually people problems.

So for example, we aren't really web developers, we're coders who work around the numerous problems introduced by decades of design by committee. We aren't game developers, we're coders who make forward progress despite a level of wheel reinvention not seen in any other industry. And so on and so forth.

Me? I've decided to take a break from programming for a while to let the churn die down. I'm still listening for possible gigs, but for the most part I'll be working on the other areas in life that I've neglected over the last ~20 years. Maybe get back into art, blog a little, invent something. Unfortunately I don't have the luxury of a large savings account or investments, so I'm doing this on faith and relying on the evolution of humanity in these times.


I really like the idea of a developer exchange - does anyone know of other places that have done this?


I recall us doing this a couple of times when I was at Etsy (2012-2015). I think we did it with Github as well.

The thing to keep in mind is that accepting non team members and getting them meaningful work to do was already part of Etsy's culture. New engineers would bootcamp with a team for a couple of weeks when they joined. So the culture was already set up for this - managers and team members knew what to expect from a teammate who may not be a domain expert.

I think the way to start a program like this is to start with an internal bootcamping type program and learn and make mistakes through that.


The Guardian (my employer) and the BBC ran sessions to exchange about how to make a news app.

It's on pause given the current situation, but it was very informative!


What kinds of problems would you face in creating a news app? Surely content delivery is the same as that through the web page and for displaying the content you just use some cross-platform UI library?

(From somebody on the outside with absolutely no idea.)


There's a little bit more to our news app than content delivery, even though content delivery is indeed the crux of the problem. Regarding the content itself, it's a more complex than a wall of text, so we need support for each type of content. Our content is also served differently for the website and for the mobile app as we've got an API in there, though both the website and the mobile api are communicating with what we call the content API.

Both the Guardian and the BBC have separate iOS and Android teams, I'm part of the server side team. We touched about how to organise the teams, how to handle request from editorial, how to test and release the app etc.


In the lingo, a "news app" is typically some sort of interactive presentation of one particular story, and depending on the story may allow the user to filter or visualize something from a database, it might change the information based on where you live and it can include interactive maps or charts or calculators or even mini games to illustrate a particular concept. So these are one-off projects made to deadline, not a generic mobile app for reading the news.


Like that coronavirus WaPo article[1] from last month with the interactive graphs?

[1] https://www.washingtonpost.com/graphics/2020/world/corona-si...


Yup!


I was thinking the scope might include news acquisition (creation!?) and not just publishing.


That wasn't part of these exchanges, but yes we produce the tools for that in house.

The Grid is a good example of what's been produced in house, and it's actually open source (and I think being used by the BBC?) https://github.com/guardian/grid


Huh, interesting idea... I guess such a thing could be quite interesting for some short form updates (i.e. live from a conference). You'd probably want that to go through some kind of editor right? I just assumed it would be a news-reader/viewer.


We do indeed have what essentially is a text-editor with Guardian specific features developed in house.

It's used for all of our content, but specifically what you're referring to are liveblogs, where a journalist can write short content and append it to the existing article.

Here's an example of such an article https://www.theguardian.com/world/live/2020/apr/23/coronavir...


Cross company is awesome but even better would be cross team within the same company. No single thing could do more to break us v. them thinking and silos than this imo


We have an infra rotation program, where application and service programmers can spend a quarter or two working under product infrastructure on whatever their infra passion is. It’s great to see somebody’s passion for a big change boil for a year or two, and then they rotate in and are given the authority to make that change.

That being said, I’d love to see people rotate between application and service teams. Some applications and services are years ahead of others in terms of their maturity and scaling problems/solutions, so there’s a lot of tacit knowledge to be spread. We have a monorepo so self-starters can read code and commit messages, but a formal program would really help accelerate and democratize the knowledge spreading.


Yeah my former company would have benefit a ton from something like this. I guess it takes a culture to make this a thing. Instead we just built abstractions around the “other” team’s product so that we didn’t have to deal with their bugs or tech debt.


Artsy did a randomized dev team swap, where team leads stayed and everyone under shuffled: https://artsy.github.io/blog/2017/09/11/DevSwap/


Etsy sounds fun to work at. I like their ideology from this guy's perspective.

Too bad we couldn't see the other perspective. From what I hear Netflix is generally nice to work at too, but from the sounds of it may be more stressful.


The article was really interesting but given it was written in 2013 I wonder how much of it is still true. They're both probably very different places to work than they were 7 years ago (for better or worse or same same but different).


> A lot of the Etsy engineers seemed to find this fascinating. I think the main reason for this is that they run in a data center and control their own hardware

I'm shocked that Etsy, then or now, is at a scale where that's even close to making sense, that it would even be a discussion.

I wonder if it still is, I don't know how its size has changed over the intervening seven years, but with the explosion in popularity of I/P aaS I think it would be even more surprising today.


This is fairly old at this point; we were in the process of migrating to the cloud for several years and completed the bulk of the effort recently: https://cloud.google.com/press-releases/2020/0219/etsy-compl...


Do you have any insights into performance or cost difference?


Not strictly, and I'm not sure what's public knowledge and what isn't, but as I understand it the biggest win was being able to scale quickly (especially during the holiday season) without having to put in orders for hardware months in advance. As I understand it, cost wasn't a driving factor in the migration. I think with regards to performance, it's a bit hard; comparing a cloud-native solution to a custom solution running on a VM on a server you manage can be really tough to compare.


Own data center rarely makes sense. Colocation can work out well for companies with a lot of demand where one location is sufficient.

Dedicated servers (where you don't own the hardware) can make sense pretty quickly, the cost of managing that can be lower than cloud offerings even for mid-sized companies. PAAS or IAAS didn't really change that. Once you have a certain size, having your own department to manage hardware isn't that expensive in comparison.


We're almost entirely migrated into GCP now, roughly 95% done. Here's a CodeAsCraft article from when we started the project in 2018 about how we selected GCP vs AWS or Azure, and what the expected effort to migrate would be: https://codeascraft.com/2018/01/04/selecting-a-cloud-provide...


This used to be true but we are now 95%~ in Google Cloud.


Sometime these decision are made because, say, the co-founder/CTO just loves his machines.


Is there anything else like this from the last year or so?


Dan McKinley's talks are not that recent, but are similar:

https://mcfunley.com/

He is another former Etsy developer.


Really cool read. I didn't know companies have these type of programs

> 8. Always Keep in Mind What You Want to Excel At This was very touching. I haven't thought about my work in this way. I have a general idea of what I'd like to do and improve on but it doesn't always translate to my day to day.


Is the site down for everyone else, or just me?


Mobile firefox gives me an untrusted SSL warning, after accepting I get a "Dreamhost site not found" error.


Same here. Tried with VPN from US, EU, no dice.


I'm now logged into work VPN (captive network) and it's reachable going out the company network routes. shrug

Edit: it's HTTPS - I use the SmartHTTPS plugin (including on mobile FF) and when trying to go this site with SSL it fails, if you force it to non-SSL it works.


works for me, loaded also very quickly.


Fantastic write up.

Curious and respectful.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: