A language that supported a platform for almost twenty years in the modern era? I can't call this anything but a wild success. Sure, it had issues, as discussed on the article, but they solved the business problem at the moment in time. No software decision is without costs, but running a major site for almost two decades indicates that they chose well.
Bravo!
I do wish they'd throw the source code up on GitHub, as it would be an interesting historical artifact.
This is the sort of programming language that I would love to read a detailed retrospective of in an ACM journal. It's a programming language designed to solve a particular problem, grew over time for that problem, and it successfully did so for almost 20 years. A programming language that is useful that long is interesting to me.
I'm not second-guessing the technical that the Times made here (I'm sure it made sense given their constraints), but I do wonder if these kinds of in-house general-purpose languages will die out eventually. Some of the comments indicating that this sort of project is a "red flag" seem to miss that 20 years ago the language landscape was a lot more proprietary/closed.
But these days, a big open source/community ecosystem is a really really strong reasons to invest in an existing language (or at least open-source your in-house language, a la Hack or Go). It's hard for in-house general-purpose languages to compete.
> Recruiting new engineers was sometimes a challenge because candidates had to accept that they were joining an organization with a homegrown programming language and build system.
I got an offer from NYT in 2009, but rejected it because of the custom language thing. My interview was conducted entirely using this custom language, and the interviewers were uninterested in discussing any other technology. Even architectural questions, which I attempted to answer in terms of industry standards, were steered towards Context.
> My interview was conducted entirely using this custom language, and the interviewers were uninterested in discussing any other technology.
Bizarre!
Writing their system in a custom programming language isn't _that_ unusual, I think. But why on earth during an interview wouldn't they let candidates program (entirely or mostly) in a language they'd heard of before that morning? Couldn't they assume that if someone has achieved a working knowledge of one or more similar languages, they can learn another in some reasonable time (days, weeks, months)?
I went on an interview one time in which the job was not for python, but in which the coding exercise was in python, the assumption being that they wanted to see how you handled something you didn't know.
The interviewer mentioned having another guy in who announced flat out he was not going to do it even though it was explained that you didn't need to know the language to solve the problem.
I of course had some python familiarity, but I still failed because I couldn't remember the name for binary search and that threw me for a loop when the obvious answer I should have gone for in the exercise was 'here we should use binary search'.
It’s usually a massive red flag of extreme tech dysfunction if someone was able to get buy-in to build entirely custom languages / platforms / compilers / etc.
(a) management was not competent enough to realize this is universally a poor investment and refute the project.
(b) tech leads either chose this to solidify job security and leverage for bonuses, or else the choice reflects incompetent tech leads you don’t want to be working with or inheriting code from (especially not in a custom language).
Some notable other examples:
- Danish startup Area 9, which wrote an utterly embarrasingly bad custom language called Flow, which was acquired by McGraw-Hill Education as part of some software that manages student progress in digital materials.
- Goldman Sachs famously with slang
- Standard Chartered with their own custom Haskell compiler and language extensions
- Dropbox (sort of) with Pyston (custom jitted Python implementation) that thankfully is defunct now.
So if Google or Facebook write themselves a new language (or two), I'm guessing you are ok with that? I have worked at Standard Chartered and their compiler was written by a world renown former CS professor, who also wrote the worlds first Haskell compiler. The quant code written in this language is some of the best I have ever seen on the street; certainly better than the python shops (who are using forked hacked-up variants of Python anyway). Goldman Sachs also had a huge advantage for many years thanks to their proprietary technology. Sometimes an in-house language is actually a sign of innovation.
> “So if Google or Facebook write themselves a new language (or two), I'm guessing you are ok with that?”
What gives you that impression? It would be equally (perhaps more) red flag if Google permitted this in-house, much like Google’s in-house monorepo tooling that grew out of a series of historical Perforce accidents is a huge red flag of dysfunction.
> It would be equally (perhaps more) red flag if Google permitted this in-house
But Google and Facebook have permitted this in-house (Go,Dart,Hack,Reason etc). They have since open-sourced them, but these projects are still run from within. I don't see this as a red flag, these languages all have good reasons to exist.
> Google’s in-house monorepo tooling that grew out of a series of historical Perforce accidents is a huge red flag of dysfunction
Google probably has the largest repo in the world, no off-the-shelf product is going to work. I fail to understand why building their own tools to accommodate their own special needs is a red flag of dsyfunction?
It’s totally disingenuous to compare general purpose languages like Go or Dart which were designed from the beginning to be general purpose, to the types of limited-purpose languages discussed earlier.
If you look upon an internal business problem and propose that the right starting point for that business problem is to design essentially a single-use language from the ground up, that is a red flag.
If you decide to create a new language as a goal unto itself or as a product, like Go, Swift, Kotlin, etc., then I’d say it’s a silly waste of time but not at all the same sort of red flag as responding to a specific business need by first building a whole language.
As for Google’s monorepo, I’m not getting drawn into more bike shedding about it. There absolutely exist polyrepo off-the-shelf solutions that would performantly handle their use case without the weird limitations of their custom tooling. Happy to agree to disagree with you about it.
Before working at Google I would have agreed with you. After working, not too sure. There's tradeoffs of course but I'm not quite as certain that it's the wrong tradeoff. Considering every similarly-sized company does the same thing it seems like it has real advantages at scale. Notably:
1. Facebook (lot of ex-Googlers, so fair if you want to criticize it as a culture thing)
2. Microsoft - they've driven mono repo on Git & have migrated Windows polyrepo flow to a mono repo one. They also gathered metrics & on the whole & users overall (30% dissatisfaction rate for completely altering the workflow over a weekend doesn't seem terrible).
3. Apple kind of. Everyone has their repo but each project submits into 1 giant repo for build purposes. Even still dependency problems crop up (team submits project A but another team forgets or is unable to submit project B that A relies on) that wouldn't in a single monolithic codebase
I work on polyrepo projects and I found the tooling to be quite lacking. Moreover it's been a giant PITA to communicate how to maintain this system to other people. A mono repo would be much easier.
If you include DSLs in what you say, then I disagree vehemently, at least as to DSLs.
For larger programming languages, it depends. Mainly it's a very long-term commitment. It requires having a strong commitment to funding a team to develop, maintain, and support it. Is that a good idea? I don't know in general. In specific cases it may be an absolutely brilliant idea, though I suspect in most actual cases it's a terrible idea mainly due either to inadequacy of the business case for the proprietary language, or inadequate funding. In order to build and keep proprietary a programming language for decades the value proposition in it has to be enormous -- that makes the business case unlikely, but not impossible.
Dropbox is widely considered a highly desirable place to work; it's odd that you describe it as an example of extreme dysfunction. Likewise, I think GS is generally viewed as the technically strongest IB.
I don’t necessarily disagree about Dropbox, and tellingly I think it means something that they realized Pyston was a bad direction and sunset it.
As for Goldman, hoo boy I have a bridge to sell you. Goldman tech is notoriously dysfunctional. Being the best among investment banks is hardly anything to be proud of whatsoever.
By all accounts, S-Lang and SecDB were massive successes and major instruments in Goldman’s competitive advantage in the industry.
On another note, what about Facebook and Reason ML, and Mozilla and Rust? Or C#, etc etc. There have been dozens of successful languages that originated from companies.
I'd love to see an example of the language, I kept reading (the story is great) but it felt like we got left hanging without any code snippets to put it all in, uh, context I guess.
I think it was this site where I remember Context from: https://www.mangaupdates.com/
Back then in my school years I remember not being able to find anything about this language. Mystery solved now. Thank you.
When I was a ten year old kid in 1973 my school did a tour. I remember looking at some programs running on green-screen terminals, and the guy doing the demo typed in commands using the keyword FOR so I assume it was all Fortran back then.
Numerous programming languages, including basic, algol, etc used for loops. Notably absent from the list of languages which used FOR as a looping construct was Fortran, which used 'do' for looping, and cobol, which used 'perform'. So whatever languages it was, it wasn't Fortran.
On a lot of systems the fortran compiler is often named fortran and usually aliased as for and fortran in the shell. I know as well on VMS, RSX, and RSTS you, at a minimum, just need to type the first 3 chars of a command for DCL to figure out what it is.
> “We probably over-engineered it, I guess,” Damens said.
Exactly. Also, creating a programming language for something as simple as a newspaper is pretty much a nightmare. Worrying about scalability when most of their content is static.
This is nearly 20 years ago. Were CGI and PHP really up for the tasks of serving up the type of dynamic pages at scale 2 decades ago? The "content" (articles) might be largely static but the pages are not, with dynamic ads, weather reports, showtimes etc. Not to mention getting non-technical people to adopt the language easily, transitioning from print publishing workflows. Do you not believe them when they say existing language and solutions weren't working?
I'm not really familiar with how things were ~20 years ago, but I still believe that creating a language to use internally just for this is overkill. Dynamic ads, weather reports ... etc. are not that difficult they could have probably done it in C (maybe a custom somewhat advanced template engine?) with much less effort.
I think you are underestimating how novel those things were 20 years ago. The NYT has been ahead-of-the-curve when it comes to the sophistication of their webpage for about that long. Perhaps their language helped.
It is pretty cool that they created their own language, but I was not sure if they really needed to do that. I guess it was worth it back then. Looking at their website in 2002[0] it is pretty impressive indeed.
[0] https://web.archive.org/web/20020209001700/https://www.nytim...
My guess is that it actually looked a lot like that, except the mod_perl app was actually a homegrown template engine using their homegrown template language which they then called "Context".
Bravo!
I do wish they'd throw the source code up on GitHub, as it would be an interesting historical artifact.