Hacker News new | comments | show | ask | jobs | submit login
The Inner Json Effect (thedailywtf.com)
291 points by dsego on July 29, 2016 | hide | past | web | favorite | 141 comments

Sometimes I think the industry's obsession with "code reuse" is what enables these architecture astronauts in the first place. If people accepted that most code is ephemeral and never going to be reused, then architecture astronauts have much less idealogical cover for their disastrous schemes.

Pretty much the only example of code reuse gone well is open-source libraries; and that's because being public forces the code to be documented and free of implicit dependencies. You pretty much never get that with janky internal frameworks and libraries, and they're usually "reusable" and yet replaced next project.

Right? I mean, special snowflake libraries that everyone can use do exist, don't get me wrong, but they're usually created by people whose sole job it was to create exactly that, a reusable API for <insert domain here>. There's very rarely an opportunity to repurpose old code in a new project if the goal of "API-izing" it wasn't in mind when it was written, and many times when people try it's a disaster.

If code could be re-used efficiently, we wouldn't need half as many programmers as we actually legitimately do.

This a thousand times.

The primary problems in that article are nothing to do with technology.

Consider these statements:

* "Tom is a genius", which implies the Tom solution is a trusted asset beyond improvement, doubt, or questioning. It also implies the solutions provided by Tom are golden unimproveable truths.

* The various statements from Scott suggest an indisputable faith in process and convention.

Clearly there are failures at multiple levels here. First of all Tom sounds like a whiny bitch. These personality types are inherently defensive and typically seek to reinforce an individual's position of self triumph in a small pond. Toxic.

Secondly, Scott has a lot of faith in process and conventions. Processes and conventions are the absolute enemy of creativity. I understand processes are necessary to establish a certain level of security, but they more typically exist to satisfy some OCD insanity where there is comfort in doing things in a particular way without consideration for why they are done in that way. Many developers cannot tell the different between security and superficial stupidity. Many technology abstractions enable that stupidity thereby convolute the differences between security and OCD stupid which only enables additional stupidity.

All of the prior mentioned failures are allowed to exist because the management doesn't want to be involved until there is a problem, such as Tom crying. This is called enabling.

Most important of all is that all technology should be questioned, doubted, and challenged. Obviously this sort of continuous improvement is utterly absent, because everybody has a competing agenda.

Process is the preventative of 'bikeshedding'. It's not so much about security as consistency, and it also prevents having the "why are we doing it this way" discussion every single time.

I think bikeshedding could exist irrespective of processes. Bikeshedding is the behavior of miss-prioritizing trivialities with greater imperative than core concerns because they are easier to think through. The solution for bikeshedding is prioritizing tasks in alignment to a written mission or plan. A documented plan described achievement of the plan where processes are the requirements to attain a certain level of conformance to a sub-goal. In other words the processes, of a well designed system, are the minimal number of barriers to get in the way just enough to prevent mission failure.

When engineers start acting like religious people it's time to run...

Please replace the term "religious people" with "people of color" and understand the harm you are doing in your discourse. It is blatant prejudice.

What I'm suggesting is that in engineering is better to test truths by applying logic and not just take them for granted by faith.

Please don't generalize my words and don't use them to activate your own prejudice..

> your own prejudice..

Which is?

By the way, I agree completely with the wording "not just take them for granted by faith." It is much more precise and to the point.

> > your own prejudice..

> Which is?

A desire to feel offended and outraged?

Weak tea, my friend. Even if it were true... that's not a prejudice. Please try again. I'm patient.

I think the correct term to use in this case, is "When engineers start acting religiously..."

As a person doesn't need a religion, to act religious; I can be religious about canned goods or martial arts, or perhaps, engineering.

I do agree that using the term "religious people", while the meaning is understood, makes a generalised assumption about a larger group of people who do not all adhere to the imagined stereotype, and it casts them in a negative light.

I suppose that the point you're making is that zer0gravity used the term "religious people", and that seems to be acceptable to most people, few object to it; whereas "people of color" or any other term that is deemed unacceptable to make that kind of negative assumption about, would not be okay. And that's a rather big double-standard.

Regardless though, the meaning that zer0gravity was trying to convey, was clear and understood. I think taking up offence at zer0gravity's comment and use of the term "religious people" just detracts from the topic being discussed; and I highly doubt any ground will be crossed on the point you're trying to make; at least on this page and topic.

True, though this story sounded more like one engineer convinced of his godhood and a bunch of clueless managers who ensure that the only way you can keep your job is to believe in the greatness of the solution and its author...

I read TDWTF as cautionary tales. I find this one to be reasonably (and scarily) believable. This 'Tom' guy probably looked at Subversion and decided, 'hey, history tracking is a GREAT thing, but I want to be able to see the history IN THE FILE ITSELF.' Thus was born JDSL.

The closing paragraph puts it perfectly. The design lends itself to production running latest trunk (because, obviously, it's assembled at runtime FROM SVN). The idea that a commit could instantly break production is horrifying.

Just another example of a developer looking for a problem to apply his solution to.

Even if I have never had this sort of Content Addressable Function "solution" I can still relate very much to this story. A story consisting of $NEW_PERSON entering some huge existing project, with $NULL_AMOUNT of documentation, where there is some established "hegemony" of some inaccessible $GENIUS, with $ZERO_AMOUNT of help and support, where bosses will likely to $YELL or otherwise treat you as a problem, where every move is very fragile and often leads to $CONFLICT... I surprises me how archetypical is this.

It is this moment when you search for the big red lever with sign: "EJECT!!!"

This is literally what I'm doing atm, on my first non-basic-bigfixing issue at a new company. Something that should have take two days has taken three, going on four weeks, I feel like a shitbag fraud for not polishing it off in two days (due to.pain time refactoring and seperating concerns). Noone can give me advice because <particular deity> left the company, and anyway they're rebuilding everything in <internal JS framework with no external deps>, but this critical feature needs to be done in place. Been in this situation before as well, thought I'd managed to escape...

This is hilarious but I don't take this as a satire.

In my modest experience most companies have something like this going on. Either the guy that everybody think is a genius because he wrote a custom (shitty) framework or just the horrible process and code base.

And remember, when you interview for companies that ask you questions about the latest javascript trends, how you would design a nice one page application, how internal Go works, etc. They most likely work with spaghetti jquery everywhere and a homemade php framework as bad as in this post.

You can tell they're a genius when only a genius could have concocted something so complicated and convoluted and still make it work. I seriously don't doubt that Tom had an off the chart IQ, but intelligence doesn't automatically lead to sensibleness.

I get to work on genius code all day long.

Reminds me of the feeling I got when a coworker professed that his favorite PHP feature was variable variables.

Two WTFs for the price of one!

The dumb programmer is the most efficient one.

That's why the industry standard in IT is to use the dumbest tool for the job. Replaceable cogs and all.

It can takes lots of smarts to make a simple solution.

See, I think the real mark of a genius is being able to make connections and see possibilities that are harder for normal people, and make something obvious and simple. You don't have to be a genius to code yourself into a corner. We've all done that before. Most of us weren't so full of ourselves that we couldn't eventually recognize our mistake though.

Lol, I'm genuinely trying to figure out if you interviewed at my company.

Meta-programming is a slippery slope. If you have a Clojure macro that pulls stuff from a DB, you're already halfway in Funky Town.

This reminds me of a similar horror story involving "the ultimate database design" https://www.simple-talk.com/opinion/opinion-pieces/bad-carma...

I have a theory that every programmer, at an early stage in their career, comes up with something approximating the 'metadatabase'. The only difference between any of us is how quickly we realise it's an appalling idea. There's a grey area: I've worked with databases on internal systems with key/value tables that just about make sense and were small enough to be manageable. It's when you see this kind of crap in production, third-party products that you want to run a mile.

Sounds like the "inner-platform effect", which this article's title seems to be a reference to. That's where a system is so soft-coded that its configuration ends up being a (poor) clone of the original programming language. That could be using a key/value table in a relational database, and using custom data and code to reimplement tables/columns/joins/etc. In the case of JDSL, it's using Javascript code and JSON metadata to implement Javascript code.

There's also the "second system effect", where the creator tries to avoid all of the bad things they encountered in the first system, and implement every spur-of-the-moment idea they ever had, all at once.



>which this article's title seems to be a reference to

The first mention of JDSL is a link to an article about it: http://thedailywtf.com/articles/The_Inner-Platform_Effect

It's extraordinary how often I have to point out that what someone is proposing is either called a "database" (to people who think it's a good idea to build a complicated file system complete with various metadata files) or a "programming language" (to the people who think it would be helpful to have some language which can logically encode business rules in a computationally understandable manner).

The interesting thing is that it goes both ways. Sometimes, when someone wants to store blobs of data in the database, you need to explain to them that it's probably better to use the file system in some manner. The common refrain is "no, we want to use the database", to which the answer often is "the file system is a database, just one that's already been optimized for what you are trying to do".

Similarly, sometimes it's useful to provide just enough primitives that a "non-programmer" can be trained on how to perform a very constrained set of operations on the data they deal with, and provide them with a simplistic language that funnels them into the types of operations they are most likely to want and away from (or disallowing) operations that are deemed problematic or complex.

The issue is that these options are usually not what you want to do. Not because they are wrong, but because they are much more likely to result in something worse, given the possible outcomes. Usually, the result is multiple of the following: too late, too slow, too complex, too inefficient, too hard to maintain, too hard to develop. That said, given a fairly lax timetable, smart people to design and develop the system with, and a clear vision of what needs to be accomplished, great things can be accomplished. But how often to do we get all that at once?

Ha, I did it in my early PHP days! At one stage I even tried using MySQL column comments to store metadata.

I think these kinds of things are great learning experiences, I'm just glad that abomination of mine was never released.

A lot of "enterprise" applications (particularly CRMs) do have everything in the database - including their own metadata and executable code (scripts and/or binaries).

Sometimes this works a lot better that you might expect and the platform works pretty well, sometimes it produces Cthulhu level nightmares.

Magento is a pretty good example, as I describe here (and oh my goodness the horrors in DCoder's followup comment): https://news.ycombinator.com/item?id=12152874

Some of these crazy designs make it into major applications. E.g. SAP stores all code in the database. http://www.se80.co.uk/saptables/r/repo/reposrc.htm

Salesforce works that way, too.

And Peoplesoft.

This is a common everyday story with NoSQL databases these days. One document/table format with 100's of fields like col1, col2, col3... so that they can stuff in whatever they want without having to think through schema design. If you try to confront it, they label you as outdated programmer from the 90's.

NoSQL sort of legitimized these kind of bad practices.

I've seen those sort of things all over the place in every NoSQL project I've ever worked on.

Wow — I've submitted it under https://news.ycombinator.com/item?id=12187520 because I think it merits its own discussion.

They describe something akin to Wordpress' custom fields

That's because custom fields in WordPress, as well as in Liferay and in a bunch of other products, are implemented as EAV, Entity–attribute–value model, which is an abomination.

It is a relational "design pattern" used to implement a database inside a database, instead of using the database itself as it was designed to be used.

This has all sorts of fun (not) and useful (neither) effects, including: worsened performance; inability to ensure data integrity through constraints; bloated / unreadable queries; hard to find bugs; needlessly complicated migrations; and so on.

So, EAV and even "inner platforms" are not always necessarily bad, so long as you know that's what you're doing.

For example, Firefox extensions could be easily considered an inner platform but they have the benefit of providing a platform which is safer and more extensible than the underlying OS.

Likewise, EAV provides a poor functional replacement for an actual database. But they also prove a lower curve for implementing very simple extensions and limit the capacity for trivial mistakes. Speaking from experience, it's the WordPress sites where plugins have created a whole bunch of extra poorly-built tables that worry me most.

The alternative to EAV for custom fields is allowing users to modify the database schema. I'd rather stick with EAV.

What’s wrong with just namespacing tables per module/app/whatever?

`wordpress_%app_table1` etc. if you want to do it ad-hoc.

PostgresSQL support namespacing out of the box[1][2].

[1] https://www.postgresql.org/docs/9.3/static/ddl-schemas.html

[2] http://jerodsanto.net/2011/07/building-multi-tenant-rails-ap...

Because giving a webapp sufficient permissions to CREATE, ALTER, and DROP tables is risky.

It gets harder to make assumptions about what tables are there when you want to migrate things. I also think it's a hassle if you want to do something like have multiple tenants in a single DB (not impossible, I suppose).

Having dealt with both, I'm split. I think, in the end, I'd rather model the DDLs and have everything point to separate database on the same server or a new namespace in the existing database. You give enough rope for the customer to hang themselves, but sometimes their requirements are to build a gallows.

One does not imply the other. Whether the schema itself is modified or just the EAV definition does not mean that everybody has the permission to do it. EAV is just the way it is stored/described.

OK... you are nevertheless talking about some end user having the power to modify the database schema, which IMO is a bigger headache.

I also meant that either way, I would not grant a end user the possibility to modify the data structure, be it through the modification of the schema, or through the addition of an EAV definition.

Granted one is more dangerous than the other, but to me it's no reason to give them access. It's still a headache.

If only the developers have access to modify the structure, the way it is done is not important (it can be done through a webform that alters a database, though the addition of a json definition, through coding, etc.) After all, you can modify the schema through PHPMyAdmin. It's some kind of admin interface. You just don't grant your users the right to do it. Just as you don't grant them the right to add custom fields, unless they know what they are doing, which is seldom the case :)

I think that EAV gives a false sense of security, in that it does not "break" much if done by a non-knowledgable user. But still, if it is done badly, good luck reconciling the data.

"Reconciling" the data isn't my job; the user can use the custom fields for whatever they want.

It's certainly true that life is easier if you never implement certain kinds of features but at that rate why make a program at all?

> "Reconciling" the data isn't my job; the user can use the custom fields for whatever they want.

I disagree strongly here and see this attitude a lot. It's easy to say "not my problem, you fucked it up, you fix it" but that is a very customer-hostile attitude.

Instead we should be giving users functionality they want in a way that prevents them from screwing things up through a combination of feedback and constraints. Sure, if a user types in blatantly incorrect data there's not much you can do. But you can do a lot more than say "here's a narrow ledge and a pit of snakes, if you fall in it's not my fault". This just results in unhappy customers who don't care whose fault it is, all they know is that your software ate their data.

> I disagree strongly here and see this attitude a lot. It's easy to say "not my problem, you fucked it up, you fix it" but that is a very customer-hostile attitude.

Very in agreement here. Almost without fail, if they knew enough how to 'fix' it, they wouldn't have fucked it up in the first place.

Much of this attitude depends on org structure. Are you talking about paying end-customers? Or joe in accounting 3 doors down who consistently never follows any directions you give him re: data?

I mean, I realize this discussion is abstract, but I can't think of a lot of ways I could possibly enforce normalization on data when I don't even control what the fields are.

Sometimes, the thing you need to model in a database behind an application you sell to a customer turns out to be... a sort of limited database with a schema the customer can control and design. When that happens, EAV is one of the common attractors in design space that everyone gravitates towards.

Of course, it's worth remembering that real databases do not typically use an EAV storage pattern for their data, so maybe it's not the only way to model a custom data store within a relational system.

It's also an option when the properties of objects going in the table are going to be wildly different or are not known at write time.

This is a candidate for something like a schemaless datastore or a PostgreSQL JSONB column. Depending on your choice of datastore you can still at least get some indexing and schema help if you choose. With EAV you generally throw all of that out.

A whole separate datastore sounds like overkill in a lot of cases.

This is stupid. Why would they write JDSL if they could just have used https://github.com/josh/cafe-js which is clearly superior because it uses Git instead of SVN? Typical case of NIH.


Actually, I've read about the content-addressable-function-idea before, but I can't remember where that was…

Joe Armstrong, one of the creators of erlang usually talks about this.


Maybe from Joe Armstrong?


Why? NIH. Or they hadn't heard of it.

Run for the hills any time anyone says, "It's our in-house developed DSL!". It's a virtual guarantee that whatever was cooked up is some sort of insanity like JDSL.

I guess also run for the hills anytime you find a place using any non-trivial amounts of Ruby, Python, TCL, Perl, Lisp, Scala, or Haskell right (all languages where embedded DSLs are rampant)?

I've worked at places that had great DSLs (both embedded and freestanding) and awful crocks that were just the right mixture of infinite funding and no accountability. The vast majority were trying to make something like database access a little nicer for their domain. Someone who doesn't want to litter their resume with the term "DSL" might even call the majority "libraries". I've certainly seen a couple that were on the level of JDSL ("Why don't we WYSIWYG build a CMS by generating XML which can expand out into PHP"), but I've also seen "writing assembler sucks! there is no C compiler for this micro! it would be way easier to write scheme ... hey look! we have a DSL for writing ECUs". In my anecdotal experience, DSLs have been the only way to stay sane in certain domains.

So how does one avoid this trap?

I've occasionally thought of using a DSL for some business logic, presumably the best way is to at least use an existing language (like Tcl) and provide business-specific verbs in it?

Don't build a DSL for a production system, period. Production quality DSLs are for academics and/or staff engineers that have enough time and resources to build and maintain them.

I don't know about you, but I have a constant backlog of features with tons of interesting problems to solve. There are places where I could build a DSL, but I am not going to because I would inevitably build something hacky and brittle that is impossible to understand and maintain (like JDSL).

I can't build new features, and increase the value of the system if my DSL is constantly breaking the system.

So don't do it. There is plenty of functionality on the .NET and Ruby platforms for me to model the concepts and objects in the domain without the need for a DSL.

Oh, this reminds me.

I once worked for a company that used RDF for all configuration and scripting for one of its main applications.

Instead of using one of the standard methods of encoding RDF (turtle, XML, etc.) our main architect wrote his own. Specifically, he wrote an encoding based on Google Sheets. The triples were the row header, the column header, and the cell value. There were two special tables; one containing namespace aliases and one containing an index of all the tables in the spreadsheet.

To deploy, you published a Google Sheet to the web, copied the sheet key, and pasted it into a variable in the Java code's main() function (and we had a few different main classes for a few different deployment types).

Later on, he added support for encoding RDF in XLSX spreadsheets so we don't need an Internet connection to launch the app.

Let's just say that architect and I didn't get along (and he didn't get along with a good chunk of the devs either; multiple people I worked very well with quit during my time at that company largely because they couldn't get along with him).

If you want to see it, the code is open-source (some of their content is proprietary, but there's an open-source RDF store in the public repos that has the proprietary stuff removed and replaced with demo content). It forms part of the appdapter project on Assembla [0] (but only part of it; there's other random stuff in appdapter too), and there are some sample main() functions in the friendularity project on Asssembla [1] (friendularity was our repository of open-source main classes; they mostly just import stuff from other open-source projects and wrap it in a main()). Honestly, though, I didn't touch the appdapter part of the codebase when I worked there, and it's been a couple of years since I've worked there, so I don't think I can help much with running it. I think org.friendularity.bundle.demo.ccmio (or maybe org.friendularity.bundle.launcher.oglweb) is the relevant main project, but I don't remember too well, and I think some things have been refactored since we left.

[0] https://app.assembla.com/spaces/appdapter/subversion/source

[1] https://app.assembla.com/spaces/friendularity/subversion/sou...

While that is probably a bad idea... it does give me a potentially good idea... using google sheets for internationalization string management... so you don't have to build a custom UI.

two-column sheets.. first sheet is "default", sheet name == lang_location and overrides... column 1, except first row, is the key, column 2 is the localized value, column 3+ unused, but can provide more context information.

It really depends. In this example they used JSON to basically create a new programming language. I know a couple of projects where a DSL is used for declarative parts of the project. For example in one projects they took one or more inputs from sensor values, applied a calculation and created a new virtual sensor output. For such highly specialized tasks a DSL can be the better option.

You could try sketching out on paper or a whiteboard what control structures you would need to pull off the DSL, and then exposing those control structures as utility classes instead of a DSL.

For example, if you are building up a queue of commands to be run in sequence with a bit of preprocessing, you might create a class that exposes: `addStep`, `preprocess`, and `run`.

At least for me, the impulse to create a DSL was always a secret desire to hide implementation details from readers so that they can see the intent of the code more clearly. Really, those implementation details are critical for maintainers to understand. Otherwise a DSL's magic is likely to hurt a maintainer when their expectations are off.

Well, I am very, very cautious when I hear "DSL" and don't hear "Lisp" in the same sentence. Most languages are not powerful enough to create full DSLs that are maintainable and actually improve something.

I saw similar levels of ridiculousness with XML during my (brief) time working with Enterprise Java many years ago, although it didn't go as far as the use of Subversion... but I did experience a lot of the "you just don't understand how beautiful it is" sentiment from the architecture-astronaut creator(s) of these behemoths. Now that JSON is the "new modern XML", I'm not surprised at all that things like this have come into existence.

Before I started my professional career I used to believe that the stories on the daily wtf were fiction.

The difference between truth and fiction is that fiction has to make sense.

What a silly system! They should have used BobX instead.


hmm. actually, the looping and conditionals in bobx look a bit like jstl.

The scary part here is that I am not actually sure whether this is fiction or not.

> Founded in 2004 by Alex Papadimoulis, The Daily WTF is your how-not-to guide for developing software. We recount tales of disastrous development, from project management gone spectacularly bad to inexplicable coding choices. Remy Porter is the editor-in-chief and needs to read your story or see your bad code.

And below:

> Submit Your WTF

Stories are not supposed to be fiction. Granted, editors can't check how true what you submit is (and often, you only have one side of the story), but there seems to be so much available material out there that we don't need fictional entries.

At one point, I believe they released a Visual Studio extension that would let you highlight a particularly WTF-worthy chunk of code and post it up easily. I was tempted with some of the code that I inherited after it had gone through a couple iterations of cut-rate outsourced freelancers...

Same here. The other stories do seem real though.

A friend of mine once worked at a software company as recently as 2014 (it went bankrupt meanwhile) where the team of around 10 developers would work without any versioning system at all! They used folders with dates instead, and every once in a while someone would go over all the folders with the respective developer and merge the code manually.

My friend tried to introduce SVN, but the developers didn't want to invest time into learning it.

Anyway, given the stories I know first-hand, I totally believe it's a true story!

In Russia, we had people running in between desks with stacks of floppy disks and flash drives up until mid 200\

Both. In the interest of anonymizing submissions and just ensuring things read well, they basically take the submissions and write a new framing story for the core WTF from the submission. In this case the names, dialogue, specific details given about the system are probably all fabricated, but the idea of assembling classes out of revisions in a source control system and not supporting comments would be from the original submission.

The general assumption when reading the Daily WTF has always been that the names have been changed to protect the guilty, but the facts and mistakes are very real.

This story could be fiction, but I don't think so. For starters, it takes some serious imagination to come up with a system so broken. And I have seen a system, written in a javascript dialect, execute by an in-house javascript engine written in Delphi, that you had to use the in-house IDE because it was the only way one could find the desired functions. You see, the 'module' system worked by referencing database IDs, from a database table that contained the javascript "file" you wanted to import...

Why is there always a "genius" behind stuff like this?

It's usually a combination of a competent but misguided senior engineer and a fleet of buisiness dudes who aren't qualified to assess whether the engineers work is actually any good.

He'll seem to weave magic initially, and the company/project will launch and make money for the people involved. It's only later, when other engineers show up, that the extent of the horror becomes evident, by which point the senior engineer has already been enshrined as the "genius" behind the companies success.

Only the new, junior engineers can see the Emperors clothes for what they are.

"It's usually a combination of a competent but misguided senior engineer"

Quibble: Anyone who comes up with and deploys that sort of system is not competent. They may be "senior", chronologically anyhow, but they are not "competent". Compenent senior engineers know about the inner-platform effect.

Similarly, to me making avoid a second reply, it's not just lack of communication that makes this system bad. The inner-platform antipattern is objectively bad, no matter how it is communicated.

We could change "competent" to "capable", which may better capture the spirit of the argument :)

The point is, this engineer, for all they're many, glaring faults, can get stuff done. Impressive stuff, even. And that's all the business dudes see in this scenario; Dave plugging away at a computer and cheques from customers showing up in the mail every month.

Fair enough. Such systems do require an impressive amount of capable labor to make work at all. Labor to staggeringly negative effect, but yes, capable labor. One could do wonders with such an engineer if you can mentor them... and if they'll take a mentoring.

> ...if you can mentor them... and if they'll take a mentoring.

I suspect these "genius" programmers are damaged beyond all hope :)

The kind of people that come up with this stuff usually are in fact really good at visualising an abstract problem and implementing it in code. What they often lack is the capacity to translate what it is their frameworks do to language others can understand.

From what I have witnessed myself, these guys are actually quite intelligent in an abstract almost mathematical way — that is, they are not dumb and can come across as geniuses — but their mediocre communication skills prevent colleagues from actually understanding their work, and more importantly, prevent themselves from ever realizing that what they are doing is way too experimental and (not rarely) wildly inappropriate for the task at hand.

You will hear these visionaries extol the virtues of their revolutionary framework time and time again, and that it will be the foundation of anything the company builds for years to come, but any criticism or even critical questions are dismissed with an air of 'you-just-don't-get-it-yet'. In the end either you leave for greener pastures, or the idiot savant cracks under the pressure of changing business needs that cannot be met by the big framework that only he/she (usually he) really 'gets' and quits.

Worked at a place where there was a 'genius' (knew him before I worked there - not a dumb guy, really) who had written some really .. bizarre AOP-style stuff in PHP4. Not AOP extensions, just... a whole lot of stuff trying to approximate AOP, but no one knew the term AOP or what it was good for. (aside: it was interesting intellectually to walk through, but was not anything that should have been used for production code. also, he didn't label himself a genius, but did have a somewhat overinflated view of how good his code was).

Anyway, he was rarely in meetings, but a couple of folks who glommed on to him early on would extol the virtues of his 'framework'.

"You know how some people are just so smart that they can't even explain their work, cause it's so beyond everyone's ability to understand it? That's who he is, and that's what we have!" This was said to me at least once, and repeated variations of it in a couple other meetings. And it was said with pride/admiration/awe, about how awesome it was.

I countered with "if you can't understand it, and he can't explain it, he's the only one who can do anything with it, and you can't really verify if it's doing what you want/need, so how is this remotely 'good' by any metric you use for any other software in this department?"

"You just don't understand how amazing this is, and what we're able to do with it" was the response.

I distanced myself from it, but was then tasked with having to build on top of the 'web framework'. And discovered it had no support for web sessions. Or security. Or database connectivity. It was literally just mapping an incoming web request to a file, while taking a logging token from a cookie and sending it to a third server for logging activity. Given the AOP-esque nature... I had a bit of a hard time trying to figure out just where I'd actually add any sort of session management, and when I asked for someone (original author maybe?) to add this in because I couldn't figure out how to do it, but needed it, I was 'aggressive' and 'not a team player'. Ugh...

EDIT: when I read dailywtf stuff, I sometimes doubt the dramatizations, then I remember back to my days on this project, and ... have to recognize there's probably very little embellishment.

When you don't understand something it's impossible to tell whether the creator was a genius or an imbecile. This is probably why hiring a sole programmer in a start-up is such a nightmare. And I speak as one who is that sole programmer.

It's also something that can accidentally happen as a cargo cult when you don't quite realize you hired a single very senior programmer and lots of very junior programmers without much of a middle ground or experience spectrum. That's a relatively easy trap for a company to fall into, especially early on, because it can be cheaper and easier to hire more junior programmers faster (ie, grads right out of college) rather than try to structure around a less steep gradient.

It fulfills complementary anti-patterns both with the developer and the management.

The developer that built it actually is smart, since you have to be to make something so bizarre and complex actually function at all. They are also full of ego, which gives them the need to prove to everyone how smart they are by building something very complex, whether the problem calls for it or not. What they lack is the wisdom to refrain from inappropriately trying to prove how smart they are and building something that is easy to troubleshoot and maintain.

They are able to stay where they are because management has their own egoic need to believe that they are employing and in charge of geniuses to demonstrate to their own peers how awesome they are. Instead of actually being awesome by coming up with a good business plan, executing well on it, and recognizing that your software problem is probably perfectly ordinary, and even if it isn't, you only hurt things by hiring ego-driven builders who want to build overly complex things instead of solving the problem at hand as simply as possible.

Replace "genius" with "senior engineer" and you might understand how this sort of things happen a little bit better.

Any idiot can create a mess that doesn't work. It takes a genius to create a mess that works.

Such an aura can be cultered.

This was a fairly entertaining read: https://news.ycombinator.com/item?id=11631118

JDSL sounds like a capital idea, but obviously Subversion is much too slow, and Javascript/JSON is not enterprise-compatible. Someone should re-implement it using EJB's, XML and an Oracle backend to store the functions in.

P.S: Dear Eris, please let me never turn into Tom!

I honestly fear the day my boss stumbles over this idea.

This is like the programming version of sarcastaball. We'll make up the stupidest possible idea, and then our bosses will find out and think they're brilliant.

"What the fuck" was the first thing I thought after finishing this article. Didn't even realize it was from "thedailywtf.com", but it sure fits there. Good article, hope it's fake.

I sincerely doubt it was fake.

It's one of those stories that might be fake but still ring true. I know I've been there, on both sides (to some degree), so I can relate.

This isn't real right? It's just a made up story to scare, right??

So many things can go wrong. But how do you assess, where to draw a line? What could be used as a guiding light?

Currently I try to use the following scenario as orientation: If a completely new developer would walk in and wanted to install or build or contribute to a current system/project, how much I would have to explain to this person? How many, "yes, true, but only if" style statements would I need to make?

The optimum here is, that a developer will contribute, without me needing to explain or motivate anything.

Do you use a certain approach? How would you describe it?

Is this really that much different from any other industry though?

The system in the story is clearly insane, but every organization (not just software companies) has it's own operational awkwardness, that might look weird for new employees.

Imagine that Google has all their code in a single repository. How "insane" is that?

My point is I don't see why is it a hard requirement for a new developer to be able to commit code on day 1. Of course if you have hundreds of developers, it makes sense to optimize the process, but personally I would prefer a new college to walk me through things (apprenticeship style) rather than throw me at a repo and go-figure-it-out.

If a competent senior developer can't come in and make a commit (even a simple one) on their first day, your system is too complicated and should be rethought.

Other things that I find useful to consider:

How big of an IDE and how many extensions/plugins to that IDE do I need to require? Recommend?

Can you take a fresh code clone and hit the big green play button in your IDE and go straight into debugging? How long does it take to build that first time from a fresh start? Subsequent "average" attempts?

This story is like a parable that condenses all my work experiences so far. The problem is that management is easily terrified and impressed by anyone who can flummox them with tech buzzwords and knows just enough programming to be dangerous. They get in early, create a horrific mess that takes other people at least a year to untangle enough to be able to maintain. The company talks forever about rewriting it - but everyone's too busy putting out fires in the spaghetti to work on the newer better architecture. Really, most of the crap these 'geniuses' write could be quickly replaced with something off the open source shelf. But the customers are too entrenched in the proprietary spaghetti, so they can't, etc. And every new 6 figure tech guy they hire just tries to create his own cozy nest of over complex code in a sad attempt to create job security. Its all a scam.

Think about all those dependency management systems (npm, Maven, NuGet). Look at a project with huge number of dependencies, picking on piece of functionality from one library, something else from another. All tied together with some depedencies file and maybe with some additional dependency injection magic. Not backed up by version control, but instead the package repository.

Not that many projects add a real DSL, but I think you can end up pretty close also with just objects and language features.

And where do you then run this system? Of course in Docker container, probably picking up Linux components with apt-get from left and right.

The bad things in the article were the inner-platform effect and that the inner-platform was especially terrible (no comment support, tons of boilerplate, abuse of the project's own source control, lack of dev/production split). I really don't think the concept of versioned dependencies was one of the bad parts.

Sure dependencies can become absurd (like the NPM left-pad story). But should we on the other hand always code everything from scratch? Where does it end?

Did anyone else notice the part about checking into the trunk?

I agree that the framework and people were definitely WTF.

However, I would at least have talked to someone before committing to the trunk, even if I thought I was just adding comments.

Enjoyable read, and I think for any experienced developer there's at least a _bit_ to relate to in there.

But seriously, this is why you have an integ/staging environment :)

And obviously, better developers.

I've come to believe that 'clever' solutions are almost always a bad idea. You should err on the side of best practices. Defying expected norms means that other people aren't going to be able understand your code or maintain it. This is a pretty spectacularly bad example of that. I've actually worked on sites that did similar stuff to this - having multiple versions of the code checked out on the same server with references to files in specific commits.

Jake should have left two weeks earlier. That JDSL was a disaster waiting to happen. Jake was lucky it happened quickly and he didn't waste his career on that system.

I think the more interesting question is: when can something crazy like this work? I started writing a comment about that here, but it got long enough I turned it into its own blog post. Here is my submission:


It may seem appealing to go down the meta meta programming rabbit hole, but let's not forget, that any intelligent fool can make things bigger and more complicated, it takes a stroke of genius to move in the OPPOSITE direction. In engineering things need to be functionally beautiful first.

This reminds me how I always seem to fall into the "find and program to the pattern" anti-pattern.

I am Tom.

Satire aside, couldn't they just revert the commits? That is the whole point of the system. ONLY if the system didn't work would Tom be angry, but IF the system didn't work then the rest of the story is contradictory.

reverting the commits won't bring back the deleted database

which brings up the other problem of why aren't there backups

What's the name of the company? Their customers deserve to know that story.

Reading these kinds of stories, I can't help but reflect on how many days of my life spent dealing with crap similar to this I'm never getting back. Hair stands on the back of my neck, literally.

Great example of abstract business logic. Totally believe it's true.

This is a work of fiction right? I feel so stupid for having to ask.

I wish these kinds of things didn't happen in the real world, but they do. Management wants it fast and cheap, wether it's good, they have no clue, they probably don't trust their engineers who speak too much gobbledigook. So they hire the cheapest sob they can find, if he screws up, the next scrub can take his place, and that one might in turn get fired if the first one made too big of a mess to refactor ("What do you mean, you fixed this bug and now there are TEN MORE? We're already over budget and you were supposed to have finished last week. And don't give me your technobabble excuses!"). Happens more often in places ran by wannabe get-rich-quick type entrepreneurs.

The first rule is: in tech, all shitthatdidnthappen.txt stories actually did happen. I once got a CD cup holder when I was working dialup tech support.

My WTF entry: I worked at a company whose main website was written in J2EE. This was here in the Bay Area, right in downtown SF.

When I joined, I soon learned about "Stephen". Stephen had already left the company but he was still very much a presence. Most of his comments contained expletives, even for something simple like changing a log level: "tell Tomcat to shut the fuck up!"

Throughout the wiki, there were references to technical "discussions" around coding standards that often had statements like "Stephen doesn't like this!", which was short for "I don't want to get yelled at so we'll do what Stephen says". There were a lot of things Stephen didn't like. He didn't like Java. He didn't like private methods. He seemed on the fence about getters and setters.

But he loved Python. Boy howdy did he love Python.

Stephen loved Python so much that much of the Java codebase was uneditable. Yes, uneditable. You see, in order to write the Java code that'd be actually, you know, run in production, you didn't simply open IDEA and write Java. You wrote Python code that emitted Java. Hibernate configs, classes, the lot. Development time wasn't the fastest.

Something else Stephen hated was MySQL. He hated it a lot. Postgres was way better in his opinion. Stephen hated MySQL so much that he stripped it out of all the Puppet files one day because he didn't think the company should be running it. Nevermind that he was editing the production files for another team, so the next time Puppet ran, the predictable outcome occurred. I believe that was one of the few times his actions had any direct consequences for him.

Beyond Stephen and his hatred of all things non-Python and MySQL, the deployment process was a true thing of horrifying beauty.

You see, one didn't simply just drop a WAR file into Tomcat and restart the server process. That'd be much too simple. When you deployed, you deployed everything. And when I say everything, I mean everything.

Apache? Compiled from source and re-installed. Java? Compiled from source and re-installed. Name a dependency other than Postgres and that ish was compiled from source and re-installed. Deploying any code change was a minimum 45-minute process. Heaven help you if a bug was found.

(Oh yeah, and of course Stephen's domain user couldn't be removed from Perforce or any of the build systems because all the scripts ran as his user. Nobody knew what might happen if his user was removed. To this day the company still exists after an acquisition and I'd bet good money that Stephen's user is still out there consuming CPU cycles.)

My takeaway from this experience is that this kind of environment is pretty common, and it's because of some basic psychology in people. Engineers don't want to look stupid and their social skills are often not developed. So when you meet someone with a strong personality who uses shouting to establish technical superiority, they can get a long way.

I would kick Tom in the jingle berries... so hard ... that every time he ever came up with a crazy idea like this he would ache.

Is there any easy way to recreate that blocked visualization of the JSON?

Didn't they have code reviews ?

Is that true story?

Really hoping this is a joke and fiction. At least I chuckled.

_And_ that short employment stint is a black-mark on Jake's resume. What an industry we have.

>> _And_ that short employment stint is a black-mark on Jake's resume. What an industry we have.

But now he can just point people to that WTF piece and say "that's what that was". Anyone who still considers it a black mark after that isn't worth working for.

I have a 5-month stint at a huge company on my resume - one that a lot of people would really like to work for. It comes up sometimes in interviews, but I've got a good story behind it and usually come off as capable and ambitious after explaining it.

If something raises a question, ask it. Don't just assume the worst of a candidate.

I agree. If a company isn't willing to give me the benefit of the doubt by asking, then maybe it's another place I'm not interested in working.

Why would Jake even list it on his resume? He was there two weeks, it's not worth it.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact