
The Inner Json Effect - dsego
http://thedailywtf.com/articles/the-inner-json-effect
======
overgard
Sometimes I think the industry's obsession with "code reuse" is what enables
these architecture astronauts in the first place. If people accepted that most
code is ephemeral and never going to be reused, then architecture astronauts
have much less idealogical cover for their disastrous schemes.

Pretty much the only example of code reuse gone well is open-source libraries;
and that's because being public forces the code to be documented and free of
implicit dependencies. You pretty much never get that with janky internal
frameworks and libraries, and they're usually "reusable" and yet replaced next
project.

~~~
nameless912
Right? I mean, special snowflake libraries that everyone can use _do_ exist,
don't get me wrong, but they're usually created by people whose sole job it
was to create exactly that, a reusable API for <insert domain here>. There's
very rarely an opportunity to repurpose old code in a new project if the goal
of "API-izing" it wasn't in mind when it was written, and many times when
people try it's a disaster.

If code could be re-used efficiently, we wouldn't need half as many
programmers as we actually legitimately do.

------
anonymousguy
The primary problems in that article are nothing to do with technology.

Consider these statements:

* "Tom is a genius", which implies the Tom solution is a trusted asset beyond improvement, doubt, or questioning. It also implies the solutions provided by Tom are golden unimproveable truths.

* The various statements from Scott suggest an indisputable faith in process and convention.

Clearly there are failures at multiple levels here. First of all Tom sounds
like a whiny bitch. These personality types are inherently defensive and
typically seek to reinforce an individual's position of self triumph in a
small pond. Toxic.

Secondly, Scott has a lot of faith in process and conventions. Processes and
conventions are the absolute enemy of creativity. I understand processes are
necessary to establish a certain level of security, but they more typically
exist to satisfy some OCD insanity where there is comfort in doing things in a
particular way without consideration for why they are done in that way. Many
developers cannot tell the different between security and superficial
stupidity. Many technology abstractions enable that stupidity thereby
convolute the differences between security and OCD stupid which only enables
additional stupidity.

All of the prior mentioned failures are allowed to exist because the
management doesn't want to be involved until there is a problem, such as Tom
crying. This is called enabling.

Most important of all is that all technology should be questioned, doubted,
and challenged. Obviously this sort of continuous improvement is utterly
absent, because everybody has a competing agenda.

~~~
zer0gravity
When engineers start acting like religious people it's time to run...

~~~
tomcam
Please replace the term "religious people" with "people of color" and
understand the harm you are doing in your discourse. It is blatant prejudice.

~~~
zer0gravity
What I'm suggesting is that in engineering is better to test truths by
applying logic and not just take them for granted by faith.

Please don't generalize my words and don't use them to activate your own
prejudice..

~~~
tomcam
> your own prejudice..

Which is?

By the way, I agree completely with the wording "not just take them for
granted by faith." It is much more precise and to the point.

~~~
stevenwiles
> > your own prejudice..

> Which is?

A desire to feel offended and outraged?

~~~
tomcam
Weak tea, my friend. Even if it were true... that's not a prejudice. Please
try again. I'm patient.

~~~
rebel28
I think the correct term to use in this case, is "When engineers start acting
religiously..."

As a person doesn't need a religion, to act religious; I can be religious
about canned goods or martial arts, or perhaps, engineering.

I do agree that using the term "religious people", while the meaning is
understood, makes a generalised assumption about a larger group of people who
do not all adhere to the imagined stereotype, and it casts them in a negative
light.

I suppose that the point you're making is that zer0gravity used the term
"religious people", and that seems to be acceptable to most people, few object
to it; whereas "people of color" or any other term that is deemed unacceptable
to make that kind of negative assumption about, would not be okay. And that's
a rather big double-standard.

Regardless though, the meaning that zer0gravity was trying to convey, was
clear and understood. I think taking up offence at zer0gravity's comment and
use of the term "religious people" just detracts from the topic being
discussed; and I highly doubt any ground will be crossed on the point you're
trying to make; at least on this page and topic.

------
gargravarr
I read TDWTF as cautionary tales. I find this one to be reasonably (and
scarily) believable. This 'Tom' guy probably looked at Subversion and decided,
'hey, history tracking is a GREAT thing, but I want to be able to see the
history IN THE FILE ITSELF.' Thus was born JDSL.

The closing paragraph puts it perfectly. The design lends itself to production
running latest trunk (because, obviously, it's assembled at runtime FROM SVN).
The idea that a commit could instantly break production is horrifying.

Just another example of a developer looking for a problem to apply his
solution to.

------
haddr
Even if I have never had this sort of Content Addressable Function "solution"
I can still relate very much to this story. A story consisting of $NEW_PERSON
entering some huge existing project, with $NULL_AMOUNT of documentation, where
there is some established "hegemony" of some inaccessible $GENIUS, with
$ZERO_AMOUNT of help and support, where bosses will likely to $YELL or
otherwise treat you as a problem, where every move is very fragile and often
leads to $CONFLICT... I surprises me how archetypical is this.

It is this moment when you search for the big red lever with sign: "EJECT!!!"

~~~
RobertKerans
This is literally what I'm doing atm, on my first non-basic-bigfixing issue at
a new company. Something that should have take two days has taken three, going
on four weeks, I feel like a shitbag fraud for not polishing it off in two
days (due to.pain time refactoring and seperating concerns). Noone can give me
advice because <particular deity> left the company, and anyway they're
rebuilding everything in <internal JS framework with no external deps>, but
this critical feature needs to be done in place. Been in this situation before
as well, thought I'd managed to escape...

------
grosbisou
This is hilarious but I don't take this as a satire.

In my modest experience most companies have something like this going on.
Either the guy that everybody think is a genius because he wrote a custom
(shitty) framework or just the horrible process and code base.

And remember, when you interview for companies that ask you questions about
the latest javascript trends, how you would design a nice one page
application, how internal Go works, etc. They most likely work with spaghetti
jquery everywhere and a homemade php framework as bad as in this post.

~~~
mark-r
You can tell they're a genius when only a genius could have concocted
something so complicated and convoluted and still make it work. I seriously
don't doubt that Tom had an off the chart IQ, but intelligence doesn't
automatically lead to sensibleness.

I get to work on genius code all day long.

~~~
nameless912
The dumb programmer is the most efficient one.

~~~
TeMPOraL
That's why the industry standard in IT is to use the dumbest tool for the job.
Replaceable cogs and all.

------
shangxiao
This reminds me of a similar horror story involving "the ultimate database
design" [https://www.simple-talk.com/opinion/opinion-pieces/bad-
carma...](https://www.simple-talk.com/opinion/opinion-pieces/bad-carma/)

~~~
oneeyedpigeon
I have a theory that every programmer, at an early stage in their career,
comes up with something approximating the 'metadatabase'. The only difference
between any of us is how quickly we realise it's an appalling idea. There's a
grey area: I've worked with databases on internal systems with key/value
tables that _just about_ make sense and were small enough to be manageable.
It's when you see this kind of crap in production, third-party products that
you want to run a mile.

~~~
chriswarbo
Sounds like the "inner-platform effect", which this article's title seems to
be a reference to. That's where a system is so soft-coded that its
configuration ends up being a (poor) clone of the original programming
language. That could be using a key/value table in a relational database, and
using custom data and code to reimplement tables/columns/joins/etc. In the
case of JDSL, it's using Javascript code and JSON metadata to implement
Javascript code.

There's also the "second system effect", where the creator tries to avoid all
of the bad things they encountered in the first system, and implement every
spur-of-the-moment idea they ever had, all at once.

[https://en.wikipedia.org/wiki/Inner-
platform_effect](https://en.wikipedia.org/wiki/Inner-platform_effect)

[https://en.wikipedia.org/wiki/Second-
system_effect](https://en.wikipedia.org/wiki/Second-system_effect)

~~~
Kristine1975
_> which this article's title seems to be a reference to_

The first mention of JDSL is a link to an article about it:
[http://thedailywtf.com/articles/The_Inner-
Platform_Effect](http://thedailywtf.com/articles/The_Inner-Platform_Effect)

------
svckr
This is stupid. Why would they write JDSL if they could just have used
[https://github.com/josh/cafe-js](https://github.com/josh/cafe-js) which is
clearly superior because it uses Git instead of SVN? Typical case of NIH.

/s

Actually, I've read about the content-addressable-function-idea before, but I
can't remember where that was…

~~~
marianoguerra
Joe Armstrong, one of the creators of erlang usually talks about this.

[http://erlang.org/pipermail/erlang-
questions/2011-May/058768...](http://erlang.org/pipermail/erlang-
questions/2011-May/058768.html)

------
WDCDev
Run for the hills any time anyone says, "It's our in-house developed DSL!".
It's a virtual guarantee that whatever was cooked up is some sort of insanity
like JDSL.

~~~
pjc50
So how does one avoid this trap?

I've occasionally thought of using a DSL for some business logic, presumably
the best way is to at least use an existing language (like Tcl) and provide
business-specific verbs in it?

~~~
WDCDev
Don't build a DSL for a production system, period. Production quality DSLs are
for academics and/or staff engineers that have enough time and resources to
build _and maintain_ them.

I don't know about you, but I have a constant backlog of features with tons of
interesting problems to solve. There are places where I could build a DSL, but
I am not going to because I would inevitably build something hacky and brittle
that is impossible to understand and maintain (like JDSL).

I can't build new features, and increase the value of the system if my DSL is
constantly breaking the system.

So don't do it. There is plenty of functionality on the .NET and Ruby
platforms for me to model the concepts and objects in the domain without the
need for a DSL.

~~~
amyjess
Oh, this reminds me.

I once worked for a company that used RDF for all configuration and scripting
for one of its main applications.

Instead of using one of the standard methods of encoding RDF (turtle, XML,
etc.) our main architect wrote his own. Specifically, he wrote an encoding
based on Google Sheets. The triples were the row header, the column header,
and the cell value. There were two special tables; one containing namespace
aliases and one containing an index of all the tables in the spreadsheet.

To deploy, you published a Google Sheet to the web, copied the sheet key, and
pasted it into a variable in the Java code's main() function (and we had a few
different main classes for a few different deployment types).

Later on, he added support for encoding RDF in XLSX spreadsheets so we don't
need an Internet connection to launch the app.

Let's just say that architect and I didn't get along (and he didn't get along
with a good chunk of the devs either; multiple people I worked very well with
quit during my time at that company largely because they couldn't get along
with him).

If you want to see it, the code is open-source (some of their content is
proprietary, but there's an open-source RDF store in the public repos that has
the proprietary stuff removed and replaced with demo content). It forms part
of the appdapter project on Assembla [0] (but only part of it; there's other
random stuff in appdapter too), and there are some sample main() functions in
the friendularity project on Asssembla [1] (friendularity was our repository
of open-source main classes; they mostly just import stuff from other open-
source projects and wrap it in a main()). Honestly, though, I didn't touch the
appdapter part of the codebase when I worked there, and it's been a couple of
years since I've worked there, so I don't think I can help much with running
it. I think org.friendularity.bundle.demo.ccmio (or maybe
org.friendularity.bundle.launcher.oglweb) is the relevant main project, but I
don't remember too well, and I think some things have been refactored since we
left.

[0]
[https://app.assembla.com/spaces/appdapter/subversion/source](https://app.assembla.com/spaces/appdapter/subversion/source)

[1]
[https://app.assembla.com/spaces/friendularity/subversion/sou...](https://app.assembla.com/spaces/friendularity/subversion/source)

~~~
tracker1
While that is probably a bad idea... it does give me a potentially good
idea... using google sheets for internationalization string management... so
you don't have to build a custom UI.

two-column sheets.. first sheet is "default", sheet name == lang_location and
overrides... column 1, except first row, is the key, column 2 is the localized
value, column 3+ unused, but can provide more context information.

------
userbinator
I saw similar levels of ridiculousness with XML during my (brief) time working
with Enterprise Java many years ago, although it didn't go as far as the use
of Subversion... but I did experience a lot of the "you just don't understand
how beautiful it is" sentiment from the architecture-astronaut creator(s) of
these behemoths. Now that JSON is the "new modern XML", I'm not surprised at
all that things like this have come into existence.

------
rollulus
Before I started my professional career I used to believe that the stories on
the daily wtf were fiction.

~~~
clifanatic
The difference between truth and fiction is that fiction has to make sense.

------
EnderMB
What a silly system! They should have used BobX instead.

[http://thedailywtf.com/articles/We-Use-
BobX](http://thedailywtf.com/articles/We-Use-BobX)

~~~
scrame
hmm. actually, the looping and conditionals in bobx look a bit like jstl.

------
petetnt
The scary part here is that I am not actually sure whether this is fiction or
not.

~~~
junke
> Founded in 2004 by Alex Papadimoulis, The Daily WTF is your how-not-to guide
> for developing software. We recount tales of disastrous development, from
> project management gone spectacularly bad to inexplicable coding choices.
> Remy Porter is the editor-in-chief and needs to read your story or see your
> bad code.

And below:

> Submit Your WTF

Stories are not supposed to be fiction. Granted, editors can't check how true
what you submit is (and often, you only have one side of the story), but there
seems to be so much available material out there that we don't need fictional
entries.

~~~
douche
At one point, I believe they released a Visual Studio extension that would let
you highlight a particularly WTF-worthy chunk of code and post it up easily. I
was tempted with some of the code that I inherited after it had gone through a
couple iterations of cut-rate outsourced freelancers...

------
outworlder
This story _could_ be fiction, but I don't think so. For starters, it takes
some serious imagination to come up with a system so broken. And I _have_ seen
a system, written in a javascript dialect, execute by an in-house javascript
engine written in Delphi, that you had to use the in-house IDE because it was
the only way one could find the desired functions. You see, the 'module'
system worked by referencing database IDs, from a database table that
contained the javascript "file" you wanted to import...

------
ahoka
Why is there always a "genius" behind stuff like this?

~~~
s_kilk
It's usually a combination of a competent but misguided senior engineer and a
fleet of buisiness dudes who aren't qualified to assess whether the engineers
work is actually any good.

He'll seem to weave magic initially, and the company/project will launch and
make money for the people involved. It's only later, when other engineers show
up, that the extent of the horror becomes evident, by which point the senior
engineer has already been enshrined as the "genius" behind the companies
success.

Only the new, junior engineers can see the Emperors clothes for what they are.

~~~
jerf
"It's usually a combination of a competent but misguided senior engineer"

Quibble: Anyone who comes up with and deploys that sort of system is not
competent. They may be "senior", chronologically anyhow, but they are not
"competent". Compenent senior engineers know about the inner-platform effect.

Similarly, to me making avoid a second reply, it's not just lack of
communication that makes this system bad. The inner-platform antipattern is
objectively bad, no matter how it is communicated.

~~~
s_kilk
We could change "competent" to "capable", which may better capture the spirit
of the argument :)

The point is, this engineer, for all they're many, glaring faults, can get
stuff done. Impressive stuff, even. And that's all the business dudes see in
this scenario; Dave plugging away at a computer and cheques from customers
showing up in the mail every month.

~~~
jerf
Fair enough. Such systems do require an impressive amount of capable labor to
make work at all. Labor to staggeringly negative effect, but yes, capable
labor. One could do wonders with such an engineer if you can mentor them...
and if they'll take a mentoring.

~~~
s_kilk
> ...if you can mentor them... and if they'll take a mentoring.

I suspect these "genius" programmers are damaged beyond all hope :)

------
Kristine1975
JDSL sounds like a capital idea, but obviously Subversion is much too slow,
and Javascript/JSON is not enterprise-compatible. Someone should re-implement
it using EJB's, XML and an Oracle backend to store the functions in.

P.S: Dear Eris, please let me never turn into Tom!

~~~
maze-le
I honestly fear the day my boss stumbles over this idea.

~~~
nameless912
This is like the programming version of sarcastaball. We'll make up the
_stupidest possible_ idea, and then our bosses will find out and think they're
brilliant.

------
marsrover
"What the fuck" was the first thing I thought after finishing this article.
Didn't even realize it was from "thedailywtf.com", but it sure fits there.
Good article, hope it's fake.

~~~
clifanatic
I sincerely doubt it was fake.

~~~
whatever_dude
It's one of those stories that might be fake but still ring true. I know I've
been there, on both sides (to some degree), so I can relate.

------
firephreek
This isn't real right? It's just a made up story to scare, right??

------
mtrn
So many things can go wrong. But how do you assess, where to draw a line? What
could be used as a guiding light?

Currently I try to use the following scenario as orientation: If a completely
new developer would walk in and wanted to install or build or contribute to a
current system/project, how much I would have to explain to this person? How
many, "yes, true, but only if" style statements would I need to make?

The optimum here is, that a developer will contribute, without me needing to
explain or motivate anything.

Do you use a certain approach? How would you describe it?

~~~
sly010
Is this really that much different from any other industry though?

The system in the story is clearly insane, but every organization (not just
software companies) has it's own operational awkwardness, that might look
weird for new employees.

Imagine that Google has all their code in a single repository. How "insane" is
that?

My point is I don't see why is it a hard requirement for a new developer to be
able to commit code on day 1. Of course if you have hundreds of developers, it
makes sense to optimize the process, but personally I would prefer a new
college to walk me through things (apprenticeship style) rather than throw me
at a repo and go-figure-it-out.

------
diyseguy
This story is like a parable that condenses all my work experiences so far.
The problem is that management is easily terrified and impressed by anyone who
can flummox them with tech buzzwords and knows just enough programming to be
dangerous. They get in early, create a horrific mess that takes other people
at least a year to untangle enough to be able to maintain. The company talks
forever about rewriting it - but everyone's too busy putting out fires in the
spaghetti to work on the newer better architecture. Really, most of the crap
these 'geniuses' write could be quickly replaced with something off the open
source shelf. But the customers are too entrenched in the proprietary
spaghetti, so they can't, etc. And every new 6 figure tech guy they hire just
tries to create his own cozy nest of over complex code in a sad attempt to
create job security. Its all a scam.

------
jpalomaki
Think about all those dependency management systems (npm, Maven, NuGet). Look
at a project with huge number of dependencies, picking on piece of
functionality from one library, something else from another. All tied together
with some depedencies file and maybe with some additional dependency injection
magic. Not backed up by version control, but instead the package repository.

Not that many projects add a real DSL, but I think you can end up pretty close
also with just objects and language features.

And where do you then run this system? Of course in Docker container, probably
picking up Linux components with apt-get from left and right.

~~~
AgentME
The bad things in the article were the inner-platform effect and that the
inner-platform was especially terrible (no comment support, tons of
boilerplate, abuse of the project's own source control, lack of dev/production
split). I really don't think the concept of versioned dependencies was one of
the bad parts.

------
new_hackers
Did anyone else notice the part about checking into the trunk?

I agree that the framework and people were definitely WTF.

However, I would at least have talked to someone before committing to the
trunk, even if I thought I was just adding comments.

------
fndrplayer13
Enjoyable read, and I think for any experienced developer there's at least a
_bit_ to relate to in there.

But seriously, this is why you have an integ/staging environment :)

And obviously, better developers.

------
WhiteNoiz3
I've come to believe that 'clever' solutions are almost always a bad idea. You
should err on the side of best practices. Defying expected norms means that
other people aren't going to be able understand your code or maintain it. This
is a pretty spectacularly bad example of that. I've actually worked on sites
that did similar stuff to this - having multiple versions of the code checked
out on the same server with references to files in specific commits.

------
pmontra
Jake should have left two weeks earlier. That JDSL was a disaster waiting to
happen. Jake was lucky it happened quickly and he didn't waste his career on
that system.

------
pjungwir
I think the more interesting question is: when can something crazy like this
work? I started writing a comment about that here, but it got long enough I
turned it into its own blog post. Here is my submission:

[https://news.ycombinator.com/item?id=12188087](https://news.ycombinator.com/item?id=12188087)

------
zer0gravity
It may seem appealing to go down the meta meta programming rabbit hole, but
let's not forget, that any intelligent fool can make things bigger and more
complicated, it takes a stroke of genius to move in the OPPOSITE direction. In
engineering things need to be functionally beautiful first.

------
mywittyname
This reminds me how I always seem to fall into the "find and program to the
pattern" anti-pattern.

I am Tom.

------
Furincer
Satire aside, couldn't they just revert the commits? That is the whole point
of the system. ONLY if the system didn't work would Tom be angry, but IF the
system didn't work then the rest of the story is contradictory.

~~~
winstonewert
reverting the commits won't bring back the deleted database

~~~
iraphael
which brings up the other problem of why aren't there backups

------
mrsheen
What's the name of the company? Their customers deserve to know that story.

------
anton_gogolev
Reading these kinds of stories, I can't help but reflect on how many days of
my life spent dealing with crap similar to this I'm never getting back. Hair
stands on the back of my neck, literally.

------
rurban
Great example of abstract business logic. Totally believe it's true.

------
Waterluvian
This is a work of fiction right? I feel so stupid for having to ask.

~~~
damptowel
I wish these kinds of things didn't happen in the real world, but they do.
Management wants it fast and cheap, wether it's good, they have no clue, they
probably don't trust their engineers who speak too much gobbledigook. So they
hire the cheapest sob they can find, if he screws up, the next scrub can take
his place, and that one might in turn get fired if the first one made too big
of a mess to refactor ("What do you mean, you fixed this bug and now there are
TEN MORE? We're already over budget and you were supposed to have finished
last week. And don't give me your technobabble excuses!"). Happens more often
in places ran by wannabe get-rich-quick type entrepreneurs.

------
paloaltokid
My WTF entry: I worked at a company whose main website was written in J2EE.
This was here in the Bay Area, right in downtown SF.

When I joined, I soon learned about "Stephen". Stephen had already left the
company but he was still very much a presence. Most of his comments contained
expletives, even for something simple like changing a log level: "tell Tomcat
to shut the fuck up!"

Throughout the wiki, there were references to technical "discussions" around
coding standards that often had statements like "Stephen doesn't like this!",
which was short for "I don't want to get yelled at so we'll do what Stephen
says". There were a lot of things Stephen didn't like. He didn't like Java. He
didn't like private methods. He seemed on the fence about getters and setters.

But he _loved_ Python. Boy howdy did he love Python.

Stephen loved Python so much that much of the Java codebase was uneditable.
Yes, uneditable. You see, in order to write the Java code that'd be actually,
you know, run in production, you didn't simply open IDEA and write Java. You
wrote Python code that emitted Java. Hibernate configs, classes, the lot.
Development time wasn't the fastest.

Something else Stephen hated was MySQL. He hated it a lot. Postgres was way
better in his opinion. Stephen hated MySQL so much that he stripped it out of
all the Puppet files one day because he didn't think the company should be
running it. Nevermind that he was editing the production files for another
team, so the next time Puppet ran, the predictable outcome occurred. I believe
that was one of the few times his actions had any direct consequences for him.

Beyond Stephen and his hatred of all things non-Python and MySQL, the
deployment process was a true thing of horrifying beauty.

You see, one didn't simply just drop a WAR file into Tomcat and restart the
server process. That'd be much too simple. When you deployed, you deployed
everything. And when I say everything, I mean everything.

Apache? Compiled from source and re-installed. Java? Compiled from source and
re-installed. Name a dependency other than Postgres and that ish was compiled
from source and re-installed. Deploying any code change was a minimum
45-minute process. Heaven help you if a bug was found.

(Oh yeah, and of course Stephen's domain user couldn't be removed from
Perforce or any of the build systems because all the scripts ran as his user.
Nobody knew what might happen if his user was removed. To this day the company
still exists after an acquisition and I'd bet good money that Stephen's user
is still out there consuming CPU cycles.)

My takeaway from this experience is that this kind of environment is pretty
common, and it's because of some basic psychology in people. Engineers don't
want to look stupid and their social skills are often not developed. So when
you meet someone with a strong personality who uses shouting to establish
technical superiority, they can get a long way.

------
JustSomeNobody
I would kick Tom in the jingle berries... so hard ... that every time he ever
came up with a crazy idea like this he would ache.

------
MWil
Is there any easy way to recreate that blocked visualization of the JSON?

------
nojvek
Didn't they have code reviews ?

------
NhanH
Is that true story?

------
ragebol
Really hoping this is a joke and fiction. At least I chuckled.

------
clifanatic
_And_ that short employment stint is a black-mark on Jake's resume. What an
industry we have.

~~~
phkahler
>> _And_ that short employment stint is a black-mark on Jake's resume. What an
industry we have.

But now he can just point people to that WTF piece and say "that's what that
was". Anyone who still considers it a black mark after that isn't worth
working for.

I have a 5-month stint at a huge company on my resume - one that a lot of
people would really like to work for. It comes up sometimes in interviews, but
I've got a good story behind it and usually come off as capable and ambitious
after explaining it.

If something raises a question, ask it. Don't just assume the worst of a
candidate.

~~~
chadgeidel
I agree. If a company isn't willing to give me the benefit of the doubt by
_asking_ , then maybe it's another place I'm not interested in working.

