
New Grad vs. Senior Dev - kogir
https://ericlippert.com/2020/03/27/new-grad-vs-senior-dev/
======
maliker
I'm the senior dev on my team, and whenever a new dev joined my team they
would look at the codebase and go "ew, python2? Just use python3."

That gave me a chance to explain the testing and refactoring cost that would
come with changing python versions, and how the benefits to users would be
almost zero. And then at some point one of the new juniors said, "hey, there's
a lot of filesystem performance improvements and readability improvements
(f-strings) in 3. I can get the test/refactor done in a month, and I think
it's a net positive." They were right, and now we're on python3.

So, sometimes we all learn something.

~~~
bogdanu
The reverse of this also happens: new team manager joins a team of 4-5 dev and
goes "eww... a monolith, we'll write an MVP in 3 weeks with microservices,
CQRS and all".

Long story short, one year and a half passes and the mvp is still not
finished, the architect leaves the company and some poor guys (from an
outsourcing company) are still going at it with the same architecture.

~~~
cameronfraser
CQRS and microservices has so much overhead, I'm amazed at how many companies
adopt microservices without having anyone know a single thing about
distributed systems. I think people underestimate the glue required to make
microservices be useful. I had a similar situation at my last company where
they spent a year and a half converting their monolith into half ass
microservices and it still wasn't working properly. They also stopped
releasing new features on their product during this entire conversion which
obviously lead to a drop in customers. The mass exodus of employees at the end
was something beautiful though.

~~~
Lapsa
I would argue that main reason for adaptation of microservices is to actually
prolong development time (moar dollars) and make devops as convoluted as
possible to safeguard data. Or foolishness.

~~~
marktolson
I don't understand the hate for microservices on hn. I have found
microservices are a great way to extend monoliths developed with legacy
frameworks. There are so many pros with the cons easily avoidable with the
right tooling / processes.

~~~
JamesBarney
Basically it comes down to

Distributed logging:hard

Distributed debugging: hard

Distributed versioning: hard

Distributed transactions: very hard

And on 95% of projects these problems aren't worth solving for the benefits of
microservices.

~~~
bogdanu
> And on 95% of projects these problems aren't worth solving for the benefits
> of microservices.

And especially with a team of 4 or 5 developers.

~~~
capableweb
I just started working on a new project about half a year ago, completely
greenfield. The backend developer (there is only one...) jumped into
microservices directly, deploying on AWS Fargate, trying to split out as many
things into containers as possible, because that's the "proper" way of doing
it. We still have few users as the project is new, but hey, at least we could
scale to 100000s users in a few minutes, instead of having easier logging,
debugging and versioning

A lot of stuff in software engineering is just driven by what's popular, not
what's needed.

~~~
bcrosby95
A monolith can handle millions of daily unique users. The database is the hard
part.

In the web world, if you have less than 10s of millions of daily users, as
long as you design an application that holds no state itself, the architecture
is usually more important for scaling your team and the size of your codebase
rather than the number of users you can handle.

~~~
redis_mlc
With microservices, you lose the ability to use database transactions across
services/tables. Later on, when people realize this, it's too late.

It's priceless to see their expressions when senior mgmt. and business owners
finally find out.

~~~
rumanator
> With microservices, you lose the ability to use database transactions across
> services/tables. Later on, when people realize this, it's too late.

I don't follow your reasoning. I mean, the ACID vs BASE problem you described
is extensively covered in pretty much any microservices 101 course or even
MOOC, along other basic microservices tradeoffs such as the distributed tax
and ways to mitigate or eliminate these issues like going with bounded
contexts. Why do you believe this is a mystery that no one is aware of?

~~~
JamesBarney
I've seen a couple of projects where the original team didn't think they
needed transactions, or underestimated how much harder eventually consistent
distributed systems are to reason about than an acid system.

------
dimtion
I find that the biggest misunderstanding happens because "new grads" (and I
happen to be one) confuse _asymptotic complexity_ with actual complexity.

I'm not sure sure why, but CS courses and interview questions mostly focus on
_asymptotic complexity_ and usually forget to take into consideration the
complexity for "little values of n". And funnily enough, in real life n never
goes to infinity!

In a strict sense big O notation only cares about what happens when n goes to
infinity. The algorithm could behave in any way up to numbers unimaginable
(like TREE(3)) but still, its big O wouldn't change.

Maybe what is missing to those "new grad" is a felling of real world data, and
how a computer behave in the real world (with caches, latencies, optimised
instructions etc...) not just having an ideal computer model in their mind
when they design algorithms.

~~~
corysama
It’s because Big O is Computer Science. Cache effects are Software
Engineering. Professors of CS do a fine job of teaching CS. They even briefly
mention that there is a implicit constant factor k in O(k n log(n)) and then
they never mention it again. They certainly don’t mention that k can easily
vary by 128x between algos. AKA: 7 levels of a binary tree. Or that most of
the data they will be dealing with in practice not only won’t be infinite, but
will actually be less than 128 bytes. Or, that even with huge data and an
proven-ideal O() algo, there is often 10x speed-up to be had with a hybrid
algo like a b-tree instead of a binary tree. And, another 2-10x with SIMD vs
scalar. 100x isn’t infinite, but it’s still counts.

So, grads listen to their CS professors and that’s what they know. It’s not
until they get lectures from greybeard software engineers that they learn
about the reality algos and not just the idealized algos.

~~~
MaxBarraclough
> briefly mention that there is a implicit constant factor k in O(k n log(n))
> and then they never mention it again

A fine concrete example of this is the Coppersmith–Winograd algorithm (and its
derivatives), a matrix multiplication algorithm with impressive complexity
properties, but which in practice _always_ loses to the Strassen algorithm,
despite Strassen's inferior complexity. [0][1][2]

(Aside: the Strassen algorithm is pretty mind-bending, but also easily shown.
If you've got 22 minutes spare, there's a good explanation of it on YouTube.
Perhaps there's a more dense source elsewhere. [3])

> It’s not until they get lectures from greybeard software engineers that they
> learn about the reality algos and not just the idealized algos.

To mirror what some others are saying here, students should also be taught the
realities of cache behaviour, SIMD-friendliness, branch prediction, multi-
threaded programming, real-time constraints, hardware acceleration, etc.

[0]
[https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_a...](https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_algorithm)

[1]
[https://en.wikipedia.org/wiki/Strassen_algorithm](https://en.wikipedia.org/wiki/Strassen_algorithm)

[2]
[https://en.wikipedia.org/wiki/Computational_complexity_of_ma...](https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations#Matrix_algebra)

[3] [https://www.youtube.com/watch?v=ORrM-
aSNZUs](https://www.youtube.com/watch?v=ORrM-aSNZUs)

~~~
msla
> To mirror what some others are saying here, students should also be taught
> the realities of cache behaviour, SIMD-friendliness, branch prediction,
> multi-threaded programming, real-time constraints, hardware acceleration,
> etc.

Which would have the positive knock-on effect of the textbook being
sufficiently obsolete every year or so that the students could no longer trade
it in for credit, saving the bookstores money!

More seriously, that knowledge (at least once you attach numbers to it) has a
shelf life, and not a very long one. Teaching big-O analysis means the
knowledge is timeless, which any good theoretical knowledge is, and moving
more towards practice would force the professors to keep on top of the state
of the art, and the state of the _mainstream_ , of hardware design in addition
to everything else they're doing.

~~~
temac
> Which would have the positive knock-on effect of the textbook being
> sufficiently obsolete every year or so that the students could no longer
> trade it in for credit, saving the bookstores money!

That's naive. E.g. SIMD is here since a good time and going to stay. So are
GPGPU, with now quite similar architectures for tons of chips.

And Computer Science can actually be about science for real computers, and
computers are not 8086 nor PDP11 anymore, and have never been a turing
machine. So there actually _is_ some existing generic CS and ongoing research
that cares about cache effects and so over. Maybe it is applied CS if you
want, and some kind of pure CS should not care about that, but I really don't
see what should be the criteria to decide which is what anyway, so IMO there
should not be any (but I do _not_ mean that all research should care about
e.g. cache effects, just that it is not really useful to attempt to
distinguish between those which do and those which don't).

We don't teach advanced math by only showing what was done at e.g. the
beginning of algebra. Neither should we stick to _only_ basic subjects in
computer science.

------
reader_1000
In real life, you will always start with simple working implementation and go
with it. Then if things are slow, you profile your code with a good profiler
while running for some kind of real life scenario and spot the slow parts
(also keep in mind that profiling may affect the program's behaviour). After
that you may want to consider alternatives with less asymptotic complexity iff
that's the part causing slowness.

Once I was asked to look for one project to see that if there is any room for
improvement to speed up the program. After I profiled the program with the
test data, I saw that program was severely affected by a "size" method call on
a lock-free concurrent list. Since the data structure is lock free, size
method is not a constant time operation and calling it in a large list takes
too much time. It was just there to print some kind of statistics, I changed
the code so that it is called only necessary not every time some operation
occurs. This immediately made program 2-3 times faster.

There were also some parts I changed with some algorithms with less
algorithmic complexith to make it faster. Overall, I made the program 6x
faster. So sometimes you need to use fancy algorithms, sometimes you just need
to change one line of code after profiling.

~~~
yashap
Although I mostly agree, there’s certain types of simple, fundamental
performance considerations you want to take when writing code the first time,
otherwise you’re just needlessly dealing with tones of performance issues in
prod. Like make sure your DB queries are well covered by indexes, if you’re
repeatedly looking up items from a list, turn it into a map or set first, in
FE code try to keep re-renders to areas of the UI that need updating, etc.

Correctness and simplicity are almost always things you want to focus on over
performance, but there’s still SOME level of simple performance best practices
that you want to stick to up front. Similar to the tradeoff between YAGNI and
software architecture - if you don’t start with some sort of coherent
architecture, your project is gonna descend into a spaghetti mess. Ignore
simple performance best practices and you’ll spend way too much time fighting
performance issues.

~~~
ehsankia
> Correctness and simplicity are almost always things you want to focus on
> over performance

Right, it's a balance good developers know how to tread. Obviously if you can
use a set instead of a list and it's one line change, go for it. But as the
meme in OPs post, if you're gonna USE SEGMENT TREES and all, then it better be
worth the amount of complexity and time you're putting into it.

Part of what makes a good engineer is being able to quickly tell where it's
worth optimizing and where it isn't. Or as the meme says, when nested loop
goes BRRRRRR.

~~~
yashap
Yeah agreed with all of that.

------
saagarjha
By the way, here’s an anecdote for the flip side: at one of my internships I
was working on a tool to process large log files, and by careful application
of Aho-Corasick I was able to make it about _50 times_ faster on the dataset
we were dealing with, which made using the tool change from “let’s go grab
lunch while this finishes” to “let’s stream the logs through this live”.
Sometimes you do know how to make things faster: just make sure to test them
before you do, and asking why something is a certain way is always a good
thing to do before proclaiming it’s broken.

~~~
lalaland1125
For interest's sake, did you try simply using a decent regex engine as an
alternative? Any DFA regex engine implicitly implements Aho-Corasick for you.

~~~
saagarjha
I think the engineer before me put a bit of effort into this, but it didn’t
work out all that well in this case. This isn’t surprising considering that
the standard regex tool on the platform wasn’t known for being particularly
performant.

------
jakear
Heh... reminds me of my first proper MS internship, when I too was responsible
for speeding up some code, this time in the VS Code Go extension. This code
was responsible for tokenization, so it affected pretty much every operation
and ran on every edit. Important shit.

Day 1: do some basic hoisting. O(n^3) => O(n^2). Tokenization times for a 10k
line file go from ~15s to 500ms. Sweet.

Days 2-30 [1]: ideate, develop, bug fix, iterate on, etc, a novel (to me)
data-structure to speed up the program even more. O(n^2) => O(n x log(n))
(expected). Great! Runtime on 10k line file went from 500ms to _maybe_ 300ms.
Oooops.

But hey, all the people working in 500k line files must really love the couple
seconds my month of toiling (more importantly, my month of not doing other,
more impactful things) saved them.

Learned a lot from that experience, and writing this out now I can see how
that impacted engineering decisions I make to this day. I suppose thats the
_real_ point of an internship, so time well spent, in a way.

[1] It probably wasn't actually a month, but certainly a significant chunk of
the internship.

~~~
TeMPOraL
> _But hey, all the people working in 500k line files must really love the
> couple seconds my month of toiling (more importantly, my month of not doing
> other, more impactful things) saved them._

This stuff matters. These couple seconds per operation may very well be a
difference between being able to open a 100k+ LOC file in the same editor
you're doing your other work in, vs. giving up in frustration and looking for
something else that can handle large files. Large files happen surprisingly
often (in particular: log files, data dumps, machine-generated code). A "month
of toiling" like this may immediately enable new use cases.

~~~
jakear
This is true, but my approach was still not a great one. It turns out the data
structure was only accessed in a very specific way, which I could have
exploited to dramatically simplify the implementation. If I had researched the
problem more before diving into my original idea, I could have achieved the
same end goal with much less work.

Lessons upon lessons :)

------
tyleo
I see these senior vs non-senior engineer contrasts pop up a lot. I’m not a
huge fan of them.

It seems that there is a spectrum of skills an engineer could excel at:
programming, infrastructure, managing, planning, etc. I’ve known senior
engineers who only excel at a particular skill. I’ve also known senior
engineers who are moderately good at many but not particularly good at one.

In my experience the only difference between a senior and non-senior is that
the senior’s distribution of skills makes them more effective.

I’ve also seen senior engineers change roles laterally and become more or less
effective, yet still maintain the senior title.

~~~
ericlippert
Well then, feel free to write your own blog post on a topic you enjoy more!

~~~
XelNika
I don't think he dislikes the topic, but rather the way it is framed as senior
vs. junior instead of subject matter expert vs. non-expert. The skipto example
from your post is not exemplary of the difference between a senior dev and a
new grad, it seems very domain-specific.

------
TeMPOraL
Oh god. That meme.

I've seen it a day or two ago. Can't find the picture anywhere now (I've seen
it in some group chat). Anyway, beyond the words quoted at the beginning of
this article, the meme's "nested loops go brrr" had a picture of a triple-
nested loop using Active Record to do some simple database operations.

To which the correct response is: "it's a 'senior developer' in an industry
where you get called a 'senior' after a total of 3 years of experience doing
programming; don't do stupid shit like this, _just use SQL like it 's meant
to_".

~~~
tspop
Hey, I made that meme.

It was based on a similar story the one in OPs blogpost. At my first job I
used to work with some really talented fresh grads that wanted to show off
their algorithms skills and ended up over-engineering stuff.

One of them implemented a trie and stored it in SQL lite to implement some
string autocomplete where the number of strings was something like 100.

The other implemented a 2D segment tree for doing some grid updates where the
size of the grid was small. This inspired the first part of the meme. Segment
trees and sqrt decomposition are topics that are popular at programming
contests and nowhere else really.

Regarding the triple nested loop, I just wrote the simplest pseudocode that
represents nested loops, not necessarily something a "senior" developer would
write.

~~~
lonelappde
New hires showing up at work and doing the one thing they were tested on in
the interview. How strange of them!

~~~
DonHopkins
I like to ask interviewees to imitate the sound a computer would make over an
AM radio while executing different algorithms.

Nested for loops go brrrrrrrrrrrr, munching squares go bweep bweep bwweeeep
bwweeeep bwweeeep bwweeeep bwwwweeeeeeep bwwwweeeeeeep bwwwweeeeeeep
bwwwweeeeeeep bweep bweep bweep bweep...

[https://www.youtube.com/watch?v=V4oRHv-
Svwc](https://www.youtube.com/watch?v=V4oRHv-Svwc)

Life goes shlup shlup shlup shlup shlup...

[https://www.youtube.com/watch?v=hB78NXH77s4](https://www.youtube.com/watch?v=hB78NXH77s4)

If they use any Don Martin sound effects, I hire them on the spot.

[https://www.madcoversite.com/dmd-
alphabetical.html](https://www.madcoversite.com/dmd-alphabetical.html)

>CHK CHK CHA-GONK BRBBRBBRING! -- Man's Eyes Being Poked Like A Cash
Registers' Keys And Jaw Popping Open Like A Till Drawer -- Mad #61, Mar 1961,
Page 18 -- Kitzel's Department Store

~~~
tartoran
Thanks for sharing the DEC PDP videos. I find them fascinating and the sound
gives them a new dimension

------
lordleft
This is a great story. I feel like talented CS students are particularly prone
to this sort of myopic thinking; their professors introduce them to functional
programming, or the philosophy of UNIX, and they carry that approach
everywhere, in a very blunt way. They also carry this little-examined
conviction that the world consists of mediocrities and mediocre code and
sometimes aren't charitable with the work of others.

~~~
gen220
Putting it another way, knowledge with the bare minimum experience required to
be effective is very sharp, and that sharpness is sometimes what’s required to
cut through old, retrospectively “wrong” ways of doing things. I don’t
disagree with what you’ve said, I think we all have met plenty of people
fitting your description, I just mean to say there’s another side of the coin.
As food for thought, much (not all) of the open source software we herald as
brilliant today was initially written by people still in or just barely out of
grad school.

~~~
chiefalchemist
The New Grad had knowledge. The Senior Dev had Understanding.

Understanding > Knowledge

It's that simple.

~~~
9wzYQbTYsAIc
Being pedantic here, but knowledge is equivalent to understanding (information
being knowledge without understanding). Wisdom is the word you were looking
for (knowledge being wisdom without experience):

The New Grad had knowledge. The Senior Dev had Wisdom.

Wisdom > Knowledge.

~~~
chiefalchemist
I see it differently. Knowledge =/= understanding. There are plenty of ppl who
know a lot but have little understanding.

Put another way, knowledge is the nodes. Understanding is grasping the
connections. Understanding is the higher power. Understanding is where the
magic happens.

Wisdom? Wisdom is next level understanding. It's the maturity of developing
connection within the connections.

------
nickysielicki
I think it's really interesting how complexity theory sort of falls apart in
the face of hardware-specifics and concrete use cases. The motto in computer
architecture is to make the common case fast (it's in like every textbook),
whereas the motto in computer science is to make the program scale/to solve
generically. When push comes to shove (as this article shows), the computer
architects seem to have the final word, at least when it comes to making
things go _more brrrrrr_.

There's so much stress and attention given to complexity theory for software
engineers, to the point that people will cram for hours to make it through
FAANG interviews. I understand that it's important and it's just something
that everyone has to go through... but data structure and algorithm
performance is a Wikipedia search away, and then you choose the appropriate
STL container and move on with your life. The same can't be said for gaining
an understanding of modern processors.

I'm not saying that one is more important than the other or vice-versa, I'm
just saying that it seems wrong to me that not knowing algorithms and data
structures can break an interview, whereas not knowing hardware is virtually a
non-factor.

The big take away for me is this: if you're not benchmarking, you're cargo
culting.

~~~
JoeCamel
I think you are wrong. Yes, better understanding of hardware implementation
(cache sizes and latency, branch prediction and pipelines, specialized
instructions) is important. But also in real life those asymptotic
complexities will often have similar c and N_0, or your input will be bigger
than N_0, so O(n) vs O(n^2) will still matter. Basic analysis of algorithms
makes your life easier because, in this simplified model, you can get a rough
estimate of time/space requirements. Of course, if you have competing
implementations, you will benchmark them (which is not straightforward to do
properly).

By the way, usually we use "complexity theory" for a part of theoretical
computer science (math) concerned with proving lower bounds (like the most
famous P vs. NP). What is required for basic analysis of algorithms (without
any fancy methods) is not very hard. Yes, you can search the web like we all
do, but first you need to know what are you searching for. Also, we reuse
algorithms/data structures but not only in the final product but also in our
own new algorightms so you cannot search it.

Regarding interviews, I think the main problem is that many companies and
interviewers try to blindly copy the questions (cargo cult) without
understanding the point of such interviews. The point IMO is that you evaluate
analytic skills of a candidate. Correct/incorrect answer is not everything.
You could do the same with math, science problems but algorithms/data
structures are closer to programming. Anyway, I feel this is becoming a
controversial topic.

------
twotwotwo
Go's strings.Index/bytes.Index also often use the "find first byte" approach;
now on amd64 it's a fancy vector instruction. If it finds the first byte too
many times without a full match, it falls back to a Rabin-Karp match using a
rolling multiplicative hash.

The strings.Index code is at
[https://golang.org/src/strings/strings.go?s=25956:25988#L101...](https://golang.org/src/strings/strings.go?s=25956:25988#L1018)
and the internal bytealg package with all the CPU intrinsics is at
[https://golang.org/src/internal/bytealg/](https://golang.org/src/internal/bytealg/)
with index_amd64.{go,s} as the relevant files for x64.

Battle-tested implementations of fundamental things like string searches,
sorts, hashtables, allocators, graphics primitives, etc. are often interesting
'cause you find out a lot about what you need (and/or don't need) to work well
in practice as well as protect from exploding worst-case scenarios.

~~~
makapuf
I sometimes think that you should be be able to hint the compiler that this is
a smallish string, this one can be big, this list should be mostly under 1000
elements so that it handles the correct version of an algorithm (bonus point
to let me provide several algorithms with different use cases)

------
daxfohl
As a senior dev, I wish that I could say I always knew more than my interns,
and that all the code that's there is because it was carefully planned to be
that way.

But more often than not, I don't, and it's not.

My take on it is that for the most part senior devs are a lot more paranoid on
breaking things and don't want to make code changes unless enough people are
asking for it and it'll make a notable difference. Even making things strictly
faster can break users if they depend on the algorithm being slow (an extreme
edge case that you'd usually say is the user's fault anyway, but could
nonetheless be relevant in things like optimizing away a spin-wait).

~~~
makapuf
Ah, spacebar cpu overheat [https://xkcd.com/1172/](https://xkcd.com/1172/)

------
rkagerer
Shockingly,

    
    
        InStr(<this page of 100+ comments>, "docum") = 0
    

I'm a dev with some grey hair who feels it would have been useful for all that
fantastic domain knowledge from Paterson to get documented in a code comment.

I'd love to hear if either of them ever went back and did that?

~~~
ericlippert
In this case I think not. However I strongly agree with your position and have
tried hard to ensure that my code is commented such that it explains why the
code works as it does. See [https://ericlippert.com/2014/09/08/comment-
commentary/](https://ericlippert.com/2014/09/08/comment-commentary/) for more
thoughts on this subject.

~~~
rkagerer
Hey Eric thanks for responding here!

Love that article, it was interesting to read your dissection of code
comments.

For anyone else reading it, here's an updated link to the CS file referenced
in his blog post (the original moved):
[https://github.com/dotnet/roslyn/blob/master/src/Compilers/C...](https://github.com/dotnet/roslyn/blob/master/src/Compilers/CSharp/Portable/Binder/Semantics/Conversions/UserDefinedExplicitConversions.cs)

~~~
ericlippert
Yeah, all the links are wrong now. I'll fix them!

------
Ididntdothis
I always envy people who work on this level instead of cobbling systems
together that integrate several systems all with their own set of flaws and
you can be happy if you can make them work together somehow. The algorithm
stuff seems pretty simple in comparison. A very local problem that can be
profiled well and you can understand most of the factors at play.

~~~
TeMPOraL
The "cobbling systems together" stuff is code bureaucracy rather than
programming. You're no longer dealing with constraints of physics, mathematics
and sound system design - you're building a bit-pushing layer in between
Kafkaesque monstrosities that were never truly intended to work together.

Sadly, most programming jobs seem to primarily involve code bureaucracy rather
than the "algorithmic" level.

~~~
capableweb
I've always seen the "cobbling systems together" stuff as actually software
engineering, as that's what you need to have a final production system
running. What you describe as "programming" I would say is computer science.

And for me, the work that solves real-world problems tends to be software
engineering (using my own definition), rather than computer science (again,
using my own definition), which seems to be more about optimizations.

------
nostrademons
This is also why C++ (and hopefully other languages by now) has the short
string optimization. On a 64-bit machine, a pointer to a heap-allocated buffer
is 8 bytes long. A stupidly large number of the strings used by actual
programs are <= 7 bytes long (it's actually more like 20 in C++, because
strings also include a length and capacity). Think about it: that's virtually
every file extension, path separator, and field delimiter, and a good number
of field names, labels, first & last names, usernames, identifiers, URL path
components, parameter names, etc. If you can fit the whole string into an
immediate value, then you can avoid the heap allocation and deallocation
entirely, you can avoid chasing pointers and incurring a cache miss, and you
can very significantly reduce the total amount of memory used.

This is also a good reason why, if you're designing a standard library, you
really want to have length-prefixed strings instead of null-terminated ones.
If you know the length of the strings you're dealing with, you can swap out
the algorithm you use, such that you might use brute force for a very small
needle, word comparisons if it's exactly 4 bytes long, SSE instructions if
it's a word-length multiple, or Boyer-Moore for very long strings.

------
04091948
Imagine that the intern didn't dare to ask such questions.

a - he could've gone out thinking that performance doesn't matter. but it
certainly does in a piece of code being used daily by thousands of devs.

b - he could've thought that the simple implementation is faster but missed
the fact that skip is implemented in assembly.

c - he could've realized both but missed the why.

and these failure scenarios are likely to happen because this is an intern
we're speaking about.

One or two tricks up your sleeve do not matter but repeat this a 100 times
which one do you think would be a better programmer?

I think the willingness to challenge authority figures and to be (often)
proven wrong and to learn from it is an essential part of becoming better.

Maybe Tim was understanding because he himself challenged people older than
him and in-process learned form them.

Advice like "don't reinvent the wheel", "avoid premature optimization", "write
boring code" promote exploitation. which is the optimal strategy for older
agents.

but for newer agents, a higher level of exploration is needed otherwise they
would converge into a suboptimal strategy

~~~
saagarjha
> he could've thought that the simple implementation is faster but missed the
> fact that skip is implemented in assembly

It’s not; the compiler is just fairly decent at transforming string
manipulation routines.

~~~
04091948
>As Tim explained to me, first off, the reason why VB does this seemingly
bizarre “find the first character match, then check if query is a prefix of
source” logic is because the skipto method is not written in the naive fashion
that I showed here. The skipto method is a single x86 machine instruction.

oh my bad then, from this text it sounded like it was written in assembly.

------
dunkelheit
I'd say that for widely used library code it makes sense to implement an
efficient algorithm. By its very nature the context in which a library routine
will be called is unknown and so it must be prepared for everything. Also for
widely used code the amount of total saved machine time can be quite
substantial.

In the story the senior dev makes a judgment call (that this routine will be
used in LOB applications where the worst case is unlikely to appear so the
implementation is OK) which is probably correct, especially considering other
priorities. And of course senior devs are much better equipped to make this
kind of calls than juniors, but they still can and will guess wrong.

> Moreover, Tim explained to me, any solution that involves allocating a
> table, preprocessing strings, and so on, is going to take longer to do all
> that stuff than the blazingly-fast-99.9999%-of-the-time brute force
> algorithm takes to just give you the answer.

That's why a good implementation will dispatch to the best algorithm at
runtime!

------
Jabbles
Of course, if this code were running in a server, or if it were part of a
library that was widely used, then suddenly you have the potential for a DoS
vulnerability.

The narrative would then be that the microseconds you saved all those devs
over the years were wiped out when hackers took down your system.

------
m3kw9
The senior dev knows the system, knows what they can get away with, and has
choices to use those knowledge powers. A newbie will always follow the
programming rules, like never going over the limit on a hwy. as they can’t
take much calculated risks

~~~
m3kw9
Senior has a lot more types of ammo to use to solve problems let’s just say

------
mettamage
> NO! YOU CAN’T JUST USE BRUTE FORCE HERE! WE NEED TO USE SEGMENT TREES TO GET
> UPDATE TIME COMPLEXITY DOWN TO O(LOG N)! BREAK THE DATA INTO CHUNKS AT
> LEAST! OH THE INEFFICIENCY!!!

I'm the opposite of this stereotype, and I think there are more like me. Two
reasons as to why:

(1) Psychological:

I never had this. As a junior dev, I don't like to optimize because I feel a
bit of pain when I need to moderately focus. I can do it, and I've done it
quite a lot. In that sense, I've experienced quite a bit of pain in my life.
It's something I've learned to live with. And when I have to focus, I prefer
to focus all the way, because whether I focus on 50% or 100%, the pain is
roughly similar. This leaves me in a state of either being lazy(ish) and
wanting to program in a simple and understandable way versus being willing to
go as deep into a topic as I would need to and focus on all the details step
by step.

When I'm intense, I also am still sympathetic towards simple code because I
know that I understand that in both states. I only understand complicated code
when I'm focused.

(2) There are enough CS grads that know better:

Also, on another note. Efficiency analysis is simply not taught at university.
Parts of it are taught, but they're never connected to real world cases.

For efficiency analysis (note I haven't done any but I've read a thing or two
and talk with a friend who is a performance engineer quite regurlarly about
it) I think there need to be a few perspectives in check:

1\. What is the user's actual behavior.

2\. What is the behavior for malicious attackers (if applicable, Intel should
do this more, to my knowledge they are doing it more now).

3\. How does the compiler translate it to assembly?

4\. How does it theoretically look? Specifically: worst-case, best-case,
average-case in both theoretical and empirical sense.

Concluding:

I only have one year of experience in software development, where no one cared
about performance yet I know better than to look only at the theoretical
speed. So I guess I'm a counter example to the stereotype. I'm not alone. Any
CS student who reads HN has a high chance of stumbling upon articles like this
and will know that performance analysis is not only about calculating the
space-time complexity.

~~~
ericlippert
The "malice" aspect is a great one and I did not go into that in this post
because of course back in the 1990s we were not at all concerned that someone
would _maliciously_ craft inputs that would slow down this algorithm.

In modern code we'd want to do a threat model that considered the consequences
of untrusted inputs.

~~~
mettamage
Neither did Intel :P

Come to think of it, neither do beginning game-designers. They think that
players will play their game as intended.

I like that you're standing still at the malice part as it is becoming
seemingly more important every day.

------
whymauri
This is the meme referenced by the blog post:

[https://i.redd.it/lmrf72ko0ro41.png](https://i.redd.it/lmrf72ko0ro41.png)

------
xenocyon
Admittedly clueless question: what does 'brrr' mean? (I'm not a software dev
and idioms that may be obvious to others are unfamiliar to me.)

~~~
sergiotapia
It originated from a meme of the Federal Reserve printing more monopoly money.
"Printing machine goes brrrr". As they turn the crank ever faster making the
printing machines smoke.

~~~
gruez
More info: [https://knowyourmeme.com/memes/money-printer-go-
brrr](https://knowyourmeme.com/memes/money-printer-go-brrr)

------
jpochtar
Can someone explain the bugs in the code samples? The author says they're
there but I honestly can't find them.

~~~
ericlippert
Hint: what is the correct behaviour of this method when given empty strings?
Every string contains the empty string as a substring.

~~~
jpochtar
Oh, I'd assumed disagreement on behavior of query="" between the two code
samples meant it was UB and was looking for crashes/invalid memory accesses.

~~~
ericlippert
A Visual Basic program is not allowed to have undefined behaviours like a C
program; InStr has a specification and that specification has to be
implemented; that spec includes defining the behaviour for all inputs.

There's also no null handling here, which was a deliberate omission for
clarity. In practice, the convention used inside the VB source code is that
null string pointers are semantically the same as empty strings, which
introduces some complexities.

~~~
jpochtar
ah neat, thanks!

------
Forge36
One of my job tasks was improving our custom string library. In VB! Sure InStr
was slow: but we were trying to do "starts with" across very long strings.
Sure, it worked, but a slightly smarter solution was 100 times faster. Upon
finding it I went on a search, I found it written nearly identical 10 years
before me, in a different version of a string library (hint, that one was
faster).

The trick to knowing we needed to improve it: profiling!

~~~
kokey
You're telling me that a bigger instance size in AWS isn't always the
solution?

------
peteri
This isn't always the case that simplistic algorithms don't cause issues Bruce
Dawson keeps on finding O(n^2) algorithms in windows that keep on biting him.
I'm always impressed with his work.

[https://randomascii.wordpress.com/2019/12/08/on2-again-
now-i...](https://randomascii.wordpress.com/2019/12/08/on2-again-now-in-wmi/)

------
aerovistae
I dislike the mentality that one must "struggle" to be patient with new devs
and that it's "more than they deserve."

Is it really so hard to help other people learn, and to accept that the only
advantage you have on them is starting earlier?

~~~
ericlippert
I take your point, but let's be fair. My attitude was "this code is bad and
I'm going to demonstrate my skill by improving it" when it should have been
"please teach me what design and implementation concerns went into the choice
of algorithm here". I was lucky to get a gentle and thoughtful correction for
my presumptions.

~~~
nocman
Kudos to you first for recognizing your own error and second for openly
admitting it.

I think there many things to consider here. For new developers, I'd encourage
you to look for those "old guys" who really know their stuff. There's a lot of
unmined gold you can discover there if you find the right ones. I think us
older developers would do well to imitate the patience and kindness of Tim
Paterson more often. I guess what I'm saying is both sides could do with a
huge dose of humility. I know at times I've been the youthful dev out to one-
up the "old guys", and I've been the senior dev thinking "these kids today" to
myself when dealing with those with a lot less experience. And both of those
are bad.

Also, there are many times when a young guy fresh out of college spots a
problem, finds a great solution, and makes things ten times better by just
doing it! If you're in the business for a long time, it can become too easy to
be cynical, and lose your enthusiasm.

I think the best thing we can do is try to let the good stuff from both sides
rub off on each other.

------
jacobsenscott
I've noticed there's a similar form of this at the sr. dev/grizzled veteran
divide.

The sr. dev will look at "old" code in horror and be sure the only way to make
it perform is to port it to a newer language/framework/architecture.

The grizzled veteran will profile the code, think, and tweak a few lines. No
fancy new things or big re-writes needed.

------
joejerryronnie
Sometimes it’s not just coding tools/techniques where new grads can benefit
from a bit of oldster wisdom, like that one time I convinced a new grad it was
wiser to slog through the bureaucratic approval process for a new api than
attempt to hack into our company’s Oracle Financials db.

------
mlazos
This reminds me of how insertion sort is the most efficient sorting algorithm
for small values of n. I remember new-grad me always dismissing insertion sort
in every situation because of asymptotic complexity. Engineers are supposed to
find the most pragmatic solution without biases.

------
jb3689
To be fair, there have also been plenty of times in this situation where the
senior dev was wrong. You were right to question things and we should make
sure to continue to encourage questions. The difference between a new grad and
a senior dev is that the senior dev has the real-life experience to make the
"yes/no" decision as to whether the new grad's input actually makes things
better (e.g. it could be true that it doesn't matter if it is asymptotically
faster if we are always working with small data). It's also a trait of a
senior dev to be able to stay level-headed in these situations

------
cheez
Edit: I thought I was responding to someone, but I can't find the comment now.

I find that being open to new ideas, and allowing some time to prove them out
rather than deciding beforehand, is the best way to find ideas that move the
needle.

Senior comes to you with an idea? Great, prototype it, prove it out. Junior
comes to you with an idea? Great, prototype it, prove it out.

Of course, you need to allow some time for prototyping.

When I ran teams, I told them part of their job was to spend the last half of
Friday (unless emergency) prototyping their ideas and presenting their
favorites at some point.

No one works the last half of Friday anyway, unless it's on this.

------
wmu
Well, this is a nice example of lack of knowledge flow. The senior developer
spent his time explaining things that should have been put in the procedure's
comment. Yes, that's what comments are for.

------
jupp0r
The real lesson here should be: don’t make assumptions about performance. Take
real world use cases and measure!

That’s what my professors drilled into me (I specialized in high performance
computing) and it’s served me well.

------
joshwa
On a larger architectural scale, see: Use Boring Technology

[http://boringtechnology.club/](http://boringtechnology.club/)

Also described as: "how to be old, for young people"

------
mkchoi212
As a new grad myself, I completely agree with the author’s freak out when
seeing “inefficient” looking code. But I feel like this applies to basically
all aspects of a company life as a new grad software engineer. For me at
least, there seems to be a lot of inefficient things going on in every aspect
of the company. Maybe the author is right in that there is a reason behind
those seemingly “inefficient” things. However, the inefficiencies are there
because sometimes, people who have been working there for years just have
become used to it.

~~~
jacobsenscott
One thing you learn over time is many of those code horrors are there for a
good reason, put there by someone solving a real problem, in the best possible
way given time constraints. It is just the reason is lost to history. This is
one reason the "big re-write" almost always fails. You are throwing away all
those hard learned lessons just because they aren't apparent on reading of the
code.

That is also true (perhaps less often) for byzantine process related
procedures.

------
habosa
Seems like the original developer should have left a comment about why their
nested for loop was the right implementation (if they knew it was). Would have
saved everyone a lot of time.

------
certera
With the world of developers becoming "senior" at faster and faster rates,
we're asymptomatically approaching the demise of these questions, no?

~~~
jacobsenscott
Certainly at any startup "sr" is simply a self assigned title.

------
kazinator
I would use the standard ISO C strstr instead of writing the function they are
calling find.

Yes, we had strstr in 1994; it was in the 1989 ANSI C standard.

On a very small system, strstr will probably just be naively coded, to save
space. On a big system, it might use Knuth-Morrison-Pratt or Boyer-Moore. I
don't have to care; by using strstr, that decision is taken care of.

~~~
ericlippert
As I noted in the text, the actual problem that we had to solve was
complicated by a great many factors not the least of which was the fact that
the library had to handle strings from different string encodings.

------
arendtio
I think it is good when new grads have the knowledge to see that those things
don't look right and can articulate how and why it should look different.

After all, when you come across a problem, that does not contain just 100
chars, it is very helpful to know what you can use to create something that
still works with reasonable resources.

------
csours
The only nested loop implementation I've ever changed for performance was due
to the fact that there was a freakin' database call in the inner loop. All I
did was refactor the data structure so that the db call was done outside the
outer loop. Instant 20x improvement in performance.

------
fulafel
Were the x86 string instructions much faster in 94? That was 486/Pentium era,
right? IIRC this has varied over the years with sometimes slow microcoded
string instructions and then faster improved instructions again.

------
hyperpallium
But Boyer-Moore is so pretty!

    
    
      abcdefghijklmnopqrstuvwxyz
    

If the 26th char isn't _z_ , jump along 26 chars! (More complex, and more
cache misses, if _z_ is repeated in the search string).

------
nsilvestri
The explanation behind the code is exactly the kind of thing that belongs in a
comment. The why, not the what. So even if it wasn't a failure of programming,
it was a failure of documentation.

------
saagarjha
> The skipto method is a single x86 machine instruction.

That’s not always a good thing, especially on modern hardware. And obviously,
the “single instruction” doesn’t mean it’ll take bounded time to execute…

~~~
asveikau
Tangentially to your point, here's something I haven't thought about much:
when these instructions get an interrupt, I imagine they've updated (r|e)si
and (r|e)di, (r|e)cx etc. to reflect where they are in their copy or scan
loop. So if you get a page fault in the middle, then the kernel does enormous
amounts of work in response to it, then resumes that single instruction, it
resumes in the middle of the loop, not the start.

So to reiterate one aspect of your point, there might be lots of work, written
in C, that occurs in response to the page fault in the middle of your
"hardware-backed" single instruction. On top of all the other complexities of
cache vs memory access etc. that make scanning and copying memory complicated
no matter which way you do it.

But probably in the heyday of Visual Basic, and especially the DOS-based
BASICs that preceded it which wouldn't have had virtual memory at all, all of
this is less of a concern. The story takes place in a simpler time which
serves as a plot device to better illustrate the point.

~~~
vardump
Pretty pointless as said page fault would occur no matter how you access the
string. Besides, the page in question would very likely be already present.

~~~
asveikau
> Pretty pointless as said page fault would occur no matter how you access the
> string.

When did I dispute this?

> Besides, the page in question would very likely be already present.

Really? How are you so sure? I guess you can just abandon all notion of
virtual memory and mmap then. 'Cause it ain't gonna happen.

~~~
vardump
>> Besides, the page in question would very likely be already present.

> Really? How are you so sure?

It was in a BASIC interpreter. Most of the time string needle in a haystack
search is done, haystack is relatively fresh, almost certainly on a page that
is present. Might not be in CPU dcache, but that's another matter.

------
08-15
The way I read this, TIM FREAKIN' PATERSON, the man who wrote the most hated
"operating system" in history, put an algorithm with a nasty performance bug
into the standard library of VB, and conveniently forgot to document that this
function is no good for large and/or repetitive inputs. Confronted with his
blunder, he not only doesn't correct his blunder, but instead condescendingly
proclaims that those inferior VB coders would never need long strings, just
like 640kb ought to be enough for everybody, and if they did anyway, they'd
surely use a library written for the purpose by a real programmer.

The arrogant prick should have listened to the new grad.

~~~
ericlippert
And that you would read it that way says more about you than it does about
Tim, believe me.

------
stackzero
There's some good wisdom in this story. It reminds me of that article posted
here about a more page cache efficient b-trees vs. Knuth's theoretically ideal
version

------
a_c
There is another type/layer of new grad - the kind without CS background and
not being able to make a right solution (let alone balancing different
concerns)

------
Lammy
The true mark of a Senior dev would be commenting the code in question with
the thought processes and decisions that lead to the chosen approach.

------
noogler67
They could use naive solution for like len(query)<12, and then implement the
asymptotically good solution for other situations.

------
jacobsenscott
A CS degree helps you understand why something is slow, and how to make it
faster. Experience tells you when it matters.

~~~
OJFord
That's not even the case in this story though - the implementation _was_
faster, _for the usual way it was used_.

------
bitwize
New Grad: I'd use a linked list here because insertion into the middle is O(1)
rather than O(n).

Senior Dev: Linked lists have very many more cache misses than do vectors, and
the difference between hitting cache and hitting main memory is such a huge
constant factor that for most reasonable list sizes it never makes sense to
use a linked list. Use a vector. Checkmate, smug Lisp weenies.

~~~
erik_seaberg
Yeah, the Lisp world figured out cdr-coding in the 1970s and had moved on to
chains of vectors by 1985 according to
[https://cpsc.yale.edu/sites/default/files/files/tr362.pdf](https://cpsc.yale.edu/sites/default/files/files/tr362.pdf).
Fortunately they avoided changing the API to make this tradeoff.

~~~
bitwize
Afaik the only modern Lisp implementation that actually uses these techniques
is Clojure.

------
andybak
Aside - I wish the examples were written in pseudocode without c pointer and
type clutter.

~~~
ericlippert
Noted.

------
empath75
Most FAANG job interviews would fail you if you did the brute force solution,
it seems.

~~~
saagarjha
I feel that most will be sympathetic if to you explain why you did something a
certain way and showed that you understood the different approaches available.

------
fjfaase
Everyone who has done profiling knows that (almost always) the performance
problem is in the part that you least expect it. But also that sometimes you
wrestle to improve the performance of an algorithm hitting a hard wall and
than someone comes up with an idea, often resulting from a fresh look on the
problem, that results in an improvement of several orders. I guess that
performance is at least as counter intuitive as statistics. The same is true
for some other things like scalability and reliability.

Actually, I think that it scares the hell out of most of the developers that
it is so difficult to get a grip on these things. It is so easy to think that
there is a simple solution, a grand idea that will fix the problem. I still
find myself falling in the trap and this after having developed software for
over 30 year. It is the Dunning–Kruger effect over and over again. I guess it
more that as a more senior engineer, you have experienced a little more often.

~~~
jniedrauer
> I guess that performance is at least as counter intuitive as statistics.

I don't think this is really true. After you've optimized enough code over the
years, you start to get a sense for bottlenecks, and your code is usually
"fast enough" even on the first try. When it isn't, finding the problem with a
profiler is usually pretty straightforward.

------
malisper
I understand this as phenomenon as the new grad and the senior developer are
optimizing for different things. The new grad is focused solely on the
asymptotic complexity of the code. It doesn't matter how slow or how
complicated it is in practice, they are solely focused on using the fastest
data structure asymptotically.

The senior developer optimizes a different set of criteria:

    
    
      1) How hard is it to understand the code and make sure it's correct.
      2) How fast the algorithm in practice.
    

There are several different reasons why the performance of the algorithm in
practice is different than the performance in theory. The most obvious reason
is big-O notation does not capture lots of details that matter in practice. An
L1 cache read and a disk IOP are both treated the same in theory.

A second reason is the implementation of a complex algorithm is more likely to
be incorrect. In some cases this leads to bugs which you can find with good
testing. In other cases, it leads to a performance degradation that you'll
only find if you run a profiler.

I one time saw a case where a function for finding the right shard for a given
id was too slow. The code needed to find from a list of id ranges, which one a
given id fell into. The implementation would sort the id ranges once ahead of
time and then run a binary search of the ranges to find the right shard for
the id. One engineer took a look at this, realized that we were doing the
shard lookups sequentially, and decided to perform the shard lookups in
parallel. This made the code faster, but we still would have needed to double
the size of our servers in order to provide enough additional CPU to make the
code fast enough.

Another engineer hooked the code up into a profiler and made a surprising
discovery. It turns out the implementation of the function was subtlety
incorrect and it was sorting the id ranges _on every call_. This happened
because the code sorted the id ranges inside of a Scala mapValues function. It
turns out that mapValues does not actually map a function over the values of a
hash table. It instead returns an object that when you look up a key, it will
look up the value in the original hash table, then apply the function[0]. This
results in the function being called on every read.

The solution was to replace mapValues with map. This dramatically improved the
performance of the system and basically brought the CPU usage of the system
down to zero. Notably, it would have been impossible to discover this issue
without either knowing the difference between map and mapValues, or by using a
profiler.

[0] [https://blog.bruchez.name/2013/02/mapmap-vs-
mapmapvalues.htm...](https://blog.bruchez.name/2013/02/mapmap-vs-
mapmapvalues.html)

------
foxyv
This Dilbert went around the office a lot when I was a new employee. The
numbing is real...

[https://dilbert.com/strip/2003-02-16](https://dilbert.com/strip/2003-02-16)

------
bfung

        Nested for-loops go brrrrrr
    

lmao! But #truth.

------
sabujp
tldr; needle in a haystack, rolling polyhash

------
pdubs1
I have no CS degree or STEM degree.

Recently I worked on the same type of project as someone with 10 yrs of
experience & a CS degree from Stanford.

A few months later, I created a project, and had a manager with a CS degree.
However, when I left, that manager was unable to pickup where I left off, and
he ended up leaving soon after.

I have less years of experience, but to me, what matters more is the time
within that experience which was put into a relevant business model, product
built, or past projects. I.e. a Senior Dev with a CS degree, vs. a New Grad
with a Business Background & SWE Experience. It's apples to oranges in many
cases.

Also, I'd echo another comment here:

>"daxfohl 1 hour ago [-]

> As a senior dev, I wish that I could say I always knew more than my interns,
> and that all the code that's there is because it was carefully planned to be
> that way.

>But more often than not, I don't, and it's not. "

~~~
downerending
There are indeed many not-really-programmers with CS degrees.

On the other hand, sometimes I'm handed a program written by a new grad to
maintain/fix/improve, and rapidly determine that it's less work to just chuck
it in the bin and start over.

One of the major differentiators between a newbie and an old hand is knowing
how to create a piece of software that those who work alongside you or come
after you can understand, maintain, and improve.

------
jeromebaek
Sure. But most senior devs are not Tim Patterson.

~~~
toolz
but it's not uncommon for jr. devs to believe every piece of code deserves the
most efficient runtime. Runtime speed causing projects to fail is very
uncommon. What does add an incredible amount of work time is combing through a
codebase looking for micro optimizations. I've never once seen a jr. dev who
claimed to care about efficiency start by writing benchmarks over large parts
of the system and using that to find real bottlenecks. No, they always comb
through the code trying to impress the sr. dev. with their algorithmic
knowledge.

~~~
TeMPOraL
> _Runtime speed causing projects to fail is very uncommon._

That's true. What is common, however, is bad runtime performance losing you
users and bleeding your money. Not doing dumb things (like using a list where
a vector would do), and taking a moment every now and then to go over your
product with a profiler and fix the biggest bottlenecks early, can save you a
ton of money in cloud bills (it might even turn out that your product actually
doesn't need to horizontally scale _at all_ , giving you further reduction-of-
complexity benefits). Or you might end up delivering features that were
impossible to do with bad performance (see e.g.
[https://news.ycombinator.com/item?id=22712103](https://news.ycombinator.com/item?id=22712103)).

