
Big O Notation – Using not-boring math to measure code’s efficiency - rspivak
https://www.interviewcake.com/article/python/big-o-notation-time-and-space-complexity
======
abalone
I've noticed there are SO many of these "coding interview prep" courses
lately. Like, obviously it's a hot job market, but there's just so many and of
a certain vibe that it seems like some are selling a dream.

There's a common narrative too, that it's all about algorithm question prep
and it's a "game" you can win. One youtuber who recently landed a ~$275K TC
gig said it's not about being "intelligent" it's about practice and rapid
pattern recognition (he also was pitching his own coding interview prep
course). Said he forgot half of it after getting the job.

This could all be true but this narrative is coming from sources that have a
financial interest in making you believe that. What do you think HN? I mean I
know algorithm prep is an important part of the interview but.. there's just
something sketch about all these outfits I can't quite put my finger on.

(I also wonder if the people who get hired based on prep vs. aptitude then
have a kind of survivor bias and hire others who are strong on this limited
range of pattern recognition... could it be a problem for the industry?)

~~~
dntrkv
It depends on the company, but most of the FAANG company (and others that try
to imitate them) interviews come down to how quickly you can get the optimal
solution on the board. It has very little to do with how well you can code day
to day.

Most interviews have 4 sessions broken up in the following manner:

1 hour: coding

1 hour: coding

1 hour: lunch

1 hour: system design

1 hour: cross-functional

In my experience though, the first two are the ones that really make or break
candidates, if you don't pass both, you're not getting hired. And yes, most of
those interviews just test your knowledge of how quickly you can figure out
the right data structure and algo for the question at hand. And even if you
know all of the data structures and the applications of each, it can be
incredibly difficult to solve the problems in the allotted time.

The other funny thing is that your previous experience doesn't do anything for
you other than maybe getting to skip the technical phone screen. Everyone goes
through the same process. e.g.
[https://twitter.com/mxcl/status/608682016205344768?lang=en](https://twitter.com/mxcl/status/608682016205344768?lang=en)

On the bright side, it does look like many companies are now realizing this
isn't a great way to evaluate someone's ability and are beginning to
incorporate more practical exercises.

~~~
spraak
I wish the usage of FAANG would die. Firstly it exlcudes obvious others like
Uber, Microsoft, etc. Secondly, and purely aesthetically, it sounds silly. I
hereby propose we call such companies the Big N, and I'm definitely open to
other ideas.

~~~
JudgeWapner
More generally, why do we have such a fetish for acronyms in general? Is it
some kind of insider feeling to know a "code" for a thing?

~~~
nothrabannosir
Pattern recognition and aversion to repetitive behaviour on a micro level.
Ironically oblivious to the repetition on a macro level that is most of our
entire lives, but that’s another story :)

------
curiousgal
> _not-boring math_

What's next? Gradient decent with no gradients? Fourrier transform with no
functions?

If math is do boring just skip the whole thing, don't try to kid yourself by
saying you're not doing it when you are in fact doing it.

/rant

~~~
thiht
What's with this gatekeeping attitude?

Big-O notation is a useful tool in a programmer's belt, knowing the math
behind it in details is less useful. What's the problem? It's the same thing
as not needing to know how an engine works to be able to use a car

~~~
curiousgal
That is not my point.

Big-O notation _is_ a mathematical notation. By using it, you _are_ using
math. It's the same as using a car and saying you are not using a car.

~~~
Toboe
The post isn‘t saying that Big-O isn't math, it is saying Big-O is "not-
boring"(->interesting/exciting/whatever) math.

------
ajmarcic
This article misses an actual definition of Big O.

This forces us to skip crucial points like why O(n^2) truly identically
_equals_ O(n^2/2 + 5n). The author instead seems to think they are
approximately the same because the lower order terms are small compared to the
highest one.

We only need to explain the definition and the reader could understand why an
O(n) algorithm is in O(n^2), why some O(n^3) algorithms are much faster than
others in O(n^~2.4), etc.

~~~
devnulloverflow
> The author instead seems to think they are approximately the same because
> the lower order terms are small compared to the highest one.

In fairness, this is actually why we care _in practice_ about asymptotic
issues. The fact that there is a rigorous mathematical viewpoint that makes
them strictly comparable is helpful for being confident about the non-trivial
results -- but the reason we care about it is because we want to know roughly
how expensive our code is (or will be).

------
falcor84
> As n gets really big, adding 100 or dividing by 2 has a decreasingly
> significant effect.

I was annoyed to see that comment about division, as of course dividing n by 2
has exactly the same effect regardless of the size of n. This is not the
reason we ignore multiplicative constants in complexity analysis.

It reminded me of the joke that Earth's circumference is pretty much the same
as its diameter, since the the size of pi is negligible at that scale.

~~~
concert-gilled
> This is not the reason we ignore multiplicative constants in complexity
> analysis.

What is the reason?

~~~
Jaxan
One possible answer is that any algorithm can be sped up linearly:
[https://en.m.wikipedia.org/wiki/Linear_speedup_theorem](https://en.m.wikipedia.org/wiki/Linear_speedup_theorem)
. (At least theoretically, for runtimes bigger than linear.)

Another answer could be Moore's law: machines do get faster over time. (And
also memory gets cheaper.) And so we wish to define efficiency (time or space)
in a way which does not depend on the current technological state.

~~~
dooglius
That theorem just indicates that Turing Machines with an unbounded/arbitrary
number of symbols is a poor model; the approach there will not work on any
actual machine. If you take out a factor of 2 in an algorithm, that's going to
asymptotically double the state regardless of what actual machine you're on,
so I don't buy machine-independence as a reasonable excuse. The actual reason
constants are ignored is because it makes the math a lot simpler when you
ignore them.

------
blattimwind
Except Big O Notation is exactly not measuring code's efficiency.

~~~
adrianN
That depends on your definition of efficiency.

------
Blackthorn
You're an engineer, aren't you? Just do the damn "boring" math.

------
raxxorrax
> Big O notation is [...] about how long an algorithm takes to run.

My former prof would have me spanked for an imprecise statement like this.

------
gameguy43
Original author here. Thanks for the post! Would stick around to answer
questions but I'm on vacation in Japan right now!

We're doing a poor job with navigation on the site ATM, so here are some "you
might also like" hits for you:

A piece where we derive most of the main data structures step by step,
starting with naked bits in RAM: \- [https://www.interviewcake.com/data-
structures-and-algorithms...](https://www.interviewcake.com/data-structures-
and-algorithms-guide)

A reference / cheat sheet for the main data structures: \-
[https://www.interviewcake.com/data-structures-
reference](https://www.interviewcake.com/data-structures-reference)

A reference / cheat sheet for sorting algorithms: \-
[https://www.interviewcake.com/sorting-algorithm-cheat-
sheet](https://www.interviewcake.com/sorting-algorithm-cheat-sheet)

------
kgilpin
It's true that most engineers are not going to implement a binary search
algorithm or an optimizing compiler during the course of their employment in a
typical year.

However, CS fundamentals are the language of software engineering. It's
important for engineers to be able to communicate efficiently about things
like runtime analysis, indexes, concurrency models, type systems, etc.

Your mechanic knows how a carburetor works, (s)he knows how brakes work.
Mechanics have a common set of fundamentals that they all know and share, and
can use to communicate with each other about their work. The purpose of this
knowledge is not to be able to fabricate new car parts, although given time
and the right equipment, it could probably be done.

The purpose of CS fundamentals is not to so much to implement complex
algorithms, as to understand how to use tools and libraries that are based on
them.

------
iandanforth
Question: At what point in a career do you expect someone to have gone from "I
know Big O" to "I know how this data structure is laid out in memory on this
OS?"

My experience says really good engineers know the latter, but I'd like to hear
if this holds for others.

~~~
bauerd
How data structures are laid out in memory has nothing to do with the
operating system

~~~
blattimwind
OS typically dictates ABI, ABI dictates memory layout of structures, among
other things.

------
MichaelBurge

        def print_first_item(items):
            print items[0]
    
        This function runs in O(1) time (or "constant time") relative to its input. 
    

If this were C where the only types are fixed-size, that might be true. But in
Python it seems like items[0] could display as 10 gigabytes of JSON.

And if stdout is piped to a process that isn't consuming its input, it might
never terminate naturally.

Also, since Python has arbitrarily large integers, an expression like "index
+= 1" probably runs in O(log(index)) time, not O(1) time.

~~~
dpzmick
in C, it items[0] could segfault, the fault could be caught by a signal
handler which does some unbounded amount of work, then populates the
appropriate memory location and returns...

More realistically, the page of memory that items[0] is sitting on could be
swapped to disk, requiring the OS to do some big operation.

trying to write a piece of code with the big-O you are looking for generally
has a huge number of caveats. I'm undecided on whether this is a feature
(abstraction) or a flaw (easily misleading)

~~~
GuB-42
And "print items[0]" is actually O(n) where n is the size of the string. Maybe
even O(n^2) if the item is a big number and a binary to decimal conversion is
required.

But it is doesn't matter here because we have chosen n to be the size of the
array, because it is what we are studying.

Here it is big-O "constant time" because the execution time doesn't depend on
on the size of the array, assuming it is really big. Not that every call of
the function will take the same time to run.

The reality is much more complicated, with things like caches to take into
account. As a result, complexities under O(n) often don't mean much in
practice. But the higher you go, the more important it becomes. As you get to
O(n^2) it is time to take a hard look at your algorithms, or make really sure
that n is small and stays small.

------
sireat
Am I the only one who signed up for interviewcake on a whim (similar to those
January fitness club resolutions)?

Then I sort of had an epiphany. When will I need this? (or rather "I am too
old for this...") I must be the perfect customer for interview cake: paid but
do not use it.

I make my own work and those kind of problems just do not come up.

I mean CLRS was fun a long time ago, but it seems to degenerate into skeet
shooting pretty quickly:

“Shooting skeet eight hours a month was excellent training for them. It
trained them to shoot skeet.”

------
smudgymcscmudge
I have a question for those with a better understanding of such things. Is it
difficult to calculate time or memory complexity of a function through static
analysis?

It seems kind of basic, but I still find myself manually checking code instead
of relying on a tool.

~~~
zuza
I would argue it isn't tractable.

f(n) = f(n-1) + f(n-1) has runtime O(2^n); f(n) = f(n-1) + f(n-2) has runtmie
O(~1.62^n); f(n) = f(n/2) + f(n/2) has O(n log n); for i = 1 to f(n) has
runtime f(n).. good luck determining a function output through static analysis

~~~
gizmo686
In both cases it has O(2^n). Normally speaking, this response would be pure
pedantry [0], but for static analysis it might be good enough. In most cases,
the naive answer will be good enough, and the tool can direct you to pay
attention to areas of particular concern.

Getting static guarantees about performance would require carefully
constructing your code with the tool in mind.

[0] Although I still find it outrageous that CS departments describe their
algorithms class as "math" when they do not accept this answer.

~~~
SamReidHughes
They’re asking for the tightest upper bound.

