
Algorithms Are Now Commodities - EvgeniyZh
http://shape-of-code.coding-guidelines.com/2020/07/05/algorithms-are-now-commodities/
======
OliverJones
This old dev has been around to witness the transition from in-depth algorithm
development to algorithms as commodities. I've pounded out my share of B-tree
search, polygon intersection, and numeric stuff. Try doing numeric stuff
reliably and quickly on a tiny computer without a Floating Point Unit and
you'll know why you might need to learn, and then program, Jack Bresenham's
line-drawing and circle-drawing algorithms.

And there's Newton and Raphson's sqrt(). Hint: if you understand your data and
your precision requirement, you can avoid iterating it all the way out to the
end of your integer precision, and get a lot more work done.

Try transposing a 4-megabyte matrix before the week is out on a machine with
32KiB of RAM and two 5-megabyte hard drives. Yes, it can be done.

When I started doing this work it took deep understanding to get anything
done. Now it takes the skills of a good reference librarian. We have to be
able to find stuff (via npm, nuget, maven) quickly, assess its quality, and
figure out how to use it. So the type of thinking we need has changed from
"deep" to "broad".

~~~
chrisseaton
> Try transposing a 4-megabyte matrix before the week is out on a machine with
> 32KiB of RAM and two 5-megabyte hard drives.

Transposing is just copying bytes from one place to another, isn't it? There's
no maths involved, is there?

To have that run in a week with the most naive approach possible you'd only
need a machine that can copy seven words a second. Are there real relevant
machines that can't do that? What algorithms could help you anyway? Something
to do with cache and the order in which you read and write? And you've got
plenty of space as you can trivially do it in-place.

~~~
smallnamespace
The RAM here is essentially a cache. You’ll want to transpose a block at a
time, something like 150 x 150 blocks, so it all fits.

Each block is not contiguous on disk, so you’ll want to load blocks in the
right order to avoid extra seeks.

~~~
chrisseaton
I suspect, since the poster mentioned having _two_ hard drives, that really
they're confusing transposition with inversion. These aren't the same things.

------
nirui
> why waste time implementing the ‘low’ level stuff when there were plenty of
> other problems waiting to be implemented.

Because:

\- The package is licensed through GPL

\- The lib allocates memory in such way that it messed up my program

\- The last update to the lib is 3 years ago, and it takes at least another 3
years for the author of the package get out of jail which is the min
punishment of some NSL

\- I don't like how it's interface is designed

\- Because I got paid even I write everything by myself

The list goes on.

Another way to look at this is, "Do I really want just to implement a Linked-
list? Or an sizable & queueable list that allow me to sort items via LRU?".
Sometime, you just have to to write some "low level stuff" by yourself to make
it exactly as you wanted.

------
scoutt
> There are companies where algorithms are not commodities

And there is an entire industry! Embedded (but not the Raspberry-pi kind of
embedded that is similar to desktop).

Try to do a dead reckoning system, or anything related to real-time
gesture/motion analysis. Try reading, processing, mixing and outputting audio
in real time. Or any DSP related stuff. What about FPGA, ASICs, etc.?

Why oh why is the embedded industry so easily forgotten? It's not all cloud,
food apps, SaaS and Electron, ya know?

Edit to add: all the above with the limited resources of embedded, that forces
you to improve your algorithms.

~~~
vanderZwan
> _Why oh why is the embedded industry so easily forgotten? It 's not all
> cloud, food apps, SaaS and Electron, ya know?_

Because the _other_ industry is inherently linked to the internet. It's like
the media telling you that the media is very important, or your brain telling
you that it's the most interesting organ in your body (I forget which comedian
I stole that from).

------
enriquto
> I date the age of the Algorithm from roughly the 1960s to the late 1980s.
> During the age of the Algorithms, developers spent a lot of time figuring
> out the best algorithm to use and writing code to implement algorithms.

Today there's orders of magnitude _more_ developers spending time figuring out
algorithms and writing core to implement them than in the 80s. The difference,
maybe, is that even more developers today do not write any algorithm at all.
But if there's an age of the algorithm, it is today. At no point in human
history have as many algorithms been written as today. And next week, many
more will be written!

~~~
dtech
You're missing the point. There is a larger absolute number of people working
on complex algorithms, but as a percentage it is much smaller. In the 60's
deep algorithmic knowledge was required for 90%+ of programming tasks, now it
is probably closer to 1%. It is perfectly possible to make a living as a
programmer nowadays without being able to inverse a binary tree

~~~
oriolid
What's the obsession with inverting a binary tree? I have never found an use
case that particular algorithm, but on the other hand I would be very happy if
more programmers were aware that having a where clause in sql statement (and
maybe and index) is more efficient than selecting entire database and looping
over the results, even if the end result is the same.

I understand that in order to come up with the inversion on the fly you need
to know about trees and recursion, but is it really that scary?

~~~
arethuza
Possibly due to the tweet by Max Howell:

 _" Google: 90% of our engineers use the software you wrote (Homebrew), but
you can’t invert a binary tree on a whiteboard so fuck off."_

~~~
oriolid
Probably unrelated, but every time I update Homebrew, I get the feeling that
there's something that should be O(log N) or O(n) but the implementation takes
O(N), O(N^2) or worse. There's no way checking a small list of installed
package versions against a large list of available versions should take that
long.

~~~
arethuza
I noted in an another discussion on HN that dependency checking is apparently
NP-hard:

[https://stackoverflow.com/questions/28099683/algorithm-
for-d...](https://stackoverflow.com/questions/28099683/algorithm-for-
dependency-resolution)

~~~
oriolid
I doubt it's just this, since on my laptop just brew list from warm cache with
102 packages takes about a second, and twice as long if files must be read
from SSD. That's a lot of reading and computation for list of 102 packages.

------
_bxg1
I remember reading an article a while back about how professional photography
has become a commodity. You can find and license any imaginable photograph,
for pennies, in minutes. From the best photographers in the world. So from a
business perspective, photography as a pure art has been devalued to almost
nothing. Nobody pays you to take the very best possible photo.

So why do people still hire photographers? To put _themselves or their
product_ into a photo. Most photographers are no longer paid to take the very
best photos, but to make their photos specific to what the particular client
cares about. That's something that can't be commoditized.

I think that's where software is now as an industry. Very few of us these days
get to innovate at a "pure art" level. Instead, we get paid to pull together a
bespoke solution that's sensitive to a given business and its needs. It isn't
novel at the macro-level, only at the micro-level.

This can be kind of depressing, but it's also a little comforting that those
micro-level solutions are so hard to fully commoditize. As long as the real
world is messy and varied, the leaves in the software tree will have to be
too, and there will be work that needs doing.

~~~
ForHackernews
I don't see how that's depressing at all. Another way to phrase what you're
saying is "Developers get paid to write software that solves problems people
actually need solved.

~~~
_bxg1
For many of us, writing something pure and generic and beautiful is much more
gratifying. And it's depressing to think that we'll probably never write
something like that that's good enough to be picked over the version that was
developed at Google and then open-sourced.

~~~
majormajor
That's not unique to software, or the modern world at all, though.

Most people have never created the best-in-the-world implementation of
anything. The math just doesn't work that way. Only one can be the best. We
need to stop turning such unobtainable things into goals.

~~~
_bxg1
That's not what I'm saying at all. Of course very few people make "the best"
of anything. What I'm saying is that the internet, and in our case open-source
software, have caused "the best" to be _all that matters_ , because it's
accessible to everyone. There's no longer a practical reason for most people
to engineer most (generic) things themselves because the better solution isn't
hidden behind a license, or proprietary to an organization, or inaccessible
due to physical media. People used to _buy_ things like linkers. Today, the
state-of-the-art is nearly always at your fingertips, for free, in seconds.

I'm not saying this is a bad thing. Progress always makes certain endeavors
obsolete over time. You don't see job postings for telegraph operators any
more because it's no longer useful. But if a person _enjoys_ one of those
endeavors for its own sake, it's bittersweet to see it relegated to a hobby.

------
jvanderbot
Well that's just not true.

I concur: Nowadays, the basic building blocks are done. Good sorting, database
indexing, O(1) set-membership, many variants of lists, hash tables, and other
higher level data structures are well covered by modern libraries. The
foundations of modern system design are algorithms and data structures, and
they are largely commoditized, _for general use_. Certainly, working on
general purpose, clever-but-foundational algorithms is becomming increasingly
niche.

I disagree with the final conclusion that somehow algorithms have faded into
obscurity.

Anyone writing any program has to consistently track their expected program
performance as it shuffles data between commodotized library calls or between
network end points. This is algorithm design. Many "foundational" algorithms
like max-flow actually make use of "foundational(er)" algorithms and shuffle
data between them.

This is what the modern programmer does when he decomposes a data-intensive
problem into sub-problems and uses commoditized libraries and re-composes
those solutions.

The push to catalog and archive a wide array of well-implemented algorithms
and data structures tracked the emergence of the personal and commodity
hardware. The emergence of new hardware will require new solutions. Mobile,
webassembly (perhaps), ASIC-based data centers, CUDA, are all exciting new
areas where our "foundational" algorithms and canonical solutions need to be
revisited, redesigned, and re-tuned. Or at least re-composed.

------
nickcw
I spent a lot of time reading Knuth's books, especially Seminumerical
Algorithms when I was implementing multi precision arithmetic for various
architectures. What amazing pieces of work they are. That was back in the time
when you could look at assembler (or even C code) and get a good idea of how
many cycles it would take.

I love the Art of the Algorithm, but I have to say I'm very happy to use off
the shelf libraries now. I've implemented binary search dozens of times and
each time spent ages fixing all the corner cases. Instructive, but a waste of
time when you can import a library which is battle tested and faster than
anything you'll ever write.

Algorithms are dead - long live the (commoditized) Algorithm!

~~~
chrchang523
Incidentally, I think there is at least one important area where Knuth is
actually still ahead of off-the-shelf libraries as of 2020: "broadword
computing". When skimming volume 4A for semi-recreational purposes a few years
ago, I was surprised to find multiple "bit-hacks" which were better than
anything I had found in other sources, and proceeded to use them in my own
open-sourced work. Every time I've encountered a situation where bitvectors
are useful, I have benefited from rolling my own implementation.

~~~
klyrs
Came here to say this. It's rare that you want to consume these things through
an abstraction layer: the layer is often heavier than the work underneath.
Sucks for readability, though, so I usually spend as much time documenting as
implementing.

------
jasode
_> This was the age of the algorithm. [...] When it comes to algorithm
implementation, developers are now spoilt for choice; why waste time
implementing the ‘low’ level stuff when there were plenty of other problems
waiting to be implemented. Algorithms are now like the bolts in a bridge: very
important, but nobody talks about them. Today developers talk about story
points, features, business logic, etc. [...] Today, we are in the age of the
ecosystem._

The blog author doesn't make it explicit but his usage of _" algorithm"_ is
actually about "low-level algorithms" which is how it seems to support his
claim that _" algorithms are now commodities"_.

But "algorithms" in the general sense are not commodities. It's just that
we're now free to concentrate on higher-level problems with new algorithms.

E.g., in the 1980s & 1990s, the Barnes & Noble section for computers would
have books with algorithms showing how to write disk-based B-trees from
scratch. But now with SQLite library in wide usage, you can just use their
B-trees instead of reinventing your own. Some more examples of low-level
algorithms I used to write by hand:

\- manually uppercase chars by adding "32" to the ASCII code because there was
no Upper() function in a standard library.

\- manually titlecase/propercase/lowercase a person's name from "JOHN DOE" to
"John Doe" by looping the the string char-by-char and subtracting "32" from
the ASCII code if it doesn't follows a space char because there was no
TitleCase() function

\- adding a drop shadow beneath a window because the operating system GUI
didn't render shadows for you

\- manually writing link lists to create dynamic size memory buffers because
there was no C++ STL library yet or built-in associative arrays in Python/C#

I don't do have to do any of that low-level tedious work anymore _but I still
write new algorithms_ because there is always some _[missing functionality]_
where a standard library doesn't exist or a consensus implementation hasn't
yet "won" in the marketplace of ideas. It could be distributed
cloud/desk/mobile data sync algorithm, or collaborative recommendation
algorithm, etc. Algorithms (general sense) will not be commoditized for
decades -- if ever.

~~~
raphlinus
I have a feeling you might enjoy this code:
[https://github.com/zbraniecki/tinystr/blob/master/src/tinyst...](https://github.com/zbraniecki/tinystr/blob/master/src/tinystr8.rs#L221)

I wrote this because I wanted (normalizing) locale comparisons to be super
fast, as they can be part of the key for cache lookups of text layout. In a
language identifier such as "en-Latn-US" the script is in title case.

So this kind of thing still does happen, just at the lowest levels of
libraries that provide nice clean abstractions for other people to use.

------
sdenton4
One of my favorite mini games at my job is rewriting classic algorithms to run
in batched mode on gpu/tpu. The speed improvements often improve model
training time by days, and it's always a lovely intellectual challenge. (The
basic challenge is to rewrite the algorithms in terms of matrix operations
which operate on many examples of the problem at once, while eliminating all
branching.)

------
tanilama
I am somewhat confused about the definition of Algorithms under the article's
context.

What is this Algorithm we are talking about? Is it about basic data
structures? If that is what we talk about, I feel at least from like 15 years
ago, people already giving up writing those things from scratch for new
projects.

I agree encapsulation is growing. As developers we are in a reality that we
lose more and more predictability over the behavior of our code.

But that seems to create more demand for nuanced solution to reign the
complexity, since commoditized algorithm created this problem, not solving it.

------
Const-me
I don’t remember last time I implemented sorting, search or trees, these are
available in all languages and runtimes.

Skill, I implement a lot of my own algorithms. Following reasons.

1\. Performance. Custom algorithms and data structures is often a requirement
to write performant code, because SIMD, memory hierarchy, and other modern
hardware-related shenanigans largely ignored by authors of standard libraries.

2\. GPGPU. Due to their massively parallel nature they need completely
different algorithms, a naïve implementation is often by an order of magnitude
slower than purposely built one, due to things like memory access coalescing
and group shared memory.

3\. Licensing. In my area of work (CAD/CAM/CAE) some good quality libraries
are only available as GPL or even AGPL. Apparently, the commercial licenses
are targeted towards companies like GM or Airbus, way too expensive.

------
jcahill
Commodities are fully fungible goods and services.

Different implementations of a single algorithm should have the same basic
formal properties, by definition.

That is, they should be roughly fungible. Else you aren't dealing with correct
implementations of the same algorithm.

If anything, the thesis "algorithms are now commodities" is less true now than
ever before.

As the complexity of software deliverables increases, the use of complex,
proprietary algorithms in software should likely increase in turn.

------
softwaredoug
Yes commodities, but I still need an intuitive understanding of how the
algorithms work to pick the right one. I might even need to read the code when
I get deeper into solving my problem. Or implement a dummy version simply for
learning.

Algorithms are commodities, but algorithm skill is still deeply needed

------
softwaredoug
I wonder if the number of people working on algorithms is about the same
proportion of the workforce, it's just that there's way way more developers
now, and they need commoditized algorithms that solve 90% of the problem (but
that 10% is still out there...).

For example, if you summed up all the 'core committers' of open source
projects, people working on operating systems, embedded software, systems
programming, and other specialized software developers that might think about
unique algorithms, you wonder if that's about the same as the number of total
software developers in the 80s

------
opportune
Personally I found it a bit disappointing moving from academia to industry and
finding out that there is not much of a market for algorithm implementation or
development/research. Sure, there are specialized products where they are
important (DB, OS, languages/libraries, trading systems) and even places for
algorithms in business apps, but it doesn’t seem you can really “specialize”
in it as a career (like distributed systems, system programming, frontend)
unless you’re very picky about which jobs you take.

------
brushfoot
All that's old is new again. The level of abstraction just keeps getting
higher.

Someday, ecosystems like AWS and Azure will be as standardized as programming
languages. You'll still be writing algorithms, but instead of telling one PC
what to do, you'll be telling a distributed system what to do. The old design
patterns will still be relevant; the implementation will just look different.

We're already there to some extent. It's just messy and fragmented right now.

------
omarhaneef
There are probably more algorithms being "figured out" today than ever before.

However, the vast majority are used during interviews, and don't see the dark
of silicon.

------
fogetti
And that's why HackerRank, Codility and all the other coding sites have zero
relevance today in regards of the actual capability of the candidate. The fact
that hiring committees, HR departments and software engineering managers rely
on them goes to show how detached all those people are/have been from reality
for a long time now.

Good software engineering skills on the other hand are pretty much more needed
than ever.

~~~
cyberdrunk
The thing is, in large tech companies like Google, you might be tasked with
designing (or modyfing a design) of something that works on Internet scale.
For such cases, thinking in terms of algorithms is essential. That's why I
think FAANGs still require algorithmic literacy. Unfortunately, other
companies, which don't have FAANG-like problems, ape its recruitment
process...

~~~
fogetti
Yes, algorithms are essential for information technology. But the question is
not that. And the discussion is also not about that.

The question is: are algorithms essential for one's career and for the
majority of business too? I don't think so.

~~~
strikelaserclaw
By definition, any code written to solve a problem is an algorithm (not just
the standard stuff found in textbooks), since a lot of people in IT write code
to solve problems, then yes they absolutely need to understand some what
performance characteristics and the correctness of what they wrote. Even if
you don't need to performance characteristics, you certainly need the
correctness part!

------
oriolid
I have the feeling that the article is really about optimization, but the
author does not know the difference.

