
Paul Graham inspired the creation of Redis - oneowl
https://twitter.com/antirez/status/1110468354542919681
======
antirez
I want to clarify a couple of things. I'm not saying that Paul Graham invented
this pattern. Actually after he mentioned it, I remembered a friend of my
father to implement exactly that in QUICKBASIC in the late 80s :-) The point
is that maybe the Redis design was already inside me, but I needed a trigger:
I often think of good things after being triggered. And smart people are more
likely to tell about good ideas, old and new. That was the point. Similarly I
believe there are a lot of simple fundamental ideas that can be re-applied to
today's technology, as the landscape changes many things become relevant
again.

~~~
michaelmcmillan
I'll just take the opportunity to say how grateful I am that the idea of
inventing Redis struck you -– regardless of how it originated. I use it all
the time, both professionally and in my free time. Awesome piece of software.
An idea is worthless by itself, execution is everything.

~~~
markbnj
Very much this. Redis is pretty much the swiss army knife of persistence
around our shop.

~~~
technics256
I've been meaning to learn redis for quite some time. Do you have favorite
resources for this? Thank you.

~~~
markbnj
Just the documentation to be honest. The basic functions of redis are quite
simple to learn and use either using the redis-cli client or a language
binding. Basically you PUT keyname value, and GET keyname to retrieve value.
There's a ton of additional features and types of structures but the basic use
of it is as a key/value store.

------
himynameisdom
Man, some of the comments on here are really disheartening. What's the deal
with trying to humble people with long-winded, tangential counterpoints,
gotchas, and condescending questions? Constantly proving one's intellect seems
to be a prevailing theme on HN and I'm not seeing how it adds to the quality
of the content. I'm sure someone will unearth the irony in this and let me
know soon enough.

~~~
coleifer
Best not to read the comments. There's literally nothing you can do about the
phenomenon you're describing.

------
DonHopkins
That's exactly how HyperCard works, and we all know what inspired HyperCard!

[https://www.mondo2000.com/2018/06/18/the-inspiration-for-
hyp...](https://www.mondo2000.com/2018/06/18/the-inspiration-for-hypercard/)

[https://twit.tv/shows/triangulation/episodes/247?autostart=f...](https://twit.tv/shows/triangulation/episodes/247?autostart=false)

------
michaelmcmillan
Hmm, I thought this pattern was really common? That is, appending everything
to a file and reading back from the file when there's a reboot. I constantly
use it when a database (or redis for that matter) is simply overkill for my
use case.

Here's a 34 line implementation I use on a node production system. It writes
struct-like JavaScript objects that represents events to disk. When reading it
back I do a fold (or .reduce) to build the state.

And yes –– it could be way smarter (writing to memory _and_ disk), but YAGNI
has been working out pretty well so far.

    
    
      class EventStore {
        constructor(file) {
          this.file = file;
          this.cache = null;
        }
      
        async appendEvent(event) {
          // Purge the cache for entries since we mutated the store.
          this.cache = null;
          return new Promise((resolve, reject) => {
            createWriteStream(this.file, { flags: 'a' })
              .on('error', reject)
              .on('close', resolve)
              .end(`${JSON.stringify(event)}\n`);
          });
        }
    
        async readEvents() {
          if (this.cache !== null) {
            return this.cache;
          }
    
          try {
            const data = await readFile(this.file, 'utf-8');
            const lines = data.split('\n').filter(line => line);
            const events = lines.map(line => JSON.parse(line));
            this.cache = events;
            return events;
          } catch (error) {
            return [];
          }
        }
      }

~~~
jrockway
It's pretty common. All the wonderful proprietary file formats from the 90s
(and probably before, and probably after) basically boil down to writing raw C
structs to disk.

You can try it yourself... mmap a file, memcpy some structs there, do the
reverse ... and enjoy!

(Obviously depending on the memory layout of one C compiler on one
architecture does not make for portable files. But that was never a design
goal of this system.)

~~~
beat
I spent part of the mid-1990s hacking a proprietary data streaming protocol
written in lexx and yacc. It was what we had back then.

------
lucideer
To offer a slightly alternative perpective on this, I actually think this type
of "article"/("listicle"/"tweetacle"?) can have negative effects. In my mind,
it lends credence to the very toxic notion of "value of ideas" over "value of
execution".

The former is something that has all sorts of knock-on effects: backward IP
laws "protecting" ideas, perverse incentives within large corporations with
outspoken "idea men" being promoted ahead of doers, non-technical founders
with "high-potential ideas" sucking up investment and expecting to execute
with technical hires on untested theories.

I'm not saying any of the above applies in this case of course, but the fact
is that it is you who built Redis, not pg, nor many others who've had similar
ideas, and I think writing the above tweet thread lends undue weight to many
of above negative trends in our industry (and also in general in recorded
history of IP/invention-credit battles).

~~~
dragonwriter
> In my mind, it lends credence to the very toxic notion of "value of ideas"
> over "value of execution".

Ideas are required for execution; they don't have independent value (idea
without execution delivers nothing), but neither does execution (you have
tohave something to execute.)

~~~
lucideer
Of course. I didn't say otherwise: what I'm talking about is value. Ideas are
often (usually?) assigned greater value than execution itself, which is
absurd.

In actual fact, even beyond "inception", most execution requires ongoing
iteration and innovation. No final product is solely the result of its
inspiration.

~~~
qaq
We might have biased opinion on the subject at least those of us heavily
involved on execution side. Execution is much more subject to available
resources.

------
userSumo
The comment should probably be still around right? Does anyone have the link?

~~~
tim333
Maybe
[https://news.ycombinator.com/item?id=14605](https://news.ycombinator.com/item?id=14605)
?

~~~
adlpz
Here's the one, I believe. Linked to parent for context:

[https://news.ycombinator.com/item?id=14754](https://news.ycombinator.com/item?id=14754)

PS: They're basically the same, really, just adding another candidate to
yours.

~~~
tim333
Wonder if HN still has all 20 million comments in a hash table?

------
chrchr
“System prevalence[1] is a simple software architectural pattern that combines
system images (snapshots) and transaction journaling to provide speed,
performance scalability, transparent persistence and transparent live
mirroring of computer system state.“ —
[https://en.m.wikipedia.org/wiki/System_prevalence](https://en.m.wikipedia.org/wiki/System_prevalence)

------
vshan
PG was also involved in the inception of Reddit: It was PG who gave Alexis and
Steve the idea to make something like reddit, and also gave them the tagline
"the front page of the internet".[0] PG had vetoed their initial idea to
create a food-delivery app and then called them back and asked them to come up
with something new. [0]:
[https://www.youtube.com/watch?v=5rZ8f3Bx6Po](https://www.youtube.com/watch?v=5rZ8f3Bx6Po)

------
quadcore
Actually, I wonder why we dont have yet a programming language and runtime
which, after a shutdown, reload exactly like it was.

~~~
tonyarkles
If I recall correctly, Emacs currently does this. During the build it (slowly)
loads all of the elisp into memory and then dumps out the memory image of it
after it’s all been compiled.

I’m a bit hazy on the details, but I think it involves calling unexec().
[https://lwn.net/Articles/673724/](https://lwn.net/Articles/673724/)

Edit: just noticed a sibling comment mentioned this too...

~~~
Nullabillity
Only for built-in elisp, as part of the build process. User configuration
(including packages) is reloaded from scratch on each start.

------
kukabynd
Paul Graham inspired many people. His Hackers & Painters is something I enjoy
going through now and then.

~~~
dredmorbius
Counterpoint:

[https://idlewords.com/2005/04/dabblers_and_blowhards.htm](https://idlewords.com/2005/04/dabblers_and_blowhards.htm)

~~~
jimbokun
Those footnotes are pretty funny.

------
stretchwithme
Doesn't AWS's Aurora database sort of work that way too?

~~~
bpicolo
The pattern in general is used by more or less every database in the form of
write-ahead logging.

[https://en.wikipedia.org/wiki/Write-
ahead_logging](https://en.wikipedia.org/wiki/Write-ahead_logging)

------
konamicode
I'd be keen to see a simple example of this pattern in Lisp (or another
language). Does anybody have a good link?

~~~
pavlov
I use it by default for new Node.js projects (most of which are experiments
but some end up in production, and have been running without problems for
years).

It's a simple pattern to implement. To make it a bit easier to use repeatedly,
I've got a small helper class called JournaledCollection. You pass it
serialize+deserialize callbacks for your item type, and it takes care of
persistence in event logs.

For a while I was thinking about releasing my helpers as a project called
LAUF, short for "Lame-Ass Un-Framework". Then one could say: "Most of my
projects are LAUFable, I don't need anything more serious." (Awful dad jokes
are a solid reason to publish open source, right?)

Never got around to it though, but if you're interested, I could put together
an example.

~~~
chrisweekly
I like the name -- and it sounds worth sharing!

------
tosh
Redis is a great example for unbundling a pattern to a lib/service/business.

------
viach
Probably the next logical idea would be to apply to YC with Redis?

------
huahe
PG inspires as usual!

------
techie128
_When the application would be restarted, reading back the log would recreate
the in memory data structures. I thought that it was cool, and that databases
themselves could be that way instead of using the programming language data
structures without a networked API._

That is literally how databases work. In Memory + WAL + Data Files on disk.
You could, in theory, live without the Data Files and just a big WAL.

~~~
kijin
Except that most databases don't store their data in anything resembling
"programming language data structures". You get tables, rows, and columns (or
maybe a bit of JSON if you're lucky) instead of native integers, strings,
lists, sets, and dictionaries.

~~~
threeseed
Object and document databases have been around for decades.

Likewise ORMs which allow for higher order types etc. have been around since
WebObjects i.e. also decades.

~~~
kijin
The primary purpose of an ORM is to overcome the "impedance mismatch" between
relational databases and programming language data structures. There's no need
for an ORM if you can store your data structures directly in the database.

~~~
mamcx
That is an angle.

Other way to say it is that ORM is a workaround to the fact most languages are
VERY poor at manipulate data.

Exist 2 main reasons for the "impedance mismatch":

\- Paradigms. 2 different paradigms will be at odds. Example: Functional and
OO. This is ok.

\- Limitations: The relational model is _absolutely_ superior and more
expressive at manipulate data than OO/Functional. You need A LOT of machinery
to recover that power.

This is not ok.

However, this not change the fact that OO is ok.Similar how a KV store is
fine, but certainly, a RDBMS store is much more capable.

~~~
heavenlyblue
How is relational more powerful than OO/Functional if you can’t define an
infinite dataset with it?

~~~
mamcx
You can't?

Most people only see the relational model as is inside a RDBMS. That is how
judge the OO for what you can do on Mongo.

RDBMS have some weird and well considered restrictions for their case.

But read a little about the relational model, and none of it depend on SQL or
say anything about how is the storage.

P.D: Is important to note for where is MORE powerful. Remember that the
creator of Pascal say:

[https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures...](https://en.wikipedia.org/wiki/Algorithms_%2B_Data_Structures_%3D_Programs)
Algorithms + Data Structures = Programs

You can say OO/Functional lean more to the "Algorithms" side but RM to the
"Data Structures". OO/Functional not say much how operate on data, most is a
exercise for the reader.

Instead, RM give a clear answer and defined operations for that.

You need to "spice up" things to make the one be more useful to the other part
of the equation. RM, too pure, certainly is incredible limited, not even can
print "hello world!", but that is just too give a solution about how transform
data...

------
threeseed
Strange title.

Does merely recalling a language pattern really count as inspiring ?

~~~
tomhoward
In the tweet, the author of Redis says "[Paul Graham] ... inspired the
creation of Redis". So it probably counts!

~~~
dang
Ok, we put that in the title above for disambiguity.

------
andy_ppp
Of course Erlang has had ETS (and of course you can use gen_servers of various
kinds for this) built in forever. Redis is fantastic but I think there have
been many examples of prior art before this tweet!

~~~
pushpop
Lots of prior art but the tweet didn’t say PG inspired the paradigm; rather
just Redis specifically.

Anecdotally, I’ve used plenty of in memory DBs (and written some too) but
Redis has been by far my personal favourite.

------
jokoon
Is this really a brilliant idea though?

Everybody knows RAM became cheaper cheaper, while mechanical disk can't be
made faster, and SSD have reliability limits.

It seems entirely logical to expect databases to work on RAM first and then
commit on disk for a large performance improvement.

~~~
antirez
When I started Redis people were like FTW data in memory?!

~~~
jokoon
I was talking to a database teacher, and I tried arguing about ACID on an in
memory database, arguing that there was a minimal window for database
corruption in a transaction system, since the worst case scenario would seem
to be a loss of very few transaction.

He was not really listening to what I was saying or my arguments, because a
system like redis seems like a more than acceptable compromise.

It still seems a few people are reluctant to an in-memory database.

~~~
benmmurphy
VoltDB is a probably a better example of an in-memory database with ACID
semantics. Redis usually won't be deployed in a way where it fsyncs after each
write operation because it runs too slow. You need to do some tricks like
batching fsyncs to get decent performance and I don't think Redis has support
for this. However, SSD fsync performance seems very high. I remember
benchmarking an EC2 i3 instance that was giving 20000 fsync/s but if redis is
giving you like 80k writes/s then even fsync this fast is going to be a
bottleneck.
[[https://redis.io/topics/benchmarks](https://redis.io/topics/benchmarks)]

I think people who are deploying redis are willing to tolerate lower
durability guarantees for the extra performance. A lot of the time redis is
some kind of cache and there is a way of reconstructing the real data from
more durable storage in the case of failure. Or people are willing to lose a
second of data or whatever fsync interval people are using.

