
Redis as a JSON store - dvirsky
https://redislabs.com/blog/redis-as-a-json-store/
======
gtrubetskoy
For the life of me I do not understand the JSON obsession. When I was young
and naive, JSON indeed looked cool because it is both human and computer
readable. Until you run into some of its limitations.

For example, AFAIK there is still no agreement on how to handle binary data,
e.g. what do you do with:

    
    
      >>> json.dumps(["\xb8\xc3\xb6\xbb"])
      Traceback (most recent call last):
      ...
      UnicodeDecodeError: 'utf8' codec can't decode byte 0xb8 in position 0: invalid start byte
    

You can base64-encode it or do a number of other things, but then that's
outside the JSON standard, and suddenly it isn't the flexible encoding
standard it seemed at first...

~~~
maxxxxx
Totally agree. XML is probably over engineered but JSON is under engineered.
No date type, no binaries, no comments, no easy append to existing files. We
jumped from one bad standard format to another bad standard format. Other than
maybe being a little easier on the eye I don't see how JSON is an improvement
over other formats.

~~~
onli
Why would you need any of that? A date type is bloat, store an epoch number.
Comments have no place in a data format. Append is exactly the same as with
XML: You take the part you want and add it at the right place in a list of the
other file. I'd argue the lack of namespaces makes it easier than with XML.

For the binary data: JSON is not a binary format, one should not try to mix
those. To encode it and store it as a string is exactly right. Otherwise a
database as blob store is the better place for that.

~~~
maxxxxx
Comments are important ina format that's being used for configuration as JSON
is. Commenting out things makes for easier revert if something doesn't work.
Having a Line of comment in front of each setting is also a good thing. This
is just basic practicality.

You can't easily append something to an already existing JSON file. You have
to decode the whole thing first and then write it back whole. XML has the same
problem. That's why I said we replaced one bad format with another.

JSON is not an improvement over XML. It replaced one set of weaknesses with
another.

~~~
davexunit
JSON is _not_ for people to write, despite what people try to do. It's for
machines.

~~~
maxxxxx
That's not what I see happening. JSON is being used a lot for things that
should be human readable. If we want formats purely for machines, we should go
binary. Easier to parse and much more efficient.

~~~
davexunit
Binary formats are hard to read. JSON can be pretty printed and read easily.

~~~
maxxxxx
I thought it's for machines? If humans are supposed to read it then comments
would be helpful.

------
macca321
>Since there does not exist a standard for path syntax, ReJSON implements its
own.

Well there's this: JavaScript Object Notation (JSON) Pointer RFC
[https://tools.ietf.org/html/rfc6901](https://tools.ietf.org/html/rfc6901)

~~~
itamarhaber
Thank you, I didn't know about that one... good thing HN has comments :) Do
you know if anybody is actually using RFC6901?

~~~
jkodumal
JSON Patch ([http://jsonpatch.com/](http://jsonpatch.com/)) also uses it.

------
chmike
The graphic type use to show performance is inappropriate because line slopes
have no significant meaning. A bar graph would be more appropriate.

------
floatboth
What's the indexing story? Do it yourself? "Any JSON store implementation will
contain a half-specified bug-ridden implementation of half of Postgres's
GIN/GiST for jsonb"? :D

~~~
traviscj
Wise words -- a good set of indices are the first thing you need for real
problems with realistic data sizes.

~~~
dvirsky
Right now the problem that ReJSON tries to solve is very specific and typical
for Redis users: You save JSON blobs on Redis (usually session data), now you
want to get just a part of a JSON key, or manipulate just one element in a
specific key.

Until now you couldn't do it without loading the entire object into memory.
ReJSON just allows you to manipulate that object or retrieve just a part of it
efficiently. For this use case we do not need and will never need indexing.

------
jsemrau
We use Redis Queue (RQ) as a listener cache at the Twitter firehose. Currently
it is processing 300,000 tweets in JSON format per day and queuing them for
further processing. Lovely piece of software (YTD experience ~ 2 months
production environment usage)

~~~
defly
Cool, Could you show me your app? Just interesting in handling Twitter
Streaming API.

------
gabriele
This is a welcome addition. I expected JSON to become a first class citizens
in REDISland for quite some time. In fact, REDIS has always been my goto store
for JSON data. In 2013 I created a microservice[0] for accessing JSON data
stored in native REDIS lists[1] via HTTP(S).

[0][https://github.com/boilerplatecode/redis-
wsgi](https://github.com/boilerplatecode/redis-wsgi)

[1][https://redis.io/topics/data-types#lists](https://redis.io/topics/data-
types#lists)

------
pfooti
Wow, this looks rad. I'm just now working on a data storage module for my
hobby project, and have been pushing strings through JSON.parse to store data
in redis. Being able to directly address values deep in the hash is great -
when setting foo.bar to 'baz', using a path selector instead of reading,
deserializing, modifying, reserializing, and writing is great.

------
aaroninsf
Naive question: does anyone know enough about this or modules generally to
tell me whether ReJSON values are manipulable from Redis Lua?

That is, I have some application support from Lua scripts manipulating native
types today. But if I can read and write into JSON stored this way from Lua,
that might be very handy!

~~~
dvirsky
You can use modules from Lua if they are loaded, there is not difference
between them and native stuff. You can certainly manipulate ReJSON objects
from Lua, parse ReJSON returned objects from Lua, etc. No limitations as long
as the module is loaded.

~~~
aaroninsf
Great, that is promising... thank you for the clarification!

------
gregmac
[https://redislabsmodules.github.io/rejson/](https://redislabsmodules.github.io/rejson/)

> Windows

> Yeah, right :)

How about just "There are no plans to build a Windows version"?

~~~
wtf_is_up
How about not policing someone else's expression?

------
rizach
Shouldn't storing a json as a string be better performance for redis? All the
extra jobb to find subkeys put extra work for redis. While pulling out the str
and parse it in application would minimize the time spent in redis. This gets
more true as the number of applications against one redis grow.

~~~
dvirsky
If you're serializing and saving the entire object always - you don't need
this extension.

But if you're storing say, a 1K user profile, and most of the time you just
want to update the access time, or get some auth token - this both allows
atomicity and speeds up what you are doing.

The bigger the JSON object is vs. the how small a piece you want to retrieve
or manipulate - the more efficient this becomes.

Also, manipulating a field in a JSON object atomically is currently possible
only with Lua scripts - and even then, Lua will have to parse, manipulate and
serialize the entire object. Where as in this case, if you are just updating
something - there is zero parsing and zero serialization going on in the
module.

------
salmonfamine
I really could have used this for my last side project. Moving JSON blobs
between Python and Redis is pretty error-prone, especially if those blobs
contain nested structures.

------
colinbartlett
The page renders as 100% black and totally blank with uBlock Origin for me.
Not complaining as much as pointing it out in hopes the author might see this
and adjust to prevent people from bouncing.

~~~
icebraining
No problems here; 7 requests blocked, but page still rendered fine.

~~~
chmike
Same here. I'm using the default ublock origin configuration. Maybe it's
specific rules or not due to ublock origin.

