Hacker News new | past | comments | ask | show | jobs | submit login
ImmortalDB – A resilient key-value store for the browser (github.com)
156 points by tosh 87 days ago | hide | past | web | favorite | 49 comments

While I appreciate the work done I feel this is conceptually problematic and is somewhat nefarious in nature, just like "evercookie" that it's based on.

> clearing cookies is a common user action, even for non-technical users. And browsers will unceremoniously delete IndexedDB, LocalStorage, and/or SessionStorage without warning under storage pressure. ImmortalDB is resilient in the face of such events.

When a user (especially a non-technical one) clears cookies, their sole objective is to remove their identity and "start fresh". That's the expected outcome and this library violates that expectation. Also, the reason the browser "unceremoniously deletes" some storage areas is because they are not meant to hold 100% persistent data. So in both cases, the "right" thing to do is respect the end goal of the action and not restore it.

Furthermore, this library should not be used for sensitive data but likely will. This is because sensitive data should not be stored in local storage but rather in cookies [0] as they have better security options with "httponly" mitigating leaks via XSS and scope narrowing with Domain and Path. Local Storage is available to any script on the domain including any 3rd party script in the page.

> Strikes an equitable balance between reliability and respect for the user. That's a fancy way of saying "Doesn't fully respect the user".

[0] https://www.owasp.org/index.php/HTML5_Security_Cheat_Sheet#L...

> Strikes an equitable balance between reliability and respect for the user.

Is it just me, or is that a novel interpretation of "respect for the user"?

"I'm going to continue to track you even after the usual and expected method of preventing tracking in the browser is used, but if you jump through an unexpected hoop that's not documented to the user, then I'll grudgingly delete my tracking data..."

It depends on how the data is used and if an easy delete option exists. There is a valid use case for working around browser behaviour not user behaviour:

> browsers will unceremoniously delete IndexedDB, LocalStorage, and/or SessionStorage without warning under storage pressure

If you have an app that stores a small amount of data client-side (preferences, data to allow operation while unable to connect) but is not used very often it's stored state could be evicted because of a few large apps because the browser drops data on a "last used" basis because it has no way of knowing that this bit of data might be more important than the data touched more recently. The user might not want a small amount of practical data for an app that they used on their morning commute being evicted because they've touched some large games over the weekend. For a large device this isn't going to be an issue as you are unlikely to be that limited for space, but for a cheap phone with limited storage it could be a real inconvenience.

Any app doing this should have a very easy and obvious "purge my data from this device" option, to maintain the user control. In fact to do it properly right, the local storage should be opt-in ("Do you want to keep some data locally to speed up this app and allow it to be used offline? Yes / Not now / No and don't bother be again") with the app continuing to function without the local data cache.

> I'm going to continue to track you even after

Of course it will get used for this, it would be naive to think otherwise, it is after all essentially a structured evercookie, but that doesn't stop the idea having good uses too. The existence of arsonists doesn't mean you need to ban all uses of fire.

> And browsers will unceremoniously delete IndexedDB, LocalStorage, and/or SessionStorage without warning under storage pressure.

I get why it's doing this, but it strikes me as fairly user-hostile to react to the browser deleting data under storage pressure by duplicating the data so it now takes up 4x the space it used to.

It's only for small chunks of data, since you have to limit yourself to the tiniest storage medium the browser allow for your replication.

E.G: on my browser, you can only store 81920 bytes per domain in cookies, so if I want the max redundancy, I have to limit myself to 80ko. Besides, cookies are sent in each request, so I want to limit myself. Let's say you want more, local storage is limited to 10 MB. It's not huge.

But the browser doesn't care about the size of the data, only how recent it is. So it may evict yours, even if it's very small, compared to the huge cache of hundred's of MB from the manifest file or the webworker from other open tabs.

What if it is something small but very important. That seems like it would be the opposite of user-hostile.

So it is not user-hostile when you don’t respect the user’s attempt to delete data? It seems condescending at the very least.

The real world equivalent that I can think of is being stalked even after moving to a far off location taking care not to leave a trace but inadvertently left one or two things around.

With many user hostile and invasive solutions, browser makers should take a stronger stance on clearing all local data simultaneously (cookies, local storage, IndexedDB, etc.). If someone wants a resilient client side data store, apps are one solution. Websites shouldn't be doing this or allowed to do this in a (standalone) browser.

This seems...really valuable, mainly for me is the case when we want to store a value for a user locally but they have browser restrictions preventing it. Primarily, this is the case with corporate users running within their environment where the domain admin has misconfigured their PCs and thus unintentionally disallowed localStorage (I've experienced working with two large corporations, I literally had to get one of them to send me the laptop to figure it out cause they didn't think they had disallowed local storage, it had happened unintentionally).

> Primarily, this is the case with corporate users running within their environment where the domain admin has misconfigured their PCs and thus unintentionally disallowed localStorage

Safari in Private Browsing also used to do this too, which happens to completely break a surprising number of websites.

Safari didn't remove or 'break' localStorage, it simply responded with QuotaExceededError, indicating there's no more space available.

Automatically recreating cookies whenever users delete them is the opposite of "respect for the user."

I think people are reading this backwards. It isn't just using local storage and indexeddb to protect cookie values from being wiped (by the user), I can see it being more useful for using cookies as an extra store to protect localstorage/indexeddb from being cleared by the browser.

The user can still clear the data manually (clear cookies and other local data) or through a properly written app (which implements a "remove my records from this device" option, and preferably makes the local storage opt-in anyway).

If you are thinking of using this maliciously then you wouldn't use this: you'd use the already existing evercookie instead as it survives more (at the expense of potentially imparting more load on the users browser).

Hey guys. I'm Ansgar. I built ImmortalDB.

Don't hesitate to let me know if you have any questions or comments about ImmortalDB. Feedback is how good products become great.

what caused you to write this library? do you believe it will be used for nefarious purposes?

thank you for open sourcing your work

> what caused you to write this library?

On smaller devices, a few kb of important data in LocalStorage was periodically deleted without user intervention.

> do you believe it will be used for nefarious purposes?

If one is nefarious of heart, https://github.com/samyk/evercookie supersedes ImmortalDB.

Hi, congratulations on your work, very enlightening. Could you please remove the "respect for user" and "anti-otherlibrary" propaganda and just explain your work as it is? Let it stand on it's own merit?

What is the storage limit? Would be nice to offer some kind of helper function to return that. For instance, I want to store a base64 encoded image. I believe there is a 10MB limit imposed by Chrome on local storage (or something similar). Bypassing that restriction locally would be great. Could be a nice feature of ImmortalDB.

That won't fit in cookie space... this is really for redundant controls for session/access tokens etc... anything large like that you're better off using caching and a service worker to handle repeated requests. Also, cookies are sent with every request to the server, and I don't think you'd want the user sending 10MB to your server for every request.

Also, if you are storing larger strings in local/sessionStorage, websql or indexdb, you should use lz-string[1] to try to minimize storage impact.

Aside: if you want something with a nicer interface for underlying storage you may want to look at pouchdb[2], which has a localized couch interface and supports remote synching to coudchdb, which is pretty nice.

[1] https://github.com/pieroxy/lz-string [2] https://pouchdb.com/

Yeah, about those cookies - I don't want them (https://github.com/gruns/ImmortalDB/issues/6). I just want to permanently store large data in the browser. ImmortalDB sounded like the right thing to abstract away everything and make it sticky.

You can tell it not to use cookies, to get around the problems of them being more limited in size and being sent on each request so wasting bandwidth.

I'm guessing a mix of localstorage and sessionstorage isn't al that useful (I can't imagine a circumstance where localstorage is being cleared when sessionstorage isn't also or hasn't already been) but localstorage+indexeddb might give you some useful level of protection.

Cool project. I wonder how it persists after clearing browser history/cookies/LocalStorage.

It doesn't: "Data is stored reliably but can also be voluntarily purged if the user designedly clears cookies and application storage".

I think the way to read it is that, if any one of them is cleared (which happens involuntarily sometimes) it comes back, but if all are cleared at the same time then the data does go away.

A user might have their browsers set to clear cookies on close, but not other saved data.. in this way most sessions are closed. This works around that case.

If a user clears all app data for a site/domain then it will be cleared. If you want to work around this, you could utilize resources/assets from other domains in conjunction but it will be more complicated. I find that most of such cases are abused for unscrupulous tracking.

Then that’s inconsistent with the beginning of their readme:

> For example, clearing cookies is a common user action, even for non-technical users. And browsers will unceremoniously delete IndexedDB, LocalStorage, and/or SessionStorage without warning under storage pressure.

> ImmortalDB is resilient in the face of such events.

I guess that’s a good thing. I was worried about the next browser supercookie being made with a library like this.

It's based on Evercookie https://github.com/samyk/evercookie which is a little less... friendly.

>Its goal is to identify users after they've removed standard cookies and other privacy data such as Flash cookies (LSOs), HTML5 storage, SilverLight storage, and others.


>Its goal is to identify users after they've taken actions to not be identified


I don't think they're being inconsistent - intentionally clearing cookies is common, but clearing app storage less so. And the browser may purge app storage to reclaim space, but that leaves cookies. It's unlikely that both things will overlap unless the user deliberately clears both cookies and storage.

I just wondered why there is only support for IndexDB and not for WebSQL. Turns out all modern browsers support IndexDB by now :)

Though websql, where available, is often faster. It's one of the more common overrides for pouchdb to override the user order in favor of websql where available.

Isn't WebSQL deprecated?

Yes, and 10 years after its deprecation, WebSQL is still shipping with Safari and Chrome (worth pointing out that these 2 browsers completely dominate the market).

Interestingly, neither Apple nor Google were pro-deprecation; Mozilla, and to some degree Microsoft, spearheaded the move to standardize on IndexedDB.

Moving forward now that Edge is based on Chrome, maybe there's hope for WebSQL after all. Not that it will be undeprecated, but if 98% of the browser market supports WebSQL then it's effectively a non-standard standard.

The "standard" would have basically read "link your browser with SQLite v3.x.x or an exact bug-compatible clone".

An attempt was made to break down the SQL dialect and exact behavior of that version of SQLite, but the objection was that it's not really great to reverse engineer a standard from a single vendor implementation even if that vendor's product is open source. (eg. what happens when sqlite changes? do browsers stay fixed on an ancient exact version?)

Yep. WebSQL is unmaintained. https://www.w3.org/TR/webdatabase/

Though many browsers still support it. https://caniuse.com/#feat=sql-storage

Can anyone explain why there's a leading semi-colon in the inline script example? AFAICT it doesn't do anything.

Some people for some reason write javascript without semicolons, if they do that you cant safely concatenate javascript files together since the file2 may modify file1's last expression, putting a ; at the start guarantees to complete the last files expression if they havent already done so. A bunch of concatenators will do this automatically / by default

You are correct that this is an issue when writing JavaScript without semicolons, but it's also an issue if you place an IIFE (immediately invoked function expression) directly after a function, even if you're using semicolons:

  function hasSemicolons() {
    console.log('hello world');

  (function() {
    console.log('this is broken, JS will call hasSemicolons()');
In the ImmortalDB example, since it's a standalone <script> tag this is just extra defensive against a minifier that would join the tag with other tags, but just wanted to call out that there's a reason to include the leading semicolon in front of an IIFE whether or not you write your JavaScript with semicolons.

See also: https://stackoverflow.com/questions/1873983/what-does-the-le...

Your snippet doesn't fail. This one does

    console.log('this doesn't work!')
    (false ? console.log : console.error)('this will fail with a TypeError')
    >> Uncaught TypeError: console.log(...) is not a function

    console.log('this works!')
    ;(true ? console.log : console.error)('hello world')

So I tested this out in Firefox. I think you're wrong. I never see 'hello world' even if I remove all the whitespace between the functions.

There is a reason why that issue with IIFEs can be hit, and it's because you've omitted some semicolons - granted they are the sort that most people omit.

Since everything in js is a statement the correct semi-colon syntax for the above is

  function hasSemicolons() {
    console.log('hello world');

  (function() {
    console.log('this is broken, JS will call hasSemicolons()');

The ECMAScript specification (https://www.ecma-international.org/publications/files/ECMA-S...) has statements in a separate chapter from function declarations. (Chapter 14) Furthermore, the function declaration grammar does not contain a semi-colon, unlike other statements.

When you're dealing with JS files it makes sense, but the example is an inline <script> tag. It might be an abundance of caution just in case you're dealing with CDN optimizers that slice and dice inline scripts. I've experienced that with Akamai a few times.


I assume, still, it should not be used to store JTW refresh tokens, or anything where secure storage is required.

I don't see why not - at least if you're comfortable with client-side code on the website having access (implicit or explicit) to the token. Certainly nothing you store on the client side is secure against an untrusted user, so you should only store the user's own secrets, not things you wish to keep secret from the user. And anything you intend to make use of fro the client side must be usable from within the website, so anyone who can get code execution on the website (e.g., via XSS) has already gained access to whatever you're trying to protect.

I see a handful of blog posts on the internet arguing that HttpOnly cookies are a more secure storage place than web storage, because HttpOnly cookies are only vulnerable to CSRF attacks and not XSS and CSRF is easier to defend against. This argument is, as far as I can tell, completely flawed: if an attacker is able to execute XSS, they're then able to use that XSS to execute same-site forged requests - they no longer have to bother with the "cross-site" part of CSRF. They can just fetch your anti-CSRF token with a normal XHR after having executed an XSS, and then they can make arbitrary same-origin requests against your website, each of which gets sent with the HttpOnly cookie.

JWT refresh tokens are equivalent to passwords. (they are active for days at a time).

I do not know of many web apps that store passwords in browser storage.

Unlike passwords, JWT refresh tokens are created programmatically (not entered by user interactively).

Perhaps,I am just missing a link, but I though browsers do not allow an application to programmatically store passwords securely

If you don't persist JWT in localStorage or sessionStorage, how would you handle page reloads? You'd have to rely on a session cookie and get a new JWT from your auth service. You'd solve XSS but get CSRF.

Storing a JWT in localstore isn't the same as storing a password in plain text. JWTs are indeed non-revocable, but they are also non-durable (ours expire after 24 hours) and cannot be renewed without a refreshToken. Most people also do not rely on JWTs to write important information (like password reset) because there are more secure alternatives for these specific use cases.

As an aside, if someone gets script access to your SPA, you're already very far up shit creek. If your user types in any sensitive information like a password to login you've lost the war. Securing the JWT is the very least of your concerns at that point.

JWT tokens, and any session tokens are used for authentication, like a password. The issue is that people don't want to entire a password every single time they want to send a request that requires authentication. Imagine google/facebook/etc. prompting you for a password every time you click a link. That's what it would be like if browsers couldn't store session tokens.

Typically the matching hostname can view local data/cookies though (along with any extensions you have that have access to it - which is most of them). That's why you shouldn't ever copy/paste stuff into the console of a website, it lets people extract your session.

There are two important attributes of a password: they're (often) human-memorable and prone to reuse / inexact reuse on other sites, and they're a long-term authentication token. JWT refresh tokens have the second attribute. But so are many other things, like regular old cookies that don't expire for weeks or months, which browsers store on a regular basis. If you're logged into HN you have one of these.

While the HN cookie is HttpOnly, this isn't really putting it in more-secure storage: an XSS attack against HN can just issue whatever API requests to HN it wants, even though it can't easily walk off with a copy of the cookie.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact