
Extended Outage at Instapaper - lenova
http://blog.instapaper.com/post/157027537441
======
msh
From the article :

After spending multiple hours on the phone with our cloud service provider, it
appears we hit a system limit for our hosted database that’s preventing new
articles from being saved. At this time, our only option is to export all data
from our old database and import it into a new one.

Seems pretty bad that it takes hours to discover that. Anyone knows what cloud
it is?

~~~
nanaujal
instapaper.com seems to be hosted @ AWS

~~~
bhouston
Which DB at AWS:

[https://aws.amazon.com/products/databases/](https://aws.amazon.com/products/databases/)

Also shouldn't AWS warn them that this was about to happen?

Edit 1:

DynamoDB says: "There is no practical limit on a table's size. Tables are
unconstrained in terms of the number of items or the number of bytes."

The SQL databases say this: "The Amazon Aurora engine will automatically grow
the size of your database volume as your database storage needs grow, up to a
maximum of 64 TB or a maximum you define. The MySQL, MariaDB, Oracle, and
PostgreSQL engines allow you to provision up to 6 TB of storage on-the-fly
with zero downtime."

So they either hit 64TB or 6TB of data I assume?

EDIT 2:

According to this they are MySQL based: [https://medium.com/making-
instapaper/rebuilding-instapaper-s...](https://medium.com/making-
instapaper/rebuilding-instapaper-search-7fae1340699e#.svigv748s)

~~~
gagabity
Think you need to set up CloudWatch to get alerts.

~~~
dhd415
And CloudWatch is free for RDS at 5-minute collection intervals which is more
than adequate for alerting on things such as "Am I about to hit the space
limit on my database?"

------
macintux
Post-mortem: [https://medium.com/making-instapaper/instapaper-outage-
cause...](https://medium.com/making-instapaper/instapaper-outage-cause-
recovery-3c32a7e9cc5f#.g7jy6jtdg)

------
dangoldin
Very odd and surprising it's taking that long to recover. I got the following
email from them this morning which seems to indicate they will only be fully
normal a week from now.

As we shared earlier today, Instapaper is experiencing an extended outage.
After 31 hours of downtime, we were able to rebuild a database instance to get
Instapaper back online! In the interest of coming back up as soon as possible,
this instance only has the last six weeks of articles. For now, anything
you’ve saved since December 20, 2016 is accessible. We’re working on getting
the rest of your articles restored, but wanted to give you access to your most
recently saved articles in the meantime. Again, there hasn’t been any data
loss. The full database exports are taking much longer than anticipated. We’re
aiming to get the full archives available by next Friday, February 17 at the
latest. We appreciate your patience and understanding in the interim. If
you've got any questions or thoughts, just reply to this email or let us know
at @InstapaperHelp on Twitter. \- Instapaper Team

------
ycombinete
I'd be interested to know what services like Instapaper are used by HN
members.

Personally I've always wanted to like Instapaper more. I really do. I've given
it a few extended runs, on it's own, and parallel to Pocket; but I always end
up moving back to Pocket.

~~~
joekrill
I recently switched from Pocket to a self-hosted instance of Wallabag
([https://wallabag.org](https://wallabag.org)) and I've been very happy with
it.

One thing I've noticed with these types of services is that they quickly start
becoming intrusive and annoying. Ads are the obvious annoyance. But as an
additional example, the Pocket extension injects code into certain sites
(HackerNews being one) so that I can "more easily" "pocket" posted articles
directly from the listing page itself. And in fairness it can be disabled, but
it's enabled by default. And with things like this, I just find that simpler
is better. I don't need all these extra features being pushed and modifying my
browsing experience "by default".

~~~
dangoldin
I had some issues getting the Wallabag app working on a smartphone. Have you
used that successfully?

~~~
joekrill
I've had no problems at all.

I use it primarily on my Android tablet (Nvidia Shield K1), and occasionally
on my phone (also Android, Moto X Pure 2015). I haven't noticed any problems.
I use it for both reading and adding using the built-in sharing functionality
in Android. For setup I simply set my URL, username, and password in the
settings and used the "Autofill" option to handle populating the rest of the
settings, and it works great. I'm running the server itself in a Docker
container, with connections proxied through Nginx (also in a Docker
container).

------
gagabity
I'm guessing they mean space limit on a single DB, you would think a service
like Instapaper would have some monitoring in place by now.

~~~
j_s
Interesting to compare both the different approaches taken in sharing details
by GitLab vs. Instapaper, and the long-term reaction to it here.

The backlash against GitLab was obvious; I'll be watching the next Instapaper
disucssion with interest.

[https://news.ycombinator.com/item?id=13608267](https://news.ycombinator.com/item?id=13608267)
etc.

