
Lorem Picsum, death by a million pixel-gigabits - dmarby
https://dmarby.se/blog/lorem-picsum/
======
boyter
Image resizer scaling is one of the more interesting problems I have worked on
in the last 10 years of so. I was part of a small team that designed and built
the resizer that powers the nine.com.au network of sites. Modest by USA
standards it gets close to hundreds of millions of views a day across the
whole network.

We ended up using shared nothing architecture. The whole thing ran on 6 T2
large AWS instances using a slightly modified version of Thumbor where if you
rotated the key the disk cache would be shared so avoid a large scale cache
invalidation. It worked quite well and we rotated the key every few weeks.

Things I learnt.

Pretty much all image resizers have the same performance as all the good ones
call out to C libraries in the end. Akamai (the CDN we used) despite having
site-shield on would still hit the back-end ~100 times for the same image on
occasion as I suspect all of the whitelisted machines could request the same
image if their internal sharing didn't kick in fast enough.

Long tail images were the ones that brought the resizer to its knees. The hot
images would quickly enter the local disk cache and were not an issue. Purge
the whole cache though and the long tail images would quickly overwhelm the
instances.

The last thing I learnt was to have a backup cloud-front ready to flip over
to. At one point Akamai had issues and the resizer was facing origin load. It
capped out at about 300 RPS which couldn't keep up with what was expected. It
got even worse when the T2 instances ran out of credit. Spinning up cloud-
front solved that issue once the DNS flip kicked in.

One good thing to come out of it was I helped write the C# thumbor library as
we had one site that was using C# and nobody could move over to the new
resizer without it.

~~~
baybal2
I once tried to pitch a one "mobile ecommerce website as a service" company in
Vancouver to go for GPU based image rescaler at around 2011.

A very dumb proposal: no caching, resize on the fly, the gpu has many gigabits
of resizing performance for as long as JPEG is involved. One GPU works in
decoding with VDPAU, one in encoding with CUDA.

That knocked down any google app engine based "elastic" service on economic
basis, but the catch is that you have to send that GPU resizer to every colo.
That did not work out as with google app engine you were getting access for
google's POP network for almost free, and they were already paying for
gigantic amount of CDN traffic.

\----

When I worked as sub-subcontractor for the Alibaba's RDMA wired DC project,
there was one demo by another team where they got DSP devs involved and they
got a 10GB/s JPEG transcoders for under 100W. I think, most of power budget
was going to the FPGA that was linking it all with the NIC :/

An expensive toy, but it again demonstrated to me just how powerful is the
"lockdown" power of all those "cloud" companies. You can not buy anything like
this on the open market.

Imagine what it could've been if they offered something more cash worthy over
the RDMA there.

I said long ago that the killed product during the Bitcoin boom was not the
mining itself, but leasing and renting the rigs. Your capital costs get
covered near instantly, and you can cash out the next week. I believe that all
that "cloud" thing will eventually follow this path.

~~~
avn2109
This is a (rambling) underrated pro comment and you should turn each of these
little vignettes into blog posts.

------
davidu
More interestingly, the github repo is just a wonderful example of a fully
built application using modern techniques in a microservices architecture:
[https://github.com/DMarby/picsum-photos](https://github.com/DMarby/picsum-
photos)

It's so hard to always find how all the pieces fit together and this repo has
it all. Really impressive.

~~~
cc81
Is it really a microservice architecture? It looks like it is a web
application with a frontend, backend api and normal modules/packages for the
functionality?

------
wlesieutre
Funny timing, I was just looking at this over the weekend.

Another option if you prefer cat pictures:
[http://placekitten.com/](http://placekitten.com/)

------
k2xl
I wonder how much it would cost to do a similar service and just upload all
images and dimension possibilities to S3 + use cloudfront cdn.

~~~
swebs
Well let's limit the possibilities to any image between 1x1 and 1920x1080.

Calculating the number of possible images is simple enough. It's just
1920*1080, or 2073600 images.

Now what's the average size of each image? Well the average of each dimension
is half of the full size, so the average area should be 1/4th to of a
1920x1080 image, or 518400 pixels per image.

So in total, we need to save 1074954240000 pixels. Now how many bytes does
each pixel take to store? I really don't know. You can save them as 8bit RGB
PNG images and assume each pixel will use 24 bytes. You'd add some for
headers, and remove some to account for compression, but let's ignore that for
now. Maybe someone else can chime in. But for now we need to store
25798901760000 bytes, or about 23TB.

Not impossible, but sounds needlessly expensive.

~~~
cabaalis
I'll bet that most image requests will be within certain parameters. 2^x by
2^y. So you could probably pre-cache most real-world image sizes, and leave
dynamic generation for 1-offs.

~~~
davinic
AND you could try to figure out how to do that beforehand, but some LRU-type
cache solves the problem without any prior knowledge of what those sizes are.

------
hinkley
One of the mysteries of my career is that everyone seems to be using Varnish
except for the companies I work for.

We are always using some other solution. Like pure CDN or our own poor man's
S3.

------
siscia
Is there any economical margin in running a similar service?

People do it just for fun or have some monetisation strategy?

------
bananocurrency
nice to see a digital ocean site with some popularity not getting nuked or
otherwise blacklisted.

~~~
jokowueu
Why would it get nuked or black listed ?

~~~
SahAssar
DO has sometimes been accused of overselling capacity and then terminating
services when they are fully used. I have never actually heard a first-hand
account of this, but I assume that is what the parent is talking about.

~~~
penagwin
I'm not sure if this is what you're talking about, or if the scale just wasn't
a problem, but anecdotally I've (accidentally) pegged a 5$/mo instance for at
least a month without any issues.

EDIT: I do know that DO/Linode/(maybe vultr) have blackholed IPs during large
scale DDOS attacks, simply because they don't have the infrastructure to
mitigate it, and you affect their other customers.

------
elcomet
I wonder how much it costs them to maintain this service.

~~~
kingkool68
I run [https://dummyimage.com](https://dummyimage.com), the first placeholder
image service which has been online since 2007. I use a 1GB, 1vCPU instance
from DreamCompute costing $6.00 per month. That's it.

To be fair dummyimage.com doesn't do any sort of image reading or resizing.
Source doe at
[https://github.com/kingkool68/dummyimage](https://github.com/kingkool68/dummyimage)

~~~
haasted
What kind of traffic does your service attract?

~~~
famfamfam
For reference: I do about 7GB of image traffic/day for placekitten (a similar
service) for about the same cost ($6/mo). I _do_ resize images.

~~~
davinic
I love placekitten. Thank you for this service!

------
narven
Awesome, its one my fav places for random images :D

------
roddux
Seems like a hell of a lot of engineering effort for something that could
easily be replaced by a gradient...

~~~
tiborsaas
It's a good exercise to test these technologies and how to run things at
scale.

A real image from third party is valuable since gradients don't have onload
events for example.

~~~
AstralStorm
An image gradient then. 1 px wide perhaps, or high. Few problems resizing it
(even on client) and it should be reusable.

~~~
tiborsaas
I think the point is not to get away with the scaling problem.

Here's one service I used in a project's development mode:

[https://dummyimage.com/2500x2000/000/09f.png&text=HackerNews](https://dummyimage.com/2500x2000/000/09f.png&text=HackerNews)

In the production build I changed it to the real host.

------
classics2
Why would a stateless service need a database?

------
fullwall
nice to see a familiar project still going strong. :)

