WarpURL – URL Shorteners Latency Monitoring

rehanvdm · on July 7, 2021

Author here, trying to get awareness to the latency URL Shorteners force onto every user that clicks on them. You can read more about it here: https://link.warpurl.com/DrMbLW but below is a quick summary.

Double Linking - Shortening a URL that has already been shortened. The biggest culprit is social media sites shorteners: - Twitter ~189ms (+-81ms), max ~420ms Mumbai - LinkedIn ~447ms (+-245ms), max ~850ms Mumbai - Facebook ~148ms (+-66ms), max ~240ms Singapore

Average URL Shortener: ~300ms (+-150ms)

When a user in Mumbai clicks on a Bitly link (~300ms) shared on LinkedIn (~850ms) then it adds ~1150ms of latency, which is more than 1 second before even reaching your website!

WarpURL(https://warpurl.com/) is ~6 times faster (<50ms) and ~5 times (~30) more consistent.

tln · on July 7, 2021

Thanks for posting!

Have you worked on optimizing DNS and TLS negotiation at all?

Just now on my laptop, the first request took 1.9 seconds!

    $ time curl -v https://link.warpurl.com/DrMbLW
    ...
    real 0m1.936s
    $ time curl -v https://link.warpurl.com/DrMbLW
    ...
    real 0m0.079s

What did you use to check the latency worldwide? I've got a quickly hacked shortener running on CF workers, curious how it stacks up :)

PS. there is a small typo on homepage... "and the number of changes you"

rehanvdm · on July 7, 2021

DNS no, and also not TLS. You can actually navigate it with HTTPS => http://link.warpurl.com/DrMbLW

The first hit any user from your country/city navigating the URL will be slow aka normal URL Shortener speed. There after it is cached in that region/city/CDN POP. That is why the second curl was so fast. If you ask a friend living in close proximity to you (same region or city) to also navigate it, they should see the fast time.

I wrote more about the logic behind it in this blog post: https://link.warpurl.com/x4BSyC I also delve into the architecture a bit. Spoiler alert, it is 100% serverless!

anonred · on July 7, 2021

Seems quite disingenuous to claim low latency and variance based on a synthetic benchmark when the real world p95 latency and variance will likely be quite high with a normal distribution of user geos.

And FWIW I requested the same link on my laptop and then phone and my phone took >1s to resolve the redirect.

rehanvdm · on July 7, 2021

I appreciate the response. I clarified the method that I am using on the comparison page, but did not want to go into too much detail because it was taking more real estate than I wanted it to.

I will extend that text by saying 1 ping is taking the average time of hitting the URL 5 times. The reason for taking 5 readings directly after each other is to filter out any network related issues and noise on the graphs.

When you see a spike in the x axis, it means that was the average latency of hitting that URL 5 times for that interval(every 10 minutes).

I am doing it every 10 minutes because I want to show the variance not only for a single region but especially across regions. If we were to take the p95 or even p99 tail latencies over say 100k clicks, WarpURL will be significantly faster.

I don't think I am being dishonest in my claim?

tln · on July 7, 2021

I don't see how you can get a sub 100ms latency average over 5 hits if your cold starts are > 1s.

Maybe hitting the URL every 10 minutes keeps the URLs you test in your cache.

If users are frequently getting >1s due to cold starts and your stats are systematically excluding them, then the stats aren't very realistic.

I'm guessing you can see how many cold starts there were in AWS Lambda somehow... I can say anecdotally that I got slow responses several times from link.warpurl.com

rehanvdm · on July 8, 2021

> I don't see how you can get a sub 100ms latency average over 5 hits if your cold starts are > 1s

I can't and I am not trying to hide it, the CDN will keep a link in its Edge Cache for about 1 day. This is what causes the smaller spikes on the WarpURLs chart, the test link gets evicted from the cache and has to be looked up in the origin. WarpURL does not optimize for the first hit, but for the thousands that follow it near that PoP.

---

> Maybe hitting the URL every 10 minutes keeps the URLs you test in your cache.

No they stay at least 24 hours

---

>If users are frequently getting >1s due to cold starts and your stats are systematically excluding them, then the stats aren't very realistic.

Again, WarpURL does not optimize for the first/single click but for a single URL that gets thousands and thousands of clicks.

---

rehanvdm · on July 7, 2021

I am using AWS Lambda functions deployed in all major AWS regions. Deployed with the AWS CDK, it was as easy as creating 1 stack (CloudFormation), then just looping over it for every region I want. Storing the numbers rolled up into DynamoDB.

I can't be an AWS Community Hero and not use AWS in my personal projects/business ;)

Thanks for noticing the typo :) I am a bit sloppy when it comes to language.

rehanvdm · on July 7, 2021

FYI, if you want to follow me along the journey, I am building WarpURL in public, https://link.rehanvdm.com/BaSR1y was last months(also the first) retro of many.

PaulHoule · on July 7, 2021

It's what I'd expect. Most people lack the attention to detail to care about UX and latency.

rehanvdm · on July 7, 2021

Yeah, as developers we want to optimize the first page the user hits but we don’t often think about how the user gets there. Just because it is fast for us, practically sitting next to a server does not mean others will experience the same. Suddenly optimizing that last 100ms until first paint becomes insignificant.

The real shocker after creating that page, was to learn that no other URL Shortener makes use of a CDN :0