It started with epiphemeral instances, available for a fixed workload, billed by the hour, such as batch jobs.
It turned out people were willing to pay $100 a month for a box worth $30 a month.
AWS was a very different product and not really comparable to other things available at the time.
"You've been developing like a beast, and you app is ready to go live"
"Availability zones" exist for AWS convenience, not for yours. It allows for cheaper and simpler DC ops, removes the need for redundant generators, simplifies network design, and makes "under-cloud" maintenance easier. It's a feature for them, a headache for you, and a (brilliantly addressed) challenge for AWS product marketing.
It sure is nice to have separate failure domains with low enough latency between them to pretty much ignore it in application architectures.
This can be mitigated through redundant service providers, careful checks on shared interconnects and other measures - but having "hard" failure isolation at the facility level will also get you there with less chance of someone doing something dumb.
You're doing it wrong. Plan to lose sites. If you plan to never lose a building, you're just setting yourself up for pain by optimizing for the wrong kind of redundancy.
 "Digital Ocean Killed Our Company", Hacker News, May 2019. https://news.ycombinator.com/item?id=20064169
Droplets (VPS) have generally worked out cheaper than EC2 (with more resources) for me.
The bit that initially sold me on it a while ago was no more worrying about CPU credits, ie consistently maxing the CPU out at 100% then getting throttled, on the small instances (T2).
Also the web interface isn’t a mishmash of 1000 services with different UX for each section.
If DO isn’t cheap enough for people there’s also Vultr which works out even cheaper ($2.50 a month for 1cpu/512mb) if you want something similar (not baremetal).
I can see that price on the pricing page but when I go the Deploy new server page, I cant see that price, the minimum is $5.
I wrote down pros/cons of various ~$5 and below VPS services in a Gist I have been maintaining for a couple of years: https://gist.github.com/frafra/4688b146ca6d55accb768c3557939...
BTW, I run my SaaS on real servers from Hetzner. I figured I won't need instant auto-scaling anyway, and if you provision with ansible it doesn't really matter if it's an EC2 instance or a real server. What does matter is the price and the performance: I get servers which are significantly faster than anything you can get on AWS, and for a much better price.
To be honest, I do not understand the drive towards AWS. It makes sense in micro-deployments (lightsail and/or lambda, when your usage is sporadic) and in large deployments that need dynamic scaling. But it does not make much sense anywhere in the middle.
As an owner of an iMac Pro (the 10c/20t CPU variant) I have to tell you that you are slightly mistaken -- the Xeon-2150B is a pretty strong CPU even compared to the bursty high-end desktop i9 CPUs. Sure I mostly do parallel compilation and work with (and produce) highly parallel programs, and there it really really shines.
But even for the occasional huge JS compilation it still performs pretty well. (Turbo boost puts it at about 4.3Ghz is the max I've seen.)
The iMac Pro's only real downside is that it has only 4 memory channels so as lightning-fast its NVMe SSD is, and as efficient the CPU with its huge caches is, the machine can likely perform anywhere from 30% to 80% faster if it had 8 memory channels.
But in general, iMac Pro is a very, and I really mean very, solid developer / design / professional software user machine.
I mean, all of that doesn't matter much in a world where the Threadripper 3960x / 3970x / 3990x now exist but still. iMac Pro is still the best Mac you can buy (Mac Pro 2019 uses the very last possible high-end desktop Xeons and I don't think Intel will be making many more of these; AMD is definitely on their tails and I don't think Apple will produce that many Mac Pros).
That being said, I am looking forward to buy a Threadripper 3990x monster machine somewhere in the next 1-2 years with dual 4k or 5k displays. Hopefully the Linux community will finally get its act together and make proper sub-pixel font aliasing by then...
I do not think I am mistaken. I haven't yet encountered a Xeon that can beat the desktop i9 in single-core performance. Looking at geekbench scores (https://browser.geekbench.com/mac-benchmarks), my iMac is 10.5% faster in single-core than your iMac Pro, and your iMac Pro is about 15% faster in multi-core. For development work, I will take single-core performance any day.
I do mostly parallel work and for me the Xeon has proved itself as a better CPU compared to several desktop and laptop grade CPUs I also tried.
There has been a lot of comparisons and benchmarks and there will be even more.
After some point monthly bills grow to the scale where it makes perfect sense to invest money and time into your own infra. Yet People will still be throwing money into the hype oven.
We definitely considered using other cloud providers like DO, linode, etc. But it was important for us to go with AWS because we needed some of the other services that AWS providers like s3, Route53, etc.
Some of our static websites are in fact hosted entirely using CloudFront + s3 combination which is something I forgot to mention in this post :)
Very good read, thanks for sharing. Have an upvote.
Once you graduate to full fledge AWS, you can peer your Lightsail VPC to your full fledge AWS VPC.
However in the Route53 case there are a few automatic integrations that only work with other AWS services. Not that big a deal though.
But for the former: AWS is a whole suite of tools -- the specified technologies (cloudfront, S3, etc) are individual (and mostly decoupled) tools within that suite
hardest part is working out how to go fully-public fully-static
Am I reading this right? 4 cores, 8 gigs of ram, 200 gigs SSD for 4.99 euro a month? That is around $5.60 USD a month. That feels too cheap to be true. What's the catch over ec2 or Digital Ocean?
mind you, it's very much a pet and not cattle
Go to hetzner.de, create an account, fire up one of their 5€/month VPS and see if it runs your application well. If it doesn't, upgrade the VPS (or whatever) till the max or until you decide that AWS or whatnot might be better suited (or cheaper).
It's nothing personal against you, but those comparisons aren't worth anything. They are highly specific to the test case, which might very likely not be like the stuff you are trying to run.
And I've just so often seen people on StackOverflow ask how to fix their AWS setup to run a simple WordPress site, it's unbelievable.
There's this old saying: "If the only tool you've got is a hammer, everything looks like a nail."
That pretty much works for GC/AWS/Azure, too.
Never had that from AWS or OVH.
https://www.hetzner.com/dedicated-rootserver/ or maybe https://www.hetzner.com/sb
For example, a dedicated "AX51-NVMe" server for 65 euro / month = 73 usd / month gives you: AMD Ryzen 7 3700X Octa-Core (8 physical aka 16 logical cores), RAM: 64 GB DDR4 ECC, NVMe: 2x 1 TB NVMe SSD, unlimited traffic. That's less than half the cost of lightsail per logical core, or less than 1/4 of the cost of lightsail per GB of ECC ram.
DO, Vultr, AWS, GCP, Azure are the odd ones out and extremely expensive.
The only reason ever not to use dedicated servers is if you're in the bay area and ops wages are so overinflated that you literally can't afford an ops or devops person.
For everyone else on the planet, the comparison between the wage of an ops person, and server costs, is always in favor of dedicated hardware with your own ops people.
I can deploy my SaaS either to VPS servers (AWS, DigitalOcean, Azure, etc) or to physical servers. Both deployments use the same ansible automation. The only difference is that I additionally use terraform to set up the cloud instances.
In every case, if a machine fails it is my problem.
Where is the savings on ops/devops?
I don't know either, I've always used european dedicated hosters.
But apparently the typical YC user needs to pay 300'000$/year for a dedicated ops person just to run an ansible script?
“If this doesn’t take off, I’m on the hook for ~$100/month when it could have been under $10/month“ just isn’t that compelling.
It's something worth optimizing, but the idea that it will break a starting initiative isn't realistic.
Also, with vpncloud I know that my data gets encrypted and is private — not necessarily the case with various VLAN setups.
I don't know why Hetzner dedicated servers are so cheap (they're also cheaper than their equivalent VPS's). I guess they take a bit longer to set up but there must be more to it than that.
You also don’t get the benefits of a virtualized instance like live migration for host maintenance. You can run your own hypervisor - but you’ll probably want extra hardware like 10Gb switches and the appropriate cross connects and as well as paying the one-time server move fee.
Are you sure?
If I look at the AX dedicated server page, I can read that the basic support is free:
"Basic support includes the free replacement of defective hardware and the renewed loading of the basic system (in so far as a disk image system can be loaded)."
The question should be, why is AWS so expensive on all fronts (not just the hardware but also bandwidth)? And the answer is again because the market lets them get away with it.
A large part of the cost of a VM on these clouds is paying their super high employee comp, for advanced software development e.g. CosmosDB/DynamoDB, but most of all, contributing to the high revenue growth that drives stock prices.
It's not paying for the actual cost of hardware. Smart people know this and don't run in the cloud unless they get a sweetheart deal. Consider that GitHub, for example, runs/ran in their own datacenter and used the cloud only for spillover capacity. Zoom also (although now Oracle cut them a sweetheart deal). Netflix built their own CDN etc.
Add this cron job:
yum -y update
At minimum you need a data replication and backup strategy. You're exposed to things like drive and RAM failures on dedicated hosts, so you'd need to think about RAID at least, unless you're running a system that clusters at a higher level (but then you need multiple machines).
However this is basically a matter of learning or hiring/renting a sysadmin to do it for you.
MTBF stats for different classes of hardware at different hosts would be valuable to have, as it can be affected by things like datacenter temperature. But I never heard of such a dataset.
Sure they're below AWS, but if you compare lightsail, they're pretty much identical.
My conclusion was to rent physical servers (I use Hetzner) and get great performance at a fraction of the price.
Here you go
It comes as little surprise to me that the AWS lightsail offering saved them money over traditional AWS services, just as they’d save even more using two dedicated servers with any reasonable provider and being able to have a hot-failover for everything.
At DNSFilter we’ve gone through the evolution of Heroku -> AWS -> VPSs -> Dedicated -> and now are setting up our first colo rack.
I did a price comparison recently between Dedicated, AWS, and Colo. Dedicated is 15% the cost of AWS for our needs, and colo will be 42% the cost of Dedicated for us.
Now keep in mind we have dedicated DevOps staff and are running at a very different scale from OP, and such a solution is not for everyone. But I personally have never understood the folks who love to brag about having spent much time and effort optimizing AWS to spin up and spin things down, when for the same cost I could just have a 10x more powerful server sitting there, on all the time to handle a spike in load, and I can utilize the extra resources or get things done faster with faster gear at 1/10th the price.
I don't ask for anything but usually 2-3 days after that he'd call to inform that I got a few thousands of AWS credit.
I ended up moving my personal projects to OVH as a trial (and due to some some gross incompetence by Singlehop). After a few months of everything looking great, I moved our company's production services and couldn't be happier. I now consider them one of my tier one options when looking for hosting options. The hardware on the higher end models is all enterprise grade - Xeon, ECC, enterprise SSD / NVME drives, etc. You also get full IPMI access with virtual media, so setting up full disk encryption etc is a breeze. Network has been very reliable, their DDoS protection in particular is almost magical - I barely even see the malicious traffic before their network filters it. I only have a handful of servers, but none of them have experienced any hardware or major network issues. Currently have servers in their Vint Hill, Beauharnois and Roubaix datacenters and it all seems to be equally reliable.
The value-add features like FTP backup space haven't been great, speed / reliability problems but I don't care too much for it anyway, I ended up using Hetzner for backup storage as they have a ridiculously good $/GB ratio. I can't speak much for OVH support as I've rarely had to contact them, the few times I did they resolved the problem satisfactorily. With IPMI access and being an unmanaged service, you're probably on your own for anything software related.
Except you have to implement that hot failover yourself, rather than using something like EC2 auto-scaling groups or an RDS multi-AZ deployment. I'm not confident that I'd get it right, and the middle of a night is a hell of a time to discover that you got it wrong.
I don't know about all RDBMS brands, but if you're running Postgres then setting up a read-only slave is trivially simple. You run one command to promote it to master.
At my work we use both AWS and colo our own gear in two geographically isolated datacenters. We use AWS for highly specialized tasks (e.g., Lex) and our colocated servers for just about everything else. The savings have been tremendous -- our costs would be ~10x higher if everything was on AWS.
I think a lot of people forget just how cost-effective dedicated servers and colocation can be. If you're not allergic to dealing with the occasional hardware failure, then there's no question in my mind that it's the right way to go.
But from what I've seen, the point may be moot. Odds are that the EC2 uptime is around the same as you'll get on a single machine VPS anywhere anyway.
The author completely failed to do his homework. An a1.large instance is only 37$ if it's an on-demand instance. You pay a high premium to be able to pull the switch in the exact minute you cease to need one.
If he's willing to go with AWS lightsail with it's monthly plan, the same a1.medium instance type is about 11.75$/month as a reserved instance, and can be had for about 131$/year as well.
Also, creating an instance on Lightsail isn't exactly a monthly commitment. You are free to delete your instance at any time and only pay for the hours you used. It's a lot more flexible than the multi-year commitment you need to make in order to get the best pricing out of EC2.
If I want to move 400 TB per month out of Amazon, why can't I just proxy it thru 100 Lightsail instances?
Isn't that much cheaper than paying directly per byte? For that much saving it's probably worth figuring out how to do this. It can't be hard.
Isn't it "free" to move data internally to a lightsail instance, as long as both are in the same DC?
There are some answers in the lightsail faq:
1. You're limited to 20 instances per account. They'll increase that number on a case-by-case basis, but probably not if you're planning on proxying all your traffic through those instances.
2. If you delete an instance and create a new one, they share the same data transfer allowance.
3. All data transfer (both egress and ingress) applies to the data transfer allowance.
It may still be worth doing the proxy setup, but Amazon seems to have pretty clearly set up limits to make it less desirable.
1. elastic beanstalk, no docker: EB comes with really nice defaults so that you can quickly whip up a flask app, upload it and it just works. It provisions a small ec2 instance by default which iirc costs 10ish a month at best. Importantly, any operation you do with EB will by definition be ready for continuous deployment since you don't get an option to ssh into the machine to deploy. It's extensible enough to add whatever extra stuff you need as well. Only thing it can't do is simple caching ( if you scale to more than one instance that is), but that can be solved by having a separate eb deployment for a worker that can take care of all these aux stuff (elasticache is expensive I think). In a pinch, it also scales well (though the default cheap deployment does not have a load balancer).
2. Just suck it up and go with RDS postgres. Again, I see too many things going wrong with spinning up your own db in an ec2 instance especially if that instance goes down. Im too lazy to write backup scripts and keep track of them! The cheapest RDS postgres costs 13 a month or so, but I just suck it up to power whatever side projects I do. Postgres means I'm working with something I know, and I get full text search and pubsub for free. And whatever I write is not locked code in anyform, and can be scaled up if needed as well. More importantly I'm only spending my time in technologies that are relevant for me in my day job so that's a win.
3. Github Actions to deploy to eb. just a few lines of yaml and you instantly get continuous deployment directly from your repo for free! Really can't beat that.
I have meant to try out heroku since it could be cheaper from what I have read. But I couldn't figure out what their S3 alternative is or how different they are from the canonical cloud offerings.
I'm sure it can be so much more cheaper, but I'm not good at advanced networking or sysadmin, and I'm too lazy / bored / disorganised to write deploy scripts or sshing into remote machines. I'm also always afraid of if/how long I need to re-provision a vps/ec2 if it goes down. Not that they do, but they can, and that scares me.
I have explored redislabs but all in all it seems much more expensive than elasticache. A similar instance from redislabs costs ~$36 vs ~$13 in AWS. My comparison was based solely on capacity
1. When I started working on this I was not fully aware of the large array of services that AWS provides and therefore our setup may not be the best possible one. So I would be checking out Elastic beanstalk for sure and see if it is feasible to use for my next venture.
2. Just to clarify, we are not running mysql on a lightsail instance, rather we are using a managed database provided by lightsail which automatically takes backups on its own and also has the option to restore a new DB from an existing backup.
3. Thanks for the insight on EB. Will be looking into it for sure.
The thought of any of these instances going down scares me as well. but I would like to believe that I have set up enough alerts everywhere so I can take immediate action :)
The biggest benefit for me is that you are not tied to a single EC2 instance anymore. theoretically this is a moot pain point - in my work we have EC2 instances with uptimes going to four years now. I assume AWS aims for even better stability with lightsail (or not? Can't remember what the SLA for lightsail is). But EB or appengine is definitely the bare minimum of what you need to truly consider yourself cloud native and given the small increase in investment it might be worth it.
How does your traffic scale? Does it peak during work hours? You could potentially setup EB with a small instance that scales up to 2-3 instances just during work hours, with just a few toggles. Also, if you at least estimate having this service for a year, you can prepay and reserve the instance which is again a huge discount.
More importantly, I think the productivity gains you see going into continuous deployment is amazing. Getting a deploy by just merging your PR to master is surreal for sure, and makes it a breeze to just work on your features instead of infra! Often if I'm lazy and want to refresh some environment state I'll jusr kill the EC2 instance with the confidence that EB will scale a new one up!
By far the biggest issue the the relatively small connection limits.
... if you did things well, it's just starting another instance and installing / copying things over with some minimum downtime. But if you don't have a 100% documented stack, you don't know which configurations you touched to make things work, you have files lying around, then you're probably going to pay back all your savings and more in the workdays needed to migrate.
At my current employer we never have time for maintenance, and we don't have professional expertise (the "I worked exclusively at this for many years" kind, I mean) at webhosting, security, system admin or dba, so I heavily lean towards more "managed" and cloud solutions.
Like, I wouldn't run Shopify like this, or anything really large, but it may get them to a better place right now.
What I really liked about this post was the discussion of what they actually needed and how to achieve their goals. It's a tradeoff of time against money, and I wish more people documented their thought processes and approaches to this kind of stuff.
Is there a way that the hypervisor allocates a dedicated portion of the shared L3 cache to just your instance, or is it a free for all for all of the L3 cache space against potentially noisy neighbors?
Otherwise, they'd do it in software patches for older CPUs and take the performance hit of the patch.
I'm not sure how much the hypervisor would reserve off of the L3, it is likely to be free for all however you'd still have quite a bit of dedicated L2 and L1 on most xenons. With AMD's first gen EPYC it's a little bit different because clusters of cores share a cache and you can get weirdly high latencies depending on which cores you're using, (i.e. cores 8 and 9 being too far apart)
Also according to this anandtech article, the average total CPU load for physical aws machines is ~60% and is actively balanced out by them. And yes, running benchmarks on a machine without noisy neighbors yields very significant improvements, up to 2x better on the benchmark scores. They measured this by comparing renting out all the cores of a machine vs only renting out the 4 or however they needed .
I'm assuming they'd put a CPU into sleep/hibernate mode in order to save power instead of having it only run at 5% utilization.
I remember a while ago reading about storing your encryption keys in L2 instead of ram and deliberate "abuse" of the L3 cache on VPS hosts however can't find that article and haven't kept up with the news on it.
Given that the intel cpu patches have reduced CPU performance by ~15% ( again sorry don't have the exact source) I'd say there has been quite significant changes in cache management in the name of security.
On the same note, in case someone is digging into how to reduce CDN bills, I wanted to share that we are quite happy with BelugaCDN. It distributes objects stored in S3 using, in a hacky way, referrals as authentication method.
Lots of money saved there.
Lightsail has pretty comparable pricing to Linode. I'm sure they could re-architect their app to use fewer instances, but they could do that and stay on Lightsail as well.
Moving to linode isn't going to give them a 90% savings.
This bandwidth pricing is very interesting because if you have apps that ship a lot of bytes, you can just run like haproxy on lightsail and get those bytes very very cheap instead of paying standard egress data pricing at like 9 cents per gig...
Does this work? It seems like using lightsail is a way to drastically reduce egress data pricing??? It's like a free lunch, what's the catch?
It's actually incredibly complicated and often very difficult (if not impossible) to predict. We have a dedicated part of our organisation which exists solely to figure out costs ahead of time for project.
I'm not sure if it's intentionally obfuscated, I would suspect not, because "pay for what you use" can be broken down into many areas.
It's actually incredibly complicated and often very
difficult (if not impossible) to predict.
It absolutely is a strategic choice and not an accident.
Lightsail and digital ocean give you 1TB egress minimum included in billing. Now try to input that number into EC2 instance. If I am not wrong, that turns out to be 130$ approx
While this is true, we wanted to go with AWS for 2 primary reasons
1. We use some of the other services provided by AWS like s3 and Route53.
2. Just the reliability and brand that AWS has is something that we had to take into consideration
No, Lightsail doesn't use Graviton2 powered instances today.
They are using Lightsail to create pets, it has not much configuration.
When you need a cattle, you need more config and complex setup which costs money.
As they say in the end, this is because they are a micro startup and don't need a huge scalable infrastructure.
It's funny as it seems to be the inverse of economy of scale. The more you grow, the higher the marginal cost. But I think that Amazon gives discount to large users
But, my advice tends to be Lambda first if it is really low volume, LightSail second, and full AWS third.
As far as Lambda, I often recommend proxy integration, where you can just use the standard API framework for whichever language you choose (Django, Flask, Express, ASP.Net, etc) add three lines of code and push the entire thing into Lambdas. This gives you the flexibility to deploy to a VM later with no code changes.
For your static assets use S3, except for the case of Lightsail where you get plenty of free bandwidth.
I totally agree on Lambda first. The only reason we did not do that is because when we started this product I was not well versed with Lambda and serverless and preferred to work with something that I dealt with previously.
If I could go back in time, I would set up all our applications on serverless.
From the perspective of an outside, boots on the ground Developer/architect, I’ve never worried about “vendor lock in”, I believe you should choose your infrastructure wisely and go all in. But, I do worry about “Lambda lock-in” for APIs. I like the optionality of being able to deploy my APIs anywhere just by changing the CI/CD pipeline.
That’s why I recommend using proxy integration. Every language supported by Lambda has a method to just throw your standard API in lambda without tying yourself to it.
Here is an example for Node/Express
Python/Flask (ignore the DDB part):
Could you go further into what this is? Do you mean the API Gateway proxy integration? Is there a code sample somewhere
Short version, instead of letting APIGW do the routing and applying its own intelligence, it just passes the event straight to your lambda. Your lambda has to process it. The plugins I cited, will translate the Lambda event to a form that your standard API framework expects.
Those plugins also work if you put your lambda behind an application load balancer.
Do you have more than one, and DNS load balance, or do you just live with the risk?
One of the main reasons why I use an ALB/ELB is so that I don’t have that SPOF. If you found a way around that, please share, I would love to know, so I can save some money :)
I think it's highly unprofessional to use a setup like this in production. Looking at his product, it seems like a product whose downtime has a big impact on their clients.
We currently handle it by setting up alerts all over the place so I can take quick action if something goes sideways, but other than this I have not really found a way around it.
We also have latest snapshots ready of all our instances so that I can get another server running ASAP during a calamity.
Depends, but it's important to know the details.
Lambdas are used for 2 things in AWS:
1) Glue for managing services and events. (Required for advanced AWS mgmt.)
2) User applications. Lambdas have improved over time, but initially had all kinds of limitations (startup time, database access, 15 minute job limits, etc.)
DynamoDB has also improved a lot. Initially hot keys could cost you a lot of money. Some companies have designed their applications to use it in a way that absolutely no administration is required, reducing DBAs and Ops people.
Besides using the free tier and other AWS services which are practically free at low uses, you can use cost effective options like Fargate Spot Instances and EC2 Reserved Instances. I highly recommend Fargate over running instances. Use Lambda with CloudWatch triggers if you need to schedule occasional jobs (or use Fargate's feature for that). Try to avoid heavy reliance on caches, ElastiCache is kind of a rip off. Move as much content to static as possible, use CloudFlare to reduce bandwidth costs. If you're gonna serve over S3, you might as well front it with CloudFront as it's actually cheaper due to caching at the edge, and also more reliable. ALBs are expensive but very useful for APIs as well as autoscaling (if you have to run instances, run them with an ASG, which also means having versioned AMIs)
I wrote a PostgreSQL DBaaS Calculator that got some traction here a while ago, and I just updated it this evening to add Lightsail to see how it stacks up: https://barnabas.me/articles/postgres-dbaas.html#calculator
No surprise, Lightsail is similar pricing to RDS with a 1-year commitment, but it's month-to-month. It's a pretty good deal until you need more than 8 GB memory, 240 GB storage, 2 cores ($115/month Standard plan). But Azure or AWS RDS are the ones to beat.
Its amazing to me that this is all included for basically free except for Application level filtering by these providers.
In my day job we saved $600 a day by cutting down on needless inter-AZ traffic.
In general I'm not sure you can really compare instances from different service by just look at cores, RAM, and disk.
I used to have my small website and email server at Rackspace, on the smallest, cheapest instance. I ended up getting out of mail hosting (moved that to Fastmail), and putting the small website on Lightsail.
The Lightsail instance and the old Rackspace instance have the same nominal specs for virtual cores, RAM, and disk. (Actually, Lightsail may be better on disk--I don't recall if the Rackspace one was SSD like Lightsail).
The main thing the website actually does is host some graphs showing the temperature in my house. An ESP8266 temperature monitor I made uploads a sample once a minute. A script was running once a minute on the server that using gnuplot to make graphs of temperature over the last hour, 3 hours, 12 hours, 48 hours, and over all samples.
At Rackspace it ran that fine, while at the same time gathering my mail from several services via fetchmail, receiving mail for my domain on its smtp server, and running spam filtering.
On Lightsail just handling the website it was locking up every few hours. I managed to catch one pre-lockup and found the load average was something like 300.
What was happening was sometimes that once a minute graph generation task was taking longer than a minute, and that would slow down the next one, and so on. Oddly, it didn't seem to be gnuplot that was taking too long, but rather the script that took the file containing all the samples, and extracted just the samples newer than a specific threshold. At least, running everything by hand that was the only step I ever saw take unusually long.
I changed it to every 5 minutes to temporarily stop the frequent hanging, and then added a check to my script to skip regenerating the graphs if a previous instance of the script was still running to fix things permanently. I also changed it from storing the data in just a big file of samples sorted by time to an sqlite DB.
In my opinion, that approach sacrifices pretty much all the benefits that cloud proponents usually talk about, like only paying for what you actually use or scaling up and down on demand.
In fact, I'm not sure what the actual difference would be between Lightsail and a good dedicated hoster that provides backup and failover services.
- They could have much better service
- And bring down their expenses by at least half again
I used lightsail, and it's a terrible service.
It is very expensive (still) compared to other hosting providers.
And if your instance ever go out of memory, it become unresponsive for as long as you don't go manually restart it.
On other providers that I use, the OS would just "sacrifice child" (kill the process) and restart it.
It's not ideal, but much better than having to go there yourself to restart the whole thing.
And yes, I fully agree with you. In addition to the other disadvantages, Lightsail also shifts the burden of process monitoring and management onto you.
But somehow on lightsail (only), the machine just goes completely unresponsive instead.
Too many startups starts with Kubernetes, microservices putting a lot of engineering into infrastruture when they should worry about product market fit and simple server would just do fine.
I've been looking for a job recently and my lack of experience with excitedly building expensive overly engineered big piles of stuff to do simple things is really hurting me.
When I suggest simple solutions proportional to the problem size people look at me like I'm some kind of simpleminded ninny.
And then when I describe working systems I've made using the approach on companies that exited north of 10 million, then they usually just think I'm a bullshitter.
10,000 users? Just do a simple lamp stack on a $10 month vps and don't code it like a moron, it's not hard. And it can be delegated to almost anyone, that's important for longevity. That's it, move on with life.
I've even offered to show logs, I mean, it's remarkably bad how hard it is to find work as a cost-cutting product-focused engineer, nobody wants that shit, really.
They want autoscale, prometheus, travis, blah blah blah... as fancy and crazy as possible, an empire of infrastructure instead of infrastructure for an empire... It's a damn cult.
I'm thinking about giving up and giving in. Maybe spend a month and drop $10,000 or whatever on a bunch of the tech just so I can talk about these things.
The second thing I'm constantly surprised by is how many obvious things are formalized with fancy names, math equations, papers, etc. Stuff I'd think like literally anyone with half a brain would do.
Recently I found out about "CRDT". Apparently I've been doing it for a decade... It's just super simple stuff wrapped in needless formalism, like back when design "patterns" were all the rage. I used to think "why is there a name for that?" it's such basic stuff.
So saying "dude, the emperor has no clothes, here, take my coat" - bad interview strategy and it's hard to rectify because there's like 50 naked emperors and silicon valley culture is on a constant rotating basis of seeing the nakedness of 5 of them and ignoring the other 45.
Maybe what I should do is just a couple throwaway interviews to focus on finding out the current fads and then be able to give the right cultural signals for the ones I actually want.
And it's been fine, profitable, revenue generating, sure. Mysql, php, a bit of python, easy stuff.
Honestly I've been wanting to move and try out silicon valley again though and it's been hard because I don't speak that language and frankly I'm skeptical and hostile to it - bad idea. That bad attitude needs to go.
I've been in practical startup land too long so I keep getting stonewalled from getting in to the sv world.
I'm trying to contribute to some companies open source projects recently and then basically earn my way in through labor since I can't do the virtue signaling. It's only been a few weeks, nothing yet.
I gotta learn how to be along for the ride without always trying to reach for the steering wheel.
Hopefully by the end of summer I'll find a lead somewhere.
It's not about the money, it's kinda actually how I want to spend my time for the next year or two.
For example, one of my companies is doing hosting and image delivery services for photographers. I started it on one dedicated server, and now 10+ years later it's running on 3 dedicated servers.
My hoster does backups and OS upgrades, so there was never any reason to go into the cloud. Except maybe, if I wanted to burn cash on slow performance ;)
> We are a team of two people
Articles like this reflect the fact that one size does not fit all.
> a good dedicated hoster that provides backup and failover services.
In theory that sounds reasonable. In practice the ability to completely manage the system via a web console and APIs makes a big difference.
But honestly, if you have only one server, what do you need an API for? It'll be much faster to just configure things by hand. It's not like you'll switch servers very often. Or at least the original article says that they committed to monthly cancelation terms.
That's a trope that I have never seen in practise.
A company might have some small percentage of clusters that autoscale, but usually there's a bunch of servers that don't.
Also, when AWS has control plane problems, autoscaling doesn't work, you get swamped, your health checks fail, and you lose all your servers and you're hosed. So it's better to always over-provision in the first place.
- kill services with low usage
- downgrade instance size
- downgrade instance type
- merge databases
- schedule services with obvious usage patterns to shut down when not used
- use EC2 spot instances
Most importantly, it requires an aggressive culling mindset. If drastically reducing the AWS bill means staying afloat, then make bold choices.
Edit - Alsocontainerize your app from the start so that if it does take off you can slam it in ecs or k8s.
I wonder how do you cover the cost for unlimited outbound messages in your free plan...
Like I mentioned in the post, there is no way to programmatically upload anything to the Shopify CDN and the CDN cache cannot be invalidated once you upload a file. If you want to update an existing resource, all you can do really is upload a new one.
This ofcourse still does not completely prevent people from abusing it, but it does restrict usage to a large extent. There is also the issue of giving out a cdn.shopify.com link to your customers instead of something that has you company branding on it. This is not a problem for us because our customers do not have to manually add this snippet to their website and we do it via an API instead, so this link is not apparent to our customers
Author is talking about few hundred dollars here.
> I would like to put emphasis on the fact that we are a micro-SaaS product that solves a small and specific use case and therefore this kind of AWS setup worked for us. This may not work for big organisations or products where the traffic is erratic.
> This setup will also not work for folks who have a ton of stuff to do already and would prefer to use managed services and not take the additional headache of monitoring, maintaining and provisioning hardware resources on a regular basis because this has a time cost to it.
It is refreshing to see a post that doesn't pretend as though its recommendations are the holy grail for anyone and everyone.
I personally prefer saving time and effort, over a ~$500 monetary discount. But it's nice to read posts like these and learn more about alternatives such as Lightsail.
Completely agree. As the general "Ops" manager, I hate getting the call at 2am because something broke... I hate it even more when it's not really fixable by me in the first place (e.g. black box appliance or software).. but I feel a heck of a lot better when I can point the finger at an external team that's paid and supposed to be experts... If you outsourced Redis or you DB, and someone like AWS or MongoDB themselves has an issue... Will they pay people a lot of money to simply provide that service reliably... I get paid to keep 20 things connected, not be the subject matter expert in all of them.
But how would you feel if you were not just the Ops manager, but the CEO as well?
I totally get that it can be cheaper to do things yourself. And I also get that sometimes you literally can't afford to go the easier route.
But I also think a lot of people of forums like this overly discount time spent or even just distraction that take away from doing the many other things you could be working on as part of a business.
I guess it depends where you live and how much your time costs (and if you have any time to spare).
> where I learnt how to scale an application to serve ads to millions of users a day
Please lets stop this 'race'.
I would like to have a workflow of really lightweight QR/barcode transactions... but with what seems to me currently a complex implementation:
1. you have a prescribed barcode ParentA
2. you scan that barcode and create a QR code ParentB
2.a: you ascribe that QR to an object ChildA
3.n: you have that ChildA go through many iterations
3.n+: ChildA may be conjoined with ChildB,C,D etc... -- but I want an iterative QR code that follows the family tree back to parent A and keeps history of all transitions...
(I know I am wording this poorly... just thinking out loud...)
The workflow that I am talking about would apply to taking the nightmare out of METRC tracking system for cannabis -- the METRC system is a piece of shit - but wants complete chain of history of how plantA becomes productA - and it is a bullshit workflow - and I have a way to make it easier - and by using a system similar in the way described in this post, it could be done very cheaply...
If it's barcodes being used by humans in the real world, get a software package to handle it, check the solution from Zebra https://www.zebra.com/us/en/products.html