Especially since “EC2” is not just a VM. There are dozen services at least that come under EC2.
Also if you start trying to use DynamoDB like you would Mongo, you’re going to be really upset. The use cases, design considerations and performance considerations of DynamoDB compared to Mongo is night and day.
Yes. But DynamoDB basically requires you to know your access patterns up front and isn’t that flexible when it comes to querying - yes I’ve watched all of the reInvent videos about how to properly model your DDB tables.
That being said, if I needed the flexibility and wanted to stay in the AWS ecosystem, I would use DocumentDB with Mongo compatibility. It uses the same storage engine as Aurora, it’s much more flexible and it doesn’t have any of the well known downsides of Mongo.
I’m really not trying to sound like the “do people still watch TV” guy, but I thought OpenStack fell off the hype curve years ago. Do people still use it.
> fell off the hype curve years ago. Do people still use it.
Hype and usage are very different. It's not like a lot of people really used it during the hype either. There are a few places which actually need something like that and can't use k8s.
Consider that your typical enterprise software project will use quite a lot of these, and that you pay for them all separately, and sometimes pay twice for them (e.g. S3 you pay for storage and for outbound bandwidth).
It's quite a tour de force how Amazon have taken "separation of concerns", applied it to web services and used it to create complex and difficult to understand or predict pricing to print money. Bravo.
I'm always amazed at the complexity of AWS billing. "Why does this cost so much" turns out to be an incredibly hard problem to answer the moment you don't have 100% perfect discipline when it comes to resource tagging.
The fact that there are consultants who specialize in figuring out AWS billing was, in retrospect, a warning sign.
I usually try to cut them some slack. I mean, they're operating this stuff at Internet scale and have never (AFAIK) raised prices. They've basically got two options:
* Price things in an intricate way sufficiently specific to actual costs of providing the service for many use cases
* Price things more generally in a way that more or less probably results in about the same revenue
The problem with option #2 is that it's dependent upon some customers implicitly subsidizing others (e.g. any "unlimited storage" backup solution). It can work a lot of times, but sometimes it doesn't, and at AWS's scale, that could change quickly. It'll upset a lot of customers if they then have to restructure pricing to account for a small subset of customers' use cases being too costly to provide.
>I mean, they're operating this stuff at Internet scale and have never (AFAIK) raised prices
Prices should have gone down because they've continued to scale up and go into custom cooling/hardware/backbones/etc.
You're basically saying that you're cutting them slack because their margins have gotten much larger and they haven't decided to raise prices to make them even bigger.
> Prices should have gone down because they've continued to scale up and go into custom cooling/hardware/backbones/etc.
Why should that make prices go down? Prices aren't based on cost. They're based on what the market will bear, i.e., what customers will pay. If you think AWS is too costly, you can go find some other solution. If enough customers start doing that that AWS's financial state suffers, AWS will have to rethink its price and cost structure.
In a highly competitive market prices tend toward marginal costs plus some small profit margin. Think agriculture or plastic trinkets. The fact their prices and profits have remained high for so long indicates lower competition, likely due to high barriers to entry from competitors and high switching costs for consumers.
> In a highly competitive market prices tend toward marginal costs plus some small profit margin.
Competition in general tends to drive prices down, yes; but competition was not what the post I was responding to claimed "should" make prices go down. That post was claiming prices should go down simply because AWS is investing more resources in building capital as they scale up. That is not true.
> The fact their prices and profits have remained high for so long indicates lower competition, likely due to high barriers to entry from competitors and high switching costs for consumers.
It could, but it could also indicate that this market simply has not reached equilibrium yet. I don't think it's plausible to claim that AWS has no competition in this area, or that barriers to entry are high; there are a number of huge corporations investing a lot in this market, and plenty of smaller players gaining customers by presenting a simpler interface to an AWS back end (e.g., Heroku). Switching costs might be high, not just for AWS but for any provider in this market, simply because there is so little standardization in how infrastructure is specified and controlled. That would lengthen the time to reach a competitive equilibrium in this market.
It is worth remembering that for new technology markets, it can take a long time for a competitive equilibrium to be established. A good example is the automobile market; in the US, for example, it took many decades for car prices to be driven down to marginal cost and for all of the various market players to search out and capitalize on all the possible competitive efficiencies and strategies. Part of that was also that it took decades for the auto market in the US to become saturated, i.e., for most new car sales to be replacing old cars instead of getting a car to someone who had never owned a car before. I don't think the market for AWS-like services is anywhere close to saturated, which means we should not expect a competitive equilibrium; instead, we should expect exactly what we see, large market players trying to capture as much market share as possible during the growth period, just as major automakers did in the US during the growth period in the mid-20th century. Capturing market share in a growth period is a very different game from squeezing out efficiencies in a market near competitive equilibrium.
Way to miss the forest for the trees. This is a discussion about how someone feels about the prices being charged. The argument is the poor old Amazon deserves slack because they haven't raised costs.
My point is that their prices could have gone down without them even feeling any pain because their costs have gone down. They don't deserve slack for keeping prices the same. Keeping prices the same just means their competitors either suck or they sit in an oligopoly together.
Perhaps. All I meant was to show there's never been a time where they've had to acknowledge pricing something sufficiently badly that they had to go back and adjust prices up, perhaps after seeing how customers were using it.
The slack I'm giving is for the complexity in their billing, not in the price itself.
Doesn't that kind of problem get easier at large scale? It's like any time you have a limited resource to share out. e.g. if an ISP has 150mbits of connectivity, it would be difficult to provide a '100mbit' service to two homes - one busy user will clearly hit the other. But if you have 1500mbits to share out to 20 homes, the service will seem much better. As you scale up, it becomes easier to provision less per customer.
Economy of scale is different when you’re providing a cloud business because your costs (and hopefully revenue) are dominated by the head customers. Imagine a company whose business model isn’t profitable for AWS becomes the next Dropbox. Suddenly the tail can’t subsidize the head. And even if it could, the business is shot.
You always need the head to subsidize the tail (e.g. free trials). You can’t let the head become unprofitable.
All these are problems, yes, but my point is they become more manageable when you have more customers. Having Dropbox as a client would be exceedingly difficult if you only have two other customers, but it gets easier as you add more customers and scale up.
I think that works for ISPs because their service area and customer base are arbitrarily defined and within their control. They don't have to worry as much about a datacenter suddenly opening up in the middle of a neighborhood that throws off the profitability of everyone else.
And they can also impose things like bandwidth caps to curb some less-profitable customers' usage. But cloud providers, by design, are charging "per use" and expected to scale "indefinitely"... at least for expected use cases.
I'm as anti-AMZN/AWS as anyone these days, but to their credit, Amazon has their own teams of consultants who will come to your business for free to analyze your usage of AWS and tell you how to save money. They openly joked about how it was their job to get customers to spend less money with their company.
If you feel like you're way overpaying for AWS, you're much likelier to look out for alternatives. So if they help their biggest accounts save some money, it'll net them way more in the long run.
We were in no way one of "their biggest accounts". We were a couple guys in a cramped office over a mattress store, with a production service so small we could have run it from my old laptop. I guarantee it cost them at least 10x more to send those consultants over to us than we spent on AWS in a year, even before the optimizations they pointed out.
Yes it is, but the gp is also right that it is part of a well calculated and proven way of keep existing customers and getting new ones. $$$ is the reason they do that, else this would not exist.
Right - it gives the illusion that you're 'getting an amazing deal and wow these people are great' when it reality you're probably getting gouged without the consulting and have to opt in for fair prices.
I mean it's a retention tactic at the company scale but for the individual consultant they're genuinely trying to help you out.
I frequent a local family-owned Italian restaurant/pizza place and my usual order jalapeños, banana peppers, and roasted garlic so I do a build you own. Totally fine, happy to pay the price. But one day I come in and the woman working the take-out counter sees my order before it goes in and tells me that if I order a "meat lovers", remove two toppings and sub the rest it comes out way cheaper.
So now I'm the vegetarian that always gets a meat lovers and she makes the joke every time I come in.
I don't think it's weird or nefarious that a business doesn't have completely consistent pricing and holding it differently can save you money. When we spec out services on AWS we're using the official calculator when deciding whether it's "worth it" so to us at least it genuinely feels like we're saving money.
> So now I'm the vegetarian that always gets a meat lovers and she makes the joke every time I come in.
I've seen that before, and I've also seen the opposite where people buy the veggie pizza and then add a meat or two on top to get a cheaper supreme pizza.
I did an online food ordering startup a ways back, and it was very surprising how few of the restaurant owners we talked to had clearly thought through pricing strategy for things like this.
We had originally gone in with clever ways for owners to describe how different changes would affect the price of an item, but ended up landing on just a giant matrix where they could input the end result price for any combination. The owners couldn't articulate their own pricing well enough to model it well.
Do you really think any company of any size doesn’t do a rational cost/benefit analysis when deciding when to build vs buy or when and what to outsource?
If you are any decent size and you pay for a partner’s “Well Architect” overview, they will give you the money back in credits. I am not at liberty to say how much we spend, but it’s not over a million.
If this team shows up, it means you're spending so much money stupidly that they are afraid a competitor will easily steal you away with an offering for 10% of the cost. Their job is to come in and get you scaled back so another sales team can't get a foot in the door.
If you have an MSP - and if you are a small company they can pay for themselves just by being able to get bulk discounts by coming under their organization - they will offer you one and you can get reimbursed via AWS credits.
I've been horrified at how bad this is on Azure, I'm sorry to hear that AWS has the same issue. I've signed in some mornings only to discover that I've been charged $20 for a MongoDB instance that's been sitting idle all night. Support can only come back with some hand-waving about RUs, a term that defies any clear definition.
AWS you can poke around and figure it out, Azure is a Kafkaesque nightmare of infinitely confusing UI/UX, a permission system that is like trying to hit a moving target in a dark room, being moved by someone that hates you, doing anything seems needlessly complex, and trying to replicate/test anything locally is horrible (looking at you Azure functions).
I'm trapped in this mess because the nature of our Client is that they'd consider anything besides Azure as a slap in the face, but it's been absolutely the most miserable task I've taken on in years.
As a bonus I'm getting side-eyed by teammates who have a naive faith in The Cloud and can't comprehend why this is taking so long and why I can't just give them a cost estimate.
As an aside, unless I'm reading this very wrong, this is the first time I've seen the use of "my sibling" as a gender-neutral replacement of "my brother/sister" to express empathy. I'm delighted by it.
Semi-shameless plug (I work for Oracle) but have you tried OCI? You might find both the console UI/UX and the cloud calculator pretty decent. That's just my personal feel, coming over to work as an Oracle cloud consultant with previous AWS/Azure experience. I'd argue it's normal those two are where they are; AWS is now a spaghetti of products, while Microsoft are anti-KISS by DNA.
AWS might be a spaghetti of products, but they get out of my way and they’re straightforward enough to use/ignore; and that’s pretty much what I want.
Honestly, I wouldn’t touch something from Oracle with a 10 foot pole, sorry. I’m sure the UX is probably pretty good, but I’m staying away from that company if I can avoid it-too many horror stories, way too “enterprise”.
Point taken. Every time Oracle is mentioned it's in the context of horror licensing stories. I get that - but would it make a difference if I said OCI is actually different?
Take this with as much salt as you want but they're really trying to put that in the past. OCI pricing is super transparent, simple metrics, cheaper than the incumbents on every front, there's no "call us for pricing", there's a transparent 30% discount for committed spending (before you even get to talk to a salesperson) and the dev team has been given free reign to act like a startup and ignore the corporate machine. I'm not a marketing person but I do believe we have a very good offering and it's a shame it's being ignored solely because of bad blood.
Another point re $$$'s. Predictable, low pricing is actually a key Oracle cloud selling point. Following the recent wins with Zoom and 8x8 we've put up a comparison sheet [1] and a workload estimator [2]. Then there's the general cost estimator [3]. No hidden costs, and no calling required.
You have a company that charges licenses based on the physical core count even if you run the product in a VM using only a single core, but then they also want to be your cloud supplier.
The only strong point of Oracle is legacy product lock in, you would be crazy to pick them for any new initiatives.
This is a problem across enterprise SaaS companies. At my last company (small enterprise SaaS biz <$50M ARR), I got so tired of this consuming so much of my time. I literally hacked together some code so I could point at any line item from any bill generated by our "system" and it would try to answer "why did I get billed $x for this?" so a five year old could understand.
I wish there were SaaS products that would effortlessly audit enterprise SaaS bills with support for more than just the top 1% SaaS products.
I was poking around with their CLI to see if I could put together a cron job - this artifact now costs more than $X so delete it. Unfortunately if there's a 'get current charges' command I haven't found it.
I have a ticket open with MS support. I suppose it's to their credit that they're listening to me but it's been a few weeks with only the aforementioned hand-waving. I started just sending them screenshots showing that their own dashboard showed no usage during the times I'd gotten charged.
It isn't simple at all! Not only is it complex and fairly daunting, you also need pretty deep knowledge of AWS to fill it out correctly. They eventually introduced an improved calculator that dropped the "simple" branding: https://calculator.aws/
> The fact that there are consultants who specialize in figuring out AWS billing was, in retrospect, a warning sign.
Well, there are whole organizations in my company dedicated to installation, procurement, provisioning, ordering(and tracking of said orders), of datacenter resources and hardware.
Billing is really hard. I've had projects that literally sat on the shelves for months while we finalized our billing model, after realizing it would significantly increase the cost for our largest customers.
Have you ever used AWS? There's a whole suite of tools they provide around pricing and budgeting.
> It's quite a tour de force how Amazon have taken "separation of concerns", applied it to web services and used it to create complex and difficult to understand or predict pricing to print money.
Is applying separation of concerns to web services really that bad? Look, if you're a small company with a simple product, you can just put your stuff on a few EC2 boxes and pay the monthly bill for that. At that size, your infra costs are going to be dwarfed by your other costs of doing business anyway. If you're a big business, you can literally pay people to keep track of this stuff. You've got the extra money, because now you don't employ data center architects, server engineers, etc. AWS is able to "print money" because it brings a lot of value to many businesses.
> Have you ever used AWS? There's a whole suite of tools they provide around pricing and budgeting.
They don't work. To elaborate, they fail to provide the user with information with enough clarity to allow him to form accurate mental models of what is being used,what it costs, and what is the implication in cost of applying some change.
Heck, at any given moment it's unexplainably hard to tell exactly which AWS services are you actually using, and being billed for.
The only reliable info on cost it the invoice you receive at the end of the month. Even then, the AWS guys felt the need to develop a full blown machine learning service to help customers predict what their next invoice will be.
It meticulously breaks out per-service costs across whatever dimension you want. "EC2-Other" is rolled up in the chart, but it's broken out by each individual item further down in the list.
Isn’t that exactly what the person you replied to said? You’re showing what you’ve been charged for previously. That part works, at a certain level of detail.
That doesn’t show what you are being charged for right now, nor does it help you predict what your total costs will be at the end of the month.
You bring up a good point - it would be a good idea for AWS to set some sensible max budget defaults for new accounts to prevent surprises.
Since they don't, if you are literally so concerned with your current costs for this month, you can set a budget and alerts for whatever you want, so that you don't exceed some cost. Also, they show you your current cost for this month, up until the moment, as well as a daily breakdown of your costs.
> sensible max budget defaults for new accounts to prevent surprises.
Amazon, in my experience, has issued organizations a one-time credit in cases of surprise bills.
The bigger crux of the issue, is Amazon has democratized CapEx into OpEx, and it's too easy to be ignorant of the details of infrastructure planning, which is rolled into AWS pricing.
Before AWS, IT teams had to plan for and estimate costs prior to procuring hardware. This is no longer needed with ondemand, programmatic infrastructure.
Engineers, who (reductively) focus on solving business problems with code, aren't necessarily thinking about the costs of deploying their solutions, the way Ops and IT would be.
The couple of AWS acquaintances I asked about this said that this kind of feature doesn't exist because:
- There's almost no demand for it from the majority of their customers.
- Billing code is a big, scary, tangled mess and also happens to be the one area where you really can't afford to introduce bugs and make mistakes, so meaningful change is already exceptionally difficult.
I already got a surprise, because in one place it says t2.micro and t3.micro instances are free for twelve months.
But, if the instance runs Windows, or the zone is this or that, or something else I don't know, the only free tier instance is actually the t2.micro.
And then there's the Unlimited option, and CPU credits, where if your CPU use goes over 25% for some time, it will either: Be throttled so slow as to be useless, or, not being free and you still have to pay for each hour of use.
> You bring up a good point - it would be a good idea for AWS to set some sensible max budget defaults for new accounts to prevent surprises.
I didn’t say anything about this at all. It would certainly be a nice feature, and one people have asked for since the very beginning of AWS.
> Since they don't, if you are literally so concerned with your current costs for this month, you can set a budget and alerts for whatever you want, so that you don't exceed some cost.
Again, this is a strawman. This isn’t a response to anything I said.
If you’re trying to understand how much a new deployment on AWS is costing you, and find ways to optimize that, you don’t care about the total budget of the account, or having some auto shutoff. You need detailed, real-time information from Amazon, which I’ve never seen in the console.
It’s been several years since I’ve had to be digging around in the billing console, so I have no stake in this discussion and things might have changed. Based on other comments being made around here, I really doubt anything significant has changed.
I was just trying to clarify a comment you completely misunderstood.
> If you’re trying to understand how much a new deployment on AWS is costing you, and find ways to optimize that, you don’t care about the total budget of the account, or having some auto shutoff. You need detailed, real-time information from Amazon, which I’ve never seen in the console.
No one who is working on this cares about real-time by the second data - the daily cost is enough.
> Again, this is a strawman. This isn’t a response to anything I said.
OK, what are you trying to say? If my response above did not answer the question, can you give me a one sentence summary of the point?
> can you give me a one sentence summary of the point?
>> I was just trying to clarify a comment you completely misunderstood.
Alternatively,
“The billing UI and billing infrastructure is a major weakness of AWS when it comes to helping people understand what their usage will cost.” You can clearly see this is true by the large number of people who complain about it.
You can opt in for detailed billing report, and get detailed daily reports on what you are being charged on. You can load it in any db of your choice and ask all the questions that you want.
This is really terrible blanket advice, CF is one of AWS' worst products in my extensive experience with it (My team manages a ~$4 million monthly bill and locked into CF well before I joined). It has a number of significant limitations that aren't applied to terraform, it's pretty important to understand those tradeoffs if you're buying in.
Given that AWS supports terraform, it's really on them to provide a calculator for it as well. We're large enough that we just spent an obscene amount of money on employee time and effort to track our billing (lots of automated tagging), but until we did that it wasn't uncommon for a couple hundred thousand dollars of unnecessary charges to hit our bill every month.
A lot of people use Terraform (my shop does) and I've found a good terraform setup way better than CF templates. You can do tags in Terraform too. You should be forced to use their already bad tooling just to get correct cost estimates.
I'm training on AWS and part of an AWS infra team - but we use Terraform partially because we intend to have production support at our company for all three big US clouds, and want the IaC layer to be uniform.
I get that there are times to dive all the way in, but there is still a part of me that says "Is this what the Internet is now?"
That is correct; I've always found it a missed opportunity in Terraform. I still like it a lot as a tool.
A unified abstraction layer would certainly work for a subset of cloud offerings. Think the more basic stuff like EC2, VPCs, etc. Platform specific extensions could be handled as optional arguments to the unified objects.
That said, even though you need custom Terraform definitions for each cloud vendor, using one IaC tool still beats having to use a cloud-specific one for each of your deployments. The parent's point stands.
Learning terraform for the various providers = learning the different syntaxes and design philosophies of different python libraries.
Learning the different IaC tools for each provider = Learning Python, Javascript, and PHP.
Also, Terraform does state management and can be used to package deployments of apps to different platforms. Our IaC pipeline uses Terraform to deploy resources to AWS and several other cloud toolsets, all in one language.
The provisioners are the easy part - understanding the ecosystem and how the different parts of the cloud providers relate is the harder part. I routinely go back and forth between C#, Python and JavaScript using the official AWS SDKs and I understand how to accomplish what I need in either language. Using those same languages, I wouldn’t know where to start with Azure or GCP.
Using Terraform vs CF is easy compared to using AWS vs Azure.
Moving from Terraform to CloudForm is as pleasant as chewing glass, and the last thing we need is more AWS lock-in unless you're running an AWS consultancy.
If you are using Terraform you are still “locked into” AWS since all of their provisioners are cloud specific.
Also if you are at any type of scale, your IAAC choice is the least of your problems.
Have you actually done a realistic project plan and estimated how much it would cost to migrate infrastructure from one provider to another once you reach any scale - including regression testing, auditing, training, etc? Do you really just use your cloud provider to host a bunch of VMs? Even then it’s not that easy at scale.
Sigh, I should've gone ahead with the longer version of my previous comment that I deleted because I felt that pre-emptively responding to this would be unnecessary by now.
Yes I've dealt with this issue (to be more precise multi-cloud rather than migration, also used other TF providers in tandem like the Kubernetes one), and yes your TF resources are provider-specific, but being able to handle it all with the same tool instead of having to deal with an awful vendor-specific tool (to be charitable, because the reality is having to deal with mutiple ones and having them inter-operate through bash glue) vastly helps in reducing and controlling lock-in, and I would argue it's the baseline step you have to take if you don't want to risk your whole business on a single cloud provider not going to the gutter either because they've grown too big for their own good or because they're being squashed.
CloudFormation means jumping into the pit of lock-in and you can only climb back out by digging your fingers into the dirt wall, Terraform means you have climbing gear to rappel down and if you have to, get back up without as much hardship. Sure you have to put the gear on but your descent is controlled and then you can climb back up at your own leisure.
Is the only thing you are doing with your cloud vendor is using VMs? Not S3? Not IAM? Not SQS? SNS? You don’t have to go through security audits? Hybrid networks via VPNs/Direct Connect? You don’t have a massive amount of data that has to be transferred? Your DNS entries? Your build pipelines?
No longer at that gig, but in order: Kubernetes applications using Kubernetes abstractions, S3 API abstracted away at the library level, RBAC done through Kubernetes (which includes IAM integration), we did use SQS and SNS but those were easy to replace given our abstractions. No security audits (third party ones at least, we did have scripts and checklists for deps and GDPR compliance). No hybrid networks, we either had an internal API/frontend for our services or we used bastion servers for SSH proxying. Wherever data transfer was a major issue we stayed within AWS, but that's our fault for going with AWS in the first place, which doesn't belong to the bandwidth alliance; a hard migration had been discussed and informally planned for but punted for later. DNS can be handled cross-vendor easily with Terraform since it's easy to set a module's outputs to another's parameters. Pipelines we ran with CodeBuild and hosted in ECR, but running a single command and docker pushing to a repo is not something I would even consider as a migration pain.
Everything seems easy until you actually get your project management organization involved, and your IT staff, your QA, your business analysts, your compliance department, start allocating cost for your staff etc....
Well that's because you're locked-in :). If you're already locked in, you're screwed and have to work your way out or live with it and the consequences that come either now or later.
If I get to make the call and I care one bit about the business beyond next quarter, I would always have a clear way out to heavily reduce risks to a situation that we have seen play out many times before with the Oracles and IBMs of the world.
I'm not sure why that would be terribly important though? A month of delay in optimizing your services isn't going to make or break an average company.
You're absolutely right, and also absolutely missing his point. As a massive user of AWS for a long time, yes you can know exactly to the sub-detail line what you pay for, no you can absolutely not have an overview or good estimation of mapping what a change in your software stack / setup will cost or save you. Unless we're talking "remove an entire service" and then duh, yeah.
First, the obvious that if a massive surge happens (attack, bug, ...) on prem stop serving, aws surprise-drain your bank account. Yes it's possible to have fail safe to avoid it, it's also very common to not have them or that they're not tested or working.
Second, the "in-between" costslike bandwidth from service to service, iops on your disk, ... They mostly don't exist on prem, you either have enough of it and it works or you don't and it doesn't work and you need to go buy a bigger pipe. They're one of the biggest source of surprise cost on AWS.
Eg take a basic data storing application, the cpu and storage space are easy to estimate and are the same as on prem, but suddenly you also need to estimate your iops, and you realize you have no idea about it. Do you have an on-prem database running somewhere ? How many iops does it need ? Unless you've had to figure it out, you have no idea and will never know on prem.
If the AWS tools fail to provide you this information than you can just tag everything (we use scope, stage, service, region, owner tags, example: global, dev, hadoop, eu-west-1, data-engineering) and it is trivial to generate a cost report where you can drill down by environments, teams, services, etc.
I’m often fielding questions over on r/aws about how to do X on AWS where X is something simple. I then ask are they trying to solve problem X or are they trying to learn AWS. If it is the former, just use LightSail. Outside of serverless applications, I wouldn’t want the complications of full fledge AWS for a small project and I get paid to know the ins and outs of it.
AWS (or GCP or Azure or ...) value proposition depends on you.
For a small/startup, the ability to execute quickly without sinking capital into a product that will likely pivot anyway is a superpower.
When you get bigger, stuff gets dicey. Paying a premium for a "X as a Service" made sense when you had no people, but at some point the cost structure may not make sense or your needs are at odds with how the service delivery works.
When you work for a big company or government agency, things change again -- you don't hire a database guy for a project, you have a project team that builds stuff, and an ops team that exerts the minimum viable effort to keep your stuff running or an outsourced ops team that does the same, but worse. At that point, you start looking at how much it costs to pay AWS to provide and SLA vs hiring more competent (ie. more $$$) ops people to keep it running.
The long and short of it is that AWS is in a position to rake in cash up and down the spectrum -- all they need to do is be competent, which they are very good at.
> There's a whole suite of tools they provide around pricing and budgeting.
It could be there are such tools. But they are impossible to find, setup or understand for someone who is not already an AWS expert. I recently got a $500 bill from AWS. It was impossible for me to find out which S3 buckets (or even specific files) caused these costs. I looked at this for half an hour and have used AWS before. They just don't tell you how to obtain this information. Or maybe they do, and it is so well hidden that it might as well not exist. This was a horrible user experience. As a customer, I want to know what I'm being charged for. The fact that AWS make it so difficult make me never want to use them again.
You can get cost breakdown on a more granular level by taking advantage of tagging. You can tag your resources logically and get a break down by tag.
The biggest mistake I see is that people jump into AWS whole hog without doing the research or if they are large enough, hiring someone who knows what they are doing.
I don't understand why I even need to do manual things such as "tagging" (besides learning what that even means in AWS) to find out something as simple as why I'm being charged as much as I'm being charged. I'm sorry. It's inexcusable.
And large organizations would complain about why they couldn’t group their resources like they want to. Having individual bucket costs would be of very little use for cost allocation and to tie back to something like a department. For instance, we tag every resource created with the same CloudFormation template with the same tag. We can get billing based on an “application” and all of its related resources.
Just like AWS is not understandable to someone new, I feel the modern front end ecosystem is royal cluster but yet and still thousands of people understand it after a 12 week boot camp.
You could have both. Let advanced users tag. But give less sophisticated users insight into what they're being charged for - without having to go through a 12 week boot camp.
Well, was most of that $500 for storage costs, or for network? Surely you don't expect the monthly bill to include details about particular files or bucket sizes? Have you enabled CloudTrail logging and/or VPC flow logs?
It's not my intention to victim-blame here, yet do bear in mind that there are many, many ways to use the services and to expose resources to the bottomless appetite of the internet. Maybe Digital Ocean or Linode or the like would be a better option for starting out.
It's literally called "Billing" and the "Cost Explorer" is the first item in the navigation when you go to the page. I've attached a screenshot and highlighted it for you. FYI, this month is the first time I've opened the AWS GUI in 4-5 months, and it took me about a minute to find the billing page and drill into our AWS costs. I've provided an image that shows how to get to the cost explorer from "Billing" (you can use the search box available on the AWS account landing page to type in "Billing" in order to access the billing page of AWS).
This does not show the costs per bucket. Specifically, I was interested in the costs for data transfer for each of my buckets. As I said, after half an hour of looking into the "Billing" in AWS, the Cost Explorer and googling around I simply could not find a way to obtain this information.
You're right, it's shitty of them to not provide that out of the box. However, if it's such a big problem for you, as others have pointed out, it's easy to solve. If you are paying $500/month for S3, you must be processing quite a large amount of data - as another poster pointed out, there's an easy to fix it. If you weren't going to use S3 for this, what would you use instead for this use-case?
Why do people think that they can give Amazon a credit card number and expect everything to be easy enough for them and suitable for a large enterprise? AWS offers something for beginners - Lightsail. When that outgrows their needs, they can peer their behind the scenes LightSail VPC to a full fledge VPC.
I look forward to migrating to Linode's S3-compatible object storage. I appreciate that you are agreeing that it's shitty of AWS. But I disagree that it's "easy to solve". Like I said, I already spent quite some time looking into how to solve it. And having to set up logging for a bucket (and learning how to do this in AWS's complex interface), and if I understand correctly creating another bucket to receive those logs, and learning how to then obtain and interpret the results? Maybe you're above my paygrade but I do not consider this easy.
What drew you to using S3 in the first place while also discounting the risk that it might get a lot of use, and thus a lot of cost? Like, if a 100MB file in a public bucket got a bazillion downloads, did you not want that to happen?
If this is an ongoing issue and you can’t get the information from something that is in front of S3 (cloudfront/your application etc) then you can enable bucket logging in the settings (which costs!)
> Have you ever used AWS? There's a whole suite of tools they provide around pricing and budgeting.
Yes, I use it extensively.
> Is applying separation of concerns to web services really that bad? Look, if you're a small company with a simple product, you can just put your stuff on a few EC2 boxes and pay the monthly bill for that. At that size, your infra costs are going to be dwarfed by your other costs of doing business anyway. If you're a big business, you can literally pay people to keep track of this stuff. You've got the extra money, because now you don't employ data center architects, server engineers, etc. AWS is able to "print money" because it brings a lot of value to many businesses.
It's not bad per se. The real problem I often see is that many people within the typical org aren't aware of how these costs can accumulate, especially when you have multiple DTAP environments, microservices, and so on.
Another problem is the days of "put your stuff on a few EC2 boxes" seem to have been overtaken by the cloud native, microservices, k8s trends of the last few years. Personally I 100% agree this is the way to go for many (if not most) people -- and for this purpose Digital Ocean is just as good -- but what I see more often is people diving straight into the deep end, because "Is applying separation of concerns to web services really that bad?"
I also think even big businesses underestimate the total costs of these all-in cloud infrastructures. You indeed need to pay people to keep track of this stuff -- multiple people. Your cloud costs + your devops/platform personnel costs can get to be a significant % of your total IT costs. Is it really necessary?
Just because your business has a big market cap or significant revenue from digital doesn't necessarily mean it needs any of that stuff. Chances are you can run it all on the same couple of EC2 boxes you ran it on when you were still a small company.
I think we are probably 90% in agreement on things, just talking about it in different terms. I think AWS/GCP have both gotten overwhelming in terms of how many different services they provide. Many people fall for it and end up playing AWS service golf (https://news.ycombinator.com/item?id=23274668).
Maybe a good analogy is to think of AWS as C. It's flexible and powerful, but it's easy to shoot yourself in the foot. That would explain why a lot of these "> magical-tool deploy" services have emerged, promising to abstract away AWS.
>> Another problem is the days of "put your stuff on a few EC2 boxes" seem to have been overtaken by the cloud-native, microservices, k8s trends of the last few years.
I am not sure where you get the notion that any serious user goes with the "put your stuff on a few EC2 boxes" approach. Every single AWS project I worked on had a cost calculation phase when we investigated what combination of services could be the most cost-efficient for a certain problem.
>> cloud-native, microservices, k8s trends
It is kind of funny that you try to put on one end AWS the company that made cloud computing popular (and microservices with it) and the biggest player in the segment and "cloud-native, microservices, k8s" on the other. Trends come and go, AWS is here for the long run.
They’re literally entire companies of 100+ employees selling a product that helps you figure out how much money you’re spending on AWS https://www.cloudability.com/
From where I stand as someone who isn't using Amazon figuring out what my costs will be is difficult, impossible and would require wasting money on various tests.
With something like digitalocean my costs are fixed. I might have misjudged how much I need but can scale up easily or down (with effort).
Will Amazon offer fixed pricing? Does Amazom offer fixed pricing?
Aws is definitely not for hobbyists, or small applications, its a replacement to a datacenter with a lot of variables, not a VM. Your better comparison would be DigitalOcean vs Lightsail (AWS). So yes.
They are awful. I have tried and tried to list the cost by resource (like you can in Azure) and I just can't figure it out. I think I might need to enable some special pricing tools (coincidentally not free) that Cost Explorer don't expose.
When every company workload and footprint is entirely different what alternative do you propose? Different companies have different needs and different cloud services suffice those needs in really different ways. You can run your database on relational, non-relational or flat text file backends. They all have different performances characteristics and different costs associated with running them.
Not to mention that if AWS doesn’t price everything correctly someone will find out how to abuse it. For example, if AWS only charged for outbound bandwidth on S3 I could store petabytes of archives for almost free. If they only charged for storage, I would host all my high access, small size files on there for free. S3 is an oversimplification but it applies to other services. If they don’t charge for every way they can lose money on a service, someone will find it out and abuse it.
Yup; if you go with AWS, expect to build a pretty complicated setup distributed over many services. Even the basic LAMP / Wordpress stack mentioned in a sibling comment and often used in tutorials starts at $50 / month once you add everything up, and that's without any traffic.
Although I guess Lightsail would be a lot more affordable.
I faced a similar issue at the company I work for (where a PM had activated an instance for months and racked up a $9k bill). I've been spending my coronavirus time hacking together a service that hopefully makes billing on AWS more clear. It does cost optimization and anomaly detection right now, and I plan on adding more features as I start to see more use cases.
Check it out if you'd like, and let me know what you think!
I am not sure what you are talking about. We have many millions of yearly AWS cost and it is both predictable and understandable down to the last cent. One additional thing, I am not sure it occurred to you but some infrastructures operate on AWS are not sending any significant traffic outwards. S3 is used as a cheap and reliable data warehouse storage layer for exabytes of data just fine.
There's a level of complexity and nuance in AWS that casual users or those on the outside do not quite comprehend. And honestly it takes quite of bit of experience and interaction to fully understand some of these things. It's not that you're getting charged "twice" for S3, it's that there are separately line items in the CUR for S3 data transfer, S3 storage (and S3 API, S3 inventory, etc.). It's confusing to a layperson but this level of detail is actually quite powerful. You can tune your application to use just what's needed.
Problem is they are all built up of many smaller pieces as well - you may need the information about them at different points. For ex. EC2 drives are from EBS. if you need a load balancer ELB. You probably use S3 for some or other reason (log exports etc at least). You want a db - RDS makes it easier. Dynamo is literally a button click away. Kubernetes? Well there is EKS.
Yes you can probably launch your own cache fleet using a bunch of ec2 boxes - but you don't get all the low level hooks Elasticache gets. You don't get the pricing discounts either (I remember cross zone replication or something being distinctly free for ec2 but you pay for network. Same with dynamo - access from all zones cost the same pretty much IIRC).
If you are going to make the dive, it is worth spending the time. I would do a small overview bootcamp somewhere so that you know what's available and then start digging in for each new service you take up.
I think it's because naming things is hard, that link proves it by coming up with worse names for almost everything they tried to rename, and often far, far worse.
Imagine the confusion if S3 were called "Amazon Unlimited FTP Server." That gets every word wrong, except that "Amazon" is merely redundant. It's not unlimited (having to pay for a thing is a limit), it's not using FTP, and it's a service, not a server.
Or if VPC was "Amazon Virtual Colocated Rack". A "colocated rack" means your computer in their datacenter. They actually have this service, it's called Direct Connect, because you can actually
Lambda does require you've got some vague notion of what lambda notation is. But "AWS App Scripts" suggests it's for mobile "apps", but it is not specific to those. And it suggests it's only for scripts, but you can run an entire application on Lambda just fine.
Or even DynamoDB. They recommend "Amazon NoSQL." They're not offering many NoSQL databases, just their proprietary one: DynamoDB. They have a service that offers many relational databases and that is called Relational Database Service.
Oracle Cloud Infrastructure has been trying really hard to make sure things are named in as straightforward a way as possible. It was a very early decision pre-launch, and unsurprisingly not that hard to stick to. Marketing people didn't argue, either, but maybe that's a difference between the marketing team backgrounds? Enterprise company CIOs etc. don't want to have a translation guide when it comes to making purchasing decisions.
Some of the "WTF, how did they come up with that name" with AWS comes entirely down to the public name being the internal project name, e.g. Snowball. Various engineers and managers have facepalmed hard when marketing decided to go with the easiest option and use that name rather than come up with something meaningful.
And then I’m stuck with the 4th (?) place cloud provider with a horrible reputation. Choosing Oracle definitely would violate the CYA rule - “No one ever got fired for choosing $what_everyone_else_chooses”. I would be safer choosing Azure or even GCP.
Er, I'd argue that aiming for safety and/or following the crowd aren't really the ingredients of a sound decision making process. On that basis no challenger would ever stand a chance.
That’s not my problem. Everyone looks out for their own self interest. Let’s say that OCI and AWS both statistically had the same uptime. If AWS went down for a day no one is going to question you as CTO for choosing AWS, besides you’re in the same boat as everyone else. If Oracle Cloud went down, everyone is going to be questioning your decision.
But, the saying initially was about IBM. The CTOs in the 60s and 70s who chose IBM instead of one of their competitors in hindsight made a good decision. You can still buy hardware from IBM today that can run COBOL programs written thirty to forty years ago. All of IBMs competitors are long dead.
From a recruitment standpoint, you can easily find someone who knows or wants to learn AWS or Azure - Oracle Cloud - not so much.
Like when you think about if a startup had to sell people on a single one of these services, they would have to actually say "like an unlimited FTP server BUT" and thats the only way they could get into board rooms and they would spend years on just that wrong and skeuomorphic branding just to get off the ground
Whereas Amazon doesn't have to do that, and doesn't have to explain the skeumorphic stuff to anyone, they'll just mention it in some conferences about what you can do now, the end.
Yeah, thats really cool. It doesn't mean there isn't a better way, but I can see how it isn't as arbitrary as I thought.
Services like S3 are referred to as „object storage“. AWS Lambda is a service for „serverless compute“ or „function as a service“ (although it‘s debatable if these are good names for the concept).
My point is that definitely are named for these concepts but AWS uses brand names which is quite confusing for people who are new to AWS.
Why? They could have simply named them with AWS Serverless, or AWS Compute, or whatever.
Using these brand names might make it easier to evolve them later, or they really wanted nice sounding names because eventually you'd have to write out long phrases with the AWS prefix, and that's probably harder to market.
Kubernetes Engine, Compute, Storage, Memory Store, Cloud SQL, PubSub.... almost all of the main services do what they say on the tin.
The only downside is - ironically - it sometimes makes googling for help a bit tricker. Eg. Are you search for generic cloud storage or the Google product with the same name?
Why is lemon sugar flavored carbonated beverage called Mountain Dew? Shouldn't the soda companies use less convoluted language?
Why is Confluent not just called Kafka+? Why isn't Kafka called LinkedIn distributed subscriber service?
I think "codenames" are easier to reference and discuss once you get immersed in the ecosystem. "lambda" vs. "serverless compute" - which one do you want to say 30 times a day?
How long has the idea of “ubiquitous language” when talking about a business domain been around?
Everyone technical at my company from the CTO down can have conversations and talk about CloudFront, API Gateway, Lambda, S3,SQS,SNS,DynamoDB, Aurora, Fargate, CodeBuild, CodePipeline, CloudFormation, IAM, WAF, EC2, ECS,ECR, Cognito, etc.
I’ve seen developers come in and within 3 months it’s second nature.
I know the ins and outs of AWS pretty well, but when I see discussions about Azure or GCP, I’m completely lost. It’s not like I’m a stranger to the development side of the Microsoft ecosystem. I’ve been developing in C# for over decade and have used Visual Studio since 1997.
Probably because parallelization is not the end-all-be-all of management strategies.
Consumers expect most companies to act like a coherent unit, (purportedly due to Dunbar's Number), and when you don't have enough oversight or leadership everything begins to look schizophrenic.
"Self-organizing" is organized chaos. If nobody picks winners at the end it never stops being organized chaos.
This is the exact reason I prefer Azure. I can use the search textbox to find something and the name is usually pretty explanatory (but they do have some daft stuff, like 3 different queue offerings with pretty vague documentation on the differences).
To add: It might actually help with commercial sales - the obfuscation makes it seem like there is more to each product than it being a cloud-managed version of what is done locally.
This is typical in finance. Many of the financial "instruments" are given funny nouns that hide the true character of what they are (usually shitty loans, or stocks being sold at a mark-up).
>Why does AWS use such convoluted language? Is it because they're dominant and it adds friction to moving to another provider?
Having gone from AWS in my last role to GCP in my current, I can tell you with 100% certainty that for me, AWS' mnemonic device naming convention is far, FAR more effective in helping me remember which service does what.
S3? Storage. EC2? VM/compute. GCP's equivalent? GCS/GCE. I don't do a whole lot with VMs in my role but it takes me a few good seconds to remember "GCE" whereas EC2 is instantly memorable. Don't get me started on Google's many "Data X" services (Datastore, Dataproc, Data Transfer, Data Catalog, Data Fusion, Dataprep, Data Labeling).
tl,dr; the lizard part of my brain very much prefers AWS' naming style, and I have a hard time remembering GCP's services despite the descriptive naming.
I think that AWS offers more distinctive names, which helps experienced users a lot. EC2 is instantly recognisable as meaning Amazon's compute service, S3 is storage. GCE and GCS could stand for all sorts, Google brings up "General Certificate of Education" and "Glasgow Coma Scale" as the first results.
I suppose because of this, Google tend to use the full name in most of their documentation. It does make learning how to use Google Cloud as a beginner a lot easier though.
This is a 100 x better than their website. I've actively walked away from Amazon products because I could barely make out what it really was and if I could use it for the application at hand. Many thanks!
The counterpart to stream processing is batch processing. With batch processing you run a job every hour or so and calculate a result, with stream processing you immediately calculate the result you want as the data is coming in.
nowhere close to 100x. actually it’s worse than the aws names across the board.
the names have a logic to them but I agree it’s intimidating to learn hundreds of things at the same time - so you don’t. you learn the bits and pieces you need.
for example:
EC2 actually comes from Elastic Cloud Compute. You have Compute in the Cloud which also happens to be elastic.
S3 is Simple Storage Service. It’s a Service for Storing things. It’s simple because it’s just a key-blob storage.
Route53 is obvious if you know what runs on port 53.
Cognito is an identity service, “incognito” means unknown, “cognito” means known. Polly is a joke about how parrots talk, the service does text-to-speech. Athena is a goddess of wisdom, the service is for querying databases. The name is frequently some sort of silly inside joke. Except Fargate, that name means nothing...
It's like Glacier, but it's smaller and moves more quickly. I think this is one of the dumb-codenames-turned-real that crop up from time to time. Fargate is another notable example.
> Chime
A chime is a notification. That's about all I got.
> Route 53
DNS runs on port 53, Route 66 is a famous highway.
> Sumarian
It's a really niche joke about the novel Snow Crash, that popularized the concept of digital "avatars".
Redshift - Oracle’s brand colors are Black and Red. Redshift was built to help Amazon “shift” from using Oracle.
Route 53 - DNS runs on port 53
Chime - when you want to talk about something “let me Chime in”.
Snowball and Sumerian. I got nothing. I didn’t even know what Sumerian was until I looked it up and I thought I had at least heard of most AWS services.
which i really appreciate when it comes to googling them for help with stuff. i can google "aws lightsail" and get results directly relating to what i'm looking for. have you ever tried to sift through the mess of seo marketing crap that comes back when your search term includes "VPS"?
look up your service. each service has a 1 line description that captures what it does.
eg Athena = query data in s3 using sql
i don’t really get why people cannot be bothered to learn what a service does and want to pretend they get it from the name. you need to learn the ins and out of the service. the name is the least of your worries.
> i don’t really get why people cannot be bothered to learn what a service does and want to pretend they get it from the name. you need to learn the ins and out of the service.
Names matter.
When your first experience with a product is an opaque name or description, it usually tells you one of two things:
1. The people behind it are not good communicators. If so, this will likely show up in many of the other product details: APIs, documentation, complexity.
2. The people behind it deliberately chose to be opaque. Perhaps they want to make the product seem more important than it is. Perhaps they thought marketing jargon would help sales. Etc.
You seem to want to frame this in terms of developer laziness, but I don't think that's right. Both of these are often reliable signals. The criticism here is well-justified.
Yes, this is a huge problem throughout the industry.
The tech world is gibberish. The real world is gibberish too, but the comparatively large number of tech products (because of the ease of releasing software) really exacerbates the problem in tech. Everything is a code name, in-joke, or initialism.
If this were limited to public products then you could write it off as overly ambitious branding, but you can tell it's an issue because internal products suffer the same problem. You know it's bad when you can't even navigate your own company's hierarchy because you don't know what half the team names mean.
Unfortunately, this will probably never change; there's too much precedent. Naming things incomprehensibly is fun, it creates a barrier to entry that makes developers feel smart, and the tech world doesn't value communication enough to change it. In a world where stackoverflow serves as de facto documentation for many major tech products because their own documentation is terrible, I hardly expect those same companies to value coherent naming.
After looking up the supposed clear names for OCI. I know what they are, but how does that help me ramp up any faster? It would still take me just as long to ramp up on the equivalent Oracle offerings as it did AWS. Heck, I know the Microsoft development ecosystem and it would still take me a few months to get any sort of proficiency with it where a company would be willing to pay me my asking salary.
I look at the front end web ecosystem and I think it’s all gibberish also and I’ve been using Javascript off and on since it was first in beta for Netscape. But thousands of people take the time to learn it.
> It would still take me just as long to ramp up on the equivalent Oracle offerings
The claim isn't that better names help you ramp up faster. It's that they make it faster to figure out which service you need, or if a service is relevant to you. That said, per the point in my previous post, it's likely that if you can't even clearly describe or name your service, you can't write clear documentation either. I've found that's often the case.
> But thousands of people take the time to learn it.
I don't understand what argument you're making. Is it: Since some subset of people will invest time in learning technology even if it's complicated, therefore it doesn't matter if it's too complicated? If so, you'd have to show that the number of people using it wouldn't increase were it simpler, with clearer names, etc.
That seems unlikely. The whole ethos of many successful companies -- Stripe, Mailchimp come to mind -- is to differentiate themselves by simplifying. It matters to the vast majority of users, developers or otherwise.
Since some subset of people will invest time in learning technology even if it's complicated, therefore it doesn't matter if it's too complicated? If so, you'd have to show that the number of people using it wouldn't increase were it simpler, with clearer names, etc.
Well, two points.
1. From a completely selfish standpoint, if people aren’t willing to learn about AWS, it helps me out a lot. I’m able to make more money by taking the time to learn it.
2. There are three credible cloud providers -AWS, Azure, and GCP - and Oracle. Amazon is twice as large as Azure and 5 times larger than GCP. It doesn’t seem to be hurting them.
Just like the obtuse clusterf%%% that is modern front end development doesn’t seem to be hurting the adoption by the $cool_kids.
> Just like the obtuse clusterf%%% that is modern front end development doesn’t seem to be hurting the adoption by the $cool_kids.
This is true, and it may even help with adoption among "cool kids." But the adoption could be much, much higher if it weren't such a clusterf%%%. There is an almost infinite amount of marketshare from a wide spectrum of technologies ranging from Wordpress down to "vanilla jquery" that could be eaten into.
Again, you can't look at some successful thing S lacking quality X and conclude "quality X doesn't matter." You have to compare "S with X" to "S without X". Sometimes a direct experiment is possible, sometimes not.
Doesn't feel like accurate description for Lightsail, nor a useful one. Maybe something like "simplified deployment and billing for some AWS resources, including VPS, databases, DNS, and load balancers"
(listing "storage" as something Lightsail does is kind of weird; of course, it does instance-attached block storage, you couldn't have a VPS without that. critically, it has no S3-like blob storage product, and I think that's what most people would associate the general word "storage" with, but maybe I'm wrong about that).
Sometimes I feel it is easier described by saying what its main competitor is.
Lightsail is basically AWS' version of Heroku and App Engine.
(i.e. a PAAS)
I wish Google would also do this. Many times on GCP's website or at Google Next you try to decipher what the product is that they are talking about, then you realise "ah, it is their version of S3, CloudFormation etc". If they just had said that at the start...
Of course, no company will do this, unfortunately.
I've seen companies post feature comparison charts listing competitors, they are very useful when switching from a different provider or discovering what the product does in terms of established player
Look for "Object Storange" for instance and in the row will be links to all the competing services, so you could pretty easily do this to learn about competitors through the one you know... At least for the big players.
You are missing one of the really amazing services: Snowmobile (https://aws.amazon.com/snowmobile/). It is a real truck, that connects to your data center, copies up to 100 PB of data and drives back to one of the AWS data centers and dumps the data there...
"Never underestimate the bandwidth of a truck full of hard-drives driving down the highway" or however the saying goes. Latency is extremely long, but the bandwidth is crazy once it's arrive!
Edit: Original quote (seems I accidentally modernized it a bit):
> Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway – Andrew Tanenbaum, 1981
I'd prefer to see this integrated into the AWS UX.
For an external crowd-sourced version, I'd like to see something like this with a column for maturity, and whether it actually works.
The classic AWS services are rock-solid, and perfectly sufficient to build a business on. Many of the newer ones are.... much less so. A green checkmark, yellow question mark, and red land mine icon would go a long way towards letting me know what I should and shouldn't use.
Sure, it should be right in the headline of the service's "about" page. The fact that people need this at all is UX problem.
If people are reading your "about" page and nobody understands what the hell your thing does, maybe your marketing, faux-tech, word salad is pointless.
>The fact that people need this at all is UX problem.
Is it a UX problem or is it intentional deep branding to further promote vendor lock-in? This is one reason I've been opposed to AWS since the early phases--I don't want to learn all of their stupid branded vendor-specific nomenclature.
If you are any sort of real business, you already “locked in” to more than likely a dozen or more third party services. I worked at a company whose entire workflow was tightly integrated into Workday via APIs, not to mention SalesForce.
If you have ever worked in healthcare, the level of “lock-in” that they have to their EHR/EMR and various other third party services. would make you cry.
> The classic AWS services are rock-solid, and perfectly sufficient to build a business on. Many of the newer ones are.... much less so. A green checkmark, yellow question mark, and red land mine icon would go a long way towards letting me know what I should and shouldn't use.
I’d like that and also the same thing for languages supported by various services and SDKs, which are sometimes supported and sometimes “supported”.
thanks. Fixed the first one. About storage gateway: as far as I could see (haven't tested it) you get a iscsi path which you can connect to from your own device. It uses S3 as the backend store for files from and to this device. Will try and take some more time to look into it (never used this myself)
I may be wrong but from what I understood, it's more or less a way to manage AWS Lambda functions (cgi-bin scripts), Docker, and a MQTT client connected to AWS on your GNU/Linux devices (raspbian on a raspberry pi for example, or a x86 pc).
However you still need Ansible or similar to manage the device so the actual value is kinda low because if you have a setup to manage the device, it's not much more work to manage docker and a mqtt client yourself. About running AWS Lambda functions on a the device, I think it makes sense for AWS to check the box "IoT edge computing with AWS Lambda" but unless you have a huge codebase in AWS lambda, it seems to be a bad idea.
In one sentence : "vendor locked half baked IoT platform".
The whole lot of IOT services are pretty confusing. Mostly because I have no experience with IOT/MQTT in general and it took a few days to actually figure out how to create and connect things (emulated). I actually bought some IOT devices (lamps, sensors) to try it out, but this turned out to be vendor-locked without any possibilities for MQTT. I reckon somebody with more experience in the IOT field could provide more insight in these things.
The IoT world is full of vendor locked solutions and platforms that you must avoid. And the joke "the S in IoT stands for Security" is very very true, though AWS Greengrass is actually pretty good for security, if you don't mind sending all your data to a USA company of course.
I would recommend to go opensource and skip the platforms that try to lock you in. There is a lot of solutions depending on how low-level you want to be. Like HomeAssistant if you want high level, or a Mosquitto MQTT broker and NodeRed and some dongles on a Raspberry pi if you want to be lower level.
Disclaimer: I'm VP of Engineering for software consultancy focused on IoT. We work with startups, midcap and several fortune 500s.
I agree vendor lock in must be avoided. I also disagree that Open Source is the best approach in all situations. Mosquito and Raspberry Pi are fine for smaller projects, but if you're going to push 100M or a billion messages, you're better off leveraging a IaaS solution like AWS IoT Core. At least, until you have a dedicated 24/7 opps team that can triage and support the solution.
The trick is to create the right abstractions and architectures to migrate your solutions off of AWS and onto another solution: Azure, your own, etc.
Do your sensors/lamps use Zigbee? I use the excellent Zigbee2MQTT library and a DIY Zigbee sniffer. Now I can use most vendor-locked IOT-devices (Hue, Ikea Tradfri, cheapo Chinese stuff) in my Home Assistant without the need of a separate bridge.
The devices have a ESP8266, and they should be easily flashed with custom firmware to connect to MQTT. But I found that OTA flashing did not work, and the only viable option is to solder apparently. I'm dangerous enough without a soldering iron in my vicinity.
Close, greengrass is about pushing real aws lambas down to intermediate site gateways (with certs and crypto and management), and then you can have very lightweight clients with almost no brain talk to that thing. My blurb would read, "vendor locked lambdas at the edge to proxy IoT widgets".
Absolutely not a criticism of your list, more a comment on how baffling AWS is these days. I stopped using AWS once they hit the point where I couldn't reasonably be expected to remember what all the three letter acronyms were. I still occasionally have to use it for S3, Route53 and IAM - but every time I log into the console I find that they've removed them from my "pinned" services in the menu bar and I have to pin them again. Even this tiny detail is enough to make me not want to have to deal with that * 1000 by taking up more of the services.
Well, to EC2 you attach storage from another AWS service. I've not used Lightsail, but from the description I imagine it's a more 'single package, quickstart' service mirroring the experience you might get from a PaaS like DigitalOcean/Scaleway/Vultr/etc. vs. the IaaS style provisioning of compute & storage independently.
Think of LightSail as being more akin to Linode or Digital Ocean a simple straight forward VPS without having to setup a VPC and being bombarded with trying to navigate 150 services. It’s also straight forward pricing.
> I couldn't reasonably be expected to remember what all the three letter acronyms were.
Just remember if you see an "S" it means simple. That means if you can't understand it you must be an idiot. S3 is one of the least simple things I've ever used.
I think pinned items are on a browser basis - because I logged into another account on the same browser and it had the same pins as my other account, without me setting them.
I only use Safari for any actual browsing, I've not noticed this happening, just that whatever I pin is gone next time I log in. Maybe it's short lived cookie.
I figured somebody has done this before. I purposely did not look for it to keep my findings unbiased. It was fun clicking on random services and tweak with it until figuring out what it was for and how to use it (and actually see some results)
New Azure services tend to have names like "Cognitive Services Speech" or "Cognitive Services Image Recognition" or "Azure Stateful Functions"... There's not a lot to riff on.
Sometimes it pays to be boring. One of the hardest parts of using AWS is learning all their silly names for everything. I know it's tempting to be cute when naming things, but everyone else wishes you would just be clear and descriptive. I've seen this play out at startups that love to name servers after galaxies or cartoon characters. It's all fine until your new employee onboarding guide comes with a massive memorization test before you can be productive. Yes, db-master and db-slave are way more boring names than Saturn and Uranus, but do everyone a favor and express your creativity somewhere else.
I worked at a company that had names for all the conference and meeting rooms based on various science fiction names and other "nerdy" stuff instead of some sort of building-floor-room_number scheme. It was neat at first, but quickly became agonizing trying to remember what _building_ a conference room was in much less where in that building. It added an extra few minutes to every meeting for me where I had to either look it up on the internal wiki or go ask someone at the front desk where I was supposed to go or make sure at least one person nearby attending the meeting already knew where we were going. It took months to get the hang of for even the more routine rooms.
I think they went through a middle phase of cutesy naming, but this seems to have subsided some (to the good).
Simple Queuing Service, Simple Storage Service, Elastic Compute Cloud, SimpleDB are names that very much make sense. Greengrass, Lightsail, not so much. EKS, Outpost, Ground Station, and Lake Formation are more toward the usefully descriptive side again, I think.
Opsworks: I thought it was using Chef under the hood. Is it really Ansible?
CloudWatch: it's actually so much more than logging, as it also provides timeseries, alerting and even scheduling. Not sure how to summarize this, though.
a bit tangential: I think a lot of AWS customers would actually benefit by hosting on their own machines in a data-center. The tools (and hardware) out there have become so good that there's minimal benefit to hosting on AWS for more than 4x the price. A lot of DCs also accept shipments so that even makes things easier. The trouble is that we've been conditioned, as an industry, not to think for ourselves or dare question certain accepted norms/practices. AWS/Azure/GCloud is great for some but I suspect it's for a much smaller subset that we want to accept.
I used to feel like "the cloud is just someone else's computer" but I have changed a bit.
If you do cloud native, there can definitely be pricing and peace of mind benefits. The idea that you can deploy a globally redundant database with automatic backups, with zero installation or server/software maintenance, is pretty amazing. Lambda is awesome and powerful, AWS IAM security model is revolutionary vs. so many different network/LDAP security policies.
That's what we (mid/small-sized company) with couple dozen machines. It is probably 10% the cost of AWS - but you do need to have at least one person that can handle the servers when/if there are issues.
That one person only needs to devote maybe 2-5% of their time to it (once the systems are setup), so it's still a net gain.
Yeah, we use a mix of AWS and self hosted machines. We've had up to 40 servers with this method and devoted similarly trivial amounts of time to it. If you want it to be long term sustainable, you probably want at least 2 people that can handle it, so they can go on vacation.
If your goal is to be the next billion dollar company, it probably doesn't matter that much. But if you're self-funding and need a sustainable business model from the beginning, there's a huge growth phase (mid-sized) where the cost savings from doing this sort of thing can be large enough to be worthwhile.
I'm not sure why so many developers seem afraid of hardware. Back in the early 2000s it definitely felt more commonplace for developers to be able to deal with it. Maybe it's because we grew up having to deal with our own in the form of desktops. Nowadays it seems like lots of people just use a macbook. And that's their only exposure to hardware
If you want it to be long term sustainable, you probably want at least 2 people that can handle it, so they can go on vacation.
The fully allocated cost of two ops people in my neck of the woods is a little over $300K. That can buy a lot from AWS.
I'm not sure why so many developers seem afraid of hardware
I want to develop. Yes I’ve been around long before AWS was a thing, I had to manage servers, load balancers, and even an elevated “server” room with a SAN with whopping 3TBs of storage back in the early 2000s. But in my old age, my time is valuable. I’ve done the on call thing before where our database server went down in the middle of the night when the night operations crew was working.
I won’t work for a company that is not on a cloud provider and doesn’t have a dedicated ops team.
But, if you are a small company. Why are you spending a lot on AWS? Two EC2 instances behind a load balancer and a hosted database shouldn’t cost you over $500 a month and that includes a decent amount of traffic and miscellaneous other charges. Heck if you are that small, just use Lightsail and when you grow, peer your Lightsail VPC that you probably didn’t know existed to a real VPC.
What are those two ops-focused folks doing with the other 95% of their time? Are we talking about two developers who are willing to spend 5-10% time on non-promotable server chores--while yet being on the hook in the event of a hardware disaster--or people who consider operations their vocation and aren't interested in software development?
I agree that it's possible to do that kind of work on a 5-10% basis, but it requires just the right set of circumstances for sustainability. Like, maybe it is your own startup or family business, or one is not particular good at sysop'ing and wants to learn more.
When I first started working at my current company, my manager wanted a highly available SFTP solution that would automatically sync with S3. I designed a relatively straightforward solution on paper involving a network load balancer, autoscaling group [1] with two instances and a SMB share (https://docs.aws.amazon.com/storagegateway/latest/userguide/...) and gave him an estimated cost.
He said okay go for it.
But then I said, or we can just throw some money at AWS and use the more expensive (on paper excluding babysitting time), AWS SFTP service. I gave him both solutions as a test to see where his head was at.
He chose AWS SFTP. We are a small high margin B2B company and sign long term contracts that are worth yearly in the six figures. We are very much focused on outsourcing the “undifferentiated heavy lifting” so we can focus on development and sales.
[1] Autoscaling would not have been used to scale on demand. We set a min/max of two so if one server fails a health check, another instance will be brought up.
Sometimes I sit back and wonder if Amazon makes these names so complicated on purpose. Hmm, now that I think about it more, maybe they had to come up with these weird names so they could trademark them?
Although AWS Braket is just in preview it's pretty safe to say it is a Quantum Computer as a Service bundled with a framework to write you algorithm in (a la IBM Q and qiskit).
The nice thing about it is that you have a choice between three hardware vendor, all featuring different architectures giving the ability to test superconducting, ion trap and annealing systems from the same place.
I have no affiliation with this other than being a Physics/CS graduate with only one Quantum Computing course under my belt.
Spark was built as a Batch job solution. Flink was built as a Streaming solution. Since Spark needed to adapt, they leveraged micro-batching to operate as a streaming solution.
They are very similar today. Maybe some remnants of the original design make Spark "slightly less appropriate" for pure streaming usecases, or possibly less able to iterate on future features/optimizations, but so-far that line of reasoning has been only speculation.
We use Camel, Spark and Beam to do much the same thing at my work, so despite using them all semi regularly, I've never known why you'd pick one over the other.
Yes.. some of them I have absolutely no clue on what it does. They might serve some edgecase specific for that domain. There are a few services in preview which I cannot see for myself what it does.
I will try to update and correct services as soon as I have more info (or somebody can provide it to me)
Hey, dev here for Robomaker. Most relevant line from our overview page is probably "AWS RoboMaker is the most complete cloud solution for robotic developers to simulate, test and securely deploy robotic applications at scale."
The main feature is the on-demand cloud hosted version of the robotics simulator gazebo ( http://gazebosim.org/ ) that allows you to run your robot's code in a 3d physics environment. You can then take that tested/verified code and deploy it over the air via our fleet management features to your actual robot.
Indeed. One of the entries like this is a dedicated robot infrastructure service (like, managing real robot hardware that does something out there - think delivery robots). Certainly out of the usual software domain. They also collaborate with a few different quantum computer companies that do different approaches to quantum computing (gate based and annealing) which is an entirely separate field of science and most likely out of the usual comfort zone of most IT people.
It seems there is a a lot of back and forth on this thread about AWS good/bad. On the one side, people seem offended people use AWS at all Which common it serves a need and generally does what it’s supposed to. On the other side it’s like Stockholm syndrome, do people seriously believe AWS is there for you and hasn’t trapped you in dependency of their services?
Most sound about right except for Global Accelerator, which isn’t a way to run your apps on edges, it’s a way to route all your network traffic through the AWS edges. Make it a bit more reliable and faster, and has really cool load balancing and routing options.
Someone do this for Oracle and SAP's products list too please. It's a mess of marketing speak and no one has a clue. With SAP you dont even get to see the product in most cases. It's just marketing you have to make your buying decision on.
That's kinda useful I guess? But I still don't know whether I need EC2 or Elastic Beanstalk. And do I need still need Batch if I have one of the other two? Still, it's much clearer than Amazon's own pages about this.
It'd be nice to include "year introduced" data as well. My AWS knowledge peaked sometime in 2012 after DynamoDB came out. I know a bit about newer things (like Lambda) but there's a lot of catching up to do...
Tested Sumerian and it's a barebones 3D web tool (modeled after Unity but more akin to ThreeJS editor) that can be controlled via script by other services. One example is being able to do some tasks like in-browser AR.
Soon we will all be back to hosting everything ourselves.
The complexity of pricing and so much different technology, which we absolutely don't really need, will eventually lead to we start hosting things ourselves.
Useful, but it contains factual errors:
>S3 File storage. Not directly used for mounting, but you can directly download files from HTTP.
You most certainly can mount S3 buckets, and its done frequently in data pipelines throughout the industry[0].
>S3 Glacier Low cost storage system for backups and archives and such
Sure but it would be good to include why there is a tradeoff in price, why is it low cost? For Glacier its intention is to provide storage, but rarely fetch data, as it is very slow.
How is that a factual error? It didn't say that mounting was impossible, just that it isn't directly used for mounting. How did you conclude that it's done frequently throughout the industry? What percentage of S3 customers do you think does this? If I would have to guess I'd say less than 1%. But I am happy to learn I am wrong.
Not OP, but my current gig is literally building streaming data pipelines on top of S3 using lambdas that only have a few hundred MBs of RAM to process gigabytes or more of data. S3FS is as close as it comes to mounting in my book.
You should be wary of any hostilities between your site and it's users. Even though it can't be brought down, your wallet sure can be if you don't set it up really carefully.
You can simply retrieve the page constantly over a residential connection to drive up the costs to over hundreds of dollars
mediaconnect - to transport live video
mediapackage - package live/on demand video content
mediastore - store video assets for live/on demand video content
EBS is actually part of the EC2 service. It's not listed explicitly in the console service menu. So I did not add it. (same goes with loadbalancing, security groups etc)
I think just expanding it would be more helpful. I had seen the term before and knew exactly what a Virtual Private Cloud was once being reminded what VPC actually stood for.
Route 53 - Holy shit! It's NSD
WAF - Holy shit! It's modsecurity
SES - Holy shit! It's Postfix
Inspector - Holy shit! It's OSSEC
GuardDuty - Holy shit! It's Snort
Data Pipeline - Holy shit! It's Cron and Bash
Athena - Holy shit! It's Prestodb
Glue - Holy shit! It's Hive Metastore and Spark
OpsWorks - Holy shit! It's Chef
VPC - Holy shit! It's a VLAN
Snowball - Holy shit! It's a truck full of hard drives
CloudWatch - Holy shit! It's syslogd
Neptune - Holy shit! It's Neo4j
ElastiCache - Holy shit! It's Redis
DynamoDB - Holy shit! It's MongoDB
S3 Glacier - Holy shit! It's DVD backup
EFS - Holy shit! It's NFS
Elastic Block Store - Holy shit! It's a SAN
Elastic Beanstalk - Holy shit! It's Apache Tomcat
EMR - Holy shit! It's Apache Hadoop
Elastic Cloud Compute - Holy shit! It's a virtual machine
Kinesis - Holy shit! It's Apache Kafka
QuickSight - Holy shit! It's Tableau