Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Does anyone else find the AWS Lambda developer experience frustrating?
422 points by tdfirth on April 18, 2021 | hide | past | favorite | 278 comments
Hey HN,

I've been using AWS lambda a bit recently, mostly as a way to glue together various bits and pieces.

Maybe I'm doing this wrong but does anyone else find the experience to be really frustrating?

I can unit test bits of the code just fine, but at some point I always end up stuck in a slow feedback loop where I deploy the code, do some manual invoking, go and dig through the logs in CloudWatch, add another print statement in my lambda... and so on.

What I want is to run the lambdas locally, ideally more than one, and then exercise them with streams of test events (perhaps captured from a real environment). It would be quite cool if I could define BDD style tests around them too.

Anyone have any suggestions or share my frustrations?

I have heard localstack is quite good although I haven't given it a go yet. Would that work for me? I did try SAM but I was a bit underwhelmed and I don't want to use a separate IaC tool for these.

Alternatively, do other FaaS providers solve this problem?

Thanks for any help.




You've discovered what many other people have: The cloud is the new time-share mainframe.

Programming in the 1960s to 80s was like this too. You'd develop some program in isolation, unable to properly run it. You "submit" it to the system, and it would be scheduled to run along with other workloads. You'd get a printout of the results back hours later, or even tomorrow. Rinse and repeat.

This work loop is incredibly inefficient, and was replaced by development that happened entirely locally on a workstation. This dramatically tightened the edit-compile-debug loop, down to seconds or at most minutes. Productivity skyrocketed, and most enterprises shifted the majority of their workload away from mainframes.

Now, in the 2020s, mainframes are back! They're just called "the cloud" now, but not much of their essential nature has changed other than the vendor name.

The cloud, just like mainframes:

- Does not provide all-local workstations. The only full-fidelity platform is the shared server.

- Is closed source. Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP. You can't peer into their source code, it is all proprietary and even secret.

- Has a poor debugging experience. Shared platforms can't generally allow "invasive" debugging for security reasons. Their sheer size and complexity will mean that your visibility will always be limited. You'll never been able to get a stack trace that crosses into the internal calls of the platform services like S3 or Lambda. Contrast this with typical debugging where you can even trace into the OS kernel if you so choose.

- Are generally based on the "print the logs out" feedback mechanism, with all the usual issues of mainframes such as hours-long delays.


While that’s generally true, the inputs and outputs and execution environment that you can count on is well-documented on AWS Lambda. That means that recreation of execution environment for the function in question is pretty reasonable locally (why is this not a thing then Amazon?!?). For the rest of AWS, you can either get credentials in place for access or mock those out somehow, but that’s not officially supported either.

I feel like the service is leaving a bunch of easy work on the table that would let people have a local development process that mirrors the remote development reality.


It is a thing; unfortunately it's not officially supported but I also find it crazy that so many developers are fine doing dev directly on "the cloud", as if that's a reasonable use of their time.

https://github.com/lambci/docker-lambda

https://github.com/localstack/localstack


Lack of understanding, I think every new dev to serverless will have this stage of frustration, then eventually will find a way to do better with creative ideas, I myself never run lambda in the cloud I write a UT for my handler and assume I know the input and the expected output, I mock everything except step functions I use the official jar from AWS.


People sometimes don’t understand the “why” of something like Literate Programming - it wasn’t really for documentation purposes. Part of it was to reduce as much as possible what we would call “compiler errors” as much as possible.

Arguably that’s a more dignified form of programming for a more elegant time but it doesn’t really mesh with the reality of “try fail print retry”.


A fantastic summary. My bread and butter is helping companies get to the cloud, but my number one guideline is that the local developer experience must be supported as much as the infrastructure improvements are.

Otherwise, you get a mishmash of fantastic visibility into your workloads and better provisioning cycle times, which then immediately get cancelled out by developers who have to work 3x as hard to do the same thing they were doing before.


IMHO leveraging serverless properly generally means writing tiny, stateless, ephemeral, single-responsibility, "pure" functions. Cloudwatch might be underwhelming, but complicated debugging scenarios, with hours-long delays, isn't necessarily representative.


IMHO leveraging serverless properly generally means writing tiny, stateless, ephemeral, single-responsibility, "pure" functions.

Unfortunately, what it most often seems to be used for is triggering side effects in response to some event, which is exactly the opposite of a pure function. Local testing of that logic is probably reliant on mocking all the other services it integrates with. Beyond that, the next step tends to be remote integration testing on the actual cloud hosting service, using some sort of separate staging infrastructure that will possibly be shared among many developers.

Some of the ideas and flexibility that should be offered by cloud hosting are appealing, but in practice, it often feels like we've regressed decades in our software development practices, discarding successful ideas like shorter feedback loops and unit testing and one-step build/deploy processes.


> the next step tends to be remote integration testing on the actual cloud hosting service, using some sort of separate staging infrastructure that will possibly be shared among many developers

One of the results of the pay-per-use pricing model of services utilized in serverless architectures is that you can easily provision dirt-cheap developer and feature-branch specific environments for integration-level testing. While you still can't attach a debugger in your function, the isolation from changes others are working on makes testing and exploration much more pleasant than when working in a shared dev environment. Things only get complicated when your architecture includes more expensive components with different pricing schemes that you probably want to share with others (e.g. relational databases, data streams).


Things only get complicated when your architecture includes more expensive components with different pricing schemes that you probably want to share with others (e.g. relational databases, data streams).

But how often will that not be the case, realistically? By their nature, serverless functions have no persistent storage or other capabilities of their own. All they can do without communicating with other services is return some data to the caller.

So, if you really are writing a lambda to do some calculation on demand, that's OK. However, literally anything else is going to be integrated with other services in way or another. That will include some sort of database or other persistent storage if there is data to be retrieved or saved for future use. It might involve services that provide other effects, whether sending an email or collecting a payment or querying someone's geographical database API and sending a message to a user's phone giving them the nearest dining options.

It used to be a pain testing integrations with third party online APIs, because most of them don't provide support for spinning up a fresh simulated environment on demand to test against. (Did I mention that as an industry we seem to have regressed by decades in our standard practices?) But with some of these cloud services, we create the same problems even with our own systems, which seems like madness to me.


I've been working on serverless applications for years now and the cases are quite rare TBH.

Static cost of data streams come up in data projects but even there it's trivial to consume data from shared streams in developer-specific environment since the stream pointer is managed in client side. The throughput required by even multiple occasionally consuming dev environments rarely requires provisioning streams above minimum capacity.

DynamoDB in on-demand mode is my go-to OLTP database until I know enough of runtime load characteristics to provision static capacity with autoscaling. For any non-prod (or load testing) environment on-demand mode is the best choice. Aurora Serverless is OK as well but has its warts and so far the cases where I really need a relational database in a greenfield project have been few and far between. Anything with complex query requirements goes to S3 and gets queried asynchronously or in batches via Athena. Not suitable for real-time cases though.

Things like queues, notifications and e-mails are handled by SQS or SNS and paid per usage. Websockets by API Gateway and Lambda.

Architecting for serverless requires a shift in mindset and tooling but as a benefit you get cheap isolated development environments and truly elastic production environments with very little ops burden.


Problem is aws or gcp or azure do not follow any standard spec for their services.

Ideally, they should first come up with a spec - and provide reference implementation.

Every provider will implement solutions for these standard specs. That way customers can use services for the spec. Customers can move between providers or work with multiple providers at the same time. Level playing field & let the best provider win the customer loyalty.

But, all providers want to lock in customers. So, we are at the wrong end of the stick.


GCP -> Kubernetes. Pretty portable.


To some extent. I have seen tons of scenarios istio/loadbalancing/other managed services that cannot be easily reproducible without 'the cloud'. I would say it is much more portable.


I should have been bit more specific. I meant managed services like s3, dynamodb etc


s3 is pretty much defacto standardized api for object store... tons of others follow minio, digital ocean etc.

dynamodb despite being a heavily promoted product at aws is not really a staple, but a crutch since rdbms doesn't scale that well


While I understand this viewpoint, and the comparison makes _some_ sense.. Kubernetes seems to solve all of these problems to me.

> Does not provide all-local workstations. - Solutions like minikube fix this.

> Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP.

Kubernetes has the same developer experience everywhere. As far as OP's serverless trouble, why not just use knative?


As long you only use k8s and no (other) managed services in the mix.


In addition to K8s, the big clouds (and even DO) all support S3-compatible storage, SQL and Redis/Memstore.


Kyma would be the FaaS replacement for k8s/knative

(I haven't tried it yet as the whole functions fad still hasn't hit me)


Haven't tried Kyma but I like OpenFaaS.


Wow I’m just amazed how you summed up the modern day development by giving an example from the past. I’m just curious we have three big cloud players at the moment Azure,AWS and GCP but do we have open source alternatives to their services that these cloud provides provide which you can host your self on large servers


k8s


> You'd get a printout of the results back hours later, or even tomorrow.

See also “big data” and machine learning.


> Is closed source. Only Amazon provides AWS. Only Microsoft provides Azure. Only Google provides GCP. You can't peer into their source code, it is all proprietary and even secret.

I'm a big fan of [OpenStack](https://www.openstack.org/) as it is open source and definitely enterprise-grade.


This weeks episode of DevOps paradox talked ... a lot about this exact "we're back to the mainframe" thought process.

https://www.devopsparadox.com/episodes/does-history-repeat-i...


That's really interesting, thanks for sharing. I did not have the pleasure(?) of working in the industry back then, as is true of most developers these days.

Do you hold out hope for a similar transition to a better local development workflow? What do you think it might look like?


The major cloud vendors have a vested interest to the tune of trillions of dollars for ensuring lock-in. All the cloud vendors want to become the next IBM, except this time with recurring revenue instead of one-time sales, something IBM never succeeded at despite trying very hard.

Openness, standardisation, commoditisation, and low-tier equivalent offerings are an anathema to the vision of these mega-corps. They will fight tooth and nail to prevent it.

Look at what Microsoft is doing: Instead of moving Azure features down to their IDEs, they are moving their IDE into the cloud! VS Code is Electron-based, which is to say it is just a local web server plus a web browser shell. It's not rocket science to move the server component into the cloud, and have the standard customer web browser be the front-end. I've noticed recently that this has started to happen. There's already a SQL query editor based on VS Code, and there's editors popping up for App Service and I think Functions also (equivalent to AWS Lambda).

This is the ultimate lock-in, where you develop "on" the platform directly, and have no hope of ever reproducing any part of it locally: the build system, the debugger, source control, tracing tools, or the IDE itself.

Other people in this thread mentioned local Kubernetes development flows. They're too busy drinking the Kool-aid. The cloud vendors are not interested in ever making this a long-term viable option and will find creative ways of making such open development practices non-tenable.


There's already good projects doing it check the https://serverless-stack.com/


Ibm cloud is especially bad. Tried api connect. There is no way! To get any logs, debug information or anything! Nothing! Zero! You only get what the api endpoint tells the client... It's insane..


This is a great insight.


Wow, you put into words what I was feeling about modern development. And software in general. God forbid you make a completely offline program and (gasp) never update it, that's backwards, unacceptable, a terrible security risk, etc.


And it's so hard to convince management, all rewarded on innovative solution like firing the data center team to replace it with an enormous Microsoft Hybrid Cloud contract, that they're only buying something way inferior for a much higher price.

I envy the geniuses at Microsoft who can convince 60k+ software companies like mine where we're all engineer or mathematicians, led by incredibly clueless IT decision makers :D I've talked face to face with people from an other continent in charge of our switch to Azure, and he was neither aware nor involved in even the decision of Azure vs AWS vs Google Cloud. But he was in charge...

I came to learn that the main reason we chose Azure (but I think the biggest problem was to choose Cloud) was because we already had teams doing Windows maintenance, server management, etc: we had way more relationship with MS already than with Amazon or Google. Now we have it, it's beautiful, nobody uses it, people grudgingly do trainings, and make big presentation about Cloud strategies, while waiting for legal and compliance to allow us to connect to it. Since we figured after the contract was signed that it would be mostly illegal for us to give Microsoft any of our client data...

Whatever, happy to see sometimes other people struggling with the bewildering fact that we somehow regressed :D


Look on the bright side, when it comes crashing down, there will be a whole lot of money to be made in undoing the damage done by frameworks upon frameworks inside of containers/serverless locked up in proprietary cloud services and APIs. And there won't be a lot of competition, either, since the old timers are retiring and many of the younger engineers couldn't tell you how a Kernel Module or a POSIX Signal works if their lives depended on it.


Yeah I’m sure the future is going back to kernel modules and posix

/s


Everything old is new. I don't think anyone predicted that the future (our now) would be consisted of massively scaled timesharing systems, hehe.


lol! This made my day.


The genius at Amazon paced the road. Azure just walked on it.

To be honest though, I never had a full production system in my dev machine in my 5yrs career.

So, it is not different. If anything with microservice approach, it would be easier to test locally than it was before


> To be honest though, I never had a full production system in my dev machine in my 5yrs career.

http://catern.com/run.html

I can understand it might not have been possible thirty or even twenty years ago but with modern systems such as the EPYC 7003[1] you can have a 64 core / 128 thread beast with 4 TB memory. Granted the processor alone costs $7890 but I can't imagine any system that won't run on a single top end modern computer, especially with dummy data.

Why don't we run the full production system on a development machine?

[1] https://www.anandtech.com/print/16529/amd-epyc-milan-review https://news.ycombinator.com/item?id=26466933


What's a full production system? Do you think you could run your company's production system on your most powerful MacBook pro?

I can probably run a few services. And then what for? Why would I run on my host something developed by a team I probably will never speak to?


> powerful MacBook pro

Likely not on a laptop/notebook computer. I was thinking more like a modern desktop. I’ll even settle for a dedicated environment on some server farm somewhere.

However, I’d like to be able to run everything that is pertinent to my work and that we have access/expectation to edit/comment on.

Especially if you are just getting started with micro services and there are like nine microservices but only like three teams you are still expected to have input on all nine micro services even though your team is only directly responsible for three of them.

I think now that I think about it, I basically want a development environment for me with an easy story to debug without getting on VPN or something.


You can achieve that with a tool like Kustomize maybe. Or docker-compose.


Can I ask what country you're in? All the big cloud providers have infra that can handle even classified projects securely.


That might be true technically, but there is a price to pay when governments insist on being able to pry into otherwise private data, and part of that price is that business entities operating within their jurisdiction may not be considered sufficiently trustworthy from a legal data protection point of view, particularly by business entities in other jurisdictions who have their own compliance obligations to worry about. This has been an issue for EU-US data sharing for years, it's a current issue for post-Brexit EU-UK relations, and if recent trends continue it may even become an issue for inter-state data transfer within the US. And those are all examples between relatively close allies with generally good diplomatic relations and comity.


> The cloud is the new time-share mainframe.

> You'd get a printout of the results back hours later.

Not sure what you are trying to say: What you describe is not time-share, but batch processing.


Eh? No, the "hours later" bit was because other people were using the mainframe. Not because it was "big data".


I've been in the AWS world for 4+ years now and my immediate feedback is don't run any local emulators. None! Write unit tests for the internals and test your system end-to-end. I say that both because AWS services can have unpredictable behavior that you need to account for but also because local emulators are at best functional, but in reality far from emulating the AWS world on a 1:1 scale (especially the unpredictable behaviors i mentioned). So instead optimize for many local unit tests and many live end-to-end tests (implies many deployments and parallel environments prod, staging, dev, etc)

When it comes to Lambdas, given the reasons above, there's only one thing that can improve the experience: PROXY. Before i went on parental leave i had the idea is creating a proxy lambda which can be configured with an IP and port number. That IP and port is for your local dev environment. This way, when developing you can instruct a live system to short-circuit and to proxy calls to a local lambda available om the respective port. Trigger end-to-end tests by invoking AWS services that will eventually call the proxy lambda, and then your local lambda with the same environment/context/input, reply with output which will reach the proxy lambda, which will output the same content forward.


Yes, unit tests are the way to go. Lambda being a single function with a defined output makes this really simple. We have unit tests for sublibraries, but the "integration test"-level test are just calling the handler with the test payload and comparing it against the expected response.


To check my understanding, you basically deploy a place holder lambda which proxies events to your dev machine?

That's pretty neat if so, I might have to try that out. Are there any downsides to this approach do you think?

Thanks for your thoughts on emulators/tests too.


Sounds a lot like https://docs.serverless-stack.com/live-lambda-development


Correct. I'm very pedantic and usually I can't help but find flaws in all proposals :)) but I really can't think of anything here. I can see limitations though: if you use lambda layers or native code specifically for the AWS architecture etc, then you need to account for that when proxying but... It's not impossible. You also need to account for having a public IP or use a service that exposes a public ip/port and redirects to your local dev machine. The idea is not super simple in practice but it is very simple compared to the emulators and it gets you very close to 1:1 parity for all those corner cases.


Are you sure?

Getting your live stack to call your Dev machine seems a sure fire way to slip up and accidentally send customers a test email. Or accidentally order $100,000,000 widgets.

In fact, I would say it sounds great, but in fact it's incredibly risky and monumentally stupid.


Read my comments in https://news.ycombinator.com/item?id=26857198

When you write "your live stack", my brain freezes. Even a simple "webapp" project in my team had 4 common stacks (prod, staging, dev, git-master which had every push to master getting deployed) plus individual stacks, one per developer, with the ability to create extra stacks for specific feature development.

Only prod had real data. Obviously (to me) you wouldn't shorcircuit lambdas in the prod env. Not even staging. Not even dev. Not even git-master. Short-circuit development environments where you explore. Very low risks there. We need to learn it's ok to fuck up, as long as we fuck up in a safe environment, nowhere near the nuclear button ;)


Not unless you have live secrets in your local application on your dev machine.

Just create a staging env, if you don't have it already, and point to staging


For a similar relay/proxy setup for development, I've found that a metadata endpoint emulation like provided in aws-vault can make it easier to have any AWS SDK calls behave the same way as they would in AWS itself. You do have to assume the role that the lambda itself would normally assume and you have to allow more than just service linked execution, but you do get to test the whole IAM chain that way as well.


Not sure that i follow you, but the setup i described would proxy the env vars as well, among them the aws credentials, so you should be able to run loudly with exactly the same liberties as the original AWS lambda.


Ha, I'm glad my simplification did not fall too short!

I did wonder with regards to the IP address. If you had a VPN set up for access to a VPC and ran the lambda in that VPC then that would be another option I think. You would just need to configure a static IP for your dev machine within the private network.


Right! I haven't explored that thought. My team kept everything serverless so we touched zero VPC/etc. From a generic perspective, having a setup to proxy into a LAN is more valuable I think - i can imagine for instance how my team had access to certain AWS services, but not all, erc etc. But these are details - the idea would be worth exploring


One "trick" I usually use with Lambdas - add a main() method that allows you to run the Lambda as a regular command line app. If you authenticate locally with a test environment in AWS, you can pretty much test everything in your code easily that way.

You can use this in your local dev env and even in integration tests. I don't need to do this a lot but sometimes it helps when troubleshooting.

Also, you might actually be building a tool which can be used from the command line. I recently took an existing command line tool (written in Go) and ported it to Lambda to expose it as a chatbot.


Modern software development: middle man attacks on AWS.


This 10x!!! Personally i find these setups a complete design failure. Make it work and then figure out how to develop with the constraints you will find later (not the ones you have designed yourself).

So - reverse engineer and monitor in-transit behavior it is...


This is a brilliant idea I'm stealing for the next time I'm doing Lambda dev.


This is the approach I use. For the AWS specific workflows e.g. SNS inbound to S3 bucket, processed and put in another bucket getting the structure of a heterogeneous set of events can be a hassle.


> and test your system end-to-end.

How do you automate this?


Living in my team's bubble i thought everyone runs or tries to run parallel environments: prod, staging, dev, but also an individual (person) or feature env. Why? Because there's no emulator or documentation that will teach you real behavior. Like others have said, AWS seems out of this world. Just like GCP and Azure i might add. Some things you don't expect and they mesmerize you how smart they are. Some you expect and you can't fathom how come you're the "only" one screaming. Random thought: this is how i ended up logging all I bumped into into "Fl-aws" https://github.com/andreineculau/fl-aws

Back to the point: reality is that many build their AWS environment (prod) manually, maybe they duplicate once (dev) also manually, maybe they use some automation for their "code" (lambda) but that's it. This implies it's practically impossible to run end-to-end tests. You can't do that in prod for obvious reasons and you can't do it in dev either - you have many devs queueing, maybe dev is not in sync with prod etc.

My team ran cloudformation end-to-end. We actually orchestrated and wrapped cloudformation (this is yet another topic for not using terraform etc) so that if smth couldn't be done in CFN, it would still be automated and reproducible. Long story short, in 30 minutes (it was this long because we had to wait for cloudfront etc) we had a new environment, ready to play with. A total sandbox. Every dev had their own and it was easy to deploy from a release artifact or a git branch to this environment. Similarly you could create a separate env for more elaborate changes to the architecture. And test in a live environment.

Finally to your question: how do you test end-to-end?

If we talk about lambdas because that's where the business logic lies in a "serverless" architecture, then the answer is by calling the system which will eventually call your lambda/s along the way. If your lambda ia sitting behind AWS gateway, then fire an http request. Is it triggered when objects land on S3? Then push some object to S3. How do you assert? Just the same - http response, S3 changes etc. Not to mention you can also check cloudwatch for specific log entries (though they are not instant).

With this type of a setup, which sounds complex, but it is not since it is 100% reproducible (also from project to project - I had several), adding this proxy-to-my-dev-machine lambda would mean I can make local changes and then fire unit AND end-to-end tests without any changes pushed to AWS, which is the main time/energy consumer imo.

PS: sorry for the wall of text. Like i said i recently realized that the development realities have huge discrepancies, so i tried to summarize my reality :)


A wall of text that's full of something interesting and useful is way more welcome than someone trying to do a funny reddit quip! Thanks!

We have the ability to spin up one-off environments per-project, including our serverless stuff - and we do it automatically on every CI run - so I guess the answer is to do that, and test against that.


Yes, this idea could save my team a ton of headache.

do you have a git repo for your proxy code?


Unfortunately i had this idea just before i changed jobs to a "server-full" position, and now I'm on parental, so i don't have this proxy lambda implemented as a generic solution :(

I only had a PoC and then told myself this would be brilliant as an abstraction


Sound a little like Tilt, Skaffold or build packs?


More like telepresence than skaffold


I don't know them in depth but yes, I'd also go for telepresence (which is smth i discovered after i had my evrika moment and so i started googling for similar ideas or maybe even implementations)


"Think of the history of data access strategies to come out of Microsoft. ODBC, RDO, DAO, ADO, OLEDB, now ADO.NET – All New! Are these technological imperatives? The result of an incompetent design group that needs to reinvent data access every goddamn year? (That’s probably it, actually.) But the end result is just cover fire. The competition has no choice but to spend all their time porting and keeping up, time that they can’t spend writing new features. Look closely at the software landscape. The companies that do well are the ones who rely least on big companies and don’t have to spend all their cycles catching up and reimplementing and fixing bugs that crop up only on Windows XP." - Fire And Motion, Joel on Software

In my own work with serverless I think about this a lot. In my day job, we built a new service and we pay AWS a few dollars per month to run. It would have cost us around $100 a month in AWS costs. However, the operational complexity is extremely high and we are also coupled to another service via Kinesis. For a billion dollar business, the trade off doesn't seem worth it.

I wonder how much of serverless is just AWS firing at Google (and Heroku, DO, etc) and other competitors. It certainly hasn't made my life as a developer easier. It's certainly much cheaper for some use cases but the complexity of the system goes up very quickly, and you're end up having to manage a lot of that complexity using an inferior tool like Cloud Formation (or Terraform).


> I wonder how much of serverless is just AWS firing at Google (and Heroku, DO, etc) and other competitors.

It depends on what you mean by "serverless". Serverless webapps? Sure, maybe a fair bit. But the real killer feature of the Lambda service is that it's a universal extension mechanism for other AWS services. Want to run some shoddy virus scanning whenever an object is added to a bucket for compliance reasons? That's a lambda. Want to manipulate Firehose records before they are sent to the sink? That's a lambda. Want to execute some crazy Python function from an Athena SQL query? You guessed it. That's a lambda.

So when you say "It certainly hasn't made my life as a developer easier" it means you haven't needed to do these bespoke gluing together of stuff, because "being able to do this somehow" is infinitely easier than "not being able to do it at all because AWS hasn't exposed an API for this use case".


...You realize you can do 90% of those usecases on your own hardware, with your own tools, within your own infratructure, with complete visibility into every bit without Amazon right?

I think the true geniuses are the ones who spent so much time trying to sell people on "Server Admin is hard, lets hire it out!" as well as building the billing and metering systems.

In a gold rush, screw panning, sell shovels (and pans).


> ...You realize you can do 90% of those usecases on your own hardware, with your own tools, within your own infratructure, with complete visibility into every bit without Amazon right?

This statement is a red herring. People don't flock to AWS because they want to use lambdas. Lambdas are adopted by AWS customers who already have their data in S3 or DynamoDB and/or already use one of AWS message brokers, and happen to stumble on a requirement for a background that runs asynchronously in response to an event, such as someone uploading a file to a bucket.

When that usecases pops up, the decision process consists of either adding a brand new service to be ran on ECS, add a background task on a service already being developed, or simply write the event handler and deploy it somewhere for AWS to run without us bothering about scaling and maintaining stuff. Writing and deploying the handler is an order of magnitude simpler to deploy and maintain, which happens to win over some technical discussions.


I'd disagree. I chose AWS over GCP precisely because of the relative ease of deploying Lambda. That in turn led me to use DynamoDB and S3.

I started off with GCP, Cloud Storage and Cloud Firestore, but it was such a hassle and in the end, I couldn't even manage to run my Cloud Functions. The 10s cold start time was another non starter factor that led me to switch.

Honestly, I don't see myself staying with DynamoDB or AWS lambda. I definitely prefer local-hosted data stores such as on a DO droplet, and I would have started off with them were it not for their shoddy service. AWS service has been extremely strong so far.


A couple of years ago I was contracting for a company, and their CTO was so fired up about Lambda that that created a project to start using it - without having any other service on AWS (yet). Small sample size, but still.


A concrete example might help. I worked at a shop that had a decent monthly spend on AWS and other hosting fees. One intermittent task was rolling up hourly/daily logs into monthly stats. We could have used our existing infrastructure to provision, run, and bring down the hosts on completion and that would have worked. Over time that could have been effective.

Instead I suggested writing a Lambda service in Go and to run it incrementally (I think it was hourly and monthly). It worked out to tens of dollars per month, basically free. [At first it cost much more than this because the original implementation wasn't specifying known sizes for initializing collections that was constantly being resized in several places in its pipeline resulting in copying/double consumption plus severe fragmentation.]

The development, in particular versioning and deployment sucked back then. It's surprising to still hear mention of challenges so much later.


If you don’t see the value that AWS (or any other cloud provider) gives you then that’s fine. However that doesn’t mean it doesn’t exist.

IAM (consistent identity and access control across all of your estate) and KMS (at rest encryption of all data, partitioned by service) are two examples of two things that would be very hard to replicate on your own hardware.


Achieving the base functionality is not hard, getting some of the most advanced features is more of a pain. Rarely I see aws services sold for their advanced features, though. Somehow aws became the default for hiring / learning how to do DevOps.

For example: Setting up different SSH keys when deploying a machine is not hard; getting granular ACL is more complicated.

Getting encryption at rest is very service specific, for example Postgres has TDE (free, with paid support if my memory serves correctly); I'm sure there are some services where EaR has been implemented by aws and there is no OSS alternative.


> Achieving the base functionality is not hard, getting some of the most advanced features is more of a pain. Rarely I see aws services sold for their advanced features, though.

Base functionality is relative. The base functionality of IAM is that every server, pod, user or AWS service has an centrally managed identity controlled via RBAC that is used to access any other AWS resource, requires no long-lived credentials, powered via a flexible policy engine that can rely on principal and resource attributes, with updates to policies applied in seconds across the globe. The actual advanced functionality is more complex and useful in certain situations.

If your benchmark for base identity functionality is “different SSH keys for different servers” then I’m not sure what to say.

The same for encryption - the point is that it’s not very service specific with AWS. That’s the base functionality: encrypt this data (queue, disk, object, database, backup, whatever) with this key. Done. Consistent, transparent encryption at rest for every service, access controlled through IAM roles and policies.


I've found lambda helpful for adding bits of code in other languages to your stack. Want to use a NodeJS library from your Python app? Jam the NodeJS in a lambda and don't worry about maintaining/scaling it.


Lambda for the web is also really useful for little one-off tasks or webhooks that don't have a big domain model behind them. Lambda@Edge is also super useful for manipulating requests and responses.


So is the solution to avoid Lambda unless you have to? Don't use it to build regular software, just use it for AWS glue when necessary?


That’s the use case where it shines, but it can be an effective part of “regular software” as well.

It’s a hammer, and if it’s a good fit for your particular nail is very dependent on it’s shape.


> So is the solution to avoid Lambda unless you have to? Don't use it to build regular software, just use it for AWS glue when necessary?

That, and also offload some somewhat computationally expensive background tasks. Instead if screwing up your deployments by ramping up CPU and memory utilization to run a one-off task, just let he service fire a lambda and, if needed, just act when the lambda replies back.


Just want to pile-on to the other replies and say yes, absolutely. When you need them as glue they're a great option with basically no alternatives -- also means your lambda functions will be just a handful of lines of code, and then you can get on with your life.


It's like the SPA craze, microservices craze, NoSQL craze or similar things.

Appropriate for a small amount of apps, but used inappropriately by a lot more for a while because it's the hot new thing.


IMO SPA is the natural consequence of API 1st design, of you can do everything via the API why reimplement it all again for the UI, as much as it sucks in some ways, it can yield really good results.


Because you can never implement everything only via the API, anymore than you can implement everything via SQL queries. The reason why people created custom business logic atop the data model is .... because they need custom business logic atop the data model. Oh, you want to require a captcha before allowing someone to view a record? Well, that's not gonna be in your api. It's business logic sitting ontop of it. The reason people "do" API 1st design is just that they haven't thought these use cases through very well and it all makes sense until you start looking at the details of what your business logic is actually doing. Then it's up to everyone else to either convince the decision makers that no, this really is API first design so they get the buzzword crowd off their back, or they just drop the ability to securely do custom business logic and hope no one notices.


I think you have an overly specific interpretation of "API".


I agree this is natural outcome when developer time is precious, though sometimes increases complexity for the user and their browser.

And if one doesn't need an API first design then SSR with a robust framework might be cheaper to stand up and maintain.


> It's certainly much cheaper for some use cases but the complexity of the system goes up very quickly, and you're end up having to manage a lot of that complexity using an inferior tool like Cloud Formation (or Terraform).

The complexity bogeyman is a red herring. What is software development if not a constant fight between increasing complexity and adding features, including making systems more robust and resilient and reliable?

Sure, you are free to claim that function-as-a-service paradigm is very complex because the code runs somewhere else, and you have to use a separate workflow than the one you're used to, and that you actually have to write unit tests to have some assurances that stuff does work as expected.

But what would be the alternative approach?

Would it be simpler to write a full-blown service that polls events and listens to message queues and launches background tasks? Obviously, no.

AWS Lambda is a fancy way to let developers develop event-driven systems by just writing tiny event handlers and plug in events. It's no rocket science. In terms of complexity, they pale in comparison with even the tiniest hello world windows GUI app, or even Qt. I'm terms of web services, they are a service where you just have to write the request controller bit after that runs after all auth things passed and request messages were parsed and validated. Considering this, isn't it unreasonable to talk about increasing complexity, when everything was made.far simpler than what it would otherwise be?


In general, I found your post very strawmany...

It's not really a red herring when Lambda mostly forces communication between different application layers to be over the network. You suddenly have to deal with rate limiting, retries, bulkheading, etc. in a lot more places than you would even with even a SOA.

> you actually have to write unit tests

I'm not really sure unit tests are a differing factor in testing functions as a service and other software. The real issue with testing is that it's very difficult if not impossible to do more than unit testing locally. You're often using docker containers or mock services to try and simulate what is going on in production which is insufficient. Things like CloudFormation can't be done locally (and this can be where a lot of gross complexity lies), so ultimately you have to set up an identical-to-prod testing environment and do the bulk of your testing there (which you probably should have anyway, but that's besides the point). Things like localstack HAVING to exist but not being supplied by AWS is a symptom of something gone wrong.

> Would it be simpler to write a full-blown service that polls events and listens to message queues and launches background tasks? Obviously, no.

Why is this obviously no? It isn't to me. You likely need a lot less these type of things if you're running a monolith.

> In terms of complexity, they pale in comparison with even the tiniest hello world windows GUI app, or even Qt.

It seems to me you're ignoring all the extra stuff you HAVE to have to deploy, secure, and maintain serverless. There is a lot more than the code for the function themselves.


Take something as simple as "I want to display a webpage that prints out some rows from a database".

Well now you're deep into Terraform or CloudFormation, setting up API Gateway, building an authorizer, coming up with an API Schema, wiring up everything with IAM, setting up Route 53 and Certificate Manager, building CloudWatch Dashboards, etc etc. I hope you got all the permissions right because there are now 25 different places someone can maliciously gain access to your AWS account. And in order to change anything you should be prepared to update your IaaC schema once every couple months or it will all stop working.

Back in days past you'd rent a hosted server or shell account (It was hideous, but bear with me), write 15 lines of PHP, connect to the included database, and you're done. It wasn't "web scale" but the time to reach a MVP was probably 1/100th that of the current era of gross complexity. And any one who built software in that time period had a very clear runway to bring it in house and run on bare-metal. No such thing exists in today's era of subscription-model infrastructure.


> Well now you're deep into Terraform or CloudFormation, setting up API Gateway, building an authorizer, coming up with an API Schema, wiring up everything with IAM, setting up Route 53 and Certificate Manager, building CloudWatch Dashboards, etc etc.

No, not really. You only use any of that if that's what you want to do. You can use a EC2 instance and do everything yourself, if that's your thing.

I mean, let's go through your list. CloudFormation is just infrastructure as code. It helps treat your deployments as cattle, but you aren't forced to use it, right? And you only need an API gateway if you want to put together an API using Amazon's proprietary stuff. Nothing forces you to do that, right? And you only need Route53 if you feel the need to manage a DNS and do stuff as DNS-based routing. And dashboards is just to show metrics in a way that fits your need.

From these services, which ones are a must-have? None.

> Back in days past you'd rent a hosted server (...)

Your "back in the day" is "right now" if you want to. You can fire up an EC2 instance, open a port, and done. Why are you leaving this out, and instead opt to misrepresent extras as required?

Hell, AWS also offers lightsail, and you can use S3 to run static sites directly. Both are far simpler than managing your own VM or baremetal service. Why are you leaving that out?

And why do you leave out the fact that with API Gateway and lambdas you can setup a fully versioned API with multiple endpoints in a few minutes? You don't even need to deploy anything as you can write the controllers for each route directly in the dashboard. If reaching an MVP instantly is what you really want, it's weird you leave that out.

> No such thing exists in today's era of subscription-model infrastructure.

I regret to tell you that you don't know what you're talking about. Even in AWS, where they try to upsell you everything, you can easily ramp up a VM, open a port, and do everything you need by ssh-ing into a linux box. And AWS is not alone on it. You can do the same on pretty much any cloud vendor. Say Hetzner: you can rent a baremetal box for around 30€ a month and leave it at that. If you prefer VMs you can get one for 3€ month. What's stopping you?


I don't think parent doesn't know this can still be done; it's just that culturally this is not valued anymore and it's often not taught to beginners.

The same thing happened on frontend: all the juniors bootcampers' only experience of doing frontend development is running create-react-app and hacking away at React. Before that it was angular.

Everybody would frown at deploying a pure frontend js app or a jQuery app, in the same way most people would frown at deploying a PHP script on Apache.

Despite this, a huge portion of the internet runs on PHP and jQuery (including shiny new things, like levels.io's million making 1-2 men saas).


Oh, and all that stuff you just set up requires arcane knowledge of how to reproduce it locally. Worthless arcane knowledge, your aws certification.

Terraform is a magical amount of arcane.


For a static site or basic CRUD some alternatives are bog standard HTML, CSS and maybe SSR framework


> For a static site or basic CRUD some alternatives are bog standard HTML, CSS and maybe SSR framework

Static sites do not involve any form of tasks being executed: you just serve dumb files and let everything happen client side. I fail to see how a usecases that involves zero computational needs, and arguably zero servers, is used as an example in a discussion about how to implement services.


Wouldn't those be a bad fit for AWS Lambda, even by the opinions of most proponents of the service? IMO, one of the Netlify clones (JAMStack) is the best solution for these, so if you're running your own VPS for a static site, you're also not using the best tool for the job.


> Wouldn't those be a bad fit for AWS Lambda, even by the opinions of most proponents of the service?

Static site? Yes. Basic CRUD? Not sure about cost efficiency, but otherwise that’s right in the APIGateway+Lambda wheelhouse.


SST is an Open-Source framework worth checking out if you are looking for a lighting fast local dev environment for serverless. https://github.com/serverless-stack/serverless-stack

I often see ppl adopt one of two development patterns:

1. Locally mock all the services that your Lambda function uses. Like API Gateway, SNS, SQS, etc. This is hard to do. If you are using a tool that mocks a specific service (like API Gateway), you won't be able to test a Lambda that's invoked by a different service (like SNS). On the other hand a service like LocalStack, that tries to mock a whole suite of services, is slow and the mocked services can be out of date.

2. Or, you'll need to deploy your changes to test them. Each deployment can take at least a minute. And repeatedly deploying to test a change really slows down the feedback loop.

SST lets you develop your Lambda functions locally while connecting to others AWS services without needing to mock them locally. Behind the scene, the lambda requests are streamed to ur local, runs ur local function, response streamed back to Lambda. So you can iterate on ur function code without redeploying.

Think of your Lambda functions are Live Reloaded. You can see it in action here — https://youtu.be/hnTSTm5n11g

I’m the core maintainer of the project. And folks in our community are saying using SST feels like "working in the future" ;)

I'd love for you to give it a try. We are also a part of the YC W21.


I use this companies services (seed.run as well). Frank and Jay are doing something truly special in this space.

Also, this guide they put together is definitive: https://serverless-stack.com/#guide

Had I found that years ago, I would have saved SO MANY headaches. I've implemented local debugging (hell, and doesn't actually replicate AWS). I've followed the CloudWatch logs, just like you, painful.

The SST local debugging with lambda is the best way forward. Deploying micro services on seed.run is also the way forward.


Wow I really appreciate that! Thank you!


Thank you, that's awesome to hear!


Hey, thanks for the link. It does look quite good - there was another comment where someone had a home-brewed set up that did something fairly similar.

I'll check it out!


Awesome! We've got a super passionate community at https://launchpass.com/serverless-stack

You can also direct any questions/feedback to frank@anoma.ly :)


I second this. SST is like magic.


Thanks for the support! Really appreciate it!


I find all of AWS frustrating. Their interfaces feel like they were designed by an 80s IBM analyst.


I use them heavily for work. IMO, they're pretty well designed for managing large numbers of servers with sophisticated coordination and management systems. If you just wanna run a single server, it's kind of ridiculously over-complex. IMO, Digital Ocean, Linode, etc have much better workflows for just spin up a single Linux server attached to this DNS and let me set everything up on it.


Aws also offers that in terms of aws Lightsail It comes with the servers , DNS system , ip management under one interface.

I find it more simplistic that Digital Ocean , tbh


> Their interfaces feel like they were designed by an 80s IBM analyst.

The purpose of the AWS Console, AFAICT, is:

(1) To be used by semi-technical ops and management types, and people doing tutorials, and

(2) To drive everyone else as quickly as possible to the APIs/CLIs/SDKs.


I wonder if that is internal motivation for design as well


This made me laugh, thanks.

If I'm honest, I do find AWS in general fairly high friction at times but it's way better than renting servers directly I think.

It's the only cloud provider I've really worked with to be honest though (other than a few experiments), maybe some others are much better in this regard?


For me the Azure management portal is frustratingly slow, AWS is at least moderately responsive in comparison. I've not tried the Google stuff for years, but in my experience everything they do is ridiculously overcomplicated.

I don't really get where you're coming from on renting servers directly, using AWS is far more expensive if you actually need a server for a while.

Though I do admit being able to fire up a server for a couple of hours and then delete it is pretty awesome.


I wasn't referring to cost as, yes, it's definitely cheaper to just rent some servers.

The operational overhead of provisioning them, configuring them (and so on) is really difficult for a lot of teams though.


There is some truth to that. They are designed to serve a number of personas in an organization, so if you just need to do one thing or another, they will seem overly complex, but I think you could say the same thing about a cockpit or any interface for a complex system.


Haha! Funny, and there is probably a kernel of truth in there.

I’ve been thinking a lot about AWS lately and it definitely seems to me like a modern take on what IBM did in the 80’s.


I'm amazed that people use the console for anything other than discovery. Any heavy lifting you want the AWS CLI or CloudFormation.


For one offs, it's... Usable. But also it's not really the user's fault because AWS pushes it.


Ultra slow too, logging into the console is one of my least favourite things to do.


Time to check out https://vantage.sh/ :)


It is considered impolite to plug your own stuff without disclosing your relation to it within the same post.


...didn't know that - thanks for the heads up.


I was just looking at your pricing page and the greyed-out-but-still-ticked items in the comparison lists are very confusing. Being greyed out makes it look like their not included, but there's still a tick, so maybe they're still included but less capable? Or available as an add-on? I can't tell what, exactly, you're trying to convey to potential customers.


I was completely lost with AWS lambda until I discovered https://serverless.com - the fact that you haven't found them yourself suggests they need to do a much better job of marketing themselves!

I run my lambdas locally with a single command: serverless offline

If the lambda has an http endpoint, it creates the endpoint at localhost/<endpoint> and I'm good to go, it even does live reloading as I edit my code.

If the lambda runs off AWS events, I can invoke the lambda locally with a command, and point it to a JSON file of a simulated AWS event. I get my local rails app to create these AWS event JSON files, so that I can test end to end locally. Works well for my purposes.

To deploy I just run: serverless deploy --stage production

Which sets up all the necessary additional services like API Gateway, cloudwatch etc.

I can't imagine using AWS lambda any other way.


I can also recommend serverless.com framework for local development and running lambdas offline.

AWS provides a docker image for emulating DynamoDB which works great for local dev and will commonly be paired with lambdas.

Another option I have used recently for implementing node web services is https://github.com/apex/up which also has a nice local dev experience.


Thanks for the recommendation, that does sound good and I'll give it a go.


+1 for serverless offline


+1 -- yes if you must develop AWS lambda, then serverless is the BEST way.

BUT .. I should also say that while I started ~5 years ago with serverless framework, in the past year I've completely transitioned everything over to the Azure Durable Serverless Functions.

AZ has their own same-same-but-better "Core Functions Power Tools", it's got a lovely & zippy cli interface built in node that is separate/decoupled from the "az cli". Just like serverless you can run/dev/test locally, then quickly deploy to azure. The debugging integration with VS code, distributed tracing, even "AZ Remote Debugging" with "Azure Hybrid Connections" creates super fast inner/outer development loops.

If you like the "serverless model" then you also need to check-out AZ "Logic Apps". Logic custom connectors totally kick ass imho since they provide a ton of insight into how data is moving through the application, basically you deploy and you get instantaneous telemetry.

Deploying software that costs nothing (except negligible storage fees) unless you're using it AND would require an ELE (i.e. meteor strike) to break is very cool.


I recently migrated from Cloudflare Workers to AWS Lambda + Dynamo for my relatively large pet project.

It was surprisingly hard - the local development of lambdas is still very raw, documentation is scarce, and there are various small issues appearing here and there. I should probably write a blogpost about how to setup decent developer environment for AWS CDK and Lambdas, because there's not much on the Internet about it.

I set up the whole AWS infrastructure via AWS CDK. I have one TypeScript file, that creates a stack with lambdas, dynamodb tables, API gateways, S3 buckets, wires up the secrets manager, etc - all of that in 2 environments - dev and prod.

AWS CLI also can generate a Cloudformation YAML file from the CDK file (via `cdk synth`), which could be fed into SAM. So, I generate a `template.yaml` Cloudformation file this way, then run SAM like `sam local start-api`, and it runs exactly the same lambda locally, as in AWS, using that Cloudformation YAML file. SAM also supports live reload, so if you change any source files, it will automatically get those changes.

So, okay-ish developer experience for lambdas is possible. There're caveats though:

* I couldn't figure out how to use "nameless" Dynamodb tables (i.e. when CDK assigns the name to them automatically), because if I omit a table name in CDK template, then the local lambda and lambda in AWS assume different names for some reason.

* Locally, the binary outputs don't work. It ignores `isBase64Encoded` by some reason, and lambda just returns Base64 output, instead of binary.

* And the main problem - local lambdas are SLOW. It seems like it restarts some container or something under the hood on each API call, which adds 2-3 seconds to each API call. So, the calls that should be like 30ms, are actually 2 seconds now. This is super frustrating.


I could have written your message word for word. I recently had a very similar experience moving to AWS Lambda using AWS CDK and SAM. It is pretty shocking how unhelpful the AWS documentation can be at times.

Each use case is so unique, that it is likely difficult for them to account for every single situation where Lambdas are being used. Instead they opt for a generalized and non-specific documentation site, which results in docs I find practically useless.

I frequently find solutions to problems for AWS on someone's obscure blog, who after many days of suffering stumbled onto the answer, rather than finding anything of use directly from those who wrote the software.

Does a site exist where programmers can submit useful findings? Similar to stackoverflow, but rather than asking for questions, you just submit answers. I would think being able to search across such a crowdsourced utility would save programmers a ton of time.


> Does a site exist where programmers can submit useful findings? Similar to stackoverflow, but rather than asking for questions, you just submit answers. I would think being able to search across such a crowdsourced utility would save programmers a ton of time.

SO kind of lets you do this. You can write a question and immediately answer it yourself - there's an option in the question UI for it.


What made you switch? How do Cloudflare workers and their development compare?


Pros of Cloudflare Workers - they're way simpler. Their docs are nice, it's straightforward to set up a dev environment for them, it's easy to set up a key-value store for them (Cloudflare KV), secrets management is simple. Pricing is nice - it's flat $5 a month, and you get access to almost all of it. It works fast and distributed across the whole world, admin console is nicely integrated with the Cloudflare CDN dashboard.

But they're too limiting. Cloudflare Worker should be a pure JS script (that looks like a service worker script in a browser). You cannot use any Node libraries. In my app I need to generate images on the fly, so I couldn't figure out how to do that in a Cloudflare worker, so I ended up creating lambdas for those, and proxying the calls from Cloudflare worker to those lambdas.

KV (their key-value store) is way too simple. No secondary indexes, no sorting, no querying (you can only sorta query list of keys, but then you have to fetch a record by each key individually). My app hit the limits of KV very quickly.

Also, there's no monitoring or any good way to do backups for KV. So, it's almost unusable in a real-world production system, except as maybe a cache layer.

Monitoring was a big pain for Cloudflare Workers. Like, even accessing production logs is weird - you have to "tail" them from your CLI. Even Cloudwatch is 100x better here.

Development environment is okay, I guess, but the local server crashes constantly. Also, it kinda forces you to use Webpack and bundle the sources. It also doesn't have anything like layers in AWS lambdas, and they have a limit of 1MB for a bundle, which I almost hit (I had like a 900kb JS file), so that part got me worried too.

Basically all of that made me worry I will hit a wall soon with Cloudflare Workers, so I moved to AWS. It's not piece of cake too, but at least I get a way to run proper Node scripts, a decent document database, continuous backups and pretty good monitoring out of the box.


I tried Cloudflare Workers and found these pain points:

- local dev server (“wrangler dev”) crashes often

- local dev server executes code always on Cloudflare -> seeing changes takes a few seconds, you need an Internet connection

- forces Webpack on you

- managing KV store is difficult (How to back up? How to update schema of millions of entries?)

- KV store is slow (400 ms in Europe) unless keys are being hit every couple of seconds to keep the edge cache warm

Currently I’m trying an alternative – Deno Deploy – and love it so far. Local dev server executes your code locally, quick compile times, native TypeScript support due to Deno runtime.

https://deno.com/deploy


> Like, even accessing production logs is weird - you have to "tail" them from your CLI.

Which seems to fail for access-protected sites since `tail` uses the same domain as the real endpoint. Also no error reporting - the exception may be in the logs, but it's missing the stack trace unless you log it explicitly.

The monitoring side of workers is extremely poor currently.

Also unit testing workers is hard to get right - there are no good mock libraries for request and kv which behave exactly like the production versions, so you end up with surprises after deployment.


This is very new, but I can see CloudFlare are trying to solve the logging problem: https://blog.cloudflare.com/observability-ecosystem/


I have not had difficulty writing code for AWS Lambda. I just write everything locally and run the code by invoking the entry point with the proper event and context dictionaries. For debugging, I just attach a debugger and run the entry point like any normal script.

I don't know why you need to deploy to test Lambda code; you can hit remote AWS stuff from local. The AWS SDK picks up your local config's IAM role the same way as it picks up the one granted to the Lambda itself. You don't need localstack for this, just an account in your AWS organization with the right role attached.

Packaging dependencies was a little weird to figure out, but the docs [0] are very good. A simple shell script can do the packaging work; its just a few lines to make the zip file.

[0]: https://docs.aws.amazon.com/lambda/latest/dg/python-package-...


Not a full solution, but when I was doing this I really got to love the awslogs utility:

https://github.com/jorgebastida/awslogs

It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)

Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.

Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.

https://github.com/spulec/moto


awslogs was a massive productivity improvement for me - highly recommended


Suggestions:

1. If you are building APIs and using Lambda functions as targets from an API Gateway API, look into libraries like serverless-wsgi (Python) or wai-handler-hal (Haskell) that translate between API Gateway request/response payloads and some kind of ecosystem-native representation. Then as long as you're writing code where all state gets persisted outside of the request/response cycle, you can develop locally as if you were writing for a more normal deploy environment.

2. Look into the lambda runtime interface emulator ( https://github.com/aws/aws-lambda-runtime-interface-emulator... ). This lets you send invoke requests to a fake listener and locally test the lambda more easily. While the emulator is provided in the AWS container base images, you don't need to run it inside a container if you're deploying with zip files. (AWS-provided container images automatically enable the emulator if not running in a lambda runtime environment, and using docker for port remapping, which is nice but not at all required.)

3. Get really good at capturing all requests to external services, and mocking them out for local testing. Whether this is done with free monads, effect systems, or by routing everything through gateway classes will depend on your language and library choices.


Thanks for these suggestions, they seem pretty solid. I'm particularly keen on the idea of capturing requests/events for use in mocking.

It would be quite a powerful workflow for running integration tests. I shall do some research!


If all you need is the ability to run a lambda function's code locally you might interested in docker-lambda[1]. I haven't really used localstack or SAM but a couple of years ago when we needed to run some lambda functions locally for development docker-lambda worked well enough.

[1] https://github.com/lambci/docker-lambda


Oh that does look quite interesting, I hadn't heard of that. Thanks for the tip.

As you say, it only does the 'host lambdas locally' bit though. If you wanted to do stuff like test the interactions between lambdas, or test them over HTTP (for lambdas that act as components in REST APIs), you'd need to build something yourself.


If I were you, I’d write a little framework that abstracts these interactions. An HTTP call is an RPC call, and putting something onto SQS is just a queue.

You’re not testing that HTTP works, you’re testing that the two functions work together. If you can abstract a calling interface between them with different backends you’ll have a better time developing lambdas, while also not being tightly coupled to them.

Create an RPC container object that has method stubs for the other lambdas you want to call. When you deploy it on AWS configure it to make an HTTP call; when running it locally, configure it to just call the code as normal.

Just think of the HTTP call and JSON blob as the calling convention. The API is really simple.

I think one of the biggest mistakes people make when developing Lambadas is designing them differently to normal software.


> I think one of the biggest mistakes people make when developing Lambadas is designing them differently to normal software.

That's probably very true (and I'm probably quite guilty of it!).

Thanks for your suggestions, that does sound like a good approach.


Don’t bother with Localstack, it’s a bunch of Docker images combined with Python’s Moto library.

AWS has Step Functions, which is a vaguely functional and declarative language for linking your lambdas together. It’s a bit verbose and annoying to write, but at least a it makes your lambada setup repeatable.

The state machine which actually runs your step functions is publicly available.

https://docs.aws.amazon.com/step-functions/latest/dg/sfn-loc...


Seconded that Localstack is pointless. So much of what I want to do isn't supported and/or throws weird errors that are nothing like the AWS service.


I actually feel a little bit of rage whenever I see it being used in a project. People think they’re Integration testing AWS, but they’re really testing the mocking behaviour of Moto over a loopback interface. But all they’re doing is making their test pipeline bloody slow and memory hungry.

What people need to understand about testing against AWS is that the SDKs provides the typing guarantee. So it’s nearly impossible to make an invalid call with any AWS SDK.

Moreover, doing a test along the lines of ‘post to S3, then get the object to verify it has been posted’ is pointless too. Because that’s not how AWS works. In a real application, you don’t have any guarantees the object exists in S3 immediately afterwards, just a high probability that it will become consistent relatively quickly. If you do such a thing in a real world environment, you’re liable to have intermittent failures.

The eventually consistent behaviour is more obvious with other APIs like the EC2 interface, where no one expects an instance to be created immediately.

There are a few exceptions to this rule, but they’re not accurately modelled by localstack.

What people need to test when developing AWS applications is that there _was_ a call to an AWS API, not that the AWS API behaved with all of its documented side effects. You can’t unit test the black box of AWS, you have to QA it as part of an application release.


I've found both AWS Lambda and all Azure serverless tooling to be extremely frustrating. "Just don't do local development" is the feedback I see a lot, which just seems bonkers to me. Like the orgs that don't run tests locally and just use CI to validate builds. If your workflow involves long periods of waiting to test even very basic functionality, it is (to me) totally broken.

We have containers which unify local environment and production environment. In my opinion (and experience), there aren't any more efficient shared mediums to work with.


It sounds like you're committed to Lambda, but we've spent extensive time across both Lambdas and Google Cloud Functions, and my off the cuff, somewhat biased opinion is that GCF gives a significantly better development experience the Lambdas. I note biased because a majority of the work we do lives in RTDB/Firestore.

However, in response to your specific pain point - after initializing a new functions project you can immediately run an 'npm run serve' and have local execution firing for any scaffolded endpoints. These - by default - will connect to a project associated RTDB/Firestore instance without the need for additional service account/IAM configuration.

I've always enjoyed that this was a part of the mainline dev flow - and it removes a huge barrier of entry for me (mostly mental) to spin up little toy projects or test endpoints whenever the thought crosses my mind.


This response surprises me. I find Google Cloud Functions to be orders of magnitude worse than Lambdas. Everything that you deal with in AWS with the added bonus that instead of taking 5 seconds to deploy it takes 5 minutes. I would love to know better options. How do you deal with that horrible deployment time?


The simple answer is we don't. I've never even considered deployment time as a base metric, and I guess it's equally surprising to me that that's an issue. The fact that we can do all of our testing locally means that we rarely have to do the "log and redeploy" dance that the OP was referring to.


Right. I definitely try and do as much testing locally as I can, but it still often takes me 10-20 updates to get everything working (missed an environment variable, permissions different in the cloud, blah blah blah). That’s usually a couple hours of my life i don’t get back.


Yeah, completely understand. I think the only issue we've run into that's forced us to go back to the trough with multiple re-deploys is very early on when we were negotiating better CORS patterns.

Other than that, the fact that you can now fully simulate not only the GCF endpoints, but also all of RTDB and Firestore locally - and with a simple debug toggle to connect to stage/prod (insert safety disclaimer here) is a really powerful paradigm.

Certainly not something you couldn't do similarly with Lambda/Dynamo/API Gateway using IAM roles or whatever - but it just feels very first class citizen with GC.

Also, as an aside - I wouldn't normally try and convert someone from AWS => GC, because I think you can be equally effective in either. I only dropped my original response because the OP specifically called out local emulation being an issue.


1) If you're using Google Cloud Functions, you might also want to check out the Running Functions Locally doc page. [1] (There's an NPM library [2] - @google-cloud/functions-framework - that handles the scaffolding for you.)

Note that this library only supports scaffolding one function at a time per process - so you'd have to create a separate process [and use a separate port] for each function you're using. (This includes event-based functions - everything is HTTP under the hood, and this library is no different.)

2) Deployment time - you're definitely not the first to bring this up. :) We're working internally on improving deployment times.

(Disclosure - per my username, I work at GCP.)

[1] https://cloud.google.com/functions/docs/running/function-fra... [2] https://www.npmjs.com/package/@google-cloud/functions-framew...


I appreciate this comment. I spent 4 hours updating-deploy-test-repeat with GCP last night trying to track down a couple of niggling errors and it was a nightmare. I'd have killed to be able to step through it just once with a debugger and everything would have been fixed.


I share these frustrations and don't have a great answer. When integrating N cloud services, at some point it gets incredibly hard to build out fast feedback / local environments. The best way I know how to deal with it is use less services and ensure the ones you stick with are simple.


I am not alone then at least! The particularly annoying thing is that I feel like N isn't even that high. Even 2 or 3 things interacting is a pain (unless it's stuff like RDS or just a plain old container of course).


I write Lambda functions in golang. In dev I run the function as local http server. In prod I map a custom domain to the lambda, routing is done internally to the function. API Gateway is only used for the top level route. This workflow is enabled by https://github.com/apex/gateway

In principle I try to avoid Amazon Web Services that lock me into the platform. So, for my dev stack I run a few other containers to give me the equivalent of the prod environment, RDS (MySQL or PostgreSQL), ES (OpenDistro), S3 (Minio), and SES https://github.com/mozey/aws-local

Dev is easy because I can test everything locally, and the resulting code should be portable to any other cloud.

Obviously this approach might not be feasible if you choose to use AWS that do not have equivalent self-hosted options.


Seems like you use Lambda to host an HTTP service? apex/gateway is really nice to implement the server using standard Go net/http interfaces instead of proprietary AWS SDK. But why not use ECS instead of Lambda and just host containers running your HTTp service?


Lambda because pricing, I don't have to pay the per hour fee if the service is not used every hour/day/etc. By using apex/gateway I can easily move the code to ECS if that becomes the better option.


The best solution I've found for local testing is the Lambda Runtime Interface Emulator (RIE)[1]. It's basically a little Go wrapper[2] that runs your handler and makes it callable through an HTTP API. I considered writing a test harness around this for convenient integration testing but in my recent evaluation, I ended up avoiding Lambda (it's been really nice for small one-off tasks but was too expensive for the most recent and somewhat fringe use-case I considered using it for).

[1]: https://docs.aws.amazon.com/lambda/latest/dg/images-test.htm... [2]: https://github.com/aws/aws-lambda-runtime-interface-emulator


Ah that's a good idea - I have actually played around with that. When they released support for running lambdas with container images I used it to have a go at making a custom lambda image.

I didn't join the dots up for this use case though, thanks!


I did the same and learned a ton about their runtime! In my case, I did similar and also had a go (pun intended) at shipping a single-binary Docker image (using the scratch base image). Best of luck!


That can be true if you are using tightly coupled event sources (like s3, SNS, etc) where you need to inspect the incoming object. If you are doing a HTTP / REST API, try to decouple the code as much as possible from the API Gateway invocation by using ExpressJS / aws-serverless-express to aid in local testing and debugging. Then testing and debugging locally becomes much easier.


This is my preferred style, and since an express app is technically just a function that accepts req/res, you can pass a whole express app in as the function when running in the cloud, so you have multiple path routes, middleware, the whole nine. You can actually host a fairly complex API in a single function subject to the package size/dependency count restrictions of your FaaS of choice


There are a few ways to improve the feedback loop, like SAM, Stackery, and Serverless Framework. That said, containers on Lambda (or GCP Cloud Run) are where it’s at. Just code your code, run local unit tests, and push the container up. The most important thing is to clearly define a mock request from your invocation source as your first step of the project. There is a vast difference between SQS, ALB and API Gateway events that need to be accounted for, and trip up people on deployment. I usually have those in my tests folder for every event the code will handle, and build from there.

‘aws lambda invoke -e myevent.JSON’ goes a long ways, because you get the full response object back.


Have you tried using serverless/serverless-offline? https://www.serverless.com/


I haven't tried it yet, no. It always struck me as being focused on making deployment quick/easy rather than on creating a good local dev workflow (i.e. the same workflow I currently have but a quicker cycle time). This never really appealed to me.

Cosmotic's comment kind of confirms my suspicions to be honest.


Couple of notes I'd like to throw out there... - The Serverless Framework does help ease quite a few of your concerns, local development is made much easier (they have a node server for js development). The company working on it is kinda funny, but I still think it's a good tool. I currently use the raw lambda environment at work and can say that the serverless framework is a valuable abstraction. - I think Cosmotic is making their deployment setup more complicated than it needs to be. I obviously don't know their use case but there are easier ways to accomplish a CICD w/ FaaS... - I would recommend trying it. Its pretty painless, and you'll know within ~2-4 hours whether or not it will work for you


It's quite limiting on: 1. It only mocks API Gateway, and not accurately (ie. IAM permissions). You also need to use a bunch of other similar plugins to mock DynamoDB, SNS, SQS, etc. 2. You need to change your code (wrapping your aws-sdk to connect to localhost) to use these mocking plugins.

I posted elsewhere in this thread but checkout SST (https://github.com/serverless-stack/serverless-stack), it lets’ you work on your Lambda functions live without having to mock AWS services locally. Here’s a short demo (https://youtu.be/hnTSTm5n11g)


Our team has; it's been very difficult. Our jenkins build agent (running as a container) runs integration tests that start a node+serverless container, that starts a localstack container, that starts a lambda contianer.


> Alternatively, do other FaaS providers solve this problem?

I worked on Netlify Dev, which does offer a simulated lambda environment for testing locally: https://news.ycombinator.com/item?id=19615546

I was also quite disappointed to see that AWS Amplify did not offer a running dev server for lambdas, best you could do was a local onetime manual invoke and even that often failed (mismatched to the live lambda environment where it succeeds)


IN a direct reply to Op: There is definitely a lot that goes into learning serverless tech like AWS Lambda.

I lead Developer Advocacy for Serverless at AWS and we're constantly looking at ways to help streamline and simplify this process for you. You can find a lot of information on a site we've launched to rollup lots of resources: https://serverlessland.com .

Right now the best dev process is a mix of using local emulation via AWS SAM CLI (http://s12d.com/sam) or tools like localstack. But it's true that you'll never be able to completely mimic all the various services and their functionality.

Would love to know more of what left you feeling underwhelmed by SAM. It has almost all the functionality of CloudFormation, which is quite a lot.

Thanks, - Chris Munns - Lead of Serverless DA @ AWS


I haven't any suggestions but can relate on the frustration part.

We recently started using AWS Athena and I was shocked to find no local test environment is offered by Amazon. Eventually I built a test container that uses Hive and Presto and allows my integration tests to deploy their test data to the Hive filesystem for Presto to access it. Unfortunately, Presto deviates from Athena so I can only approach the production environment during development, which leads to much lower confidence than we would have had otherwise.

Essentially, after all the tests are green, I'm still not sure if my code is production ready. Which is bewildering to me, we made all this progress in the past decade with continuous deployment, this just seems like a large step back.

It's not like it's that hard to fix for Amazon, they could at least offer local test containers for their cloud offerings. They just choose not to.


Probably lost in all the replies but use can use AWS SAM start-local[1] to run a lambda locally.

There's also a docker container to mock dynamodb locally.

When I'm just working logic... Unit tests are best, but you could also keep the lambda invitation endpoint to lambda_handler, then write your own main() function to do some setup (like assume role to the same role as the lambda execution role, set up env bars)... Then end by invoking your lambda_handler locally... Basically main() is your pathway to running the lambda locally.. with only the minimum code required to "fake" being an AWS lambda.

1: https://docs.aws.amazon.com/serverless-application-model/lat...


> do other FaaS providers solve this problem?

Cloudflare's Workers have a good dev experience. It takes a second to build and locally preview a function. It live-reloads when you make a change. Later, it takes a second to build and deploy. On the flip side, Workers can't run arbitrary Docker containers. Your code has to be JS/TS or WebAssembly.

I'm not affiliated with Cloudflare, just a satisfied customer.


> Cloudflare's Workers have a good dev experience

My experience has been the opposite.

Wrangler doesn't give you any meaningful errors (since it converts JS errors to Rust) and it crashes way too often.

Recently I worked on a Workers project and I had an undefined missing variable. The only error Wrangler would output is "error".

IMO Workers are only viable for really small and simple projects.


> On the flip side, Workers can't run arbitrary Docker containers.

Containers are coming to the Cloudflare edge: https://blog.cloudflare.com/containers-on-the-edge/ (inevitable really)

Nit: Workers should really be compared to Lambda@Edge instead.


That's interesting, thanks for the recommendation. I have heard good things about cloudflare workers too and the local dev setup you've described sounds quite good.

Am I right in saying that they serve a somewhat narrower use case than lambdas though? My understanding is that they are mostly used for things like modifying requests/responses rather than for building a whole backend.


You're not alone. I've had this exact experience so much so, that we stopped using AWS Lambda, and switched to using the the various Amazon API calls individually and gluing together a pipeline using a job queue.

It's just so slow to debug and develop on AWS Lambda


We run Azure Functions with C#, it's not perfect but it's very easy to test, runs with a single click locally, deploys fairly easily to Azure as well(though it has some quirks) can use 99% of our codebase as a dependency without any issues.


Yes, the AWS serverless is essentially half-baked for this reason it would be considerably more powerful - to the point of changing how we make apps - if there was a local environment provided by them that could really make testing and deployment seamless. Being able to see a debug of 'live' lambdas in your own cli etc. would be ideal.


My tinfoil hat theory is that making local development difficult is the goal rather than a side effect. Why not just run the stack yourself if it's that easy? Better for Amazon if the development process for using their services involves as many other AWS services as possible.


I wouldn't put it past them to do something like that if that were indeed going to work out, but the more reasonable thing is that they're just bad at it. Lambda was weak for years. Also, a 'local stack' for testing probably wouldn't cover nearly enough basis for people to 'just to it themselves'.

Literally today I had a billing problem with AWS and it was one of the worst, broken UI experiences I've ever seen. Involving money to boot (usually those things have higher standards).

They are really bad at quite a lot of things, it is what it is - there might be some rhyme or reason to it, for example, they might not care about documentation, or may be much more focused on bread and butter ec2, bit it's totally plausible that there are a lot of frayed edges.

You know that difficult-to-attribute quote: 'Don't attribute to malice that which can just as easily be attributed to incompetence'.


Wow, that is exactly what we are making. I posted elsewhere in this thread but SST (https://github.com/serverless-stack/serverless-stack) lets you work on your Lambda functions live without having to mock AWS services locally.

Here’s a short demo (https://youtu.be/hnTSTm5n11g)

I'd love for you to give it a try.


That looks cool I will check it out.


Awesome! We also have a super passionate community at https://launchpass.com/serverless-stack


AWS Lambda is a great replacement for cronjobs when gluing various AWS infrastructure components together, but I would hesitate to use it for an actual application.

Pre-packaged lambda functions are a pain to deal with if you're standardizing on something like terraform for all AWS infra provisioning - you need to jump through extra hoops to repackage the lambda functions into something you can manage.

Replacing the zip deploys with Docker images is a good first step. I just wish they'd support inter-account ECR -> Lambda usage - ATM you still need to replicate the images to each account separately.


We're using it "the right way" with SAM, but still very frustrating.

We had to detect if running locally or deployed in our code because Cognito auth was giving us trouble when running locally, and Step Functions can only be tested deployed.

Cloud Watch logs take several minutes to appear, funcs take some time to package and deploy (even with caching) - making feedback loop very slow.

I personally think the best way is to develop serverless funcs on platform - using the built in editor. You get logs instantly. But can't get devs to give up their IDEs...


+2x - The logs are funny. they'll duplicate, show up late, etc


I completely agree, especially having just come from working with Azure Functions. This likely also applies to other runtimes, but definitely for dotnet, you can run all Functions in a solution locally via the default F5 debugging feature in Visual Studio. Stellar developer experience.


It’s astonishing how entrenched is the HN type in AWS that this no one said it before.

P.S you can even attach the VS debugger to the live, deployed remote function and get full debugging (although quite slow). Don’t forget to deploy with the Debug (rather than Release) profile if you wanna do that. Breakpoints and all!


And even if you're on Linux, there is a VSCode addon that is equivalent [1]. And even if you can't / don't want to use VSCode or the addon, you can download and run the Functions host yourself via the `func` tool [2]. (The latter is what I do for my own Functions.)

One caveat with these options is that the Function won't get a Managed Service Identity since it's running locally, so if you have code that relies on an MSI you'll have to create a Service Principal and make your code support SPs too.

Apart from that, the beahvior of the Function running locally or in the cloud should be identical.

[1]: https://marketplace.visualstudio.com/items?itemName=ms-azure...

[2]: https://github.com/Azure/azure-functions-core-tools/releases Azure.Functions.Cli.linux-*


Hi tdfirth. I am with the team at Serverless, Inc, creators of the Serverless Framework. I can totally understand the frustration, especially with new development architectures such as serverless.

Over the years of building serverless applications there are some unique differences in building them compared to traditional applications you build entirely locally then "throw over the wall" to be run on a server somewhere. And for this reason there are many teams and organisations working to make the developer experience better, including making deployments to the cloud to test easier and faster. Unlike mainframes of old, new code can be deployed in seconds and logs available in less than that when testing invocations of your Lambda functions. For example, the Serverless Framework has a command to deploy just the code changes of a Lambda and does so in seconds. There are other tools that do similar.

Its a really broad topic to get into but am more than happy to do so if you wish or have any specific questions I can help you answer. If its easier to ask privately feel free to do so

And I would just like to leave saying that I am a proponent of serverless development not because I happen to work for a company bearing the name, but because I sincerely think its the future. They just happened to spot that and asked me to join them.


I actually find extremelly easy to develop with lambdas, there's many options for development, testing and debugging. things like serverless, localstack and even Amplify can help you with local testing and local development.

For debugging in the cloud, besides cloudwatch you can use x-ray and have a full picture of your services and segments.

And if this ins't enough and you really need to test in the cloud, then use CLI or serverless to upload your lambda(few seconds tops)

invooke the lambda with test data via cli https://docs.aws.amazon.com/cli/latest/reference/lambda/invo...

and tail the logs via AWS CLI.

https://awscli.amazonaws.com/v2/documentation/api/latest/ref...

I mean, those steps aren't much different to running tests locally, just create a script for it =)


Try knative on k8s.

1. Some great tooling out there to shorten the dev loop on k8s. I use tools like skaffold or apache kamel k, but telepresence looks interesting too. 2. Minikube solves the "Localstack" issue. It's an official project that runs the exact infra, not just a replica of the APIs. 3. logging is far more straightforward than cloud watch. 4. It runs the same everywhere.


Yes, I tried it once for a small job for a client, thinking this was so small that it would be really nice not to deal with servers or anything. Boy was I wrong. NEVER AGAIN. God what an unpleasant workflow. Even on a tiny job the disruption to my usual productive code workflow ate up way more time than standing up an EC2 instance and installing stuff. ugh.


I'm reminded of a suggestion from someone else a few days go if you don't mind PHP

>https://bref.sh/ -- it allows running PHP apps on AWS Lambda effortlessly. With all the serverless craze, it's a very important piece of the ecosystem. After writing PHP apps for two decades, I am absolutely stunned how easy it is. Amazon blogged about it at https://aws.amazon.com/blogs/compute/the-serverless-lamp-sta...

(comment https://news.ycombinator.com/item?id=26827819)


If you are ok using a ‘higher level’ serverless provider, vercel offers first class support for running and debugging serverless functions locally, in an environment which closely mirrors their production environment (which I believe is mostly AWS under the hood).

See the following for more information:

https://vercel.com/docs/serverless-functions/introduction#lo...

I tried a few solutions earlier in the year for a side project (vercel, netlify functions, Cloudflare workers, begin) and found vercel to be the outstanding choice for developer experience, features (wildcard DNS for example is fully supported on their free plan) and support.


We operate a serverless infrastructure (lambda + dynamodb) at secapps.com with a super small team, and we could not be any happier. Mind you, currently, we have over 30 services (not microservices but products we sell to clients) and well over 30 applications.

While you should take my comment with a grain of salt, for me it is a live testament that the serverless development paradigm works.

That being said, I also believe that it depends on how exactly you work and develop products/features. That is probably more important than the technological approach used. We have some very strict mental model about lambda which do not quite fit the mental model used for microservices.


Would you please share more details about your mental model for working with Lambda?


Yes. I only use Lambda when forced to by a client. 99% of the time it's a poor fit for their use case, anyway, but they've still got to have it to make someone happy.

I feel as though there are similar amounts of friction across AWS offerings that make them unpleasant to work with in general, too. I'm numb to it now, but imagine my surprise when I actually went to use it for the first time after being hyped up about AWS.

For these reasons, and because you pay a significant premium for network traffic on AWS, I never use AWS for personal projects and I'm very happy with that choice.


(Disclaimer: I work on Google Cloud Functions)

With Google Cloud Functions, you can run your function locally using the Function Framework libraries (available in all Cloud Function languages) [0]. These libraries are used in production Cloud Functions, so you're not emulating the runtime. The feedback loop / devX is pretty simple with just 1 command to run the function locally like a library for every language.

We have some guides on local development here [1]. Our samples have system tests that use the FF.

You also can bundle your function in a container after building it with `pack build` [2]. This provides an exact container environment with things like environment variables and system packages.

We're working on improving eventing devX, but we have some `curl` examples here [3]. Connecting multiple functions together eloquently is still something to be desired though.

[0] https://github.com/GoogleCloudPlatform/functions-framework [1] https://cloud.google.com/functions/docs/running/overview [2] https://cloud.google.com/functions/docs/building/pack [3] https://cloud.google.com/functions/docs/running/calling


Here are a few libraries that make testing lambdas (and simulating adjacent infrastructure like queues) very easy in tests.

localstack

mintel/pytest-localstack

charlieparkes/boto3-fixtures


I found it extremely frustrating in that what I expected/hope for was that I'd write a function that got called with some object that had the headers and body of an HTTP request to which I'd output a response.

Instead what I got was a (massively over engineered?) front end (the API Gateway) where the default is to add 10s or 100s of rules to try to have Amazon parse the request. As a JS dev I can only guess this was designed by Java programmers where strict typing might suggest such a convoluted solution?

It took a while to figure out how to turn off all that stuff and just pass the request to my lambda JS based function.

I feel like API Gateway should be doing absolutely nothing except routing requests to the appropriate service. That should be the default and all that parsing should be a separate service. In other words it should be

API Gateway (no parsing) -> lambda

and if you want parsing then you'd

API Gateway -> Parsing service -> lambda

Or for that mater just make the parsing a library you can use if you want.

OTOH I'm not a network person so maybe there's valid reasons for API Gateway to have such a giant configuration


I typically write my Lambda functions as Nest apps, develop locally, and then deploy with the aws-serverless-express (or fastify) NPM package, and I enable The ability to trigger a kinesis/s3/etc events locally as well. The fact that my app gets deployed to Lambda doesn't really have any impact on my developer experience. What stops you from working locally?


This has worked very well for me, too. I suppose it might start to break down if you're using some of the more complex AWS-specific offerings.

But if your function more or less resembles a little web app that maybe talks to S3 and DynamoDB, this works really well.


> What I want is to run the lambdas locally, ideally more than one, and then exercise them with streams of test events (perhaps captured from a real environment).

lambci has dockerized near perfect replicas of the lambda runtime environments, so as long as you aren’t relying on VPC endpoints tied to specific lambdas, or things like that, you should be able to, even without SAM.

But you can also: (1) deploy separate test copies of lambdas and fire test events at them the same way you would locally, which has full fidelity and no third-party deps. (2) throw test events at your lambda entry points locally and capture and validate what they return.

I do both regularly with python lambdas (I find that the latter is adequate for testing lambdas and their downstream interactions, but the former is useful for validating interactions upstream, particularly—for most of the lambdas I work with—with APIGateway and some auth stuff we have hooked in there.)


Yor sentiment is beautifully captured in the essay "The Eternal Mainframe" [1] by Rudolf Winestock.

It gets reposted every year or so here on HN.

[1] http://www.winestockwebdesign.com/Essays/Eternal_Mainframe.h... (2013)


I’ve got about 300 c# lambdas and dont have any development friction.

Curious what sort of errors OP is running into that cause friction.


Probably because you realized that lambdas are in fact functions as the name indicates and you know how to compose software with functions. Most people are trying to build out monolithic scripts that should be broken up into...functions. It's like kicking a dead horse sometimes.


I had exactly the same experience you describe. I’ve developed serverless code on Netlify and AWS Lambda.

I managed to get what I needed working but it was laborious compared to typical local server development.

I feel like most of the tools were written by devops folks because it’s quite easy to run these deployments in production, but the dev experience is lousy. The basics for developers is to be able to run the code locally and be able to run everything in a debugger. And that needs to include some sort of way to run services like dbs, queues etc. That’s just not possible as far as I can tell.

They say that cloud is somebody else’s server, well serverless is just somebody else’s monolith, and you don’t get a copy of it to develop on.

Not sure if it helps but here’s a demo IoT project I developed, has unit and integrations tests, it’s simple to understand, deployed using serverless:

https://github.com/mjgs/serverless-books-api

A few things I found useful:

netlify-cli - CLI tool for interacting with the Netlify API, great for deploying to Netlify when you aren’t building your site on their infrastructure.

netlify-lambda - CLI tool for building and running your Netlify functions locally.

serverless framework - A CLI and web tool to build and deploy serverless architectures, includes supports for Amazon, Azure, Google among others. Makes it easy to configure and provision all the components of your Cloud Native application.

These were taken from a blog post I wrote which some might find relevant:

Cloud Native web application development: what's it all about?

https://blog.markjgsmith.com/2021/01/12/cloud-native-web-app...


Is AWS a requirement? I've just started a project that is roughly... 75% Azure Functions and so far it's been a dream. Between the triggers and the outputs, and being able to run functions locally in VSCode I feel like it's the perfect way to "glue together various bits and pieces" as you said. Obviously a lot of the triggers are Azure-specific but between HTTP and Service Bus Queue/Service Bus Topic you should be able to do just about anything you need.

On the plus side of having a lot of the pieces in Azure, if you're lucky enough to have an input and output match (e.g. you want to persist SB Topic messages into a SQL API CosmosDB instance) it's a half dozen lines of code and you're done.


A big part of this is the switch from traditional programming (just write code) to cloud based programming (combine cloud resources together like "lego blocks"; code is just one block among many others).

This reduces the importance of writing code and shifts focus into new development tools like AWS CDK. More and more of development work is about defining Step Functions logic and setting up various cloud events that hook all the pieces together.

It's not trivial to replicate ALL the hundreds of cloud services at home. So we accept that development is done in batches and deployment takes a while. It could be better if course, but it's not only about Lambda any more. It's about CloudFormation in general.


People shove too much complexity into a lambda. It seems like a great idea to have this infinitely scaling lambda with no infrastructure to run, but what you're trading off is developer experience.

It sounds like your lambdas are calling other lambdas. If this is true, rethink your architecture.

If you have a thin, minimal code wrapper that translates your lambda invocation event into some sort of function invocation, it shouldn't be hard to test. Your external dependencies should be modeled as interfaces that can have fake implementations injected that either return pre-canned responses or implement things in memory.

Without more info about what you're doing, it makes it really hard to know what specifically is stopping you.


My company uses separate stages (through serverless framework) for each branch of feature development. CI uses these to test, running within AWS infrastructure. It works well for isolating test environments. We lean more on unit testing to debug the service with external dependencies mocked. We try to avoid situations where we are adding log statements one-by-one; CloudFormation deployments take several minutes, so iterating small changes is indeed frustrating. We thought about localstack, but haven't tried it. I'm worried that it will increase our IaC complexity; everything will have to deploy to both AWS and also localstack.


Yeah I wouldn't go down the local stack path. IMHO, you should develop/test against real AWS services, and it falls on the tooling to make that experience fast/smooth.

I posted elsewhere in this thread but checkout SST (https://github.com/serverless-stack/serverless-stack), it lets’ you work on your Lambda functions live.

Think of it like your Lambda functions are Live Reloaded as you make changes to your code. You don't need to mock AWS services locally, and you don't have to wait for CloudFormation to deploy your changes.

Here’s a short demo (https://youtu.be/hnTSTm5n11g)


Live reloading sounds like it would relieve some pain with deployment. I'll try it out with SST.


Awesome! We also have a super passionate community at https://launchpass.com/serverless-stack


Have you considered just using lambda to http server adapter? That way you can just write a normal http server and test against that locally as normal, then when you publish to a lambda, the adapter turns AWS lambda json to http requests?


I don't understand the benefit of that. Could you explain please? It sounds to me like doing what you describe would prevent all the wins of serverless - scaling and only paying for the runtime you use. In what you describe, you get zero auto-scaling unless you implement it yourself, and you pay more than just lambda or just a http server. It sounds like adding another layer of complexity on top for no benefit


I think the idea is to run a regular http server locally that forwards requests to your lambda handler function for the purpose of testing. When you actually deploy your lambda, you just hook up the lambda handler function and leave the http server out of the picture.


I have considered it, yes, and there are a couple of projects that make the HTTP aspect of lambda easier, it's true. However running HTTP requests through lambdas is just one use case for them! For other kinds of events there doesn't really seem to be an equivalent.

It would be great if there was a consistent way to do all of them.


What I’m reading from this thread:

- cloud native is weird/hard - old ways were better! - misconceptions about boundaries and aspects

Remember folks. When choosing technology, your selections always depend on a list of concerns, including:

- cost - complexity - developer audience - support audience - change management - vendor relations - mapping out dev strategies - getting consensus from team

No matter what tech is used, you still need to go through planning and design.

More often than not, cloud enables us to hack first. That is never a good strategy.

There’s nothing inherently bad about cloud native. It’s just different and _can_ be highly beneficial to outcomes.


Are you sure the folks deciding the tech are the ones who browse HN?

I was in that position once, with Microsoft functions, which my boss wanted to use, but fortunately convinced my boss to get rid of it. Maybe others don't have the same luck.


I personally like SAM, but there is https://github.com/dherault/serverless-offline as well


I am working on a side project (more of a way to learn go than anything) that makes “serverless” functions as simple to deploy as possible. Basically, a developer posts some JavaScript to an API endpoint, and they are returned a function id and a management token. The function can be run at URL.com/functionid and to delete or modify the function the management token must be used. It all runs in a V8 isolate (ala Cloudflare Workers). It’s a self contained go binary, so it should be super easy to run locally, too.


You can run your code locally and still interact with real AWS services. For example, using Serverless Framework, you can do "sls invoke local", and your calls to DynamoDB etc. will work fine.

The problematic bit is the input event. You can't call real API Gtw. to trigger your Lambda code locally. You need to write/copy the request as JSON and provide it as input. It's more annoying with triggers like Kinesis Stream, but doable.

I personally don't like Localstack, it's safer to use real services.


Maybe someone else has already mentioned but anyway... The things I do: - Lambda event passed to your lambda_handler is just a big json, get it and copy it somewhere, and then you can test your code locally. ( lambda_handler(your_event, None) ) - If there are some things you cannot do locally, use some flags to handle them. (You may use: MessageGroupId from the lambda event for example) - Prepare a deploy script and use it. (aws lambda update-function-code ...)


It’s just not a realistic way to debug and test. Data is very different across environments and use cases. Unless it’s a bumble fuck add(x, y) lambda, you are effectively suggesting an unrealistic solution. I NEED to reproduce the system locally.

And, why is this such a wild expectation in 2021? Is it that so much to ask?


I feel your pain. I developed an event handler and messaging buss framework that allows me to run end-to-end workflows in my IDE. The framework is smart enough to know if it is running in the cloud or locally.

The other challenge the framework solves is that payload wrappers are different if there is there is an sqs component in front of the lambda, or an sns in front of that. The execution code of the lamda should be completely agnostic to how the payload arrives.


For developing API Gateway web servers on Lambda in Go, I use an adaptor, so it becomes a standard HTTP request, and I can just write a web service like any other. https://github.com/carlmjohnson/gateway For event based stuff… I have always avoided it for exactly this reason. Seems completely untestable and a pain to get working right.


Maybe you should give OpenFaas a spin. You can run it completely locally on your machine with Docker . It's mostly based off of Kubernetes. But you can also test it on EC2 or any virtual machine using faasd.

https://docs.openfaas.com/ https://github.com/openfaas/faasd


One way to make your life easier is to use a JetBrains IDE with the AWS Toolkit extension. Many of the pain points you’re experiencing are greatly eased by that.


My co-founder and I built https://napkin.io for this reason. We thought the experience of onboarding and stitching together services with AWS was too complex. With Napkin you can deploy a Python API endpoint from our landing page with a single button click.

We're actively developing this so any feedback of how we could make this better for your use case is welcome!


Part of my frustrations with FaaS lead me to develop serverless cells where there is no deploy step.

https://observablehq.com/@endpointservices/serverless-cells

Source code is loaded dynamically each invocation.

Still, some work todo, it's still annoying to program. Next hope to make them easy to use is pipe the runtime to Chrome devtools


Isn’t that why aws created docker packaging for lambdas or am I missing something.

The zip file packaging and debugging in the cloud has always frustrated me as well. But I like the docker packaging, still a long way to go but atleast I can somewhat mock some inputs and output triggers.

I have no idea how people managed the zip file uploads especially for python projects, it’s an insane waste of time uploading and debugging on the cloud.


I haven't use it yet but CDK seems to make it easy to build and tear down infrastructure for development and testing. My strongest language is python and I like the idea of less context switching and being able to spin up an aws account just for development and delete it after prod deployment.

For lambda based API's I am curious how it compares to cloudformation and terraform?


I write Go lambdas and run HandleFunc normally as an AWS user who has assumed the appropriate role.

Then when I'm done, I push `lambda.Start(HandleFunc)`

Frequently, though, when I want to change some code I already have, I just run through the edit-build-push-invoke loop. It isn't optimal but with fast Internet it is not much slower than a Scala build and I can iterate with that.


Firebase Functions has a local emulator and a unit testing module. The emulator is quite good for testing your code locally quickly.


I am a fan of localstack. I know it isn’t perfect and I hear a lot of the complaints that others have posted. However, it covered everything I’ve had to use so far, mostly lambda, rds, sqs, sns, iam. Sure, won’t work for everyone, and there might be easier ways for me to test, but it fits my needs and has saved me so much time and money.


Not sure if anyone mentioned this, but I write a good bit of Lambdas by getting the boilerplate for them to run through Express.js for local development. Makes it possible to interactively debug with Visual Studio Code pretty easily. It requires a little initial startup cost for a project to connect them but works great afterwards.


We’re testing lambdas by running it with Node and enabling the local testing code with an environment variable. Internally we’re just calling the lambda handler with fake parameters to enable bdd like testing. Entire test suite runs before deploy, always worked flawlessly for us.


Use serverless framework to deploy your lambdas, use serverless-offline to launch them locally.


`sls invoke local` is great - do you know if there's a good way to mock dyanmoDB, S3, RDS etcc, with Serverless Framework as well? our python stack is lots of SF services and we are struggling with reliable testing


I've seen the dev team using https://www.npmjs.com/package/sls-plugin-dynamodb-offline

Not sure about RDS or S3, possibly use the local or dev resources in aws


Have you tried the recently added container support? There's a component provided called the Lambda Runtime Interface Emulator that exposes an API for the locally running container. In my experience this is miles better than the original approach.


You should check out Vercel.

It uses AWS Lambda underneath but it makes the dev experience just awesome.


I thought Vercel was for building static sites/backends, or would it also work for generic AWS glue code? (e.g. run this lambda when the bucket changes).


It's a combination of static + cloud functions (AWS Lambda) + CDN.

AFAIK even though it's running AWS Lambda, the cloud functions can't be integrated directly with other AWS services.


https://github.com/localstack/localstack

Why not try to use this toolchain to local build/test your server less application


Depending on the language you're using, one thing that works is: 1) use a framework to handle the nitty gritty (like Serverless, Zappa, or similar) 2) definitely use "devDependencies": { "serverless-python-requirements": "^4.3.0" } for python 3) don't be afraid to use Serverless 1.84 instead of 2.xx 4) I use pipenv (but poetry is probably fine too) and these: black = "" ipython = "" mypy = "" moto = "" pytest = "" pytest-cov = "" python-lambda-local = "*" The last one lets me drop this in my Makefile: try: pipenv run python-lambda-local -t 60 -f identify handler.py event.json event.json is a recorded event (or a carefully crafted one)


There are some plugins for PyCharm (and possibly other jetbrains editors?) that let you see all your lambdas and invoke them right from the IDE.


c# with lambas you just click 'play' locally and you can invoke debug step by step etc, could not be any easier. Which lang?


That sounds really neat, I must admit I've never written a single line of c# although I keep meaning to give it a go. Maybe this is my moment...

I'm using python mostly atm. I've played around with go/node too (both of which I'm familiar with in and of themselves) and I was thinking about switching over to one of those already.

The dependency management is easier for both of them than with python I found.


Very! serverless-framework helps a little bit but overall is missing a lot of what we are used to in the development cycle.


I was blown away that the service has been around for so long yet the process of installing packages was so manual


You can start a local lambda server and invoke the functions without a deployment.


Why can't you test the whole lambda ? Just invoke the handler in the test.


There are many monitoring/observability services that help.

- Dashbird

- Thundra

- Lumigo

Cloud9 has direct Lambda integration, which helps too.


begin.com helps address a lot of the issues here. If you want to deploy your own arc.codes is completely open source, works locally and generates standard aws sam flavored CloudFormation.


Lambda works fantastically when you're working in AWS' little box.

Processing kinesis streams, monitoring and munging S3 objects, auditing dynamo tables, etc. Web lambdas are not worth it imo, very complex and difficult to get right.


I swear on glitch.com!


You can use serverless framework


Ye


Full disclosure: I work for AWS, in fact I work with the Serverless team as a Developer Advocate This is a great thread and I understand some of the pain points that have been talked about. Serverless changes the model for how we develop and test code. There is a want to have everything local. But it is tough to build and maintain local emulators for all services. With that in mind, I encourage you to change the mindset of bringing the cloud to the developer to bringing the developer to the cloud.

When building serverless applications, the most tested and iterated upon part of the application is our code which usually resides in a Lambda function. Testing Lambda functions breaks down to to angles. 1) invocation: testing services invoking a Lambda function, and 2) action: what is the Lambda function doing. This is the only part of the application that should be tested locally through local emulation. The rest of the application is best tested in the cloud.

IMHO the best way to test a Lambda function locally is with AWS SAM: https://aws.amazon.com/serverless/sam/

For testing invocation:

A Lambda function can only be invoked through the AWS Lambda service. SAM has three ways to emulate the Lambda service: 1) invoke - locally invoke the Lambda function one time [https://docs.aws.amazon.com/serverless-application-model/lat...] This functionality is helpful if you want to mock invoking a Lambda function from a service like S3, SNS, SQS, etc. Simply add an event. To create the proper event structure, SAM provides a command called generate event. [https://docs.aws.amazon.com/serverless-application-model/lat...] 2) start-lambda - start an emulator of the Lambda service that can be reached via SDK or AWS CLI [https://docs.aws.amazon.com/serverless-application-model/lat...] 3) start-api - start the Lambda service emulator with a basic API Gateway emulator wrapped around it. This creates a local endpoint for each Lambda function that uses an API GW event source [https://docs.aws.amazon.com/serverless-application-model/lat...]

For testing action:

Using one of the above commands will invoke a Lambda function. The Lambda function will run locally and provide logs as well as stepping through the code in an IDE like AWS Cloud9 or VS Code. The Lambda function can also call out to service like DynamoDB, SQS, SNS, etc that reside in the cloud. Once the Lambda function is working as expected locally, it's time to deploy to a development environment and run E2E tests.

One other tool I would suggest it SAM Logs. SAM Logs can output logs for a specific Lambda function from CloudWatch to your terminal. This is a great way to debug async Lambda functions in the cloud.

I encourage you to visit https://serverlessland.com where we are constantly adding content to help developers with Serverless on AWS. I also have a series of SAM videos at https://s12d.com/sws. Additionally, we host Serverless Office Hours every Tuesday: https://twitch.tv/aws or https://youtube.com/serverlessland. During this time we answer any and all serverless related questions.

Having said all this. Our team is continuing to work towards making the development experience better. Threads like this are critical to our understanding of developer needs and we read them and take them to heart. If you would like to have a longer conversation please reach out to me at ericdj@amazon.com or @edjgeek on Twitter.


I tend to start with a type-checked language (usually Scala) since the AWS libraries have a lot more structure than the arbitrary dicts-of-lists used in JS, Python, etc. (I've written AWS code in those too, but it's not my preference). One annoyance is that they can take a while to 'spin up', compared to "slow" languages like Python. Ideally Lambda would support (with native SDKs) some well-typed languages which don't rely on runtime behemoths like the JVM (e.g. Rust, Haskell, StandardML, etc.)

I try to use the AWS 'resource API' rather than 'service API', since it's usually easier to understand. The latter can do anything, but deals with fiddly 'Request' and 'Response' values; the former isn't as expansive, but provides high-level things like 'Tables', 'Buckets', etc.

I wrap all calls to AWS in a failure mechanism, and check for nulls immediately. I usually use Scala's `Try[T]` type, which is essentially `Either[Exception, T]`. Note that there are some cases where null is expected, like an empty DynamoDB.get result. Those should be turned into `Try[Option[T]]` values immediately.

I'll aggressively simplify the interface that an application depends on. For example, a Lambda might be completely independent of DynamoDB except for a single function like:

    put: Row => Try[Unit]
Even things which are more complicated, like range queries with conditional bells and whistles, etc. can be hidden behind reasonably simples interfaces. In particular, the application logic should not instantiate AWS clients, parse results, etc. That should all be handled separately.

I'll usually wrap these simple type signatures in an interface, with an AWS-backed implementation and a stub implementation for testing (usually little more than a HashMap). These stubs can usually be put in a shared library and re-used across projects.

My current approach to dependency injection is to dynamically bind the overridable part (e.g. using a scala.util.DynamicVariable). This can be hidden behind a nicer API. The "real" AWS-backed version is bound by default; usually wrapped in a `Try` or `Future`, to prevent failures from affecting anything else.

All business/application logic is written against this nice, simple API. Tests can re-bind whatever they need, e.g. swapping out the UserTable with a HashMap stub.

I tend to use property-based tests, like ScalaCheck, since they're good at finding edge-cases, and don't require us to invent test data by hand.

For each project I'll usually write a "healthcheck" lambda. This returns various information about the system, e.g. counting database rows, last-accessed times, etc. as well as performing integration tests (e.g. test queries) to check that the AWS-backed implementations work, that we can connect to all the needed systems, etc.


Yes.


don't use AWS lambdas, problem solved.


I developed on Lambda with SAM and testing was pretty dang quick. IMO, Lambda works great for light weight event processing. I don’t know your use case. I wrote some Java based Lambdas, and despite not being a huge Java fan, the Lambda experience was fairly easy for me. I was essentially processing events coming in from an SQS queue, performing some enrichment, and dropping things in other queues for downstream services to consume. Are you sure Lambda is the right solution for your problem?


Your comment basically says "I did it and it's great" but I don't see actionable advice for OP.


Op asked:

> Anyone have any suggestions or share my frustrations?

I shared my experience and asked them a question. I at least contributed to the discussion.

Your comment basically adds nothing to the discussion and is essentially cyber bullying.


Most of the top comments are saying how Lambda is half baked. It might very well be, but it’s funny to see the one comment so far that is positive about Lambda get downvoted and the poster singled out for not providing a clear solution to ops problem. The HN hive mind disapproves of your experience!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: