
Show HN: Serverless logging to S3 - ellimilial
https://github.com/ellimilial/sqs-s3-logger
======
apexkid
Why not go for App -> AWS Firehose -> S3 architecture. Firehose will
automatically take care of batching logs and putting them together. Also it
can create partitions. No need to use Lambda at all.

~~~
guitarbill
Possibly because the app is running on Lambda? Cloudwatch logging is not great
for Lambdas.

~~~
szalonypiekarz
In that case, why not Lambda -> Cloudwatch Logs -> Firehose -> S3 ?

No bespoke logging solution is needed.

~~~
joemaller1
Don’t assume the author made an informed but poor choice, they likely made
something work with the tools they know or found. It’s really difficult to
keep up with all the various AWS offerings if your not part of a team that’s
immersed in it. Constructively suggesting alternates is welcome, I’ve got some
new tools to read up on this morning.

------
CSDude
I am not trying to be rude but SQS is not meant for this kind of usage,
Firehose to S3 is much more useful.

~~~
ellimilial
Sure, Kinesis is a clear candidate for a larger scale scenario, hence all the
disclaimers.

This is aimed at small-scale operations, utilising hopefully simple
components.

~~~
leevlad
Kinesis works well for low volume, and isn't any more or less complicated than
SQS. You should give it a try, it will certainly make some things simpler

~~~
jamiesonbecker
Even though it'd work, kinesis would probably not fit this use case very well
(sporadic/low volume logging), since you have to pay every hour, whether
you're using it (or barely using it) or not.

Firehose would work, though... no per-hour charges.

------
mooreds
Cool idea. I love that you're very clear where this makes sense and where it
doesn't. For higher volume logs files I'd definitely consider something like
kinesis agent: [https://github.com/awslabs/amazon-kinesis-
agent](https://github.com/awslabs/amazon-kinesis-agent)

But for lower volume logging this seems like a cost effective solution, and
then you can use Athena or EMR to further process or analyze the logs. I
didn't see any partitioning of the logs by date--that might be a nice
enhancement (unless I missed it).

------
haydenkshaw
Would probably use CloudWatch Logs for this personally

~~~
alex_duf
Yes if the goal is to archive stuff, event per event I'd use cloudwatch logs.
You can create the stream you want in order to log to it.

It's probably cheaper but lambdas and s3 are equally dirt cheap so hey why
not.

Edit: Two possibilities here. Either the choice of technology is deliberate
and there's a reason I can't quite figure out, or AWS has reached that state
where they do so much it's really really hard for people to know what service
to use for each problem.

Which actually makes me wonder what am I doing right now that could be done
easily using a pre-baked AWS solution?

~~~
mooreds
I teach some AWS courses and that's something I always bring up: if you have a
system you are thinking of building it always makes sense to see if AWS has a
prebaked solution and evaluate it. The solution may not meet your needs but is
likely to be available at a price/performance ratio (especially considering
fully weighted operational costs) that will make it compelling if it does meet
the requirements.

------
cosenal
This should have the «Show HN» header

~~~
ellimilial
Done, thanks for the pointer.

