Hacker News new | past | comments | ask | show | jobs | submit login

Could you share your experience with FFMPEG+ Lambda ? I ran into trouble with this when dealing with large files, especially when some of the files were being pulled off non S3 sources. Also what EC2 cores were you using. ?



Sure: you likely don't want to be dealing with large files on Lambda. Why?

* The maximum timeout window for a function invocation is 300 seconds

* The maximum available temp disk space per instance is 500MB

* Memory is maxed at 1.5GB

In the function invocation time window you need to:

* retrieve the file (to memory or disk)

* transcode the file (outputting to memory or disk)

* upload the file (as the disk is not persistent)

This along with the following facts make it infeasible:

* transfers from S3 are fast, but non-s3 sources are likely to be much slower.

* assuming you mean large as in GB - you have nowhere to put the files (disk too small, memory too small).

* transfers to S3 are fast, but uploading the transcoded video to a non-s3 source will likely me much slower.

Hope that helps.


I don't know much about video transcoding, but if FFMPEG can utilize streams it's easy to work around the lambda size constraints.

You can process several GBs in the 5 minute window by piping your S3 download stream through your transformation steps then directly into an S3 upload stream. Nothing ever persists to disk, so your only worry if anything is managing your stream buffers so you don't run out of memory.

As long as any single step of your pipeline doesn't exceed the time limit, you can make really nifty pipelines for large file processing by using the S3 upload as "temp space" then an S3 event to automatically trigger the next step of your pipeline.


ffmpeg can utilize streams, in both input and output. The trouble comes from different codecs and containers, especially on output. Some formats aren't append-only—the prime example being MP4 + h.264—and so ffmpeg needs to be able to write to a seekable output device, ruling out streaming output in those cases.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: