
Tools for converting Python code to AWS Step Function JSON - ocfnash
https://github.com/bennorth/pyawssfn
======
striglia
Great to see open source tooling around Step Functions. Personally I'm still
holding out for someone taking the States language spec ([https://states-
language.net/spec.html](https://states-language.net/spec.html)) and turning
out higher-level tooling for writing/inspecting/manipulating state machines.

Agree w/ bpicolo (hi Ben!) that the tooling still can come a long way for
those who can't use Lambdas directly. This code is interesting, but it misses
my major use case for Step Functions -- running a workflow across several
services transparently.

If I had to guess the future, I suspect that tools like Step Functions will
become more and more crucial as we realize that in a FaaS world the job of
coordination is complex and essential.

Interesting video from Tim Bray showing off the variety of Step Function uses
in the wild
[https://www.youtube.com/watch?v=sMaqd5J69Ns](https://www.youtube.com/watch?v=sMaqd5J69Ns).
Lots of Lambda, but not solely!

------
ak217
Step Functions have great potential for orchestrating complex executions of
Lambdas or even fleets of them. My co-workers and I designed a threadpool
pattern with step functions that we use for orchestrating complex long-running
tasks:
[https://github.com/kislyuk/domovoi/blob/master/domovoi/examp...](https://github.com/kislyuk/domovoi/blob/master/domovoi/examples/state_machine_threadpool_app.py)

However, there are currently some big issues with step functions: compared to
Lambdas, they are not very scalable, not very economical, and have low I/O
size limits (because all I/O gets interpreted by the executor, so they limit
the state size to 32KB). Running Lambdas, we're used to be able to burst to
many thousands of concurrent executions. The Step Functions burst start limit
is 500, with just 25 per second after that (and 400 for state transitions).
This can be seriously limiting if you need to process more activities in your
system.

------
bpicolo
Step functions have a lot of good use cases - strong guarantees, concurrent,
distributed, good at recovery in all the right ways. It's a terrific queueing
layer abstraction for complex business processes. I suspect there are a lot of
businesses out there with adhoc state machine definitions in queue workers
that would benefit from formalization.

I had a closed-source proof of concept for a little framework for building
these in Python while at a former employer that worked out pretty neat. The
gist was arranging / building them more in pure code and automatically
generating/deploying lambda tasks and state machine.

    
    
        @task(name='foo')
        def some_lambda(input):
            ....
        
        state_machine = StateMachine(
          {...define_some_steps}
        )
    

then it was able to take advantage of Zappa to immutable deploy steps as
lambdas and build new state machine definitions with a little bit of python
magic. Was pretty nifty and not particularly tricky to make it work.

Amazon would do really well releasing a maintained open-source lib / docker
container that would be able to emulate state machines locally. The definition
language is pretty simple, but it's something that you definitely want to be
able to test end-to-end. I at one point did a little PoC for that bit too
([https://github.com/bpicolo/local-step-
functions](https://github.com/bpicolo/local-step-functions)).

The one big miss for me in step functions right now is that Tasks can't just
POST to an HTTP endpoint for request/response to take advantage of existing
infrastructure/http services easily. That'd be a real killer feature. They're
currently distributed in nature requiring lambdas or polling daemons, which is
powerful but also requires a lot of operational / configuration / deployment
overhead where a simpler model would suffice for many use cases - seems like
there's no reason you shouldn't be able to just deploy a monolith and get a
lot of use out of step functions in that regard. There are ways you could sort
of shim it with lambdas but the overhead is pretty high to do that (you'd need
to carry / edit routes in the task data which would ruin the declarative
nature of task definitions)

------
brylie
This seems similar to Apache Airflow:

[https://github.com/apache/incubator-
airflow/blob/master/READ...](https://github.com/apache/incubator-
airflow/blob/master/README.md)

