
Easily Create Complex Workflows with AWS Step Functions - vrudyi
https://agilevision.io/blog/serverless%20architecture/2017/02/12/easily-create-complex-workflows-with-aws-step-functions.html
======
forrestbrazeal
My team has been using Step Functions for a couple of months now to
orchestrate application deployments. It is way better than our old SWF-based
solution - I especially love how easy it is to implement dynamic backoff
between states.

Biggest gripe: updating a state machine is still not supported (you can manage
state machines through CloudFormation now, but I believe it updates with
replacement, meaning the state machine gets blown away and replaced by a new
one with the same name.) We've had to roll our own versioning to get around
this.

If you'd like to know more about how we are using Step Functions, as well as
some specific benchmark comparisons against our old SWF architecture, feel
free to check out this post:
[https://forrestbrazeal.com/2016/12/29/serverless-
workflows-o...](https://forrestbrazeal.com/2016/12/29/serverless-workflows-on-
aws-my-journey-from-swf-to-step-functions/)

~~~
PKop
Step functions has pre-defined parallel states, but have you come up with a
solution for dynamic parallel states?

Such as: a parent state's output will determine the size of the input to pass
on to multiple copies of identical parallel state children (the number of
which is determined by the parent's output). These get the same input or a
specific slice of the same input from their parent.

I guess these could be created dynamically within the parent, then deleted
afterwards. Not as clean as declaring this in the states markup though.

~~~
forrestbrazeal
Right now, we have to coordinate our own batch processing, since dynamic
parallel states aren't supported, as you note. I've actually asked the Step
Functions team about this feature, and I know I'm not the only person to do
so, so hopefully it's on their roadmap :)

------
casperc
I am actually starting to look into creating a batch workflow, which would be
very suited for this (or SWF). However, I would like to know what the
alternatives are that are non-amazon specific. Are there any open source
projects that cover this space of defining workflows?

------
cle
Step Functions is a massive improvement over SWF. As the author said, there
are major differences and areas for improvement. Lifecycle management of state
machines is pretty poor, especially considering that deleting state machines
is hit-or-miss. You'll need to version your state machines yourself--which is
still less painful than SWF versioning. Also, because there are no "domains"
as in SWF, development can be more of a pain. The last SWF feature that would
let me drop SWF completely would be signaling ("sleeping" a workflow until an
event occurs).

~~~
2_listerine_pls
I have been awaiting for the wait-for-signal feature too.

~~~
nyxie
I've worked around this with Lambda tasks that write a result to a presigned
S3 URL and a wait loop that polls it (like CloudFormation custom resources do)
but it would be nice if this was built-in.

~~~
2_listerine_pls
[http://i.imgur.com/A5QnFkO.png](http://i.imgur.com/A5QnFkO.png)

~~~
nyxie
Sorry, that's what I meant.

~~~
2_listerine_pls
Have you tried to handle it as an error and retry?

    
    
      Retry": [ {
        "ErrorEquals": ["S3resourceNotFound"],
        "IntervalSeconds": 1,
        "MaxAttempts": 2,
        "BackoffRate": 2.0
      },
    
      "ErrorEquals": ["OtherErrorName"],
        "IntervalSeconds": 1,
        "MaxAttempts": 2,
        "BackoffRate": 2.0
      }]
    

Simpler.

~~~
nyxie
I couldn't figure out how to return a named error from a python2.7 Lambda
function; as far as I could tell it would always return UnhandledError. How'd
you do it?

~~~
forrestbrazeal
I don't think you can set custom error names from a Lambda function:

"Important Custom Error Names, like those shown above ("ErrorA", "ErrorB",
etc.) can be generated by an Activity, but not Lambda Functions. (See
"FunctionError" field in the Lambda Response Syntax for more information"[0]

[0] [https://docs.aws.amazon.com/step-
functions/latest/dg/concept...](https://docs.aws.amazon.com/step-
functions/latest/dg/concepts-error-handling.html)

------
djhworld
The blog post talks about triggering the workflow off an upload to S3, but
neglects to discuss the critical step around the fact that you need a Lambda
function to trigger the step function state machine (if you have a look at the
github it's there)

I wish step functions just supported the triggers directly tbh, i.e. have the
ability to invoke a state machine on an S3 put, passing the event information
to the first step in the machine.

~~~
forrestbrazeal
Step Functions does integrate directly with API Gateway now. [0] I'm not sure
if that helps your specific use case, though.

[0] [https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-
ap...](https://aws.amazon.com/about-aws/whats-new/2017/02/amazon-api-gateway-
integration-with-aws-step-functions/)

------
striglia
Very much looking forward to an upcoming project with Step Functions at
$DAYJOB. I suspect there are many use cases for turning complicated
intersections of batch jobs and blobs of code into discrete, independent tasks
like SFN requires.

Particularly interested in composing higher and lower level state machines for
better separation of concerns, but haven't yet come up with a compelling use
case :)

