
Workflow software: I'm calling the bluff. - bootload
http://www.secretgeek.net/workflow_jerkflow.asp
======
wanorris
I personally build workflow software. My current employer sells vertical
systems to small government and nonprofits, and we built a dynamic workflow
engine (along with a dynamic ORM layer and a dynamic presentation layer). This
allows us to only hire programmers to build the underlying engine, and hire
only moderately technical people as project managers and deployment
specialists to actually customize the system for existing clients.

The programmers get to focus on building a pretty interesting code base, and
the project teams get to focus squarely on the customer. This turns out to be
a very effective division of labor. We're also in the process of rolling out
our tools to VARs and clients.

Summary of lessons learned so far: a workflow engine can be a big win, but you
really need to understand what you're going to use it for and how it's going
to integrate with the rest of your tools to define requirements for one. For
us, it slots right in at the business logic layer, and we just build UI on top
of it as needed. A standalone workflow engine may or may not do what you want.

~~~
gruseom
I think what you're describing may be in a different class than the products
the OP is castigating. The latter are not vertical systems. They're middleware
that the client expects to integrate with existing systems (paying huge
consulting bills in the process, though they may not know it yet). And once
that integration is done, then magic happens and the productivity boost is
supposed to kick in.

What you're describing sounds more like an in-house framework that is used to
boost _your_ company's efficiency in delivering systems to customers. That's
different. When you're doing a lot of similar projects, abstracting out the
common parts can be a big win. I've seen this succeed, up to a point.

I say "up to a point" because I've noticed a pattern. Sooner or later the
vendor decides that their in-house framework is so great that they've really
solved the problem of enterprise software development in general. Now they
"just" need to package it up as a platform.

The problem with this is that the value is not in the framework, it's in the
team that's built up around the framework. The framework may make _you_ more
efficient ("you" meaning the programmers, managers, analysts who know it,
identify with it, and belong to an organization based on it), but it doesn't
follow that it will make others more efficient, because the framework
abstracted away from _you_ is not the same thing at all. The overhead for
anybody else to use it is considerably greater and may (and usually does)
exceed the benefit.

Vendors do this because they want to shift away from being a consulting
company (where the product is you, and the framework is your secret weapon) to
being a product company (where the product is the framework and the "you" part
is vastly diminished, ideally to zero). Of course even in the success stories
(SAP?) the consulting role is still huge. And I can't shake the impression
that the successes are mainly in sales and marketing rather than technical
substance.

Anyway, these are just things I've observed. I don't claim they apply to your
situation.

~~~
wanorris
> The latter are not vertical systems. They're middleware that the client
> expects to integrate with existing systems (paying huge consulting bills in
> the process, though they may not know it yet).

Agreed. Tools to abstract workflow can be useful. My impression is that
whether middleware tools to abstract workflow can be useful depends largely on
how well they integrate with the overall toolset being used. Ours works
because it's fully embedded in the tools. Standalone solutions may or may not
work. I'm hearing good early returns from a friend using Microsoft WF in an
all-Microsoft shop, but (1) it's still too soon to tell whether it's really a
success, and (2) solving the all-MS solution is a much smaller problem than a
universal workflow engine or even a general-purpose Java workflow engine.

> Sooner or later the vendor decides that their in-house framework is so great
> that they've really solved the problem of enterprise software development in
> general. Now they "just" need to package it up as a platform.

Actually, we're exactly at the point of trying to figure out whether our tools
useful to VARs and client-level users, or whether it's really only useful to
us. And, yes, figuring out whether we can increase product revenues as a
percentage of our overall revenue mix.

> Of course even in the success stories (SAP?) the consulting role is still
> huge.

Unequivocally. However, if we succeed in working with VARs and technically
savvy clients, we would be offloading most of the consulting work to them by
building expertise outside our organization.

I won't pretend that I think it's a lock to succeed at a problem a lot of
people have failed at, but since we're not really risking our core business
model to try, it seems like a reasonable gamble as a way to grow the business
faster.

------
gruseom
There is a long and ignoble tradition in the software industry of vendors who
prey on the ignorance of managers and executives to sell expensive tools that
allegedly eliminate or reduce the need for programming. Executives fall for it
because they know that programming is hard and expensive but not _why_ it is.
The prospect of a tool that will eliminate their dependency on programmers is
very seductive. What programmers produce is usually late and over budget and
works like crap. They themselves tend to be weird, geeky, and erratic. Who
wouldn't want to be rid of them? Meanwhile, the vendors with the magic
programmer-replacement tool are wearing nice suits and know all about the
problem. They are full of stories about how great it would be if business
people - normal people - could "just" make the computer do what it should.
"Defining business processes" shouldn't have to require _programming_ , after
all. Normal people know about business processes.

This is a sales job made in heaven. At no point in the process - the _sales_
process - is it required for the tool to actually work. All they need is
enough of a demo to convince the stakeholders - who already want to believe -
that it will take away their pain.

At this point the well-chosen trivial visual example inevitably makes its
triumphant appearance. Who can look at the diagram the post cited
(<http://secretgeek.net/image/NXTCode_small.jpg>) and take anything associated
with it seriously? Yet people do. Important people.

In this dysfunctional dynamic, a new crop of tools that will work "this time"
makes a predictable appearance every 5-10 years. One reason this persists is
that it's in no one's interest to publicize the failures. When such tools are
purchased, rolled out, forced on organizations, and result in an even greater
clusterf--k than what was there before, who's going to say so out loud?
Obviously neither the vendors nor the executives. And only programmers listen
to programmers.

~~~
osipov
There is a long and ignoble tradition in the software industry of programmers
who prey on the ignorance of managers and executives to convince them in
uselessness of productivity enhancing technologies so that the programmers can
enjoy building their own productivity enhancing technologies that are
subjectively (and very rarely objectively) superior to alternatives.

At least we agree on the ignorance of managers and executives :)

It is difficult to pin down your argument, since it isn't clear if you are
talking in anecdotes about specific instances of product failures or if you
are describing the entire marketplace. Assuming your claim is about the
latter, you are missing the context of the general trends: yes, most of the
"new crop of tools" that come out every few years fail. However that is in
line with most new products -- over 90% of new software products that are
released every year fail, so it shouldn't be surprising that tool products
fail at a similar rate.

However, at what point are you throwing baby out with bathwater? Just because
most of the workflow products are crap, doesn't mean there aren't a few
innovative ones that truly improve productivity and deserve to be used.

Finally, you are confusing programming and writing workflows or designing
business processes. Very much like BNF grammars are an effective way of
describing parsers, flowcharts (or workflows) are an effective way of
describing invariant, sequential operations. Neither are an effective general
purpose programming language.

~~~
gruseom
_At least we agree on the ignorance of managers and executives :)_

Heh.

 _programmers who prey on the ignorance of managers and executives to convince
them in uselessness of productivity enhancing technologies so that the
programmers can enjoy building their own productivity enhancing technologies
that are subjectively (and very rarely objectively) superior to alternatives_

I'm not sure I agree with that. It's true that programmers often want to build
something instead of buy off-the-shelf solutions. But in my observation,
they're right as often as wrong. Then again, it's true many programmers prefer
their own intellectual gratification above all else.

You're quite right that I haven't surveyed all recent workflow systems and my
generalization shouldn't be applied to all of them. I'm talking about a
pattern I've observed and am highly skeptical of.

------
osipov
A very shallow argument. The author doesn't take into account realities of
most companies out in the marketplace. I'm exaggerating of course, but not
every company is like Google which can afford to hire Ph.Ds to write high
performance code for distributed systems. In a typical mid-market business
(500-5k employees), if the company can afford it, the IT dept. is staffed with
"Bob"s from accounting who started programming using scripts in Excel and then
moved on to basic Java. These aren't the people who know about lambda
calculus, Petri nets or even nuances of ORM. So for them, a workflow product
that works well in a distributed environment, e.g. supports a range of
databases, web service integration, long running processes, asynchronous
invocation is a huge time saver and productivity boost.

~~~
gruseom
You're assuming your conclusion. Obviously, a product that dramatically
simplified programming to the point where anyone (or "Bob") could do most of
what they need, would be valuable. The question is whether these products
really do that, or merely appear to.

~~~
osipov
>Obviously, a product that dramatically simplified programming to the point
where anyone (or "Bob") could do most of what they need, would be valuable.

You are attacking a strawman :) My claim pointed out specific types of
problems that are solved by workflow software, and solved in substance, not in
form.

~~~
gruseom
_My claim pointed out specific types of problems that are solved by workflow
software_

It did? I wonder how I read it ten times and missed that. "Distributed
systems" doesn't seem like a very "specific type of problem" to me.

~~~
osipov
>supports a range of databases, web service integration, long running
processes, asynchronous invocation

those are specific examples of problems that a workflow product would address
and consequently a programmer wouldn't need to spend time on building from
scratch.

~~~
gruseom
That's not a list of problems, it's a list of buzzwords.

Edit: nor is it specific. The class of programs encompassing those things is
enormous.

~~~
osipov
>>The class of programs encompassing those things is enormous.

Heh. Welcome to the real world.

If you'd like to call them buzzwords, I don't care, it is your prerogative.
They are what they are. People who have done workflow software know what they
mean. If you were to spend some time on Google you'd get a sense of the
meaning too.

------
cousin_it
A long-running web 2.0 startup for business process management: Thingamy
<http://www.thingamy.com/> by Sig Rinde <http://thingamy.typepad.com/> . I'm
not yet sure if this approach makes sense, the blog sounds intriguing but
vaporware-y.

------
mosburger
We experimented with a workflow engine at my last job (jBPM), and it was a
colossal waste of time. I have a colleague who uses it very effectively at
another job.

I think it's important to use workflow engines only in situations where it
makes a lot of sense. In our case, we were using it to control the order in
which network components were provisioned, and it was a lot more difficult
than just implementing it in code. In his case, they're modeling complex sign-
off business processes at financial institutions.

So I think it's most useful in modeling business processes in software in the
rare cases where you'd want to do such a thing, especially if those processes
can be dynamic (EDIT: and need to be validated by non-programmer types). But
it's not a suitable replacement for "if statements" when good old code will
do. We threw out flowcharts years ago for a reason.

~~~
osipov
It helps to think of a workflow as a domain specific language, then it becomes
natural to ask yourself "which part of the system that I am building should be
described using this language?". By "natural" I mean in the same way that if
you were to build a parser, you'd use BNF as a domain specific language to
describe it.

------
ShabbyDoo
There are two fallacies at play here:

1\. "Changing code is difficult. Our organization has all these 'change
management' processes in place. If we can just make the change in the workflow
engine, we can deploy it straight to production."

Perhaps a workflow engine keeps you from shooting yourself in the foot by
constraining kinds of things you can modify in your system. However, change
control processes also protect your organization from all sorts of things:
fraud, business rule errors, etc. I once worked for an organization where
project managers wanted to put business logic in the database because they
were allowed to change it at will! Just because changes don't fall within the
scope of your change control processes doesn't mean that those changes are
safe to do at will. This is the same fallacy that led people to believe that
SOAP was great because it could get through firewalls. The only reason admins
would punch large HTTP holes in their firewalls was because they believed the
risk of serving up web pages was relatively low, monitored, and contained.

2\. "Coding is done by expensive software developers. Business uses can make
these changes if we implement a workflow engine."

It's not syntax that keeps non-coders from becoming decent developers. It's
the ability to think in a structured, logical way. And, for anything but the
most trivial uses of a workflow engine, one must be able to think through
business processes in a way that only software development trains you for.
What are the holes in this process? What happens if an employee quits who has
active workflow items? How are they re-assigned? For years, business rule
vendors (iLog, Blaze, etc.) have been making similar claims. In practice, I
have seen that business rule system users are only allowed to make
configuration changes (15% discount on Tuesdays instead of 20%) rather than
more dynamic changes to rules (which are just declarative code).

With all this said, I still like the concept of jBPM and other workflow
engines. State machines are a powerful way of ensuring correctness, and I
would hate to throw the baby out with the bathwater.

------
nebogeo
Very different, but you can turn artists into programmers via visual
programming: <http://www.cycling74.com/products/maxmsp>
<http://www.apple.com/shake/> This seems to work well, but this maybe due to
the constrained nature of the problems.

------
mooneater
Confirmed. Most discussions I've encountered of applications "needing"
workflow software, could be easily served by simple state machine. I was very
confused for a while before I realized this.

