
Show HN: Free hosted open-source alternative to Zapier/Airflow - newcrobuzon
https://cloud.titanoboa.io/index.html
======
sixhobbits
I've used Zapier a fair amount and I wrote an article about Airflow so I have
a fairly good understanding of that too.

I would never really consider them alternatives though? To me Zapier is a
low/no code tool that offers a bazillion integrations and Airflow is a
workflow orchestration tool.

So comparing to both of them confuses me and I guess choosing one would give
you a more nieche audience but one that you can connect better to as well.

~~~
lewisjoe
Very true. We really should stop overloading the term _Workflow_ in software.

As of now these are the broad categories that abuse the term workflow.

1\. State Machines

For example in Jira a bug/ticket moves through different states to reach a
final stage. This type of state-machines can be found in a lot of different
softwares - most CMS/bug-trackers/CRMs where the entity is different
(document/bug/lead).

The motive of these systems who call themselves as workflow engines - is to
provide a structure to an otherwise ad-hoc movement of entities so that a
lead/manager is able to ensure a process and collect statistics.

2\. Automations

For example "apple workflow" app or Zapier or Zoho Flow. These softwares
define a sequence of steps that are triggered when an event occurs in the
system.

The motive of these systems are to enable automation and integrations between
different software components with zero code (thus, no-code).

3\. Process Designers (Bad term but can't think of anything better at the
moment)

For example Airflow/Camunda. These systems are not necessarily low-code but
they mostly deal with arranging individual components of code such that a
process can be assembled as quickly as possible. These systems usually are
accompanied by a visual designer like what Zapier has, but the intentions are
mostly to ease out the process, than being a complete no-code tool to create
automations. However, their marketing tries to sell themselves as no-code
platform for business folks.

The motive is not yet very clear to me but from my initial intuition they can
be used to initiate some data-processing pipeline, I guess? If anybody can
throw more clarity, please leave a reply.

Now as you can see, much like how a "Process" can mean many things in many
different context, the term "Workflow" can mean a lot depending on the
context. Any software that calls itself the ultimate workflow solution is just
a lie. It's like calling something an "ultimate process engine" \- doesn't
make sense.

~~~
zacwest
I'd like you to listen to this file, which is your first 'headline' read by
`say`:
[https://www.dropbox.com/s/fanpqs8lv2d9nvl/say.m4a?dl=0](https://www.dropbox.com/s/fanpqs8lv2d9nvl/say.m4a?dl=0)
Now imagine yourself in the shoes of a vision-impaired person relying on a
technology like VoiceOver. Do you understand how unbelievably frustrating that
is?

~~~
lewisjoe
Thanks for pointing this out. Never knew this would break screen readers.
Sorry. I'm changing the font right now.

------
newcrobuzon
Hi HN peeps! I single-handedly built Titanoboa (a workflow automation for JVM)
and now I am experimenting with hosting dockerized Titanoboa instances - so I
built a free hosting service. If you want to play with Titanoboa in your
browser without installing anything, give it a try.

Feel free to check out Titanoboa on Github:
[https://github.com/mikub/titanoboa](https://github.com/mikub/titanoboa)

Also: This is an early beta so please do let me know if something breaks or
you spot a bug. Atm I have load-balancers set up only on West coast and in
Europe, so apologies to folks from down under & similar locations, let me know
if it's too laggy :)

~~~
cateye
Why do you describe it as "automation for JVM" instead of "automation that
runs on JVM"?

~~~
newcrobuzon
Good point, not a native speaker so I thought this would roughly mean the
same.

The main message I wanted to convey was that this runs on JVM and multiple JVM
languages (java and clojure) could be used.

------
newcrobuzon
Wow, this has blown up a bit, so I have added few more servers (now two
servers on the West coast and two in Europe), but ultimately I think there is
a limit cap of 100 Titanoboa instances in each geography in parallel.

So if we break that level please don't be mad if you don't get your instance
:)

Instead give me a star on github and come back later :)

Cheers Miro

~~~
newcrobuzon
Ok, I have just reached 100 concurrent containers in Europe, which is max at
this stage and I don't think I can change the aws limit on the fly (will check
though) so I might start killing the oldest instances.

Apologies for inconvenience folks, but I am sure people rarely play with each
instance for the whole 3 hours anyway at this stage.

Honestly did not expect this kind of "public beta" rollout, still surprised
how well the service holds so far under the load.

~~~
newcrobuzon
Approaching 100 containers cap in US, so if something does not work, dont be
mad :)

~~~
newcrobuzon
If anybody's curious about this "public beta hosting - HN bear hug"
performance load test:

I just (sequentially) restarted all 4 servers behind my 2 load balancers -
seems that by reaching 100 containers in each geography the provisioning
threads may have died after many (unsuccessful) retries, so need to restart
the thread pools.

That is one more error I need to handle (or increasing my limit in AWS).

~~~
erulabs
Heya - just a note that I build these kinds of PaaS platforms for a living -
if you need a hand building a super-massive cluster for lots of users, let me
know!

------
caymanjim
First impression: rename it now while the cost is low. The name is unwieldy.

~~~
everfree
Ti-ta-no-bo-a

5 syllables.

Agreed that this is unwieldy to say in normal speech.

~~~
newcrobuzon
I mostly considered naming to be a distraction from getting the real work
done.

I also am probably not good with names as my other alternative was Megalodon
(4 syllables) - I just wanted to have some megafauna name.

Happy to hear what you folks would suggest :)

~~~
afarviral
"Dinornis" or "Moa" in maori.

~~~
StavrosK
If you're going with the former, you might as well spell it "deinornis", it
looks cooler and is more correct.

------
PascalW
Nice work!

How does it compare to n8n? [0]

n8n is the closest OSS alternative to Zapier I've seen so far.

[0] [https://n8n.io/](https://n8n.io/)

~~~
lulzx
It's license (fair code) sucks.

~~~
newcrobuzon
That is a fair criticism, I am aware of the AGPL license shortcomings.

I just picked up the more restrictive license at the beginning - being a sole
funder and not working on this full time etc. I simply did not want somebody
(e.g. a big company with a big team) grabbing my code along the way and
running away with it.

Since now Titanoboa got to the shape I envisioned it to be in I am starting to
focus more on adoption, so yes I am definitely thinking about switching to
less restrictive license since it will probably help.

Also at the beginning I was not aware how badly AGPL is perceived (I always
thought if it was good for Mongo it could work for me, but I may have been
wrong).

~~~
cstuder
You misunderstood, it's not you who's being criticized.

n8n used to write "open source" on their webpage while publishing the product
with a proprietary license. A lot of discussions ensued.

[https://github.com/n8n-io/n8n/issues/40](https://github.com/n8n-io/n8n/issues/40)

~~~
newcrobuzon
Oh really? Wasn't aware of that. Thanks for the link!

------
jchook
@newcrobuzon you have some kind of comment string output at the top of your
html body tag. Looks like something you do not want exposed.

Congratulations on reaching public beta and for submitting to HN! Looks very
promising!

~~~
newcrobuzon
Thanks! If it is the recapcha string then it is just an old public string for
local testing.

But I would upvote you 100x or send you a medal or something if I could, this
is really super nice of you letting me know!

------
kevsim
Impressive feat of engineering for one person!

But is it really an alternative to Zapier? I mean, while Zapier is obviously
about automation, it always felt that their big selling point was the sheer
number of integrations they provide.

~~~
newcrobuzon
My aim is to make this easier to customize - i.e. develop custom steps on the
run and have an instant feedback loop as you develop or customize existing
integrations.

Target audience might be slightly different ultimately, but we'll see - I
would envision this to be more useful in an enterprise environment where you
pretty much always have to customize the integrations that were provided out
of the box.

------
slotrans
This looks very interesting. The visual component in particular seems very
well done.

I have a deep interest in DAG-structured ETL tooling and had a couple of
questions that the documentation didn't seem to address...

1\. Can I execute workflows without a server running? Something like... $ java
-jar titanoboa.jar MyWorkFlowName arg1 arg2 ... ...and then my workflow
executes, as a program, on my machine, until it's done and then exits? Or does
every workflow always execute within the context of a running server?

2\. Is there any notion of resuming a partially failed workflow? As a point of
comparison, Luigi structures its DAG concept using Tasks which create Targets,
and invoking a Task whose Target already exists is a no-op, so if you have a
big execution graph that gets 80% finished and then dies, you can easily
restart it. I find that many competing tools are missing this concept.

~~~
slotrans
Now that it's Saturday I've had time to play with this and I believe I've
found the answers to my questions...

1\. Based on the Clojure REPL example in the main README, I think the answer
is YES, though not exactly the way I had imagined. It seems what you would
need to do is write a top-level script (or java "main") that starts a
"system", starts "workers", runs your job using that system, then stops the
system and exits. A little clunky compared to how Luigi does it, but usable.

2\. Best as I can tell the answer is NO. Neither the documented API, nor the
implementation of the API in src/clj/titanoboa/handler.clj contain any hint of
an ability to operate on a job id, beyond retrieving the result of its
execution.

Additional commentary: 1\. Resuming failed jobs As implied by my question
above, the ability to resume a failed job is essential. One of the major
reasons to adopt DAG-structured code is parallel execution, and Titanoboa has
that. But the OTHER major reason is to allow partially-failed computation to
retry/resume without repeating already-completed work. In particular in the
ETL space, we often have job graphs composed of hundreds of nodes, with total
runtimes measured in hours. If my 100-node job graph fails due to an error in
node #78, preventing an additional 15 downstream nodes from running, I don't
want want to run all 100 nodes again after I fix the problem. I want to resume
executing my graph at #78, and expect only the 16 total affected nodes to
execute, since everything else ran correctly the first time (and presumably
persisted their outputs). Luigi gets this one right. Airflow sorta tries but
it's clunky and you can tell it's not a priority.

2\. Flow/Dependency direction When designing a workflow, either in the GUI or
as EDN, you tell Titanoboa what jobs are "next". This is intuitive because it
comports with our notion of execution flow through a graph of jobs, but it
gets things backwards. That is, when we write A->B->C, we are thinking that A
will execute, and then B, and then C (perhaps results will be passed from step
to step). It is often better though to describe this as A<-B<-C, which reads
as C depends on B, B depends on A, and A depends on nothing. Structuring our
thinking in this way focuses the mind on what inputs a node requires in order
to perform its effect or compute its output, rather than on what operations
should follow it in time. Luigi and Airflow both get this one right.

3\. Properties The way Titanoboa defines workflow-level "properties", into
which job-level properties are merged, and the way properties flow along the
path of execution, is very nice. A constant problem with Luigi is how to flow
values from one Task to the next without using an excessive number of
Parameters. I can't say for sure that Titanoboa's properties construct doesn't
have the same problems, without taking the time to actually use it to build a
large project, but on the surface it looks good.

4\. Logging I noticed that when a step's function returns a map, to be
integrated into "properties", that return value is not logged. The message in
the log is like "Step [my-cool-step-name] finshed with result []" which is
both unhelpful, and not even literally true, as it most certainly did have a
result! When a step returns a scalar value, it does get logged. I found this
inconsistency frustrating.

Also, the stdout/stderr of each step function apparently goes to /dev/null. I
find this odd as the placeholder function when you build a new workflow is
(println "Hello World!") but if you actually execute that you'll discover that
our classic greeting vanishes into the void. This is a major shortcoming. As a
point of comparison, one of the biggest value-adds of using Jenkins as a job
scheduler is how it automatically captures the output of anything you run,
saves it in a durable log file, AND lets you view it in real time. Job
orchestration systems that don't match that level of log-friendliness drive me
nuts.

5\. Versioning The built-in versioning system is great. Two thumbs up. I don't
know how it would work if I were writing my jobs in proper Clojure or Java
code in their own repo, but I kinda don't care because the value of storing
and versioning what I do in the UI is so great.

6\. UI -> data I love the way the interactive UI is just there to generate
EDN. In a way this mirrors how Jenkins' UI builds its job XML files, but you
have to go hunting for those and they're hard to read (because XML). Being
able to see what EDN is generated by your actions in the UI, _right there in
the UI_, is fantastic.

7\. UI issues The UI is great but it has quite a bit of low-hanging-fruit
improvements that could be made. \- the run job popup forces you to choose a
system every time, even if there is only one \- being able to draw arrows in
the visualization is cool, but I could not figure out how to delete them
there. Needs work. \- the UI doesn't lay out well on small screens (I'm on an
old 13" Air), I had to zoom to 80% just to be able to see the X to delete a
property. It would help if the Workflows panel on the left (the least
important UI element by far!) could be collapsed (edit: it can be collapsed,
but the collapse button is on the _other_ side of the screen which makes no
sense) \- the box that pops up after starting a job has nothing clickable in
it. I have to close it and go to the jobs tab \- the jobs tab doesn't refresh
when it loads, even if I just started a job, which needlessly adds clicks to
the main workflow \- the jobs tab has an "archived" sub-tab but no apparent
way to actually move a job to the archive

Overall, there's a lot of promise here, and it's amazing to me that you built
this by yourself. Still, it has a long way to go. I recommend spending some
time with Luigi, which I still think is the best general way of DAG-
structuring real world workflow code, and with Jenkins which remains far and
away the best UI-driven job orchestration system. It seems you're already
familiar with Airflow, but I would recommend you treat it mainly as an example
of what not to do.

~~~
newcrobuzon
Really appreciate your time looking into this and apologies, missed your
original post. You got the answeres right. Re 2 - it shouldn't be that hard to
add, since everything is just data. It is however partially covered by the
retry property on each step. Will read through your additional comments and
will respond tomorrow (busy day, plus it's midnight here in Europe)! Cheers
Miro

------
ForHackernews
How does this compare to Prefect? [https://medium.com/the-prefect-blog/why-
not-airflow-4cfa4232...](https://medium.com/the-prefect-blog/why-not-
airflow-4cfa423299c4)

~~~
newcrobuzon
To be honest Prefect just got on my radar last week, so I will need to look at
it more closely, I was not aware of them till then.

Obviously this is for JVM, plus I am not sure how Prefect addresses following
two points I mainly focus on:

\- make rapid prototyping of new steps possible during runtime, without any
need to restart/redeploy

\- focus on distributed processing, where in master-less Titanoboa cluster you
can have pretty much any number of nodes

~~~
ramon
Prefect is like an hour just to put up the kubernetes environment required for
it. The way you have it you can just do a simple docker run and start coding
this is the comparison and you're coding on the web no need to steup VSCode or
nothing it's fast.

------
zomglings
This looks really well done, newcrobuzon. Very impressed.

I especially love your web UI. It makes it very easy to start experimenting
with workflows without the overhead of having to set up a local development
environment.

I am curious how you store secrets (e.g. AWS access key id, secret access
key). There isn't a login wall and it's not clear that the values will be
protected from you, so I am loathe to put my credentials on there.

Great work overall. Wish you the best.

~~~
zomglings
After clicking logout, I had no way of logging back in. :)

~~~
newcrobuzon
Ooops, good catch, I will fix that :)

------
juliend2
Very interesting! What library did you use for the SVG powered diagram tool?

~~~
newcrobuzon
I am using D3. AFAIK nobody has used it for worfklow visualization so far:
[https://www.titanoboa.io/visualizing-
workflows.html](https://www.titanoboa.io/visualizing-workflows.html)

~~~
X6S1x6Okd1st
Luigi does some workflow visualization with d3

[https://luigi.readthedocs.io/en/stable/index.html](https://luigi.readthedocs.io/en/stable/index.html)

~~~
newcrobuzon
Thanks, wasn't aware of them!

~~~
X6S1x6Okd1st
It's a very code forward one, but I've certainly found it useful and easier to
understand than something like airflow.

------
j45
Being a JVM friendly tool, it would be great to unlock other JVM syntaxes to
allow scripting in any lanugage.. Jython for Python, Quercus for PHP, JRuby
for Ruby, etc.

~~~
newcrobuzon
yes, I am considering this and it should not be _that_ hard (also Kotlin,
Nashorn javascript...), but at this stage Titanoboa is just a one-man-show -
so patience, I might get there ;)

~~~
mdaniel
FWIW, I would have to check all the ones listed but if they're sane scripting
languages on the jvm then they'll implement the JSR-223 interface allowing one
to pick them at runtime without a huge amount of drama:
[https://jcp.org/aboutJava/communityprocess/final/jsr223/inde...](https://jcp.org/aboutJava/communityprocess/final/jsr223/index.html)

------
phonon
How does this compare with
[https://github.com/openintegrationhub/openintegrationhub](https://github.com/openintegrationhub/openintegrationhub)
?

------
foolinaround
is there some recipes or some common scenarios one can 'import' in to check it
out?

~~~
newcrobuzon
You can check out [https://github.com/mikub/titanoboa-
tasklets](https://github.com/mikub/titanoboa-tasklets) And also
[https://github.com/mikub/titanoboa-sample-
workflows](https://github.com/mikub/titanoboa-sample-workflows)

There are few blog posts I wrote about some sample workflows, e.g.:

[https://github.com/mikub/titanoboa/wiki/Getting-Started-
with...](https://github.com/mikub/titanoboa/wiki/Getting-Started-with-GUI)

[https://www.titanoboa.io/using-titanoboa-for-it-
automation.h...](https://www.titanoboa.io/using-titanoboa-for-it-
automation.html)

[https://www.titanoboa.io/using-titanoboa-for-cloud-
integrati...](https://www.titanoboa.io/using-titanoboa-for-cloud-
integration.html)

[https://www.titanoboa.io/java-repl.html](https://www.titanoboa.io/java-
repl.html)

------
WrtCdEvrydy
Honestly I really like this, I can't wait to use it, but my concerns are
around certain features being behind a paywall, especially HA and Clustering
on the self-hosted version.

I understand that the cloud hosted version should cost more for these features
but without a way to get pricing for the self-hosted HA and clustered version
it will be a tough sell for me to deploy it, use it and have to scrap it in
the future.

------
ramon
I liked the video demo really nice and I would say this should be called not
low-code or no-code but rather aPaaS which is Application Platform as a
Service.

So the title should be "Host your own aPaaS" instead.

------
itpragmatik
How does it compare with Cadence/Temporal?

