
Show HN: TaskBotJS – TypeScript and JavaScript background job processing - eropple
https://github.com/eropple/taskbotjs/
======
eropple
Whee! My first Show HN. I've only been here, like, seven years and all.

But, yeah, I've been tooling away on TaskBotJS in my metaphorical garage for a
while. I've found Kue and Bull really frustrating to use (no shade intended;
they feel like they made a lot more sense in a pre-ES6 universe?) and `node-
resque`, while scratching a little more of that itch, just made me miss the
comfortable guardrails of Ruby offline job queues. So I made one.

As my day job is devops and devops accessories, making it easy to run and
administer TaskBotJS was a big priority for me and I've tried to get the best
of both worlds: mostly turn-key to run in simpler cases but less prescriptive
and more pluggable at scale. TaskBotJS is pretty extensible and supports job
middleware and event listeners; I have a Sentry plugin coming in the near
future as an example (and also 'cause I like Sentry).

I also like to write documentation[0]. While it's not yet complete, what's
there is pretty exhaustive and I try to explain my thinking as well as just
"what it does"; capturing the whys is as important as the whats, to me, to
better foster conversation around the project and to make for more
constructive issues when systemic, design-focused concerns are raised.

It's running in at least two production environments (only one of which is
mine; I dogfood but I wouldn't have released it as a "you should check this
out" version if somebody else wasn't checking me) and is being rolled out to
at least one more in the near-ish future.

Thanks for taking a peek, and I'd love to hear any actionable feedback that
comes to mind.

[0] -
[https://github.com/eropple/taskbotjs/wiki](https://github.com/eropple/taskbotjs/wiki)

~~~
disiplus
sorry i don't get it. why would somebody use this over a rabbitmq.

~~~
ChrisCinelli
I think it is a legit question... I poked a little around and I did not find
an answer.

------
sequoia
After a few minutes I gave up on evaluating this tool. It looks interesting
but there's no "hello world" or even a simple use-case explanation in the
first few pages of docs. Couple that with the no-warning switching between JS
& TS in the code examples, and it is too tedious to try to figure out.

I have no objections to TS, but if this is a "JS" library for consumption in
JS, please write your example jobs in JS please! Don't make me try to
translate TS to JS in my head–I don't care how easy you might consider it to
be, I don't want to do it.

I like the idea and will revisit when docs improve. I recommend user testing
your docs: put an unfamiliar JS dev in front of your README, set a timer for
~7 minutes, and when the timer dings ask if they think they'll use your
library. If they're still confused after that time, your intro docs are not
clear enough. That may seem like a small amount of time but there are 10,000
JS libs out there, 7 minutes is far more time than I can devote to developing
an initial impression of each one.

~~~
eropple
Thanks for checking it out and for the feedback! Agreed on intro docs--that's
a big reason it's "0.9.0" instead of "1.0". ;) "1.0", to me, is "the API is
stable" (though the current API should be) and "it's documented to the best of
my ability for novice users." The current userbase is a little more willing to
dig, as it's mostly friends and colleagues, and that'll change as it moves
towards 1.0.

I'll look into a JS-based example job; I have never found the translation
particularly onerous, but horses for courses, I guess. I'm not switching
between JS and TS in the documentation, though, save for the definition of a
configuration file--it's pretty silly, IMO, to transpile a config file,
especially when you get 99% of the way there through something like VSCode's
IntelliSense. I didn't make clear enough the distinction, apparently, but I
didn't expect it to be a stumbling block. Lessons learned.

------
deedubaya
How about a language agnostic background processing framework with clients in
multiple languages, including Javascript?

[https://github.com/contribsys/faktory](https://github.com/contribsys/faktory)

~~~
eropple
I like Mike a lot and I think Sidekiq is _awesome_ \--not having it in Node
was largely why I wrote TaskBotJS. And I evaluated Faktory before writing it,
because hey, this has been a lot of work, buy instead of build when you can!

As it stands, and things could change in the future, but I think a dedicated
ecosystem-specific system is a better bet for the 90th, 95th, and probably
98th percentile of products and projects. I don't feel that centralizing
background processes makes a lot of sense in a service-oriented architecture
and that takes away a lot of Faktory's benefits of agnosticism. The embedded
use of RocksDB also makes me itchy; Redis--while by no means perfect--is, IMO,
more understood as a backing store, and it's separate from the runner itself.
(TaskBotJS is also likely to eventually grow a Postgres backend. I've also got
a good chunk of one written using SQS/DynamoDB, but that's further out for
sure.)

All that said: Mike is crazy smart and I wouldn't begrudge anybody who wanted
to use Faktory. When I write Ruby, I will continue to use Sidekiq. ;)

------
JohnDotAwesome
Very nice job! I'll definitely be trying this out in a personal project.

How's your experience been with TypeScript and oao? I've got a very large
monorepo (using yarn workspaces) of TypeScript modules and it's been sort of a
PITA. For instance, a typical yarn install will unnecessarily duplicate some
dependencies across packages causing tsc to complain about duplicate
identifiers. Running yarn --check-files fixes this, but it's still annoying.
Also, following symbols with yarn workspaces is sometimes annoying since
linked packages will use "main" and "types" fields in the package.json, thus
following a symbol takes you to the generated type definition. I have a
generated tsconfig.json that sets paths to their appropriate package paths to
fix that. Again annoying.

~~~
eropple
Thanks for the kind words. =) And, good question! I...probably use `oao`
wrong, to be honest, or at least not as fully as it could be; my main use of
it is to power a workspace-wide `yarn watch`. I have had a lot of problems
trying to use its more advanced feature set and, tbh, if you look in
`packaging` I am certain there are functions in there that `oao` totally does,
especially around version bumping.

100% agreed with regards to yarn workspaces and linked packages. Symbol
navigation is Not Great, and I also have to be careful using VSCode's quick
import feature because sometimes it will just go wild and import
"../../../../dist/client/foo" or whatever instead of using the package import.
Once done correctly in a given file it's fine, though.

But converting TaskBotJS to TypeScript was really easy, and it catches a lot
of issues, so I think the future's bright on the tooling front. I found
TypeScript unusable, like, six months ago. I wrote a couple React projects and
a fairly large React Native app[0] in ES6 (which I Do Not Recommend
Doing...ow) because getting TypeScript working was just way too much work.
Still kind of is with regards to React, to be honest; the web panel for
TaskBotJS is just a create-react-app app because of it. But--progress.

[0]: [https://bit.ly/buymyapp](https://bit.ly/buymyapp)

------
alexchamberlain
I don't get it. I'm primarily a Python developer, so this looks like Celery to
me. I've never understood why I'd want to tie myself to a particular library,
rather than encode messages to an agreed encoding, then write workers in the
technology most appropriate for the job?

------
ilaksh
Does it run the tasks in the same or a different process? Why is redis
necessary? Can it pick up where it left off if the server restarts? Why so
much boilerplate (still) for simple use cases?

------
bdcravens
I love seeing projects with a docker-compose.yml, so I can check it out
quickly. However, yours only spins up redis - wouldn't it make sense to spin
up TaskBotJS in a separate container? Yes, I know the proper answer is "PRs
accepted", but just curious about the rationale.

~~~
eropple
Hey, awesome feedback! The root docker-compose file is for development; it
stands up Redis and the example config files, unless overridden, look for a
Redis on the port specified in that file.

TaskBotJS does have an integration environment, though. Take a look in
`packaging`. When I do a release I use the functionality in there to spin up
`@taskbotjs/example` and the associated functionality. On that it does some
very basic smoke tests to make sure nothing's obviously broken, but it spins
up a set of producers, consumers, and the webapi/web panel.

If you run `NO_CLEANUP=1 ./packaging/build-and-integrate.bash`, it'll stand up
the integration environment and you can play around with an active
producer/consumer/web panel system.

I should definitely break that out and make it easier to play with right off
the bat. Thanks for the idea.

~~~
bdcravens
I'd definitely try to go with docker-compose - it's nice to be able to try
something out without having to install it in your dev environment. Perhaps
example/docker-compose.yml.

~~~
eropple
That's a really good point, and I can still use it during packaging. Will see
what I can do. Thanks.

------
ex3ndr
Trying to solve something like this always puzzled me: how to make job
publishing transactional? There are no simple way to do transaction across
redis and, say, postgres. If you wish to make thing reliable, you will have to
put job to an sql first and then pull (or subscribe) from database to a worker
queue.

Am i right or there are another option?

~~~
ryanworl
Yes, this is a major problem most people simply ignore and deal with in a one-
off manner when it fails. The only way to do this safely that is practical is
to create a jobs table which is written to in a transaction with the other
writes. Another process reads the table and writes the job data into the
queueing system.

For example, the process starts up, reads the highest ID it has written from
Redis, then starts reading the table for jobs with IDs higher than that. The
job itself is written to Redis in a transaction along with the ID from the
database. Ignoring the fact that Redis can lose data during failover, you’re
pretty safe and duplicates will be minimal. You can add some idempotency on
top if desired.

Attempting to write to both a queue system and a relational database at the
same time _will fail_ at some point and you will be left to pick up the pieces
of inconsistent state.

If the write to the queue is to send an email, maybe you don’t care care. If
it is to calculate a very important value that happens to be expensive to
compute and needs to be done outside the request... good luck.

~~~
eropple
I think calling it a "major problem" is overblown for the 90/95/98th
percentile use cases. It certainly exists, and given a long enough time
horizon it _will_ happen, but I have never worked on a system where datastore
non-transactionality has caused issues that need to be resolved with more
than, as you say, a one-off fix.

I tend to implement what you suggest with _absolutely critical_ jobs, but the
level of juggling it requires is a little much for, again, the 90/95/98th
percentile jobs. Few things are so critical that you can only do them once
_and_ don't have significant visibility into them that retry logic can't bail
you out--though, I hasten to note, I do explicitly advise the use of
idempotent job logic with this or any other task-running solution.

~~~
ryanworl
99.99% of the time at small scale it will work fine. That is completely
accurate. But random network hiccups are so common in today’s cloud
environments that, even at medium scale, a few minutes of partition between
your app server and Redis, while still allowing access to the database, will
result in a huge number of dropped jobs.

The fix is so tiny I don’t see why it would be an issue for a team to
implement. It uses one new database table and a long lived process for writing
to Redis, which could be just running a script on the same node as Redis
polling the database. The rest of the code can be the same outside of that.

~~~
eropple
Yeah, that's a fair criticism. For me, why I'm not comfortable with that
approach for everything is that--if I'm understanding you correctly--means
storing a lot of state with the job (state which can then go stale).

~~~
ryanworl
That would depend on the application’s needs, but isn’t a part of the approach
itself. It is only guaranteeing delivery of the job to a worker at some point
in the future, on average half the duration of the polling interval reading
from the database. You can still use the strategy of only encoding object IDs
in the job to avoid stale data.

If the worker can’t reach the database, the job will just fail as it would at
any other time and retry later.

This is changing an at-most once delivery of the job into the queue into at-
least once. Combine with idempotency you can get “exactly once semantics.”

------
andreygrehov
Not criticizing, but just curious what was the reasoning behind picking Redis
rather than something like sqlite, which would keep the application a little
bit more self-contained so to speak?

~~~
aclave1
I am in no way affiliated with this project, but my assumption is that since
this uses workers(separate servers) to process the background jobs - they
could be on completely separate machines. If you use redis for job storage,
they can all communicate without having to talk back to the master server.

~~~
askmike
You can also do that with sqlite: have workers query for data and lock records
when they are working on it. (people have been implementing job queues in SQL
based environments for decades). The point of using sqlite is that it's not
something you need to install and manage separately, or something that acts as
a broker. (sqlite would be very slow as the workers need to poll for jobs).

~~~
eropple
Polling is not the worst thing in the world; when using reliable queueing and
BRPOPLPUSH TaskBotJS does poll, and Redis handles it okay. SQLite is more just
emphatically Not A Good Network Database, and the developers themselves say
so.

------
anothergoogler
Not a fan of this sort of "open core" offering, and it's heavy on the
marketing. Recurring jobs are an "Enterprise" feature? Hard pass. That's a
core feature of Celery and readily available for Resqueue, latter of which
seems to be your project inspiration? Pro/Enterprise should be tool offerings
that depend on your hosting and support. Arbitrary feature gating isn't
encouraging for a project like this.

~~~
eropple
Celery requires the system administrator to ensure that there's only a single
scheduler in the cluster at a time; the documentation specifically calls that
out. This additional administrative load is not a requirement of TaskBotJS.
Because of that recurring jobs requires underlying, distributed-environment
infrastructural features I'm not prepared to open source right now. Past that?
TaskBotJS is LGPL. If you'd like to write and maintain your own cron-type
scheduler, I would be happy to link to it as a plugin, and you can own the
difficulty of maintaining it when users come yelling. ;)

That last bit is important, IMO. The best way to keep software maintained is
to create a financial incentive for its maintenance. The pro/enterprise
feature set is borne out of actual use, and to signal the very strong intent
to maintain this as unsurprising software that will be there and be supported
for the long haul.

~~~
anothergoogler
> Because of that recurring jobs requires underlying, distributed-environment
> infrastructural features I'm not prepared to open source right now.

I don't see how that follows. Dedicating a single node is trivial. If you're
talking about redundancy, nobody's going to steal your leader election code.
That's why the feature gating feels arbitrary.

~~~
eropple
Sorry, maybe I was unclear. I don't want to have to support it in the open-
source version ("I tried to run it on three DigitalOcean droplets and I didn't
read the instructions and it's all sad" is just a time sink) and I do want
feature incentives, in addition to support guarantees, to encourage corporate
users to support the project.

I want to align the financial incentives to maintain this software; features
are part of how that can be done. At commercial scales, companies do need
support. But support is hard to quantify by itself and feature gates provide
additional arguments for buying the software and helping to ensure that the
project survives and is maintained.

------
anothergoogler
Off topic, is it normal now to include a "Code of Conduct" in new, single-
contributor projects? I thought it was more of a reactive thing.

~~~
eropple
I don't know what's "normal", but I believe very strongly in specifying
acceptable behaviors from the jump when trying to build a community.

Thanks for taking a look.

~~~
alangpierce
@eropple Sort of a nitpick, but you might want to replace "[INSERT EMAIL
ADDRESS]" with your actual email address. :-)

[https://github.com/eropple/taskbotjs/blob/master/CODE-OF-
CON...](https://github.com/eropple/taskbotjs/blob/master/CODE-OF-CONDUCT.md)

~~~
eropple
I just hit my forehead hard enough to wake the dog.

Thanks. :)

------
waibelp
Great logo!

------
mjuszczak
This is great!

------
vanwalj
How can "The best TypeScript and JavaScript background job processing on the
planet" only have 8 stars on GH ? :o

~~~
realPubkey
How can you measure a project by the amount of github stars?

~~~
optimuspaul
I think given the size of the potential user base for such a thing that a low
number like 8 would be a contrary indicator of the claims being made.

But in general I'd agree that stars can be a deceptive metric.

~~~
eropple
I mean...there are 102 as of this writing.

The project has not, I stress, changed in the interim. ;)

