
What Is Dark? - stanzheng
https://medium.com/darklang/the-design-of-dark-59f5d38e52d2
======
Alex3917
I think this is fundamentally the correct solution to the problem, so I'm
excited to see how this develops.

What's the business model? E.g. open tooling, pay for
hosting/monitoring/optimization?

~~~
pbiggar
Thanks Alex!

We operate the infra, so we charge for it like aws, firebase, etc. We plan to
be pretty cheap, charge for usage, and have a generous free tier.

~~~
Alex3917
Will the code be theoretically runnable elsewhere though?

I guess my two concerns about solutions like this are always A) vendor lock-in
B) whether it's too big of a project for any one company and requires a
community of contributors to realize the vision.

In the short term it doesn't really matter because clearly being able to build
an app is a huge step up from not being able to build an app regardless (which
is why companies like Bubble.is are doing so well), but in the long term it
seems like there are 10 - 15 companies working on this so eventually there
will be more options and those factors will be key differentiators.

~~~
pbiggar
Our goal is making it super easy to build and run backends, and so if the
first step was "install kubernetes" I don't think we'd have realized our goal.
We don't have a goal of running these apps elsewhere. However, we don't like
the lock-in either, as it's a blocker to people trying Dark, so we're trying
to figure out ways to avoid that.

A current thought is perhaps allowing you to export your application compiled
to Node or similar if you decide to stop using Dark. (Data is already easy to
export, so we're mostly worried about code).

This is probably going to be the next blog post I write about this, because
it's pretty front and center to the business.

This is the first time I've heard the concern of the project being too big for
one company. I personally don't worry about it because it's so hard to move a
small team in the same direction that I don't think a large community of
volunteers will be a better fit.

~~~
Alex3917
Yeah I agree Dark should definitely run apps by default. And just to be clear
I'm guessing that currently vendor lock-in is probably mostly a concern for
folks who already know how to code, who wouldn't necessarily use it anyway if
that specific blocker were removed. E.g. since I'm already a full stack
developer, the things I'd be looking at are more like whether it would make
hiring easier rather than whether it would be better than hiring a dev shop to
build an MVP.

I guess another option would be having Dark function more as an app-store-like
intermediary between the hardware and software. E.g. telling users they need
X, Y, and Z infrastructure and letting them deploy it to whatever cloud they
wanted, and having dark just charge for orchestrating that and for providing
whatever other services.

In terms of a community, I guess I meant more in terms of getting developers
to create libraries for data processing, in terms of things like HTML parsers
for scraping webpages and other stuff that is essential for choosing a
language but where it's hard to see how it would be viable for one company to
build out all that stuff. Unless the plan is just to let folks run code in
other languages.

~~~
pbiggar
> mostly a concern for folks who already know how to code, who wouldn't
> necessarily use it anyway if that specific blocker were removed

Yes, I think that's right. There's a lot of folks who have strong feelings
about this but who aren't necessarily our target user. I know that's going to
cause a lot of problems but what can you do :-/

> I guess another option would be having Dark function more as an app-store-
> like intermediary between the hardware and software.

Yeah, this very much isn't what we're trying to build, or the business we want
to be in :)

> In terms of a community, I guess I meant more in terms of getting developers
> to create libraries for data processing

Gotcha! Yes indeed, there will be a package manager very much like
npm/cargo/rubygems, for just this sort of thing. We plan to make it extremely
easy to create and share packages.

~~~
Alex3917
> I know that's going to cause a lot of problems but what can you do

That's a good point, in the sense that even if less technical people don't
care about certain issues, they're often not going to adopt the product
without first soliciting advice from people who do care about those issues. So
that just means potentially needing to build out a lot of stuff knowing that
people won't actually use it for a long time.

Sort of like our product and the OAuth issue, which we're finally on track to
(mostly) solve with a Gmail add-on but it's obviously been several years at
this point.

------
shalabhc
Looks promising! Since this subsumes the database - what kind of transnational
data consistency is provided? How do upgrades to your data model work?

~~~
pbiggar
Good questions!

We haven't decided on transactional consistency yet - if there are particular
goals there, I'd love to hear them. Ostensibly, we want to allow users to
design the right distributed system for themselves, and provide ways of saying
"we want this DB to prioritize availability over consistency", for example.
But we haven't yet decided how to do that or what to support at launch, nor
where the bounds of a transaction will be.

We have a DB migration tool. The DB has a current type, and you specify the
new type, and add rollforward and rollback functions. That creates a new name
for the datastore: any use of the old name uses the old type, and the new name
uses the new type (both on the same datastore, applying rollforward and
rollback as necessary for the interface you need). Once you've removed all the
references to the old name, you mark the migration as complete, and we'll
handle the underlying transformation of the data behind the scenes.

~~~
shalabhc
I think operation level consistency policies are more useful than DB level.
For instance, saying bounded stale reads are OK for some operation could be
useful for the high read scenarios and hopefully the system will materialize a
cache.

~~~
pbiggar
Gotcha. Yeah, we're planning on allowing users to tweak their performance by
providing this sort of constraint to the system, and allowing the system
optimize it. We are far from that, but that's where we're aiming for.

------
pbiggar
Author here - would love to answer any questions people have!

~~~
lliamander
> In Dark, making an API call is as simple as a function call. Our tools for
> building connectors to 3rd party services handle rate limiting,
> authentication, error handling, and retries, with great default behaviour.
> Visibility around costs and failures are built into the editor, as is
> support for secret keys and Personally Identifying Information.

The history of RPC is...checkered. It's possible that every attempt to make
remote process calls as simple as local function calls has resulted in
terrible abstractions that leak like a sieve.

My suggestion (for what it's worth) is to bake the notion of _message passing_
into the language, and make it as simple as possible.

~~~
pbiggar
I'm talking about using 3rdparty APIs, like Twilio or Stripe or something,
that we don't have control over and that we need to handle. So we want it to
feel like calling functions, though in practice it needs to be able to handle
all the stuff that goes wrong.

Can you link to something about message passing? I'm familiar with a concept
with this name, but I don't quite see what you mean.

------
craftoman
Why you called it "Dark" and not "Light" or something more optimistic?

~~~
pbiggar
Who says Dark isn't optimistic? It makes me happy whenever I see it :)

