
Faster and more flexible pipelines with a Directed Acyclic Graph - caution
https://about.gitlab.com/blog/2020/05/12/directed-acyclic-graph/
======
colonwqbang
This is funny to me since essentially all build/test/etc systems in the past
40+ years have been based around a DAG model.

They are rediscovering the basic features of Make et al.

~~~
throwaway02318
The fact that everyone goes out of their way to reinvent this really goes to
show how awful the Make experience is.

~~~
gnulinux
I have a pretty awesome experience with make. If you use it for what it's
designed for i.e. building files from a graph of dependencies, it's pretty
handy. I use for auto-generated code, binary code as well as tasks that depend
on other tasks, such as logging into system A requires getting this key from
AWS etc. Most people have bash shortcuts for simple scripts to automate
things, I instead write Makefiles, since it seems closer to my mental model,
you want to do X but doing X depends on resources Y Z T, make figures out the
rest.

------
xyzzy_plugh
I didn't read too deeply, but I'm glad to see all these sorts of systems
slowly converge on a sane model.

The part that saddens me is that they all end up looking like a glorified
distributed make.

~~~
ptsneves
The problem of DAG is solved by make very well. So well that it is why people
insist on using it. In theory they would have made a good choice.

The issue is that "make" made sure to put enough traps so that people with
less knowledge of the pitfalls tend to fall through. How many times I have
seen recursive make calls without the $(MAKE) variable lead to funny races or
lost compilation flags? How many paralelizable targets running in single
thread because somebody did not understand some strange out of order
execution(due to wrong Dependency mapping)? How many programs which are only
partially built? How many projects were not actually built for huge periods of
time because somebody forgot to put PHONY target? How many Frankenstein build
systems exist which create Meta languages around make(kbuild) ? My latest
adventure was a dormant fork bomb for 4 years in a daily updated project in
the clean target! Whole development server farms being resource starved
because the clean target started being executed as part of the normal build.
How did the fork bomb happen? The usual recursive make.

Make is perfect actually,but we are not perfect in its usage.

------
epage
My problems with this feature:

\- You can't have a DAG of stages. Sometimes, especially with a monorepo which
this touts, the maintenance of the "needs" becomes burdensome and you want to
just block on a stage rather than explicitly named jobs

\- The visualization of this is subpar.

So far my favorite implementation of this feature is Azure Pipelines. No idea
if its coped into Github Actions yet or not (haven't switched over). I hate
that is the case because of my underlying caution about Microsoft after the
90s and early 2000s though supposedly they are better now.

~~~
jayd16
Can't you do something like:

    
    
        -Stage1 needs A, B, C
        -D needs Stage1

~~~
epage
Basically faking a stage by making a job?

That helps but the "definition" of a stage is still far away from the job
definition when you have enough of them and people won't know to update this
when they copy/paste a job definition for adding a new test.

Plus, visualization is still terrible :)

~~~
jayd16
I don't disagree. Just trying to help.

------
new_realist
So, just like Concourse, Drone and any other modern CI system. About time.

~~~
hardwaresofton
Yeah so I just want to point out that most other CI systems are harder to
understand, less documented, harder to bring your own hardware to bear on, and
do less for the general software community than GitLab does.

I've used CircleCI, Travis, Drone (I don't think I've used Concourse actually)
-- and I _by far_ prefer GitLab CI, it is more featureful _and_ more easy to
use.

------
deckar01
This feature is currently broken. It can let deploys run without builds/tests
passing.

[https://gitlab.com/gitlab-
org/gitlab/-/issues/31526](https://gitlab.com/gitlab-
org/gitlab/-/issues/31526)

~~~
lindsayolson
Hi! GitLab Community Advocate here: I noticed the issue you linked is closed.
I believe the conversation is continued here: [https://gitlab.com/gitlab-
org/gitlab/-/issues/213080](https://gitlab.com/gitlab-
org/gitlab/-/issues/213080)

Hope that is helpful. If so, feel free to add a comment so the team can see.

~~~
deckar01
Due to that issue being closed maintainers assumed it was fixed and went
around closing related issues without testing if the problems were fixed.

[https://gitlab.com/gitlab-
org/gitlab/issues/198570#note_2936...](https://gitlab.com/gitlab-
org/gitlab/issues/198570#note_293646250)

I have had a really negative experience reporting bugs to GitLab in the last
12 months. I have spent tons of time investigating bugs in new features only
to have the devs ignore it. It feels like they are sweeping things under the
rug so they can hit deadlines for officially announcing new features.

[https://gitlab.com/gitlab-
org/gitlab/-/issues/39534#note_283...](https://gitlab.com/gitlab-
org/gitlab/-/issues/39534#note_283794210)

~~~
furkanayhan
Hello Jared!

GitLab Senior Backend Engineer - Verify (CI) here - Thanks for reaching out!
I'll try to explain current situation of these issues.

1\. [https://gitlab.com/gitlab-
org/gitlab/-/issues/198570](https://gitlab.com/gitlab-
org/gitlab/-/issues/198570)

We know that this bug with DAG/needs is quite annoying. That's why we've been
actively discussing about this bug in other issue: [https://gitlab.com/gitlab-
org/gitlab/-/issues/213080](https://gitlab.com/gitlab-
org/gitlab/-/issues/213080)

Can you please take a look at the discussions there?

2\. [https://gitlab.com/gitlab-
org/gitlab/-/issues/39534](https://gitlab.com/gitlab-
org/gitlab/-/issues/39534)

Unfortunately we had no choice but to give an option to fix this behavior,
instead of fixing it. That's because, if we fixed this bug immediately, it
would probably break existing pipelines.

That's why we allowed users to fix this behavior themselves
([https://gitlab.com/gitlab-
org/gitlab/-/merge_requests/24605](https://gitlab.com/gitlab-
org/gitlab/-/merge_requests/24605)).

Hope this is helpful and please feel free to contact us if you have any
questions/feedbacks. Thanks!

~~~
deckar01
These two issues illustrate that GitLab has a quality control problem. If devs
aren't dog fooding these features, the community should be given more time to
beta test them before the APIs are declared stable.

------
qppo
I'm confused, a DAG is a model for the relationships between data. What makes
this a separate feature for the product? And it's pretty fundamental to the
problem you try to solve, usually, so what were they using before, a list?
Tree?

~~~
detaro
Which part of the explanation of what they mean in the article is unclear?

~~~
corndoge
They're sort of coopting the name of a datastructure and using it as if it
were a product name which is a bit weird

------
jayd16
Sounds good. My guess as to why this has been late coming is because git
monorepos aren't that popular. The seem to be growing in popularity as LFS and
submodules stabilize.

Anyone want to share their experience with git and monorepos these days?

