Hacker News new | past | comments | ask | show | jobs | submit login
Jenkins 2.0 Beta (jenkins.io)
415 points by sciurus on Mar 25, 2016 | hide | past | web | favorite | 154 comments



It's cool that they're promoting the "pipeline" plugin to a built-in feature, but the devil is in the details.

Under the hood, it's implemented by taking a script written in a "DSL" (it's actually Groovy code), transforming it into continuation-passing style, and serializing its state. This is pretty cool from a theoretical CS perspective, but having played with it a little bit, the implementation seems very fragile. There are a number of long-standing unfixed bugs that cause the transformed code to misbehave in ways that you wouldn't predict just from looking at it, even if you're experienced with Groovy. I ran into a couple of them just during a brief period of experimentation. For instance:

https://issues.jenkins-ci.org/browse/JENKINS-27893 (varargs are mishandled)

https://issues.jenkins-ci.org/browse/JENKINS-28277 (currying doesn't work)

https://issues.jenkins-ci.org/browse/JENKINS-26481 (calling a closure in a loop doesn't work right)

https://issues.jenkins-ci.org/browse/JENKINS-28183 (killing a pipeline job doesn't clean up its state properly)

And there are inherent limitations to the approach; for instance, you can't store any non-serializable values in local variables, which means simple things like foreach loops don't work (because you can't serialize an iterator).

I really like the idea behind the Pipelines feature, in principle. But I think building it around a general-purpose programming language, and then failing to support all of that language's features, is a recipe for headaches. If you never try to do anything more complicated than what's shown in the examples then you should be fine, but the borders between what works and what doesn't are very ill-defined.

EDIT: Oh yeah, and there's a surprising amount of functionality that isn't documented anywhere except for blog posts and JIRA tickets.


the broken for loop functionally wasted an entire day for me because I didn't know if the problem was me or groovy or Jenkins, being unfamiliar with groovy.


> Under the hood, it's implemented by taking a script written in a "DSL" (it's actually Groovy code)

It's Groovy code? The announcement page doesn't mention "Groovy" anywhere, let alone "Apache Groovy" which is what the backers of Groovy are meant to refer to it as since last November (2015) when it was promoted from the Apache incubator by the ASF.


Yep. You'd think this kind of thing would be considered significant enough to mention, but apparently not. They do mention it in the "tutorial", which AFAICT constitutes the entirety of the official documentation: https://github.com/jenkinsci/workflow-plugin/blob/master/TUT...

> A pipeline is a Groovy script that tells Jenkins what to do when your Pipeline is run.


DSL are certainly the future of this kind of tools but not with a dynamically typed language like Groovy. The fact that it's dynamically typed makes it pretty much impossible to get decent support from IDE's. Gradle's support in IDEA has been broken forever, and you know that even if the JetBrains guys can't pull it off, it can't be pulled off.

I think the future belongs to statically typed languages that can offer DSL's as terse as Groovy but with actually types in them, so that the IDE can actually do the job it's being paid for.


> I think the future belongs to statically typed languages that can offer DSL's as terse as Groovy but with actually types in them, so that the IDE can actually do the job it's being paid for.

Not as easy as it sounds. If the DSL is rich enough, and if the types are handled by the host language, type errors can become just as cryptic and unhelpful as with an untyped DSLs, and it's just as hard for the IDE to do something meaningful with them.


Sure, but at least, auto completion works automatically.

With dynamically typed languages, you get cryptic (and vague) errors and auto completion just can't work.


I use auto-completion in Python every day with PyDev. It's not as advanced as for other languages, but "just can't work" is not really true either.

http://www.pydev.org/manual_adv_complctx.html


That also depends on the namespaces used by the language. E.g. auto-completion in Clojure works a lot better than in JS, because every symbol has a unique resolution, known statically (or as unique as in many typed languages). True, you get less helpful suggestions, but OTOH, the error messages can be much clearer than in the typed case. The former can leave error reporting to the DSL, while the latter is restricted to the cryptic types of the host language's compiler. I'm not saying this can't be resolved with some clever compiler plugins, but if we're comparing options available now, the advantage is not so clear-cut (I do favor types, but only as long as they don't get too complex).


It's great to see that Jenkins is following the path blazed by GoCD[1] and Concourse[2] to make the pipeline concept more central.

That said, this appears to be achieved by promoting the plugin into the default installation.

It also misses some of additional the advantage Concourse holds over Jenkins and GoCD: build configuration is purely declarative and can be checked in with the project. You know what version of your pipeline built the software at any point in its history. And you have a reasonable shot at recreating that build, because every task in every job is run in a fresh container.

These are killer features, in my view. Jenkins can be extended with plugins to try to sorta-kinda do either or both, but it's not part of the central organising concept of how it works. Windows can run some POSIX apps, but it's not a nix.

Further out, the Jenkins pipelines are tricky to do fan-out/fan-in with; in Concourse it's trivial. You have to lay out your pipeline in Jenkins, Concourse will lay it out automatically based on declarative information about each job. Rather than a very rich API for plugins, Concourse boils the unit of extension down to "resources", which with three actions (check, get, put) can model points in time, git commits, S3 files, version numbers, interacting with a complex product repository and so on.

I used to tolerate CI/CD, as a necessary and worthy PITA. Now I find myself actively looking for regular tasks, resources, sanity checks and so on I can put into Concourse, so that I don't have to remember them or write them up in a wiki.

Disclaimer: I work for Pivotal, which sponsors Concourse development. But I wouldn't rave about it if I wasn't such a convert.

[1] https://www.go.cd/

[2] http://concourse.ci/


I've been a heavy Jenkins user for the last three years and I can completely see where you (and Concourse team) are coming from. The page comparing Jenkins and Concourse hits all my right buttons (complexity in build specification, minimal pipeline, etc) but I find that having to BOSH the hell out of a new system (pun intended) just to get CI running seems like a PITA to me (it's the one thing preventing me from recommending it as a solution atm). Maybe I'm missing something, but it seems like a lot of complexity is added to the deployment/worker management system that is pretty much a requirement if you want slaves external to the master. Is there a way to not have to use BOSH (BOSH Lite still tries to shoehorn things I don't want/need) to run Concourse builds? We use a combination of VMWare/KVM VMs, LXC and Docker containers for our builds and we have our own working deployment system (Puppet manages all our state successfully for VMs and LXC containers) which I would like to integrate with Concourse. I really hope I'm missing something because what I saw when I tried out Concourse for a week, made me swoon.


@vito from the Concourse team here.

We've recently started building standalone binaries which should lower the barrier to entry. Concourse itself has never been too tightly coupled to BOSH, it's just been the quickest feedback loop for us during development, so it ended up being the first thing we documented, primarily for internal use as we haven't really "launched" yet.

Binaries are available for download in the GitHub releases[1]. Check the README.md[2] for details. We'll be launching 1.0 very soon and part of this will include a major docs/website revamp which promotes the binaries to the "main stage". It also switches to BOSH 2.0, which drastically simplifies the deployment configuration necessary, but it still takes a backseat to the lower-upfront-cost distribution formats in the new site.

Glad you liked Concourse otherwise, and hopefully this helps. :)

[1]: https://github.com/concourse/concourse/releases [2]: https://github.com/concourse/bin/blob/master/README.md


Does this mean it will be (is?) possible to deploy Concourse on a single machine without the headache of BOSH Lite? I've wanted to use Concourse, but when all you've got is a Mac Mini, doing a full BOSH deploy (or even BOSH Lite) is quite a big ask.


Yup (is). You'd just run `concourse web` and then `concourse worker` on the same machine. If all you have is a Mac Mini there's one gotcha, though: currently none of the resources will run on OS X, as they're built as Docker images. So you'll still need at least one Linux worker somewhere.

I think the next step from us may be to start building Vagrant boxes that just spin up a worker, parameterized with credentials to register with a Concourse `web` instance. That way you can run Concourse on OS X for iOS testing/etc. and still have all your resources and Linux containerization when you need it via a tiny local VM.


OK, that makes sense. Anything that makes it easier to deploy would be awesome, particularly when dealing with iOS-related CI/CD.


Are there plans to support other container managers (other than Garden-based managers)?


The nature of Garden is to support container managers as Garden backends. Garden itself is just a client/server API spec.

For example, [Guardian](https://github.com/cloudfoundry-incubator/guardian-release) is in the works to replace the Linux backend with a thinner runC-based backend.

The main value we get from it is having a nice Go API and not having to overhaul everything using Garden every time some shiny new container tech comes out.


Yeah, version control of Jenkins itself has always scared me. There seems to be a pattern that we go through.

(in the beginning, there was light...)

* Create a small, tight, single-purpose Jenkins job

* Add a small tweak to it

(repeat adding tweaks)

(realize the Jenkins job now contains MANY different configurations options and the job itself is now a shell script in its own right)

* Sweep the "job" into a shell script. Check in said shell script

* Back up the Jenkins config, and hope no one asks why something's happened.

I now have a plugin that automatically checks in the Jenkins config to source control, but it again doesn't solve the problem of matching up a particular jenkins artifact to exactly what built it, and why.


We use Netflix's Job-DSL to keep Jenkins job configuration in source control (and to allow easier reuse than offered with job reuse plugins).

https://github.com/jenkinsci/job-dsl-plugin


I use http://docs.openstack.org/infra/jenkins-job-builder/, which is great as well: Jenkins configuration in a simple YAML file under source control.


Apache Bigtop relies on this, and does a pretty spiffy job of configuring itself out of the box this way.

Will look more closely...


We're using this as well, it's got warts but it's 100x better than authoring jobs in the web ui.


At my work we're running all Jenkins jobs in Docker containers using some simple scripting [1].

Works really great. Jobs can run on any slave, no snowflakes, full CI config is versioned in the repo along with the code. Jenkins job only point to a single script and that's it.

[1] https://github.com/kabisa/jenkins-docker



I think you would find Concourse to be a very appealing alternative; out of the box it gets you a lot closer to reproducibility than Jenkins does.


GoCD pipeline config is actually (sort of) declarative and stored in XML format in an internal git repo so it's versioned and you can recover/replay any version of it (in fact you can visualize the very first version of a pipeline you ran 3 years ago in the Value Stream Map visualisation and execute it again).

Agreed that the format isn't ideal but there are non-trivial problems in breaking it down due to the advanced way GoCD manages to tackle the diamond dependency problem (aka the Fan-In functionality).

Totally agree with you re pipeline as a first class citizen: I wrote this back in 2014 so it's kinda old now and surely Jenkins got better but I think it still applies by and large: https://highops.com/insights/continuous-delivery-pipelines-g...


Concourse avoids the fan-in/fan-out problems of GoCD's XML configuration by performing planning for you.

That is, you define the inputs and outputs of a job, then Concourse derives how to carry out your jobs in the correct order under the correct conditions.

A Concourse yaml file is pretty flat, as a consequence.

That's a great writeup. I think you'd find Concourse attractive.


I played with Concourse when it was initially released but it wasn't there for us yet and at the moment there is no reason to change but definitely need to test it again.

I couldn't find any docs on intra pipeline dependency management though, any links?

Thanks!


I'd have to ask around; I imagine the Cloud Foundry release engineering team have some tricks up their sleeves.

As for my own project, we have an increasing group of downstream consumers of our API. There are a few ways I could signal upgrades. They could just watch my repo and narrow it to the directories they're interested in. I can also publish a file on S3 or push into a git repo that they watch.


I have been using GoCD exclusively and have been quite satisfied. The pipeline as a first class citizen concept really resonates well.


Cloud Foundry used to be built on GoCD, and before that Jenkins. But it's close to 100% Concourse now.


> You know what version of your pipeline built the software at any point in its history.

Source control can tell you when changes to the pipeline were checked in, but they actually take effect when they're applied with 'fly set-pipeline'. That may be before or after they were checked in. Perhaps a sensible team would set up a pipeline to watch the pipeline repository for changes and apply them automatically. Mine hasn't.

> Concourse boils the unit of extension down to "resources"

Which you can only actually use by forking the Concourse BOSH release to add them to the workers [1]. I'm not sure i'd honestly call it a point of extension.

[1] https://github.com/concourse/concourse/blob/master/jobs/grou...



Oho! That's excellent. I see that came in 0.74; we're still on 0.70 i think, so i look forward even more to upgrading.


Yeah, a few of us griped at the Concourse team about this large escape hatch from historical reproducibility and they took us seriously.


Wait, what? Jenkins can do whatever pre-processing/post-processing you need by means of tasks. Regular Ant/Maven/Java tasks. And that was there, in production, from the time immemorial. People have been doing that for a decade now.

GoCD and Concourse add... what?


Or I could use Makefiles. Or just write a giant single bash script which only works on three workstations in the office.

GoCD and Concourse make the connections as important as the individual jobs. Concourse goes further and makes checked-in configuration and disposable containers the basis of the build environment.

This sounds like no big deal, but it's critical.

The point is not whether Jenkins can "do" these things. With the right stew of plugins, it can. The point is that Jenkins does not really think this way. It starts with the Single Ball of Build as the unit of work and retrofits other possibilities.


> build configuration is purely declarative and can be checked in with the project.

Seems like you are missing the point of pipeline and the Jenkinsfile concept, namely that the pipeline is part of the SCM.

Yes you can have the pipeline defined in the old-style text box in the job config, but that is intended to be used only while you develop the pipeline. Once you get it developed it gets checked in to source control.

Of course there are down sides to having the build instructions in source control, e.g. drive-by hack via Pull Request... granted this is nothing new... You can always do the drive by hack in a unit test... but it does take a little more work. In that regard having the Jenkinsfile in a text box (or better yet in a separate SCM) can be a useful protection (as can mandating that PRs use the target branch Jenkinsfile rather than the Jenkinsfile in the PR... again a feature in Jenkins pipelines)

For me the real advantage in pipeline is the organization folders support. You can tell Jenkins to scan all the repositories in your GitHub org and automatically create Jenkins jobs for any branches of any repositories that have a Jenkinsfile... PRs will automatically (subject to basic drive-by hack protection) be built and the commit will be tagged with the result.

So as with all things Jenkins, you have choices... we are providing some opinionated defaults (which is a change from the 1.x series)

> That said, this appears to be achieved by promoting the plugin into the default installation.

So the thing to remember is that the core of Jenkins is really better viewed as a platform. The plugins are really where functionality for Jenkins lives. I would expect to see more of the "current core" functionality get shipped out of core and into plugins. There is no reason why the Freestyle job type needs to remain in core. The advantage of having these functionalities outside of core is that we can be more reactive with regards to developing features.

> Further out, the Jenkins pipelines are tricky to do fan-out/fan-in with

I might have a different view on that claim, but hey I'm significantly biased.

OTOH my personal view is that for the 99% of jobs pipeline is overkill and literate is actually a better fit... but sadly most people don't seem to like the idea of actually making your source control have a README.md file with a - shock horror - "build" section that actually has the verbatim command required to build the software in source control (perhaps with an "environments" section that describes the build / test toolchains and environments)... I guess there's too many people who signed up to the mortgage-driven development manifesto [1] to actually want to leave a README file in source control explaining how to build and release the software!

Disclaimer: I created the weather report column and I am an active Jenkins developer

[1] https://refuctoring.wordpress.com/2011/01/13/the-mortgage-dr...


> Seems like you are missing the point of pipeline and the Jenkinsfile concept, namely that the pipeline is part of the SCM.

I guess I missed that Jenkins is heading that way. It's what Concourse does and I'm a fan of having CI/CD live in the repo.

> For me the real advantage in pipeline is the organization folders support. You can tell Jenkins to scan all the repositories in your GitHub org and automatically create Jenkins jobs for any branches of any repositories that have a Jenkinsfile... PRs will automatically (subject to basic drive-by hack protection) be built and the commit will be tagged with the result.

For PR-building on Concourse, the resource I'd recommend is: https://github.com/jtarchie/pullrequest-resource

> we are providing some opinionated defaults (which is a change from the 1.x series)

I see this more and more in the Java ecosystem and I think it's a good thing.

> I might have a different view on that claim, but hey I'm significantly biased.

Me too! :)

> OTOH my personal view is that for the 99% of jobs pipeline is overkill

I am starting to head in the other direction. We've historically fallen into creating "big ball of mud" build systems because it was just too hard to easily decompose and manage them as smaller units that could be rearranged quickly and safely.

Concourse makes it so trivial that the gradient for what is easy points in the other direction. It is less painful to lay out a pipeline (a graph, really) of builds that are composed of small pieces, than to have one gigantic Build To Rule Them All.

In Pivotal the practices around Concourse are evolving extremely quickly, because teams are discovering that it's really easy to delegate more and more to it. You start with a simple git->unit->feature->deploy pipeline, but soon you realise it's easy to assemble all sorts of things. The best is yet to come.


I've been tasked with implementing a large Jenkins deployment to support a ton of teams and I don't think I've hated a piece of software so much in such a long time. The past few years I've been using other CI systems like Circle and I totally forgot how much you have to fight Jenkins.

The UI is atrocious, job state is spread out among tons of crappy xml files, and the plugin system causes tons of headaches. If you're going to have a system that forces you to use the UI for the most part, rather than scripting up config files that I can load with some automation, at least make that UI nice to use.

Hopefully 2.0 is fixed up, but personally I'd never reach for Jenkins as a CI system if it wasn't part of a client's requirements.


You should definitely check out Jenkins Job Builder - http://docs.openstack.org/infra/jenkins-job-builder/. It lets you store job configs as nice yaml files which can be checked in to version control. It's what openstack uses to manage jobs for the many many openstack projects. I've been using for a year or so at work and highly recommend it.


I'm not in charge of writing the jobs, just standing it up and migrating a few dozen teams onto the system. Some are migrating from previous Jenkins systems, so the majority have old jobs in config.xml files. The Job Builder makes everything a little bit better, I've seen a few people using it already.

I'd still never use Jenkins on my own though, I'd just use Drone or pay CircleCI.


I wrote these posts recently about stateless Jenkins deployments:

https://zwischenzugs.wordpress.com/2016/01/24/ci-as-code-sta...

https://zwischenzugs.wordpress.com/2016/01/30/ci-as-code-par...

https://zwischenzugs.wordpress.com/2016/02/25/922/

Does anyone know if any effort has been put in to make Jenkins more 'programmable' in 2.0? I had a quick look at the bumph recently but couldn't see anything pertinent.


I agree that the lack of an easy, declarative deployment capability is one of Jenkins' biggest flaws and doesn't get near enough attention.


Thats mostly why I use hakyll, it's just a library that gives functions to build your own generator, so you can write out your own rules in haskell (Pretty simply for the most part) And then compile it to get your static website generator.


Very exciting -- a new major version of Jenkins has been a long time coming.

> "Jenkins 2.0 is a drop-in replacement of the Jenkins 1.x series of releases and fully backward compatible. There is practically no reason not to upgrade once 2.0 is released."

Skeptical, but optimistic about this claim.

I hope that the Jenkins configuration format is something that can be manipulated more easily in the new version.


I'll believe it when I see it -- the Jenkins ecosystem is one of the worst offenders I've worked in with regards to things getting arbitrarily broken by a point release. I shudder to think what a major release will do.


We look at the list of available plugin updates and cringe. You never know if one plugin update will break something else, and then it's a game of downgrade/upgrade until things mostly work.


I don't update or install plug-ins without doing a taking a snapshot of the Jenkins machine. A new plug-in version may force various other plug-ins to update, and thereby break the many plug-ins that have incomplete dependency specifications.

I also found that downgrade may not fix the problem, as the upgrade may change your configuration files to use newer features, causing Jenkins to choke on them once downgraded.

Jenkins is a spiderweb of plug-ins held together by shoestring. It's a very useful tool, but extremely fragile.


I hope Jenkins 2 will fix Jenkins 1's tendency for unclear and ambiguous user interfaces. Many configuration input fields have weird names, and their explanation fields can confuse me even more, and when using plugins, the whole configuration layout can quickly become a mess - Jenkins 1 often gives me the feeling of a mighty but hard-to-use Japanese pro tool lost in translation.


Jenkins by itself is eating too much memory on our buildservers, which can be used more wisely. Using compiled language could help that - e.g. Drone.io CI written in Go: https://drone.io/

You can find it's sources here: https://github.com/drone/drone

I just hope it would be able to work without docker, like jenkins.


Jenkins too is built on top of a compiled language. Too bad it is JVM, which is a hog when it comes to memory... Point being that a "compiled language" is no guarantee of a low memory usage.


OK, I meant "compiled into native code" language.


Still doesn't change much when said language can be JIT'ed.


But Go and Java are both compiled, garbage-collected languages.


RAM is so cheap, why not upgrade it on your build boxes.


It is clear that build stages should be first class citizens. We based GitLab CI on the awesome work that GoCD and Concourse did in this respect. Our DSL allows you to assign jobs to stages https://gitlab.com/gitlab-org/gitlab-ce/blob/43e49f52e30199c... and jenkins seems to have picked a similar syntax.

What is interesting is that Jenkins has a DSL but that this is not part of the repository. This means that it is hard to extend when you introduce new functionality and when you push an old branch it might not work anymore. I think that the model that Travis CI pioneered clearly wins.

I'm torn on the plugins. One one hand it is great to be able to plug so much in. But the plugins have access deep into the Jenkins internals, preventing a rewrite of core functionality. Our idea is: "Because GitLab is open source the enhancements can become part of the codebase instead of being external. This ensures the automated tests for all functionality are continually run, ensuring that plugins always work. It also ensures GitLab can continue to evolve with it’s plugins instead of being bound to a plugin API that is hard to change and that resists refactoring. This ensures we have many years of quality code and great monthly releases ahead of us." https://about.gitlab.com/direction/#vision


Almost all of what you said here is totally wrong.

>What is interesting is that Jenkins has a DSL but that this is not part of the repository.

Netflix job dsl plugin, Jenkinsfile pipleline, dotci all of these let you put a .travis.yml like file in your repo.

>But the plugins have access deep into the Jenkins internals, preventing a rewrite of core functionality.

Plugins don't have deep access to jenkins internals they use clearly defined "extension points" .

I know you are trying to plug gitlab but 'our stuff is better because we don't allow plugins' is not exactly a plus. Jenkins is so popular because it can customized via plugins to wide range of use cases.


Thanks for educating me!

1. Cool that there are different plugins that let you check the configuration in a dotfile.

2. I was not aware they used extension points instead of something more intrusive.

3. Obviously I like GitLab and try to promote it. But I post this too to see what others think and to learn. Turns out I had a lot to learn in this case. I still think that some items in Jenkins should be part of the core, such as the pipeline. But I agree that plugins are needed for some cases. For deploys we're thinking about plugins for GitLab CI too in https://gitlab.com/gitlab-org/gitlab-ce/issues/3286#note_402...


I might be an absolute DevOps Newbie, but getting GitLab CI to do a simple "copy to this ssh destination" buildscript seems very complicated.

From what I could gather it means that I have to create a specific runner whose login credentials are in its config, which aren't accessible via the WebUi, only as a file on the server where the runner is running and give it an unique tag. Then in my repofile i have to select that specific runner so it then "runs the build" on my deployment target.

Hopefully i'm completely wrong about this, but this seems horribly complicated and very inflexible for deployment purposes. Why would I specify my deployment target in my repo?

Maybe simple deployment jobs like I'd build them in Jenkins aren't in the scope for GitLab CI?


Thanks for the feedback. Deployment with GitLab CI is still harder than it should be and we're working on it in https://gitlab.com/gitlab-org/gitlab-ce/issues/3286

If you want to securely store login credentials consider using secure variables http://doc.gitlab.com/ce/ci/variables/README.html


Why so complicated? As a first step Please give me a Deployments Page that gives me the option to run simples SH Scripts in the Runner on click, parameterised please. All I need is a nice GUI for my QA people that allows them to deploy(which for our PHP site basically just means copy and DB Import) Branch x on Server y.

Jenkins is total Overkill for that but GitLab CI doesnt leave me with enough GUI to skip it either.


I agree that we should have a simple page that shows you an overview, our issue for this is https://gitlab.com/gitlab-org/gitlab-ce/issues/750910


I've been using Jenkins for years, and have been overwhelmingly happy. 2.0 is a big step for the project, congrats!

The flexibility Jenkins provides as build system is enormous. The plugin community is also a huge benefit -- chances are, if you have a need, there's a plugin for it.


I feel the same. Using Jenkins since 5 years now on a daily basis. Yet I also feel it may be coming too late. During the past 5 years our projects have grown to a size were we're expecting to have more than 1m DAU end of this year, and I don't think Jenkins will survive this journey with us. I'm speaking about scripted Docker container baking & deployments to e.g. ECS, ECR, asset uploads and versioning, basically release management. I'm looking for modern solutions for the latter which is basically as stateless and reliable as our product Docker infrastructure. I would dismiss Jenkins in case I find an equally flexible but safer tool.


Well, I don't know your specific circumstances, but I'll say one should strive to have as little logic in their build system as possible.

It essentially should be running shell scripts and/or language-specific build scripts/tools. All logic should be self-contained in your project(s) repos, so that it doesn't really matter what build system you use, and/or one could build the project without the build server if required.

With that said, I don't see any reason why release management, containerizing, versioning of assets, etc... could not all be done with Jenkins (or any build system).


> I'm looking for modern solutions for the latter which is basically as stateless and reliable as our product Docker infrastructure.

I'm a broken record in this thread: Concourse. Stateless, containerised builds. Declarative config checked into repos. Pipelines as the central concept.

All of these as the core paradigm, not from plugins.


Every random piece of software expects some kind of file in the root directory of a project - this is not acceptable and leads to messy project layouts.

Could developers of these tools please stop dictating where people have to put the "Blahfile" dsl config files and finally allow people to configure these kind of details.

Also I would like to propose a common standard directory "projectroot/config" for these kind of files, NOT the project root directory itself.

Thanks for your attention!


I've had this thought too, but I think the horses have well and truly bolted.

Concourse lets you put the file anywhere; the act of applying configuration is distinct from committing it.

Initially people put `pipeline.yml` or `concourse.yml` files in the top of their repo.

It's rapidly evolved so that typically you'll find either `ci/pipelines/{some,number,of,pipelines}.yml` in the main repo, or a separate repo for the CI/CD system. I prefer the former, most teams I've seen so far prefer the latter.


"projectroot/config" it's too common and already used in different ways that will produce conflicts. If something is established it should be done in a proper way through a spec like semver was established.


you could avoid conflicts by not naming your project files "Dockerfile" or "Jenkinsfile" or "travis.yml"...


I don't like the Dockerfile or Jenkinsfile either, but I can kind of understand the .travis.yml. The good thing about the "Dockerfile" is that with most of the docker commands that interact with it, you can use the '-f' option to specify the file. For example "docker build -f dockerfiles/application.dockerfile" -t myapplication .". I don't see any easy way to do that with the Jenkinsfile.

The amount of top-level files in these projects has started to become quite large, and it is unclear at first glance to see what it is important when exploring a new ecosystem.

I'd like to see these products would move in a direction where these were would do any/some of the following:

1) use a file extension rather than "important" describing filename (.docker for Dockerfiles and .groovy/.jenkins for Jenkinsfile)

2) moved into a product specific folder (travisci/ci.yml, github/ for the issue and pull request templates)

3) use yet another top level file for indirection that is agreed upon (my initial thought is something named integrations.yml)


You don't need to call it Jenkinsfile. You can call it anything you want and put it anywhere you want.



As the website got HN'd, a question: does Jenkins support now pass-through of parameters in pre/post build actions to other jobs?

Background: I have "deploy frontend" and "deploy backend" job, both of which need to invalidate some layers of caching, and a job parameter that specifies the environment (it's the docroot).

For now I have a metric shitload of shared code between the jobs as there is no way to move the cache clearing code to its own job and pass the environment parameter through in pre/post build actions.


I can't imagine what you couldn't accomplish using the Parameterized Trigger Plugin https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Tr...


Cool, thanks for that. I'll mail this one to our infrastructure team, but it's sad to see that this isn't installed by default.


how does it being an optional plugin change anything vs. Jenkins having every possible useful thing pre-installed resulting in a much more complicated setup screen? Plugin architecture is total win IMHO.


Parameters without the ability to pass them on is kind of pointless, imho the right solution is to provide a single "parametrization" plugin.


That's nice but even the current version of Jenkins is buggy as hell and it's been around for a while now. Every place I've been at takes the base Jenkins worker management and then layers routing and job orchestration on top of it through something else because Jenkins itself just can't deal with anything that is beyond triggering and canceling a job. Even canceling doesn't work half the time.

So given that's the foundation they're layering even more complicated stuff on top of it. Thanks but I'll pass.


Having been using jenkins several times since it was Hudson, as well as Bamboo, travis-ci, python buildbot, and TeamCity, Jenkins is a loser from my perspective. Not worth becoming invested in.

Its simply too difficult to use by a laymen developer or sysadmin when accommodating complex buildchains, which only grow more complex over time in most software businesses.

If you're going to program your CI with a web administrative interface, and provide an easy to navigate interface for users, expand with build metrics and external systems, I suggest TeamCity over any others.

Though jenkins is free, my experience is maintenance teams of 2 or more people with tribal knowledge develop around the tool that far outweigh the license cost of TeamCity, which solve many of these homegrown engineered problems as part of the product cost.

If you have a strong developer team, I would suggest using something like python buildbot and programming your CI end-end in a single unified language, much better than shoe-horning plugins and groovy code inside your jenkins textarea fields!

That said, all CI systems are just software, and any requirement can be accommodated with sufficient amount of additional software, forks, or changes. Any will do.


If you're putting lots of logic into ANY CI server you're doing it wrong!

Script your build so that you can run it with a single command on any build server or any developer machine. Then have the build server invoke said scripts (with arguments for any build-server specific paths etc).

Having spent a week fixing problems with inter-plugin dependencies I do agree with your point - Jenkins just isn't very good, like a lot of the free software from that generation (FitNess etc).


Absolutely, tasks/system commands should always be stored in VCS as a shell script, so that feature branches can exercise changes in the task itself. Unfortunately theres the other 90% to worry about, and thats where the highest complexity is, such as configuring a build chains, access control, reporting, etc.

Glad to see jenkins is moving towards a travis-ci ".travis.yml" kind of format, but I'm not hopeful to see it catch up to anywhere near TeamCity, even Bamboo in user or developer UX.


Could use something like this if you are using github https://github.com/groupon/DotCi Job configurations can be version controlled and reviewed like everything else.

For multijob pipelines checkout https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin


JJB is also a nice tool that seems like it allows similar functionality (I haven't worked with DotCi, so I can't say how they compare)

http://docs.openstack.org/infra/jenkins-job-builder/


Well...

Jenkins 2.0 will provide support for build chains via the pipeline plugin which will be included as a core plugin.

Access control can be configured quite nicely with the Role Strategy Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Role+Strategy+Pl... + AD/LDAP integration.

Reporting automation is covered by the pipeline plugin. Custom reporting is quite easy to implement on top of Jenkins, its APIs cover almost everything you need.


You have simple problems to suggest simple solutions. CI logic can gets complicated:

What about cascade builds? The pipeline can't be handled in your build script.

What about reporting? (list of compiler warnings in the build? List of tests that failed? In general extracting information from the log)

What about efficient email alert? (with blame list, list of changes, etc.)

What about IRC reporting?

Jenkins is not very good (and written in Java :() but the other available free alternatives are just not better.


Huh? I really don't understand your argument. Jenkins is better than scripts because you can do more in Jenkins than in scripts? O_O

You can do a heck of a lot more in scripts; you've got a full language with all of the libraries it comes with in order to do stuff.

We've build our current build pipeline with Python scripts:

0. Bootstrap the build environment (installs everything you need to build). 1. Build projects order, order is derived from the intermodule dependency graph. 2. Execute unit test sets based on build parameter + save results in output. 3. Execute integration tests based on build parameter + save results in output. 4. Build distribution packages + save in output. 5. Deploy distribution packages to server based on parameters. 6. Run UI tests against the deployed packages + save results in output. 7. Publish the bits if they're OK.

Most of these steps are in different scripts, which can be called either from orchestration scripts or one-for-one in Jenkins.

All we have Jenkins do is invoke the script(s) then process and report the results.

We've basically got it wrong with the design of our CI tools; they do too much. They should just provide scheduling, invocation, reporting and dashboards over the build process. Expose that as an API and let scripts integrate with it.


So, let's say you have 8 projects. In each of the 8 projects, you want to run the same static analysis tool. If the violations go up, you want an email sent out. If any critical errors are there, you want the build failed.

In any project, if the build fails, you want the latest commit automatically reverted.

Now, do you just copy and paste to each script, or do you have a chain of build processes and logic in the build system?


I was basically answering "If you're putting lots of logic into ANY CI server you're doing it wrong!".

There is a separation of concern: the CI server logic is a refactoring of things that are common to any project. If your project have scripts that sends email on build failure, I think you're doing something wrong.

You mention at some point "orchestration scripts", this is exactly what Jenkins is supposed to provide help with: orchestration.

The important thing IMO is that the boundaries of responsibilities are clearly defined between what a project does and what the orchestration does.


This is a critical component of releasing production software. I would not expect to see "free" in the conversation here.


There are good reasons to prefer free and open systems, even for "production" software. Those reasons are part of why people choose to use GNU/Linux, nginx, and all the rest of the open infrastructure.


It was not clear whether it was free as in beer or speech. I do understand there are benefits to using open software.

Too often I see engineers bias heavily toward inexpensive, free/cheap, or home-built solutions simply because they personally don't think things should cost money, when the understated anchoring costs of building and maintaining, plus the opportunity cost of valuable people time, might shift the toward other alternatives if a wider business context is taken into consideration.


Yeah, I parsed them as wanting a free system as in one that's open source, but I could be wrong.

If we're talking about hosted solutions, I agree it's mostly silly to refuse to pay... unless we're talking about open source projects, where many CI vendors do have a free tier...

Anyway, I also see an opposite bias toward costly hosted proprietary solutions and a refusal to even think about how one could accomplish one's goal with, say, a couple of shell scripts and cron jobs—or a self-hosted open source system.

That bias can be very sensible, but it can also be a hindrance, especially when it turns out that the proprietary solutions aren't quite what you want anyway, but since you're opposed to writing your own scripts, or running a more flexible system on your own servers, you settle for a suboptimal setup...

This bias I'm talking about is also a way of undervaluing something. It's saying "I'll pay $99/month for some hosted thing, but I'll never spend a week of engineer effort on really getting it right."


It's never the cost for me, it's the mutability, oos I can hack on, and make it do exactly what I want it to do, proprietary code I only ever seem to get somewhat close but never exactly there.


We made the same mistake with Anthill Pro a long time ago. The plugins were of various quality and caused us a lot of headache.

Now our build and test suite is just a script each and it can pretty much on on any CI server.


The biggest thing is that Jenkins is a job engine, not a CI server. Because it CI happens to be nothing more than executing jobs when triggered and then tracking/reporting the results Jenkins gets used for it as well as a number of other things.

If you're looking at Jenkins purely as a CI then absolutely, it's overkill. I've seen Jenkins used as a distributed cron replacement, CI server, pingdom replacement, deployer and even as a means to give non-programmers permission controlled access to execute certain jobs via parameterized builds. Just having granular control to decide how to report on cron failures from a central location is a huge help.

It's ability to combine job execution, detailed result tracking, scheduling, scripting, SSH and failure reporting from a central location with flexible permissions makes it an extremely useful tool.

The extensions that the Cloudbees folks have put together are pretty sharp too.


>If you have a strong developer team, I would suggest using something like python buildbot and programming your CI end-end in a single unified language, much better than shoe-horning plugins and groovy code inside your jenkins textarea fields!

This is obscenely bad advice. We run thousands of builds/day and none of the configuration is maintained inside jenkins. All of our job configurations resides in repository alongside all the code.


TeamCity is cute, but at scale its pricing is... kind of bad. $12k for 50 agents. And it's not that hard to have 50 Jenkins slaves, especially if you're testing on multiple platforms or on different versions for those platforms.


If you have 50 Jenkins slaves, you probably have someone who's job is almost exclusively to babysit Jenkins. At my two previous workplaces, we had entire release teams focused on that, and still Jenkins was a never ending source of problems. And they did everything "right", all the Jenkins scripts and configs were in git etc.

$12K/month for a much (I mean, it's like going from cvs to git) better solution in that situation should be seriously evaluated; perhaps you could give every team in that org their own dashboards and eliminate/distribute the whole release engeering org. $12K/year, which is the actual TeamCity pricing, is a no-brainer, which unfortunately most teams are way too cheap to pay for, headcount looks so much better. Fortunately at least cloud based CIs are starting to seriously take off, hopefully I never have to deal with a home-grown Jenkins deployment again.


Well, from my experience as the person who "babysits Jenkins" unless you appoint an actual owner for Jenkins, you'll get a real tragedy of the commons with it. Devops sounds cool and all but most devs want to write cool code: usually product features, not infrastructure stuff, so everyone will do CI/CD related work half-heartedly.

So in my obviously biased opinion, if your team is big enough, either get an actual build/release engineer/devops guy or appoint one of the devs as an owner for this, otherwise you won't get the best results.


Jenkins compiles jars with Apache Ant, Maven, and checks code out from all the relevant SCMs, which, if you don't need more than that, works perfectly.

Also, arbitrary shell scripts, (as usual) provide duct tape for everything else as needed.


Yes, software is harder to manage (long-term) when the primary interface is a GUI.

However, the new Jenkins 2.0 fixes many of the issues you've described. Build pipelines are a major new feature, as well as the integration of a "Jenkinsfile" which allows configuration of jobs via a proprietary DSL.


the integration of a "Jenkinsfile" which allows configuration of jobs via a proprietary DSL

This makes no sense to me. Only how my project is built should be checked into the project's repo, not the whole CI/CD deployment pipeline! If I start deploying to Azure instead of AWS I need to commit that change to my project? It's nonsense.


We use the pipeline plugin in my job and we use a seperate repo for pretty much the same reason. It's not a detail that's really part of the application (do you keep your chef cookbooks in the same repo?) and not only that but we use the same scripts across multiple apps.


The DSL can pull in other scripts from elsewhere, so you could just have a stub Jenkinsfile in your project, with the bulk of the pipeline config version-controlled in another repo.


And if I change where I keep my pipeline config, I have to commit a change to the project so it grabs it from the new place? Or pass it in as a parameter, so you need to know ahead of time what parameters the DSL will be expecting?

Come on now, we don't need to go to these contortions just to avoid saying that Jenkins is doing it wrong.


In this situation, yes.

You could indeed just use the Jenkinsfile for building a project. You can also specify the DSL config directly in a job, e.g. in a separate deployment job.

What would be the right place to define the deployment config?


In the job itself is fine... unless you want to version control it. This is a historical Jenkins weak-point. The plugins that exist to try to solve the problem are clumsy.

This Jenkinsfile is their solution to the problem. I'm saying it's not a very well thought-out solution, because now they're forcing us to put this thing that's not part of the project in the project, if we want it version-controlled.


So don't use the Jenkins file. I have mine in a separate repo.


Ugh. Not another proprietary DSL.

There is something amiss in the tech world these days. Too much esoterica and too many specialized tools, too many points of failure and dependencies, and too much duct tape. But, mostly, too much reinvention for frequently incremental "gains".

It seems normal, but it's really a mess. Environments are overly complex and it's a wonder anything works half the time.

At some point we will completely jump the shark. From there, my bet is there'll be a powerful movement towards simplicity.

/rant


But all the lisp people always talk about how incredible it is that one can have all sorts of DSLs....


Completely different. The Groovy DSL is a limited subset of Groovy. Lisp DSLs are embedded DSLs. They're libraries that help you write code in what looks like a new language, but the entirety of lisp remains at your disposal.


There are a lot DSLs writtn in Lisp, which don't use Lisp syntax and are not embedded.


being able to config jenkins via code is huge! looking forward to trying it


You can do this today with the Job DSL plugin.


You can do this today with the same Pipeline plugin they are announcing as a major feature for 2.0. It's been available for months.


If we were hypothetically "too big" for TeamCity, with scaling difficulties all over, do you have (or have heard) any recommendations? I hear the biggest companies all roll their own, but is there anything between TeamCity and Google-scale?


Jenkins + generated jobs (i.e. jenkins-job-dsl or the new Pipeline DSL), and keep as much as the build configuration and logic in your project where it belongs. The configuration that lives on the CI server should be pretty minimal.

We have a custom wrapper around the jenkins-job-dsl that encodes our conventions and templates to generate several hundred jobs, and it works very well for our medium-scale instance.


There's http://go.cd if you are looking at something focused on CD (Pipelines etc)


Yes, we've found gocd does a terrific job of allowing you cleanly model your full continuous delivery workflow and organize your servers.


We use jenkins at medium scale. > 400 jobs, 20k-30k builds/day.

Works just fine.


Jenkins works fine with dozens to hundreds of slaves per master, thousands of jobs, and tens of thousands of builds per day. Jenkins isn't perfect for everything, but it can be a very useful task runner. People have adapted it to do almost everything these days, from database maintenances to builds to restarting infrastructure. Given the almost unlimited triggering and reporting, you could use it to build many ETL apps as well (not saying that's a good idea).

As much as Jenkins sucks sometimes, it's what many big shops use for many things. There is tons of community around it. It works, it's battle-tested, and it's been that way for years. Anything else will have a hard time catching up for as many use-cases as Jenkins currently solves for people.

If you hate a part of it, fix it and open a PR :)


What is too big? Too many concurrent jobs? We found the integration with AWS to autolaunch workers to be quite easy to setup.


I tried buildbot, and found that the documentation didn't match observed behaviour, and their insistence on not providing any examples (combined with some in-house jargon) because "everyone is different" made the whole process particularly painful. Buildbot required a lot of attention and time to get working for us, and after all that, wasn't particularly end-user friendly. You're going to get exactly the same '2 people with tribal knowledge' if you use buildbot.


Jenkins always struck me as one of the few software projects that don't use OSGi but should be using it.


I've become a big proponent of Drone CI. It's very configurable and Docker powered.


Is there a potential release date, so that we could plan for the adoption of the beta for future releases?


ugh. this is still alive? i was done with jenkins the first time i had to use it.

since everbody's at it, i'll recommend github.com/drone/drone. drone really gives docker a pretty use case.


If there was ever a webpage that needed a facelift, it would be Jenkins, so hopefully this is not just a backend rewrite.


Take a look at the Usability Section[1]

Looks like a massive facelift, and lots of work done to make things more intuitive, faster, and more pleasing to the eye.

[1] https://jenkins.io/2.0/#ux


Unfortunately, the new design still looks almost a decade old.


So does hackernews. If it actually addresses the usability problems, it doesn't matter how 'old' it looks.


Couldn't agree more. Jenkins is a tool - it doesn't need to look "uber modern", but it needs to be functional (which it is).


Seriously. In FF editing a job is painfully slow.


I came here to write sth like "still in Java... :/" but I can't even check it, the site seems to be hugged to death.


> I came here to write sth like "still in Java... :/"

And the problem with that is...?


Typical issues surrounding Java projects:

- High startup time. Anything bigger than a hello world takes seconds, more typically tens of seconds, to start. My Jenkins setup takes 3 or 4 minutes before it is fully initialized. Even starting the JVM in client mode doesn't help much. I know, it's a server, and startup time doesn't matter once it's running, but it feels annoying.

- High memory usage. Heaps in the 1-2 GB range are not uncommon. This can probably be tweaked but as a user who is not a Java expert, figuring out how to do this (and figuring out what value is safe) is frustrating.

- XML configuration files. XML was hip in 2004 but these days it is frowned upon and users generally don't like it. This is technically not Java's fault (and I'm sure newer projects use something else) but it is something traditionally heavily associated with Java projects.

- Non-Unixy feel. Lots of things just behave like they're not really native.


> - High startup time.

This is really not a problem. If you're starting and stopping your Jenkins often, you're really doing it wrong. Jenkins is a server, and really should only get restarted when there's been an update. Not to mention, other large servers that are not Java still take time to startup... there's a lot going on during initialization.

> - High memory usage.

Perhaps more of an issue for some, but again, Jenkins is a build server, and should be on it's own hardware/vm, with it's own dedicated resources. If you're trying to squeeze by with 1GB of ram, you can, but you should expect the obvious results. The more complicated your builds, the more resources they're consume. Compiling GCC usually takes 1-2GB ram by itself. I don't consider this a problem.

> - XML configuration files.

If you're using Jenkins properly, you should not have to interact with any config files. Sure, if you're running it behind Tomcat or something, you'll need to dip into the configs, but not liking Jenkins because it has XML config files is shallow and superficial, in my opinion.

> - Non-Unixy feel. Lots of things just behave like they're not really native.

I'm not sure what to expect from a deliberately cross-platform system. Of course it's not "native Unix-like". But neither is any software written to support OS's that are not Unix-like. And again, you manage and use Jenkins from a web interface, so if you're on the command line, you're doing it wrong. (as an aside, Jenkins has native support for running shell scripts, which I make heavy use of... can't get any more unix-like than that).


the config is a huge problem, not just because of XML files. Jenkins (pre 2.0 at least) job configs have huge problems with config drift and doesn't play well with configuration management tools.

Trying to automate jenkins job configurations requires using the terrible job config xml format and a poorly designed rest API which breaks all the time when you add in a plugin or plugins get upgraded.


Then this should be a life saver for this scenario: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin

Jenkins has a ton of issues but with a bit of careful plugin selection it provides a ton of bang for your buck.


The new integrated DSL to replace that plugin is going to be great. Unfortunately, managing jenkins plugins themselves via config management/orchestration is still surprisingly difficult.


fwiw, this is very much on the roadmap - https://issues.jenkins-ci.org/browse/JENKINS-31094 - don't know when it'll land, but it will be done. =)


I would agree on some of your points : few minutes initialization times and 1-2 GB heaps may sound not that bad, and when we are talking about product that just provides a fancy, over engineered way to run bunch of shell scripts (considering jenkins doesn't integrate sandboxing from the box) ... honestly there is no excuse to spend 2 gigs of ram just to run shell scripts with web UI. It would be much better in a long term to improve energy/resource efficiency of software like this, just to make it more sustainable and reliable.

Edit: configuration-as-code feature is available in jenkins 1.x with job dsl plugin [1], I'm using it at work, because managing 50+ jobs by hand was just insane. Nowadays we just have few dsl files :)

[1] https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin


Jenkins is not my cup of tea, but in its defence:

1. High startup time is amortised over long run times. Jenkins is a general CI server, not an interactive build tool.

2. High memory usage is true, though again tolerable given the role and importance Jenkins will assume in any sane project.


> High startup time. Anything bigger than a hello world takes seconds

Oh, no! Seconds? Like five? Seven? For a process that you start less than once a day? Unacceptable.

> High memory usage. Heaps in the 1-2 GB range are not uncommon.

So... what? Spend an additional $30 on your build server and add 8Gb to it. Problem solved.

> - XML configuration files.

XML is fine. As opposed to Groovy, IDE's do a great job at offering auto-completion on it. But if Groovy is your bag, you can use that too.

> - Non-Unixy feel.

I have no idea what that means and I've been using UNIX for about thirty years now.


"Heaps in the 1-2 GB range are not uncommon."

Is this really still a complaint in 2016? Your phone has that much RAM.

Honestly if you're worried about Java heap size, spin up a t2.medium instead of a t2.small for your build pipeline. Running Jenkins on a Raspberry Pi is probably not a great idea if you're writing any serious application.


Actually it's becoming more of a complaint in 2016 with rise AWS instances that pay by the container size and with the rise of languages like go, etc. which have smaller runtime sizes and are more compatible with cloud computing environments like docker.

The JVM adds something like 400MB to every layer of a docker container resulting in multi gigabyte containers. Additionally the memory requirements for running a JVM put you immediately into the t2.medium range which will cost you 4X cost over a t2.nano for simply choosing to run Java instead of something with a smaller runtime footprint.


Don't run production code on a nano.

Don't run production code on a nano.

Don't run production code on a nano.

I don't care what language you're using, if you find yourself saying "well, I could run on a nano" just stop.


> The JVM adds something like 400MB to every layer of a docker container

Perhaps this is the "writing on the wall" that containers aren't what you should be using to deploy your app.

Just because it's the "latest craze" doesn't mean it's the best solution for the task at hand.


In the simplest case, Jenkins is just polling and cloning git repos, and then spawning some tests. This plus config parsing and web frontend is not a 1GB RAM job.


This is a horrible solution. No, my phone does NOT have that amount of memory available. It has it in its specs, but there's no way any ONE app should require that much. Look at what Jenkins is actually doing. Git/SVN checkout, build, other shell tasks. This should not take more than 16 megs!!! Let's be generous and add overhead for the GUI... so... 128mb. Java can certainly be tuned, but by default it wants a ton of memory.


In my opinion

Using XML is always wrong. If you want to automate Jenkins setup with sth like Docker or Ansible, you have to touch the configuration, and that means dealing with XML, which makes me cry. There are many ways to do configuration reasonably. Not so related to Java but hand in hand, as somebody else put it: "Java is a DSL to convert large XML files to stack traces".

It's impossible to debug, and its plugins are impossible to debug. If it would be in a scripting language, you could see what's going on easily, maybe print debug messages, and wouldn't have to clone-edit-compile-(deploy)-run in order to debug.

When some Jenkins plugin doesn't work out-of-the box, I straight up give up on it, because I know there's no simple way to interactively dive into the code. Based on my experience, I really tried. YMMV.


It works for me.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: