
Stop Filing Bugs, File a Container - tolmasky
http://blog.runkit.com/2017/02/01/stop-filing-bugs-file-a-container.html
======
feld
I hate to be negative, but this feels like an invitation for very poor bug
quality reports where now you have to debug an application _AND_ an OS (the
container).

~~~
Me1000
Hi there, I'm on the RunKit team.

I think we've all been in a situation where we have a bug reproducing in front
of us but when we ask for help by describe or even give code to someone else
they say "hmm, it works on my machine". Since software rarely runs in a
vacuum, having better system level access can sometimes be the only way to
debug it.

Our hope here is that RunKit notebooks make it easier to see what's wrong,
even when the problem is less obvious.

~~~
philipov
The most common situations for having a bug that works on my machine but not
other people's have to do with memory access violations. But it seems like
when you containerize the environment, you will change the memory layout and
risk the same situation where the bug disappears because you've moved it to a
different machine.

~~~
coldtea
> _The most common situations for having a bug that works on my machine but
> not other people 's have to do with memory access violations._

Not even close. Different classpaths, different versions of packages, some ENV
setting like LOCALE causing havoc, a different configuration between the two
installations of the programs that are debugged, a single core vs multi core
cpu that masks some race conditions, there are literally MILLIONS of things
that can, and in my experience, have, cause something to work in one machine
and not another.

In fact memory access violations do not even register as a blip on the top-100
reasons...

------
voiper1
Wow. Not sure why all the hate. This is amazing.

It's like the live javascript/html/css editors, but for systems!

[https://runkit.com/npm/lodash](https://runkit.com/npm/lodash) and poof, you
can run queries against lodash and share and they can edit it. (which you
could do in a plunkr too, if you loaded it). I didn't even need to sign up...

Looks like [http://code.runnable.com/](http://code.runnable.com/) does similar
for more stacks, but you have to sign up for that.

------
chriswarbo
I find Nix invaluable for this kind of thing; especially how it rebuilds
anything who's dependencies have changed, and caches everything else.

I tried to play with docker, but haven't seen much use for it in for my
current workflows.

~~~
swift
I don't work on projects where Docker is useful in production, but I've found
it tremendously useful in continuous integration. It guarantees a reproducible
environment, which a VM could do, but just as importantly it _documents how to
produce that environment from scratch_. I've found that hugely valuable.

I've always liked the idea of Nix but I've never used it. How easy is it to
extract and understand the details of the environment you've constructed to
build your project?

~~~
chriswarbo
> Docker... documents how to produce that environment from scratch. I've found
> that hugely valuable.

> How easy is it to extract and understand the details of the [Nix]
> environment you've constructed to build your project?

The "extract" part is easy: Nix uses a declarative language to describe the
environment, so in that sense it's like docker, but more fine-grained: each
package is built in isolation, and installed to a read-only filesystem.

As far as "understanding the details", it depends on what you want to know.
You can use "nix-shell" to enter a build environment in a shell, e.g.

    
    
        nix-shell -E 'some Nix expression'
    

will enter the build environment of 'some Nix expression',

    
    
        nix-shell -p 'some Nix expression'
    

will create an environment with 'some Nix expression' installed, etc.

For example, I'm currently tracking down a space leak in a Haskell program by
running the following command after each edit of the source code:

    
    
        nix-shell -p 'with import <nixpkgs> {}; profiledHaskellPackages.callPackage (runCabal2nix { url = ../myTester; }) { myDependency = profiledHaskellPackages.callPackage (runCabal2nix { url = ./.; }) {}; }' --run 'tester +RTS -M100M -xc'
    

The command 'tester +RTS -M100M -xc' is a Haskell program which dies if it
uses more than 100MB of heap, exposing the space leak and dumping a stack
trace.

This will be run in an environment containing Haskell packages built from the
source code in "./." (the current directory, for "myDependency") and
"../myTester" (for the "tester" command). These packages and all of their
dependencies will be built with profiling enabled.

Running this command over and over will just re-use the previously-built
environment. If I edit the source in ./. or in ../myTester, Nix will notice
that the hashes have changed, rebuild the packages and their dependents and
make a new environment using those. This makes it easy for me to test
potential fixes.

------
Animats
So now someone can reproduce the bug, but can't test a fix to it. They can
take apart the container and attempt to reproduce in their own environment,
but that's a lot of work.

~~~
tolmasky
We provide a download link on the left which will download the file with all
the dependencies shrink-wrapped, so that you can certainly start hacking on it
locally too. This is a great start and a lot easier than manually trying to
npm install the right combination of sub-dependencies, etc. If you have any
other ideas we're happy to implement!

~~~
contingencies
The point is that complexity is most effectively managed by discrete (modular)
consideration, not another layer of abstraction, which tends to hide it. Build
process is a discrete development process problem space with its own group of
established solutions. It should not be conflated with bug reporting.

 _All problems in computer science can be solved by another level of
indirection, except of course for the problem of too many indirections._ \-
David Wheeler

~~~
sametmax
If you are in the type of bug runkit targets, getting the exact same env right
will take you days. With runkit, you do have a little setup to do, but the
days can be used to debug.

This is a great concept : remove the work to reproduce an environment, and use
the freed resource to understand said env.

Major advantages are:

\- the bug reporter may not know his/her case is special and won't provide you
with the information you need to see it. Providing a running container with
the bug just obliviate that.

\- the running container can come with a docker file, which gives you
everything: libs, versions, settings. It's basically a summary of the stuff
you need for it to go wrong.

\- you can try locally the stuff, even if your setup is completly different,
without messing with your setup. Because complex bugs rarely are reproducible
with a bunch or pip install, apt get and other yum incantations.

It's not good for all bugs. I would say it's actually bad for most bugs. But
for the bugs that needs it, it seems fantastic.

~~~
contingencies
_getting the exact same env right will take you days_

If your codebase has significant undocumented environmental dependencies and
these are hard to script in place, there's something far more fundamentally
wrong with your development process than bug reporting.

~~~
sametmax
Lol you react like most of us have software deployed in controlled
environments.

I create Python libs that are used in so many various configurations by so
many people I don't know. They come back and say "hey, I got this stack trace"
(in the best case).

Now I have to play the guessing game. Is it me, is it you ? Wrong path ?
Permission problem ? Bad conf ? Network is having trouble ? Server is this
particular linux version and and there is something important about the SEL
setup here ? Oh but upstart/systemd don't behave the same way. Output is
redirected here look. Stuff is not the expected encoding. Na it was not a bug
but you're file is corrupted. What the heck is this data format ?

Etc, etc.

You thing you though about every single thing ? Your error handling is perfect
for all IO ? You deal with all encoding, all user inputs perfectly ? You know
all the little OS peculiarities that will make your subprocess run in the
exact way you think ?

Of course you don't, nobody is perfect, we don't have infinite resources. But
there are infinite ways to fail.

And for many things, you can figure it out with with just your code base and
the error because it's a simple cause. But from time to time arrive this
terrible bug that is a mix between a strange LOCAL, this particular version of
the VM you use but only in one time zone with this env variable set. And for
that, yes, a good container with a reproducible bug in the proper env is an
interesting idea.

Apparently, you don't ship software that is wildly used enough or you would
not be that arrogant.

Deploying on your own 100 servers is hard. Try seing your code deployed on
1000 servers that you don't own, nor configure.

~~~
contingencies
I'm trying to help point you in the right direction based on my own
experience, not be arrogant. It sounds like you would get a lot out of
spending some time working on CI/CD concepts.

In short throwing your hands up in frustration at the complexity of software
is not a solution and gets you nowhere. The way we deal with "infinite ways to
fail" is to control the environment. These days, quality projects are expected
to version control their environments and conduct test deployments within a
representative set of environments using a representative set of
configurations.

Docker provides an easy way to do this ("always deploy on
<distro>-<os>-<version>"), but it's only one approach. Another free and
relatively straightforward place to start getting up to speed would be
automating build and test processes with Travis CI for an open source project.

Deploying to any number of tested environments is trivial.

~~~
sametmax
You are completly missing the point.

If you create a Python lib, 10000 people will pip install it. You have no
control on the env.

If you create a deb package, you will see ppa and it will be installed on many
various env. You have no control on the env.

Again, you definitely have no experience in shipping software outside of your
bubble.

A lib is not "a web project". A cmd line tools is not either. You still need
to debug them. People will run them on windows, linux, mac, bsd, and who knows
where. And they will come for you.

Now you can choose to simplify the problem and only support a limited number
of env. But I guess I'm quite happy the guys who created apache, ffmpeg didn't
force me to only used them on Linux with LOCALE set to accept only ascii and
CEST.

~~~
contingencies
No. As I said, in these cases you use a CI tool to test on a broad range of
environments.

For example, here is a library I maintain with 12,000+ installs per month that
is tested on 7 different environments every commit using Travis:
[https://github.com/globalcitizen/php-
iban](https://github.com/globalcitizen/php-iban)

~~~
sametmax
7 env. LOL.

Don't get me wrong. It's a good thing. Most people I know don't even have unit
tests. Having a CI is fantastic. 7 env is more than a lot of people do.

But that doesn't even scratch the surface of the combinations of factors you
can get. It can't.

You can't install all the locals, all the lib versions, simulate all network
conditions and all user inputs.

You will get strange bugs that your CI didn't take in consideration. And some
of them will be very hard to reproduce. That's just a fact.

Now knowing you do have all this setup I just can't understand we still
disagree. It's impossible you didn't run in those.

------
saosebastiao
I wish we had better reproducibility tools. It should be possible to write a
failing test case, and everything that is not of value to that test case gets
stripped out, containing just the parts of the program/environment that can
affect the test case (including sources of non-determinism like clocks and
RNGs). Kinda like C-Reduce, but on steroids.

And we should be able to look at the problem probabilistically too: given the
form of the test case, the value pipeline through the debugger, and the
structure of the program, we should be able to narrow down places to look as
much as possible using some form of bayesian search theory.

~~~
mrkgnao
Minimal test cases sounds like QuickCheck.

------
mattwhilden
We have something "similar" for the .NET Native compiler. You pass a special
flag to the compiler and it will generate a file with more or less everything
we need in house to diagnose your issue.

Compared to cut down repros, the packages are huge (200+MB) and debugging
requires more gumption but we save a huge amount of time going back and forth
with customers. It's not ideal but we'd rather have a "bad" repro than never
hear about it.

We've been taking customer issues this way for almost 2 years and it's been a
good thing to have in the tool belt.

------
tolmasky
Hi I'm part of the RunKit team and am happy to answer any questions!

------
eridius
This looks really cool. I wish I could do this with other languages.

~~~
Myztiq
You could do something similar with
[http://code.runnable.com/](http://code.runnable.com/)

[Disclaimer - I work at Runnable]

------
TheAceOfHearts
This looks really awesome. I really like that you allow users to try it out
without having to sign up.

I think probably the biggest hurdle with something like this gaining traction
is having the users remember that it's available.

Is it possible to fork from an existing container? If that's possible, you can
probably get maintainers to include a link to a working container with the lib
already setup in the project's new issue template.

~~~
tolmasky
You can link to an existing notebook and add "/clone" to the end of the URL to
basically get this behavior. That's how the 'Try It Now' button at the bottom
of the page works, we should definitely make this more obvious. Additionally,
maintainers can embed RunKit on their pages with our embed API:
[https://runkit.com/embed](https://runkit.com/embed). For example, Lodash uses
embed on their documentation:
[https://lodash.com/docs/](https://lodash.com/docs/).

------
amelius
Or send a coredump :)

------
shade23
I think this would make a lot of sense on platforms that undergo massive
fragmentation(all fingers pointing at android here). Browsers are also
fragmented , but websites like Can I Use It[1] should be able to reduce the
problem to a manageable extent. I love the idea of reproducible bugs. Because
often the hardest part of solving any bug is first being able to reproduce it
followed by finding the source. Since I presume these containers would also
help in reliably reproducing those one-off bugs which happen only on first
install/ first open / on some other event.

[1]:[http://caniuse.com/](http://caniuse.com/)

------
corndoge
Down in microcontroller land we generally just send an email describing the
problem. But not knowing much about web development, I suppose problems in
that area are much more complicated.

------
dullroar
Am I the only one put off by the bad "s-t" ligatures as being too twee? It may
or may not be a good approach, but I stopped reading literally ("Archer"
literally) got too annoyed by the distraction of that font face. Edit: I want
to downvote myself-my wife's W10 box show no annoying ligatures. My Linux Mint
18/Firefox setup is otherwise. Sorry for the distraction!

