
BugJail: New Java debugger – debug across time with mouse - jacquesm
https://bugjail.com/
======
kij
Bugjail was mentioned in a thread about a xor bloomfilter alternative. Or at
least that's where I saw it first mentioned. My bet is it triggered op's
posting.

~~~
dang
That was quite interesting:
[https://news.ycombinator.com/item?id=21841921](https://news.ycombinator.com/item?id=21841921)

------
HenriNext
Hey, I'm Henri, founder and lead developer of BugJail. I just noticed traffic
spiking and came to see what is going on :)

The landing page wasn't quite ready for spotlight, but such is life :)

All feedback is highly appreciated, no matter how brutal.

And if you have any questions, feel free to ask!

~~~
lallysingh
I think you need to make the experience more concrete.

On the visceral side: put up some example videos of use. If you can't do it to
the level of polish you like, get someone else to do it for you as an
unofficial one. IntelliJ is already slow, and I'm worried about event-
recording debuggers being only useful for small programs -- larger ones eat up
too much RAM and cpu overhead to be tractable.

On the power side: show some sophisticated use cases using real libraries and
real problems to fix. What stuff can I find out directly that I can't easily
do in IntelliJ?

Can I directly query the DB? Can I use those queries' results in regression
tests?

When I read the page, I thought "Oh, like the valgrind family of tools for
Java, nice." So perhaps some pre-baked validations using the tech you've built
could be useful.

~~~
HenriNext
Yep, I think video would be a perfect fit.

About performance impact and program size: BugJail can be used also with
hardcore programs, like databases and compilers, where event based record-and-
replay debuggers typically struggle. There's still significant performance
hit, around "twice as slow", but it won't be orders of magnitude slower.

Also, BugJail is tested with large applications. For example one test is
startup sequence of Eclipse, on 5 year old box with 32GB. The result is ~700
million rows in database, in ~30 seconds. It does consume all memory, and your
workstation will scream for mercy, but it gets there :)

Direct DB query is not available in current beta, but it is partially
implemented in dev version: it shows the results, but there is no navigation
from the query results to CallTree, Source code view etc. So: coming, but not
yet.

I hope I answered your questions at least partially? It's 7AM in the morning,
after one of the hottest days in Australian history, and a surprise appearance
in HN front page, so I might not be quite the sharpest tool at the moment :)

~~~
hinkley
From what I understand, some time sequence databases use some sophisticated
compression techniques. Are you there already or is that on the roadmap?

I think it would be kinda cool in a way to go back to dual-machine debugging
but I’m not sure everyone else would agree with me.

~~~
HenriNext
There's fair bit of compression already, but actually I haven't done that much
industrial spying into time-series databases, good idea, thanks.

Also, I've tested dual machine debugging with BugJail, as part of spike
towards CI/QA/SIT/UAT/Pre-Prod use. It works okey-ish but the (crappy 1GB)
network became bottleneck and it was slower than single box.

~~~
lallysingh
If 1GB wired doesn't work well, WiFi will be worthless. Compression is a
really good idea

~~~
HenriNext
My "Fair bit of compression" was massively simplified: there isn't one true
canonical "compression", but dozen different places and ways where and how to
compress.

BugJail does compression around storage layer, but nothing before the network
hop from capturing agent to processing server.

And fully agreed of course; when BugJail graduates to CI/QA/SIT/UAT/PreProd
use, it will need compression before the network hop. But that is some time
away. Currently the agent and server run in same box, and compression between
them has negative net effect (compression uses more CPU than what shifting
around less data saves).

~~~
hinkley
statsd might be a good example there. Local coalescing before reporting
upstream.

Basically they didn’t make the client smart, they made a proxy and spam
loopback.

------
datastoat
Database-backed debugging is great! It means that debugging can be done as a
data science exercise, which is very useful when, for example, a rare
confluence of external factors make the program behave badly.

Earlier examples: "Debugging in the (very) large", which describes how
Microsoft used Windows telemetry, especially for debugging weird slow-down
glitches. In the networking domain, where there is AppDynamics for trying to
debug the interactions between network and distributed services.

I'd love to see data-science style debugging much more accessible (especially
in Python). And it's great to see that you're making a serious effort to get
the user experience right -- though I hope that the underlying data will also
be accessible from more capable data science platforms.

~~~
HenriNext
BugJail writes its data files in open source format, Apache ORC.

You can load the data files directly to Spark, Hive or anything else that
reads ORC, and then use Python to query, visualize or get statistics from the
captured program execution.

Give it a try and let me know how it goes?

~~~
rmetzler
I think it might be really useful to run BugJail as an (optional) CI step and
offer the user to download, integrate with the IDE, or have historical data.
But I didn't fully understand the license regarding such usecases.

~~~
HenriNext
Absolutely. The long term plan is to do separate edition "BugJail for
CI/QA/SIT/UAT/PreProd/etc", so that when things-are-not-fine on server, you
don't need to repro but just load the capture in BugJail and see what happened
[edit: that in addition to your "post CI build step" use case; both uses seem
compelling].

But the current beta version is not robust enough and doesn't handle long
captures well enough. I want to make sure that BugJail doesn't waste people's
time, hence the license allows only capturing from local workstation.

------
trishume
[https://pernos.co/](https://pernos.co/) is making a very similar omniscient
debugger based on rr for AOT compiled languages with DWARF info.

------
ctataryn
How is this different than
[http://chrononsystems.com/](http://chrononsystems.com/) ?

------
notkaiho
The page needs an editor, stat. The grammar is _not good_. I'd be happy to do
it to be fair, doesn't seem that big a job.

~~~
HenriNext
Thanks, actually I'd really appreciate your help! My email address is the
contact page.

------
wellpast
I wonder how they are able to trace and capture variable and stack state at a
rate of 20M invocations — per second!?

How is this possible on the JVM?

(I’m aware of the instrumentation API that profilers use but I thought they
had to do sampling to limit interference with program performance and they are
not capturing all the data req’d for debugging either.)

Anyone know?

~~~
HenriNext
There is no single trick, just lot of hard work.

If there's interest, I can write a blog post about this.

~~~
wellpast
I would love to read this. Thank you.

------
wheresvic3
This is really neat and I wish I had this around 10 years ago when I had to
migrate Java apps.

However, these days I rarely need to debug programs while developing - I
usually use some combination of TDD with unit/integration testing and it gets
the job done with the added bonus of being easy to maintain down the road.

Of course if you're on a project where time to market is critical and you need
to sacrifice tests, I suppose a debugger is required. But then again, you're
probably not using Java either.

~~~
bni
I don't see a conflict between using debugger and TDD.

I use the debugger to set breakpoints when running my unit tests all the time.
It lets me understand the code better when developing it.

Also convenient when pasting in data from production into the unit test to
find what is causing the bug.

------
knutwannheden
Since "database" and "relational model" is mentioned, I assume the collected
data is indeed being written to an RDBMS. Would be interesting to be able to
poke around in that with some SQL, even if not available as an official API.

~~~
HenriNext
Sure, I can add that to next major beta. Thanks for the suggestion!

------
aarbor989
I tried using it and my first impression is that it's pretty difficult to use
for a webapp (or maybe just specifically a Spring application since there are
a ton of added proxy classes) without the ability to search. You need to know
exactly which thread accepted your http request and then dig your way down the
call stack to find your method invocation. You should be able to search for
your method invocation and then go back up the call stack from there.

Other than that, seeing exactly how long every single method call took is
awesome. I think there's a lot of potential, but without the ability to search
I can't see myself reaching for this again

~~~
HenriNext
I totally understand your point.

The search functionality is currently under development, and it's by far the
most critical 'missing' feature.

As a workaround for webapps: using the 'Capturing Assistant' and configuring
BugJail to capture only your own packages helps a lot. That way you usually
see clearly which thread handled the request, and also have less app-server /
framework level crap that you need to expand through before you get to your
own code.

If you'd like to get notified when the search functionality is available, just
shoot me an email (my email is in the contact page).

Thanks for providing the feedback, and sorry that the initial experience
wasn't better.

~~~
aarbor989
Thanks for your reply. I actually did specify my packages in the agent
-javaagent:/Users/aa/.BugJail/agent.jar=--snapshot:.--primary:io.mycompany.
__but it didn 't seem to help too much. I'll shoot you an email to get
notified - is there an estimated timeline?

~~~
HenriNext
Hmm, if setting --primary didn't help, then it might not have worked (aka Bug
in BugJail..). I'd love to check that further tomorrow, if you have couple of
minutes?

As for the search/query timeline, I'm afraid to say anything more exact than
"month or two". The core search/query is working on dev version, but it needs
a lot of work.

------
thethirdone
I tried playing around with this and I can't see it being useful for me
currently. If the source and tracing information were more tightly integrated
it would be a lot more useful. Offering a more traditional debugging
experience with the ability to rewind time to the last time a value is set
would also make it valuable.

A specific issue I ran into was that multi dimensional arrays do get logged as
changing when a value is set in the bottom layer. This meant for my
application the were a bunch of init calls setting that value, but none of the
actual logical changes were tracked.

~~~
HenriNext
The source and tracing sides will definitely get fully integrated. This is
only the second beta that even has the source side at all, so it's just "early
days".

The line/statement level traditional debugging is possible to implement with
currently captured information, but the implementation is seriously hard,
likely in 2021+ timeframe.

About the specific issue: I'd really like to get that fixed, do you think you
could shoot me an email so that I can followup in case that i cannot repro the
bug (my email is in the contact page)?

Many thanks for the feedback & apologies that the initial experience wasn't so
great

~~~
thethirdone
I made a gist [0] that demonstrates the issue. Line 5 does not make a node in
the timeline.

I look forward to seeing what bugjail becomes in the future.

[0]:
[https://gist.github.com/TheThirdOne/7af9356dfb89d21816250017...](https://gist.github.com/TheThirdOne/7af9356dfb89d218162500179514b2c6)

~~~
HenriNext
Thanks, appreciated. I get this fixed.

------
baud147258
No Java 11 support? I was interested in giving it a try after the holidays,
since it look like it would be a good tool for the type of bugs we have.

I think it could be a big barrier to adoption, considering how Java 8 is EOL.

~~~
HenriNext
Yeah, Java 9+ is on roadmap for next beta.

When the previous major beta was released there was one dependency that didn't
support Java 9 and locked everything to 8.

If you'd like get notified when Java 11 support is ready, you can just send
one line ping email (my email address is in the 'contact' page).

Many thanks for the feedback!

------
ccleve
Two comments on the website:

There is some reference to a beta in 2018. Should remove that.

Also, a video of the thing in use would be helpful. Put it right up at the
top. Show me something I can't do now. Include specific examples.

~~~
HenriNext
Thanks, that 2018 is a very unfortunate typo, should have been 2019.

And agreed, video would be great, but it takes fair bit of time away from
development, so I'm undecided.

~~~
ccleve
Shouldn't take long. Google "screencast software". Record yourself: "Hi, this
is Henri. Let show you how find bugs with BugJail..." Do a 3-minute video with
some simple examples. Put it on YouTube and embed it in your site.

If I were doing it, I'd budget 4-8 hours of work and close to zero cost.

------
brian_herman__
This looks really good. I like the fact you can drill down and have a replay
debugger with a slider! Great job! I wish it worked for python!

------
jhvkjhk
The `direct answers` looks fascinating! Do we have some equivalent tools for
c++ or python? I'm tired of step step step...

------
localhost
I would imagine that the execution logs would get rather large. Is there a way
to dynamically trigger logging as well?

~~~
HenriNext
The execution logs (or "Captures", in our Jail terminology..) do get fairly
large if you capture something hardcore, like database or compiler.

For normal web/business/enterprise applications you can configure BugJail to
only capture your own packages, which massively reduces the file size (and
performance impact).

To give concrete example of hardcore scenario, capturing startup of
Elasticsearch results in about 85MB file.

There is no way to enable capturing dynamically, and this is likely never
going to be supported, due to the way how BugJail works (it reconstructs the
program execution, so if beginning of the execution is missing, "things are
not fine").

------
giorgioz
As a constructive feedback on just the presentation of the landing page I
think the screenshots are showing some code with theme colors that are kind of
ugly. I understand this is a landing page but if you care about Developer
Experience DX I'm sure you can find a theme that is better looking or
inspiring.

~~~
DailyHN
Those images aren't visible on mobile. And I was expecting much worse when I
viewed them in desktop mode.

Based on the stage they're at, I'd save optimizing the landing page design for
the future.

Though, I would put effort to better describe the problem on the landing page.

------
Alupis
Looks like a pretty nice way to debug Java programs! I look forward to testing
it.

One quick note though, you might want to consider removing the politics from
your screenshot before really releasing this product. You've got a great
product here, but there's no need to inject politics - even if it's amusing to
some. It takes away from the seriousness of your project and makes it feel
more like a personal side project instead of something I should pay for and
build into my workflow, in my opinion. Just my two-cents.

~~~
HenriNext
Yep, it was already mentioned somewhere here, but basically: I agree & will
do.

I look forward to your feedback on the actual product, be as brutal as you can
/ want! :)

~~~
Alupis
Great! I've shared your link with a few other java devs and the consensus so
far is this looks like an amazing tool. I can't wait to take it for a spin!
Keep up the great work!

~~~
HenriNext
Thanks, appreciated!

------
stevemk14ebr
This doesn't work if source code is not available. I can get a trace of the
function calls but no arguments, or return values or anything. This makes it
entirely useless as a reverse engineering tool, something i was very excited
to use it as due to it essentially being a tracer. It also NEEDS modern java
support.

~~~
HenriNext
Actually, it does work without source code (or, should work)

The left side 'CallTree' is fully functional, and shows owner instances,
arguments and return values. If you click those, then the Object/Array is
displayed at bottom and is expandable just like in normal debugger (+slider to
change debug time). So only the right side "Source" view is missing if you
don't have source.

If the above doesn't work, could you get in touch and I'll get it sorted out?
[https://support.bugjail.com](https://support.bugjail.com)

And fully agreed on Java 9+ support, coming in next beta.

Thanks for the feedback!

------
bni
I find this very interesting. Can you use it integrated in IntelliJ. How about
as a paid plugin?

~~~
HenriNext
There's no plugin yet, but definitely coming (I've been using IntelliJ almost
20 years now, so also want that integration for my own use).

------
lvh
Sounds similar to sayid, though generalized to the entire JVM?

[sayid]: [https://github.com/clojure-emacs/sayid](https://github.com/clojure-
emacs/sayid)

------
petagonoral
How does it compare to Chronon DVR[0] for Java. Seems similar functionality
wise

[0] [http://chrononsystems.com/](http://chrononsystems.com/)

------
lowercased
android debugging to start in late 2018...

is there any progress on this project (bugjail in general I mean, not android)

~~~
HenriNext
Auh crap, that is just typo, should have been 2019. BugJail is under very
active development.

~~~
netsharc
Late 2019 is right now, any time after now is really really late!

~~~
HenriNext
I finally got the courage to deploy new version under heavy traffic, so it's
mid 2020 now. No more snarky comments for half a year :)

------
carapace
Can I get the source? In other words, how do I debug the debugger?

~~~
cryptonector
I'm guessing "no", but it sounds a lot like DTrace/BTrace with many enabled
probes married to a DB.

~~~
cryptonector
See
[https://news.ycombinator.com/vote?id=21846545&how=un&auth=99...](https://news.ycombinator.com/vote?id=21846545&how=un&auth=995cf76915ecf63357da8f609c96704d4e305e99&goto=threads%3Fid%3Dcryptonector#21846545)

------
ylyn
This looks like rr or GDB reverse debugging but for Java.

~~~
dhlin
Have a question to rr or any replay debuging tool, does it still work if the
program has database access while the replaying machine doesn't have or not
have the same database?

~~~
cfallin
Yes, it records all syscall return values, so the replay doesn't actually need
to perform them. This means that IO (GUIs, network accesses, etc.) shouldn't
happen on replay, in general. (The replay does show stdout/stderr data, so
perhaps those are special-cased?)

