Hacker News new | past | comments | ask | show | jobs | submit login
Pernosco Omniscient Debugger is now available for individual developers (pernos.co)
62 points by modeless on Nov 10, 2020 | hide | past | favorite | 44 comments

We use rr extensively for Julia development [1] (for a sense of scale, I probably look at hundreds of rr traces a month). There's two big challenges for us there 1) Many cloud environments are broken in various ways for rr recording - I spend way too much time guiding our users to the few that do work 2) The GDB frontend really isn't the best way to view all the underlying information

Pernosco is well on its way to address the second part of this. Its replayer is quite a bit nicer than the open source rr one and I have high hopes for the platform in general.

For the first one, all I can do is plead with the cloud vendors not to mess up the environment. Since this is HN, if you're working for a cloud provider, please ask your engineering team not to break rr. Otherwise I'll have to spend time migrating our users off your platform, which will be fun for neither of us.

Anyway, congrats to Robert and Kyle on the launch. Wish you all the best.

[1] https://julialang.org/blog/2020/05/rr/

Can you be more specific about which cloud environments do work with rr? In your blog post you said it works on Amazon EC2 instances above a certain size. Which instance types are known to work?

The main thing is that rr recording needs access to the PMU. So generally AWS Nitro instances which occupy a whole CPU socket (or more than one) work. In particular AWS c5(d).9xlarge and larger are known to work. Other instance types (including metal instances) should work too.

Could you share examples of how cloud providers break rr?

Usually they break rr by either blocking access to the hardware PMU entirely (e.g. on AWS, any instance that shares a CPU socket with other instances; on Azure/GCP, everywhere AFAIK), or by introducing subtle errors into the virtualized performance counter values that rr depends on (e.g. VMWare: https://robert.ocallahan.org/2015/11/rr-in-vmware-solved.htm..., Xen: https://lists.xen.org/archives/html/xen-devel/2017-07/msg022...).

Another feature rr benefits from is CPUID faulting, which is not virtualized by any of the major cloud providers (though Linux KVM does virtualize it): https://patchwork.kernel.org/project/kvm/patch/2016110101374.... That'd be "nice to have" but we can live without it.

The video references and recreates a debug session which is originally from a great blog post by Brendan Gregg: “gdb Debugging Full Example” [1]. This original blog post uses only free tools to reach the same conclusions as the Pernosco session. So it’s interesting review and compare them both.

[1] http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncur...

Also, that Pernosco session is public so anyone can try their hand at it: https://pernos.co/debug/e9UFXkGq__8KQB-N6n59lA/index.html

I’ve been an extensive user of time travel debugging for most of my career, and Pernosco looks great. However, please, please change your business model. As is, it is useless for employees of corporations with sane legal departments.

We would love to talk about on-prem with those corporations --- contact us!

What’s the best time travel debugger you’ve ever seen? (any language)

Out of curiosity, has anyone else noticed that the videos and screenshots on the About page are not actually videos or even images? They're iframes with the actual Pernosco UI inside them! That is so cool.

Is there a way to compare two sessions? Basically, do a diff and see where the execution flow differed between two sessions? At high level first, then drilling down.

They list that feature as a TODO for the future at https://pernos.co/about/vision/#comparing-executions

If it's done, it will allow cracking software that uses basic protection methods for $10 flat.

But for the normal use both the pricing and its off-prem nature renders the whole thing all but useless. A pity really, because the tech looks fantastic.

I agree that comparing different runs of the same program will be amazing.

I think that running Pernosco off-premises will be acceptable for many people. I use it a few times per month while working on open-source projects like Reposurgeon and Flex, so I'm not revealing any trade secrets by uploading those recordings. I've learned that the database built by Pernosco from a large recording can be 100GB and take an hour to generate on a 36-core AWS instance. $4 per debugging session is a pretty reasonable price compared to buying that hardware myself.

And they are offering an on-premises solution, running on your own hardware. For a tool this good, the price for that is bound to be large enough to give individuals like us reason to think twice, and then probably a third time. Given the potential time savings, I think that it should be easy for even a small company to justify the purchase.

On the gripping hand, this is non-Free software. Some will avoid it primarily for that reason, and I can't really fault them for sticking to their principles. I think that it's entirely possible that Pernosco, or debuggers like it, could effectively eliminate traditional debuggers like GDB. GDB would still exist and still work, but without the nearly-omniscient viewpoint that Pernosco offers very few will bother to use it. It's like looking through a keyhole. I've thought about this a lot over the last few months.

This is insanely expensive. 20 usd for 5 submissions?

    "To use Pernosco, you must consent to collection, transfer, storage and use of your data"
I understand the need to monetize but honestly seems like a huge commitment in terms of both trust and money to use a tool like this.

Perhaps a nagging screen model would work better? (better => more users using the tool, a percentage of which would def. pay to remove the nag and support the creator/s).

A list of successful paid software I can think of from the top of my head:

* https://www.reaper.fm/ (nagging screen)

* https://www.linqpad.net/ (PRO versions with additional features)

* https://www.jetbrains.com/idea ("Community" and "Professional" versions)

* https://cursive-ide.com/ (30 day trial? I think? Personal, Commercial and Non Commercial licenses avail.)

* https://www.sublimetext.com/ (no limitations but purchase is expected, not enforced)

* https://github.com/cognitect-labs/REBL-distro (used to _require_ joining a Patreon to use the software, but did not enforce it... Also I think it used to have commercial licenses available).

The service model is not just about monetization. It enables collaboration features that are really useful, e.g. https://pernos.co/about/browser

Resource utilization is also an issue. To accelerate the debugging experience we build huge databases of all program states. You can't run Pernosco on your laptop.

Having said that, we understand the trust issue, we are very open to offering an on-premises version of Pernosco and we are keen to talk to people interested in that.

It's definitely a factor worth considering before purchasing the service. It's going to record an entire program running on your local machine, including all the data that it reads and writes. That includes local files, network connections, events from your window manager, pixels put on the screen, any environment variables that happened to be defined, etc. If any of that is private then you shouldn't upload the recording to Pernosco. If you're working on a public GitHub repository then your concern is pretty much limited only to your local environment. Make sure you don't have security tokens in your environment variables when you record. If your program opens files, make sure it opens something harmless. If your program visits webpages, don't record it visiting a page you want kept private. It's only natural then that the EULA would want you to give Pernosco permission to receive that data, store it for you, and then present it back to you during your debugging sessions.

There's also the on-premises option.

Note that the pernosco-submit script supports detecting "sensitive" environment variables leaking into the recording: https://github.com/Pernosco/pernosco-submit/blob/a6119a1b291... And we accept PRs :-)

That's true! I ran afoul of that several times. It was easiest to clean up my environment, but I thought about writing a patch to rr that would bail out before recording anything if it detected those variables. Having that would have saved me from having to rerecord a few things.

I suppose an option to run the same checks in rr at recording time would be a fine feature to have ... though you could implement it pretty easily as a wrapper script too.

We often get users estimating that a Pernosco session saves them half an hour or more. Let's say on average it saves 15 minutes. Let's say a reasonable developer's time is worth at least 50 USD an hour. Then Pernosco is giving 12.5 USD of value and we're charging 4 USD. Is that insane?

To me it feels like it's a weird scale. 5 a month is something where I actually have to think when and how to use it, and that's not something I want to do for a tool like that.

I assume it's still usable as a local rr-GUI (that hopefully adds something) without submitting?

I get that you are still looking for a model, but I would much more expect something like "my employer pays $X00 or $X000 and we get a volume where I don't have to think about using up a scarce resource" (potentially even can have customer systems run it if there's issues) - the market for single licenses feels pretty small to me?

EDIT: cleaned up word salad

> I assume it's still usable as a local rr-GUI (that hopefully adds something) without submitting?


> I get that you are still looking for a model, but I would much more expect something like "my employer pays $X00 or $X000 and we don't get a volume where I don't have to think about using up a scarce resource" - the market for single licenses feels pretty small to me?

We like that model too but it's been hard to sell that. We're certainly happy to offer such a deal if someone asks for it!

People in related businesses have said they've had more success with "bottom-up" individual-developer sales than with enterprisey deals, at least initially, so we are giving it a go.

I guess that makes sense regarding slowly getting a foot in the door and leaving enterprise to "let's talk about it".

I don't think it's for me, but I'll try it out and either way good luck, I hope it works out for you!

To help with this we've added a new "volume subscription plan" for individual users: 50 submissions per month for $50.

Do you realize how insane things would be if everything was charged as a percentage of value provided ?

So you charge, 1/4 of the value given. Should my IDE charge me 1/4 too? My ISP ? My computer OEM ? My electricity provider ? My renter ? My OS ? That's a lot of 1/4th.

OK my workflow might be run, debug, change code, recompile, run, debug... Like many many times over. I can't imagine having to pay every time I use the debugger. Like it's weird that you pay per submission.

Yeah, Pernosco may not be a great fit when you have to do that. That's not just due to the pricing model --- which we can change! --- but the entire debugging approach, where we do a lot of analysis up front, speeding the debugging experience but adding latency to the workflow.

Now, sometimes people are doing that because to track down a bug they have to go through many cycles of adding instrumentation or logging to their code. In many cases Pernosco can just make those cycles go away, e.g. using https://pernos.co/about/expressions.

However, if you develop new code using cycles of "add new code, compile, debug", and each of those debugging sessions are short, Pernosco isn't a great fit for that currently.

>Is that insane?

I have no idea, but I’ve already lost interest.

Same. Punishing tool usage is a really dumb mistake. You want your tool to be used all the time and become indispensable, so you can charge a high monthly fee for it. Corporations are used to and even expect fixed monthly fees. Pernosco doesn't at all understand how large firms work, and what it takes to get a budget for developer productivity tools.

I suggest you consider giving them a little benefit of the doubt?

I agree with you that for a larger corporation it makes much more sense for the tool to be prepaid and for the developers to not worry about their usage. There's definitely a learning curve to switching to a new style of debugging, even if you're accustomed to rr, and so encouraging use is more likely to get people to actually get to the point where they're benefiting from the tool.

But that's a tough sales cycle, they've tried it, they've said they would be happy if someone came asking for that. You could make the argument that they should hire an enterprise sales person or something, which may be true but also has obvious costs. (Let's just say that it would be a large percentage increase over their current staffing.)

I'd think that their reality wins out over your "shoulds". You may be right that they're incompetent about selling into the enterprise, but I'm certain that they've had more experience trying to sell Pernosco to larger corporations than you have and so I'm not going to ding them for this additional purchasing opportunity.

Source: I work for a mid-sized company that does have a contract that gives me unlimited access to Pernosco. It's quite useful. I don't know the exact terms. I know I have solved some bugs that I would have been unlikely to ever fix and would be ongoing stability and maintenance concerns.

I'm working on a bug right now where the submitter is able to reproduce easily but I am not. They have managed to capture it in an rr recording, so I'm uploading it to Pernosco to take a look even though my desktop's CPU is mismatched enough that I can't just run the recording myself. From past experience, if I have it in Pernosco, I will be able to fix it, even though the root cause is far far removed from the resulting crash. (It's subtle GC corruption.)

Oh I was under the impression that was the exclusive purchasing opportunity. In that case, good for them. From personal experience, I just very rarely had the situation that even if it made perfect sense and would have saved money, that it was easy or even possible to buy dev tools with company money. Its stupid but I know its the same in many places, the bigger the worse it gets (unless FAANG).

It looks cool but i dont adopt platform services in general. Seems like a profitable business if that port doesnt scare devs away.

Are you saying that the average session saves the developer 15 minutes, or the average successful session, in which the bug is actually solved, saves 15 minutes? How likely is it that a given bug will be solved with Pernosco? What do I do if I pay and only realize after that the session wasn't helpful?

The average session. Of course it's impossible to know the actual average number for any given developer but we often get developers estimating Pernosco saved hours, occasionally days, or occasionally the bug couldn't have been solved any other way. So I just tried to make a conservative guess.

> How likely is it that a given bug will be solved with Pernosco?

There's no way to put a number on this but I think it's unusual to not eventually figure it out with rr or Pernosco (but Pernosco will be faster) if conditions under the user's control are met: the recording really captures the bug you care about, it's in code in a language we support, and the debuginfo and sources are right.

> What do I do if I pay and only realize after that the session wasn't helpful?

"Sorry". Hopefully the reasons it wasn't helpful are mostly under your control and the benefits average out in your favour.

Worth it.

I'd prefer to pay once and then install it on my own computer, even though that would surely cost several thousand dollars. It would still be worth it.

I am not sure I would want to self-host this. It does a lot of analysis on the cloud. https://robert.ocallahan.org/2019/07/auckland-rust-meetup-bu... gives the scale of "100s of GB of data produced saturating 36 cores for hour", now that's a heavy compute.

Cool, I hadn't seen these numbers before. I have the disk space, but not the cpus. Do you think I could get it down to just 17 minutes if I buy a dual-Epyc machine to run Pernosco on? I'm asking because I've been looking for an excuse to buy a dual-Epyc machine for a while now…

For the curious, a small discussion about this from last year:


Amazing name. Has my Latin nerd stamp of approval.

Is it possible to use something like rr for microcontrollers, assuming we can simulate the microcontroller?

Yes, you can instrument simulators to get record-and-replay. Simics has a record-and-replay feature like that, so does QEMU lately. I don't know of anything specifically for microcontrollers but it certainly could be done.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact