Hacker News new | comments | ask | show | jobs | submit login
Debugging Node.js in Production (netflix.com)
173 points by aaronbrethorst on Dec 4, 2015 | hide | past | web | favorite | 72 comments

I am fairly new to node but find that it is hard as heck to debug. node debug index.js seems really weird and not at all like python's pdb. Is there something I am missing?

Trying to figure out some websocket stuff, I somehow managed to get nodejs, autobahn, and whatnot running locally (client and server) using Jetbrains' WebStorm on a Mac.

This tool stack makes me want to knock my teeth out.

That WebStorm's debugger (with break points!) works at all is a marvel. Props to the JetBrains team.

Check out node-inspector

Yeah, there are better debuggers out there for node. Hopefully others will come along and make a list, but there's a number of graphical ones you can find with a little searching. Even visual studio has step through debugging at this point.

Nope. node and server-side JS in general is a pretty gross fad.

"Name your functions": Easier to know what the code does, flattens the "Christmas tree from hell", no need for ugly closures, + Makes it's easier to debug. Anonymous functions is probably the most abused syntax in JavaScript.

The argument that stack traces are easier to debug when functions are named is always brought up when talking about the issue of naming functions. However, I don't quite understand why it's such a big issue. Whenever I get a stack trace the issue is almost always easy to debug. That has been my experience working with JavaScript full time for two and a half years.

And every difficult to debug bug hasn't been related to knowing which functions are in the stack, but are instead related to figuring out why frameworks do something or race conditions and so on.

I just find it surprising that people are having problems with anonymous functions in stack traces, when I've never seen that.

Yeah I don't understand why "name your (short callback) functions" gets repeated as a truism so much. Do people have stacktraces that are missing line numbers?

Line numbers don't help you with memory leaks, seeing the name of your function there does.

This. They don't show up in heap dumps and sometimes CPU profiles.

How do you easily glance at a stack trace and know if it is your own code or a 3rd party library's code? Named functions definitely help a lot.

Because at a glance, it's easier to see the path of the file as an identifier than the name of the function. Function names are not unique throughout the codebase and dependencies. Lots of dependencies have the same named functions.

I've never understood JScripters' insistence on anonymous functions instead of named ones.

As someone just delving into Node from another platform what're "proper best practices"-based JS patterns?

Lambdas make code really easy to read compared to making the reader skip around the code. In addition, it reduces boilerplate code and requires less time to be spent on naming functions. Of course, it all depends on situation. Simple logic which is not beneficial elsewhere can be implemented as an anonymous function in my opinion.

For example: if(objects.find(object => object.id === id)) {}

Also: anonymous function, file x, line y tends to be enough for me.

>Lambdas make code really easy to read compared to making the reader skip around the code.

You can have lambdas that are also named, e.g:

if(objects.find(function idCheck(object){ return object.id === id})

Not sure if it's possible with the new ES6 syntax [Edit: apparently it's not, although arrow functions will automatically be assigned a name in some common scenarios].

That defeats the whole purpose of fat arrow functions.

I'm not sure what purpose you have in mind.

The main purposes of fat arrow functions is the auto-binding of this and succinctness.

A named fat-arrow function would still retain both, and in fact the feature was considered for ES6 back in 2013, but didn't made it. Syntax would have been like:

if(objects.find(functionName(object) => object.id === id)) {}

It's also important to note that these are not quite the same thing. If you name a function, it will be hoisted to the top of the scope where a possible naming conflict could occur.

http://callbackhell.com is a good starting point.

I was on board with that website until I couldn't find a single semicolon. Then my eyeballs started twitching.

Love me some non-semicolon JavaScript.

Best (professional) practice is to use named functions, with lambdas reserved for one-offs which are so genuinely trivial that promoting them to named functions would do more to clutter your code than to streamline it.

It's telling that, while you do often see Javascript (not JScript, which is a Microsoft product and quite obsolete) developers pull some crazy nonsense with deeply nested lambdas, you almost never see Javascript developers do that in an environment that requires peer code review.

I used JScripters as a short-hand. I'm well aware of the stigma surrounding JScript :P

I've come across many JavaScript devs casually using and advising interns, etc to use anon functions to avoid cluttering code. It was almost preached as a design pattern. I was taken aback (as a back-end dev), but given the crazy stuff most JS devs did I just shrugged it is one of the platforms' quirks.

JScript isn't a shorthand for Javascript. It's a name for an ill-favored heterodox implementation of ECMAscript. Using it in the fashion you have here will therefore cause knowledgeable engineers to assume you must not know what you're talking about. Of course, you're welcome to continue doing so nonetheless.

Using lambdas in the fashion you describe is a dangerously poor practice that, over the long run, produces code that's much harder to maintain than it should be. That's why you don't see it in environments where changes have to pass peer review by experienced Javascript engineers before being accepted into the codebase. Unfortunately, such environments are far from prevalent and the JS dev world lacks rigor even by comparison with web development as a whole, so you see a lot of people abusing lambdas and encouraging others to do likewise. That doesn't make it any less a bad idea.

I totally agree, and my use of JScript(er) has been the first in the last few years, the last was when I was teaching a bunch of school kids about Web technologies. I don't intend to use it again and only arose out of laziness than ignorance.

Are there any other such bad practices in the JS dev world? I've only just considered moving onto Node as I've grown disillusioned with Rails and the concept of using a single language across the entire app & mobile (React Native, etc) sounds enticing.

Javascript is a fun language, but its laissez-faire nature and occasionally weird semantics lend themselves easily to the development of poor practices. Fortunately, there are several linting tools, such as jshint, which analyze code and complain about such things; I'd suggest taking a look at jshint's documentation, which is a veritable catalog of things it's easy to get wrong in JS and have come back later to bite you. I'd suggest even more strongly installing jshint and using it on your code; it's easy (just "npm install -g jshint" and you're off and running) and highly beneficial.

I'd also suggest that it's worth your while to take advantage of the JS dev community's excellent culture around automated testing; while that's a slightly more complicated topic, it is only slightly so, and I've found that a solid test suite, plus a coverage tool to ensure it doesn't leave anything out, is even better for improving and maintaining code quality than just a linter by itself -- jshint can catch stylistic and structural errors, but only unit tests will catch logic errors and regressions introduced by a new change. If you're (understandably) bewildered by the variety of testing tools available -- it seems like somebody comes out with a nifty new one every week or so -- then I'd suggest you can hardly go wrong with these:

* jshint, as already described

* Mocha, a flexible and reliable test framework

* Istanbul, a coverage report generator

* Grunt, a build tool whose plugin architecture simplifies integrating them

For an example of how they work together, you might take a look at https://github.com/aaron-em/same-encoder, where I use them all in support of a library providing functionality which I think is neat and probably no one else cares about. No doubt there are much better examples, but none come to mind quite so readily, and this codebase being as small as it is, it should be able to serve as a decent overview without being so big you'll get bogged down in it. In particular, /Gruntfile.js demonstrates how to tie all these disparate tools together for quick invocation with a single command like 'grunt validate', which I find makes it a lot more likely I will actually remember to run them; also take a look in /test/unit for an example of how Mocha tests are actually written.

Perhaps they aren't clearly reasoning about first class functions and think that is the way it needs to be written.

Some complaints I see about Python's weak lambda have a similar undercurrent.

this is my problem with Reactjs right now. I like it but debugging it is often difficult as hell. Thankfully my problems right now are small. With Hot module reloading I am constantly watching for signs of trouble. And can spot many quickly as they are introduced. But the stack trace is damn near foreign to me right now.

I am using those.

i use that + sourcemaps, and it seems pretty easy to debug.

using Chrome + React Dev Extensions

setting up sourcemaps to work properly is quite a pain though, most the browserify tooling via gulp or grunt does a pretty poor job of it.

Actually gulp has many great ways to work with sourcemaps, browserify, and babel. Google it and you should find a gist. I'm using through2.

I'm using typescript, which may impact my success, but really I don't get proper sourcemappings if I use uglify/browserify, or even the sourcemaps plugin.

I only get proper sourcemaps if I use the tsify plugin + setting browserify debug=true. adding anything else causes sourcemaps to go out-of sync. That's OK though, as I get good sourcemaps in development and can live without it when in production.

Yeah, I've always been suspicious of anon functions in any language, but jscripters take it beyond absurdity.

Surely you jest! Impossible-to-parse trees of anonymous function is a crucial part of the JavaScript revolution.

Signed: someone who occasionally has to maintain a massive CoffeeScript clusterfuck. If you want to be trendy that bad, buy a turtleneck. Don't use CoffeeScript.

I rarely look at function names. I almost exclusively rely on the stack trace's filenames and line numbers to find out where the code is that I need to look at.

even more abused now w/ es6 arrow shorthand

I wish named arrow functions made it into the spec.

A workaround for this to get the best of the two worlds:

    let func = () => {};

That seems to do it.

  let func = () => {};
  console.log(func.name); // func
and in stack traces:

  let func = () => foo();
results in:

  Uncaught ReferenceError: foo is not defined
  func @ test.js:1

Wow. I had no idea that would work.

All this time I've been using the full named function expressions over arrow functions just for the stack traces.

Still though, in situations where I care about the stack trace, I don't see myself writing:

  let doStuffCallback = () => {};

  doStuff(function doStuffCallback() {

Does that only work with something like babel? Because using the node 5.1 repl (and chrome console) you get:

    > let func = () => {}
    > func.name
    > function longfunc() {}
    > longfunc.name

The .name property doesn't seem to get set in Firefox either, but error stack traces do show the variable name as the function name. That's the important part IMO.

Have you tried it with a « var » instead as Chrome imposes certain restrictions on the new ES6 keywords?

I also checked the online Babel REPL and verified that it works fine.

var results in the same behavior in chrome (empty string for a function name)

It looks like babel does this little nicety for stack-trace readability.

I think Babel converts the code to ES5 and then execute the statements and therefore the function's name is printed to the console just like what you expect from good old ES5 code.

The most direct ES5 translation of

    let func = () => {}
would be

  var func = function() {}
But even in that case, func.name is going to be "".

    var func = function func() {};
That's what the Babel REPL spits when you input the « let » statement.

sadly, babel is taken for granted these days.

Which is the same as var func = function(){...}.bind(this); no es6 required.

OP is saying that `let func = () => {};` creates a named function where func.name equals "func" and "func" appears as part of the stack trace.

Your example is still an anonymous function just bound to a different context.

and more verbose as well which was the concern of person to whom I replied.

on the other hand, es6 classes encourage named methods ...

Interesting that they choose the brute force way. And that it works that good and maybe has less overhead than instrumenting the code.

Sorry if this is slightly off topic but can anyone tell me why I would not be able to access the netflix tech blog?

Every time I try to go to it I get a "Web page unavailable" despite sites like downforeveryoneorjustme.com telling me it is not down.

Not sure if its just my laptop as I am abroad at the moment in Italy and have no other computer to check the site.

"abroad at the moment in Italy"

Here is the basic checklist for something like this:

check your hosts file ping the website tracert the website clear dns cache check dns server up/down temporarily disable av/firewall

If all the above indicate normal operation but it still fails, it could be blocked anywhere upstream of you.

It's strange, nothing in the hosts file, any ping says it can't find the host http://techblog.netflix.com/, or even anyone of those check ip sites says the same. Asked some work colleagues here and in Ireland and it seems our company is blocking it, I guess through the firewall which unfortunately I can't disable.

I guess it is unfortunate though not too surprising that an employer would be blocking netflix.com.

My employer blocks this blog as well. Not sure if they just decided to block *.netflix.com or what.

It's hosted on Blogger. Do you have problems with any other Blogger sites?

If you are viewing from work they may block the domain for obvious reasons...

Restify, interesting.

Netflix is a large supporter and active contributor (perhaps the most active?) to Restify, so it really isn't that surprising when you think about it.

I thought the same thing.

Why not use New Relic?

My early impressions of New Relic were that they were a really fast and easy way to correlate all the default logging of the web application stack. I imagine Netflix would want something a little deeper and more customized then this, also deploying New Relic at scale would be costly and at the very least require an audit of potential performance implications.

always bad practice to be forced to debug in production, but I guess that's what you get in the "DevOps" model.

On the contrary: there are many bugs you will not find until software is in production. (That's not a reason not to test or roll out to staging environments first and the like, because you can continue to find new bugs that way. But despite all of our efforts, there remain bugs that only manifest for the first time in production.)

sure, but that is paramount of environments not being in sync, and inaccurate pre-production testing.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact