

How we used Node.js to build real-time award flight search - nmeyer
http://blog.milewise.com/post/8792373531/how-we-used-node

======
unshift
they'd do much better with twisted (python) than tornado. using Deferreds is
nothing like opening keys with a beer bottle -- it makes a lot of sense.

and as a former ITA Software employee, i can scoff and say they aren't
actually doing anything resembling flight search. they're just scraping
websites and APIs. node.js might be fine for that, but so is anything else. at
that level it's more a matter of preference than capability.

~~~
IsaacSchlueter
There are deferreds in node as well.

~~~
sausagefeet
They aren't part of the core API like they are in Twisted, is this correct?

~~~
IsaacSchlueter
No. Node is a lower-level API than Twisted.

The "fibers" utility is quite popular. There are also several different
promise libraries, and a few code-translation approaches that simulate
coroutines.

They all build on top of the very simple EventEmitter and continuation-
passing-style ("callback") style that the node-core API uses.

As a JavaScript luddite, I prefer passing functions around and listening to
events :)

------
justinsb
So does milewise scrape the airline sites for award travel? Does it also
integrate with ITA or similar?

I ask because (1) the airline websites tend to be _really_ slow and (2) they
tend not to find the cheapest/best flights - an ITA search can often beat
them.

~~~
nmeyer
We actually do both, and then spent a few cycles de-duping etc.

1) Part of our reason for using Node+Socket.IO is to deal with the problem of
having a bunch of different sources returning data over a wide spread of time.

2) You'd be surprised about the airline sites. While they don't necessarily
have the widest variety of good fares, airline will often reserve their
cheapest fares for their own sites (called Last Seat Availability).

~~~
justinsb
Thanks ... looks like it's time to sign up for MileWise!

------
blakeperdue
Not related to Node, but your homepage is really well designed. Attractive,
simple and clean. Nice.

~~~
nmeyer
Sacha Greif did our homepage. (<http://www.sachagreif.com/>)

He's awesome, super talented and incredibly fun to work with.

~~~
sgdesign
Thanks Nick! You guys are great to work to as well, glad you appreciate the
result :)

------
ftartaggia
Could you describe the difficulties you encountered with Tornado and
asynchronous Python code?

~~~
vinayp
Don't want to turn this into a Node.js/Python flame war (we love Python as
well here at MileWise), but regarding asynchronous Python code:

It's more just an annoyance of the language. Specifically, anonymous functions
are must easier in JavaScript, so it was just easier to write read and write
maintainable code.

In Python, you are limited to only passing in simple lambda functions
(although deferreds do get around this issue somewhat). It's not a show
stopper but when combined with all the other advantages of Node.js, it just
made the decision to switch a lot easier.

We also ran into a lot of issues related to poor garbage collection in Python
and were using up a lot of memory. This was not related to dangling references
but rather Python's permanent allocation of Integer and Float objects.

Finally, we just saw better performance out of V8 than Python (although we
never tried anything else besides vanilla Python 2.6).

For our specific problem, Node.js worked well. It's not always the right
solution. We will write a more detailed blog post soon with specifics of how
we used Node.

~~~
ljlolel
>> It's more just an annoyance of the language. Specifically, anonymous
functions are must easier in JavaScript, so it was just easier to write read
and write maintainable code.

That's pretty specious. You can easily write a function defined inside a
function that you can pass by just naming it. "Anonymous" functions are just
syntactic sugar. Furthermore, writing a new python named function is easier
(fewer chars) to write than a javascript anonymous function, so the "sugar"
gained is moot.

Finally, as you build more and bigger systems, you realize that those multi-
line anonymous functions you had you actually want to be named (for
documentation) and tested. So you'll end up de-anonymizing the complex (multi-
line) ones anyway.

As for speed, why not compare a jit (v8) to a jit (pypy) ?

~~~
icey
This is getting ridiculous. These guys were busy launching a product, not
trying to write a balanced comparison of the pros and cons of every evented
framework out there.

Node worked better for them, what's the problem?

------
chefsurfing
This is a killer product and very useful for people who travel often. Way to
go! Also, thanks for sharing your code. You all seem like a class act.

------
jscheel
Gah, right after I finish a site requiring me to write my own node soap
interface, you guys come and release yours ;) Thanks for contributing back!

~~~
vinayp
:) Would still love to see your implementation though. Ours only covers a
subset of the spec so lots of room for improvement...

------
apaprocki
One comment: I typed in an email/password and right away it is asking me for
my airline credentials but it isn't explaining what it is that the site wants
to do with them or if it will store my info and collect information from now
on. I'm curious about the site, but I thought twice about entering my info and
skipped it. Just saying you might want to explain what it is you are doing to
make people more comfortable.

EDIT: This information does seem to be covered in the FAQ, but you don't see
the link to the FAQ anywhere when you are trying to capture user accounts on
initial signup: <https://secured.milewise.com/faq>

~~~
nmeyer
Great feedback - we weren't sure whether at this point people would have a
clear idea from the homepage of what we were doing w/ the info and they'd just
want to get on with things, or if they'd want more contextual explanation.

This is definitely an area for metrics and user testing, but probably better
for us to err on the side of caution =)

~~~
apaprocki
I mean, I'd like to think you were going to provide me some great service..
but then that means I could register a domain name and put up a slick looking
homepage and start collecting passwords :)

Just a side note, but do you use some other collection API such as Yodlee or
are you maintaining your own data scrapers for all the airline sites?

~~~
nmeyer
Aw shucks you've uncovered our master plan.

We have our own account scrapers augmented with a partner API from AwardWallet
for the more esoteric programs (CVS Rewards anyone?)

~~~
apaprocki
Interesting... I'd like to talk to you more offline. If you want to chat,
shoot me an email, andrew at ishiboo dot com.

------
dcaylor
Great use case for Node.js.

~~~
sausagefeet
Great use case for Twisted, Erlang, Ocaml/Lwt, Lua, Python/Gevent too.

~~~
IsaacSchlueter
Great point! I bet that now thanks to your razor wit, dcaylor will think twice
before saying something nice about node without also saying something nice
about every other programming platform.

~~~
sausagefeet
Node gets a lot of attention despite not doing much of anything new, sometimes
it can be helpful to point this out. I was not rude or insulting about it,
unlike others.

~~~
dcaylor
I laughed at both your initial response and IsaacSchlueter's comment. Valid
points from both.

It would have been a little odd for me to say "Nice use case for Python"
considering the original post was about Node, and specifically said they tried
writing it in Python but "writing asynchronous code in Python is like using
your keys to open a beer bottle." That also made me laugh.

In fact, the original article specifically talked about why Node.js was a good
choice for this use case. I agree. Not that you couldn't do it in other ways.
Erlang would also be a good choice here, I'm sure. There is always more than
one way to do most anything. Still, and in spite of the fact that Node might
be getting too much hype, it is an excellent tool for this type of async need.

~~~
sausagefeet
Well, I hope you learned your lesson.

------
fnl
Nicely done! Just, the "How" in the title is misleading... ;)

------
retube
I have to sign up to search? Is this in production yet?

------
sausagefeet
Can you go into more detail about how js on front and back helped? What
percent of your codebase is actually shared? What type of functions are they?
Many people claim that it's a big win while others call shenanigans but I
haven't heard empirical data from anyone.

------
danecjensen
hey nmeyer im interested in building some similar stuff could you email me at
danecjensen@gmail.com so we can talk more. Thanks.

~~~
nmeyer
Awesome. Emailed!

