
Why We Abandoned Crowdsourcing - todsul
https://flightfox.com/community/why-we-abandoned-crowdsourcing
======
bretthopper
Your pricing/trips page
([https://flightfox.com/trips](https://flightfox.com/trips)) is really lacking
some direction and information. I can easily see new Flightfox users being
really confused. I was slightly confused and I've used Flightfox twice before.

I would suggest carrying over the 3 steps graphic/info from the main page to
the top of the /trips page. But more importantly, you really need a better
breakdown of exactly what those prices are. Even the "Guaranteed" keyword
seems out of place and confusing. What exactly is the guarantee? That you'll
charge me $149 no matter what?

Your create trips page is also lacking a bit of focus to me. Would be useful
to have some more tangible examples of trips and example trip descriptions.
I'm sure if you provided a simple example/template, you'd reduce some initial
friction on first contact with an expert.

~~~
todsul
Hi brett, we're working madly on this now. We built the new version from
scratch (very little in common), so we're a few-week-old startup all over
again.

We can imagine these products being their own page with details, testimonials,
examples, etc. Something like [http://elto.com](http://elto.com), whereas our
current page is more like the 99-designs one.

Just wanted to mention, "clarifying" our products/pricing is priority #1.

~~~
prawn
Should that be something you work on before announcing the change? Seems quite
critical to me.

~~~
todsul
Hi prawn, rightly or wrongly, we live by the "launch fast" maxim. Sure we had
some test data, but a change this big was always going to be a huge leap of
faith. So we needed to get it out to confirm previous findings before worrying
too much about conversions.

On that note, the risk wasn't just a different format but everything that goes
with real-time communications. Not just web sockets or whatever tech, but a
real-time human marketplace. The previous format afforded us a lot of luxuries
with regard to resource planning and "load balancing". Now with real-time,
it's quite tricky. I suspect this is a challenge for Uber too.

------
nlh
I've been a huge FlightFox fan and used them for several big (and complicated)
trips - they've gotten me spectacular deals each time. This was all in the
competitive phase, btw - haven't tried the new format yet.

One thing I just saw -- they dramatically raised the price if you're doing a
"miles flight" \-- it's now $99. I suppose I understand the thought process --
if you're not spending $$ on the flight, you're more willing to spend $$ on
the consultation?

But honestly it makes me less-likely to use them. I use miles on some flights
precisely because I DON'T want to spend any cash. Adding $100 in cash to the
cost of the trip feels counter-productive.

Anyway, still a fan of the service, even if I've been priced out of using it
:/

~~~
todsul
Thanks nlh. First, thanks for the feedback on the pricing. We're talking about
the miles product right now and will update it very soon. The high price was
to account for most miles trips being fairly complex, but we'll segment a
little more and offer you much better pricing.

UPDATE: just spoke to the experts and made it $69 for now. We'll change the
pricing to be +$19 to use miles, but that'll take a little time. Reason it's
more is that the experts have a few tricks that take considerable time.

Great to hear you received value with the competitive format. We still have
the top 0.3% of experts from the previous system, they just get paid more and
can spend a lot more time on your trip now. But we'll keep analyzing and make
sure it's working well.

~~~
nlh
Sounds totally reasonable. +$19 to use miles feels completely fair. I'm back
in baby! ;)

(but seriously, thanks for the update. I know pricing is an imperfect science
and takes time to optimize. Keep on kicking butt.)

------
encoderer
Congratulations. You invented travel agents.

Modernized them, I suppose.

Not exactly an exciting and thrilling turn of events but it seems the metrics
all point toward this as a better solution so I suppose "exciting" is
overrated.

~~~
todsul
Hi encoderer, we were saying the same thing before testing this format.

However, unlike travel agents, our guys are genuine flight experts. If you
travel wide, fly a lot, or use miles, you'll understand why this matters.

Behind the scenes, it's all about commissions. Agents work on commissions and
most flights don't pay commissions. So it makes little sense for agents to
spend hours on your flights.

This manifests in the wide skill gap between (most) agents and our experts.
Most agents won't save you a cent, they'll just be there for good service. But
our guys get insane deals you won't find anywhere else. Often they involve
shorter flights with fewer stops too.

~~~
eddmc
A lot of travel agents are genuine flight experts. If you don't believe that
then you haven't spent much time with good travel agents.

Also, it's not about the commissions. That went out with the '90's

Source: I've worked in this field for 13 years now and sell a SAAS product to
travel agents.

~~~
jeremyt
Okay then, how do I find a travel agent that will book me the cheapest flight
to London for 50 bucks?

~~~
njr123
You don't need a travel agent for that, just use skyscanner or something. The
case where a TA adds value is something like: 'I am going from Australia to
Canada for work, and I need to visit a bunch of small towns that are only
served by regional airlines, and I need to be there on these exact dates.'

------
RyanZAG
Honestly the pricing is the real unnerving part here and it's definitely going
to be a deal breaker for much of your audience.

Personally I'd prefer something like this: I enter my current flight details
and the price I was going to pay (your experts should be able to verify this
price as correct pretty quickly?), and then your experts would give me a
better flight plan and I'd pay you 10% of the difference or something, and
nothing if your experts don't find a better plan. This way I'd always feel
like I'm 'winning' by using your service and not taking an expensive shot in
the dark.

~~~
todsul
Hi Ryan, segmenting is our greatest challenge and (planned improvements aside)
your experience was intended. It's the old startup adage: better to have 1,000
customers who love you than 1,000,000 who don't care.

What you're suggesting is what we call "gambling against Kayak". That's not
our target market because it's not the market we can help most. We often say
internally, it's not whether we can _possibly_ help a customer, but whether we
can _likely_ help them.

Our idea to create "products" was to help segment customers into those we can
help most. For example, complex trips, flying premium, using frequent flyer
miles, traveling with pets, euro-city hops, etc. These are the people we can
"likely" make happy and will help us grow to then target a wider audience.

------
aridiculous
Good to see that crowdsourcing, one of the darlings of web commerce, is
getting exposed for what it is: Cheap and crappy. The web capitalists loved
the idea of labor competing against itself, maximizing value for the customer
and minimizing labor costs. I.e. free work by 90% of the labor.

Good luck with the new system! Although you're doing it for practical reasons,
I still applaud you moving away from this practice.

~~~
todsul
Hey aridiculous, we really wanted the economics of the competitive format to
be fair. We got to the point where our experienced experts were winning
30-50%, so for them it wasn't so bad. Customers sometimes complained about
"only" getting 2 or 3 experts, but we quickly realized it was better for
everyone, including the customer.

It's a bit like being a software consultant. Most clients don't pay enough for
you to do your absolute best work. That's what happened in the contest format.
The best experts didn't make enough to do their best work. With the current
format, we see them really sinking their teeth into big trips and having fun.

------
mgkimsal
Wish I'd known about this change a few weeks ago - would have tried the new
format - wife just booked a flight. I used flightfox once before, and while
the idea was novel, the execution was... a bit weird.

Something that wasn't addressed in this article write up was how customers
felt. I felt _bad_ for having 4 or 5 people doing work on my behalf, giving
them feedback ("change these days, find it cheaper", etc) then only 'awarding'
one of them. I did not enjoy that experience - it made me feel like I was
abusing their time.

Additionally, I never quite got the price my FF person found me - perhaps I
waited too long, or something else happened. I think he'd sent me a screenshot
showing that _he_ was being shown that price as available, but when I went to
the airline (delta, iirc) I wasn't getting that price. I was still getting a
better price than I'd been aware of before, but not quite as good as what the
guide claimed to have found.

Again, the contest format was not just bad for the reasons listed - users
(well, at least this one) felt uncomfortable using it.

Good luck with the new format!

------
bhuga
I used the new 'consultant' model last week and loved it. I really liked that
the consultant could help me get what I wanted, not what I asked for ("I'm
cold, put me in South America." "Buenos Aires is expensive. How do you feel
about La Paz or Lima?").

I hope they standardize on it. The rest of the site seems to reflect the old
models. For just one consultant, why do I need 27 different pricing options
for 'kinds of flights'? I don't know what I want. Just charge me one price and
be done with it. And the ratings system needs a field akin to 'money saved vs
kayak' in addition to the 1-10 scale, because I'm no longer sure I got the
lowest price as opposed to a consultation.

But I loved it all the same and will be back. Flightfox is doing a great job.

------
chx
Disclaimer: I was a Flightfox expert. It was one of the best computer games I
ever played and it occasionally paid me to play. (In a similar vein, Drupal
development is the best MMO I could imagine -- the graphics surely leaves
something to be desired but the social aspect is literally life changing and
the quests keep you in flow.)

I am not sure whether flight search in a crowdsource format could or should
arise again but Flightfox made the (deliberate?) blunder of revealing all
expert email addresses to the other experts in an email announcing big changes
coming on January 14. If this would be a bigger industry, that'd been quite
some news... I know at least one person who is thinking on continuing --
thanks Flightfox for keeping the community alive.

~~~
todsul
Hey CHX, apologies for the blunder with the emails. It was an honest mistake
of not BCC'ing and certainly not deliberate. As you saw, a handful of people
replied to the whole group and talked about cloning our product, which wasn't
really in our interests. We don't get worried about that stuff, we were just a
little sad we messed up for you guys.

~~~
chx
I honestly say I liked this. I would've agreed to disclosing my address to the
others to keep the experts together.

------
esteb_li
I think crowdsourcing's issue is with the experience. For a customer,
crowdsourcing requires more effort than plan google searching for example. You
need to submit your issue, come back, edit/comment. Bad experience is much
more frequent than good ones. And a neutral experience is closer to a bad
experience which might result in the user not going through crowdsourcing
again. Good luck with the new direction

~~~
todsul
I think this is spot on. We used to underplay the experience issue, but then
we realized a poor user experience often hinders access to the best savings.

What made this worse was the animosity between customers and experts. It was
expected between competing experts, but the concept of the "award" (and by
necessity "switching award") caused tension in all directions. We still had
many great results, but not to the level needed to build a great business. The
difference with the current system is night and day. Everyone is on the same
side; it's even a little erie since we're so used to the previous animosity.

------
bartkappenburg
Very interesting read! Especially when we are almost ready (as in next week)
to launch our own crowdsourced travel advice platform on
[http://www.voyando.com](http://www.voyando.com).

Let me share what we found during our test period.

We have no 'application process' for experts (flightfox has) which makes the
'crowd' somewhat smaller. In our experience this removal of the process
increases the competition and the chances of a better outcome. Take
stackoverflow for example: limiting it only to top-programmers would harm the
concept. (i see SO as a crowdsourced programming advice platform :-))

Our platform supports more than just flights. We have hotels and complete
trips as well. The philosophy of our platform is more on the convenient side
of things instead of the 'hacker' mentality that FF has (no pun intended, FF
is great for HN crowd I think). Our service could be used by your mom and dad.

Given the fact that we have no selection procedure of experts, a broader range
of travel types and aiming to be a convenience service, we are convinced, also
looking at our test phase, that crowdsourcing IS the right way to go for us.
But hey, only time will tell and maybe you'll see a blogpost about our pivot
in 18 months as well!

Congrats to FF for this pivot and the really great and open blogpost.

------
jdp23
Very interesting learnings, and well-presented too!

I have to say, I don't see this as abandoning crowdsourcing -- the "crowd" of
experts is continuing to work together. To me it looks more like investigating
various approaches to getting good results from the crowd, and discovering a
better alternative for the particular problem. Terminology aside, it's very
interesting to think about what other situations this pattern could work in.

------
emhart
Great write up. It left me wondering - do you think you could have started
here? Or did the competitive format bring you the talented experts & public
attention you needed to make consultation work?

Congrats on the re-launch.

~~~
todsul
hi emhart, interesting question. The contest format was certainly novel and
helped garner a lot of early attention from press. So there's that. But I
think you may be right that the contests helped us a) find experts, but also
b) learn what was possible.

That somewhat fits with the idea that crowdsourcing is best when you have less
info. Once you have more info, then (at least we think) it's best to work with
fewer specialists. Although this is mostly referring to the customer, maybe we
can apply the thinking to our product lifecycle too.

~~~
emhart
Thanks for the thoughtful reply.

------
OoTheNigerian
So I am traveling to 2 continents in the next month (Including the US). Most
likely a minimum of 3 cities.

I have always wanted to use FlightFox but what is holding me back is the
uncertainty of what I will get AFTER I have committed to pay.

What happens if I want to change my flight? Cancel? With the existing travel
agent model, I am certain stuff like this will be taken care of.

When is the relationship over after the purchase or after the journey.

PS: Like Brett said, your pricing is really confusing. Is

London ---> New York international or Multi city?

How about,

London---> New York ----> San Francisco ----> Accra International or Multi
City or both?

Do i pay different prices?

I plan to book my flight over the next few days. I am really tempted to prove
I am an early adopter :)

~~~
todsul
Hi nigerian, apologies for the confusing pricing, we're working on that now.
Anything more than one destination is multi-city. So your first example is
just international, but the second eg is multi. Your second itinerary should
be perfect for our guys.

------
cocoflunchy
Has anyone tried FlightFox? I wonder if it's worth using for simple flights
like NYC-Paris or NYC-SF.

~~~
kingnight
I used it a 1.5 years ago, during the competitive phase, for a two destination
trip on a different continent. I came away not entirely thrilled by the
results of the experts. After paying the fee, what they found was slightly
cheaper than what I was able to do independently, but had a funkier itinerary
that didn't justify the slight price reduction (that may have been actually
offset by the cost of using the service).

That said, I did receive tremendous feedback from Todd. Reading this post has
me reassured that he and his team are on the right track.

My wishes—

I wish I had the money and time to seek out such types of fares/trips more
often so that I could try the service again. I may have one such trip coming
up but I still feel reluctant paying a fee up front for what might not turn
into anything.

I wish FlightFox/someone would offer services to deal with the airlines'
kerfuffle on multi-destination, multi-carrier flights. I swear, if you have to
deal with any inconveniences in a situation like this, it is 100% "Hi, this is
ABC Airlines, you'll have to call XYZ Airlines as this leg of the trip is
under their control" and vice-versa.

~~~
todsul
Hi kingnight, our experts will refund you if they can't help or you're not
happy.

In the old system, there was always a lot of tension when we'd refund
customers, because the experts would have already spent a lot of time. But in
the new system, we made it so we (admin) can't actually refund you, only the
experts can. This has worked wonders because they now use refunding as a form
of segmentation. Often they'll refund after 30 seconds if it's clear they
can't help, which is a far cry from the amount of effort they'd previous
expend before a refund.

~~~
kingnight
That's interesting about the tension — I didn't really think about it that
way, and the new system does indeed sound much better for all parties
involved.

------
fibbery
I tried flightfox when I was planning a vacation last year. I had thought it
was more like taskrabbit where people would compete for the lowest price and
was confused/disappointed when it told me I had to pick someone. Unfortunately
none of the results seemed particularly promising.

In the end, either I should be working for flightfox or the service isn't as
good as it should be, because the "expert" results were worse than what I
found on my own... and I had the impression that the lack of competition was
part of it.

~~~
todsul
hi fibbery, we made this change exactly for this reason: so customers would
get great results more of the time. With the new format, not only do you get
one of the best experts, but that expert can spend more time just on you. That
last part is key.

This idea that "one person digging deeper" is better than "more people
competing" first came to us at an in-office hackathon (flight hacking, not
software hacking). We noticed with more "iterations", experts could hack even
the most mundane trips. In this particular example, they got NYC<->Berlin
(round-trip) for about $400 with rigid dates. The rigid dates part is key
because it simulates the real world (we've seen < $300 with flex).

Once we worked closer with some of our experts, we realized the top ones are
miles more analytical. They either write their own software or compile huge
spreadsheets of their findings.

So this change of format was to put you in touch with one of them, not with
someone who does this on the side. We wish we could have created a new income
stream for the average person, but to ensure customer value, we had to go this
way.

------
Aloha
I don't understand the pricing.

Is it say 29 dollars to consult for me, then another unknown dollars for the
fare, or is that 29 dollars all inclusive or per leg?

~~~
todsul
Hi Aloha, thanks for the feedback. We'll try to make this more obvious asap.

To explain, our pricing is "from" $29. That's for the simplest trip. It goes
up with complexity because the experts must work harder. This doesn't include
the fare, this is the fee for consulting.

As you can imagine, this doesn't make sense for short flights for say $100.
But our experts really start to work wonders on trips over $300-500. For
multi-continent or business class, the savings often get into the $1000s.

If you click on each product, you'll see tweets from real customers talking
about different results.

~~~
Aloha
I'd suggest giving ranges for total out the door pricing based on mileage or
some other formula. I know this is a hard thing to guesstimate because of how
complex airfare pricing is.

I personally love the idea of paying for a travel consultant to take care of
my trip for me - and more importantly, be there to take care of me if
something goes awry - its like a modern day travel agent, but with more
transparency in pricing. Showing what someone would have paid for said trip
thru kayak or whatever would also show the added value - if you can figure out
a way to give semi accurate numbers of the savings on the front side based on
mileage and actual savings when the trip consultant had set up the trip for
you would go a long way to getting return customers.

Offering travel insurance that actually does something (versus what most of
the common carriers offer) might worthwhile, especially for complex fare
hacking trips split across several carriers.

The biggest issue with the travel industry is a lack of transparency - none of
the existing online services really help with that.

------
slaven
I just wanted to say how slick their service is, I just used it earlier today
and was paired with a very competent agent. I'm now waiting for the day
Flightfox can take care of the whole thing, including booking a flight. There
is a definite opportunity there!

------
katowulf
Great, insightful article for techies; very interesting insight into your
business strategy and R&D. Thanks for sharing!

------
coldcode
How does FlightFox makes money? Given that both the expert and FF needs a cut,
how does this scale?

~~~
jusben1369
how many airline tickets are sold a day?

~~~
coldcode
Airlines pay us at most $10 per ticket no matter the price. Clearly this isn't
what make FF money because it's not enough to scale if the profit is shared.

------
DorintheFlora
It is a pleasure to see this analysis of the business strategy, what worked,
and why. Tweeted. :-)

------
danbmil99
Sounds like we're orbiting back to the travel agent model...

