
Dropbox saved almost $75M over two years by moving out of AWS - shaklee3
https://www.geekwire.com/2018/dropbox-saved-almost-75-million-two-years-building-tech-infrastructure/
======
dzdt
AWS sells optionality. If you build your own data center, you are vulnerable
to uncertain needs in a lot of ways.

(1) your business scales at a different rate than you planned -- either faster
or slower are problems!

(2) you have traffic spikes, so you to over-provision. There is then a
tradeoff doing it yourself: do you pay for infrastructure you barely ever use,
or do you have reliability problems at peak traffic?

(3) your business plans shift or pivot

A big chunk of the Amazon price should be considered as providing flexibility
in the future. It isn't fair to compare prices backwards-looking: where you
know what were your actual needs and can compare what it would have cost to
meet those by AWS vs in house.

The valid comparison is forward looking: what will it cost to meet needs over
an uncertain variety of scenarios by AWS compared to in-house.

The corallary of this is, for a well-established business with predictable
needs, going in-house will probably be cheaper. But for a growing or changing
or inherently unpredictable business, the flexibility AWS sells makes more
sense!

~~~
chiph
> A big chunk of the Amazon price should be considered as providing
> flexibility in the future.

A recent place I worked at didn't understand this. They were going to the
cloud as a strategic move because they didn't want to run a data center any
more. Their computing needs were stable and predictable - no "Black Friday"
sales that needed triple the servers for a short part of the year. They were
going to end up paying far more for the total computing cost than if they had
just kept buying blade servers and VMWare licenses.

~~~
opportune
I worked somewhere where the resident DevOps/Sysadmin guy would rather
repeatedly write me emails about using an extra 10GB of disk for a data
science project than just buy more storage. And this was on an instance with
like half as much memory as disk available. There are some people in this
industry who just have zero business sense.

~~~
mrep
Then there are people on the opposite side of the spectrum who don't even look
at server costs. I have a task sitting in our backlog to optimize our instance
usage which could save us over $100,000 a year. It has been constantly
deprioritized in order work on feature development which is stupid because we
could hire another developer to do it in a few days and then keep him to
increase our feature development all while costing our company the same amount
of money.

Granted, I think this is more of a case of the prisoners dilemma [0] between
lower management and upper management (lower management doesn't want to work
on it because it doesn't produce features that make him look good but he also
does not want to propose the additional developer for it because then upper
management will just tell lower management to do it without the additional
developer).

[0]:
[https://en.wikipedia.org/wiki/Prisoner%27s_dilemma](https://en.wikipedia.org/wiki/Prisoner%27s_dilemma)

------
Someone1234
There's two good reasons to use a service like AWS:

\- You're too small for efficient economies of scale on your own equipment
(i.e. AWS is cheaper when considering total cost of ownership).

\- You need to scale rapidly to meet demand

The second one is largely a data issue, if you have enough historical data on
your customers, their habits, usage, and so on then scaling becomes
predictable and even when it isn't you could offload only part of your
infrastructure to a cloud vendor.

What's interesting is that several companies that I know which rely on
AWS/Azure/et al aren't on it for either of the two above stated "good"
reasons.

They are large businesses and do almost no automated scaling. They're on it
for what can only be described as internal political limitations, meaning that
they are on these services to remove the politics of technology
infrastructure, one less manager at the top, a shorter chain of
communications, an external party to blame when something does go wrong, and
issues like HR/benefits/etc for infrastructure employees is outside the scope.

In effect they view themselves as "not a technology company" so look to employ
the fewest technology employees as they can. Even in cases where technology is
paramount to their success. It is very interesting to watch, and I'm not even
claiming they're "wrong" to handle their infrastructure this way, just that it
is hard to quantify the exact reasoning for it.

~~~
mdasen
Those are definitely two good reasons, but I think there's another: you are
starting up and your core competency isn't object storage.

I'd argue that the core competency of Dropbox is its easy syncing. Dropbox
wanted to get that to market quickly. If they had spent the time building out
a data storage solution on their own, it would have meant months or years of
work before they had a reliable product. Paying AWS means giving Amazon some
premium, but it also means that you don't have to build out that item. It's
not only about economies of scale and rapid demand. It's also about time to
market.

I think it's a reasonable strategy to calculate out something along the lines
of "we can pay Amazon $3N to store our data or store it ourselves for $N.
However, it will take a year to build a reliable, distributed data store and
we don't even know if customers want our product yet. So, let's build it on
Amazon and if we get traction, we'll migrate."

S3 is a value-added service and creating your own S3 means sinking time. Even
though data storage is very _very_ near to Dropbox's core competency, it's
really the syncing that was the selling point of Dropbox. To get that syncing
product in front of customers as fast as possible, leveraging S3 made a lot of
sense. It gave them a much faster time to market.

As time went on, they had traction, and S3 costs mounted, it made sense for
them to start investing in their own data storage.

It's about figuring out what's important (the syncing is the product) and
figuring out what will help you go to market fast (S3) and figuring out how to
lower costs after you have traction (transitioning to in-house storage).

Yes, a lot of companies use cloud services when they don't need them. However,
Google Cloud's compute pricing is reasonably similar to DigitalOcean (with
sustained usage discounts) and from what I hear these companies will often
negotiate discounts. AWS can seem a bit pricy compared to alternatives, but
I'm guessing that Amazon offers just enough discounts to large customers that
they look at the cost of running their own stuff and the cost of migration and
Amazon doesn't look so bad.

Still, when you're trying to go to market, you don't want to be distracted
building pieces that customers don't care about when you can rent it from
Amazon for reasonable rates. You haven't even proven that someone wants your
product yet and your time is better spent on delivering what the customers
want rather than infrastructure that saves costs. As you mature as a company,
the calculus can change and Dropbox seems to have hit that transition quite
well.

~~~
candiodari
> Yes, a lot of companies use cloud services when they don't need them.
> However, Google Cloud's compute pricing is reasonably similar to
> DigitalOcean (with sustained usage discounts) and from what I hear these
> companies will often negotiate discounts. AWS can seem a bit pricy compared
> to alternatives, but I'm guessing that Amazon offers just enough discounts
> to large customers that they look at the cost of running their own stuff and
> the cost of migration and Amazon doesn't look so bad.

Of course, the dollar amounts dropbox saved are compared to those negotiated
prices.

In my experience it isn't so much the storage price itself, but the network
transfer that makes AWS absurdly expensive.

Most companies don't need something like S3. They can perfectly suffice with
one server, maybe using RAID-1, or just using backups. Data corruption mostly
happens through logical errors anyway and nothing in S3 will protect you from
that.

~~~
jon-wood
> Data corruption mostly happens through logical errors anyway and nothing in
> S3 will protect you from that.

S3 supports object versioning, which very much will protect you from anything
other than writing the wrong data for the entire history of an object.

~~~
candiodari
Versioning is optional and just becomes part of the file data. Write garbage
into that and it'll still take a lot to fix it.

Anyway, there's plenty of filesystems that have versioning built in as well.

Besides, I've been tasked with recovering the specific state of a database
where every version of every object was available, and it was only some 200k
or so records. That took me about 2 weeks. (mostly for writing code that could
find a consistent version of the whole thing)

------
ChuckMcM
This isn't surprising. At Blekko where I ran operations we did the math not
once but twice (once when we got our series C to show that it made sense, and
once when we got acquired by IBM to show why moving it into Softlayer was
going to be too expensive). If you can build reasonable management tools and
you need more than about 300 servers and a bunch of storage, cost wise its a
win to do it on your own.

The best win is to build out in a data center with a 10G pipe into Amazon's
network so that you can spin up AWS only on peaks or while you are waiting to
bring your own stuff up. That gives you the best of both worlds.

~~~
fizwhiz
So, rent the spike a la hybrid cloud?

~~~
vidarh
Yes. The beauty of it is that building in the capability makes hosting your
own even _more_ cost competitive. Because while you might have to plan for 2x
or even 3x your average to handle normal peaks if you don't have anywhere else
to send traffic, if you can spin up cloud instances to handle spikes you can
plan for far higher utilization rates for your own equipment.

~~~
danmaz74
Makes a lot of sense for the customer. I just wonder what Amazon could try to
do to prevent this model becoming prevalent.

~~~
sah2ed
Contrary to what you think, they activately encourage this style of usage.

AWS has a product called Direct Connect [0] to reduce bandwidth costs between
AWS and your infrastructure.

[0]
[https://aws.amazon.com/directconnect/](https://aws.amazon.com/directconnect/)

~~~
ChuckMcM
This is the local data center connection option I was referring to, the
Coresite data centers on Stender Way (in Sunnyvale) had this option.

There is a latency spike between local and Amazon infrastructure so it would
be critical to build your system so that putting this spike in the mix didn't
impact your own flow path.

I had thought a bit about how we might do that at Blekko and I would probably
shift crawling into the cloud (user's don't see that compute resource) and
move crawler hosts over to the frontend/index side. But I'm sure there would
have been a bunch of ways to slice it.

------
sah2ed
From reviewing the thread's discussion, it looks like businesses turn to
cloud-based infrastructure for a number of reasons:

1\. To outsource non-core activities to experts and reduce risk, for firms
that see IT as a cost center. A cost-cutting measure.

2\. To provide dynamic capacity for mature businesses that experience
anticipated workloads that are short-lived (seasonal or computational needs).
A cost saving measure.

3\. To provide dynamic capacity for new ventures growing in popularity i.e.
fluctuating capacity requirements. This saves on large upfront infrastructure
costs when long-term viability hasn't been established. A risk management
measure.

4\. To be described as "innovative" because peers are doing it, for firms that
see IT as a revenue center (in industries that view such investments as a
source of differentiation). A form of virtue signalling.

5\. To make the accounts look good to investors by accounting for it as OPEX
instead of CAPEX [0]. A seemingly irrational but valid reason. High OPEX
numbers are easier to justify and more importantly, can be pared down with
less friction than CAPEX, if things go south from intense competition for
instance. Another risk management measure.

[0]
[https://news.ycombinator.com/item?id=16458863](https://news.ycombinator.com/item?id=16458863)

------
jeffwilcox
Data centers aren't cheap, so unless you have the economies of scale to offer
R&D investment and stock-based compensation to your employees to build a
modern cloud DC, good luck with that... done right you can save operating
expenses, but it'll take a huge investment that would not scale for others.

~~~
pathorn
I strongly disagree. Datacenters are super cheap compared to EC2. (I'm not
talking building your own: start by leasing space from existing datacenters).
There are a surprising number of places where you can go and lease a rack or
ten or a whole room and be up and running in a couple of months.

I make the case that colocating pays off at just about any scale, assuming you
have $10k in the bank, have a use for at least 40 cores and are able to pay
upfront to handle anticipated scale.

Hurricane Electric has prices online of $300/mo for a rack. On AWS, a single
full c4 machine (36 threads) costs $1.591 per Hour x 24 x 30 = 1145/mo -- this
is more than the cost of running a whole rack with 40 machines. Decent
internet can be gotten for hundreds per month.

Ok, so how about buying your own machines? E5-2630 with 20 threads is $700 x 2
= $1400 + motherboard + disk + ssd brings it to several thousand, so it will
pay off in at most 6 months, and we're not even talking bandwidth or storage
costs. Depending on the application you could be looking at a payoff after 2-3
months.

Worried about installing or remote management? IPMI, iDRAC, etc included with
basically every server make this a piece of cake.

The only good case for cloud are if you may suddenly scale 10x and can't
predict it; don't have $10k in the bank; or don't have 1-2 months to order
machines and sign a contract for rack space.

~~~
viraptor
What you're missing in that comparison is an extra engineer (or two) who can
deal with power needs, networks hardware config, firmware updates, plans for
rolling hardware, stock of replacement drives, seamless base system updates,
dealing with the platform (virt or containers) itself and other things that
the managed cloud gives you included in price. Ah, and they need to be
available on call. Hardware may be cheap - people aren't.

~~~
throwaway2048
You need people to manage AWS instances too, ive never seen any evidence that
it actually takes less people at scale.

Sure for a few instances, its likely to be true because there is a certain
amount of fixed overhead for dedicated hardware, but it remains a relatively
low constant, but in reality there isnt much difference in terms of labour
between wrangling hundreds of AWS instances, and hundreds of servers, and the
servers will be many times cheaper to run.

------
bluedino
So it only takes how many _exabytes_ of storage for it to be cheaper than S3,
by only < $40M?

For all this talk about how expensive S3 would be for a filesystem, and how
poorly suited AWS is for this kind of thing, Dropbox has seemed to make it
work just fine.

------
_pdp_
Well we removed all our servers from AWS and replaced them with lambda
functions and dynamodb tables which resulted in 4.5 reduction in cost and
increased performance by multiple factors. I suppose it all depends on what
you are building and how you are building it. If you run servers I think it is
no secret that AWS is not the cheapest option around.

~~~
smt88
Did the switch to DynamoDB make a big cost difference? I've never really
thought about the cost of Dynamo vs. RDS as being huge, but honestly I don't
know.

For most people (including every commercial project I've ever worked on), the
time-saving and safety benefits of relational schemas was far greater than any
theoretical infrastructure savings.

~~~
guitarbill
> Dynamo vs. RDS

This is so dangerous even to type out :) In some, limited cases, Dynamo can
replace tables in an RDS, and completely outperform it, too. I'm a big fan,
but it's fundamentally different from an RDS, and you can get burned, badly.

Oversimplified, DynamoDB is a key-value store that supports somewhat complex
values. If you have large values, use S3 instead [0] - I think that's a good
way to think of it, a faster S3 for loads of small records (with a nicer
interface for partially reading and updating those records).

If you need to look up on anything but the primary key, be careful, costs can
get out of control by having to provision extra indicies. If your primary keys
aren't basically random, you'll run into scaling issues because of they way
DynamoDB partitions. If you need to look at all the data in the table,
DynamoDB probably isn't the right technology (it can, but scans absolutely
tank performance = $$$).

[0]
[https://docs.aws.amazon.com/amazondynamodb/latest/developerg...](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForItems.html#GuidelinesForItems.StoringInS3)

------
meritt
While S3 storage costs are fairly reasonable, request costs are unrelated to
file size, which makes it an absolutely terrible choice for filesystem-like
activities. I say this from experience: S3 PUT requests make up a very
significant portion of our AWS bill as we write a massive number of very small
files to S3.

Does anyone know of a middle-layer solution that automatically concatenates
files into larger chunks (similar to HDFS) prior to persisting to S3 and for
retrieval utilizes HTTP Range requests to fetch the individual file? I'm going
to build it if it doesn't exist. 100MB chunks would reduce our S3 PUT costs by
~1000x.

~~~
dvfjsdhgfv
Do yourself a favor and use a Hetzner Storage Box:
[https://www.hetzner.com/storage-box](https://www.hetzner.com/storage-box)

Even if you don't include any extra costs, just keeping 10 TB on Hetzner will
cost you 40€ max, whereas with S3 it will cost you _at least_ $230.

~~~
blfr
A single Hetzner box is not really comparable to S3 in terms of durability and
availability.

~~~
dvfjsdhgfv
My experience is that they go down every 2-3 years, usually the problem is
with the hard rive, occasionally with some other component. Under normal
circumstances it takes them up to one hour to replace the hard drive, and then
the array gets rebuilt for a few hours with the system being online. That
means something like 2 hours downtime every 2-3 years, that's quite acceptable
for my usage scenarios. In terms of money spent during that time, the
difference is enormous.

------
encoderer
Last year I wrote about the AWS spend of my SaaS business, Cronitor [1]. I
couldn't imagine building a service like this without modern cloud providers,
but it is no wonder to me why AWS generates all of Amazon's profit.

Essentially our migration over the years looks like:

1) Moving to EC2 from a VPS, The Ec2 instance with same specs is notably
slower and you need to add a second instance where one worked before.

2) Moving to a managed service like RDS after running the service on Ec2,
managed service with same specs is notably slower and you need a second
instance where one worked before.

In the end, it's worth it, in the RDS example you're getting millisecond
replication times, point-in-time database recovery, hot failover, etc. But
still, it would be great just once if you got the same performance from an RDS
2xl as you'd get running your own DB on a 2xl of your own.

[1] [https://blog.cronitor.io/the-aws-spend-of-a-saas-side-
busine...](https://blog.cronitor.io/the-aws-spend-of-a-saas-side-
business-30bd5dbd91b)

~~~
betterworldb
> it is no wonder to me why AWS generates all of Amazon's profit.

Going out on a limb here but I'm guessing that Amazon makes a good amount of
profit from their online retail store.

------
dmitriid
AWS (and other cloud provider) costs are non-uniform as you grow.

It goes something like this (from when you small to when you're huge):

\- very/quite cheap \- prohibitively expensive \- much cheaper than building
and maintaining own infra \- prohibitively expensive/more expensive than
building and maintaining own infra

So various companies will end up at various stages of this cycle. Dropbox is
big enough to be in the latest stage.

~~~
nolite
Sorry, what do you mean non-uniform? The costs are digressive with volume. If
you have non-linearly increasing costs with linear workload, then you have a
fundamental math problem with your architecture / business model

~~~
dmitriid
Non-uniform with respect to what you're willing to spend on different parts of
your business.

AWS is not a "once the sweet spot, always a sweet spot".

------
anubhavmishra
I feel after getting to a certain scale moving to your own data center is a
very smart choice. Its all about how well you can run your data center there
after. So, good for them for scoping it out and saving money.

~~~
romanovcode
If it becomes cheaper to pay Sysadmins then it is always a good choice to move
to your own.

------
NightlyDev
Using the cloud is a waste of money as long as you don't need to scale up and
down, or change the infrastructure frequently.

Bandwidth has always been insanely(!!) overpriced with most cloud providers.

~~~
styfle
What kind of business doesn’t need to scale up or down?

Maybe if there will only ever be one client connected 24/7 and that never
changes but I can’t think of a real product that would avoid taking on more
that one client.

------
brudgers
For Dropbox, it probably doesn't even matter that there was a cost savings to
moving from AWS. There is strategic value in not taking a critical dependency
on a potential competitor, and Amazon is an obvious potential competitor in
the for consumer and small business online file storage.

Using AWS meant that Amazon had IP logs for Dropbox's users. It had detailed
information about Dropbox's business velocity. It had the ability to shape the
customer experience of Dropbox's customers. In short, it made sense for
Dropbox to move off of AWS for the same reasons that it makes sense for
obvious Amazon competitors like Home Depot and Target not to build on AWS.
However, unlike Home Depot or Target, the other major cloud providers, Google
& Microsoft are also potential competitors in the file storage service market.

~~~
guitarbill
> Using AWS meant that Amazon had IP logs for Dropbox's users. It had detailed
> information about Dropbox's business velocity.

That doesn't seem fair to say, without proof. Why would they risk all the
other business they do, plus GovCloud, etc, to spy on a company? Are AWS and
Dropbox even competitors? I don't think that assertion holds at all.

------
z3t4
The difference between self hosting and running in "the cloud" is that when
there's an error in "the cloud", you can't do anything about it ... But full
stack debugging can even make a greybeard cry.

~~~
closeparen
“The cloud” makes it easier to build an architecture with multiple failure
domains. You don’t want to be debugging in an emergency, you want to be
hitting the failover button.

~~~
z3t4
I think hiring dedicated servers or VPS is a nice middle ground between self
hosting and all out "cloud" _cough_ lock-in. The cloud services do the
failover and auto scaling for you, but often when something goes down it turns
out they didn't even have any backups and all your data will be lost unless
you actually backed it up yourself.

------
hayd
Do we know how much they were spending on AWS before? Not that $75m isn't a
lot but it would be interesting as a %!

~~~
mdasen
> which reduced spending on “our third-party datacenter service provider” by
> $92.5 million offset by increased expenses of $53 million for its own data
> centers.

They reduced spending on AWS by $92.5M, but still store 10% of data there so
~$103M?

Looking at the $53M cost for their data center vs the $92.5M decrease to the
"third-party datacenter", they saved around 43% moving to their own data
centers.

------
oneplane
I wonder what Dropbox is using. Of course they mention 'custom' software and
possibly hardware, but I'm pretty sure it'll be some sort of Intel-based
setup, maybe OCP, and Linux. The might be some hypervisory things going on,
possibly Xen, maybe KVM. Then there will be some sort of firmware/OOB
management as well, and perhaps some non-standard connectivity.

Those are mostly given. What runs on top of it, that's what interests me. Some
orchestration/ private cloud software, some container platform, and perhaps a
storage platform, which/what are those for Dropbox? Would it be standard
OpenStack, Kubernetes and the true in-hous stuff: dropbox object storage?

~~~
maccam94
I don't think there's much public info about the compute infrastructure, but
here's a blog post about the object storage system:
[https://blogs.dropbox.com/tech/2016/05/inside-the-magic-
pock...](https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/)

------
z1mm32m4n
Do they still use AWS for compute, with their own data center just for
storage? Or do they have their own infrastructure for compute?

If I remember correctly, they noticed they could dust costs significantly in
storage compared to S3 because the vast majority of files in your Dropbox
folder just sit there and hardly need to be read or written again (photos,
etc.).

If they’re not using AWS for compute, it’d be interesting to see what sort of
similar reasoning they have for why the costs are cheaper in house.

~~~
vidarh
> If I remember correctly, they noticed they could dust costs significantly in
> storage compared to S3 because the vast majority of files in your Dropbox
> folder just sit there and hardly need to be read or written again (photos,
> etc.).

Bandwidth costs alone for S3 are high enough that I've set up storage systems
for clients that used S3 for durability but put it behind huge caching proxies
that held a fairly substantial proportion of their total data, where the
reduction in S3 storage paid for the cache setup several times over.

Storage costs are harder to cut, but yes, S3 is typically expensive there too,
but it's also riskier to do your own because of the risk of data loss. At
Dropbox's scale it'll be worth it, but letting Amazon worry about durability
is one of the few cases where I'll happily recommend AWS despite the high
costs.

------
tedivm
This doesn't surprise me at all, considering storage is what Dropbox does and
AWS has not really been competing on storage costs lately (as much as it's
great when instances get cheaper, EBS has been 10 cents a gb/month for awhile
now which is really problematic considering most of the new instance types are
EBS backed).

------
zenonu
What would be the cost difference moving to Google Cloud or other cloud
provider? The reason why I'm asking is that the move itself requires a re-
consideration of all production configuration. The indicated savings assumes
that Dropbox was using Amazon 100% efficiently to begin with.

------
WheelsAtLarge
Yes, it's cheaper but that's because Dropbox needs lots and lots of
infrastructure and they can hire people to run it. At some point, if you have
the capacity, it cheaper to build than to outsource. Dropbox found that point.

Smaller companies don't have the option to hire a whole IT team.

------
pankajdoharey
The article lacks a detailed analysis of how it saved so much money, there
must be other stories like this when Cloud hosting doesnt make sense for
really high volume data operations. The cost savings come purely out of large
volume or is there something else to it?

------
m3kw9
The cost of switching including? What’s the percentage of saving?

------
patsplat
Didn't Zynga report the same savings prior to collapsing?

Is reporting massive savings on an infrastructure project a signal of
softening demand and a weak product pipeline?

~~~
late2part
Wot Mate? You think when a company announces they are saving money on costs,
lowering COGS and increasing margin by cutting costs by investing in a lower
cost, higher performance solution, it's a signal of softening demand and weak
product pipeline?

There's a kernel of truth in what you're saying, perhaps. Any growing company
has to choose between devoting resources (engineers, cash, executive overhead)
between different projects.

Alot of the time companies expand by capturing market share through adding
features and expanding their TAM.

But it's foolish to suggest that lowering COGS and increasing margin means a
company is in trouble.

Let's say your company is now making $1B/year (Dropbox) and you can choose
between adding 3% more revenue or lowering your costs by 7%.

In most cases, for your IPO, you'll get a higher aggregate shareholder value
by cutting the costs and increasing your margin.

~~~
greenyoda
Also, once a company is no longer being funded by venture capital and has to
finance its own growth out of its profits, those millions it saves by cutting
infrastructure costs can be used to hire more staff, rent more office space,
and all the other things required for growth.

------
njwi332
Notably, the cost of doing so was over 50 million. So it's perhaps not
realistic for majority of companies using AWS that are much smaller than
Dropbox.

------
justicezyx
This is a success story for both aws and Dropbox.

------
mkagenius
It’s understandable that lot of companies stick with AWS only because of
inertia (read laziness) when they will be better off it.

------
mankash666
This is actually great P.R for AWS. Dropbox relied on AWS primarily for a
decade, focusing on features and growth. Only when cost cutting became
relevant to valuation, i.e. pre-IPO, they decided to invest in their own
infrastructure.

Put another way, use the cloud unless your valuation depends on cutting
infrastructure operation costs

------
kesor
I'd rather see a figure of how much more money they MADE than how much they
saved by doing something that is not directly related to making more money.

