
QuickBooks Cloud Hosting Firm iNSYNQ Hit in Ransomware Attack - DemiGuru
https://krebsonsecurity.com/2019/07/quickbooks-cloud-hosting-firm-insynq-hit-in-ransomware-attack/
======
pgrote
Textbook on how not to handle the issue. Deleted Twitter, sanitized Facebook
and more than 2 days to admit the issue.

There is discussion on twitter that the company said the backups were on the
same network as the data. Hopefully there is an offsite backup available.

[https://twitter.com/ConleyU/status/1151862278909825024](https://twitter.com/ConleyU/status/1151862278909825024)

[https://twitter.com/MRasconCPA/status/1151894366291734533](https://twitter.com/MRasconCPA/status/1151894366291734533)

[https://twitter.com/hockeygirlPDX/status/1151945932935585792](https://twitter.com/hockeygirlPDX/status/1151945932935585792)

Ouch. This is the sort of stuff that can kill a company.

Does Quickbooks with the cloud option offer local backups?

~~~
jacquesm
It is interesting how often I see companies that refuse to make back-ups of
their cloud hosted data on the assumption that this is now someone else's
problem. I also have a - recent - case of a very large manufacturer of storage
solutions that managed to fuck up a restore of a raid array to the point that
a whole pile of companies lost their data.

Backups are so simple, and yet the only times people seem to realize their
true value is when they don't have any.

~~~
PopeDotNinja
> Backups are so simple

It gets more interesting when you're trying to backup a distributed database.
When your durability strategy is to spread your risk across a lot of nodes in
a cluster, and you're federating your cluster across multiple sites,
traditional backups may not be a practical option. However, as I write this,
I'm realizing that there's nothing simple about distributed anything, and
saying "it's hard to..." may not be all that interesting.

~~~
scurvy
Cassandra handles this pretty nicely. Files are immutable, so files are hard-
linked to a snapshot directory. Copy out the snapshot directory, delete it,
and things are eventually consistent.

We restore our MySQL and Cassandra data every day in our production DC from
backups in the DR DC. It's the only way to ensure things are working. It's
about 100TB of data, but things zip along nicely now that 100gig networks are
(more) common.

~~~
PopeDotNinja
Neat!

------
AdmiralAsshat
This is a question I've thought alot about, so many some Sys Admins can give a
good idea about how to approach it:

How do you create a backup server that is reachable by production servers (so
that they can back up to it) _without_ then being vulnerable to the same kind
of ransomware attacks that infect the production servers? You can't exactly
make them read-only, or else they can't accept the "legitimate" writes that
might occur during the normal backup process.

~~~
johngalt
A generally safe backup process looks like this:

Production has no access to backup.

Backup has read only access to production.

Backup writes are append and not overwrites.

Deletes/archival are governed by a retention process.

~~~
ipython
One absolutely diabolical mechanism that was used (at least 5 years ago when
this scourge of ransomware started to rear its ugly head) goes something like
this:

1\. Gain access to change the code on the front-end web servers (usually PHP)

2\. Change the database access layer to transparently encrypt data being
written to the database, and decrypt data being read from the database. The
key would be loaded into memory by curl'ing an attacker-controlled website at
startup.

3\. Wait 30 days

4\. Notify the company that they're compromised, turn off the attacker-
controlled key service, and restart the web front end

Now step (3) ensures that most data in the database has been re-written, and
if your backups are dumps of the production database, you now have a month of
encrypted backups that you can't read... If you're lucky, you may have a
month-old backup to restore from; if you're unlucky, you rotate every 30 days.

~~~
landryraccoon
If there's an engineer who can pull this off why are they screwing around with
ransomware? Have them send me their resume, if they can deploy code to our
frontend transparently without customers or devops having to spend time or
even noticing it we'll happily pay them more than they're asking for in
ransom.

Edit : thinking about it, this story doesn’t add up. Doesn’t this mean the
client must have a copy of the decryption key, meaning any cached client would
render the ransom demand worthless? It’s just also so much easier, if someone
has that level of access, to make a copy of the database, encrypt it, then
blow away the old one. Doing a silent deploy of client code with no one
noticing seems way harder.

~~~
bin0
Honestly, many of them are in nations where they are working for a state-
sponsored cybercrime agency and have few other options, if any. North Korea
and Iranian gov'ts are major purveyors of ransomware; I've known many clients
who were breached by them (FBI guys said it was usually one of them each
time). If not, it's probably Russia or China, especially if it's a higher-
value target. If it's just a private citizen flinging exploits at ip ranges,
it's possibly Brazil too.

Honestly, it used to be Eastern Europe, until we started realizing they've got
some serious programming talent and contracted some work out. Now it's less
bad, though still a bit sketchy. Of course, this doesn't help nation-state
attacks.

------
floatingatoll
iNSYNQ is a third-party service platform operator who specializes in hosting
instances of QuickBooks in the cloud.

This does not affect non-iNSYNQ QuickBooks instances, such as those operated
by Intuit (the creator of QuickBooks).

~~~
BinaryIdiot
I had no idea this was how QuickBooks even worked in the cloud. Could Intuit
be in any way liable (like is this a sort of franchise type of thing)? Or is
it closer to Word Press where different companies can install "QuickBooks
Cloud" and then offer it?

~~~
floatingatoll
Think “Microsoft Access in the cloud”. How would you do that without
Microsoft’s cooperation? Virtual machines running Windows. Full service
hosting, the works. This is classic ISP stuff, back when ISPs were service
providers and not just Internet connections and other off the shelf no-touch
products.

------
basilgohar
It's not immediately obvious, but iNSYNQ provides hosting for QuickBooks
Desktop as a virtual desktop service (think VNC or RDP). This is distinct from
Intuit's own QuickBooks Online cloud service, which I believe is unaffected by
this breach.

I was concerned because one of my clients' customers rely heavily on
QuickBooks Online and her app integrates heavily with it.

~~~
chx
I can confirm QBO is up (I am in Canada). Holy Batman, the chaos if that data
would be gone... I did a backup, quickly. I need to ask my accountant whether
they back this data up regularly, if not then I need to. I am a very small
company but my invoices are international and while I have my invoices at
hand, if I would need to reconstruct the exchange rates for an audit a few
years back... _shudders_

~~~
wildduck
ALWAYS make OFFLINE backup! USB USB USB!

------
tracker1
Was this service really less expensive than just using actual QB on an RDP
available cloud server from Azure or any number of other services?

Also, what kind of hacky backup system takes this much time to sort through to
identify issues. They should have a clean image, and a clean way to
backup/restore data for the application being hosted as a pull from
production/active deployments.

In the end, this will or maybe even should kill the company in question.
Beyond this, it is an opportunity for others. For that matter, really
surprised Intuit doesn't have this as a cloud service at this point.

~~~
miles
> really surprised Intuit doesn't have this as a cloud service at this point.

They do have a cloud offering, QuickBooks Online:

[https://quickbooks.intuit.com/online/](https://quickbooks.intuit.com/online/)

But it does not have all the same features as the Desktop version, giving rise
to a number of third party offerings, like Right Networks' "QuickBooks Desktop
Cloud": [https://www.rightnetworks.com/cloud-solutions/accounting-
sol...](https://www.rightnetworks.com/cloud-solutions/accounting-
solutions/quickbooks-desktop-cloud/)

------
ForrestN
"After the third day of outages, customers were saying Bye Bye Bye."

~~~
pixl97
One company was already in the process of moving after iNSYNQ got hit earlier
this year by ransomware. This isnt their first attack.

------
nabilhat
It's not impacting all of Insynq's services. I work with an Insynq customer.
Their Insynq services are still running, by some generous stroke of fate. The
only outage we noticed was in the middle of the day on the 16th. For about an
hour, users weren't able to access the service. I called support, and was
diverted to a recorded message saying that they were doing normal maintenance,
would be finished shortly, and were aware of and sorry for the disruption.

It's been my outspoken opinion that this was an inevitable outcome for as long
as I've been familiar with their product.

------
Keverw
Wonder if they obtained any people's financial data or social security
numbers. Probably mostly self employed people and small businesses. Pretty
scary how people use their SSN for everything. Seems so insecure to have a
number you just openly pass around... Need to get a ID or license, credit
card, bank account, your doctor, dentist asks for them, your phone company,
cable company, of course when getting paid, and even police officers ask for
them sometimes and write down in a notepad if your name happens to match
someone else's name who has a warrant. Many other uses probably too I didn't
think of off the top of my head.

I was randomly one day looking at dentist new patient forms and one even
wanted to know your relationship status, not sure how that's relevant if a
single or married guy gets a cleaning... I know home alone when the internet
went out, so called the local cable company to see if an outage and the lady
wanted the social security number on the account before continuing, which I
didn't know. Just insane how many things use the same number, it's like single
sign on for real life.

Same issue with bank account numbers. To pay someone with direct deposit, they
can use the same number to withdraw from your account. I'm surprised banks
haven't figured out a way to offer deposit only option... Just create a new
account number but linked to another account, where deposits to account 4321
goes to account 1234 instead, but can't ever withdraw from 4321.

I got a feeling Facebook's account system is probably more secure than my
local bank. Pretty sad when someone's hobby blockchain project has more
technology in it than banks with billions of dollars of assets under
management.

~~~
mschuster91
> I was randomly one day looking at dentist new patient forms and one even
> wanted to know your relationship status, not sure how that's relevant if a
> single or married guy gets a cleaning...

This is for spousal rights - i.e. if your spouse is allowed to request access
your data.

> I'm surprised banks haven't figured out a way to offer deposit only option

They have, some German banks assign an IBAN also for "Sparbücher" (saving
plans). These cannot be withdrawn from.

For withdrawal security, under SEPA rules you have 8 weeks to (instantly!)
reverse a transaction. If you misuse this, you can get your account closed and
criminal proceedings filed so that is a relatively effective fraud prevention.

~~~
Keverw
Interesting, don't know much about the spousal rights thing as never been
married but felt like they are asking too much info. I figured it's to try and
collect if you don't pay since also employment info was a question.

I know I heard in Europe there's some law called PSD2 that banks would provide
standard APIs too, but haven't been following that space since not in Europe.
I know there's budgeting and other apps but they login to your bank account
and scrape the data. I was using a app that categorize your spending for a
little while but got sick of it making me relog my accounts over and over. I
think one of my credit cards was thinking their servers were trying to hack my
account. So a actual official API sounds like the move in the right direction.

~~~
ownagefool
We call PSD2 "openbanking" in the UK.

It basically means things like YANB (budgeting tool) or freeagent (
accountancy tool for small business) can get access to data you want them to,
without you giving them your credentials.

Historically, most banks offered all or nothing access, and the aggregators
would screen scrap using credentials that can steal all your money, and the
links usually broke, or needed you to contioniously re-auth.

Its actually working. Modern banking apps will likely start to pull in all
your financial services in the future.

------
not_a_cop75
The cloud is just someone else's computer.

~~~
meristem
The cloud is a magic data layer in the sky, powered by unicorns in hamster
wheels and cold, hard marketing cash.

------
julianlam
The article seems light on details about how QuickBooks is involved. Does
iNSYNQ host QB Cloud data for them, or does iNSYNQ white-label QB software?

~~~
unreal37
Quickbooks is not involved.

It's like a Wordpress hosting provider getting attacked. Wordpress is just
software hosted there.

~~~
gowld
Quickbooks (the software) _is_ involved. _Intuit_ is not involved.

------
blendo
What are the odds their vulnerability was due to IT staff choosing to not
patch their Windows servers, as doomed Baltimore in May?
[https://krebsonsecurity.com/2019/06/report-no-eternal-
blue-e...](https://krebsonsecurity.com/2019/06/report-no-eternal-blue-exploit-
found-in-baltimore-city-ransomware/)

~~~
belltaco
The article does not say the attack was due to IT staff choosing to not patch
their Windows servers, am I missing something?

~~~
blendo
You are correct, the Krebs article I cited does not say that. But it's my
strong hunch.

There was much discussion of the poor state of Baltimore's IT infra (see
[https://www.baltimorebrew.com/2019/05/21/baltimores-out-
of-d...](https://www.baltimorebrew.com/2019/05/21/baltimores-out-of-date-and-
underfunded-it-system-was-ripe-for-a-ransomware-attack/)), and at the same
time Microsoft released a patch for an RDS flaw (CVE-2019-0708, [https://msrc-
blog.microsoft.com/2019/05/14/prevent-a-worm-by...](https://msrc-
blog.microsoft.com/2019/05/14/prevent-a-worm-by-updating-remote-desktop-
services-cve-2019-0708/)) which says "Vulnerable in-support systems include
Windows 7, Windows Server 2008 R2, and Windows Server 2008. Downloads for in-
support versions of Windows can be found in the Microsoft Security Update
Guide. Customers who use an in-support version of Windows and have automatic
updates enabled are automatically protected. "

So unpatched servers are where I'm laying my money.

------
snthd
Is there a way of freezing a compromised machine, such that the contents of
RAM are kept around for forensics, but the compromised system can't do any
more damage to itself? SysRq? Intel ME?

~~~
yjftsjthsd-h
In the common case of a VM it should be easy. Bare metal, less so.

------
WheelsAtLarge
Slowly but surely ransomware is becoming an absolute threat to all data. It
seems more like not if but when it will hit.

It seems to me that it's time the OS providers start providing a very easy way
to restore the state of data. We all know that backups are the answer but as
long as people, have to think about it, there will always be some that don't
do them. And now that you can get a 1TB HD for less that $100 then it's a no-
brainer.

Virus protection is now automatic with Windows when will backups become
automatic on all OSs?

~~~
swiley
>It seems to me that it's time the OS providers start providing a very easy
way to restore the state of data.

Wat?

How are you imagining this would be implemented? Right now you can have a
cronjob that runs rsync which allows exactly that but doesn’t protect you from
ransomeware. Are you thinking if it were part of the file system in the kernel
it would be better protected? I guess that’s partially true but in practice
probably will do very little (also, that already exists on Linux in the form
of things like ZFS.)

Once something like that infects the machine there really isn’t a whole lot
the OS can do to prevent the damage, all it’s doing is handling user requests
for operations on things like the file system and the malware is operating as
a proxy for the user.

------
anbop
Why would someone use a third party to host Quickbooks? Won’t Intuit host it?

~~~
the_svd_doctor
This. Is it the same Quickbooks ?

~~~
tornquist
No, this is a desktop version hosted offsite instead of the browser-based
cloud version from Intuit. See this above post for examples:
[https://news.ycombinator.com/item?id=20481074](https://news.ycombinator.com/item?id=20481074)

~~~
gowld
How is "QuickBooks Desktop Hosted in the Cloud" different from "Anything you
darn feel like, hosted on a remote Desktop (like Citrix)" ?

~~~
andrewf
The former is probably easier to sell to accountants who want QuickBooks
Desktop plus the benefits of a remote managed service (accessible over the
Internet, someone else takes care of backups - in theory! - etc)

------
trollied
If anyone wants a recommendation for an alternative, I can happily suggest
Xero.

What a mess, though. Worst part of a business to be crippled is its core -
financials. A part often overlooked by techies. If you can’t invoice, you
can’t pay the wages.

~~~
pavanagrawal123
It's not Intuit that's down... Just a 3rd party hosting provider.

~~~
PKop
What are they hosting? Backups of the desktop version of Quickbooks?

~~~
pavanagrawal123
AFAIK, there are legacy versions of a "networked" quickbooks that can be
hosted by third parties. Not really sure though. Some are doing app streaming
of desktop quickbooks

~~~
npo9
I wouldn’t call the networked versions of quickbooks legacy. (Quickbooks
Desktop and it’s ancillary products) That’s like saying Excel is legacy
because Office 365 will let you edit spreadsheets in the cloud. Sometimes
local operations are just better, or cloud versions don’t fully match
features.

~~~
pavanagrawal123
Yep a more appropriate term would be "older".

------
roshanravan
For an accounting software this is disaster.

------
nodesocket
My Quickbook Self-Employed (hosted) seems unaffected (at least so far).

~~~
pavanagrawal123
that is different from this. The affected is a 3rd party hoster, not Intuit.

------
ryanmercer
I lost all interest in the topic when I read the hosting companies name, can
we just talk about THAT? Like, really? Who picks that name for their company,
certainly the band nsync has to be older than the company.

~~~
benjaminbrodie
It's certainly very search engine friendly. You essentially get a curated list
of all and ONLY items of potential interest to you.

~~~
ryanmercer
My point was more "hey that seems like a MikeRoweSoft [1] potential legal
quagmire, terrible for branding and SEO and a great way to be mocked and not
taken seriously at all by potential clients".

[1]
[https://en.wikipedia.org/wiki/Microsoft_v._MikeRoweSoft](https://en.wikipedia.org/wiki/Microsoft_v._MikeRoweSoft)

