> but enabling acoustics and power saving really helped on the seeking noise level.
Do you know what's the approximate power draw (in Watts) of each drive with these settings enabled? I'd like to move to a lower power system sometime in the next year, but I'll also need to add some more drives. Power is extremely expensive where I live.
I wonder if the power saving features reduce the drive lifetimes (because of quicker spindowns)... might not be a good idea for an always on NAS.
So I just recalled I had a DC-capable clamp meter, and I was able to isolate the disk on the last stretch of a power connector chain.
I tried running with both the power management set to "Level 128 - Minimum power usage without Standby (no spindown)" and Acoustics Managment to "Maximum", as well as "Disabled" for both. Didn't make a noticeable difference on power draw, at most 0.1A more during writes, primarily just noise.
Idle
5V 0.27A
12V 0.40A
Write
5V 0.53A
12V 0.50A
So that's just over 6W idle and 8.7W under load. I'm surprised about the high idle draw, both due to being significantly higher than the specs[1] and due to them running so cool. Did they spin down the disks to get the 4W figure perhaps? I did check my clamp meter against my electronic load and it reads pretty accurate.
I didn't manage to test reads properly, since the disks are part of a ZFS pool and it spreads the reads all over, along with aggressive caching.
Toshiba datasheet says 8.1W under Q1 4k random R/W, I did my testing with streaming writes and queue depth around 10, so my write figure seems reasonable.
For idle the datasheet says 4.53W typical, however I admit my disks were idle only for a few minutes when I did my readings as dinner was almost ready, so possible they would go lower after longer idle periods.
I recently built a NAS that sits 50cm away from me so noise was my #1 priority. During my research I realized that there are really two types of NAS drives:
- the enterprise ones that are meant to sit in a datacenter. They offer the best performance but are noisy and power hungry.
- the SOHO ones, that are often 5400 rpm. They are lower performance but optimized for noise and power consumption.
I ended up with ST6000VN001 and the noise level is very reasonable.
I'm not terribly happy with the 8TB Red model, given it only lasted 3 years. It's also been fairly noisy and running hot.
I have eight 3TB Reds, so far I've had two of those developing pending sectors after over 6 years of power-on time (no spindown), and of those one developed uncorrectable sectors a year later and I replaced it. The other one is chugging along happily so far with 27 pending sectors.
In the case of the WD Red, it went from pending sectors to failing SMART self tests in less than two weeks.
I have a fanton drives 2tb HDD model from like 2010. i dont use it frequently. it seems to run fine to this day. however, if its going to die soon, will it give me signs of failure, or will it just completely lock my data one day? i would like to get a replacement at the earliest sign, but idk how these things die.
I've seen Backblaze Hard Drive Stat articles in the past, and they were a lot longer and had a lot of tables breaking things down by manufacturer. This has one chart and a lot of links, but the newest one is from January. So what is being shared here that's new?
This looks like a shift away from their old “editorialized” blog-style updates to a data-sharing-centric approach. I’m guessing that this takes less time for them and it allows various commentators and communities to create their own opinions based on the data.
I liked the tone and approach of their old blog posts but this is pretty cool too. It’s just good to see them continuing to share their data since it’s arguably relevant to a wide range of audiences.
Off topic but related - a great podcast episode with the author of Restic, an open source backup system (that can connect to online services like Backblaze).
I have some ~50 TB total in a NAS, all WD drives. I bought some 3 & 4 TB WD Red drives around 2014, and they have all been going 24/7 with absolutely no problems at all.
I recently needed to expand (to the point I am at now) and bought & shucked 14 TB WDs, so I'm curious to see whether there will be any long term difference in terms of reliability between the "official" red drives and the shucked whitelabel drives.
> I bought some 3 & 4 TB WD Red drives around 2014, and they have all been going 24/7 with absolutely no problems at all.
It may seem like they are running fine, but are they actually? Have you run a zpool scrub or equivalent?
What happens with these old drives is that one dies and then you have to replace it, which is very hard on the other drives as the array is rebuilt. Then while the array is being rebuilt, another drive dies. It's better to replace drives when they are EOL (usually 4 years if running 24/7) rather than waiting until there is a problem.
The vast majority of SMR hard disks are 6TB capacities and below, and I'm only aware on one 8TB Seagate SKU that is SMR. Though I'm aware HGST shipped a 20TB SMR drive late in 2020.
In general, check the R/N of shucked white-label drives, and you should be able to quickly find a corresponding datasheet (hundreds of pages) from the manufacturer.
For example, some 16TB WD Elements I'm in the process of testing after shucking are R/N US7SAR160, which are 16TB HGST Ultrastar DC HC550. These are SATA 6Gbps 7200rpm helium-filled CMR drives with 512MB of cache.
I always wondered: what happens to customer data when these drives fail? I would imagine they would be using fault tolerant systems like RAID of ZFS to mitigate. I admire their transparency in the usual style of: 'We're a backup company, look how many drives we have failing!'
You can read on their blog. They use custom software with Reed-Solomon* erasure coding, similar to RAID but distributed between different disks, servers and racks. Their EC library is on Github (Java). The most enjoyable posts are the ones describing how they built their pods and architecture.
I guess the could encrypt the data that ends up on the platters so that it would look like random noise to any attacker recovering the platters from a broken drive that was not physically destroyed?
Of course that has both performance and data corruption scenarios that one needs to take into account.
I am avoiding Blackblaze as they don't allow you to do due diligence. Any requests to get security audit reports, pentest reports under NDA are all ignored. If anyone know if they are available through their website I am looking forward hearing it.
> I am avoiding Blackblaze as they don't allow you to do due diligence. Any requests to get security audit reports, pentest reports under NDA are all ignored.
What they should do is make you cut a half a million dollar check, refundable when you spend half a million dollars on services.
It removes so many headaches from dealing with people who think their $100 over a year is Very Big Money.
I don't understand why it matters if I only spend $100 or half million? If I am going to use third party I want to verify it meets my own security requirements. I think that's totally reasonable to ask.
Actually I would expect that such a big company have these things readily available on request.
Because in the vast majority of the cases those that want these docs after getting a generic one start asking questions/want interaction while having the attitude that "Why should it matter if I only spend $100?" thus expending way more company resources than their account is worth.
You make it sound like reviewing these documents is free.
If a company ask me for similar paperwork (which they have) I have the paperwork in order and ready, they sign a NDA and I am sending them. It's just a step in the sales process imho
Trixter, yes, I have it ready because I did all these audits already as part of company security policy; and not because of customers and I can share them on request.
A company that never did a pen test or security audit or doesn't want to share them doesn't give me much trust to use them as a partner.
I can't tell you how many times I've been able to implement solutions for my organization based off of experience with personal projects. If I'm satisfied in my due diligence of the provider then I'm a lot more likely to turn around and suggest it for use in my corporate environment.
In this case it's not so much about a $100 spend, it's about them potentially leaving a lot of money on the table if they are incapable of delivering the reports in question.
The money is in the head, not in the long tail. It is possible to make money off the long tail by never treating anything other than the head as a potential head. You will miss some middle of the distribution customers, sure, but you won't spend resources of hundreds if not thousands "influencers" that don't actually influence anyone.
Based on what I have experienced, those that have real decision making power in companies that will make a high six to seven figure purchases simply do not have time to vet their home projects where they are going to spending $100/year. The grandstanding arguments about importance of their projects come from people who probably won't even spend $100/year
Of course, I wouldn't ask this for a home project but if its considered for a business archival solution were government regulation requires me to store things for multiple years and client data for same period. Of course, I will make sure this data is safe. I am not going to depend on their marketing pages.
You make it sounds this is ridiculous to do a security assessment or to ask for such paperwork. I can tell you that my company insurance even demands it. At the moment I prefer to pay 3x more and store things at a cloud provider which shares these kind of documentation.
After https://news.ycombinator.com/item?id=26536019 I pulled off all my files from backblaze. Is there a good alternative for long term storage other than amazon glacier? For about ~6tb-8tb of data.
Isn’t this a bit hasty. Backblaze fixed thr issue immediately, wrote a detailed blog posted and it was also found to have only been in a single webpage of theirs.
Now while I certainly agree that I don’t like this info leaked, there was no mal-intent and they quickly took action once notified. I feel like their action is what makes me like them so much.
Yes and no. This issue was perhaps X lines of code away from accidentally transferring the contents of your files (or login email and password!) to Facebook, or a malicious attacker.
For one, this highlights they don't have automated tests detecting arbitrary/malicious JS injection on their web app. This is a serious security risk: we are talking about your cloud filesystem here, and "spear phish / bribe your marketing intern to adding malicious.js" is a real attack vector.
Alarms should be going off internally whenever a new external JS file gets included in your webapp, either ststically or dynamically. Facebook pixel today, a malicious hacker tomorrow.
I'd expect a private cloud to have better security procedures than "wait till someone on Twitter discovers a bug".
(Personally, I will continue to use Backblaze, but just highlighting why it's a serious security concern.)
Ahh, fair enough. They definitely snippets to be vetted by their security team. I'm sure they've learned from this mistake though, and I'm not sure I would trust anyone else any more than them.
It would be far better if they described how they managed to let Google Tag Manager be included for so long, and how they've changed their deployment process to avoid third-party scripts from being included in the future.
Out of curiosity, do you have a recommendation of a tool / structure for an automated test to catch that type of injection? This is something I haven’t considered in my own pipeline that I’d like to address.
For external files in particular you can use Content Security Policies [1] in the server configuration.
Injecting third-party content then requires editing both your site and the server setup. Of course, you can make the policies more or less strict depending on how much you want to tighten this kind of attack vector.
Everyone gets mad when there’s a breach or a bug that doesn’t get full disclosure.
When a company provides full disclosure people are shocked and abandon the platform.
Every company has bugs. Everyone eventually gets breached. If a company is honest with what happened, follows up with their users, and fixes it, what more can you ask for?
Should it have happened? No. Did they find the issue? Yes. Did they fix it? Yes. Did they disclose it? Yes. Perfect, thanks.
That was not a deliberate action though, that's just the default behavior of the analytics snippet that someone included on the page. If there's humans involved there will always be bugs, the question is how well you deal with them and react. I think they did a good job once it was pointed out.
Because corporation didn't deliver on promises. Trust was broken and their attempts at fixing said trust is meaningless given what they've already done to break said trust.
There's a difference in character between a "sorry we messed up, we fixed it" mistake and a "sorry we got caught, we fixed it" mistake. A pattern of the former often suggests the latter, but as far as I know Backblaze has no pattern of these.
Yes, 500+ Comments [1], you can quite literally put it out as HN hate Backblaze. Part of the Cancel Culture where they expect everyone is saint and can do no wrong.
I mean, worst of all, many of them are Web Developer means there is a very high probability they have make mistakes /bugs in the pass, big or small. Which put them in the category of hypocrite.
That's a good example of the paraphrased quote "don't make software for privacy people, they'll never be happy". They fixed it, yet every other service is doing far worse behavior. Dont hold backblaze on a higher pedestal than amazon unless you only want amazon to exist.
Backblaze is the most affordable and available remote storage/cdn service available, by miles. Go look at a calculator between amazon and backblaze.
The actually give half a care and even partner with various providers so your outgoing bandwidth is free to them.
What bothers me is using facebook tracking scripts in the first page, particularly for internal customer pages. Of course I'm not happy with them working with facebook at all.
Other services doing worse things is not an excuse. I don't want to go with amazon either, hence my question here.
For a large number of Windows and Mac users who have less than 1 TB to backup (or less than 6 TB if they are willing to put up with a fair amount of hassle) there is a far less expensive option.
I speak of Windows and Mac users who got a Microsoft 365 subscription for the Microsoft Office apps. That comes with 1 TB of OneDrive, or 6 TB if you get the family subscription.
OneDrive has API access which is supported by a fair number of commercial and open source backup programs.
You can get a family subscription without being a family. All you actually need is the ability to control 5 extra email addresses, one for each fake family member. Your 6 TB would end up partitioned into 6 1 TB buckets so as I said, would be somewhat of a hassle to use.
But if you only need 1 TB, and want or need Office anyway, and don't really have a lot of non-backup cloud needs, then OneDrive is a good, often overlooked option.
Considering that Backblaze was totally open and correcte this mistake immediately, what you are asking for is a backup company that never makes mistakes. They dont exist. Buy a synology NAS. Slap some ironwolfs in it and do your own backups.
How many years at $60/year of Backblaze service will it take to equal that NAS setup? Also, will that NAS survive a disaster? These are the types of questions that make the Backblaze services so hard to walk away.
Tim Apple could stand in the middle of 5th Avenue and shoot someone but HN would still use their products. Backblaze accidentally deploys a facebook pixel to its pages and HN cast it off to the tech shadow realm.
Hetzner has storage boxes and storage services. Not as cheap as Glacier of course. Otherwise there is Acronis, IDrive (nothing to do with Apple), and Carbonite. I've never dealt with the latter three only heard of them.
Well, the idea with Backblaze/Glacier is cheap long term storage in exchange for high request costs, no?
For me anyways, this is a tertiary backup of data that is duplicated in a zfs raid 10 pool and in external drives at a different house. A high request cost isn't much to get back my family photos, documents, etc, in a situation where neither of the first two backups are available.
I initially wanted to pull all my data, but then realised that I in fact encrypt everything locally so what they see is just garbage, so in my case if the provider is unsafe and leaks all my data, that would be of no use to anyone.
If you want to nail a company to the cross for being in bed with Facebook... Backblaze probably shouldn't be at the top of your naughty list.
IMHO, over-ado about a small something.
"A new campaign was launched beginning on March 8, 2021 on the marketing web pages using Google Tag Manager which included the Facebook pixel. That new campaign resulted in the Facebook advertising pixel being accidentally configured in Google Tag Manager to run on all platform pages instead of just the marketing web pages."
"We’ve confirmed that there was only a single page (b2_browse_files2.htm) where the Facebook advertising pixel had the ability to access certain metadata. We tested this on Chrome, Safari, Firefox, and Edge. Our investigation determined that 9,245 users visited that page during the window when the Facebook campaign was active (March 8 at 12:39 p.m. Pacific time, through March 21st at 11:19 p.m. Pacific time when we removed the offending code)."
"If users were browsing their B2 Cloud Storage files on b2_browse_files2.htm during that period, AND clicked to preview file information, then the Facebook pixel pulled the following metadata: folder/file name, folder/file size, and the date the folder/file was uploaded. The folder/file metadata was limited to file information that was currently loaded in the browser.
No actual files or file contents were shared at any time. The data that was pulled did not include any user account information."
In previous threads about this leak, commenters gave examples of how exposing filenames could be harmful to certain classes of paying customers. It is not a small thing by any stretch of imagination.
Then those users probably shouldn't be using something with servers in the US that can be accessed with a court order. They should backup to an offsite store they have full control over.
> We use Google Tag Manager to help deploy key third-party code in a streamlined fashion. The Google Tag Manager implementation includes a Facebook trigger. On March 8, 2021 at 12:39 p.m. Pacific time, a new Facebook campaign was created that started firing a Facebook advertising pixel, intended to only run on marketing web pages. However, it was inadvertently configured to run on signed-in pages.
I can see how that can happen. The moment that GTM was pushed for at my last company it made me really uncomfortable but ultimately got pushed through. I wish it had a built in process for review and approval.
It looks like it does but the permissions are so frustrating to use that it just becomes overly permissive as a side effect.
I just spent the last 12 hours migrating everything to B2. BB seems to be somewhat immature relative to the competitors, and the product certainly feels that way, but as long as they try and the data integrity is there, then I think I am okay with it.
The rest of my stuff is in S3 with Glacier auto archives.
We’re so pissed at Google Tag Manager being included in a page of a website in error, that we’ll move our backups to a google service (that with certainty will track analytics at google indefinitely).
I looked at rsync.net, but they are 2.5 cents/gb versus backblaze's $0.005/gb or glacier's $0.004/gb per month. I see rsync.net for active use, while b2/glacier are for set-and-forget backups.
Tape is also incredibly slow and difficult to scale. Also, the equipment needed to operate a cluster of tape archivers costs a lot, while they are using their own designs dor servers, with coXmmodity hardware in them.
Tape is slow only when you need frequently to read some random part of it, because you might need a couple of minutes to go till the other end of the tape.
The reading/writing speed of tapes is excellent, better than that of hard drives, so writing a backup takes less time.
For archives that are accessed only infrequently, tape is more reliable and also faster.
If you want to have access in less than a minute to any part of your backups, then yes, tape is inappropriate and you must use HDDs.
And to backup a NAS, it requires to regularly change the tapes. Any backup that requires manual steps is likely to be delayed and to be too infrequent to be useful.
They're massive beasts, with 9 platters and 18 heads, but enabling acoustics and power saving really helped on the seeking noise level.
Not sure how much of a performance hit that leads to though, haven't had time to fully investigate.
They run significantly cooler than the 8TB Reds tho, from ~45C down to ~30C in the same bays.