I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount/repair) it until you've made your copy.
With journaling and the like (soft-RAID)... mounting at all can induce more writes than you expect; yes, even read-only. That's probably not what you want. Consistency is key.
'Just' (I know) use "dd", "ddrescue", or simply pv/cat/shell in+out redirects (some limitations apply) to evacuate it first. This gets you two things:
1) infinite tries: files/loop devices are cheap and can be copied for free
2) better odds: the new home isn't questionable, right?
Stop thinking in terms of devices. You can write this drive to a file. You can write files to drives. Same for reading/checksums. It's all made up, yet it works. Magic.
By skipping past the block level to the filesystem, one is limited in what they can recover. You're at the mercy of it being more sane than truly necessary.
By mounting it, and potentially/indirectly writing [with bad media/connectors/whatever], you may be irrevocably breaking the filesystem.
Perhaps they get into this, but wow. I know I'm not 'The One'. I hate to pretend to be; but this is day one data-preservation stuff. I'm almost certain we've all taken compliance training that covers this.
Cutting corners like this is pretending to be The One; you don't know better. Nobody is perfect, I'm on my soap box because I've done the same thing - and want to save others. It hurts.
I had to post when I got to "LOL it went RO, it's probably fine" (my words)... when no, the kernel interjecting to protect the data is not a reasonable first line of defense.
Aside from what they chose to do... damage well-and-truly could already be done by their inaction. Having left it running/hoping for re-allocation.
I wish I could have as much faith as they've shown in what little I've read so far. I close my rant with this: a little mechanical sympathy goes a long way. Now back to reading.
I can't edit now... but one thing I'd like to change - it's not fair to call this their first line of defense.
Rachel did the most important thing right: taking backups at the first sign of trouble, presumably beforehand too.
I just feel they're off the mark foregoing block devices and having a strange process as a result... "because images are big". Speeding up doesn't normally quiet a squeaky wheel. Not to say it can't... but let's place bets :)
There are options for this concern without compromising the efficacy of the whole process; for example, {de,}compression.
Yes, you care about the files, but to recover them... you don't get to ignore the details of the filesystem - or the constraints you're allowing it (and yourself) to operate under.
The takeaway from this isn't "substitute well-established recovery processes..." but, make sure you have backups so you can afford two perks:
* ignore a failing drive for weeks
* not care *at all* about your attempts to recover the source filesystem
The filesystem, your storage, everything will lie.
Prepare for it; make copies and accept this will take time. This chestnut still applies: "If you don't have time to do it right, you must have time to do it again."
The right thing is not storing data on a single 2.5 consumer SSD because the endurance on those devices are trash and the fail at super high rates.
They aren’t designed for a linux server, they’re designed for a disposable Windows laptop that will be chucked in the bin after a year or two. I’ve learned this lesson the hard way over, and over, and over again with these little cheap SSDs.
Most consumer 1TB SSDs with 600TBW should last at least within 10% or so of 600 days at 1 DWPD. If your ZQQTIAN or KingDonxi Amazon specials, populated with Micron QA rejects or SpecTek super economy grade or random e-waste upcycling chips failed in a month, that's on you.
> The right thing is not storing data on a single 2.5 consumer SSD because the endurance on those devices are trash and the fail at super high rates.
What do you mean by "endurance"?
The only definition I found was the number of writes during the warranty time, and you're talking about a use case involving a very low volume of writes (i.e., save a file, keep it up for eternity).
This means that warranty time (more specifically, any failure mode at all beyond excess writes) is the most likely failure mode. Not writes, just the SSD being old.
Samsung advertises between 3 years and 10 years of warranty. The typical warranty of HDDs is between 3 and 5 years. Doesn't this suggest that buying you a decent SSD buys you at worst the same reliability?
How do I know an SSD is reliable or not? How do I know it's a consumer SSD or a real SSD? I'm particularly interested in drives which contain firmware that doesn't lie to the operating system about whether crucial operations have completed or whether errors were found. Is there a way to know?
I want to buy equipment that will last at least 10 years. It won't be heavily written but it will be heavily read. To me it feels like SSDs should last a long time for write-once read-many use cases.
Meh. The right tool for the job. That's engineering.
I hint at that by not caring about how the copy is made. Just that it is made... and that it's checked. Perhaps against some friends/additional copies if the situation allows for it.
Maybe all you have is an SD card or floppy drive, y'know? Each project or situation is different, I find dealing with the matters at hand more practical than dwelling over what I'd prefer.
Laptops would all be better off with four drives in RAID10 but we just don't do that.
Right, you need at least some dm-linear device mapper device in between to ensure a device is RO I think. Also, ext4 has separate "noload" option that actually instructs it not to touch the disk (ofc relying solely on "ro,noload" to not mess with your broken FS is a bad idea)
It's not a soap box, it's good advice!! Most people use default options which means they mount with access times enabled where every read causes a write. File systems are more complex than they seem. They also do lots of automatic house keeping in the background. Even the relatively simple ext4 has journaling. Modern file systems will even move and rebalance data in the background.
My laptop fell down on the floor recently. Hard disk started reporting errors immediately afterwards. I picked it up to assess the damage and immediately noticed that Linux remounted the drive read only. I turned it off, bought a 4 TB SSD, flashed a live Linux ISO and started reviewing the GNU ddrescue documentation.
When recovering data, things should be as static as possible. I'd keep the disk platters from spinning if I could.
Dunno, man. It's an old Mac-mini, single disk (the SSD is explicitly called out) fairly obviously used for private stuff. You can wring it.
In a production environment you cut you losses and restore from back-up. No-one wants to deal with half-baked recovery operations which might or might not yield a consistent state with (a little) less, but unknown, data loss.
I don't think this was wheel-spinning so much as yarn-pulling, finding interesting tid-bits and seeing where they go. The most interesting things (to me) are when something doesn't quite line up with expectations, and then digging until I understand what the difference was - usually I misunderstood something (or failed to recall something, like in this case, disk/filesystem caches) - and I find myself more enlightend.
I totally get doing this to learn because it was an interesting process. But yea, if this were for money? I'd just restore from backups and follow the SOP - because for money, the _why_ something failed is often unimportant, the remedy is the same -- reboot, replace, correlate, correct.
Playing with Schrodinger's stuff. The storage industry/equipment lies so pervasively, even when it's working as intended
More importantly, perhaps, is this is a "brain worm" being spread to others. Not to armchair too much, but there's a certain responsibility by publishing your writing.
I'd hope for more dutiful research before publishing/risking data with anything even remotely nearing suggestion. Loud/clear warnings.
We're all playing the Telephone Game whether we want to or not.
> mounting read only can induce more writes than you expect.
Have you got a source for that? I can only find speculation on various sites, but nothing that explicitly confirms that outstanding journal entries would be written at RO mount time.
"Note that, depending on the filesystem type, state and kernel behavior, the system may still write to the device. For example, ext3 and ext4 will replay the journal if the filesystem is dirty. To prevent this kind of write access, you may want to mount an ext3 or ext4 filesystem with the ro,noload mount options or set the block device itself to read-only mode, see the blockdev(8) command."
The remaining nugget/memory on this I have is a fairly open ended... "it depends [on the filesystem]". It's been a while since I dove into this earnestly.
IIRC there's no single journal mechanism - they all choose to do it (or not) in their own way.
Journaling was just one example, too.
Let's not ignore other quirks, like how BTRFS conflates volume/filesystem management... and may exhibit behavior impossible with XFS/ext{2,3,4} [under N failure conditions] WRT soft-RAID.
By not mounting it, you're avoiding the "kernel/software filter" that adds entropy to an already-rich process.
Side note: you caught me in a series of edits for phrasing :D
> I know they mean well, but if you have a filesystem you want to preserve on questionable media... don't interact with (mount) it until you've made your copy.
A further clarification of your point...
If you are a wearing a forensics hat and engaged in formal recovery effort, your point is significant good advice.
But when you are a regular joe trying to deal with a failing device, it's usually not clear when you've put on the forensics hat; you just find yourself struggling with a device that is not returning your data.
And while you are in this moment of discovery the failure mode, the system is trying to do all its normal stuff, including maintaining the journal.
So if playing the journal is a hazard to your data, it has likely already been encountered and any corruption as a result of journal processing been done.
At some point the regular joe may decide to put on the forensic hat and formally rescue the device.
In my experience, this is not necessarily the most productive next step. It depends on the device, its failure mode, the data and the usage scenario. Too many variables to generalize about tactics.
No matter the value of your data, it's wise to begin a disciplined recovery as soon as possible. Where backups are the best place to start any recovery effort.
However, once you decide to rescue the device — as opposed to nurse it along to gain access to something — that's the clear threshold to become scrupulous about read-only access: It's important to keep both the source and the destination devices in a read-only state from the beginning to end of the rescue to avoid further corruption of the copy.
In this light, any journal processing that occurs for read-only mounts, presents a possibility hazardous condition with implications beyond the scope of casual system operation. This is the province of a proper forensics regime, with a rescue system configuration and protocol designed to prevent tainting the target, and also beyond the scope of a user comments on a help thread.
Given all the other uncertainties, getting to the bottom of what a read-only mount actually does with the journal on your system at hand is probably not productive. But arranging for read-only mounts is given the uncertainty is certainly productive, so prevent auto-mounts while rescuing.
If you want to use GNU ddescue,
I've written a bash script to help with preventing auto-mount for Linux and macOS.
github @c-o-pr ddrescue-helper
Basically the script just writes /etc/fstab with the UUID of the filesystem and "noauto" option and unmounts the device. But there's other logic in the script that may be useful.
That's the least important part about my post. I mentioned several things, one being the shell you probably already have running, that all will do a perfectly fine job - to hopefully demonstrate that.
'cp' is sparse aware. Yay. Is that a good thing, actually? If the source has zeroes, I want to write zeroes. Not sure... probably fine. I've been awake too long.
'ddrescue' is special, but again, consistency is key. I don't really care how it's copied. I just question 'should' - there are advantages elsewhere.
edit: My main gist is this - living off the land applies here, too! Whatever option is available. Make the copy, check the checksums. Pull out the bigger gun if necessary.
Never know when you have to rely on some binary that's surviving only in caches. Stuck with a broken system and no live environment in sight.
There is, however, something special about `ddrescue` (note that there are 2 different projects named thus; I don't remember the difference) - it does all the okay sectors, then goes back and retries bad sectors.
Just be sure to store the ddrescue state file somewhere persistent.
dd can copy block by block from one block device to another
ddrescue can do the same, but will handle errors better
e2image understands ext2/3/4 filesystems and can copy only the blocks in use, which can really help (though don't know how it works if it encounters an error)
The correct abstraction is that a file is a device. A file is a section of a hard disk which behaves like a whole disk. It also happens to be expandable.
cp's default behaviour for device files also happens to be very stupid.
I assume the only reason for not turning off and taking the drive out to image "offline" is because SSDs have been known to permanently brick themselves when power-cycled, but I agree with the others here that trying to interact with the filesystem on a dying storage device is a really bad idea.
Incidentally, having experienced an SSD and HDD death within the past few months, the HDD definitely gave far more early warning signs (including some very audible ones) than the SSD, and failed in a "softer" fashion.
rsync has the (important) advantage it can generally be stopped and resumed.
If the source drive's actually failing, I would instead ddrescue asap, and work with the (make it a read-only file) image.
Should the filesystem need fsck, qemu-img can create a read-write image that is an overlay backed by a read-only file, and qemu-nbd can expose that through nbd, so that your kernel can use it.
I forget the exact details now, but in the past I found that cpio was the best option at times.
It may have been circumstances when I didn't want the entire tree, but rather files that met specific criteria, and I was able to combine find and cpio (-o option) together.
Sort of orthogonal anecdote. I ran freebsd on a failing usb stick for about a year. It had copies=3 or similar on it and zfs just did it's best with fixing things up on the fly. Ran scrub fairly often, it told me things were bad, system kept running. That is to say, flash can stumble on with pretty high failure rate for much longer than I expected.
journald also logs to /run/log/journal/ (a tmpfs) so until reboot you can still extract logs from it even if it didn't manage to write them to disk. See the 'storage' option in `man journald.conf`
options if you are concerned about journald overusing tmpfs. But yes, it seems unconditional that it writes the logs always somewhere, which is not unreasonable design imho.
Obligatory recommendation to use ddrescue as a first pass. Personally I've been helped in at least three ways by this:
* ddrescue doesn't retry immediately; it tries to get as many readable sectors/blocks as possible first and then goes back and retries. When there are many bad sectors this is way more efficient than retrying N times at each bad sector while additional readable data is sitting on the bad device.
* ddrescue has configurable retries and its map file to perform rescue over any interruptions. If a device loses power or overheats and stops responding or for whatever other reason you don't have to get everything off the failing device in one pass.
* If you have two failing RAID1 devices you can ddrescue from one and then from the other to get as much data as possible.
If you need to recover audio CDs, use cdparanoia which takes a similar approach but uses a lot of speculative re-reads to avoid bad reconstruction efforts by some CD drives as much as possible.
>Even if you tell it to continue after read errors ("noerror"), you've just silently corrupted something on the target disk.
The idea of using `dd` / `ddrescue` is to get the contents of the drive off it ASAP before it fails. You don't want to mount it. You don't even know if the filesystem on it is sane. The priority right now is to get all the bytes off that you can, then analyze it at your leisure to recover what you can. For the same reason you would also not `dd` directly to the new disk. You'd do it to a storage area where you can inspect and manipulate it first.
And:
>Finally, there is the whole matter of geometry. [...] Will the partition table wind up in the right place? What about the boot sector data and secondary stuff like the locations where your boot loader might be looking?
... is not a problem for the same reason. Once you have the disk image off the failing drive, you're welcome to inspect it and copy individual files off it to the target drive at the filesystem layer (rsync, cp, etc).
And so:
>Second, you're dealing with individual files. If your tar or rsync (or whatever) fails to read a file due to some disk issue, you'll know about it.
This is the right idea. It's just not optimal to do it from the failing drive itself.
My god, that article is actually filled with terrible advice. I hope they don't do this professionally. It's bad enough they're misinforming people and directing them to do things in a worse, more fraught failure-prone way.
> First of all, 'dd' is going to try to read the entire drive, including the parts which currently do not contain any useful data. This happens because you have probably told it to read /dev/hdc. That refers to the entire readable portion of the disk, including the partition table and the parts which are not associated with living files.
Make a sparse image with dd. Then you have a 1:1 copy while skipping all the unused sectors to save space. You can then mount the image read-only or write it to a new disk for recovery. That means that you have all the time in the world and you can start over if need be without affecting the source data.
Doing all this on a possibly actively failing drive, particulary an SSD which can literally die 100% into an unrecoverable state without access to a lab in the blink of an eye? That's insanity. And it's even worse SUGGESTING that to random people who don't know any better. You want to read everything off the SSD or HDD once. As fast as possible. As completely as possible.
In fact, make sure you do a non-trim pass first, so you can be sure you have as much as possible before you try retrying sectors which might actually kill the drive dead.
Once all the recoverable data is off the failing drive as an image, you're free to do whatever you want however you want without fear of any data loss caused by possible mistakes, oversights, hardware issues, etc.
> Make a sparse image with dd. Then you have a 1:1 copy while skipping all the unused sectors to save space.
Filesystems don't zero out deleted data and dd isn't aware of the filesystem mapping, so unless it's a completely fresh drive, you'll still pull off garbage data.
ddrescue is one tool that combines dd with the ability to read the filesystem metadata to only extract out the allocated filesystem blocks.
Sorry, you're right, I definitely meant ddrescue. That's what I always use, no idea why I wrote dd. Especially since ddrescue -d -n -S /dev/sdb sdb.raw sdb.logfile is permanently burnt into my brain. I guess I just took it from the previous comment and article and didn't give it a second thought before posting my comment.
> Filesystems don't zero out deleted data and dd isn't aware of the filesystem mapping, so unless it's a completely fresh drive, you'll still pull off garbage data.
Sure, but if there's unused space left zeroed, such as partitions that you didn't use, then you get to skip it. It's not a huge priority but it doesn't hurt.
>Filesystems don't zero out deleted data and dd isn't aware of the filesystem mapping, so unless it's a completely fresh drive, you'll still pull off garbage data.
That may have been true a very long time ago.
Today, some filesystems such as ZFS can effectively, and automatically, zero out deleted data using trim on devices that support that. (I say "effectively" here because we don't actually know if the data is literally zero'd by the underlying device, and I say this even though that distinction doesn't actually matter in this context. Once trimmed, those logical sectors will always read as zeros until something new is written to them, and this is the functionality that is important in this context.)
This function is useful for SSDs, and is also useful for SMR spinny-disks.
Tons of other combinations of filesystems and operating systems also deal quite well with trimming of unused space, though this more-often happens as a scheduled task instead of something that is taken care of by the filesystem itself.
Trim (in various implementations) has been broadly used for well over a decade, and a trim'd device can lead dd to be able to produce sparse files.
---
Now, that said: It probably doesn't matter much if a particular dd-esque tool is set to create sparse output files or not. Sure, some space may be saved, and sparse files sure are cute and cuddly.
But it's probably a fool's errand to even plan such an operation on a machine that has less free space than the total maximum capacity of the thing being rescued: Either there's enough room to write a non-sparse image, or there isn't enough room to even think about starting the process since it might not be able to complete.
(If space becomes an issue later on down the road, the output file can be "sparsified" in-place using "fallocate --dig-holes" in instances where that makes sense.)
And I definitely want the whole disk imaged, which means that I definitely do not want ddrescue's interpretation of metadata to determine filesystem allocation and limit the scope of that image: This is the first step of a data rescue operation, and that makes it the worst place for data to be intentionally thrown away or disregarded.
If things are failing hard enough that any of this work is on the table, then obviously the combination of the source disk and filesystem is untrustworthy -- along with the metadata.
Getting all of the bits backed up -- regardless of their apparent lack of importance -- should always be the prime directive here. Any extra bits can always be tossed later if they're eventually deemed to be actually-unimportant.
There is no purpose in sending TRIM after each delete: at worst (with a naive implementation) you would get a solid write amplification and at least your drive would be doing GC instead of serving the data.
Most of the time it's just sent every once in a while, often triggered by the schedule and/or the amount of writes[0].
[0] my current machine says it was 9 days since 'the last retrim' and T440 which 'works' as a glorified dashboard (ie minuscule writes overall) says it's 24 days.
I'm pretty sure that OSes usually send TRIM right away and let the drive figure out optimization. Wikipedia says "some distributions" of Linux turn it off but my various ubuntu-ish systems have `discard` or `discard=async` in the output of `mount`.
"last retrim" on Windows is an extra feature, because TRIMs can get dropped when a drive is busy enough. It goes through all the free space and TRIMs it again once a month.
Also even if you did only TRIM once a month, I think most of your free space would still be zero.
> You want to read everything off the SSD or HDD once.
I mean, no, not really. You want to read everything off at least once. Because a "corrupt sector" is actually often in a non-deterministic state, reading differently with each read — but that state may still be floating just far enough toward logical 0 or 1, that you can get a bit-pattern out of it through statistical analysis, over multiple reads.
You do want to capture the whole disk once first, in case the disk spindle motor is about to die. (This isn't usually the problem with a dying disk... save for certain old known-faulty disk models — the "DeathStar" et al. But it's good to be safe!)
But after that, you want to read it again, and again, and again. And save the outputs of those reads. And then throw them into a tool like ddrescue that will put the results together.
Basically, it's the same principle that applies to archival preservation of floppy disks (see e.g. https://www.youtube.com/watch?v=UxsRpMdmlGo). Magnetic flux is magnetic flux! (And even NAND cells can end up with approximately the same "weak bits" problems.)
---
Mind you, it'd be even better if we could get several analogue dumps of the disk, and then merge those together using analogue-domain tools, like the ones used for floppy preservation mentioned in the video above.
Sadly, unlike with floppy preservation, it'd be very difficult to get an analogue dump of an HDD or SSD. With floppies, the flux is right there for the reading; you just need a special drive. HDDs and SSDs, meanwhile, are far more self-contained. For an HDD, there might be a path to doing it, at least on a disk-specific basis... it would likely involve tapping some signal lines between the disk controller and the read head. But for an SSD, I don't think it would be possible — IIRC with most NAND/NOR flash packages, the signal is already digitized by the time it comes off the chip.
---
Also, I should mention that if your HDD disk spindle motor is about to die, then you're actually on a bit of a ticking clock. You want to get your data off that disk as quickly as possible, before the motor locks up and the head crashes on the platter.
Tools like ddrescue have the option to first read a disk's filesystem metadata to build a map; and then use that map to only bother to read filesystem-allocated sectors, seeking past any unallocated ones.
Doing this can greatly speed up the time it takes to make a complete copy of "any potentially-useful data on the disk" (as opposed to "the entire disk".) Which means you might be able to get a capture done on a disk "just under the wire" using this method — where otherwise the disk would have died in the middle of reading some unused sectors, with some good data stored further down the disk left un-captured.
But of course, there's the counterpoint that the filesystem metadata itself might have holes in it, where that would mean that the map made by the tool will fail to capture some of the data.
I think this isn't a weighing-pros-and-cons situation, though, but pretty clearly a "right tool for the job" situation:
• If you're getting fsck errors, weird data corruptions, or files are just disappearing — and it's clear that you're going to need to run one of those forensic data recovery programs against the disk image to extract what you can from it — then you'll need the whole disk, including the "seemingly unallocated" parts.
• If all your data seems fine, but you're getting SMART errors, writes are slowing down, and your disk is making funny noises... then you're probably better off grabbing just the filesystem-allocated sectors ASAP, before your drive stops talking to you.
> Tools like ddrescue have the option to first read a disk's filesystem metadata to build a map; and then use that map to only bother to read used sectors, skipping the unused ones.
I don't know how many versions of "ddrescue" there might be, but GNU ddrescue doesn't know anything about filesystems.
The "map" file is just a list of device LBAs (extents) and recovery status.
I've seen a tool for NFTS that can pre-populate a GNU ddrescue map file with unused extents to avoid ddrescue reads of unused areas, but you have to find this tool and manage it yourself.
ddrescue might be a good change for you, in particular specifying to not retry (because you're going to come back later) and to specify the 'log' file that's actually a map of every sector on the source drive and whether it's been successfully imaged.
I don't think I've ever imaged an SSD, but for HDDs your ideal is a single pass through with no retries on bad blocks, possibly skipping ahead if you hit a bad block, and a log file writing to the destination drive where you're also storing the image in a file (NOT doing direct output of the input disk). This gets you everything you can get with minimal disruption of the source.
After that first run you'll specify the same log file for every time you run it, and (on HDDs) you can tell it to do things like trying to read the higher-numbered sectors first (so if you got an error on block 10,000 and told it to skip 100 on errors, now you start at 10,099 and work backwards, perhaps getting down to 10,005 before you get another error. Basically, get the dropped fruit, then the low-hanging fruit, then work harder for anything remaining.
Before you do those followup runs though, it's a good idea to look at things with the 'ddrutility' tools to see if you even need to bother. ddru_findbad will let you use the image file and 'log' file to identify what files might have pieces missing so you can decide if you need to progress further; ddru_ntfsfindbad does basically the same thing for NTFS volumes. It's always nice to determine that 1) yes there's a chunk of bad blocks and 2) we don't have to care because they're all in the Hibernation file so we can restore that image to a new drive, boot, disable and re-enable hibernation just to be cautious, and have no concerns about lost data on the old drive without having to beat the crap out of the drive hoping that on the 1000th attempt it's finally going to read that one failing sector.
sorry if the latter part of that got preachy, I've seen some bad advice in the past on imaging failing drives and it's made me sensitive....
I recently had the pleasure to work with ddrescue. Unfortunately the SSD disappeared on encountering read errors and could only be brought back by power cycling (at least as far as I found out in the time allotted) :/, so it went to my 'defect hardware' pile.
same reaction, and the author not using ddrescue just makes this a tale about not following any sort of documentation when installing their distribution. There's really nothing in there that anyone should take away besides making sure they didn't hack up fstab and remove tmpfs.
Having a "real" /tmp is not an uncommon preferences amongst sysadmins who've been at this for a while.
Too many things that can wedge a system only leave evidence of why/how they did that in /tmp so still having it around post-reboot can be a huge aid to root cause analysis.
It's a non-trivial trade-off but calling deliberately choosing that trade-off "hacking up fstab" doesn't strike me as a remotely fair description thereof.
Generally, things that write tempfiles while processing data in /tmp and manage not to log enough because they wedge the system before it occurs to the code that anything's gone wrong enough to log.
Yes, absolutely, "bad software, no cookie," but the usual culprit is some sort of vendor binary where the poor sod running the system has no control over that.
BSD systems generally clean out an on-disk /tmp during the normal boot process, yes. There are ways around this, but when I've been responsible for babysitting craptastic vendorware it's always been on Linux or Solaris.
Personally I've (after quite some grumbling about it) accepted /tmp being on tmpfs and just live with it; my current source of crankiness is "people who don't configure their systems to write to syslog" since if the box gets wedged by an I/O storm systemd will shoot systemd-journald in the head and then journald sometimes deletes all of your previous logs as it starts up.
One example that springs to mind was a vendor antivirus system that unpacked email attachments into /tmp - generally when it died on its arse the only way to figure out why was to dig into /tmp and look at what it had left, then try and infer backwards to the culprit email from there.
Yes, the problem isn't disk usage, the problem is that if journald's writes get too slow a watchdog timeout will cause systemd to assume it's crashed and shoot it in the head, which leaves the journal part written, which means on restart the new journald process throws away the old journal as corrupt.
(this may have been fixed in the last couple of years, but it leaves me somewhat untrusting of it in terms of actually being able to read my logs)
You're right, but this was introduced around 2012 [1] when that old laptop was manufactured.
Also no one is forcing you to store GBs of data on it and indeed in practice most programs use only a couple of MBs.
Apparently I'm illiterate. Indeed, not only is the Mac Mini a desktop, but also the author says in that same paragraph how they did in fact upgrade its RAM once in the past.
> smartd logging some complaint and then also barking about not having any way to send mail. What can I say - it's 2024 and I don't run SMTP stuff any more.
Not hard if you use exim4, postfix, bsd-mailx, and a curses-based client like alpine to keep a low energy, simple local mail service running for just such things.
Reminds me of those monitoring dashboards printing lot of unexpected output from checks, which in turn doesn't expect not being able to interact with state files on a read-only filesystem, reporting the wrong state (i.e. critical instead of unknown), etc.
More fan of agents and an HA core for monitoring, than from direct push notifications, but still, used this smartd notifications like 15 years ago in Debian, and never realized of this issue (maybe because once the OS disc fails, we cared more about other issues than about the smartd notification).
Those were the years were you could try to put the mechanical disc in the fridge for some minutes and try again, or with some Seagate disks, change the electronic circuit for other disk of the same model and recover the data (they used to fail in the electronic part more than the magnetic plates).
> Those changes weren't managing to get flushed to the old disk with the now-RO filesystem, but they were apparently hanging out in buffers and were available for reading... or something? That makes sense, right?
This is a fun issue that I've run into before. We had remote FreeBSD 11 boxes using ZFS. ZFS said the pool was fine, OS said the drive was fine, writes appear to work, but they are only going to RAM and are lost on reboot (which happens once you run out of memory, the box locks up, and someone on-site pulls the plug). Dealing with drives going bad with no indication is an interesting problem, especially when on reboot they appear to be working expect they have "time-traveled" backwards (to when the drive first went bad).
TIL that systemd/journalctl is supposedly able to handle an read-only rootfs gracefully. I'm used to SysRq+R causing a lockup after a few seconds, on Debian & Ubuntu.
With journaling and the like (soft-RAID)... mounting at all can induce more writes than you expect; yes, even read-only. That's probably not what you want. Consistency is key.
'Just' (I know) use "dd", "ddrescue", or simply pv/cat/shell in+out redirects (some limitations apply) to evacuate it first. This gets you two things:
Stop thinking in terms of devices. You can write this drive to a file. You can write files to drives. Same for reading/checksums. It's all made up, yet it works. Magic.By skipping past the block level to the filesystem, one is limited in what they can recover. You're at the mercy of it being more sane than truly necessary.
By mounting it, and potentially/indirectly writing [with bad media/connectors/whatever], you may be irrevocably breaking the filesystem.
Perhaps they get into this, but wow. I know I'm not 'The One'. I hate to pretend to be; but this is day one data-preservation stuff. I'm almost certain we've all taken compliance training that covers this.
Cutting corners like this is pretending to be The One; you don't know better. Nobody is perfect, I'm on my soap box because I've done the same thing - and want to save others. It hurts.
I had to post when I got to "LOL it went RO, it's probably fine" (my words)... when no, the kernel interjecting to protect the data is not a reasonable first line of defense.
Aside from what they chose to do... damage well-and-truly could already be done by their inaction. Having left it running/hoping for re-allocation.
I wish I could have as much faith as they've shown in what little I've read so far. I close my rant with this: a little mechanical sympathy goes a long way. Now back to reading.