> If newpath already exists, it will be atomically replaced, so that there is no point at which another process attempting to access newpath will find it missing.
More: https://sqlite.org/howtocorrupt.html , https://sqlite.org/tempfiles.html
Typical (mis)behavior includes correctly-sized files that consist partially or entirely of zero bytes (because NTFS does guarantee newly-allocated space will be zero-filled).
Nevertheless, I agree with you, and, therefore, when dealing with application-managed files (as opposed to user-visible documents) on local NTFS filesystems, I typically
1. Write to a temporary file filename.temporary, say.
3. Rename the original file to e.g., filename.original.
4. Rename the temporary file to the original filename.
5. Delete filename.original.
Then, in some thread-safe manner at some point before accessing any such files — at application startup, say, when performance permits — I remove lingering ".temporary" files and restore any orphaned ".original" files to their original filenames.
This, of course, assumes FlushFileBuffers works properly, so, in particular, still fails in the presence of storage devices, drivers, and configurations (e.g., "Turn off Windows write-cache buffer flushing") that break buffer flushing, which are not nearly as rare as they should be.
On *nix you need to sync before renaming as well.
do you mean you add the sqlite library, load the file and then use sqlite to read/write data to a database instead of a text or bin file? I like it...
I do simulations in physics and we're really not familiar with these types of things.
Some extra time to learn SQL but it will save you a lot of time in the long run.
Another benefit compared to ASIC is that you get: File compression, easy to access random parts of the file and fast (random) inserts.
It is not the default (default is usually FULL).
I have all my mail since forever online, that is, tens of thousands of posts including attachments in multiple inboxes belonging to multiple accounts: everything since my Internet day one (20+ years), the earlier messages imported from Eudora before moving to Linux.
It's all in my $home, and searches are damn fast: fractions of seconds for header searches which are indexed, tens of seconds to minutes for searches into the body. As far as I recall, I never lost a single file.
~$ du -chs Mail
Would love to see a few (:
I've had firstname.lastname@example.org since 1995 but sadly I haven't kept the stuff older than 15 years.
Subject: YeS It Works!.....Gotta Have It 15643
Date: Wed, 18 Nov 1998 02:53:45 -0500
X-Mailer: QUALCOMM Windows Eudora Pro Version 4.1
Try This POTENT Pheromone Formula That Helps
Men and Women To Attract Members of The Opposite Sex
Click here to learn more:
Ever wonder why some people are always surrounded by members
of the opposite sex?
Now YOU Can...................
Attract Members of The Opposite Sex Instantly
Enhance Your Present Relationship
* Meet New People Easily
Give yourself that additional edge
Have people drawn to you, they won't even know why
Click Here For Information
Read What Major News Organizations Are Saying About Pheromones!
To be removed from our mailing list,
Spamming was occasional at that time, and I probably deleted most of them, but in a few years it became unsustainable and I installed IpCop to identify junk and redirect it to its own folder.
One more example:
From: "InvestTXXXX@uasc.com.kw" <InvestXXXX@uasc.com.kw>
To: "XXXX@prodigy.com" <XXXX@prodigy.com>
Subject: Fwd: Investor's Alert
Date: Sun, 9 Jun 2002 21:07:41 -0400
Cal-Bay (Stock Symbol: CBYI) Watch for analyst "Strong Buy Recommendations" and several advisory newsletters picking CBYI. CBYI has filed to be traded on the OTCBB, share prices historically INCREASE when companies get listed on this larger trading exhange. CBYI is trading around $.30¢ and should skyrocket to $2.66 - $3.25 a share in the near future. Put CBYI on your watch list, acquire a postion TODAY.
REASONS TO INVEST IN CBYI
• A profitable company, NO DEBT and is on track to beat ALL earnings estimates with increased revenue of 50% annually!
• One of the FASTEST growing distributors in environmental & safety equipment instruments.
• Excellent management team, several EXCLUSIVE contracts. IMPRESSIVE client list including the U.S. Air Force, Anheuser-Busch, Chevron Refining and Mitsubishi Heavy Industries, GE-Energy & Environmental Research.
RAPIDLY GROWING INDUSTRY
Industry revenues exceed $900 million, estimates indicate that there could be as much as $25 billion from "smell technology" by the end of 2003.
ALL removes HONERED. Please allow 7 days to be removed and send ALL address to: XXXXXAgain@btamail.net.cn
Certain statements contained in this news release may be forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. These statements may be identified by such terms as "expect", "believe", "may", "will", and "intend" or similar terms. We are NOT a registered investment advisor or a broker dealer. This is NOT an offer to buy or sell securities. No recommendation that the securities of the companies profiled should be purchased, sold or held by individuals or entities that learn of the profiled companies. We were paid $27,000 in cash by a third party to publish this report. Investing in companies profiled is high-risk and use of this information is for reading purposes only. If anyone decides to act as an investor, then it will be that investor's sole risk. Investors are advised NOT to invest without the proper advisement from an attorney or a registered financial broker. Do not rely solely on the information presented, do additional independent research to form your own opinion and decision regarding investing in the profiled companies. Be advised that the purchase of such high-risk securities may result in the loss of your entire investment. The owners of this publication may already own free trading shares in CBYI and may immediately sell all or a portion of these shares into the open market at or about the time this report is published. Factual statements are made as of the date stated and are subject to change without notice. Not intended for recipients or residents of CA,CO,CT,DE,ID, IL,IA,LA,MO,NV,NC,OK,OH,PA,RI,TN,VA,WA,WV,WI. Void where prohibited. Copyright c 2001 *
Totally undeterministic behaviour, because it would turn around and work on other searches.
Yeah, files, are hard.
It stores inboxes in mbox-like format, then some metadata in very weird .msf format, and on top on that SQLite database for search indexing. And it has some bewildering bugs like mixing up titles and content from two different messages, or slapping on messages date pulled from thin air..
IME, this happens when the message doesn't have a date (that is, Date header missing or invalid). There's an open request on bugzilla to use the last Received date in these cases.
- atomic file create with content
- atomic directory create
- atomic file move
- atomic file delete
- single-level file enumeration
No file modification in place which is the actual hard path. Not even appending to existing files.
Incidentally, reliable databases tend to also not rely on file modification, opting for a log-and-compact strategy instead.
It's when you start modifying existing data that 'just use a filesystem' becomes unviable.
No FS guarantees this. You can emulate this by writing to a tmp location, syncing and performing a move.
It is not technically atomic, because there is a period where oldpath and newpath both refer to the same file, but I don't think that is an issue for this usecase.
At work they were using outlook for mail and wondering why it crashes now and then (past 2 GB it starts writing over the start of the file or some such). We get a lot of big documents on email so switched to thunderbird and all is well since. Had to switch mailserver though since that one also only handled 2 GB. No idea what they were running but most likely something MS-related. Most of their stuff was MS at the time.
It uses a Xapian index, not an RDBMS, but it's impressive. RDBMSes aren't known for producing good natural language search results.
Rdbms for the email use case is kind of orthogonal to fast full text search.
Oh, and I also dislike even the idea that I have to find some "specialized software" to have basic functionality in my e-mail client. That's why most still suck a*.
If all you want is to grep your email, that's different, I think you can accomplish that with pine or other text mode email clients.
links in my stack: https://github.com/jakeogh/gpgmda-client
I recently switched my work notebook to XFS, so looking at this table makes me kinda happy, even if I had bad experiences with XFS in the past (a forced turnoff thanks to a crash in X11 was sufficient to put my system in a unbootable state, trying to recover it with xfs_repair broke it completely; also I suffered with 0-size files randomly appearing after forced reboots quite constantly).
I know for a long time that XFS is probably one of the most well written filesystems for Linux, even if the user case seems to be more focused in servers with uninterrupted power supply than desktops and notebooks.
And the thing is that even with uninterruptible UPS, you better handle crashes more gracefully than what XFS is apparently able to do, because e.g. kernel panics can occasionally happen.
So on my side, I'm moving from XFS to ext4. I'm not even sure this will be better, I'll see...
Back when the original POSIX specs were being worked on, a common assumption was that anyone serious about their data ran all disks/raid controllers/filesystems/etc in writethrough (or equivalent) mode. Combined with what the spec doesn't guarantee leaves us in a world where its pretty much impossible to really make any data retention guarantees. A large part of this is that write() style api's should never have been allowed to be anything but synchronous with the data being commited to non-volatile storage. Thats because if you move the error handling to when fsync() or close() (or pick your place) if one of the writes fails its impossible to report accurately to the application what failed sufficiently to know how to recover it. This goes beyond just filesystems these days, there is a major raid chipset vendor that is/was shipped by a couple tier 1 vendors which only ran in write-through/FUA mode if it wasn't provided a battery, the performance was so abysmally bad that nearly everyone ended up buying the battery to enable write back. The problem with this controller in write back mode is that it didn't honor any kind of fencing or FUA when in writeback mode, instead depending on a intermittent timer flush based system to force the data to disk. If there was a disk failure on write, it was impossible to know what had actually been flushed and depending on firmware it would go from either reporting the error to silently dropping the data. Neither helps the end application that might have quit, and had definitely passed its close/rename/flush/etc sequences by the time the error occurred.
Bottom line, there is an API mismatch from top to bottom of the storage stack. Starting with the simple idea that if a filesystem operation doesn't provide an async completion notification it should be forced to be consistent at completion. Anything else _WILL_ create the opportunity for data loss over even simple power loss, much less the more complex cases of delayed writes in raid controllers/async replication/etc.
Put another way, like this article points out and others linked here reference, there are a ton of "bugs" in most OS's storage stack. Enough that they generally behave badly in the face of actual failures.
An Analysis of Latent Sector Errors in Disk Drives (2007) https://research.cs.wisc.edu/adsl/Publications/latent-sigmet...
Failure Trends in a Large Disk Drive Population (2007) https://static.googleusercontent.com/media/research.google.c...
Cycles, Cells and Platters: An Empirical Analysis of Hardware Failures on a Million Consumer PCs (2012) https://www.microsoft.com/en-us/research/wp-content/uploads/...
Flash Reliability in Production: The Expected and the Unexpected (2016) https://www.usenix.org/system/files/conference/fast16/fast16...
IRON File Systems (how filesystems behave on disk errors) (2005) https://research.cs.wisc.edu/wind/Publications/iron-sosp05.p...
EIO: Error Handling is Occasionally Correct (2008) https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
SQCK: A Declarative File System Checker (2008) https://www.usenix.org/legacy/events/osdi08/tech/full_papers...
All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications (2014) https://www.usenix.org/system/files/conference/osdi14/osdi14...
Filesystem error handling (2017) https://danluu.com/filesystem-errors/
Files are fraught with peril (2019) https://danluu.com/deconstruct-files/
And a DRAM reliability paper, for a more complete picture:
DRAM Errors in the Wild: A Large-Scale Field Study (2009) https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf
write(/dir/log, “2,3,foo”, 7);
pwrite(/dir/orig, “bar”, 3, 2);
write(/dir/new, “foo”, 3);
> Update: many people have read this post and suggested that, in the first file example, you should use the much simpler protocol of copying the file to modified to a temp file, modifying the temp file, and then renaming the temp file to overwrite the original file. In fact, that's probably the most common comment I've gotten on this post. If you think this solves the problem, I'm going to ask you to pause for five seconds and consider the problems this might have.
> The main problems this has are:
> rename isn't atomic on crash. POSIX says that rename is atomic, but this only applies to normal operation, not to crashes.
> even if the techinque worked, the performance is very poor
> how do you handle hardlinks?
> metadata can be lost; this can sometimes be preserved, under some filesystems, with ioctls, but now you have filesystem specific code just for the non-crash case
> The fact that so many people thought that this was a simple solution to the problem demonstrates that this problem is one that people are prone to underestimating, even they're explicitly warned that people tend to underestimate this problem!