In case anyone was wondering why a "red book" audio compact disc isn't designed to be resiliant in a way that is familiar to a modern technologist (or even compared to a CD-ROM disc) the basic reason is that the former is designed to stream with no buffering directly into a DAC chip built with technology from the late 1970s. It had to handle data rates that were incomprehensibly massive for the time, so only the most insanely basic error handling was feasible.
Whereas the "yellow book" CD-ROM standard got to take advantage of over four years of technology advancement. Plus it doesn't have a real time requirement, so there's plenty of time to calculate more advanced checksums and error correction codes.
In many respects, an audio CD is more like a vinyl record than a data storage format.
> In many respects, an audio CD is more like a vinyl record than a data storage format.
The design is nearly identical to what one would get if one simply literally (almost) digitized a vinyl record. A single long spiral of data on a disk. The most significant shift is that vinyl records read from outside in, while audio CD's read from inside out. Even the method of manufacture is essentially a clone (audio CD's are "pressed" just like vinyl records are "pressed", just in different pressing machinery).
> built with technology from the late 1970s. It had to handle data rates that were incomprehensibly massive for the time, so only the most insanely basic error handling was feasible.
Yes, the limits of late 1970's technology had a huge impact on the overall design, but at the same time the design also took advantage of the fact that audio when being played back in real time is reasonably tolerant of a small level of error resulting from the read-off of the disk. Most listeners would never hear the random errors because they either are below the noise floor of the amplifier tech. at the time, or because the resulting analog wave was close enough to not be noticeably different.
> The most significant shift is that vinyl records read from outside in
Equally significant (and equally interesting) is how they spin. Whereas vinyl has constant angular velocity—they spin at the same rate from beginning to end, resulting in lower information density as you get towards the inner grooves—audio CDs have constant linear velocity, meaning that the rotational speed gradually changes as the disc is played so that the information density is equal from beginning to end.
For the era, this is very cool tech. (I'm also awestruck about how they managed to get the laser head alignment to work perfectly back then, it's practically magical.)
You're right for audio CDs but very fast CD-ROM readers use constant angular velocity (CAV), mainly because they start hitting mechanical constraints (including the integrity of the CD itself).
> resulting in lower information density as you get towards the inner grooves
It's actually the opposite: highest information density (musical notes played per surface unit) is in the innermost grooves.
Since linear velocity is lower in the center than the border, the size dedicated to each piece of data is larger in the outermost places, which may increase the quality of songs placed there.
I don't really know if the machine making the disc is precise enough and if the human ear is capable of hearing the difference.
Hmm, I would arrive at the opposite definition of "information density" than you, and I'm guessing the GP does, too.
Your definition is "musical notes played per surface unit", but I think that's inverted. Taking a musical note here to mean a fixed note length, more surface per note allows for additional expressiveness. You can even "fit more notes in" if the notes are a shorter duration.
A time span on the outer part of the record has more groove length dedicated to the same amount of playing time, so it has a higher information density in my book. I guess my definition for density would be groove length divided by playing time.
With your definition the information density is constant as in the outer part of the disk the wavelength need to be longer to compensate the higher velocity
>The design is nearly identical to what one would get if one simply literally (almost) digitized a vinyl record.
I don't really agree with that, or at least it only makes sense to me if you're willing to go at a level of abstraction so high that makes it relatively meaningless.
For one thing CDs store digital data, zeroes and ones, not analog like a vinyl. On top of that the data is written using 8-to-14 modulation (every 8 bits of data is written using a unique 14 bit pattern on disc). This provides very basic error correction and ensures that you never have very long stretches of zeroes. Then on top of that you have a cross-interleaved Reed–Solomon coding. You also have subchannel data that contains things like position in the track and other metadata. You also have a table of content at the beginning of the session to easily seek to individual tracks.
Besides the fact that it's disc shaped and read in a spiral I don't find many similarities with vinyls. Sure the format itself is largely driven by the music track format (which is why you still address CD-ROMs with "minutes" and "seconds" even though it's meaningless there) but then so is Spotify.
What's interesting is that LaserDisc (the optical ancestors of DVDs that look like giant CDs) are analog. The video and audio streams are encoded with Pulse Width Modulation (PWM)
Well that's what the rest of my comment was about, "digitizing a vinyl record" makes it sound like you take a vinyl, pass the analog audio through an ADC and output that directly onto an other disc and you get a CD. In practice it's a lot more complicated than that so I disagree with your original comment:
>The design is nearly identical to what one would get if one simply literally (almost) digitized a vinyl record.
The rotating disc format is simply due to technical constraints and not caused by vinyl mimicry (unless you also consider that hard disc drives and floppies are nearly identical to vinyls as well). The rest is completely different from a vinyl on almost every level, besides the fact that they both store stereo audio.
It's pretty interesting to read the red book and try to figure out why things are done the way they are, it really gives some perspective on the progress we've made in a few decades. An other example of somewhat "obsolete" technology in CDs is that all the track numbers and positions (minute:second:sector) are stored as binary coded decimal which is a bit annoying when you try to manipulate CD data on a modern system. It probably made a lot of sense at the time though, it meant that you could "stream" the BCD data directly to the display without any conversion while binary data would have needed a binary-to-decimal conversion which, I assume, would have been quite costly at the time.
Wait a minute, you mean the guy in the mall audio store in 1982 or so who claimed you could break the disc and still play it might have been lying to a kid?!
This is a little awkward. The Touhou album in question is already in the Touhou Lossless Music Collection (at least the last release: http://www.tlmc.eu/2018/01/tlmc-v19.html since it's from 2005, it's probably been in most of them), and has track 3 ("The End of Theocratic Era" by "弘世"). I just checked the TTA and track 3 sounds fine.
A decent amount of the albums in the TLMC were ripped and uploaded to Japanese P2P programs, a large amount of these rips were ripped "losslessly" with EAC, but they had the "normalize" option turned on in the settings, which adjusts the volume of the rip if it's above or below a certain threshold, basically ruining all the work EAC put into getting a bit-exact copy. This is likely why the rip in the TLMC doesn't match OPs rip.
Edit: If anyone is more curious about this, a large amount of the rips in the TLMC were ripped by one uploader (or a group of people using a singular alias) on the Japanese P2P programs Share[0] and Perfect Dark[1].
He was so prolific in the scene that he has his own Baidu wiki page[2] (albeit, it's not as strict as Wikipedia with notability, I still find it impressive).
All of his rips, as far as I know, were affected by this "normalize" setting, so even though he ripped hundreds (thousands?) of albums, a large majority of them probably wouldn't be considered "archive quality".
Normalizing alone with no compression/limiting involved shouldn't ruin fidelity at all. Actually in some (admittedly extreme) situations it could enhance it. A track that is very low from start to end could leave one or more most significant bits unused, therefore reducing bit depth, so especially during mastering I would welcome normalization if it brought the highest peak to 0 dB or slightly less.
Of course I mean just normalizing without any compression/limiting, ie the dynamics curve stays the same: I'm well aware of the loudness wars plague.
I believe you're correct that raising the volume is not necessarily a destructive process, since it would just be increasing the amplitude of the samples already there.
Unfortunately, the majority of rips fall over EAC's normalize threshold, which means the volume is decreased, which ends up truncating some of the bits (assuming the album was using the full range of the 16bit audio).
Edit: I noticed you used the term "fidelity", so you might be talking about being able to perceive the damage to the audio, which is not necessarily what I'm talking about.
The point of EAC (Exact Audio Copy) is to make archival quality copies of audio CDs, this means as close to bit-exact as possible. Modifying the audio in any way would be counterproductive to that purpose, even if it doesn't necessarily affect the actual sound quality.
The CUE data in TLMC is a real bummer. I don't know if it's my music player, Clementine, but a good portion of the tracks I listened to in TLMC didn't advance correctly and instead just kept playing with the metadata of the first song in the album that I listen to. At this point I've mostly just given up on it... It took me 3 months to download it too, haha (curse you, Comcast 1TB monthly data limit!)
Yes: you can use Audacity (https://www.audacityteam.org/) to "diff" two waveforms by subtracting one from the other.
For comparing .wav files you can also treat them as plain binary data, and use standard diff tools. For example I used `cmp -l` to count differences between rips, and `vbindiff` to view them.
Thanks for mentioning vbindiff. I examine/compare binary files a lot (reverse engineering for security) and I've used a bunch of tools, but somehow I've never come across vbindiff.
First thought: Invert the phase of one of the tracks 180 degress, then sum them together. Anywhere the is a bump in the waveform would show a difference between the two sources.
Funny, you can use this technique sometimes to pull clean vocals (for remixing) from certain kinds of music (they have to be panned center in an otherwise stereo mix):
a) reverse one of the channels and them sum them and you'll sum will contain only stereo data
b) reverse and sum (a) with the original track to get only the mono content
Finding the signal difference is easy, the nature of PCM means we can just subtract one recording from another.
However human hearing doesn't just import the signal, the initial processing is done during acquisition itself. Psychoacoustic lossy audio encodings like MP3 rely on this, an MP3 sounds to you very much like the original but a naive audiodiff would find huge differences.
As a fellow ripper of physical CDs in 2018, the point is that a correct rip of CDDA from the CD is the one and only way to be sure you won't ever have to rip that music again.
Any other file, any other delivery/download mechanism, any other format ... you'll be fooling around with that song again sometime. But not if you have the correct, error free rip of the WAV/PCM from the CD.
So it's really not about the sound vs. some other format ...
> you'll be fooling around with that song again sometime
I've been ripping in AAC 256kbps for 15 years now and have yet to see any reason to revisit any of those CDs. Even if a more efficient encoding mechanism gains mainstream support, I'll just use that for new rips only since storage is cheap enough.
It's not that cheap though. My laptops 1 TB SSD can comfortably store 60 GB of AAC files and I forget they even exist, but if those were uncompressed, the 300 GB they take up would be a major problem.
I'm aware - my collection is nearing 1 TB of lossless music at this point.
However I store it on a local NAS as well as a an external SSD (in case I need it in a portable way, which rarely happens) - I opted for an expensive SSD because it's very light, small and portable, but a traditional external drive would handle music just fine if price is a major concern.
At the end everyone has different requirements, but since I consider ripping a huge amount of music a major task I wouldn't want to repeat, I'd make sure I do it right from the beginning - and opting for a lossy format doesn't fit into that idea.
also being lossless you are futureproofed on new formats you might need to convert to.
I mantain both: a tree of lossless files and an exact copy already encoded as 320kbit mp3 that is automatically mantained (just because i can quickly cram those into a usb thumbdrive and they'll play anywhere -- i can't say the same about flac support).
How do you _automatically_ maintain a shadow-library? I always argued against the parallel-collection-approach as you'll likely run into inconsistencies while trying to maintain it manually.
I'm glad iTunes offers to transcode my lossless collection to lossy AAC on the fly before syncing to the iPhone (yes, some people still do that).
Of course my concern is storage in mobile devices. Yours seems to be flac support? However is that really an issue these days? I use ALAC (due to using iOS devices), but even then I never run into compatibility issues. Any worthwhile software/hardware supports either format - with Apple being the big outlier requiring ALAC.
> also being lossless you are futureproofed on new formats you might need to convert to.
Part of my original point was that I've been using AAC exclusively for 15 years now, and with all the content encoded in AAC now I don't see support for it disappearing in the next 15 either.
As another fellow ripper of physical CDs - is there a good way to check for quality of my rips besides looking at the bitrate and listening to all of them (talking mp3s here, of course)?
I'm not interested in perfect lossless quality, but I'm quite sure I should re-rip the CDs I did 10-15 years ago, but for VBR mp3s it's kinda hard to decide what is 'good enough', as compared to say 128kb/s rips - so I'm looking for a scriptable way to give me a rough "keep/maybe/definitely throw away" metric...
Rephrasing the question: What are people typically using these days when not going for FLAC? Some of my mp3s are definitely from the "HDD space is expensive" age 15+ years ago...
MP3, with LAME (-q0). A couple of the really old rips used average bit rate instead. Will probably re-rip those at some point, but the rest sounds fine to my ears.
The article says "there were about 20,000 bytes different betwen each rip", which if it happened in one stretch corresponds to ~113ms of audio. Depending on the exact nature of the error and how it changes the samples, it may be very obvious (e.g. a 113ms burst of silence or static) or inaudible (one different bit in a few hundred thousand --- probably won't even get past the lowpass filtering on the DAC.)
That 20,000 bytes might not be a real difference, as I understand OP. It sounds like it's the XOR or Hamming distance, byte or bitwise - in which case you could get very large differences in the number by very small errors like dropping 1 bit (thereby putting all following bits out of alignment) yet the perceptual difference would be unhearable (and maybe not exist at all depending on how the encoding works). eg 0101010 vs 101010. You'd want to calculate some sort of edit distance which takes into account the possible bit flips and burst errors to see how big the real delta is. And of course, as pointed out in the other comments, for all of OP's hard work, there's no way to know which source is right or how many errors remain in his final version.
This matters for long-term music archival when you want to sample it and play with the audio, which could exagerrate the defects as you edit it. We should preserve pristine source material.
Are you aware of CUETools, CUERipper and the ctdb for ripping and repairing? Just wondering if it would have helped your case. It sure helped me recovering some damaged old cds.
Available at http://cue.tools/wiki/Main_Page
Not the author but a very happy customer
If the TLMC is lossless and gapless, then you could paste the last few seconds of 2, all of track 3, and the first few seconds of track 4, then search for an offset that yields a minimum diff.
He has no idea if he has an exact copy of the actual track. He has a copy of the data that is consistently ripped by his drives. There is nothing to assure that these are the same. Sounding fine -- for some values of fine -- is the only way to tell if it's correct -- for some values of correct.
Sounding fine says nothing, it's rather that not sounding fine tells you something, namely that ripping failed. Having some of the least significant bits of a sample flip for example is inaudible and will sound fine.
Sure, op has no absolute guarantee, but the fact that his crc32 matches that of the AccurateRip DB is, apart from his statistical approach, another strong indicator that he actually got it right. The alternative is that either the other user who submitted the crc32 to the database coincidentally ended up with exactly the same read errors, or that it's a hash collision (not entirely unlikely given the size of the checksum).
He didn't get a match against the AccurateRip database:
> That single AccurateRip entry for this album matched my CRCs for all tracks except track #3 – they had 0x84B9DD1A, vs my result of 0xA595BC09. I suspect that original ripper didn’t realize their disk was bad.
I've enjoyed the Touhou music fandom for a good many years now. I don't play the games, I just like the music - it's fascinating how big and diverse it is while still being unified around a canon of Zun themes/melodies.
It reminds me of traditional Japanese court poetry, where half the esthetic work is done by knowledge of allusions and borrowings and the poet is trying to express a familiar theme in a slightly better way rather than seeking for novelty like most forms of literature.
A very large lossless (.ogg) collection of doujin [0] Touhou [1] music. Googling the name should have taken you to the website [2] which might provide a bit more information. It's split into three torrents: One for just the music (1.65TB in size), one for Album Scans (76.5GB in size), and another for extras like lyrics, bundled item scans, and other misc. extras (24.2GB in size).
Or in simple terms: A large collection of cover songs for the soundtracks of a popular Japanese bullet-hell game series.
Thanks for the correction. I converted everything to .flac to maintain consistency with my personal library, so I must have forgotten which other lossless format I had converted from.
Whether the emoji is colour or not, depends on the combo of Browser+OS+Font. On Linux, pretty much every mainstream browser renders emojis in colour. On Windows, it's a bit more complicated: Windows 7 always renders emojis in black and white regardless of the browser, whereas Windows 8.1 and Windows 10 render them in colour. Unless it's Internet Explorer, where they are in black and white again.
A LOT of sites also do 'on-the-fly' unicode emoji swap-outs for little png images. Twitter and Facebook are a good examples of this. That way the emoji is always in colour regardless of the platform.
at least on Linux, it's much less a factor of the program used, as long as it uses freetype (i.e. all programs since 2000 except for crappy Java programs), and much more a factor of simply what fonts are installed by default on the particular distribution. on Arch for example, no emoji fonts are installed by default. in fact, no graphical fonts at all are installed by default, but installing a browser will pull in a ttf-font, by default ttf-dejavu (which also does not contain emoji characters).
I saw an emoji-decorated reddit thread about the new release of a library I really like.
I commented how maybe emojis are not appropriate for serious code releases,
I was ignored.
The next release was also announced with an emoji-decorated title...
Egyptians hieroglyphics are phonograms though, not [very loosely defined] ideograms -- I thought that was the point about their translation, that people wrongly assumed they were pictorial ideograms.
As the other comment said, why can't they be complementary? Emojis are great for handling subtext and seem to (partly) offer a solution to Poe's law. That being said, I too can't take anyone who uses emojis in his comments seriously.
Note that the availability of emoji domains is limited. As of August 2017, there were eight top-level domains for which registration of emoji domain names was possible, all of which are ccTLDs: .cf, .fm, .ga, .gq, .ml, .tk, .to, and .ws.
You must have missed the recent additions of 'unvote'/'undown', deleting posts (I am not sure anymore, but don't think this was possible a few years ago), limiting edit time, much fewer (in my experience: no more) 'no such thing' errors when the cache expired, collapsible comment trees, etc.
These are all small improvements that made it a lot better. That the look hasn't changed for as long as I'm here, I'm quite happy about. If the alternative is something with loads of whitespace and Javascript like Reddit, Skype, etc., then please let us not update "the comment system from the early to mid 2000s".
It supports unicode. And indeed, for a while you could post emoji, until they were added to filters disallowing them. You can still find old posts containing them, rendering correctly.
Browser vendors have instituted a variety of different rules for this problem, include at least:
1. Decide (policy, enacted by humans) for each TLD if its registry has rules that will prevent abuse, if so whitelist this TLD and show IDNs as text for this TLD, everything else is punycode.
2. Algorithmically detect "confusing" IDNs and show punycode instead for those.
Yeah, it’s also a common example for why showing punycode for IDNs is a super bad idea.
e.g., take fluege.de (a flight comparison site) and flüge.de (a competing flight comparison site). The plain-ascii transscription of ü is ue.
Now, flüge.de can’t buy fluege.de, or use that domain, and has no alternative domain.
But if you show the punycode version in the URL bar, you just increased the phishing risk significantly, because you’ve trained users to enter their payment data on a xn-randomblabla.de page.
But I think CDDA possibly uses different (less) ECC... so info in links might not be 100% relevant.
It's a shame this cool tech is not being used anymore. As an information theorist, I used to be able to point to optical disks as an application of my field, and be like "see all that math is useful for something," but now I don't have anything shiny to point to anymore :(
> As an information theorist, I used to be able to point to optical disks as an application of my field, and be like "see all that math is useful for something," but now I don't have anything shiny to point to anymore
I strongly disagree, there's many shiny things you can point towards.
Point to a smartphone (and also the relevant sections of the LTE/HSPA/UMTS/WCDMA/GSM / 802.11 standards documents); there's a literal panoply of coding/error-correction maths that's crucial to every single one of those standards. You can actually go to the ETSI website and find the LTE standard and get free PDFs with the exact parameters of all the codes they use, in enough detail to reimplement them yourself.
In those standards, there's all sorts of lovely little state machines for coping with errors, like LTE's "HARQ loop" where your phone will tell the cell base station that it didn't receive a chunk of data correctly; at which point the tower will resend a differently punctured version of the chunk of data, and your phone's modem will try to soft-decode the data with both versions to work with. Oh, and that exchange (including processing on both ends and radio transmission latency) takes under 10 milliseconds to complete -- the standard places an exigent and strict deadline on how long your phone has to respond with its acknowledgement/request.
Also hard-drive platters are extremely shiny (more so than optical disks) and those also use error-correcting codes, as do SSDs. Did you know that bit cells in modern SSDs are so small that the number of electrons it takes to affect a measurable voltage difference for the sense amplifiers is less than 100 (https://twitter.com/whitequark/status/684018629256605696)? All your data lives in those differences of tens of electrons per bit cell; no wonder there's ECC machinery hard at work in SSDs!
> ...like LTE's "HARQ loop" where your phone will tell the cell base station that it didn't receive a chunk of data correctly; at which point the tower will resend a differently punctured version of the chunk of data, and your phone's modem will try to soft-decode the data with both versions to work with.
Cooool.
I googled "HARQ loop" and because I visit HN so much the first result was https://news.ycombinator.com/item?id=11151232, so I learned that the data-processing portion (where the decode attempt is retried) must complete within 3ms!
I've been wondering for some time about good (bandwidth-efficient) ways to do error correction/recovery in the area of general-purpose high-efficiency byte transports, and just in case, wanted to put this here in case you're/anyone is interested. How good would it be to throw TCP's "flood of ACKs" out the window and instead compare frame checksums (is CRC32 good?) of every say 32 frames, and send a bitmask (in this case 8 bytes) every n frames noting which were correct and which aren't? This would send ~32 times less data (and I just learned a TCP ACK is approx. 74 bytes! https://stackoverflow.com/questions/5543326/what-is-the-tota...)
(My interest/focus is within the domain of good ideas lacking widepread implementation/uptake. I've found that these always seem to be hiding in the woodwork.)
Your TCP ACK approach is confusing. If you mean to check all the frames, but only say if they're OK every 32 frames that only makes sense if you somehow have a medium with high bandwidth AND high error rates. If your error rates are low conventional TCP will just send occasional ACKs saying e.g. "Yup, got the last 500 packets, keep it up".
But if your error rates are high and your bandwidth is high you should already know how to fix this and get very slightly less bandwidth without errors, so why is this suddenly TCP's problem and not your transmission medium?
Because I for one can't change the DOCSIS parameters my ISP set for the segment I am on, and neither can I affect radio communications reliability without impaired movement.
While the error detection is rather weird, and a simple erasure code might well work better, this is a good idea.
Thanks for the suggestion. I've been meaning to take the time to stare at how erasure coding works and keep staring until my eyes stop glazing over and I get it :)
Thankyou for helping me understand that I don't actually understand how TCP works :) I had no idea ACKs apply to the last n bytes as a group - although now I think about it, it makes a lot of sense that no other approach would be viable (even with 80s/90s era line rates).
I want to design a protocol capable of withstanding both high and low error rate scenarios; everything from usage on 64kbps satellite to flaky home Wi-Fi to wobbly 4GX cellular performance around tunnels and buildings. The (slightly pathological) application I have in mind (mostly as a theoretical test) involves video and interactivity, so basically requires simultaneous high throughput and low latency (essentially worst case scenario). My goal is to build something that responds well to that, perhaps by being able to have its performance metrics retuned on the fly.
I read somewhere a while ago about how TCP is a poor general-purpose catch-all protocol, and that eschewing it where possible (cough where there aren't any proxies) will generally boost performance (if the replacement protocol is well-designed).
It would be awesome if I could to assign significance to and act on each incoming network packet. This would mean I wouldn't need to wait for enough packets to come through and be reassembled first. So packets could arrive out of order or get dropped and I'd be able to recover the data from what did make it through instead of throwing what I did receive away and just starting over.
Yes, a lot of work; I'm trying to figure out if it's worth it depending on the situation.
My original musing was about TCP ACK. If I'm following correctly, I now understand that apparently TCP ACKs apply to 728KB of data (1492 * 500 = 746000), so if a failure occurs... does 728KB of data get resent?! I presume not where the link is slow or congested or has a high error rate, but if that's still happening where a lot of data is being sent, that still feels quite high to me.
Thanks for replying; you helped me properly think through my idea and realize it would never actually work: I hadn't realized that I was assuming packets would arrive in order. If I use a 64-byte string to represent a 512-bit bitmask of ok/fail checksum tests, that would only work if both sides of the link are considering the same set (and sequence order) of packets.
Another thought, with far fewer moving parts (/ chances for imposition/assumption...), was to simply report a running success/failure percentage, with the remote end using that information to get as close as possible to 0% errors. That doesn't narrow down what needs retransmitting though. Back to the drawing board, then...
I have no delusions of reinventing TCP/IP, and I'm not even interested in reinventing the wheel. I do find networking very fun, but I guess I'm still working on understanding how it works, building an accurate mental model, and (in particular) honing/tuning the (very important) sense of knowing what solution approaches will work and what will fail.
When you have no idea about something, time spent envisioning how to improve it is time entirely wasted. "Hmm, I wonder if elephants are used to pick chocolate bars from tall trees. Maybe we could improve production by creating a type of stilts for elephants that makes it easier to reach upper branches. Ah, but when the elephants fly home to Canada, would the stilts interfere with their wings?".
TCP has Window Scaling, allowing high bandwidth links to have up to 2GB or so on the wire, and Selective Acknowledgement, allowing recipients to acknowledge ranges of correctly received packets if some are lost. I picked 500 entirely out of the air, actual numbers will vary enormously.
A TCP/IP "stack" automatically tunes the settings for each connection, if your stack is modern it will do a pretty good job. Work on improving this automatic tuning in consideration of both theory and real world practical experience is an ongoing topic of network engineering.
I non-rarely have links with ~0.1% packet loss. If you get even just that much packet loss, and a long-distance link, you _very_ quickly drop below 10 Mbit/s, if the other side can't do fancy selective ACK and similar additions. While I might be able to do something against this, as long as I control one node on both sides of the lossy channel, I can compensate with a little bit of FEC. I might even be able to operate lossless, using a fountain code to add more parity for each data bundle as long as my recipient node continues asking my sending node to supply more parity.
You are right though, for unicast networks there is little use of L3/L4 FEC. Maybe with the age of IPFS and other content-addressed networks we will come back to broadcast-FEC networks, like those used back when for 3GPP multicast services like e.g. TV.
> You can actually go to the ETSI website and find the LTE standard and get free PDFs with the exact parameters of all the codes they use, in enough detail to reimplement them yourself.
Does this include things like turbo codes? I've been curious about FEC for a while now
A CDDA has 2352 bytes of main data and 96 byte of sub-channel data per sector, so a total of 2448 bytes per sector. A sector is made up of 98 sub-frames. Each such sub-frame has 24 bytes of main data, 4 bytes of C1 error correction and 4 bytes of C2 error correction, and 1 "byte" of sub-channel data. The first 2 sub-frames have a special pattern in the sub channel data to recognize the start of the sector, leaving 96 bytes of sub-channel data available for other things. The error correction only covers the main data, the sub channel data just has a CRC.
The standard API to talk to the CD-ROM does not have a way to get the C1 and C2 error correction data. There is only a way to get 1 bit for each of the 2352 main data bytes after the C2 error correction to indicate that there was an error with it or not. But getting that data doesn't actually seem to be useful on the drives I tested it with, they all seem to have weird firmwares.
> As an information theorist, I used to be able to point to optical disks as an application of my field, and be like "see all that math is useful for something," but now I don't have anything shiny to point to anymore :(
Surely there's some forward error correction built into, well, pretty much any physical media these days? I can't imagine BluRay would need less of it than a CD.
I'm sure pretty much all storage media (SSD's and such) have it, as well as internet protocols and digital media (now that having storage locally is becoming less cool)
High-bandwidth Ethernet uses forward error correction, but I'm not aware of any higher-than-physical internet protocols that do, unless you count erasure codes a la QUIC
IP, TCP, and UDP all have checksums, though (a) that's less powerful than error correction and (b) the above are weaker than the Ethernet checksum (Ethernet is 32 bits while the others are 16 bits, and also I think these three are all addition- or XOR-based instead of a polynomial CRC).
> As an information theorist, I used to be able to point to optical disks as an application of my field, and be like "see all that math is useful for something," but now I don't have anything shiny to point to anymore
Xiph.Org has a command line program called cdparanoia, which is built with the same philosophy as Exact Audio Copy (EAC). The FAQ page explains a lot about why the program is necessary.
Thanks for the name drop. I went Linux-only on my primary laptop, but I still have a Windows 7 laptop with EAC installed that I dig out every time I buy a new CD and need to do a rip. Maybe I'll finally be able to put it to rest, soon.
I've found that EAC works really well in Wine, and it's still my favorite ripper, even on Linux. Similarly, Foobar2000 in Wine is the best Replaygain scanner/tagger.
cdparanoia was the cornerstone of my favorite CD ripper back in the day. The package was a Perl script that would
1) watch the CD drive for a CD.
2) When one appeared, launch cdparanoia to rip the tracks
3) Grab some ID key from the CD and look up the 'album.'
4) Encode the tracks to MP3s, renaming to the information fetched about the album.
5) Copy the album directory to the desired target destination.
6) Eject the CD.
7) Repeat from 1.
(I may not have the exact order - I just needed to watch for it spitting out one CD and swap in the next.
I used it to rip a CD collection consisting of dozens and dozens of disks. It was a true zero click interface. It took no command line arguments - customization was via code editing - so it was not particularly user friendly in that regard.
Nowadays look for the downloads of purchased CDs on Amazon or use some GUI app - at least it retains settings from one execution to the next.
Do you by any chance mean abcde? I just used that a couple of weeks ago when I noticed that I still had a few CDs that I had not ripped.
A custom CD ripper script was the first program I ever wrote that actually served a useful purpose. Back then, somebody told me not to bother and just use abcde. 16 years later, I have finally come around to follow that advice. :-)
The most modern CD ripper frontend for Linux I've found is whipper, which is a fork of morituri. It also has detailed log files, no ideas if "certain sites" accept them or not.
Due to dependency shenanigans I couldn't run it on Debian so I created an LXC container on my server running Ubuntu 18.04, passed the /dev/sr0 block device and it worked.
I'd be curious how well cdparanoia handles his CD if at all. Furthermore I read a post on a forum where someone tested a bunch of drives with scratched CDs and the Samsung SH-2xxxx drives performed the best. No idea of the exact methodology he used.
(Before you ask. I didn't buy this drive based on his post, bought it 7 years ago for my desktop on a whim) Still gotten 100% track quality out of whipper with some minor scratched discs.
They can parse it, but seem to want some proprietary plugin to create signed checksums to prevent (though in reality only for the most unknowledgeable) log spoofing.
Does the default log file list the locations of sectors that failed to read? There's also a question of interpolation for obliterated sectors. Many would prefer there to be no data than interpolated data because that's a much more obvious failure mode.
EAC was the standard because it was the only free tool that could bypass firmware error correction (with 'secure' mode) and log the whole process securely, in a way that could not be tampered with easily.
Well, that isn't the program's fault. There should be no series of commands it can send that would destroy the CD drive--I bet it wasn't even running as root.
« EDIT: After further investigation, I no longer believe it’s a factory defect. If I write the beginning or end of the affected track to a blank CD-R and rip it, the rip fails with the same error! Give it a try yourself with minimal.flac. »
Uh... so what's going on there? Something that breaks the error correction algorithm, or what?
It is likely "weak sectors", the bane of copy protection decades ago and of which plenty of detailed articles used to exist on the 'net, but now I can find only a few:
OK, wow! So basically CDs are not capable of recording arbitrary data, or at least most burners will fail on certain inputs, and the song in question causes trouble. Fascinating!
This comment just made me realize that I haven't had a system with a burner in it for probably about 10 years.. I'd be SOL if I needed to burn a CD for some reason.
My mid-2000s car has a CD player but no aux port. Instead of going out to buy CD-Rs I asked friends on social media for any spare CD-Rs they were willing to get rid of. Lo and behold I now have over 400 blank CD-Rs that I didn’t pay a single cent for!
I actually had to rip out the burner from my last computer and install it in my new computer the other day to help a friend try to diagnose an issue he was having with burned DVDs of his own work.
Sadly, I wasn't any help other than confirming his issue.
I'm not very familiar with telecom or radio protocols, but I think the extra information in an LTE packet (the parity bits + redundant data bits) might be critical to that sort of error correction working. The raw PCM coming out of an optical drive doesn't provide that kind of metadata, so at best I can count how many values appear at each position and try to guess on that. If the values after 20 rips look like this:
CD1 position 0xFFFF
A0: XXXXXXXXXXXXXXX
A1: XXX
A2: XX
CD2 position 0xFFFF
A0: XX
A1: XXX
A2: XXXXXXXXXXXXXXX
then I'm not sure how any sort of combining could usefully recover the original value.
You have to look at the effect of the noise on the statistics of the bits, since those are what's physically changing, and not the PCM output after all the error correction layers.
> You have to look at the effect of the noise on the statistics of the bits,
Yes. Soft combining (in systems designed for it) isn't even done on bits, it's done on the actual analog (well, a digitised version of the analog value, sometimes termed a "log likelihood ratio") values of the signal in question -- before any quantisation happens.
This isn't to say that you can't apply the techniques here (where stuff obviously gets quantised unless you can find a way to get the raw signal from the photodiodes), but you need access to bits with the least amount of error-correction involved (or find a way to infer what the statistics of the original raw bits are from what the decoding machinery outputs).
That assumes the noise is uncorrelated though (which the noise on a camera sensor generally would be). If the noise is correlated then averaging doesn't help.
There is lots of noise you don't get removed by averaging, ie the base noise of the sensors. For those you usually (for amateur astrophotography) take four sets of pictures (usually about 20); with the lens removed and cap on the mount, lens mounted with lens cap, lens mounted into telescope but telescope cap on and lastly telescope aimed at target but a thick cloth over the opening.
All those compensate for the various ways light may enter the camera (lens mount, lens frontmount, telescope housing, telescope lenses and mirrors) and let you subtract those out after averaging your result.
I really don't think that's the same thing. Taking multiple shots of the same subject is just taking a shot with a really long shutter time, except in slices, a little bit at a time.
What OP was talking about was something akin to using the noise in a photo of one subject to reduce noise in another photo of a completely different subject but with the same camera.
In astro-photography, this is quite common. You take a series of normal images. You then cover the the lens, and take the exact same exposure. After a series of processing, the noise pattern from the dark frames is subtracted from the normal images. It works quite well.
FWIW, some cameras can use that technique to help de-noise long exposure shots. The control image is a long exposure with the shutter closed, and that image (which should be completely black) is subtracted from the long exposure shots.
It's only the same as a really long shutter time if the subject is stationary, and in the case of Earth-based astrophotography it never is, because atmospheric turbulence is always moving the image. If you take multiple separate shots you can combine only the good ones:
I really hope there are a few people still out there that have the full catalog of what.cd archived and either keep it alive by sharing it in the darker recesses of the internet, or are able to secure it somewhere.
There really needs to be a media museum, an international library of all that has ever been produced, from a hundred different recordings of the old composers to whatever crap your 15 year old neighbour is uploading to soundcloud.
If its anywhere like the scene and DCPP used to be then there's archivists who have the extremely rare content, including for example MP3.com rarities. A full archive is very convenient, but if you're a collector who's after certain rarities all it requires is a good network (or networking skills). Social network, and some bandwidth.
It's nice to see that people still remember OiNK and what.cd. Of course, I know I can't be the only one who remembers, but I've never met someone in real life who does. Both shut down thanks to corruption in British and French police.
two years ago, i think? i stopped torrenting music when i got a streaming subscription, because i discover music easier if it's streamed and i'm not enough of an audiophile to discern between 320kbps mp3 and flac.
This didn’t seem to be an issue of wear or damage – the CD itself was probably defective from the factory.
It would be very interesting to look at the surface of the disc under a microscope, to see if you can find the defective area --- knowing which track it's in, you can determine the approximate diameter at which it occurs, and then rotate around that diameter looking for abnormalities. The pits and lands are invisible to the naked eye but easily viewable with a light microscope:
At 1200 bits/mm linear density of a CD, the 5KB (40Kbit) defective section will correspond to ~33mm of track --- probably quite obvious if it is a physical defect. (A "logical" defect, where the bits are OK but the ECC is wrong, will not be apparent at the physical level.)
I don't have the equipment for this, but it might not be needed. The disc has writing that goes "through" the top cover, such that you can read the text by holding the disc up to a normal lightbulb.
The text is across the entire diameter and I don't know why it would cause problems for only this track. Possibly the shape of the text?
That could be an area with insufficient contrast, a similar problem that early drives had when CD-Rs started appearing. In that case I would've tried a mirror or black pad (about the only thing those audiophile "disc mats" are good for...) on the top, to shift the offset level.
Note that CD uses 780nm infrared so what the human eye sees is not necessarily what the drive sees --- this is why "transparent" or "black" CDs work.
Fry's Electronics should pay this man for the plug at the end. He made a good case to go there if you need to conduct an experiment requiring an unreasonable number of unique disk drives right now.
This reminds me of trying to rip Paul Simon's Concert in the Park some number of years ago. Everytime I ripped it, there was a minor glitch in The Obvious Child at 2:02-2:03. I also heard it on the CD. So I tried different drives, different machines, Linux, Windows, iTunes, Exact Audio Copy, etc., to no avail.
Finally I reassesed my assumptions and found a copy of the track in the wild -- hey! Same glitch. Then I emailed a friend who I knew also had a copy of the album -- same glitch!
Eventually found an obscure forum where someone also complained about the same glitch.
I always wonder how that made it out of the studio -- was it a glitch in the recording equipment of the concert? Surely they used multiple recorders?
Fun times. Once upon a time I was working at a university in New Zealand. At that time Apple had created a way to legally stream your iTunes catalog to other LAN users—mostly as a way of enabling the college campus music sharing experience without the illegalities of OG Napster, Limewire, and the ilk.
I got a nastygram from the IT department for sharing my iTunes catalog. I replied that it was my understanding that the iTunes streaming was legal. They concurred, but said that format-shifting my albums had been illegal. I countered that I format shifted my CDs in the US, where it was legal.
In the end, the IT security people agreed that I was probably totally legal, but they asked me to please refrain from leaving the iTunes sharing turned on because it made their lives hard. That was a compelling argument, and so I turned off iTunes sharing. :-)
EDIT: I should note that format shifting was later made legal in NZ last I knew. Hopefully it still is.
It is in the UK, parliament made it legal, but media interests realised parliament had erred and the people were saved from the grave harm of format-shifting media they had paid for.
What a beautiful story of devotion! Imagine someone going to this much trouble over your work, it's quite impressive.
I used to worry about this kind of thing, using cdparanoia and so forth on Linux to manage my music. When I went Apple I eventually went iTunes, which is a mixed blessing. I feel like it has made me able to appreciate more music more easily by lowering the cost barrier, but it has reduced the amount I appreciate an album substantially. And it's hugely inconvenient when they disable my account for strange reasons or my internet connection vanishes. So I may return to manual management someday.
I recently found a backup of an old portable mp3 player and realized that the manual curation and limited storage really made me appreciate my collection more. Gave me a wonderful warm feeling going through that collection.
If you want to avoid having to do this you might want to look in to embedding error correction data on to your CD/DVD (or even on to separate media) at burn time, using dvdisaster:
In my experience: as the box size gets bigger, the treatment gets rougher. Single CDs in bubble wrap envelopes come through fine, as do small hardwall boxes. CDs in medium boxes (10-20 CDs) often arrive with box damage and maybe some torn internal packaging. I once used a shipping aggregator that put 50 CDs in one box, they were so banged up that I had to buy replacement jewel cases.
If you ship CDs, I have only one request: please please do _not_ put bubble wrap _inside_ the jewel case! Some sellers do this and it just destroys those little plastic retainer teeth.
Thanks for the feedback. Bubble envelopes don't really protect CD jewel cases very well, which is why I create my own packaging by wrapping the cases individually in recycled cardboard. Simple and cheap. CDs are amazingly resilient and with a proper polishing machine scratches can be completely removed without changing the data on the disc. It is damage on the non-playing side of CDs which is impossible to recover from.
the time of utter popularity has passed, but it is still plenty popular both in the world and Japan specifically. There's a few dozens conventions and Touhou music concerts/DJ-ing per year, the biggest of which is the Reitaisai and its accompanying events. And a new spin-off game is coming out this August!
As a side note and not necessarily related, one of the devices I keep using in case I need to read a (data, not music) CD is a very, very old CD-ROM drive, of the original rype that use a caddy cartridge, 1x in speed, with a SCSI connection.
Till now it never failed on discs that current, modern and newish fevices had issues with.
AFAIK the whole doujin scene is all about "go on, violate our copyrights but don't complain when people violate yours".
ZUN's blanket ban on commercial distribution makes it really difficult and expensive to legally obtain anything when the only distribution channels are specialized doujin stores -- it's still commercial, but the ones getting the profits are various middlemen and not the arrangers and performers.
ZUN has granted blanket permission for fanworks distributed in the spirit of the original games, but explicitly forbids distributing arrangements of his music through mainstream sales channels (music stores, iTunes, HDTracks, and so on).
Maybe ZUN himself is cool with people distributing his work on the internet, but I don’t think that all the doujin circles are cool with people distributing theirs.
Is, uh, there a version of this, perhaps called the Touhou Lossy Music Collection, for apparent plebes like me who don't care about codecs discarding data that I can't hear anyway?
I have about 200 DVDs, most 10 years old. Probably 1/4 is unreadable by now: MacOS does not recognise the disk when inserted (oddly enough, much older CDs still are).
The data is not terrible valuable and probably already mostly stored elsewhere, but I'd like to give it a try. Is there hope for ddrascue? I've had mixed results in the past. Are there better alternatives?
If you take away one thing from the article it is this; try different readers. Some readers perform far better than others. I have one 'magic' DVD reader that will overcome damage that makes most other drives choke with errors, and the result is usually a flawless rip.
It's very likely! There's a couple approaches one can take to detect this:
* The rip log generated by EAC includes a CRC of the audio data. I can calculate this independently to ensure the data is the same as what EAC wrote.
* The rip log itself has a checksum that can validate the CRCs are intact.
* I archive my rips in Google Cloud Storage along with a sha256. If my NAS's copy goes bad I can fetch the backup, and validate that it's the original data.
* Use a file system with strong data checks and repairability, basically meaning, ZFS with a sufficient redundant set up. "zpool scrub" can do wonders, and you can guarantee backups to a different pool are identical to the source.
In the scene they've used SFV (CRC32) for ages. It works very well and quick except it isn't useful against tampering. PAR2 provides additional parity, and it also useful if for Usenet. You can also use RAID. For example, Synology NAS easily allow RAID1 plus btrfs.
What I'd do is the same thing I do for scratchy records. Look for the scratch in a waveform editor. Interpolate the data across the scratch using cubic splines. You can also paste in data from the other channel if it is similar enough.
True, but he was using statistical techniques, which are not guaranteed to produce the actual data.
My technique is akin to what an artist would do in "restoring" a damaged photograph. Often the glitch is only one data value being off, and it does produce an audible click.
Thrift stores are a cheaper and very reliable alternative to Fry's, if you are ever looking for one. I've never not seen a CD/DVD player at a thrift store.
Good point, I’m curious if older drives produce better results. I remember some discussion 10-15 years ago saying that modern floppy drives were junk compared to older drives. The argument was that the prices dropped so much that build quality also suffered. I wonder if the same applies to CD/DVD-ROM drives.
It has always struck me as odd that the powers that be have yet to figure out how to harness the love (obvious since they toil to preserve these media works at the best possible quality, albeit illegally) of data hoarders.
I see that no one has mentioned yet cuetools and the cuetools database in this thread. That seems the perfect tool for this kind of repairing (provided someone else has ripped an accurate version in advance, of course)..
Whereas the "yellow book" CD-ROM standard got to take advantage of over four years of technology advancement. Plus it doesn't have a real time requirement, so there's plenty of time to calculate more advanced checksums and error correction codes.
In many respects, an audio CD is more like a vinyl record than a data storage format.