Hacker Newsnew | past | comments | ask | show | jobs | submit | css's commentslogin

I ran into this issue recently [0]. Apparently the integrated VSCode terminal sets its own (very high) cap by default, but other shells don't, so all of my testing in the VSCode shell "hid" the bug that other shells exposed.

[0]: https://github.com/ReagentX/imessage-exporter/issues/314#iss...


Yeah I ran into this too when testing a new feature. My colleague sent me this: https://apple.stackexchange.com/questions/32235/how-to-prope...

But I reckon its unreasonable for us to ask our users to know this, and we'll have to fix the underlying cause.


Semi-related, one of the `imessage-exporter` contributors provided a great write-up on reverse engineering the handwritten and digital touch message protobufs [0]. The reconstructed proto files are [1] [2].

[0]: https://github.com/trymoose/handwriting2svg/blob/0eb56cf4582...

[1]: https://github.com/ReagentX/imessage-exporter/blob/beeb853b2...

[2]: https://github.com/ReagentX/imessage-exporter/blob/beeb853b2...


The plist is probably a binary plist (header bytes `bplist00`) generated by NSKeyedArchiver, and then the specific data you need is encoded inside. Edited iMessages are stored in the exact same way. Luckily the plist itself is not that complex–but typedstream is pesky to work with.


The plist is no issue, but it's the values therein where you run into typedstreams. For every setting for the Script Editor's formatting, is a separate dictionary, with an NSColor and NSFont key set to a data type value. The data is a base64 encoded `streamtyped` file. Passing it through base64 decode and running `file` on the output gives back `NeXT/Apple typedstream data, little endian, version 4, system 1000`, just as in the OP.


The only reason I want to do this is because I wipe a Mac nearly weekly, and need it setup more or less the same way again. I could probably just drop the .plist in that directory and bobs your uncle, but I also would change the fonts Script Editor is using to a third-party font not installed, so I don't want to have to worry about weird order of operation BS, and also find a way to set it to any arbitrary font, as I often change out the "fixed width" font I use in all the editors for that week (I have favorites, not just a favorite, gotta keep it fresh, ya know).

I figured that since Script Editor, and the AppleScript components of macOS are so old and creaky, forgotten leftovers in the Yellow Box that no one bothered to fix. I had no idea typedstreams were still being used in modern Apple software.


iMessage uses a very strange amalgamation of typedstream (message content), keyed archives (app messages, sticker data), and protobufs (Digital Touch, handwriting) for different features. I wonder what motivated all of those design decisions.


This is stuff is such a PIA to parse. I assume it's just different teams doing different features over the years, and being alternately repulsed/seduced by each format. Probably features are implemented as libraries so there isn't a master oversight - they aren't trying to make iMessage's internal formats follow a consistent plan, just let all the libs coexist...


Maybe they should be repulsed, considering all of the journalists that are getting persecuted and/or murdered because they are getting pwned through iMessage serialization bugs :)


As someone who used to work on that team, it’s so interesting hearing thoughts from external public on the team.


I would love to hear your thoughts as an insider.


"Those who don't understand ASN.1 are doomed to reinvent it, poorly."

That said, it could be much worse --- JSON, or XML.


Apple provides Messages for Business [0], but if you have a machine that can read the iMessages as they come in, you could use the library [1] that powers `imessage-exporter` as a bridge.

[0]: https://register.apple.com/messages

[1]: https://docs.rs/imessage-database/latest/



I see how that helps with the usual case of inserting a value under the original key if it wasn't there, but I don't see how it helps in this case of checking a different key entirely if it wasn't there.


If you want something a bit more robust and cross-platform, including support for the undocumented TypedStream-stored text data, you can use my open source software: https://github.com/ReagentX/imessage-exporter


Awesome work!



Cool, is there any good OSINT info/tools for Chinese "world"?


666 is also a common number over here. Perhaps he's into heavy metal.


You can probably do this using the library: https://docs.rs/imessage-database/latest/imessage_database/


The California Poppy release [0] of `imessage-exporter` is the biggest update yet. Most notably, it implements a novel `typedstream` deserializer [1], allowing it to understand the data stored in `NSAttributedString` binary data that is stored in the iMessage database.

I've spent a long time reverse engineering nearly every aspect of Apple's iMessage SQLite tables to build this program. As far as I know, there are no other tools that support the full corpus of iMessage features [2], including edited messages, app messages, reactions, text range formatting, and threads.

[0]: https://github.com/ReagentX/imessage-exporter/releases/tag/2...

[1]: https://docs.rs/imessage-database/latest/imessage_database/u...

[2]: https://docs.rs/imessage-database/2.0.1/imessage_database/me...


FWIW, I have built something like this--including safely reimplementing the deserialization of that old "streamtyped" format, though not necessarily dealing with "the full corpus of iMessage features", as I thankfully don't experience people using newer formatting ;P)--and, to me, the hardest part wasn't/isn't dealing with a single database, or adding file format features like that deserialization: it is that the user almost certainly has a handful of these database files collected from a hodgepodge of devices and backups that are all slightly out of sync (as always happens for various reasons, both reasonable and unreasonable) and were last touched by different versions of macOS/iOS, and you have to merge all of their data into a single unified export. I found this particularly hard when going back and dealing with ancient database files from early versions of iPhoneOS, before "iMessage" even existed, as well as weird corner cases where old database files were on devices which happened to be upgraded to newer versions of iOS, as the migrations are sometimes a bit more awkward than a de novo message. Like, to me, "table stakes" for this kind of program is that it takes a list of database files, not a single one, or it is simply never going to work well. (I would be happy to help test such with my database files--though probably a tad slowly right now as I've been overly-booked recently--as well as, if it would somehow be at all helpful, to show you the somewhat disorganized code I developed, particularly so if you would be willing to support an SQLite or JSON output format... it would be nice to never have to maintain my version of this stuff again, as I only wrote what I have so far for my own personal use cases, and mostly in a frantic panic over a few days in 2021 when dealing with discovery for a lawsuit against Apple, not to distribute to random users, though I otherwise might some day if this use case is never fully served.)


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: