It's ridiculous that a public-facing agency like the FDA does not make every piece of data available in a database that is easy to access and downloadable. You can download all of PubMed in compressed files, ClinicalTrials.gov gives you both API access as well as a publicly available read-only Postgres database, and the National Cancer Institute has excellent publicly available databases. There simply isn't an excuse for requiring someone to spend ~4 months to download this!
Hi, author here! Completely agree, I would love to see the FDA publish an official dataset of predicate device relationships. I'm not sure if they have this data available internally, but if not, they could validate my dataset and republish it, which might be easier than starting from scratch.
You did not have to scrape the pdfs using the website link. They have bulk downloads indexed by year (IIRC). I remember grabbing these because I had to OCR a bunch of summaries to extract data for some NLP I was running. Also, IIRC grabbing the predicates was pretty easy w/ tesseract.
I do use this resource for 510k.fyi to populate the device and recall data, but unfortunately it does not contain the 510(k) summaries, or even links to them. The web scraping was still required to get those documents so that I could run OCR on them.
