The main bottleneck is the machine learning to extract metadata, the transcriptions are actually handled very fast at an early stage.
Someone mentioned that it might be a good idea to create a federated network where people just link their feeds and share them with each other on demand, which I have no real experience with but I think could work quite well.
I've only been working on this for a few months, so this is all still very experimental and there's no way I can build something like that in a short amount of time, but we'll think of something that makes sense and gets people as much access as possible.
What sort of metadata are you extracting with machine learning? Doing more than simple keyword indexing? I'll admit, I don't know very much about the particulars of building a large search engine.
How large is the ML model that you're using to extract the metadata? If that can be done in a distributed way, then it might let people pre-process their own feeds and share them with others -- either on-demand, like you mentioned -- or caching them centrally so that your servers are still the authoritative repository.
Anyways, everything you've built here is wonderful, and I definitely want to keep tabs on your progress and support you however I can! Thank you for sharing!
We take the transcripts and run natural language processing models on them to find keywords, n-grams, and pronouns. We then categorize the pronouns according to type (person, organization, media, location, event, etc.)
The search engine itself is just transcripts in Elasticsearch.
From the feedback I've gotten so far, I think the best thing to do is actually integrate this with YouTube through a browser extension, and have users call the backend to do this processing on-demand (with caching of course). It seems the video breakdowns are more important than searching for most people, since with YouTube you can do some kind of content searching, just not very transparently.
Someone mentioned that it might be a good idea to create a federated network where people just link their feeds and share them with each other on demand, which I have no real experience with but I think could work quite well.
I've only been working on this for a few months, so this is all still very experimental and there's no way I can build something like that in a short amount of time, but we'll think of something that makes sense and gets people as much access as possible.