I built LexGrep as a way to see how quickly I can prototype a working ML based project using the APIs which are widely available and hosted online.
- I first started with whisper to generate the transcript but eventually found a dump online where someone did the same thing :V (first step done)
- Next I tried speaker diarisation which is really easy to do in this scenario (as there are mostly 2 speakers at any time) and I could see Lex saying "welcome to my podcast" in almost all of his podcasts which made annotating the transcript easier.
- After this, its the usual create mini embeddings from a single podcast (I used OpenAI's text-embedding-ada-002) to generate the embeddings. I had to model the text in a specific way "This is a transcript between Lex Fridman and John Carmack.." (this adds the context for the conversation and helps powers query like: What did John Carmack think about {blah}) as the embeddings from the podcast with John Carmack will be closer than say some other podcast.
- Pinecone DB provided a free tier for doing cosine similarity of the generated embeddings which worked out perfectly for my use-case and I could generate with a high enough accuracy the related embeddings from the transcript.
- The final piece here is using davinci-text-003 which takes in the transcript and context and after a bit of prompt engineering (more like getting the LLM to give me the right thing and also say "no" when it can't), I have a working end to end search.
Its really exciting how this all comes together and the APIs allowed me (someone who has been a systems engineer for so so long) try out Machine Learning and quickly see it being beneficial and also fun.
Didn’t quite understand the last bit with what davinci-text-003 is doing! Can you ELI5 that part? Very cool otherwise, that I can ask all sorts of questions and get back contextual information from the conversation they had.
So the davinci-text-003 is basically prompt engineering the model to generate our answer. I tried being very specific and used a template which after a bit of debugging I figured out works best which was basically intuition based as you would talk to a human kind of ... https://github.com/theskcd/spchengine/blob/b2bf55268e9245f70...
- I first started with whisper to generate the transcript but eventually found a dump online where someone did the same thing :V (first step done)
- Next I tried speaker diarisation which is really easy to do in this scenario (as there are mostly 2 speakers at any time) and I could see Lex saying "welcome to my podcast" in almost all of his podcasts which made annotating the transcript easier.
- After this, its the usual create mini embeddings from a single podcast (I used OpenAI's text-embedding-ada-002) to generate the embeddings. I had to model the text in a specific way "This is a transcript between Lex Fridman and John Carmack.." (this adds the context for the conversation and helps powers query like: What did John Carmack think about {blah}) as the embeddings from the podcast with John Carmack will be closer than say some other podcast.
- Pinecone DB provided a free tier for doing cosine similarity of the generated embeddings which worked out perfectly for my use-case and I could generate with a high enough accuracy the related embeddings from the transcript.
- The final piece here is using davinci-text-003 which takes in the transcript and context and after a bit of prompt engineering (more like getting the LLM to give me the right thing and also say "no" when it can't), I have a working end to end search.
Its really exciting how this all comes together and the APIs allowed me (someone who has been a systems engineer for so so long) try out Machine Learning and quickly see it being beneficial and also fun.