Show HN: Medical LLM API on par with Google Med-PaLM 2. 92% USMLE accuracy

[mpmisko]

https://medisearch.io/api

Hello HN!

I’d like to share a medical question-answering API which has state-of-the-art performance on the USMLE self-assessment exam. You can try out the MediSearch API in 10 seconds in this Colab [https://tinyurl.com/medisearch-colab], or test the search engine live at MediSearch [https://medisearch.io]. See our API page [https://medisearch.io/api] to apply for access. Some technical highlights:

1. State-of-the-art accuracy of 92% on the USMLE self-assessment exam [https://tinyurl.com/medisearch-eval]. 2. Ranked 2nd on MedQA benchmark, closely tailing Google's Med-PaLM 2 (within ~2%). 3. The API provides article references as an output.

We’d love to hear feedback from this community! We’ve seen people use the MediSearch API to implement a chat bot or search field within their app.

Please note that MediSearch does not have a 100% accuracy. Therefore, a human professional should always make a final decision on the information suggested by MediSearch.

You can use the MediSearch API similarly to the OpenAI API [https://pypi.org/project/medisearch-client/]. If you’d like access to the MediSearch API, please email michal@medisearch.io or visit our API page [https://medisearch.io/api]. Please specify your use case in the application.

[gardenfelder]

I asked it a simple question: "what is known about PSC?" and it returned two citations.

Interesting.

Right now, nothing is known about the business model behind the product. What are the plans for it?

[mpmisko → gardenfelder]

Thanks for giving it a go! We will charge for the API and we may have a pro subscription with some nice features :).

[mackopes]

Having an API with medical information is amazing!

Just out of curiosity, is a React client or even a javascript library on your roadmap?

[mpmisko → mackopes]

Hey, a React client is definitely on our roadmap! The demo (https://tinyurl.com/medisearch-docs) in our docs is written in JS, so it should be useful out of the gate! A proper client is coming in the next few weeks.

[zopper]

This looks really cool. Does your API also support follow up questions and a longer conversations?

[mpmisko → zopper]

Glad that you like it! We do support follow-up questions. Implementing a chat-style interaction should be very straightforward :) (see our Python client for examples https://pypi.org/project/medisearch-client/).

[hzoli42]

Wow, really cool to see citation support! What are the different sources that are cited?

[mpmisko → hzoli42]

We primarily cite medical scientific papers and health guidelines. We occasionally cite health blogs (such as healthline.com), but this is mostly for simple health questions (e.g., "tell me some nutrition tips").

[brandonb]

Awesome! Can you talk a bit about how you trained or fine-tuned MediSearch's LLM?

[eoravkin → brandonb]

Thanks! Glad that you like it:)

We haven't trained or fine-tuned on neither of USMLE or MedQA. We use several LLMs in our system, and honestly I don't think there is any single thing that I can point to which is different to how other people do it. I guess it's a combination of many small improvements.

Sorry for the vague answer. We are thinking about publishing a report at some point.

[eoravkin]

We'd be interested to hear examples or ideas of how this could be useful for you!

[dredev]

This is amazing! It almost never hallucinates. How did you guys accomplish that?

[eoravkin → dredev]

Thanks a lot! We put a lot of energy into solving hallucination. The tldr is that we have a eval set to test hallucination and tweak stuff to optimize for performance on this eval set.