One of the most non-obvious discoveries we made was that for such long documents, turning it into a Markdown (with marked headings), as opposed to plain text, made a night-and-day difference in LLM's reasoning performance. I have my guesses as to why this could be the case, but I'm curious to hear your hypothesis and whether you've seen similar effects in the wild?
Today, we launch Hotseat AI: an AI-powered Q&A service for the 226-pages-long EU AI Act[0][1]. We launch Hotseat AI as a collaborative FAQ where anyone[2] can ask a question, and the bot will answer. The questions and answers will be public to build a high-quality community reference on AI regulation.
Hotseat is not your typical "chat-with-document". It started as such, and the earlier iteration of this project relied on embedding-based retrieval. We quickly found embeddings fall short of connecting a user question to relevant chunks of the regulation. Today's version doesn't use embeddings at all and is built on a bespoke pipeline of models. GPT4 is at the heart of Hotseat, and we heavily rely on function calling. We also use chain-of-thought and step-by-step reasoning to increase the working memory of LLM. We're performing whole-document reasoning first to make a plan for answering the question and proceed with the execution of that multi-step plan. All combined, Hotseat gets nuanced questions right.
My overarching lesson from this project is that to squeeze the most out of current LLMs, you need to focus on the retrieval and build upon that.
Our answers include a "legal trace": a series of AI Act quotes and explanatory comments. We're "pinning down" an LLM to reduce hallucinations by forcing direct quotes. This response format also reduced the chance of LLM taking a wrong turn when reasoning.
AI regulation is a hotly debated topic, and Hotseat can help folks poke at it with questions without plunging into legalese - plain language works great!
To wrap up, I'm wondering if this is a seed of a viable business. Would you find ‘directly ask the regulation’ useful, especially as a non-lawyer, like a startup founder or engineer. We had to cut a few corners to get Hotseat AI out, but it's unclear how much these matter in practice. Let me know if you find Hotseat useful to you or try to poke holes in it.
[0]: we're on the far end of "focus on one thing"
[1]: the latest AI Act version
[2]: I'll be doing light moderation to prevent spam and keep the quality high
Man I the part about having to get a license to release a foundational model really friggin sucks and I hope it doesn't make it to the final version of the text
Is the licensing requirement actually in the bill? I've seen a confusion around the distinction between foundational and high-risk models - they're not the same.
(on a larger point of the AI Act leaving much to be desired, I agree)
It takes 90-120s to compute the answer. I just checked and the bot died mid-way computing answers earlier in the day and picked up from later point in the queue. I fixed it and you should get an answer soon.
Re email: the submission form has a second step where you can opt-in to leave your email address to get notified. Did it not show for you?
One of the most non-obvious discoveries we made was that for such long documents, turning it into a Markdown (with marked headings), as opposed to plain text, made a night-and-day difference in LLM's reasoning performance. I have my guesses as to why this could be the case, but I'm curious to hear your hypothesis and whether you've seen similar effects in the wild?