> On the other hand, llms.txt information will often be used on demand when a user explicitly requesting information about a topic
I don't fully understand the reasoning for this over standard robots.txt.
It seems this is looking to be a sitemap from llms, but that's not what these types of docs are for. It's not the docs responsibility to describe content if I remember correctly.
Infact it would need to be a dynamic doc and couldn't be guaranteed while also allowing bots on robots thus making the LLM doc moot?
I don't fully understand the reasoning for this over standard robots.txt.
It seems this is looking to be a sitemap from llms, but that's not what these types of docs are for. It's not the docs responsibility to describe content if I remember correctly.
Infact it would need to be a dynamic doc and couldn't be guaranteed while also allowing bots on robots thus making the LLM doc moot?