On: ScrapeGraphAI: Web scraping using LLM and direct graph logic

[spaniard89277]

How do you handle the context window limit? If you push the entire Dom to the LLM it will exceed the context window by far in most cases, isn't it?

[aleksiy123 → spaniard89277]

My guess is you do some preprocessing on the DOM to get it down to text but still retains some structure.

Something like https://github.com/Alir3z4/html2text.

I'm sure there are other (better?) options as well.

[tarasglek → aleksiy123]

I wrote https://markdown.download as a general helper for this

[msp26 → spaniard89277]

Trim unwanted html elements + convert to markdown. Significantly reduces token counts while retaining structure.