On: Large Enough

[freediver]

That would make each API call cost at least $3 ($3 is price per million input tokens). And if you have a 10 message interaction you are looking at $30+ for the interaction. Is that what you would expect?

[coder543 → freediver]

Gemini 1.5 Pro charges $0.35/million tokens up to the first million tokens or $0.70/million tokens for prompts longer than one million tokens, and it supports a multi-million token context window.

Substantially cheaper than $3/million, but I guess Anthropic’s prices are higher.

[reitzensteinm → coder543]

You're looking at the pricing for Gemini 1.5 Flash. Pro is $3.50 for <128k tokens, else $7.

[coder543 → reitzensteinm]

Ah... oops. For some reason, that page isn't rendering properly on my browser: https://imgur.com/a/XLFBPMI

When I glanced at the pricing earlier, I didn't notice there was a dropdown at all.

[freediver → coder543]

It is also much worse.

[coder543 → freediver]

Is it, though? In my limited tests, Gemini 1.5 Pro (through the API) is very good at tasks involving long context comprehension.

Google's user-facing implementations of Gemini are pretty consistently bad when I try them out, so I understand why people might have a bad impression about the underlying Gemini models.

[rkwz → freediver]

Maybe they're summarizing/processing the documents in a specific format instead of chatting? If they needed chat, might be easier to build using RAG?

[impossiblefork → freediver]

So do it locally after predigesting the book, so that you have the entire KV-cache for it.

Then load that KV-cache and add your prompt.

[tr4656 → freediver]

This might be when it's better to not use the API and just pay for the flat-rate subscription.