So text embeddings are kind of random like LLMs?

[gandan]

When I was messing around with random words in Cohere's playground for embedding, I noticed that the way words related to each other seemed to change every time I ran it. So, I tried the same thing with OpenAI's text embeddings and got different results for the same text, even though they were pretty similar. I always thought text embeddings were like a pre-calculated, static thing. Guess I was wrong?

[sk11001]

> got different results for the same text, even though they were pretty similar.

Exactly the same text or similar text? There's a big difference between the two. The same text should produce identical results every time.

[gandan → sk11001]

I actually got different embeddings from the exact same text. I should have been clearer about that.

[fithisux]

They appear random because in reality they are hash values.

Hashing is like "deterministically" randomizing.

[gandan → fithisux]

I stumbled upon a discussion in the OpenAI community about why OpenAI's embeddings sometimes yield different vectors for the same text. https://community.openai.com/t/why-openai-embedding-return-d...

Skimming through, it seems the issue might be due to random timing in the GPU, as highlighted in one of the posts. https://community.openai.com/t/why-openai-embedding-return-d...

I'm quite new to this field, but it appears that this issue is different from hashing. I'd appreciate your thoughts on this!

[fithisux → gandan]

Ah ok, you talk about something very specific. Not why embeddings for parts of a body of text appear random. But, why computing embeddings in the same way for a specific body yield different results.

The link is revealing. If f is approximately an associative function and you compute in parallel a,b,c

with the goal of computing

X = f(f(a,b) ,c) ~ f(a, f(b, c)) = Y

in some case you may end up a,b finishing first and then c of b,c finishing first and then a.

So, you end up computing X in some cases and in other Y.

The more values you compute in parallel, the move variations.

Some computations like multiplication, addition fall into this category ....