On: Jev Does Not Play Dice: 83% probability, 19% accuracy on a hidden fair die roll

[kantahayashi]

Author here. I tested Jev with a fair die 400 times without telling it the die result. The true probability of face 1 is 1/6, but Jev always chose face 1 and the probability it returned was about 83%.

Code and data are here: https://github.com/KantaHayashiAI/jev-does-not-play-dice

Happy to answer questions!

[magicalhippo → kantahayashi]

Maybe it watched Rosencrantz & Guildenstern Are Dead[1] once too many.

[1]: https://www.youtube.com/watch?v=C_TfdNAXOwE

[kantahayashi → magicalhippo]

I found that in the play it's 92 heads in a row [1], and in my coin test Jev returned around 92%. Maybe Jev was actually trained on it.

[1]: https://en.wikipedia.org/wiki/Rosencrantz_and_Guildenstern_A...

[__m → kantahayashi]

why would you use ai for that though? how does that translate to other tasks?

[kantahayashi → __m]

I used a die as an example of problems whose answers can't be known at all, unlike problems with correct answers like MMLU questions. In a more practical test, a document said "30% risk" and Jev returned 5%. The point of the post is that Jev is weak at some kinds of tasks, so you should check the calibration for your own use case.