I never saw the second letter

zefparis · 2026-08-14T19:00:32.000000Z

Appreciate the report — I've reviewed the logic and it's working as intended on my end (tested across multiple devices). If you run into it again, let me know your browser/device and I'll dig in further.

gus_massa · 2026-08-14T21:53:19.000000Z

After looking at it again...

I expect:

  Letter 1
  Letter 2
  Is letter 1 equal to letter 2?
  Letter 3
  Letter 4
  Is letter 3 equal to letter 4?
  Letter 5
  Letter 6
  Is letter 5 equal to letter 6?
  Letter 7
  Letter 8
  Is letter 7 equal to letter 8?

But the test is actually something like

  Letter 1
  Is letter 1 equal to [I'm very confused]?
  Letter 2
  Is letter 1 equal to letter 2?
  Letter 3
  Is letter 2 equal to letter 3?
  Letter 4
  Is letter 3 equal to letter 4?

Is this correct? What should I compare in the first question?

zefparis · 2026-08-14T23:19:03.000000Z

You called it exactly right — it's a true 1-back sliding window, not paired comparison. And digging into it after your question, you also uncovered a real UI bug: the first trial showed a dash and asked the comparison question even though there was no previous letter yet. Scoring was already handling it correctly (first trial never counted as a target), but the UI didn't make that clear. Just shipped a fix — first trial now shows a 'this is the first letter, just remember it' hint with a Continue button instead of the Yes/No question. Thanks for pushing on this, made the product better.

gus_massa · 2026-08-15T02:37:54.000000Z

The dash is very confusing. Try a version that shows nothing instead of the dash (it's useful to drag few friends to test small variations).

zefparis · 2026-08-15T06:22:49.000000Z

Confirmed, much cleaner now — thanks again.

Quick context since you clearly care about getting this right: this test is part of a bigger project, training a bot-detection model that currently has 22k+ verified bot sessions but only a handful of real human ones. Classic data-starvation problem: the model knows what a bot looks like, barely knows what a human looks like. Every real person who plays through honestly (like you did, bugs and all) helps balance that out.

Appreciate you taking it seriously enough to actually find the rough edges.

gus_massa · 2026-08-17T14:13:43.000000Z

A few more comments:

In the last test, 1->2->3->4->5 the lines are not aligned with the dots.

Are you tracking the mouse? In the yes/no test I move the mouse to the correct position before the buttons appear.

zefparis · 2026-08-17T20:19:54.000000Z

Good catch again, found it and fixed: the node dots were positioned by their top-left corner instead of centered, so the connecting lines were terminating ~24px off from the visual center of each dot. Just shipped a one-line CSS fix (missing transform: translate(-50%, -50%), dots and lines should now line up correctly.

On the mouse tracking: no, free cursor movement isn't tracked at all, the only pointer data captured is during active drag (touch/click-held), not hover position. So your pre-positioning isn't logged as a signal one way or the other. Fair UX point though: the buttons always render in the same fixed left/right position, so a user who's learned the layout can anticipate it. For this specific test the display window is long enough (2s) that it shouldn't meaningfully skew results, but it's a reasonable thing to randomize in a future pass.

Really appreciate the depth here,three rounds of testing and you've caught two real bugs and one solid UX observation.