The dinner was organized by John Brockman, literary agent for many of the figures who bragged about his ties to Epstein to his clients (e.g. https://newrepublic.com/article/154826/jeffrey-epsteins-inte...).
Brockman organized the dinners as part of his running the Edge Foundation, a foundation funded substantially (reportedly some years entirely) by Epstein.
> As Brockman himself put it after one such dinner in 2004, “last year we tried ‘The Science Dinner’. Everyone yawned. So this year, it’s back to the money-sex-power thing with ‘The Billionaires’ Dinner’.”
"How could they know" he was there? Because this was _Epstein's dinner_.