> and was taken at least somewhat seriously by people there, to the point that Yudkowsky censored it.
Roko isn't taken seriously. What was taken seriously is ~ "if you've had an idea that you yourself think will harm people to even know about it, don't share it".
Not necessarily, it also comes from taking your users seriously before you even know what they were saying.
Consider: If you trust the people on your forum to be sensible, and one day one of your users writes:
"I have realised something which is dangerous to know. Here it is: …"
then the sensible choice is to (1) stop reading immediately, don't even find out what's after the ellipsis, (2) delete/hide/otherwise remove their message, and (3) put in a general statement of "don't write things that you think are dangerous for people to know".
Roko isn't taken seriously. What was taken seriously is ~ "if you've had an idea that you yourself think will harm people to even know about it, don't share it".