They can't be completely arbitrary without user trust. My guess is security directives like invoker vs creator will need to be specified and checked appropriately whenever the assemblies are installed.
The whole point of having a sandboxed runtime like WebAssembly is that the code can be executed without user trust, as the user cannot pierce the isolation boundary or exceed the resource limits provided by the sandbox.
I don't see why UDFs and RLS would need to interact, they're orthogonal concepts that can work in tandem. The DB engine filters out the rows that fail the RLS check, then apply the UDF to the remaining rows.
You’re exactly right that these are orthogonal. The data isolation is the much harder part (the type of sandboxing provided by WASM has long been available via interpreted languages, see pl/v8 et al).
Without exception the existing APIs mentioned in this article don’t attempt to handle “data sandboxing” since it’s so application specific. So it remains the case that exposing your db to user defined functions is dangerous without a ton of work, and WASM in no way reduces the amount of work needed.
Not necessarily true. E.g. in PostgreSQL, the planner is free to evaluate predicates prior to imposing RLS constraints, so long as the functions in the predicate are marked LEAKPROOF [1]. (They aren't by default.)
This is because UDFs can write to tables, and perform I/O in other ways depending on system configuration and security policy. Just because the UDFs are in WASM instead of one of the other UDF languages doesn't eliminate the need for a security system.
(Aside -- curiously in PostgreSQL, the privilege system isn't even strong enough to prevent something as innocuous as e.g. reading a view owned by another user from dropping your own tables.)