Alternatively, if DNS is not applicable, users could use filters and autoresponders to send back an automatic response with some code.
In theory that would basically mean "Portier sends Login Mail" -> "Login Mail gets bounced via filter with additional code" -> "Portier constructs OAuth Link" -> "Portier sends OAuth back"
That way, the autoresponder doesn't need to know anything about the process nor does it authenticate the user.
It merely sends back a mail that portier can interprete as "Ok, I send an auth email there and this response means they want me to construct an oauth link to this domain"
it's stateless and only requires a rather simple autoresponder that can include email bodies.
Furthermore, portier can also verify the login link, so that replay attacks aren't feasible and the sender and link-owner must have matching emails.
In this case, the worst-case is an attacker injecting a TXT record to force OAuth logins via that domain... which is a bit pointless as far as I can see, it either won't work or use the victim's own oauth service, for which they are responsible themselves IMO.