I need AVIF converters to get faster before I start using it. I added it to a project recently built on an SSG and it added about 1-5s per image to build time - which adds up on 200 images.
Yes, AVIF encoding is quite slow. Are you caching/storing the images? Converting 200 images will take a few minutes, but it should be a one-time process, from which other people will benefit.
Have you tried libheif? According to https://github.com/joedrago/avif/issues/11, it's a bit faster. Probably will need CPU/GPU chips that can accelerate the codecs more efficiently before the encode times really start coming down.
libavif (and the libraries that back it) have a speed parameter to control the CPU/quality tradeoff. Have you tried tweaking that and seeing if you get acceptable speed?