Long story short, pytest-xdist was not a good fit with a large scale Python project which has a fairly fragile test suite so I ended up implementing a new pytest plugin, pytest-split, which makes it possible to split the full test suite into n "sub suites" based on the execution time of the tests. GitHub offers 20 concurrent jobs (free plan) for GitHub Actions based workflows. By combining these two, it's relatively easy to boost the CI of a large Python test suite.
Blog post (e.g. more about the motivation behind pytest-split): https://blog.jerrycodes.com/pytest-split-and-github-actions/
pytest-split: https://github.com/jerry-git/pytest-split
Demo project (pytest-split + GitHub Actions): https://github.com/jerry-git/pytest-split-gh-actions-demo
Opinions and thoughts are very welcome :)