That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.
To make this more clear as I have had the same experience and want more people to enjoy it: Mamba is more or less a drop-in replacement for the conda tool with the guts rewritten in C++ and as a result is much faster to resolve dependencies -- I want to say an order of magnitude faster or more. It's even noticeably faster at downloading and installing. You just
conda install -c conda-forge mamba
in your base environment and then you can start running mamba instead of conda.