the "missing matter" is not just at the galactic core but assumed distributed more thorough the galaxy. if it just were at the center, star rotational speed would follow keplerian predictions, instead they observed the same rotational speed across whole galaxy disks.
Baryonic matter accelerating into black holes also emits electromagnetic radiation vs. dark matter which does not interact electromagnetically by definition (and observation).
https://en.wikipedia.org/wiki/Galaxy_rotation_curve