I've worked as a developer/engineer both in private sector and in a research department at a university (but not AI/ML). There are a couple of problems with working in an academic context.
First, as you say the incentive structure is wrong: projects are led by researchers and the incentives for researchers are around publishing not around creating good software. There's virtually no credit for making software that is well-written, robust, has a great user interface etc.
Secondly, the funding structure for projects tends to be grant-based. It's extremely hard to get funding for general ongoing development. I saw lots of projects get initial funding for a couple of years, the funding stopped and then the project had no money even for the most basic support or server costs etc. Sometimes you could grant funding for new features and could funnel a bit of that into support but it did require a constant regular stream of grant applications, as often of course you don't get them.
My salary was certainly largely funded by grants - I had a permanent job and there was some allowance for gaps between funding to cover that and because of the type of work I was doing I could do some internal work too in those gaps. It's hard to manage funding that comes with short-term grants. You just can't recruit good people to short-term contracts that quickly before they are supposed to start, especially for what universities can pay (in the UK there are set pay scales, so if you don't manage anybody you basically can't be paid more than a certain amount, plus there are strict processes about recruiting, for good reasons, so it's hard to do quickly).
First, as you say the incentive structure is wrong: projects are led by researchers and the incentives for researchers are around publishing not around creating good software. There's virtually no credit for making software that is well-written, robust, has a great user interface etc.
Secondly, the funding structure for projects tends to be grant-based. It's extremely hard to get funding for general ongoing development. I saw lots of projects get initial funding for a couple of years, the funding stopped and then the project had no money even for the most basic support or server costs etc. Sometimes you could grant funding for new features and could funnel a bit of that into support but it did require a constant regular stream of grant applications, as often of course you don't get them.
My salary was certainly largely funded by grants - I had a permanent job and there was some allowance for gaps between funding to cover that and because of the type of work I was doing I could do some internal work too in those gaps. It's hard to manage funding that comes with short-term grants. You just can't recruit good people to short-term contracts that quickly before they are supposed to start, especially for what universities can pay (in the UK there are set pay scales, so if you don't manage anybody you basically can't be paid more than a certain amount, plus there are strict processes about recruiting, for good reasons, so it's hard to do quickly).