Most moral and legal systems hold genocide in a special place, and this is natural, because systematically killing all members of a religious or ethnic group is more damaging than killing some members.
Eliminating a disease like smallpox is a much more significant achievement than simply mitigating it or treating it. When we really eliminate a disease it may never come back!
This list of experts is worried about us building something that will do to us what we did to smallpox. For the same reasons as above, that is more worrying than extreme poverty and the comparison you are making is a false equivalence.
Another way to look at it is, you can't fight to end poverty when you no longer exist.
We can argue about whether the risk is real, but if this set of experts thinks it is, and you disagree for some reason, I would spend some time thinking deeply about whether that reason is simply based in a failure of imagination on your part, and whether you are sure enough to bet your life and everyone else's on that reason. Everyone can think of a security system strong enough that they themselves can't imagine a way to break it. Similarly, anyone can think of a reason why superhuman AGI is impossible or why it can't really hurt us.
Is AI going to replace the lowest paid jobs though? I imagine, it rather has potential to move the white collar workers down the social ladder, which is unfortunate, but wouldn't cause extreme poverty.
Are the 4 million truck/taxi drivers in the US white collar? Janitors? Fast food workers? Automation is relentless and not everyone can be a plumber.
Zoom out. It's a big problem that most people derive their social power from labor while the demand for labor is clearly on a long term downward trend. Even if progress slows way down, even if the next wave of progress only dispossesses people who you hate and feel comfortable farming for schadenfreude, we will have to deal with this eventually. Defaulting means our society will look like (insert cyberpunk nightmare world here).
I am not hating anyone, being a white collar worker myself. My point is that a whole lot of people already live like that, without having much power from their labour, and the sky is not falling. More people might be joining them, and the illusion of meritocracy might be harder to maintain in the future, but extreme poverty, hunger etc. is something we will likely be able to avoid
The risk of individuals suddenly falling into extreme poverty is a very real one.
But none wants to talk about how to mitigate that problem.