I assume that you could have asked how this was relevant to your position/future work, what they were evaluating with it, and if they could do something different that might not be so closely linked with the product.
However that said i have done work recently (10 hours) that was a task set out to solve an issue on the product - something that could 100% go into production but actually is very relevant to the position and skills i need to demonstrate. They are also of a size & expertise that makes me worry less about them just making me do it for un-paid work.
Indeed. I politely raised an objection prior to the code test that I found work like this irrelevant. The company nevertheless told me it wants to push ahead with the homework.
Can I check: how irrelevant was the work to the applied for position?
But to answer your question: cancelling the recruitment process is totally fine. It's much better that you politely tell them now than go through the process and tell them later.
The company itself admitted the irrelevance when I raised an objection. The nature of the work was fundamentally different (and not even same language/technology).
However that said i have done work recently (10 hours) that was a task set out to solve an issue on the product - something that could 100% go into production but actually is very relevant to the position and skills i need to demonstrate. They are also of a size & expertise that makes me worry less about them just making me do it for un-paid work.