The modern way to talk about this stuff is via categorification;  is a good high-level introduction.
Further, the "modern" way to discuss this is not categorification, despite what Baez says; people who work and publish on the foundations of mathematics almost universally do so in the language of set theory.
Formally, let N be the natural numbers object (in some topos), let z : 1 -> N be the zero arrow, and s : N -> N be the successor arrow. Then z;s;s : 1 -> N is the arrow which chooses 2 as an element of the NNO. Since z and s are unique up to unique isomorphism, so is 2. Moreover, since geometric morphisms between topoi preserve finite limits, the NNO should also be preserved, and that includes 2.
When the topos we choose/define is (equivalent to) Set, then we get the standard ordinal-number definition of 2.
To use a pun, this lets us upgrade from Dedekind-categoricity to a more modern and natural sort of categorical categoricity.
As I said previously, it is true that set theory provides a way to encode the natural numbers as sets (or features of a topos, etc.), so that questions about natural numbers can be stated as questions about sets. It is further true that this endeavor can be incredibly fruitful, for instance for studying the foundations of mathematics. But it does not mean that natural numbers "are" sets (or objects in a topos), any more than Quicksort "is" a piece of C++ code.
Because we limit ourselves to First Order logic, which can quantify over only individuals.
In order to quantify over multitudes (sets, groups, whatever) you either need Second Order logic or else a First Order axiomatization of how these multitudes behave.
We chose the latter rather than the former because Second Order logic has no Sound, Complete, and Decidable proof theory. First Order logic does.
That is why we bother with set theory.
For instance, that there is a special, transcendent meaning to "oneness" or "twoness" — or more generally, that there is a basic harmony within mathematics that manifests in the harmonies of the cosmos.
Here is a nice article on the sources of his mathematical contributions. https://www.sciencedirect.com/science/article/pii/0315086089...