It seems like this is backwards: you have an address and username, they have an address and username. They see the source address and destination username.
So suppose an attacker sees two packets:
Encrypted("Indian for dinner?"): Their username, your address.
Encrypted("Sure, sounds good."): Your username, their address.
From this they could be reasonably sure you two are talking but it's less data than they had before and now that attacker needs to either know you're using the source address or see both messages to get the full picture.
What do you mean here by "address" ? And indeed "username" ?
Suppose Alice is asking about Indian and Bob replies that it sounds good.
A passive attacker sees TCP/IP packets between Alice's IP address and Signal's and between Bob's IP address and Signal's. They get no other information from this beyond that Alice and Bob use Signal (and perhaps not even that if Signal shares IP addresses with other services)
The Sealed Sender feature makes no difference in that layer
If an attacker has control of Signal's servers (or perhaps Signal's server admins are secretly bad guys) ordinarily they would be able to see the sender and recipient of every message.
With Sealed Sender, the servers don't know the Sender any more, only that the Sender seems to be someone permitted to send messages to this recipient.
You could try to correlate IP addresses to Signal users, but you have absolutely no guarantee that they're correlated, much less that there's a nice 1:1 correspondence.
In a mundane example, Alice proposes Indian food then leaves her office, her phone disconnects from WiFi and goes to a 3G network. The reply from Bob is picked up by Alice using a completely different IP address, because she isn't in the office any more.
So suppose an attacker sees two packets:
Encrypted("Indian for dinner?"): Their username, your address.
Encrypted("Sure, sounds good."): Your username, their address.
From this they could be reasonably sure you two are talking but it's less data than they had before and now that attacker needs to either know you're using the source address or see both messages to get the full picture.