Kafka doesn’t scale well with partitions, throughput falls over quickly especially if using replication and acks.
If each user has a topic your partitions are unbounded. You have at least one partition per user.
I’d use a single notification topic, set a reasonable number of partitions on it and partition by user id.
Use the topic as a firehouse.
You can have a consumer sending push notifications.
You can have a consumer writing to a database and have the user inbox in the db with a read flag on the message. Etc. Users query the db, Kafka queues your writes. You may end up with consistency issues if using push notifications as mentioned above. If sending push notification you’d want to do cdc off your inbox table so the notification is only sent once it’s in the inbox.
You don’t necessarily need Kafka for this. It’s being used as a queue here instead of a log unless you want to keep every notification event sent to rebuild the inbox tables but then you’ll need to publish read states and start event sourcing and treating the inbox table as a materialised view.
Big rabbit hole. FIFO queue sounds simplest if you want asyn notification handling
If each user has a topic your partitions are unbounded. You have at least one partition per user.
I’d use a single notification topic, set a reasonable number of partitions on it and partition by user id.
Use the topic as a firehouse.
You can have a consumer sending push notifications.
You can have a consumer writing to a database and have the user inbox in the db with a read flag on the message. Etc. Users query the db, Kafka queues your writes. You may end up with consistency issues if using push notifications as mentioned above. If sending push notification you’d want to do cdc off your inbox table so the notification is only sent once it’s in the inbox.
You don’t necessarily need Kafka for this. It’s being used as a queue here instead of a log unless you want to keep every notification event sent to rebuild the inbox tables but then you’ll need to publish read states and start event sourcing and treating the inbox table as a materialised view.
Big rabbit hole. FIFO queue sounds simplest if you want asyn notification handling