
Event processing for an ad network - xstartup
I am not creating a new ad network - it&#x27;s my friend&#x27;s ad network with 60B monthly clicks. We are looking to make it better, piece at a time but still need a big picture. Task mostly involves interfacing with a DSP. Logging impressions, generating reports for ad buyers&#x2F;sellers, ultra-fast redirects. Doing this very cheap, reliable. I am thinking about using Kafa + Go binary (for serving impressions, Kafka producer&#x2F;consumer). But generating 3 dimensions (group by X, group by Y, group by Z, finally filter) reports in real-time for ad buyers is a bit difficult. Is there any reference architecture for ad networks?  Can anyone share their experience with something similar?
======
robin_philip
We had a similar problem building an ad network. We ended up logging events
into a HDFS using Kafka. Backend for serving was built using Java (Go Seems
fine too). As for serving reports in real time, I don’t think that would be
doable. We need up refreshing the reports every 30 minutes. Building these
analytics views can be accomplished using Cassandra, Hive or ElephantDB or any
other. People have used apache storm also. You can look up the book Big Data
principles by Nathan Marz for discussions on scalable architecture.

Disclosure: I did not code this. I only participated in architecture reviews.

