On: Twitter open-sources a high-performance replicated...

[mgodave]

A DL stream is a single partition if you are thinking in terms of kafka. We stitch them together as a partitioned stream a la Kafka using a different system. So, to answer your question, no, there is no limit to the number of DL streams you can put together into a bigger logical stream, we have some very large ones.

[alexdean → mgodave]

Ah! That actually explains the strange syntax I was seeing in the tutorials:

  // Create Stream `basic-stream-{3-7}`
  // dlog tool create -u ${distributedlog-uri} -r ${stream-prefix} -e ${stream-regex}
  ./distributedlog-core/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/distributedlog -r basic-stream- -e 3-7

You use a regex to progammatically create N streams, which would be N shards in Kinesis or N partitions in Kafka.