I would like to hear more from application programmers here, about their experience in dealing with incast. For example, Facebook uses UDP to throttle messages at the application layer to combat incast: https://www.facebook.com/note.php?note_id=39391378919.
Also, a few catches with DCTCP:
1. It's not completely an end host solution and requires ECN support from switches, which should be widely available. Can someone pitch in about the availability of ECN in their networks?
2. DCTCP and plain TCP don't mix well, and hence not incrementally deployable. DCTCP and TCP+ECN also don't mix well!
I'll let others chime in, but the DCTCP paper mentions "our application reduces the amount of data each worker sends and employs jitter. Facebook, reportedly, has gone to the extent of developing their own UDP-based congestion control [29]."
I recently talked with someone on the Facebook memcached team about this problem and they mentioned that moving to UDP has been useful, but if I'm not mistaken, I believe they gave up on 100% in-order reliability in return for low-latency.
Also, a few catches with DCTCP:
1. It's not completely an end host solution and requires ECN support from switches, which should be widely available. Can someone pitch in about the availability of ECN in their networks?
2. DCTCP and plain TCP don't mix well, and hence not incrementally deployable. DCTCP and TCP+ECN also don't mix well!