
Topicctl – an easier way to manage Kafka topics - mrbbk
https://segment.com/blog/easier-management-of-Kafka-topics-with-topicctl/
======
rbranson
Segment employee here. Our Kafka runbook looks 100X cleaner after this tool
was rolled out by Ben and folks. One of my favorite parts is how it makes
dangerous things harder :)

~~~
xref
Is there a reason you chose not to use ACLs? We did enable them, but it feels
like it is causing more headaches than any security benefit it might
provide...

~~~
pryz
I guess we just haven't got the need to use ACL yet. This may be something we
will look later on. We have been started to implement what's needed to
leverage things like SASL in our instrumentation but we are not there yet (see
[https://github.com/segmentio/kafka-go](https://github.com/segmentio/kafka-
go)).

------
tekkertje
Thanks for this! Managing Kafka and topics can be messy at times, so a really
welcome addition to apply more robust Kafka changes.

------
retzkek
> Replica placement strategy

This is huge, currently you have to manually specify placement for all
replicas when you increase the replication factor for a topic. Do this for
many topics and it quickly becomes very tedious and error-prone. Thanks for
this, I'm very excited to try it out!

------
abledon
"""Many of these were shaped by our experiences making AWS changes with
Terraform and Kubernetes changes with kubectl. We wanted an equivalent to
these for Kafka!"""

so use this topicctl for advanced topic changes.... and the more primitive
kafkactl[1] for basic stuff?

What about the 10 or so other GUI managers like this one?
[https://github.com/cloudhut/kowl](https://github.com/cloudhut/kowl) [2]

[1][https://github.com/deviceinsight/kafkactl](https://github.com/deviceinsight/kafkactl)

[2]
[https://news.ycombinator.com/item?id=24099037](https://news.ycombinator.com/item?id=24099037)

~~~
bhyolken
Author here, thanks for the comment. topicctl is really motivated by a desire
to support rigorous, git-based topic management. The read-only views (tailing,
repl, etc.) are secondary to that and are useful for our command-line-based
workflows inside Segment, but are definitely not intended to replace all of
the other good tooling out there, including the ones you've referenced.

------
infogulch
kafka-gitops [1] is another declarative topic/acl management tool that enables
a planfile/apply strategy.

[1]: [https://github.com/devshawn/kafka-
gitops](https://github.com/devshawn/kafka-gitops)

------
caiobegotti
That looks very interesting! About managing YAMLs like we'd do with kubectl as
explained in the article, how does topicctl differ both in concept and
operation from Strimzi's Kafka Operator for Kubernetes? I know, they are a
different company etc but that Kafka operator is gitops-friendly and
everything so I'm honestly trying to compare them as managing Kafka is a
REALLY HARD problem, so kudos for trying to make it easier!

~~~
pryz
At Segment we manage clusters which are both on ECS and EKS. We didn't want to
tie our Kafka management to the container orchestration.

If I'm right the Strimzi operator is also managing the brokers. Here we only
focus on the topics.

