Description
Cadence 1.3.6 upgraded sarama to 1.46.0 which has introduced Kafka API version negotiation. When the Kafka version is not specified, it defaults to protocol version 0.10.2.0 which seems too old.
Steps to Reproduce / How to Trigger
Run Cadence 1.3.6 with advanced visibility using Kafka 3.9.1, without setting kafka.version.
Expected Behavior
Cadence 1.3.6 should startup successfully just like version 1.3.5.
Actual Behavior
Cadence crashes due to kafka: client has run out of available brokers to talk to.
Cadence was able to startup after I applied the Cadence base.yaml setting:
I suspect a version as old as "2.0.0" may also work, but haven't tested it.
Logs / Screenshots
{"level":"fatal","ts":"2026-02-25T05:24:17.766Z","msg":"Index consumer state changed","service":"cadence-worker","component":"indexer","lifecycle":"StartFailed","error":"kafka: client has run out of available brokers to talk to","logging-call-at":"indexer.go:119","stacktrace":"github.com/uber/cadence/service/worker/indexer.NewIndexer\n\t/cadence/service/worker/indexer/indexer.go:119\ngithub.com/uber/cadence/service/worker.(*Service).startIndexer\n\t/cadence/service/worker/service.go:400\ngithub.com/uber/cadence/service/worker.(*Service).Start\n\t/cadence/service/worker/service.go:231\ngithub.com/uber/cadence/cmd/server/cadence.execute\n\t/cadence/cmd/server/cadence/server.go:303"}
Environment
- Cadence server version: 1.3.6
- DB & version: Kafka 3.9.1
Description
Cadence 1.3.6 upgraded sarama to 1.46.0 which has introduced Kafka API version negotiation. When the Kafka version is not specified, it defaults to protocol version
0.10.2.0which seems too old.Steps to Reproduce / How to Trigger
Run Cadence 1.3.6 with advanced visibility using Kafka 3.9.1, without setting
kafka.version.Expected Behavior
Cadence 1.3.6 should startup successfully just like version 1.3.5.
Actual Behavior
Cadence crashes due to
kafka: client has run out of available brokers to talk to.Cadence was able to startup after I applied the Cadence base.yaml setting:
I suspect a version as old as "2.0.0" may also work, but haven't tested it.
Logs / Screenshots
Environment