Background
I encountered this problem because I had debugged the log before and created a topic and consumer groups. Later, I deleted the topic and consumer groups because they were no longer needed. However, the deletion command did not report an error, but when I checked the topic, it was still there. This made me depressed. What’s going on?
My usage environment is as follows：
| Hostname | IP | Port |
|---|---|---|
| kafka01.koevn.com | 10.100.10.1 | 9092 |
| kafka02.koevn.com | 10.100.10.2 | 9092 |
| kafka03.koevn.com | 10.100.10.3 | 9092 |
| kafkaclient01 | 10.100.8.11 | 随机 |
View Kafka topic
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka01.koevn.com:9092,\kafka02.koevn.com:9092,kafka03.koevn.com:9092 --list \--command-config /opt/kafka/config/kraft/client.properties
---------------- output ------------------__consumer_offsetstest_koevn
--command-config: Specify the encryption certificate path to add parameters
The topic displayed at this time has a test_koevn, which is the topic to be deleted. Execute the following command to delete it
Delete the specified topic
/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka01.koevn.com:9092,\kafka02.koevn.com:9092,kafka03.koevn.com:9092 --delete --topic test_koevn \--command-config /opt/kafka/config/kraft/client.properties⚠️ 注意 Note that I specified three brokers. Since I use the three-node Kafka KRaft cluster mode with multiple replicas and partitions, if you only add one broker to delete a topic, you will only delete the topic of this node. The other node replicas will still synchronize the data. Therefore, to complete the topic deletion, you need to specify all Kafka cluster nodes to delete the topic of this node.
After executing the command, there is no error in the terminal, but when checking the topic again, it is found that test_koevn is still there, which means that the deletion failed!
And when deleting, check the log.dirs path defined in the kafka server.properties configuration file, and you will find that the topic directory shows this situation.
test_koevn-0test_koevn-0-deletetest_koevn-1test_koevn-1-deletetest_koevn-2test_koevn-2-deletetest_koevn-3test_koevn-3-deletetest_koevn-4test_koevn-4-deleteDirectories like test_koevn-0-delete will be automatically deleted after a while, but when you delete a topic, the same situation occurs.
Positioning Problem
Check the kafka logs/server.log log, there is such a message
2025-03-12 01:46:31 -0400 [ReplicaFetcherThread-0-1] WARN kafka.server.ReplicaFetcherThread - [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Received UNKNOWN_TOPIC_OR_PARTITION from the leader for partition test_koevn-2. This error may be returned transiently when the partition is being created or deleted, but it is not expected to persist.This is the warning message displayed by the replica node after deleting the topic. I checked a lot of information and many of them said that it was caused by not adding delete.topic.enable=true to the configuration and restarting. I tried it but the problem was not solved. I also checked the producer and consumer processes started in the previous test and stopped them. Finally, I checked all consumer group information to see if there were other terminals connected.
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server \kafka01.koevn.com:9092,kafka02.koevn.com:9092,kafka03.koevn.com:9092 \--all-groups --describe --command-config /opt/kafka/config/kraft/client.properties | grep test_koevn
---------------- output ------------------test_koevn test_koevn 2 0 0 0 kafkaclient01-3-af11eb2f-f44b-431c-812a-652535730358 /10.100.8.11 kafkaclient01-3test_koevn test_koevn 0 0 0 0 kafkaclient01-3-01ec3757-a139-42df-bc29-5b62aeae86d3 /10.100.8.11 kafkaclient01-3test_koevn test_koevn 1 0 0 0 kafkaclient01-2-4ff5161a-0363-4623-9fb3-8e916d8d0fa8 /10.100.8.11 kafkaclient01-2test_koevn test_koevn 3 0 0 0 kafkaclient01-2-fa058ab1-eafd-46df-8892-5d1d85fa23e4 /10.100.8.11 kafkaclient01-2There is another service that is always connected. I found the IP address and logged in to see that it is logstash.
Solving the problem
This is a section of logstash configuration. Change the topics and group_id items in the configuration file and reload the logstash configuration.
input { kafka { bootstrap_servers => "kafka01.koevn.com:9092,kafka02.koevn.com:9092,kafka03.koevn.com:9092" client_id => "kafkaclient01" topics => ["test_koevn"] group_id => "test_koevn" auto_offset_reset => "latest" partition_assignment_strategy => "org.apache.kafka.clients.consumer.RoundRobinAssignor" security_protocol => "SSL" ssl_truststore_location => "/opt/logstash/config/certs/kafka_trustchain.jks" ssl_truststore_password => "${kafka_truststore_password}" ssl_keystore_location => "/opt/logstash/config/certs/kafka_client.jks" ssl_keystore_password => "${kafka_keystore_password}" consumer_threads => 4 decorate_events => true }}View all consumer group information again
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server \kafka01.koevn.com:9092,kafka02.koevn.com:9092,kafka03.koevn.com:9092 \--all-groups --describe --command-config /opt/kafka/config/kraft/client.properties | grep test_koevn
---------------- output ------------------Consumer group 'test_koevn' has no active members.test_koevn test_koevn 1 0 0 0 - - -test_koevn test_koevn 3 0 0 0 - - -test_koevn test_koevn 0 0 0 0 - - -test_koevn test_koevn 2 0 0 0 - - -You can see that there are no clients connected. Then execute the deletion operation in step 2 above, and the topic test_koevn will be deleted normally.
Conclusion
This kind of problem occurs because we are not rigorous enough, especially in the multi-cluster node architecture. The configuration file content is modified incorrectly without paying attention. We thought we were doing test-related steps, but we changed to a non-test environment. Fortunately, the deployed logstash is in multi-node mode with good fault tolerance. Otherwise, if we consume the wrong kafka topic, the log panel will be filled with countless data!