kafka Offset相关命令总结

wtnyihg

一、查询topic的offset的范围

1、查询某个topic的offset的最小值

[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -2
rabin-kafka:8:0
rabin-kafka:2:0
rabin-kafka:5:0
rabin-kafka:4:0
rabin-kafka:7:0
rabin-kafka:1:0
rabin-kafka:9:0
rabin-kafka:3:0
rabin-kafka:6:0
rabin-kafka:0:0
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -2|wc -l
10
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -2 
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -2
rabin-channel:2:0
rabin-channel:1:0
rabin-channel:0:0
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -2|wc -l     
3
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -2 
　#很显然，第一列为topic名称，第二列为partition名称，第三列则是offset。为估计仔细的你应该已经发现了最小的offset为0，分区数为30个

2、查询某个topic的offset的最大值

[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -1
rabin-kafka:8:1
rabin-kafka:2:1
rabin-kafka:5:1
rabin-kafka:4:0
rabin-kafka:7:0
rabin-kafka:1:0
rabin-kafka:9:0
rabin-kafka:3:1
rabin-kafka:6:1
rabin-kafka:0:1
[root@Hexindai-C11-71 ~]# 
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -1|wc -l
10
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-kafka --time -1 
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -1
rabin-channel:2:0
rabin-channel:1:0
rabin-channel:0:0
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -1|wc -l
3
[root@Hexindai-C11-71 ~]#
[root@Hexindai-C11-71 ~]# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.20.11.73:9092 --topic rabin-channel --time -1

二、查看consumer group列表，使用 --list参数

　　查看consumser group列表有新，旧两种命令，分别查看新版（信息保存在broker中）consumer列表和老版（信息保存在zookeeper中）consumer列表，因而需要区分指定bootstrap-server和zookeeper参数：（该博客测试的kafka版本为0.10.2.1）

[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --zookeeper 172.20.11.74:2181/kafka --list
Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).

[root@Hexindai-C11-71 ~]# 
##很显然，为这里使用的是kafka 0.10.2.1版本，我们是查询不到对应的信息的！
#老版本的consumer信息都保存在zookeeper中，很显然，0.10.2.1这个版本已经将consumer信息自己保存啦。
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --zookeeper 172.20.11.74:2181/kafka --list 
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --bootstrap-server 172.20.11.74:9092 --list
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).

console-consumer-61283
console-consumer-83065
console-consumer-4362
[root@Hexindai-C11-71 ~]# 
#查看新版consumer列表，很显然，我们可以查询到对应的数据
[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh --bootstrap-server 172.20.11.74:9092 --list

三、查看指定group.id的消费者消费情况（kafka-consumer-groups.sh)

[root@Hexindai-C11-71 ~]# kafka-consumer-groups.sh  --new-consumer --bootstrap-server 172.20.11.74:9092 --group rabin-kafka --describe

Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).



Error: Consumer group \'rabin-kafka\' does not exist.


没有组，需要查看是什么原因，是需要创建？




四、查询__consumer_offsets topic所有内容


1、运行下面命令前先要在consumer.properties中设置exclude.internal.topics=false






[root@Hexindai-C11-71 ~]# cat /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties                                             
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# see kafka.consumer.ConsumerConfig for more details

# Zookeeper connection string
# comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
zookeeper.connect=127.0.0.1:2181

# timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

#consumer group id
group.id=test-consumer-group

#consumer timeout
#consumer.timeout.ms=5000

exclude.internal.topics=false
[root@Hexindai-C11-71 ~]#



[root@Hexindai-C11-71 ~]# cat /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties 



2、0.11.0.0之前版本






[root@Hexindai-C11-71 ~]#  kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 172.20.11.73:9092 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties --from-beginning
[2019-07-26 15:22:57,224] WARN The configuration \'zookeeper.connect\' was supplied but isn\'t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2019-07-26 15:22:57,224] WARN The configuration \'zookeeper.connection.timeout.ms\' was supplied but isn\'t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[test-consumer-group,__consumer_offsets,22]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,30]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,8]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,21]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,4]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,27]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,7]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,9]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,46]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,25]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,35]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,41]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,33]::[OffsetMetadata[712,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,23]::[OffsetMetadata[93,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,49]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,47]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,16]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,28]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,31]::[OffsetMetadata[1244,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,36]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,42]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,3]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,18]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,37]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,15]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,24]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,38]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,17]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,48]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,19]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,11]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,13]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,2]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,43]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,6]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,14]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,20]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,0]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,44]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,39]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,12]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,45]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,1]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,5]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,26]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,29]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,34]::[OffsetMetadata[93,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,10]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,32]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
[test-consumer-group,__consumer_offsets,40]::[OffsetMetadata[0,NO_METADATA],CommitTime 1564125765533,ExpirationTime 1564212165533]
.........



[root@Hexindai-C11-71 ~]# kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 172.20.11.73:9092 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties --from-beginning



3、0.11.0.0之后版本（含）




[root@Hexindai-C11-71 ~]# kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server 172.20.11.74:9092 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" 
--consumer.config /export/servers/kafka_2.11-0.10.2.1/config/consumer.properties --from-beginning 




 


 


 


 


 


 


 


 


 


https://www.cnblogs.com/yinzhengjie/p/10514206.html





            

            
        

	    
	    
    


 
                    
            
                

                  
 
        

	 
        

            

                                
分类：
 
                                                                
技术点：

                                            

        

        
相关文章：

        

      
            
 
                    
    
                                                    
  • 
                                 
                                2021-06-20
                            
    • 
                                                        
  • 
                                 
                                2021-09-30
                            
    • 
                                                        
  • 
                                 
                                2021-06-03
                            
    • 
                                                        
  • 
                                 
                                2022-01-23
                            
    • 
                                                        
  • 
                                 
                                2022-12-23
                            
    • 
                                                        
  • 
                                 
                                2021-04-22
                            
    • 
                                                        
  • 
                                 
                                2021-11-14
                            
    • 
                                                

               
   
         

    


  






 
          

    
猜你喜欢 

        
    
                                 
  • 
                     
                    2021-11-28
                
    • 
                                     
  • 
                     
                    2021-09-06
                
    • 
                                     
  • 
                     
                    2021-12-24
                
    • 
                                     
  • 
                     
                    2021-11-17
                
    • 
                                     
  • 
                     
                    2021-12-04
                
    • 
                                     
  • 
                     
                    2021-07-05
                
    • 
                                     
  • 
                     
                    2021-07-01
                
    • 
                            

   
 



    
相关资源 

        
   

   
    

     
相似解决方案 

         
    
                          

    

  
   

    

 
   

    
热门标签  

    

                    Java
                        Python
                        linux
                        javascript
                        Mysql
                        C#
                        Docker
                        算法
                        前端
                        SpringBoot
                        Redis
                        Vue
                        spring
                        设计模式
                        .net core
                        .net
                        kubernetes
                        c++
                        数据库
                        数据结构
                        大数据
                        js
                        机器学习
                        微服务
                        Android
                        Go
                        程序员
                        面试
                        JVM
                        ASP.net core
                        云原生
                        人工智能
                        后端
                        PHP
                        git
                        CSS
                        golang
                        k8s
                        Nginx
                        Django
                        mybatis
                        深度学习
                        多线程
                        React
                        架构
                        devops
                        爬虫
                        云计算
                        Spring Boot
                        LeetCode