Kafka修炼日志(一):单节点使用问题一二

原创
2017/03/01 12:03
阅读数 1.7K

       根据官方文档修炼Kafka神器,详细内容请戳这里。本文简要介绍Kafka特性和优势,以及在实战当中遇到的两个问题及解决方式。本文为Kafka系列文章,请关注后续更新。

       Kafka版本:2.12-0.10.2.0

       操作系统:CentOS 6.5

Kafka的三个特性:

    (1)可以发布和订阅Records流,相当于一个消息队列或者消息系统;

    (2)可以存储Records流,并且具有容错性;

    (3)可以对Records流进行处理。

Kafka的两个优势:

    (1)通过建立实时的流数据管道,可靠的在系统间或应用间获取数据;

    (2)通过建立实时的流应用,转换成或者反应成数据流。

问题1:Kafka的Source和Binary:

       在使用官方教程学习当中,进行到Step 2时,运行如下命令,出现找不到main class的错误。

[root@localhost kafka-0.10.2.0-src]# ./bin/zookeeper-server-start.sh config/zookeeper.properties
Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

       百思不得其解,考虑是否因为Linux自带JDK,于是卸载JDK,安装JDK1.8,问题依旧,开始Google求助,在StackOverflow官网找到解决方案,原因是下错了包。使用的是未经过编译的Source,下载Binary后一切恢复正常。

问题2:按照官方教程找不到server.properties进程:

       问题1解决后,一切操作正常,直到运行到如下命令,没有显示相应的进程信息。

[root@localhost kafka_2.12-0.10.2.0]# ps aux | grep server1.properties
root     28779  0.0  0.0 103252   876 pts/2    S+   11:14   0:00 grep server1.properties

       开始抓狂,怀疑是命令问题,于是开始研究ps命令,试了ps -ef依旧无果,使用jps命令发现三个Kafka进程运行正常,但无法和server1.properties对应起来。

[root@localhost kafka_2.12-0.10.2.0]# jps
22596 Kafka
23221 Kafka
25689 Jps
22909 Kafka
22318 QuorumPeerMain

       想到可以利用PID查找对应的进程名,结果如下,三个Kafka进程之间无任何区别,依旧无法将PID和server1.properties对应。

[root@localhost kafka_2.12-0.10.2.0]# ps aux | grep 22596
root     22596  1.4  4.5 7892660 679216 pts/3  Sl   15:12   2:08 /home/jdk1.8.0_121/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/kafka_2.12-0.10.2.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/kafka_2.12-0.10.2.0/bin/../logs -Dlog4j.configuration=file:./bin/../config/log4j.properties -cp .:/home/jdk1.8.0_121/lib/dt.jar:/home/jdk1.8.0_121/lib/tools.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/argparse4j-0.7.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-api-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-file-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-json-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-runtime-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-transforms-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/guava-18.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-api-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-locator-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-utils-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-core-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-databind-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javassist-3.20.0-GA.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-client-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-common-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-core-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-guava-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-media-jaxb-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-server-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jopt-simple-5.0.3.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-test-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-clients-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-log4j-appender-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-examples-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-tools-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/log4j-1.2.17.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/lz4-1.3.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/metrics-core-2.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka_2.12-0.10.2.0/b
root     25632  0.0  0.0 103252   832 pts/2    R+   17:42   0:00 grep 22596
[root@localhost kafka_2.12-0.10.2.0]# ps aux | grep 22909
root     22909  0.9  5.5 6837340 821064 pts/3  Sl   Feb28  11:48 /home/jdk1.8.0_121/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/kafka_2.12-0.10.2.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/kafka_2.12-0.10.2.0/bin/../logs -Dlog4j.configuration=file:./bin/../config/log4j.properties -cp .:/home/jdk1.8.0_121/lib/dt.jar:/home/jdk1.8.0_121/lib/tools.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/argparse4j-0.7.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-api-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-file-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-json-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-runtime-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-transforms-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/guava-18.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-api-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-locator-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-utils-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-core-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-databind-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javassist-3.20.0-GA.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-client-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-common-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-core-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-guava-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-media-jaxb-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-server-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jopt-simple-5.0.3.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-test-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-clients-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-log4j-appender-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-examples-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-tools-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/log4j-1.2.17.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/lz4-1.3.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/metrics-core-2.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka_2.12-0.10.2.0/b
root     28818  0.0  0.0 103252   840 pts/2    R+   11:22   0:00 grep 22909

      思考之后,决定使用lsof命令,查询端口号和PID之间的对应关系,因为按照教程部署的Kafka三个节点都在本机上,以端口号区分,server1.properties的端口号设置的是9093。

[root@localhost kafka_2.12-0.10.2.0]# lsof -i:9093
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    22909 root  102u  IPv6 278744      0t0  TCP *:9093 (LISTEN)
java    22909 root  103u  IPv6 292045      0t0  TCP localhost:9093->localhost:49648 (ESTABLISHED)
java    25991 root   84u  IPv6 292044      0t0  TCP localhost:49648->localhost:9093 (ESTABLISHED)

      问题解决,可以继续教程进行fault-tolerance测试了。

      本文属作者原创,转贴请声明!

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部