文档章节

Kafka修炼日志(一):单节点使用问题一二

鬼谷半桃
 鬼谷半桃
发布于 2017/03/01 12:03
字数 1886
阅读 917
收藏 0

       根据官方文档修炼Kafka神器,详细内容请戳这里。本文简要介绍Kafka特性和优势,以及在实战当中遇到的两个问题及解决方式。本文为Kafka系列文章,请关注后续更新。

       Kafka版本:2.12-0.10.2.0

       操作系统:CentOS 6.5

Kafka的三个特性:

    (1)可以发布和订阅Records流,相当于一个消息队列或者消息系统;

    (2)可以存储Records流,并且具有容错性;

    (3)可以对Records流进行处理。

Kafka的两个优势:

    (1)通过建立实时的流数据管道,可靠的在系统间或应用间获取数据;

    (2)通过建立实时的流应用,转换成或者反应成数据流。

问题1:Kafka的Source和Binary:

       在使用官方教程学习当中,进行到Step 2时,运行如下命令,出现找不到main class的错误。

[root@localhost kafka-0.10.2.0-src]# ./bin/zookeeper-server-start.sh config/zookeeper.properties
Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

       百思不得其解,考虑是否因为Linux自带JDK,于是卸载JDK,安装JDK1.8,问题依旧,开始Google求助,在StackOverflow官网找到解决方案,原因是下错了包。使用的是未经过编译的Source,下载Binary后一切恢复正常。

问题2:按照官方教程找不到server.properties进程:

       问题1解决后,一切操作正常,直到运行到如下命令,没有显示相应的进程信息。

[root@localhost kafka_2.12-0.10.2.0]# ps aux | grep server1.properties
root     28779  0.0  0.0 103252   876 pts/2    S+   11:14   0:00 grep server1.properties

       开始抓狂,怀疑是命令问题,于是开始研究ps命令,试了ps -ef依旧无果,使用jps命令发现三个Kafka进程运行正常,但无法和server1.properties对应起来。

[root@localhost kafka_2.12-0.10.2.0]# jps
22596 Kafka
23221 Kafka
25689 Jps
22909 Kafka
22318 QuorumPeerMain

       想到可以利用PID查找对应的进程名,结果如下,三个Kafka进程之间无任何区别,依旧无法将PID和server1.properties对应。

[root@localhost kafka_2.12-0.10.2.0]# ps aux | grep 22596
root     22596  1.4  4.5 7892660 679216 pts/3  Sl   15:12   2:08 /home/jdk1.8.0_121/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/kafka_2.12-0.10.2.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/kafka_2.12-0.10.2.0/bin/../logs -Dlog4j.configuration=file:./bin/../config/log4j.properties -cp .:/home/jdk1.8.0_121/lib/dt.jar:/home/jdk1.8.0_121/lib/tools.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/argparse4j-0.7.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-api-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-file-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-json-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-runtime-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-transforms-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/guava-18.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-api-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-locator-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-utils-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-core-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-databind-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javassist-3.20.0-GA.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-client-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-common-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-core-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-guava-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-media-jaxb-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-server-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jopt-simple-5.0.3.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-test-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-clients-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-log4j-appender-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-examples-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-tools-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/log4j-1.2.17.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/lz4-1.3.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/metrics-core-2.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka_2.12-0.10.2.0/b
root     25632  0.0  0.0 103252   832 pts/2    R+   17:42   0:00 grep 22596
[root@localhost kafka_2.12-0.10.2.0]# ps aux | grep 22909
root     22909  0.9  5.5 6837340 821064 pts/3  Sl   Feb28  11:48 /home/jdk1.8.0_121/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/home/kafka_2.12-0.10.2.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/kafka_2.12-0.10.2.0/bin/../logs -Dlog4j.configuration=file:./bin/../config/log4j.properties -cp .:/home/jdk1.8.0_121/lib/dt.jar:/home/jdk1.8.0_121/lib/tools.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/argparse4j-0.7.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-api-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-file-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-json-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-runtime-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/connect-transforms-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/guava-18.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-api-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-locator-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/hk2-utils-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-core-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-databind-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javassist-3.20.0-GA.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.inject-2.5.0-b05.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-client-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-common-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-container-servlet-core-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-guava-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-media-jaxb-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jersey-server-2.24.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/jopt-simple-5.0.3.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka_2.12-0.10.2.0-test-sources.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-clients-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-log4j-appender-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-streams-examples-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/kafka-tools-0.10.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/log4j-1.2.17.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/lz4-1.3.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/metrics-core-2.2.0.jar:/home/kafka_2.12-0.10.2.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka_2.12-0.10.2.0/b
root     28818  0.0  0.0 103252   840 pts/2    R+   11:22   0:00 grep 22909

      思考之后,决定使用lsof命令,查询端口号和PID之间的对应关系,因为按照教程部署的Kafka三个节点都在本机上,以端口号区分,server1.properties的端口号设置的是9093。

[root@localhost kafka_2.12-0.10.2.0]# lsof -i:9093
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    22909 root  102u  IPv6 278744      0t0  TCP *:9093 (LISTEN)
java    22909 root  103u  IPv6 292045      0t0  TCP localhost:9093->localhost:49648 (ESTABLISHED)
java    25991 root   84u  IPv6 292044      0t0  TCP localhost:49648->localhost:9093 (ESTABLISHED)

      问题解决,可以继续教程进行fault-tolerance测试了。

      本文属作者原创,转贴请声明!

© 著作权归作者所有

鬼谷半桃
粉丝 0
博文 3
码字总数 2959
作品 0
嘉兴
程序员
私信 提问
分布式架构开发套件 - jeesuite-libs

简介 jeesuite-libs分布式架构开发套件。包括缓存(一二级缓存、自动缓存管理)、队列、分布式定时任务、文件服务(七牛、阿里云OSS、fastDFS)、日志、搜索、代码生成、API网关、配置中心、统一...

vakinge
2018/11/27
1K
0
Kafka安装部署(Linux环境)

环境准备 JDK:建议版本在 1.7 及以上,否则可能会报如下错误:java.lang.UnsupportedClassVersionError (安装步骤略) ZooKeeper:Kafka 的安装包中自带 zookeeper,但并不建议使用之,最好...

qq_29116427
2018/04/15
0
0
mongodb启动很慢:[initandlisten] found 1 index(es) that

故障现象 mongodb启动后,等了几十分钟还一直没启动完成,单节点副本集,状态一直处于startup 原因分析:查看mongod的错误日志,发现:[initandlisten] found 1 index(es) that wasn't finis...

qianghong000
2017/12/13
0
0
Kafka+Log4j实现日志集中管理

引言 前段时间写的《Spring+Log4j+ActiveMQ实现远程记录日志——实战+分析》得到了许多同学的认可,在认可的同时,也有同学提出可以使用Kafka来集中管理日志,于是今天就来学习一下。 特别说...

摆渡者
2015/12/08
16.3K
25
kafka节点假死

一、前言 早上醒来打开微信,同事反馈kafka集群从昨天凌晨开始写入频繁失败,赶紧打开电脑查看了kafka集群的机器监控,日志信息,发现其中一个节点的集群负载从昨天凌晨突然掉下来了,和同事...

navyaijm2012
2017/06/03
0
0

没有更多内容

加载失败,请刷新页面

加载更多

分布式协调服务zookeeper

ps.本文为《从Paxos到Zookeeper 分布式一致性原理与实践》笔记之一 ZooKeeper ZooKeeper曾是Apache Hadoop的一个子项目,是一个典型的分布式数据一致性的解决方案,分布式应用程序可以基于它...

ls_cherish
今天
4
0
redis 学习2

网站 启动 服务端 启动redis 服务端 在redis 安装目录下 src 里面 ./redis-server & 可以指定 配置文件或者端口 客户端 在 redis 的安装目录里面的 src 里面 ./redis-cli 可以指定 指定 连接...

之渊
昨天
2
0
Spring boot 静态资源访问

0. 两个配置 spring.mvc.static-path-patternspring.resources.static-locations 1. application中需要先行的两个配置项 1.1 spring.mvc.static-path-pattern 这个配置项是告诉springboo......

moon888
昨天
4
0
hash slot(虚拟桶)

在分布式集群中,如何保证相同请求落到相同的机器上,并且后面的集群机器可以尽可能的均分请求,并且当扩容或down机的情况下能对原有集群影响最小。 round robin算法:是把数据mod后直接映射...

李朝强
昨天
4
0
Kafka 原理和实战

本文首发于 vivo互联网技术 微信公众号 https://mp.weixin.qq.com/s/bV8AhqAjQp4a_iXRfobkCQ 作者简介:郑志彬,毕业于华南理工大学计算机科学与技术(双语班)。先后从事过电子商务、开放平...

vivo互联网技术
昨天
24
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部