文档章节

Setting Up and Running Apache Kafka on Windows

jiejie2050
 jiejie2050
发布于 2016/07/11 16:10
字数 1056
阅读 56
收藏 0
点赞 0
评论 0

Introduction

This article is all about configuring and starting an Apache Kafka server on a Windows OS. This guide will also provide instructions to set up Java and Apache Zookeeper.

Apache Kafka is a fast and scalable messaging queue, capable of handeling heavy loads in context of read and write, i.e. IO-related stuff. You can find more about Kafka on http://kafka.apache.org/. Apache Kafka requires a running Zookeeper instance, which is used for reliable distributed cordination. Please find more about Zookeeper on https://zookeeper.apache.org/.

You can find help from this video for setting up Kafka on Windows.

 

Downloading the Required Files

For this tutorial we are assuming that Zookeeper and Kafka are unzipped in C: drive, but you can unzip them in any location.

Here we are using full-fledged zookeeper and not the one packaged with Kafka because it will be single-node Zookeeper instance. If you want you can run Kafka with packaged zookeeper located in Kafka package inside \kafka\bin\windows directory.

Installation

A. JDK Setup

1. Start JRE installation and hit the check box “Change destination folder”, then click install.        

Image title

2. Change the installation directory to any path without spaces in folder name. E.g. C:\Java\jre1.8.0_xx\. (By default it will be C:\Program Files\Java\jre1.8.0_xx) then click Next.

3. Now open the system environment variables dialogue by opening Control Panel -> System -> Advanced system settings -> Environment Variables.

4. Hit the New User Variable button in User variables section then type JAVA_HOME inVariable name and give your jre path in Variable value. It should look like:

        Image title(Java path and version may change according to the version of Kafka you are using)

5. Now click OK.

6. Search for a Path variable in the “System Variable” section in “Environment Variables” dialogue box you just opened.

7. Edit the path and type “;%JAVA_HOME%\bin” at the end of the text already written there just like image below

Image title

8. To confirm the Java installation just open cmd and type “java –version”, you should be able to see version of the java you just installed.

Image title

If your command prompt somewhat looks like the image above, you are good to go. Otherwise you need to recheck whether your setup version is matching with the correct OS architecture (x86, x64), or if the environment variables path is correct.

B. Zookeeper Installation

1. Goto your Zookeeper config directory. For me its C:\zookeeper-3.4.7\conf

2. Rename file “zoo_sample.cfg” to “zoo.cfg”

3. Open zoo.cfg in any text editor like notepad but I prefer notepad++.

4. Find & edit dataDir=/tmp/zookeeper to :\zookeeper-3.4.7\data

5. Add entry in System Environment Variables as we did for Java

    a. Add in System Variables ZOOKEEPER_HOME = C:\zookeeper-3.4.7

    b. Edit System Variable named “Path” add ;%ZOOKEEPER_HOME%\bin;

6. You can change the default Zookeeper port in zoo.cfg file (Default port 2181).

7. Run Zookeeper by opening a new cmd and type zkserver.

8. You will see the command prompt with some details like the image below.

Image title

Congratulations, your Zookeeper is up and running on port 2181!

C. Setting Up Kafka

1. Go to your Kafka config directory. For me its C:\kafka_2.11-0.9.0.0\config

2. Edit file “server.properties”

3. Find & edit line “log.dirs=/tmp/kafka-logs” to “log.dir= C:\kafka_2.11-0.9.0.0\kafka-logs”.

4. If your Zookeeper is running on some other machine or cluster you can edit“zookeeper.connect:2181” to your custom IP and port. For this demo we are using same machine so no need to change. Also Kafka port & broker.id are configurable in this file. Leave other settings as it is.

5. Your Kafka will run on default port 9092 & connect to zookeeper’s default port which is 2181.

D. Running Kafka Server

Important: Please ensure that your Zookeeper instance is up and running before starting a Kafka server.

1. Go to your Kafka installation directory C:\kafka_2.11-0.9.0.0\

2. Open a command prompt here by pressing Shift + right click and choose“Open command window here” option)

3. Now type .\bin\windows\kafka-server-start.bat .\config\server.properties and press Enter.

.\bin\windows\kafka-server-start.bat .\config\server.properties

Image title 

4. If everything went fine, your command prompt will look like this one:

Image title
5. Now your Kafka is up and running, you can create topics to store messages. Also we can produce or consume data from Java or Scala code or directly from the command prompt.

E. Creating topics

1. Now create a topic with name “test” and replication factor 1, as we have only one Kafka server running. If you have a cluster with more than 1 Kafka server running, you can increase the replication-factor accordingly which will increase the data availability and act like a fault-tolerant system.

2. Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows

3. Type following command and hit Enter:

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Image title

 

F. Creating a Producer and Consumer to Test Server

1. Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows

2. To start a producer type the following command:

kafka-console-producer.bat --broker-list localhost:9092 --topic test

3. Again open a new command prompt in the same location as C:\kafka_2.11-0.9.0.0\bin\windows

4. Now start a consumer by typing the following command:

kafka-console-consumer.bat --zookeeper localhost:2181 --topic test

5. Now you will have two command prompts like the image below:

Image title

 

6. Now type anything in the producer command prompt and press Enter, and you should be able to see the message in the other consumer command prompt.

Image title

7. If you are able to push & see your messages in the consumer side, you are done with Kafka setup.

Some Other Useful Commands

  1. List Topics: kafka-topics.bat --list --zookeeper localhost:2181
  2. Describe Topic: kafka-topics.bat --describe --zookeeper localhost:2181 --topic [Topic Name]
  3. Read messages from beginning: kafka-console-consumer.bat --zookeeper localhost:2181 --topic [Topic Name] --from-beginning
  4. Delete Topic: kafka-run-class.bat kafka.admin.TopicCommand --delete --topic [topic_to_delete] --zookeeper localhost:2181

本文转载自:https://dzone.com/articles/running-apache-kafka-on-windows-os

共有 人打赏支持
jiejie2050
粉丝 2
博文 51
码字总数 9570
作品 0
广州
Windows 安装运行 Apache Kafka 教程

下面是分步指南,教你如何在Windows OS上安装运行Apache Zookeeper和Apache Kafka。 简介 本文讲述了如何在Windows OS上配置并启动Apache Kafka,这篇指南将会指导你安装Java和Apache Zookee...

xrzs ⋅ 2012/08/26 ⋅ 0

Apache Kafka 0.11.0.0 发布,分布式消息发布订阅系统

Apache Kafka 0.11.0.0 发布了。kafka 是一种高吞吐量的分布式发布订阅消息系统。 New Feature [KAFKA-3487] - KIP-146: Support per-connector/per-task classloaders in Connect [KAFKA-42......

君枫 ⋅ 2017/06/29 ⋅ 5

Spring for Apache Kafka 1.1 GA 发布

Spring for Apache Kafka 1.1 GA 发布了,与先前的 Milestone 2 相比没有太多的变化,下面是更新内容: Apache Kafka upgrade; The batch of messages support via as well as via annotatio......

局长 ⋅ 2016/09/21 ⋅ 2

Apache Kafka 0.10.0.1 发布

Apache Kafka 0.10.0.1 发布,这是一个 bug 修复版本,主要改进记录包括: New Feature [KAFKA-3538] - Abstract the creation/retrieval of Producer for stream sinks for unit testing I......

oschina ⋅ 2016/08/11 ⋅ 4

Apache Kafka 0.10.1.0 发布,大量更新

Apache Kafka 0.10.1.0 发布了,该版本更新了大量内容,主要改进如下: 新特性 [KAFKA-1464] - Add a throttling option to the Kafka replication tool [KAFKA-3176] - Allow console cons......

局长 ⋅ 2016/10/21 ⋅ 4

Kafka 快速安装使用

Quick Start This tutorial assumes you are starting fresh and have no existing Kafka or ZooKeeper data. Step 1: Download the code Download the 0.8.2.0 release and un-tar it. > ta......

啊莫 ⋅ 2015/11/05 ⋅ 0

Apache Apex Malhar 3.3.1-incubating 发布

Apache Apex Malhar Release 3.3.1-incubating 发布了,Apache Apex 是一个企业级的统一流和批处理引擎。提供高度可伸缩、高性能、容错、有状态、安全和分布式的大数据处理,同时操作起来非常...

oschina ⋅ 2016/03/08 ⋅ 2

Apache Storm 0.9.6/0.10.0 发布

Apache Storm 0.10.0 发布,此版本是个稳定版本,相比之前的 Beta 版本主要包括 bug 修复和改进: STORM-1108: Fix NPE in simulated time STORM-1106: Netty should not limit attempts to...

oschina ⋅ 2015/11/06 ⋅ 8

Apache Kafka 0.10.1.1 发布,分布式消息发布订阅系统

Apache Kafka 0.10.1.1 发布了。 更新内容: 提升 [KAFKA-4302] - Simplify KTableSource [KAFKA-4331] - Kafka Streams resetter is slow because it joins the same group for each topic ......

淡漠悠然 ⋅ 2017/02/22 ⋅ 2

Install Redmine on Centos 6.5

Install Redmine on Centos 6.5 - 64 bit Install Redmine on Centos 6.5 - 64 bit The System Requirements During the installation process we will use the Centos 6.5 - 64 bit OS, the......

Ericklee ⋅ 2015/01/22 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

如何使用serverchan微信推送告警

之前实现推送告警信息到微信的方法有如下几种: 1、通过企业公众号实现----收费: 2、通过QQ邮箱,在微信平台上开启收到邮件进行提醒; 3、第三方告警平台API,一般也是收费的; 不过最近看文...

问题终结者 ⋅ 19分钟前 ⋅ 0

TCP的RPC

RPC就是远程方法调用(Remote Process Call ),包含了客户端和服务端,涉及了对象的序列化传输。 1.服务端启动,注册远程调用的类2.客户端发送请求信息包含类、方法、参数的一些信息、序列化传...

Cobbage ⋅ 40分钟前 ⋅ 0

IOS-UI UI初步代码布局添加事件

ISO开发界面,UI是必须学习的一部分,其实很早之前想学来了,一直没有沉下心来学习。看到IOS的代码风格和布局就别扭的不行,跟java代码和android布局比较显得不是那么方便,所以一直到现在。...

京一 ⋅ 50分钟前 ⋅ 0

浅谈OpenDaylight的二次开发

OpenDaylight作为一款开源SDN网络控制器,依托于强大的社区支持以及功能特性,成为了目前主流的SDN网络控制器开发平台。在比较稳定的OpenDaylight Helium版本中,已经为开发者提供了大量的网...

wangxuwei ⋅ 59分钟前 ⋅ 0

API 开发中可选择传递 token 接口遇到的一个坑

在做 API 开发时,不可避免会涉及到登录验证,我使用的是jwt-auth 在登录中会经常遇到一个token过期的问题,在config/jwt.php默认设置中,这个过期时间是一个小时,不过为了安全也可以设置更...

等月人 ⋅ 今天 ⋅ 0

Java NIO之文件处理

程序要操作本地操作系统的一个文件,可以分为以下三个部分: 对文件位置的操作 对文件的操作 对文件内容的操作 其中,对文件内容的操作在 Java NIO之Channel 中已经有了介绍,通过FileChann...

士别三日 ⋅ 今天 ⋅ 0

Maven的pom.xml配置文件详解

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.......

小海bug ⋅ 今天 ⋅ 0

解决httpclient超时设置不生效的问题

最近公司有项目需要通过http调用第三方服务,且第三方服务偶有超时,故需要设置一定的超时时间防止不响应的情况出现。 初始设置如下: [java] view plain copy //超时设置 RequestConfig re...

Mr_Tea伯奕 ⋅ 今天 ⋅ 0

过滤器Filter和拦截器HandlerInterceptor

过滤器 依赖于servlet容器。在实现上基于函数回调,可以对几乎所有请求进行过滤,但是缺点是一个过滤器实例只能在容器初始化时调用一次。使用过滤器的目的是用来做一些过滤操作,获取我们想要...

hutaishi ⋅ 今天 ⋅ 0

Redis入门详解(转)

Redis入门详解 Redis简介 Redis安装 Redis配置 Redis数据类型 Redis功能 持久化 主从复制 事务支持 发布订阅 管道 虚拟内存 Redis性能 Redis部署 Redis应用场景 Redis总结 Redis简介: Redi...

xiaoyaoyoufang ⋅ 今天 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部