文档章节

hadoop 集群环境搭建

清清浅浅
 清清浅浅
发布于 2017/04/26 15:42
字数 1208
阅读 11
收藏 1

参考文献:

http://www.tuicool.com/articles/NvueEjM

 

http://blog.csdn.net/dangyifei/article/details/8920164 比较好

 

http://blog.csdn.net/w13770269691/article/details/24457241

 

http://www.tuicool.com/articles/AbQBz2

 

 

前提条件:

1.下载

下载hadoop和zookeeper、jdk

hadoop-2.8.0.tar.gz

zookeeper-3.4.10.tar.gz

jdk1.8.0_121

2.修改/etc/hosts文件

增加node 节点。

node1 172.22.14.107

node3 172.22.14.172

node4 172.22.14.169

3.修改hostname

注意:在命令行把 hostname 设置成 本机的ip,例如:

机器172.22.14.107执行

hostname 172.22.14.107

机器172.22.14.172执行

hostname 172.22.14.172

机器172.22.14.169执行

hostname 172.22.14.169

4.设置公钥

ssh 公钥配置

在node1上执行

ssh-keygen

ssh-copy-id root@node3

ssh-copy-id root@node4

类似在 node3,node4上执行

====================================================================

搭建步骤:

在 /etc/profile 问价下面修改

1.配置JAVA_HOME

export JAVA_HOME=/opt/jdk1.8.0_121

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

2.配置HADOOP_HOME

export HADOOP_HOME=/hadoop/hadoop

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native"

 

 

 

3. 修改zookeeper的配置文件

zookeeper/conf/zoocfg

# milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/hadoop/zookeeper/data

# the port at which the clients will connect

clientPort=2181

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=60

#

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

#

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

 

server.1=node1:2888:3888

server.3=node3:2888:3888

server.4=node4:2888:3888

 

在/hadoop/zookeeper/data 建立 myid文件

172.22.14.107(node1) 机器的的myid文件里面是 :1

172.22.14.172 (node3)机器的的myid文件里面是 :3

172.22.14.169 (node4)机器的的myid文件里面是 :4

这个id适合 server.id 对应的。

4.启动zoopeeper

在node1,node3,node4 上执行

./zkServer.sh start &

=========================以上是zookeeper的配置====================

=========================以下是hadoop hdfs的配置==================

5.修改hadoop的配置文件

(1)/hadoop/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

 

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

 

http://www.apache.org/licenses/LICENSE-2.0

 

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

 

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>fs.defaultFS</name>

<value>hdfs://hdcluster</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/hadoop/tmp</value>

</property>

 

<property>

<name>ha.zookeeper.quorum</name>

<value>node1:2181,node3:2181,node4:2181</value>

</property>

</configuration>

(2)hadoop/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

 

http://www.apache.org/licenses/LICENSE-2.0

 

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

 

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>dfs.replication</name>

<value>3</value>

</property>

<property>

<name>dfs.nameservices</name>

<value>hdcluster</value>

</property>

<property>

<name>dfs.ha.namenodes.hdcluster</name>

<value>node1,node3</value>

</property>

<property>

<name>dfs.namenode.rpc-address.hdcluster.node1</name>

<value>node1:9000</value>

</property>

<property>

<name>dfs.namenode.rpc-address.hdcluster.node3</name>

<value>node3:9000</value>

</property>

<property>

<name>dfs.namenode.http-address.hdcluster.node1</name>

<value>node1:50070</value>

</property>

<property>

<name>dfs.namenode.http-address.hdcluster.node3</name>

<value>node3:50070</value>

</property>

<property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://node1:8485;node3:8485;node4:8485/hdcluster</value>

</property>

<property>

<name>dfs.client.failover.proxy.provider.hdcluster</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<property>

<name>dfs.ha.fencing.methods</name>

<value>sshfence</value>

</property>

<property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/root/.ssh/id_rsa</value>

</property>

<property>

<name>dfs.journalnode.edits.dir</name>

<value>/usr/local/hadoop/tmp/journal</value>

</property>

<property>

<name>dfs.ha.automatic-failover.enabled.hdcluster</name>

<value>true</value>

</property>

</configuration>

 

(3)yarn-site.xml

 

<?xml version="1.0"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

 

http://www.apache.org/licenses/LICENSE-2.0

 

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

<configuration>

 

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

 

</configuration>

 

(4)mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

 

http://www.apache.org/licenses/LICENSE-2.0

 

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License. See accompanying LICENSE file.

-->

 

<!-- Put site-specific property overrides in this file. -->

 

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

 

(5)设置环境变量 hadoop-env.sh

 

# The java implementation to use.

 

export JAVA_HOME=/opt/jdk1.8.0_121

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

 

export HADOOP_HOME=/hadoop/hadoop

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native"

 

export JAVA_HOME=${JAVA_HOME}

 

6.hadoop启动

(1)启动journalnode

hadoop-daemon.sh start journalnode

(2)启动zkfc

在node1,node3上启动namenode选举进程

hadoop-daemon.sh start zkfc

(3)启动namenode

格式化namenode

在node1,node3上

hadoop-daemon.sh namenode -format

hadoop-daemon.sh start namenode

(4)启动datanode

在出了node1,和node3的节点上启动datanode

hadoop-daemon.sh start datanode

 

 

namenode 节点查看地址

http://172.22.14.172:50070/dfshealth.html#tab-overview

hbase 节点查看地址

http://172.22.14.72:16010/tablesDetailed.jsp

© 著作权归作者所有

清清浅浅
粉丝 7
博文 21
码字总数 4394
作品 0
西安
高级程序员
私信 提问
(14)Spark 2.0.0 集群部署

1.引言 在前面的博客中介绍了Spark的伪分布安装,在本篇博客中将介绍一下Spark的集群部署,在本博客中Spark的调度采用Hadoop的Yarn平台,因此在安装Spark之前,需要先部署的集群,因为我在安...

LoveCarpenter
2017/12/15
0
0
Spark2.0.0集群部署教程

1.引言 在前面的博客中介绍了Spark的伪分布安装,在本篇博客中将介绍一下Spark的集群部署,在本博客中Spark的调度采用Hadoop的Yarn平台,因此在安装Spark之前,需要先部署Hadoop的集群,因为...

Liri
2017/12/16
0
0
(11)Hadoop 2.6.1 集群部署---手工切换HA

1. 引言 在生产环境中的高可用是非常重要的,一旦宕机,那么整合集群就不能用了,在本篇博客中就主要介绍一下的部署(手动切换),在本篇博客中博主使用五台干净的机器从零开始搭建集群环境,...

LoveCarpenter
2017/12/15
0
0
hbase分布式集群搭建

hbase和hadoop一样也分为单机版、伪分布式版和完全分布式集群版本,这篇文件介绍如何搭建完全分布式集群环境搭建。 hbase依赖于hadoop环境,搭建habase之前首先需要搭建好hadoop的完全集群环...

纯洁的虫纸
2017/08/11
0
0
(12)Hadoop 2.6.1 集群部署---自动切换HA

1. 引言 在上篇博客中介绍了的手动切换,这在生产环境中还是不尽人意,因为工作人员可能并不会及时的知道什么时候宕机?因此在生产环境中实现的动态切换是十分有必要的,在本篇博客中就主要介...

LoveCarpenter
2017/12/15
0
0

没有更多内容

加载失败,请刷新页面

加载更多

OSChina 周二乱弹 —— 开发语言和语言开发的能一样么

Osc乱弹歌单(2019)请戳(这里) 【今日歌曲】 @花间小酌:#今日歌曲推荐# 分享The Score的单曲《Revolution》 《Revolution》- The Score 手机党少年们想听歌,请使劲儿戳(这里) @批判派...

小小编辑
今天
1K
17
oracle ORA-39700: database must be opened with UPGRADE option

ORA-01092: ORACLE instance terminated. Disconnection forced ORA-00704: bootstrap process failure ORA-39700: database must be opened with UPGRADE option 进程 ID: 3650 会话 ID: 29......

Tank_shu
今天
3
0
分布式协调服务zookeeper

ps.本文为《从Paxos到Zookeeper 分布式一致性原理与实践》笔记之一 ZooKeeper ZooKeeper曾是Apache Hadoop的一个子项目,是一个典型的分布式数据一致性的解决方案,分布式应用程序可以基于它...

ls_cherish
今天
4
0
聊聊DubboDefaultPropertiesEnvironmentPostProcessor

序 本文主要研究一下DubboDefaultPropertiesEnvironmentPostProcessor DubboDefaultPropertiesEnvironmentPostProcessor dubbo-spring-boot-project-2.7.3/dubbo-spring-boot-compatible/au......

go4it
昨天
3
0
redis 学习2

网站 启动 服务端 启动redis 服务端 在redis 安装目录下 src 里面 ./redis-server & 可以指定 配置文件或者端口 客户端 在 redis 的安装目录里面的 src 里面 ./redis-cli 可以指定 指定 连接...

之渊
昨天
2
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部