HADOOP集群部署

原创
2017/02/12 10:38
阅读数 619

hadoop集群部署指导

#基础环境&软件

软件版本

* jdk-1.8u111-x64.tar.gz
* hbase-1.1.8.tar.gz
* apache-hive-1.1.1.tar.gz
* hadoop-2.6.5.tar.gz
* zookeeper-3.4.6.tar.gz

环境

  • OS CentOS release 6.6.x64
  • 服务器3台
server1:
hostname:bigdata1.localdomain 
ip:169.24.2.100


server2:
hostname:bigdata2.localdomain
ip:169.24.2.102

server3:
hostname:bigdata3.localdomain
ip:169.24.2.103

修改hosts

使用root用户登录每台服务器后执行下面的命令

echo 169.24.2.100 bigdata1.localdomain bigdata1 
echo 169.24.2.102 bigdata2.localdomain bigdata2
echo 169.24.2.103 bigdata3.localdomain bigdata3

新增hadoop用户

使用root用户登录每台服务器后执行下面的命令

useradd hadoop 
passwd hadoop #设置hadoop的密码

创建hadoop的部署目录并赋予hadoop权限

mkdir /hadoop_install 
chown -R hadoop:hadoop /hadoop_install

增加无密码登录

使用hadoop登录到bigdata1 执行下面的指令

ssh-keygen -t rsa #不要输入密码,避免在远程执行指令时要求输入密码

chmod 700 .ssh/authorized_keys 

将id_rsa.pub的文件分发到其他两台服务器上

ssh hadoop@bigdata2 "cat>>~/.ssh/authorized_keys"<~/.ssh/id_rsa.pub #本次是需要输入远程密码的

ssh hadoop@bigdata2 uname #验证是否需要输入密码

其他两台服务器上做同样的操作

时间同步

这一点十分重要,一定要保证服务器的时间是同步的,否则hbase启动会失败

配置基本软件

软件均部署在/hadoop_install目录下,请保证系统盘提供了足够大的空间

分别将软件

  • jdk-1.8u111-x64.tar.gz
  • hbase-1.1.8.tar.gz
  • apache-hive-1.1.1.tar.gz
  • hadoop-2.6.5.tar.gz

解压到/hadoop_install目录下,注意权限

chown -R hadoop:hadoop /hadoop_install 

### 配置hadoop

创建数据目录

mkdir -p /hadoop
chown hadoop:hadoop /hadoop 

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://169.24.2.100:9000</value>
        </property>
        <property>
                <name>io.file.buffer.size</name>
                <value>4096</value>
        </property>

</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
            <name>dfs.replication</name>
            <value>2</value>
    </property>
    <property>
        <name>dfs.datanode.max.xcievers</name>
        <value>4096</value>
    </property>
    <property>
            <name>dfs.nameservices</name>
            <value>hadoop-cluster1</value>
    </property>
    <property>
            <name>dfs.namenode.secondary.http-address</name>
            <value>169.24.2.100:50090</value>
    </property>
    <property>
            <name>dfs.webhdfs.enabled</name>
            <value>true</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/hadoop/hdfs/data/</value>
    </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->

 <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>169.24.2.100</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.resourcemanager.address</name>
                <value>169.24.2.100:8032</value>
        </property>
        <property>
                <name>yarn.resourcemanager.scheduler.address</name>
                <value>169.24.2.100:8030</value>
        </property>
        <property>
                <name>yarn.resourcemanager.resource-tracker.address</name>
                <value>169.24.2.100:8031</value>
        </property>
        <property>
                <name>yarn.resourcemanager.admin.address</name>
                <value>169.24.2.100:8033</value>
        </property>
        <property>
                <name>yarn.resourcemanager.webapp.address</name>
                <value>169.24.2.100:8088</value>
        </property>
        <!-- Site specific YARN configuration properties -->
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
                <final>true</final>
        </property>
        <property>
                <name>mapreduce.jobtracker.http.address</name>
                <value>169.24.2.100:50030</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>169.24.2.100:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>169.24.2.100:19888</value>
        </property>
        <property>
                <name>mapred.job.tracker</name>
                <value>http://169.24.2.100:9001</value>
        </property>
</configuration>

配置hbase

创建目录/hadoop/hbase

mkdir /hadoop/hbase  

配置zookeeper

zoo.cfg

文件如果不存在可以自行创建(zoo.cfg)

内容如下:

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# the port at which the clients will connect
clientPort=2181
dataDir=/hadoop/zookeeper

server.3=169.24.2.103:2888:3888
server.2=169.24.2.102:2888:3888
server.1=169.24.2.100:2888:3888
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

注意事项:

  • 需要在每台机器上的dataDir对应的目录下手动创建myid文件,文件的内容与server.x一致 比如server.1=169.24.2.100 则需要在169.24.2.100上创建myid文件且写入1,其他的类推

配置好后在每台机器上运行

zkServer.sh start 

配置HIVE

在每一个集群节点都需要同样的配置

  • hive-site.xml配置如下
<property>
    <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://169.24.2.40:3306/metastore_db?characterEncoding=UTF-8</value>
    <description/>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    <description/>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
        <value>dev</value>
    <description/>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
        <value>1qaz2wsx</value>
    <description/>
  </property>
  <property>
    <name>hive.metastore.uris</name>
        <value>thrift://169.24.2.100:9083</value>
    <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
  </property>

启动顺序

  • 分别在3台服务器上启动zookeeper
	zkServer.sh start 
  • 启动hadoop(仅在169.24.2.100上执行)
	start-dfs.sh 
	start-yarn.sh
	mr-jobhistory-daemon.sh start historyserver
  • 启动hbase(仅在169.24.2.100上执行)
    start-hbase.sh 
  • 启动hive元数据(仅在169.24.2.100上执行)
     nohup hive --service metastore -p 9083 & 

停止顺序

  • 停止 hbase
    stop-hbase.sh 
  • 停止hive的metastore
    netstat -tunlp|grep 9083|awk '{print $7}'|awk -F '/' '{print $1}'|xargs kill -9
  • 停止 hadoop
	mr-jobhistory-daemon.sh stop historyserver
	stop-yarn.sh 
	stop-dfs.sh 
  • 停止 zookeeper

分别到zookeeper集群的服务器上执行

zkServer.sh stop 

踩过的坑

  • hbase master is initializing.

本机引起的原因是多机器的时间不同步导致

  • Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

kylin在hadoop集群上运行build动作时提示以上错误,原因是yarn-site.xml中的yarn.application.classpath没有进行配置,处理方法如下

hadoop classpath  # 找classpath

修改 $HADOOP_HOME/etc/hadoop/yarn-site.xml

请根据实际情况进行修改

 <property>
                        <name>yarn.application.classpath</name>
                        <value>/hadoop_install/hadoop-2.6.5/etc/hadoop:/hadoop_install/hadoop-2.6.5/share/hadoop/common/lib/*:/hadoop_install/hadoop-2.6.5/share/hadoop/common/*:/hadoop_install/hadoop-2.6.5/share/hadoop/hdfs:/hadoop_install/hadoop-2.6.5/share/hadoop/hdfs/lib/*:/hadoop_install/hadoop-2.6.5/share/hadoop/hdfs/*:/hadoop_install/hadoop-2.6.5/share/hadoop/yarn/lib/*:/hadoop_install/hadoop-2.6.5/share/hadoop/yarn/*:/hadoop_install/hadoop-2.6.5/share/hadoop/mapreduce/lib/*:/hadoop_install/hadoop-2.6.5/share/hadoop/mapreduce/*:/hadoop_install/hadoop-2.6.5/contrib/capacity-scheduler/*.jar
                        </value>
                </property>

展开阅读全文
打赏
0
0 收藏
分享
加载中
大佬能给个QQ号码吗...有些kylin问题想请教,谢谢
2018/08/17 12:28
回复
举报

引用来自“胡佐治”的评论

引用来自“呼啦_小呆”的评论

引用来自“胡佐治”的评论

引用来自“呼啦_小呆”的评论

引用来自“呼啦_小呆”的评论

rm nodemaner 都重启了?然后就OK啦?

引用来自“胡佐治”的评论

如果有错误信息可以贴出来,一起分析分析
嗷 我的意思是 您把 resourcemanager nodemanager 都重启了 就好了吗?

回复@呼啦_小呆 : 博客里面有停止顺序的,我重启是将整个集群停止后重新启动。不光是resourcemanger node 和 nodemanager,当然你可以试一试。我当时是测试环境可以随便搞
刚才我看了一下 hadoop的源码,上面的办法可以解决 ,还有个办法是在kylin_job_conf.xml 增加一个mapreduce.application.classpath配置,这两个的作用都是在生成MRAPP的时候添加环境变量,所以我用后面的办法解决了,这样就不用修改集群了

回复@呼啦_小呆 : 你的这种方式不是不是不需要重启集群?
是的 因为是kylin提交MR 参数配置在提交mr的conf里面就好了
2017/05/24 18:29
回复
举报
胡佐治博主

引用来自“呼啦_小呆”的评论

引用来自“胡佐治”的评论

引用来自“呼啦_小呆”的评论

引用来自“呼啦_小呆”的评论

rm nodemaner 都重启了?然后就OK啦?

引用来自“胡佐治”的评论

如果有错误信息可以贴出来,一起分析分析
嗷 我的意思是 您把 resourcemanager nodemanager 都重启了 就好了吗?

回复@呼啦_小呆 : 博客里面有停止顺序的,我重启是将整个集群停止后重新启动。不光是resourcemanger node 和 nodemanager,当然你可以试一试。我当时是测试环境可以随便搞
刚才我看了一下 hadoop的源码,上面的办法可以解决 ,还有个办法是在kylin_job_conf.xml 增加一个mapreduce.application.classpath配置,这两个的作用都是在生成MRAPP的时候添加环境变量,所以我用后面的办法解决了,这样就不用修改集群了

回复@呼啦_小呆 : 你的这种方式不是不是不需要重启集群?
2017/05/24 18:19
回复
举报

引用来自“胡佐治”的评论

引用来自“呼啦_小呆”的评论

引用来自“呼啦_小呆”的评论

rm nodemaner 都重启了?然后就OK啦?

引用来自“胡佐治”的评论

如果有错误信息可以贴出来,一起分析分析
嗷 我的意思是 您把 resourcemanager nodemanager 都重启了 就好了吗?

回复@呼啦_小呆 : 博客里面有停止顺序的,我重启是将整个集群停止后重新启动。不光是resourcemanger node 和 nodemanager,当然你可以试一试。我当时是测试环境可以随便搞
刚才我看了一下 hadoop的源码,上面的办法可以解决 ,还有个办法是在kylin_job_conf.xml 增加一个mapreduce.application.classpath配置,这两个的作用都是在生成MRAPP的时候添加环境变量,所以我用后面的办法解决了,这样就不用修改集群了
2017/05/24 18:17
回复
举报
胡佐治博主

引用来自“呼啦_小呆”的评论

引用来自“呼啦_小呆”的评论

rm nodemaner 都重启了?然后就OK啦?

引用来自“胡佐治”的评论

如果有错误信息可以贴出来,一起分析分析
嗷 我的意思是 您把 resourcemanager nodemanager 都重启了 就好了吗?

回复@呼啦_小呆 : 博客里面有停止顺序的,我重启是将整个集群停止后重新启动。不光是resourcemanger node 和 nodemanager,当然你可以试一试。我当时是测试环境可以随便搞
2017/05/24 18:11
回复
举报

引用来自“呼啦_小呆”的评论

rm nodemaner 都重启了?然后就OK啦?

引用来自“胡佐治”的评论

如果有错误信息可以贴出来,一起分析分析
嗷 我的意思是 您把 resourcemanager nodemanager 都重启了 就好了吗?
2017/05/24 18:08
回复
举报
胡佐治博主

引用来自“呼啦_小呆”的评论

rm nodemaner 都重启了?然后就OK啦?
如果有错误信息可以贴出来,一起分析分析
2017/05/24 18:07
回复
举报
胡佐治博主

引用来自“呼啦_小呆”的评论

rm nodemaner 都重启了?然后就OK啦?
rm nodemaner?什么操作?
2017/05/24 18:03
回复
举报
rm nodemaner 都重启了?然后就OK啦?
2017/05/24 17:57
回复
举报
rm nodemaner 都重启了?然后就OK啦?
2017/05/24 17:57
回复
举报
更多评论
打赏
12 评论
0 收藏
0
分享
返回顶部
顶部