文档章节

Create a hadoop1.1.1 Cluster on Cent OS 6.5 (host

幸运的幸福
 幸运的幸福
发布于 2014/08/12 11:55
字数 545
阅读 12
收藏 0

1. Prepare three Cent Os hosts for poc

10.28.241.174 shuynh-gecko1
10.28.241.172 shuynh-gecko2
10.28.241.175 shuynh-gecko3

root@shuynh-gecko1:~# cat /etc/os-release


2. Get source code of docker-scripts on each node


root@shuynh-gecko1:/# git clone https://github.com/jay-lau/hadoop-docker-master-cluster-111.git
Cloning into 'hadoop-docker-master-cluster-111'...
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 16 (delta 1), reused 16 (delta 1)
Unpacking objects: 100% (16/16), done.
Checking connectivity... done.

root@shuynh-gecko2:/# git clone https://github.com/jay-lau/hadoop-docker-master-cluster-111.git

root@shuynh-gecko3:/# git clone https://github.com/jay-lau/hadoop-docker-master-cluster-111.git

3. Build Hadoop docker on each node

#enter the folder of  the hadoop-docker-master-cluster-111 scripts.

3.1 build the images on shuynh-gecko1

[root@shuynh-gecko1 ~]#cd /root/gyliu/hadoop-docker-master-111
[root@shuynh-gecko1 hadoop-docker-master-111]# pwd                                                                                  
/root/gyliu/hadoop-docker-master-111                                                                                               
[root@shuynh-gecko1 hadoop-docker-master-111]# docker build -t="sequenceiq/hadoop-cluster-docker:1.1.1" .

[root@shuynh-gecko1 ~]# docker images | grep 1.1.1                                                          
sequenceiq/hadoop-cluster-docker    1.1.1               742b4ff50735        About a minute ago   805.7 MB


3.2 build the images on shuynh-gecko2

[root@shuynh-gecko2 ~]#cd /root/gyliu/hadoop-docker-master-111
[root@shuynh-gecko2 hadoop-docker-master-111]# pwd                                                                                  
/root/gyliu/hadoop-docker-master-111                                                                                               
[root@shuynh-gecko2 hadoop-docker-master-111]# docker build -t="sequenceiq/hadoop-cluster-docker:1.1.1" .

3.3 build the images on shuynh-gecko3

[root@shuynh-gecko3 ~]#cd /root/gyliu/hadoop-docker-master-111
[root@shuynh-gecko3 hadoop-docker-master-111]# pwd                                                                                  
/root/gyliu/hadoop-docker-master-111                                                                                               
[root@shuynh-gecko3 hadoop-docker-master-111]# docker build -t="sequenceiq/hadoop-cluster-docker:1.1.1" .   

4. Configure /etc/hosts, and ssh passwordless file  for each node

#configure /etc/hosts file on every nodes
10.28.241.174 shuynh-gecko1
10.28.241.172 shuynh-gecko2
10.28.241.175 shuynh-gecko3

#ssh passwordless for each node

....

5. Create Hadoop Cluster

# Start a container

docker run   --net=host  sequenceiq/hadoop-cluster-docker:2.4.1 $1 $2 $3
Params definition as below:
$1:Type of Namenode or Datanode, such as N | D
$2:Master Node IP address, such as 10.28.241.174
$3:Default command, run as service "-d", run as interactive "-bash"


#If we need run interactive, please add "-i -t " options.

5.1 Create NameNode on shuynh-gecko1:

[root@shuynh-gecko1 ~]# docker stop $(docker ps -a -q)
[root@shuynh-gecko1 ~]# docker rm $(docker ps -a -q)
[root@shuynh-gecko1 ~]# docker run -i -t --net="host" sequenceiq/hadoop-cluster-docker:1.1.1 N 10.28.241.174 -bash

bash-4.1# jps                                                                                                                       

119 JobTracker                                                                                                                     
596 TaskTracker
528 DataNode                                                                                                                       
389 NameNode

5.2 Create DataNode (backend service, using -d) on shuynh-gecko2:
[root@shuynh-gecko2 ~]# docker stop $(docker ps -a -q)
[root@shuynh-gecko2 ~]# docker rm $(docker ps -a -q)
[root@shuynh-gecko2 hadoop-docker-master-cluster]#docker run  --net="host" sequenceiq/hadoop-cluster-docker:1.1.1 D 10.28.241.174 -d

5.3 Create DataNode (backend service, using -d) on shuynh-gecko3:
[root@shuynh-gecko2 ~]# docker stop $(docker ps -a -q)
[root@shuynh-gecko2 ~]# docker rm $(docker ps -a -q)
[root@shuynh-gecko2 hadoop-docker-master-cluster]# docker run  --net="host" sequenceiq/hadoop-cluster-docker:1.1.1 D 10.28.241.174 -d

6. Check the cluster status

6.1 Access the WEB GUI

Access http://10.28.241.174:50070/dfshealth.html#tab-datanode


6.2 Using command line to check the status

bash-4.1# $HADOOP_PREFIX/bin/hadoop dfsadmin -report

7. Run a sample hadoop case

#create testing data

bash-4.1$HADOOP_PREFIX/bin/hadoop fs -mkdir -p /user/root
bash-4.1$HADOOP_PREFIX/bin/hadoop fs -put $HADOOP_PREFIX/conf/ input
#bash-4.1$HADOOP_PREFIX/bin/hadoop fs  -rm -r output

#run sample testing

bash-4.1$HADOOP_PREFIX/bin/hadoop jar $HADOOP_PREFIX/hadoop-examples-1.1.1.jar wordcount input output

#check the output

bash-4.1$HADOOP_PREFIX/bin/hadoop fs -cat output/*



© 著作权归作者所有

幸运的幸福
粉丝 0
博文 5
码字总数 3821
作品 0
朝阳
私信 提问
Teamtalk问题请教

@南湖船老大 你好,想跟你请教个问题: 我今天部署了Teamtalk,系统是cent os 6.5 但服务也启动成功,但是访问web的时候提示404??这是什么问题?

chrisyang
2016/06/21
373
2
mycat-1.5.1[NIOREACTOR err Got packets out of order code:1156]

mycat:1.5.1 cent-os:6.5 jdk:1.7 mysql:5.6.26 偶然会出现以下异常: WARN [$_NIOREACTOR-3-RW] (MultiNodeHandler.java:134) -error response from MySQLConnection [id=58496, lastT......

thinkingtime
2017/03/27
313
0
Install Redmine on Centos 6.5

Install Redmine on Centos 6.5 - 64 bit Install Redmine on Centos 6.5 - 64 bit The System Requirements During the installation process we will use the Centos 6.5 - 64 bit OS, the......

Ericklee
2015/01/22
0
0
Install Redmine on Centos 6.5 - 64 bit

Install Redmine on Centos 6.5 - 64 bit Install Redmine on Centos 6.5 - 64 bit The System Requirements Update the System Install the dependencies packages Install Apache and MySQ......

xiaoxin
2014/08/25
1.8K
1
VirtualBox中Cent OS 6.5 桥接模式设置

目标:为虚拟机设置静态ip,虚拟机和宿主机能相互ping通,虚拟机能访问外网。 桥接模式:虚拟机就像是局域网中的一台独立的主机,它可以访问网内任何一台机器。在桥接模式下,你需要手工为虚...

fishlord
2016/09/05
78
0

没有更多内容

加载失败,请刷新页面

加载更多

MySQL-基于SELECT查询的UPDATE查询

我需要检查(从同一张表)基于日期时间的两个事件之间是否存在关联。 一组数据将包含某些事件的结束日期时间,另一组数据将包含其他事件的开始日期时间。 如果第一个事件在第二个事件之前完成...

javail
47分钟前
60
0
将PostgreSQL数据库复制到另一台服务器

我正在将生产PostgreSQL数据库复制到开发服务器。 什么是最快,最简单的方法? #1楼 pg_dump the_db_name > the_backup.sql 然后将备份复制到您的开发服务器,并使用以下命令进行还原: ps...

技术盛宴
今天
39
0
[译]软件架构师之路

今天给大家带来一篇自己翻译的干货《软件架构师之路》。本周Github上升很快的项目。其内容对致力于成为软件架构师(不论前后端)的同学应该都会有极大的帮助。 项目地址: 中文地址 https://...

gamedilong
今天
41
0
聊聊artemis的ExpiryScanner

序 本文主要研究一下artemis的ExpiryScanner startExpiryScanner activemq-artemis-2.11.0/artemis-server/src/main/java/org/apache/activemq/artemis/core/postoffice/impl/PostOfficeImpl......

go4it
今天
51
0
MySQL概述

MySql主要概念 参考:https://snailclimb.gitee.io/javaguide/#/database/MySQL MySQL 是一种关系型数据库,在Java企业级开发中非常常用,因为 MySQL 是开源免费的,并且方便扩展。阿里巴巴数...

Las_Vegas
今天
42
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部