docker部署hdoop
博客专区 > centrald 的博客 > 博客详情
docker部署hdoop
centrald 发表于2年前
docker部署hdoop
  • 发表于 2年前
  • 阅读 14
  • 收藏 0
  • 点赞 1
  • 评论 0

新睿云服务器60天免费使用,快来体验!>>>   

docker基本用法:

2>下载镜像

docker pull ubuntu

1>启动镜像

docker run -ti ubuntu

2>列出所有镜像

docker images

3>列出所有的容器

docker ps -a

4>列出正在运行的容器

docker ps

5>启动容器

docker start  容器ID

6>进入容器

docker exec -ti 容器ID  /bin/bash

7>停止运行中的容器

docker stop 容器ID

8>删除容器

docker rm 容器ID

9>删除镜像

docker rmi 镜像ID

10>提交镜像

docker commit -m "提交信息" 镜像ID 镜像:标签

eg:docker commit -m "java install" c024e9ecb174 ubuntu:java

11>镜像导出

 docker save -o ubuntu_hadoop.tar.gz ubuntu:hadoop
12>镜像导入

 

安装Docker

1>apt-get install apt-transport-https

2> apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

3>bash -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"

4>apt-get update

5>apt-get install lxc-docker -y

安装JAVA环境

1>使用Docker镜像创建容器

docker run -ti ubuntu

2>更新数据源(使用网易的源)

# vim sources.list

 

deb http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse

deb http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse

deb http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse

deb http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse

deb http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse

deb-src http://mirrors.163.com/ubuntu/ trusty main restricted universe multiverse

deb-src http://mirrors.163.com/ubuntu/ trusty-security main restricted universe multiverse

deb-src http://mirrors.163.com/ubuntu/ trusty-updates main restricted universe multiverse

deb-src http://mirrors.163.com/ubuntu/ trusty-proposed main restricted universe multiverse

deb-src http://mirrors.163.com/ubuntu/ trusty-backports main restricted universe multiverse

# apt-get update

问题

  1. docker容器ping不同外网

    原因:默认DNS为谷歌服务器

    解决:#vi /etc/resolv.conf

    nameserver 223.5.5.5
    nameserver  223.6.6.6
     

3>.JAVA安装

    1>apt-get install software-properties-common python-software-properties
    2>add-apt-repository ppa:webupd8team/java
    3>apt-get update
    4>apt-get install oracle-java7-installer

    问题:java安装失败

     解决:下载tar包,进行安装配置

      一.让docker容器支持ssh服务

  1. 安装openssh-server

            root@d010c842a739:/# apt-get install openssh-server

        2.要正常启动SSH服务,需要/var/run/sshd存在,

            创建目录

            (1)root@d010c842a739:/# mkdir -p /var/run/sshd

            (2)启动sshd服务

            root@d010c842a739:/# /usr/sbin/sshd -D &
             [1] 593

        3.查看监听端口,22端口处于监听状态

        root@d010c842a739:/# netstat -antlpeu
        Active Internet connections (servers and established)
        Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode       PID/Program name
        tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          1114522     593/sshd        
        tcp6       0      0 :::22                   :::*                    LISTEN      0          1114524     593/sshd        
        4.修改SSH服务的安全登录配置,取消pam登录限制

        (1)root@d010c842a739:/# vim /etc/pam.d/sshd

        (2)注释pam限制

        #session    required     pam_loginuid.so

        5.创建ssh自动登录密钥

        (1)mkdir /root/.ssh

        (2) cd /root/.ssh/

        (3)将公钥拷贝到auth中

        vim authorized_keys

         (1>.ssh-keygen -t rsa 2.> vim id_rsa.pub)

        6.添加开机自启动ssh服务

        # vim ~/.bashrc

            #ssh autorun
            /usr/sbin/sshd  

        二.将jdk-7u80-linux-x64.tar.gz配置到docker中

            环境变量:

            1.#vim /etc/profile

            JAVA_HOME=/usr/local/jdk1.7
            export JAVA_HOME
            PATH=$JAVA_HOME/bin:$PATH
            export PATH
            2.使环境变量生效

            #source /etc/profile

            3.查看java版本,则配置成功

            # java -version
            java version "1.7.0_80"
4>将装好java环境的容器打包为镜像

# docker commit -m "java install" c024e9ecb174 ubuntu:java

配置hadoop环境

1>使用打包好的镜像创建容器

docker run -ti ubuntu:java

2>下载安装hadoop

安装目录:

/usr/local/apache/hadoop

3>配置hadoop环境变量

#vim /etc/profile

export HADOOP_HOME=/usr/local/apache/hadoop
export HADOOP_CONFIG_HOME=$HADOOP_HOME/etc/hadoop
export PATH=.:$HADOOP_HOME/bin:$PATH
export PATH=.:$HADOOP_HOME/sbin:$PATH

4>使环境变量生效

source /etc/profile

5>配置hadoop

1)创建三个目录

  • root@0c6acf88bac7:/usr/local/apache/hadoop# mkdir tmp
    root@0c6acf88bac7:/usr/local/apache/hadoop# mkdir namenode
    root@0c6acf88bac7:/usr/local/apache/hadoop# mkdir datanode

2)修改配置文件

1>root@0c6acf88bac7:/usr/local/apache/hadoop/etc/hadoop# vim core-site.xml

<configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://master:9000</value>
                <final>true</final>
                <description>master node</description>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/local/apache/hadoop/tmp</value>
                <description>Hadoop temporary directories</description>
        </property>
</configuration>

2>root@0c6acf88bac7:/usr/local/apache/hadoop/etc/hadoop# vim hdfs-site.xml

<configuration>
<property>
        <name>dfs.replication</name>
        <value>2</value>
        <final>true</final>
        <description>datanode count</description>
</property>
<property>
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/apache/hadoop/namenode</value>
        <final>true</final>
</property>
<property>
        <name>dfs.datanode.data.dir</name>
        <value>/usr/local/apache/hadoop/datanode</value>
        <final>true</final>
</property>
</configuration>

3>root@0c6acf88bac7:/usr/local/apache/hadoop/etc/hadoop# vim mapred-site.xml

<configuration>
<property>
        <name>mapred.job.tracker</name>
        <value>master:9001</value>
</property>
</configuration>

4>root@0c6acf88bac7:/usr/local/apache/hadoop/etc/hadoop# vim hadoop-env.sh

export JAVA_HOME=/usr/local/jdk1.7

6>格式化namenode

root@0c6acf88bac7:/usr/local/apache/hadoop/etc/hadoop# hadoop namenode -format

7>.将安装好hdoop环境的容器提交为镜像

docker commit -m "docker install" 容器ID ubuntu:hadoop

docker commit -m "hadoop install" 8ef06706f88d ubuntu:hadoop

 

1>配置IP与地址映射

vim /etc/hosts

172.17.0.2 master
172.17.0.3 slave1
172.17.0.4 slave2
 

2>

root@master:~# source /etc/profile
    javaapi下载

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class app {
	public static void main(String[] args) throws Exception {
		FileSystem fileSystem = FileSystem.get(new URI("hdfs://115.28.140.177:9000"), new Configuration());
		InputStream in = fileSystem.open(new Path("/dir1/hadoop-1.2.1.tar.gz"));
		OutputStream out = new FileOutputStream("e://hadoop.tar.gz");
		IOUtils.copyBytes(in, out, 4096,true);
	}
}

 

标签: docker hadoop
  • 打赏
  • 点赞
  • 收藏
  • 分享
共有 人打赏支持
粉丝 10
博文 67
码字总数 113149
×
centrald
如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
打赏人
留言
* 支付类型
微信扫码支付
打赏金额:
已支付成功
打赏金额: