文档章节

cdh6安装

弘_轩
 弘_轩
发布于 01/10 17:10
字数 2246
阅读 77
收藏 1

打通SSH,设置ssh无密码登陆(所有节点)

1、在每台服务器上都执行ssh-keygen -t rsa生成密钥对:				
ssh-keygen -t rsa,一路回车,生成无密码的密钥对
//将公钥添加到认证文件中:cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
//设置authorized_keys的访问权限:chmod 600 ~/.ssh/authorized_keys
2、在每台服务器上生成密钥对后,将公钥复制到其它结点上:
ssh-copy-id -i  ~/.ssh/id_rsa.pub root@hd241
ssh-copy-id -i  ~/.ssh/id_rsa.pub root@hd242

配置主机名和host(所有节点)

1、vim /etc/hostname -> sjhl-ad-hd1
2、hostnamectl set-hostname sjhl-ad-hd1 # 使其立刻生效
3、vim /etc/hosts 增加
192.168.1.29    sjhl-ad-hd1
192.168.1.30    sjhl-ad-hd2
192.168.1.185   sjhl-ad-hd3
192.168.1.186   sjhl-ad-hd4
192.168.1.187   sjhl-ad-hd5
192.168.1.188   sjhl-ad-hd6
192.168.1.189   sjhl-ad-hd7
192.168.1.190   sjhl-ad-hd8
(
4、配置 vim /etc/sysconfig/network
NETWORKING_IPV6=no
PEERNTP=no
HOSTNAME=sjhl-ad-hd1
重启网络:service network restart
测试:host -v -t A $(hostname)
) 

在线安装-配置 Cloudera Manager 仓库(所有节点) wget https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/ rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera


离线安装

1、CM6 RPM:https://archive.cloudera.com/cm6/6.0.1/redhat7/yum/RPMS/x86_64/ 
下载
├── cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
└── cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm

2、下载ASC文件:https://archive.cloudera.com/cm6/6.0.1/allkeys.asc

3、CDH6 Parcels:https://archive.cloudera.com/cdh6/6.0.1/parcels/
下载CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel和manifest.json这两个文件

配置Cloudera Manager yum库(一台master节点)

1、安装httpd和createrepo:
yum -y install httpd createrepo

2、启动httpd服务并设置开机自启动:
systemctl start httpd
systemctl enable httpd

3、在/var/www/html/下创建目录
mkdir /var/www/html/cloudera-repos/

4、把下载的CM6 RPM 和 ASC文件 移到 /var/www/html/cloudera-repos/
/var/www/html/cloudera-repos/
├── allkeys.asc
├── cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm
├── cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm

5、cd /var/www/html/cloudera-repos/
生成RPM元数据:
createrepo .

[root@sjhl-ad-hd1 cloudera-repos]# createrepo .
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

6、确保通过浏览器http://sjhl-ad-hd1/cloudera-repos/ 可以看到
或者:curl http://sjhl-ad-hd1/cloudera-repos/
<html>
<head>
  <title>Index of /cloudera-repos</title>
 </head>
 <body>
<h1>Index of /cloudera-repos</h1>
  <table>
   <tr><th valign="top"><img src="/icons/blank.gif" alt="[ICO]"></th><th><a href="?C=N;O=D">Name</a></th><th><a href="?C=M;O=A">Last modified</a></th><th><a href="?C=S;O=A">Size</a></th><th><a href="?C=D;O=A">Description</a></th></tr>
   <tr><th colspan="5"><hr></th></tr>
<tr><td valign="top"><img src="/icons/back.gif" alt="[PARENTDIR]"></td><td><a href="/">Parent Directory</a>       </td><td>&nbsp;</td><td align="right">  - </td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/text.gif" alt="[TXT]"></td><td><a href="allkeys.asc">allkeys.asc</a>            </td><td align="right">2018-10-11 20:35  </td><td align="right"> 14K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-age..&gt;</a></td><td align="right">2018-12-19 18:22  </td><td align="right"> 25M</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-dae..&gt;</a></td><td align="right">2018-12-19 18:22  </td><td align="right">1.0G</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-ser..&gt;</a></td><td align="right">2018-12-19 18:24  </td><td align="right">8.0K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/unknown.gif" alt="[   ]"></td><td><a href="cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm">cloudera-manager-ser..&gt;</a></td><td align="right">2018-12-19 18:24  </td><td align="right"> 11K</td><td>&nbsp;</td></tr>
<tr><td valign="top"><img src="/icons/folder.gif" alt="[DIR]"></td><td><a href="repodata/">repodata/</a>              </td><td align="right">2018-12-20 09:29  </td><td align="right">  - </td><td>&nbsp;</td></tr>
   <tr><th colspan="5"><hr></th></tr>
</table>
</body>
</html>

配置本地源(所有节点)

1、vim /etc/yum.repos.d/cloudera-manager.repo
编辑以下内容:
[cloudera-manager]
name = Cloudera Manager, Version
baseurl = http://sjhl-ad-hd1/cloudera-repos/
gpgcheck = 1

2、保存,退出,然后执行yum clean all && yum makecache命令

关闭防火墙或者设置iptables集群内部白名单(所有节点)

service iptables stop

systemctl stop firewalld.service
systemctl disable firewalld.service

查看iptables规则:iptables -L -n -v --line-number

关闭SELinux(所有节点)

sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
setenforce 0

禁用透明大页面压缩(所有节点)

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

禁用交换分区(所有节点)

echo "vm.swappiness = 0" >> /etc/sysctl.conf
sysctl -p

修改文件句柄数和缓存页(所有节点)

vim /etc/security/limits.conf

*       soft    nofile  655350
*       hard    nofile  655350
*       soft    nproc   102400
*       hard    nproc   102400
*       soft    memlock unlimited
*       hard    memlock unlimited

安装jdk8(所有节点)

1、下载地址:https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html

2、解压,如果不是安装在/usr/java/ 下,创建软连接
ln -s /data/jdk1.8.0_181 /usr/java/jdk1.8.0_181

2、设置环境变量
JAVA_HOME=/usr/java/jdk1.8.0_181
JRE_HOME=$JAVA_HOME/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH

下载mysql jdbc 驱动(所有节点)

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz

mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

安装ntp服务(所有节点)

yum install ntp
systemctl start ntpd  
chkconfig ntpd on #设置开机启动

1、选master作为ntp服务,vim /etc/ntp.conf 配置如下:

driftfile /var/lib/ntp/drift   #这是预设时间差异分析文件,我们暂时用不到,就这样留着
pidfile    /var/run/ntpd.pid
logfile    /var/log/ntp.log


restrict    default kod nomodify notrap nopeer noquery   #拒绝ipv4的用户
restrict -6 default kod nomodify notrap nopeer noquery   #拒绝ipv6的用户
restrict 127.0.0.1  #放行本机来源
restrict 192.168.0.0 mask 255.255.0.0 nomodify notrap nopeer noquery
restrict 120.133.0.0 mask 255.255.255.0 nomodify notrap nopeer noquery
# restrict 0.0.0.0 mask 0.0.0.0  nomodify   放行所有来源,这里可以写192.169.0.0. mask 255.255.255.0 只放行一个网断


server 127.127.1.0  #外界同步源无法联系时,使用本地时间为同步服务
fudge  127.127.1.0 stratum 10  #这行是时间服务器的层次。设为0则为顶级,如果要向别的NTP服务器更新时间,请不要把它设为0

server 0.cn.pool.ntp.org prefer  #以这个主机最为优先
server ntp1.aliyun.com iburst minpoll 4 maxpoll 10
server ntp2.aliyun.com iburst minpoll 4 maxpoll 10
server ntp3.aliyun.com iburst minpoll 4 maxpoll 10
server ntp4.aliyun.com iburst minpoll 4 maxpoll 10
server ntp5.aliyun.com iburst minpoll 4 maxpoll 10
server ntp6.aliyun.com iburst minpoll 4 maxpoll 10

2、其他节点配置为客户端即可
driftfile /var/lib/ntp/drift
pidfile    /var/run/ntpd.pid
logfile    /var/log/ntp.log

restrict    default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict 192.168.0.0 mask 255.255.0.0 nomodify notrap nopeer noquery
restrict 120.133.0.0 mask 255.255.255.0 nomodify notrap nopeer noquery


server 127.127.1.0
fudge  127.127.1.0 stratum 10

server sjhl-ad-hd1 prefer
server sjhl-ad-hd2 iburst
server 0.cn.pool.ntp.org iburst minpoll 4 maxpoll 10

3、为每个节点增加手动配置 crontab -e
01 * * * * /usr/sbin/ntpdate 0.cn.pool.ntp.org >/tmp/updateclock.log 2>&1

4、查看同步状态: ntpdc -np

安装mysql(CM Server节点)

1、CentOS 7.6 自带的 MariaDB:
yum install mariadb-server -y
修改mysql配置(vim /etc/my.cnf)
[mysqld]
datadir=/data/mysql/data
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED

symbolic-links=0

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550

log_bin=/data/mysql/log_bin

server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M


# InnoDB settings
# innodb_file_per_table = 1
# innodb_flush_log_at_trx_commit  = 2
# innodb_log_buffer_size = 64M
# innodb_buffer_pool_size = 4G
# innodb_thread_concurrency = 8
# innodb_flush_method = O_DIRECT
# innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid

!includedir /etc/my.cnf.d
启动数据库
chown -R mysql:mysql /data/mysql/

systemctl start mariadb
systemctl enable mariadb
初始化数据库
/usr/bin/mysql_secure_installation
按照下面提示输入
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
[...]
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
[...]
All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!
创建数据库
#初始化密码
mysqladmin -u root password "pwd"
#set password = password('pwd');
mysql -u root –p

grant all privileges on *.* to 'root'@'%' identified by 'pwd';
flush privileges;

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'pwd';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'pwd';

FLUSH PRIVILEGES;

安装CM Server(server 节点)

yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

安装CM agent(所有节点)

#先安装,后面通过cm安装的时候会快,不用等待
yum install cloudera-manager-daemons cloudera-manager-agent

配置本地Parcel(CM Server 节点)

1、cm server安装后,进入Parcel目录
cd /opt/cloudera/parcel-repo

2、将之前下载的Parcel文件移进该目录
CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
manifest.json

3、执行命令生成sha文件
sha1sum CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel | awk '{ print $1 }' > CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha

4、修改文件权限
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*

/opt/cloudera/parcel-repo
├── CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
├── CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
├── manifest.json

设置Cloudera Manager 数据库(CM Server 节点)

执行命令:/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

启动Cloudera Manager Server服务

systemctl start cloudera-scm-server

#监控服务启动状态
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

访问Cloudera Manager WEB界面

http://sjhl-ad-hd1:7180/cmf/login

© 著作权归作者所有

上一篇: GreenPlum 使用
下一篇: kafka 性能调优
弘_轩
粉丝 5
博文 16
码字总数 10368
作品 0
福州
高级程序员
私信 提问
使用Apache Impala(CDH6)查询OSS的数据

CDH6 目前CDH的最新版本是6.0.1,支持Hadoop 3.0.0,本文将介绍如何使CDH6的相关组件(Hadoop/Hive/Spark/Impala等)能够读写OSS。CDH5对OSS的支持在这篇文章介绍。 CDH6支持读写OSS 搭建CDH集...

冷月_wjh
2018/10/30
0
0
CDH6在安装agent时,提示安装失败 无法接收 Agent 发出的检测信号

我在三台虚拟机上安装CDH6,进行到在web界面安装agent这一步,在“等待新安装的 Agent 检测信号...”这一步等待了1分钟左右,提示如下错误: 安装失败。 无法接收 Agent 发出的检测信号。 请...

随便填个用户名
2018/10/09
1K
1
Spark读写OSS并使用OSS-Select来加速查询

Spark读写OSS 基于这篇文章搭建的CDH6以及配置,我们来使Spark能够读写OSS(其他版本的Spark都是类似的做法,不再赘述)。 由于默认Spark并没有将OSS的支持包放到它的CLASSPATH里面,所以我们需...

冷月_wjh
2018/11/01
0
0
CDH6安装后,oozie的webui打开之后报错Error in Javac compilation for JSP? 求大神指点。

安装CDH6.0.0之后,服务都正常的(附截图)。只有oozie的webui打开有问题,报错如下: http://XXX.XXX.XX.XXX:11000/oozie/docs/ docs文档打开正常的 http://XXX.XXX.XX.XXX:11000/oozie 有问...

sxl_youcun
2018/10/23
470
3
0009-如何升级Cloudera Manager和CDH

1.文档编写目的 本文档讲述如何升级Cloudera Manager和CDH,通过本文档,您将学习到以下知识: 1.如何对Cloudera Manager进行停机升级 2.如何对CDH进行停机升级 3.如何在不影响集群作业的情况...

Hadoop实操
2018/11/15
37
0

没有更多内容

加载失败,请刷新页面

加载更多

可见性有序性,Happens-before来搞定

写在前面 上一篇文章并发 Bug 之源有三,请睁大眼睛看清它们 谈到了可见性/原子性/有序性三个问题,这些问题通常违背我们的直觉和思考模式,也就导致了很多并发 Bug 为了解决 CPU,内存,IO ...

tan日拱一兵
42分钟前
3
0
网络七层模型与TCP/UDP

为了使全球范围内不同的计算机厂家能够相互之间能够比较协调的进行通信,这个时候就有必要建立一种全球范围内的通用协议,以规范各个厂家之间的通信接口,这就是网络七层模型的由来。本文首先...

爱宝贝丶
45分钟前
7
0
Jenkins World 贡献者峰会及专家答疑展位

本文首发于:Jenkins 中文社区 原文链接 作者:Marky Jackson 译者:shunw Jenkins World 贡献者峰会及专家答疑展位 本文为 Jenkins World 贡献者峰会活动期间的记录 Jenkins 15周岁啦!Jen...

Jenkins中文社区
今天
10
0
杂谈:面向微服务的体系结构评审中需要问的三个问题

面向微服务的体系结构如今风靡全球。这是因为更快的部署节奏和更低的成本是面向微服务的体系结构的基本承诺。 然而,对于大多数试水的公司来说,开发活动更多的是将现有的单块应用程序转换为...

liululee
今天
8
0
OSChina 周二乱弹 —— 我等饭呢,你是不是来错食堂了?

Osc乱弹歌单(2019)请戳(这里) 【今日歌曲】 @ 自行车丢了:给主编推荐首歌 《クリスマスの夜》- 岡村孝子 手机党少年们想听歌,请使劲儿戳(这里) @烽火燎原 :国庆快来,我需要长假! ...

小小编辑
今天
1K
14

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部