文档章节

HDP集群搭建

 勺猫子
发布于 2018/08/25 22:50
字数 1933
阅读 131
收藏 0

下载包:

http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.0.0/ambari-2.6.0.0-centos7.tar.gz
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0/HDP-2.6.5.0-centos7-rpm.tar.gz
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz

//如果出现以下结果说明启动了THP

[root@node-41 local]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never

[root@node-41 local]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

//永久关闭THP

[root@node-41 local]# vi /etc/rc.d/rc.local 

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
        echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

修改文件测执行权限:

[root@node-41 ~]# chmod +x /etc/rc.d/rc.local

//检查是否生效

reboot

重启后查看

[root@node-41 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@node-41 ~]# cat /sys/kernel/mm/transparent_hugepage/defrag 
always madvise [never]

说明关闭成功

二、克隆机器及其配置

在VMware中克隆5台及其:node-42、node-43、node-44、node-45、node-46

点击每台机器中的“网络适配器”->“高级”->"生成"(mac地址)->“确定”->"确定"

开机,修改IP地址: vi /etc/sysconfig/network-scripts/ifcfg-ens33 

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=62acf16a-30fd-4909-a68b-92c63408b5b2
DEVICE=ens33
ONBOOT=yes
IPADDR=192.168.194.46
GATEWAY=192.168.194.2
NETMASK=255.255.255.0
DNS1=8.8.8.8

修改hostname:

[root@localhost ~]# hostname
localhost.localdomain
[root@localhost ~]# 
[root@localhost ~]# hostnamectl set-hostname node-46
[root@localhost ~]# hostname
node-46

关闭防火墙:

//临时关闭

[root@node-42 ~]# systemctl stop firewalld

//禁止开机启动
[root@node-42 ~]# systemctl disable firewalld

//查看防火墙状态
[root@node-42 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

安装Mysql,具体可见 https://my.oschina.net/u/3491962/blog/1934088

修改mysql配置文件:[root@node-41 local]# vi /etc/my.cnf 

[mysqld] 下添加:

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
collation_server=utf8_general_ci
character_set_server=utf8
default-storage-engine=INNODB

[client]
default-character-set=utf8

重启MySQL服务:
[root@node-41 local]# systemctl restart mysqld

连接Mysql查看编码是否替换:
mysql> show variables like '%character_set_%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8                       |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8                       |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.01 sec)

创建ambri相应的DB以及其用户名:
mysql> create database ambari character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> create user 'ambari'@'%' IDENTIFIED BY 'Ambari123'; 
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

创建HiveDB 以及用户:

mysql> create database hive character set utf8;        
Query OK, 1 row affected (0.00 sec)

mysql> create user 'hive'@'%' IDENTIFIED BY 'Hive123';             
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';            
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

安装Mysql-java

[root@node-41 local]# yum install mysql-connector-java

配置免密登陆:

在node-41 中:

[root@node-41 local]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Ri8XZHtAAQUTWADi5zcW2qRUsDEqJKto/V0MXJH2ULk root@node-41
The key's randomart image is:
+---[RSA 2048]----+
|... =oo+*O@o.    |
|oo o =o .B +     |
|o o + ooo = o    |
|o..+ = oo. E     |
|o. .+ = Soo      |
|.   .o.o.o       |
|     . .         |
|                 |
|                 |
+----[SHA256]-----+

将公钥拷贝至node-41、node-42、node-43、node-44、node-45

ssh-copy-id node-41、ssh-copy-id node-42、ssh-copy-id node-43、ssh-copy-id node-44、ssh-copy-id node-45

在node-42 中:执行同样的操作(生成公钥、私钥,然后将公钥拷贝至各个机器上)

二、配置本地的yum源

在本地yum源上安装相关yum工具

[root@yum ~]# yum install yum-utils -y

[root@yum ~]# yum repolist

[root@yum ~]# yum install createrepo -y

安装Apache httpd

[root@yum ~]# yum install httpd -y

安装完成后会自动生成目录:[root@yum ~]# /var/www/html/,相当于Tomcat 的 /webapps目录,进入该目录,创建ambari和hdp文件夹 

[root@yum html]# cd /var/www/html/

[root@yum html]# mkdir ambari
[root@yum html]# mkdir hdp
[root@yum html]# mkdir hdp/HDP-UTILS-1.1.0.21

解压相应的文件:

[root@yum html]# tar -zxvf /root/ambari-2.6.0.0-centos7.tar.gz -C /var/www/html/ambari/

[root@yum html]# tar -zxvf /root/HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/hdp/HDP-UTILS-1.1.0.21/

[root@yum html]# tar -zxvf /root/HDP-2.6.5.0-centos7-rpm.tar.gz -C /var/www/html/hdp/

解压完成后启动 httpd 服务:

[root@yum html]# systemctl start httpd

启动后可以访问链接:http://192.168.194.40/

设置开机自启动:

[root@yum html]# systemctl enable httpd 

配置ambari yum 源:

[root@yum html]# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari.repo -O /etc/yum.repos.d/ambari.repo

[root@yum yum.repos.d]# vi ambari.repo 

#VERSION_NUMBER=2.6.0.0-267
[ambari-2.6.0.0]
name=ambari Version - ambari-2.6.0.0
baseurl=http://192.168.194.40/ambari/ambari/centos7/2.6.0.0-267/
gpgcheck=1
gpgkey=http://192.168.194.40/ambari/ambari/centos7/2.6.0.0-267/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[root@yum yum.repos.d]# vi HDP.repo 

#VERSION_NUMBER=2.6.5.0-1
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0
baseurl=http://192.168.194.40/hdp/HDP/centos7/
gpgcheck=1
gpgkey=http://192.168.194.40/hdp/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.21]
name=HDP Version - HDP-UTILS-1.1.0.21
baseurl=http://192.168.194.40/hdp/HDP-UTILS-1.1.0.21/
gpgcheck=1
gpgkey=http://192.168.194.40/hdp/HDP-UTILS-1.1.0.21/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

分发HDP.repo ambari.repo 到各个节点

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-41:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-42:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-43:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-44:$PWD

[root@yum yum.repos.d]# scp ambari.repo HDP.repo node-45:$PWD

生成本地yum源

使用createrepo命令,生成本地源

【createrepo用以创建yum源(软件仓库),即为存放于本机特定位置的众多rpm包建立索引描述各包所需以来信息,并形成源数据】

[root@yum yum.repos.d]# createrepo /var/www/html/hdp/HDP/centos7/

[root@yum yum.repos.d]# createrepo /var/www/html/hdp/HDP-UTILS-1.1.0.21/

 

三、安装 Ambari Server

1、安装服务

[root@node-41 yum.repos.d]# yum install ambari-server

2、设置服务

[root@node-41 yum.repos.d]# ambari-server setup

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk1.8.0_161/
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? Y^H
input not recognized, please try again: 
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): node-41
Port (3306): 
Database name (ambari): 
Username (ambari): 
Enter Database Password (bigdata): 
Re-enter password: 
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.6.0.0.267.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root

[root@node-41 yum.repos.d]# ambari-server setup

四、设置ambari用户远程登陆权限

[root@node-41 yum.repos.d]# mysql -uroot -p123456

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'localhost' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.17 sec)

mysql> GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%' IDENTIFIED BY 'Ambari123';
Query OK, 0 rows affected, 1 warning (0.01 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.09 sec)

mysql> exit
Bye

使用ambari账号登陆MySQL
[root@node-41 yum.repos.d]# mysql -uambari -pAmbari123 -hnode-41

mysql> use ambari;
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

启动Ambari

[root@node-41 yum.repos.d]# ambari-server start

[root@node-41 yum.repos.d]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.......................................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

访问Ambari服务:

http://192.168.194.41:8080

安装Agent

在所有节点上安装

yum install ambari-agent

安装成功后可进入Ambari Web端,初始化设置

默认账号密码:

admin

admin

设置集群,点击Launch Install Wizard :

输入集群名字:

© 著作权归作者所有

下一篇: 机器学习算法
粉丝 1
博文 8
码字总数 6885
作品 0
珠海
私信 提问
HDP2.6 Hadoop如何支持读写OSS

HDP和Ambari HDP(Hortonworks Data Platform)是由Hortonworks发行的大数据平台,里面包含了Hadoop、Hive、HBase等很多开源组件,目前有不少用户直接使用HDP版本的Hadoop。Ambari是一个分布式...

冷月_wjh
2018/10/29
0
0
Setting Up Hadoop NameNode High Availability

Hadoop2.0以后,2个NameNode的数据其实是实时共享的。新HDFS采用了一种共享机制,Quorum Journal Node(JournalNode)集群或者Nnetwork File System(NFS)进行共享。NFS是操作系统层面的,J...

candon123
2018/01/03
0
0
Setting up Oozie high availability

本文介绍如何搭建四节点的Oracle Linux 7集群以及oozie的High Availability。环境如下图:一、创建HA集群1、安装集群软件包分别在4个节点上安装,如下: 2、设置集群用户密码集群软件包安装完...

candon123
2017/12/15
0
0
Configuring Zookeeper Cluster

本文介绍的 Zookeeper 是以 3.4.10 这个稳定版本为基础,最新的版本可以通过官网下载。Zookeeper 不仅可以单机提供服务,同时也支持多机组成集群来提供服务。实际上 Zookeeper 还支持另外一种...

candon123
2017/12/13
0
0
Hadoop深度运维:Apache集群原地升级Ambari-HDP

作者介绍 冯武,目前在瓜子从事大数据集群运维管理,主要负责Hadoop、Ansible、Ambari、Cloudera Manager等技术栈的维护调优工作。 公司从建设Hadoop起,采用了Apache社区版本的Hadoop,随着...

DBAplus社群
03/14
0
0

没有更多内容

加载失败,请刷新页面

加载更多

CSS定位

CSS定位 relative相对定位 absolute绝对定位 fixed和sticky及zIndex relative相对定位 position特性:css position属性用于指定一个元素在文档中的定位方式。top、right、bottom、left属性则...

studywin
28分钟前
4
0
从零基础到拿到网易Java实习offer,我做对了哪些事

作为一个非科班小白,我在读研期间基本是自学Java,从一开始几乎零基础,只有一点点数据结构和Java方面的基础,到最终获得网易游戏的Java实习offer,我大概用了半年左右的时间。本文将会讲到...

Java技术江湖
昨天
5
0
程序性能checklist

程序性能checklist

Moks角木
昨天
7
0
VUE 计算属性

本文转载于:专业的前端网站▶VUE 计算属性 1、示例代码 <!DOCTYPE html><html lang="zh"> <head> <meta charset="UTF-8" /> <title>vue示例</title> </hea......

前端老手
昨天
7
0
快速搭建LNMT平台和环境部署 Tomcat详解

Tomcat部署的基本概念 1. CATALINA_HOME与CATALINA_BASE分别指什么?     CATALINA_HOME指的是Tomcat的安装目录     bin:\\Tomcat一些脚本存放目录,比如启动脚本startup.bat/start...

网络小虾米
昨天
9
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部