Hadoop生态圈-HUE环境部署

2018/08/09 00:31
阅读数 76

                  Hadoop生态圈-HUE环境部署

                                      作者:尹正杰

版权声明:原创作品,谢绝转载!否则将追究法律责任。

 

 

 

 

 

一.HUE简介

1>.HUE的由来

  HUE全称是HadoopUser Experience,看这名字就知道怎么回事了吧。没错,直白来说就是Hadoop用户体验,是一个开源的Apache Hadoop UI系统,由Cloudera Desktop演化而来,最后Cloudera公司将其贡献给Apache基金会的Hadoop社区,它是基于Python Web框架Django实现的。通过使用HUE我们可以在浏览器端的Web控制台上与Hadoop集群进行交互来分析处理数据。

2>.官网地址

  HUE官网地址可参考:http://gethue.com/overview/。

 

二.HUE安装

1>.部署环境介绍(我使用的是CentOS 7.2的操作系统)

 

2>.安装开发JDK版本

  现在你生成环境用的Java开发版本,关于Java的下载地址可参考官网:http://www.oracle.com/technetwork/java/javase/downloads,下载后通过上传工具(如winscp等工具,或者你也可以使用linux自带的rz命令进行上传)

 

[yinzhengjie@s100 ~]$ sudo yum -y install lrzsz
[sudo] password for yinzhengjie: 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.huaweicloud.com
 * extras: mirrors.huaweicloud.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package lrzsz.x86_64 0:0.12.20-36.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================
 Package                Arch                    Version                         Repository             Size
============================================================================================================
Installing:
 lrzsz                  x86_64                  0.12.20-36.el7                  base                   78 k

Transaction Summary
============================================================================================================
Install  1 Package

Total download size: 78 k
Installed size: 181 k
Downloading packages:
lrzsz-0.12.20-36.el7.x86_64.rpm                                                      |  78 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : lrzsz-0.12.20-36.el7.x86_64                                                              1/1 
  Verifying  : lrzsz-0.12.20-36.el7.x86_64                                                              1/1 

Installed:
  lrzsz.x86_64 0:0.12.20-36.el7                                                                             

Complete!
[yinzhengjie@s100 ~]$
安装linux自带的上传工具命令([yinzhengjie@s100 ~]$ sudo yum -y install lrzsz)
[yinzhengjie@s100 ~]$ ll
total 0
[yinzhengjie@s100 ~]$ rz                                    #会弹出对话框,选择你要上传的文件即可

[yinzhengjie@s100 ~]$ ll
total 181296
-rw-r--r--. 1 yinzhengjie yinzhengjie 185646832 Aug 11 18:26 jdk-8u181-linux-x64.tar.gz
[yinzhengjie@s100 ~]$ 
[yinzhengjie@s100 ~]$ sudo mkdir /soft
[yinzhengjie@s100 ~]$ sudo chown yinzhengjie:yinzhengjie /soft/
[yinzhengjie@s100 ~]$ tar zxf jdk-8u181-linux-x64.tar.gz -C /soft/
[yinzhengjie@s100 ~]$ ln -s /soft/jdk1.8.0_181/ /soft/jdk
[yinzhengjie@s100 ~]$ ll /soft/
total 4
lrwxrwxrwx. 1 yinzhengjie yinzhengjie   19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/
drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul  7 04:09 jdk1.8.0_181
[yinzhengjie@s100 ~]$ 
[yinzhengjie@s100 ~]$ sudo vi /etc/profile
[yinzhengjie@s100 ~]$ more /etc/profile | tail -3                #添加java的环境变量
#ADD JAVA_PATH
JAVA_HOME=/soft/jdk
PATH=$PATH:$JAVA_HOME/bin
[yinzhengjie@s100 ~]$ 
[yinzhengjie@s100 ~]$ source /etc/profile                        #重新加载系统配置文件,使得配置文件在当前shell下生效
[yinzhengjie@s100 ~]$ 
[yinzhengjie@s100 ~]$ java -version                                #查看java版本
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
[yinzhengjie@s100 ~]$

3>.准备环境依赖

  查看官网安装文档(http://archive.cloudera.com/cdh5/cdh/5/hue-3.7.0-cdh5.3.0/manual.html),我们就可以看到相关的依赖环境如下:(CentOS具体依赖包参考:http://cloudera.github.io/hue/latest/admin-manual/manual.html#installation)

  Centos安装操作系统环境如下:

[yinzhengjie@s100 ~]$sudo yum -y install apache-maven ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi gcc gcc-c++ krb5-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel
[sudo] password for yinzhengjie: 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.huaweicloud.com
 * extras: mirrors.huaweicloud.com
 * updates: mirrors.aliyun.com
..........      接下来的信息我就省略了,安装过程以及成功后会有以下输出:(见下图)

   安装过程如下:

  安装成功如下:

4>.下载HUE安装包

[yinzhengjie@s100 ~]$ sudo yum -y install wget
[sudo] password for yinzhengjie: 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.huaweicloud.com
 * extras: mirrors.huaweicloud.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.14-15.el7_4.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================
 Package               Arch                    Version                          Repository             Size
============================================================================================================
Installing:
 wget                  x86_64                  1.14-15.el7_4.1                  base                  547 k

Transaction Summary
============================================================================================================
Install  1 Package

Total download size: 547 k
Installed size: 2.0 M
Downloading packages:
wget-1.14-15.el7_4.1.x86_64.rpm                                                      | 547 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : wget-1.14-15.el7_4.1.x86_64                                                              1/1 
  Verifying  : wget-1.14-15.el7_4.1.x86_64                                                              1/1 

Installed:
  wget.x86_64 0:1.14-15.el7_4.1                                                                             

Complete!
[yinzhengjie@s100 ~]$
安装下载命令wget([yinzhengjie@s100 ~]$ sudo yum -y install wget)

5>.删除安装后的openJDK 

  温馨提示:使用yum安装这些包的同时,也会自动安装openJDK的依赖,所以,请自行删除安装后的openJDK 

[yinzhengjie@s100 ~]$ rpm -qa | grep java
javapackages-tools-3.4.1-11.el7.noarch
java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64
python-javapackages-3.4.1-11.el7.noarch
tzdata-java-2018e-3.el7.noarch
java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$ 
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps python-javapackages-3.4.1-11.el7.noarch
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps javapackages-tools-3.4.1-11.el7.noarch
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-devel-1.8.0.181-3.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$ sudo rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64
[yinzhengjie@s100 ~]$ rpm -qa | grep java
tzdata-java-2018e-3.el7.noarch
[yinzhengjie@s100 ~]$

6>.解压HUE安装包

[yinzhengjie@s100 ~]$ ll
total 216276
-rw-r--r--. 1 yinzhengjie yinzhengjie  35818977 Sep 19  2016 hue-3.7.0-cdh5.3.6.tar.gz
-rw-r--r--. 1 yinzhengjie yinzhengjie 185646832 Aug 11 18:26 jdk-8u181-linux-x64.tar.gz
[yinzhengjie@s100 ~]$ 
[yinzhengjie@s100 ~]$ tar -zxf hue-3.7.0-cdh5.3.6.tar.gz -C /soft/
[yinzhengjie@s100 ~]$ ll /soft/
total 8
drwxr-xr-x. 9 yinzhengjie yinzhengjie 4096 Jul 28  2015 hue-3.7.0-cdh5.3.6
lrwxrwxrwx. 1 yinzhengjie yinzhengjie   19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/
drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul  7 04:09 jdk1.8.0_181
[yinzhengjie@s100 ~]$

7>.编译HUE

[yinzhengjie@s100 ~]$ ll /soft/
total 8
drwxr-xr-x. 9 yinzhengjie yinzhengjie 4096 Jul 28  2015 hue-3.7.0-cdh5.3.6
lrwxrwxrwx. 1 yinzhengjie yinzhengjie   19 Aug 12 03:33 jdk -> /soft/jdk1.8.0_181/
drwxr-xr-x. 7 yinzhengjie yinzhengjie 4096 Jul  7 04:09 jdk1.8.0_181
[yinzhengjie@s100 ~]$ cd /soft/hue-3.7.0-cdh5.3.6/
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ ll
total 64
drwxr-xr-x. 21 yinzhengjie yinzhengjie  4096 Jul 28  2015 apps
drwxr-xr-x.  3 yinzhengjie yinzhengjie  4096 Jul 28  2015 cloudera
drwxr-xr-x.  5 yinzhengjie yinzhengjie    54 Jul 28  2015 desktop
drwxr-xr-x.  6 yinzhengjie yinzhengjie  4096 Jul 28  2015 docs
drwxr-xr-x.  3 yinzhengjie yinzhengjie    23 Jul 28  2015 ext
-rw-r--r--.  1 yinzhengjie yinzhengjie 11358 Jul 28  2015 LICENSE.txt
-rw-r--r--.  1 yinzhengjie yinzhengjie  4715 Jul 28  2015 Makefile
-rw-r--r--.  1 yinzhengjie yinzhengjie  8505 Jul 28  2015 Makefile.sdk
-rw-r--r--.  1 yinzhengjie yinzhengjie  3498 Jul 28  2015 Makefile.vars
-rw-r--r--.  1 yinzhengjie yinzhengjie  2192 Jul 28  2015 Makefile.vars.priv
drwxr-xr-x.  2 yinzhengjie yinzhengjie    20 Jul 28  2015 maven
-rw-r--r--.  1 yinzhengjie yinzhengjie   801 Jul 28  2015 NOTICE.txt
-rw-r--r--.  1 yinzhengjie yinzhengjie  1517 Jul 28  2015 README
drwxr-xr-x.  4 yinzhengjie yinzhengjie    65 Jul 28  2015 tools
-rw-r--r--.  1 yinzhengjie yinzhengjie   932 Jul 28  2015 VERSION
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ make apps            #开始编译HUE
--- Creating virtual environment at /soft/hue-3.7.0-cdh5.3.6/build/env
python2.7 /soft/hue-3.7.0-cdh5.3.6/tools/virtual-bootstrap/virtual-bootstrap.py \
    -qq --no-site-packages /soft/hue-3.7.0-cdh5.3.6/build/env
--- /soft/hue-3.7.0-cdh5.3.6/build/env ready
make[1]: Entering directory `/soft/hue-3.7.0-cdh5.3.6/desktop'
make -C core env-install
make[2]: Entering directory `/soft/hue-3.7.0-cdh5.3.6/desktop/core'
--- Building egg for markdown
running bdist_egg
running egg_info
.........   #编译时会出现下图的报错信息

  解决方案:

解决方案:
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ sudo vi /usr/include/openssl/x509.h 
[sudo] password for yinzhengjie:             #删除以下两行:
    751 X509_REVOKED *X509_REVOKED_dup(X509_REVOKED *rev);
  752 X509_REQ *X509_REQ_dup(X509_REQ *req);
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$

  使用root权限删除图中的751和752行代码,然后在重新编译即可。编译成功截图如下:

8>.配置HUE

[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ vi desktop/conf/hue.ini 
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep secret_key= desktop/conf/hue.ini 
  secret_key=jFE93j;2[290-eiw.KEiwN2s3['d;/.q[eIW^y#e=+Iei*@Mn<qW5o
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ 
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep http_host= desktop/conf/hue.ini 
  http_host=172.30.1.100
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep http_port= desktop/conf/hue.ini 
  http_port=8888
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$ grep time_zone= desktop/conf/hue.ini 
  time_zone=Asia/Shanghai
[yinzhengjie@s100 hue-3.7.0-cdh5.3.6]$

 

  具体配置如下:

9>.启动HUE

10>.关闭服务器防火墙以及selinux服务。

[yinzhengjie@s100 ~]$ systemctl stop firewalld                    #临时关闭防火墙
[yinzhengjie@s100 ~]$ systemctl disable firewalld                #禁用防火墙的服务开机自启
[yinzhengjie@s100 ~]$ 
[yinzhengjie@s100 ~]$ su root
Password: 
[root@s100 yinzhengjie]# cat /etc/selinux/config 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 


[root@s100 yinzhengjie]#
[root@s100 yinzhengjie]# getenforce 
Enforcing
[root@s100 yinzhengjie]#
[root@s100 yinzhengjie]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config 
[root@s100 yinzhengjie]#
[root@s100 yinzhengjie]# cat /etc/selinux/config  | grep SELINUX= | grep -v ^#
SELINUX=disabled
[root@s100 yinzhengjie]# 

11>.访问HUE的webUI并设置管理员账号

   Duang~如果你出现以下界面说明你HUE启动成功啦!

 

 三.通过git方式安装HUE

  经查阅资料,发现github上也有教程教你如何安装HUE,需要你本地服务器安装git命令,详情请参考:https://github.com/cloudera/hue#development-prerequisites

[yinzhengjie@s100 ~]$ sudo yum -y install git
[sudo] password for yinzhengjie: 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.huaweicloud.com
 * extras: mirrors.huaweicloud.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package git.x86_64 0:1.8.3.1-14.el7_5 will be installed
--> Processing Dependency: perl-Git = 1.8.3.1-14.el7_5 for package: git-1.8.3.1-14.el7_5.x86_64
--> Processing Dependency: rsync for package: git-1.8.3.1-14.el7_5.x86_64
--> Processing Dependency: perl(Term::ReadKey) for package: git-1.8.3.1-14.el7_5.x86_64
--> Processing Dependency: perl(Git) for package: git-1.8.3.1-14.el7_5.x86_64
--> Processing Dependency: perl(Error) for package: git-1.8.3.1-14.el7_5.x86_64
--> Processing Dependency: libgnome-keyring.so.0()(64bit) for package: git-1.8.3.1-14.el7_5.x86_64
--> Running transaction check
---> Package libgnome-keyring.x86_64 0:3.12.0-1.el7 will be installed
---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed
---> Package perl-Git.noarch 0:1.8.3.1-14.el7_5 will be installed
---> Package perl-TermReadKey.x86_64 0:2.30-20.el7 will be installed
---> Package rsync.x86_64 0:3.1.2-4.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================
 Package                                  Arch                           Version                                    Repository                       Size
==========================================================================================================================================================
Installing:
 git                                      x86_64                         1.8.3.1-14.el7_5                           updates                         4.4 M
Installing for dependencies:
 libgnome-keyring                         x86_64                         3.12.0-1.el7                               base                            109 k
 perl-Error                               noarch                         1:0.17020-2.el7                            base                             32 k
 perl-Git                                 noarch                         1.8.3.1-14.el7_5                           updates                          54 k
 perl-TermReadKey                         x86_64                         2.30-20.el7                                base                             31 k
 rsync                                    x86_64                         3.1.2-4.el7                                base                            403 k

Transaction Summary
==========================================================================================================================================================
Install  1 Package (+5 Dependent packages)

Total download size: 5.0 M
Installed size: 23 M
Downloading packages:
(1/6): perl-TermReadKey-2.30-20.el7.x86_64.rpm                                                                                     |  31 kB  00:00:00     
(2/6): libgnome-keyring-3.12.0-1.el7.x86_64.rpm                                                                                    | 109 kB  00:00:00     
(3/6): perl-Error-0.17020-2.el7.noarch.rpm                                                                                         |  32 kB  00:00:00     
(4/6): perl-Git-1.8.3.1-14.el7_5.noarch.rpm                                                                                        |  54 kB  00:00:00     
(5/6): rsync-3.1.2-4.el7.x86_64.rpm                                                                                                | 403 kB  00:00:00     
(6/6): git-1.8.3.1-14.el7_5.x86_64.rpm                                                                                             | 4.4 MB  00:00:02     
----------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                     1.7 MB/s | 5.0 MB  00:00:02     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 12 pre-existing rpmdb problem(s), 'yum check' output follows:
ant-1.9.2-9.el7.noarch has missing requires of java-devel >= ('0', '1.5.0', None)
ant-1.9.2-9.el7.noarch has missing requires of jpackage-utils
ant-1.9.2-9.el7.noarch has missing requires of jpackage-utils >= ('0', '1.7.5', None)
xalan-j2-2.7.1-23.el7.noarch has missing requires of jpackage-utils
xerces-j2-2.11.0-17.el7_0.noarch has missing requires of java
xerces-j2-2.11.0-17.el7_0.noarch has missing requires of jpackage-utils
xerces-j2-2.11.0-17.el7_0.noarch has missing requires of jpackage-utils
xml-commons-apis-1.4.01-16.el7.noarch has missing requires of java
xml-commons-apis-1.4.01-16.el7.noarch has missing requires of jpackage-utils
xml-commons-apis-1.4.01-16.el7.noarch has missing requires of jpackage-utils
xml-commons-resolver-1.2-15.el7.noarch has missing requires of jpackage-utils
xml-commons-resolver-1.2-15.el7.noarch has missing requires of jpackage-utils
  Installing : 1:perl-Error-0.17020-2.el7.noarch                                                                                                      1/6 
  Installing : rsync-3.1.2-4.el7.x86_64                                                                                                               2/6 
  Installing : perl-TermReadKey-2.30-20.el7.x86_64                                                                                                    3/6 
  Installing : libgnome-keyring-3.12.0-1.el7.x86_64                                                                                                   4/6 
  Installing : perl-Git-1.8.3.1-14.el7_5.noarch                                                                                                       5/6 
  Installing : git-1.8.3.1-14.el7_5.x86_64                                                                                                            6/6 
  Verifying  : git-1.8.3.1-14.el7_5.x86_64                                                                                                            1/6 
  Verifying  : libgnome-keyring-3.12.0-1.el7.x86_64                                                                                                   2/6 
  Verifying  : perl-TermReadKey-2.30-20.el7.x86_64                                                                                                    3/6 
  Verifying  : 1:perl-Error-0.17020-2.el7.noarch                                                                                                      4/6 
  Verifying  : rsync-3.1.2-4.el7.x86_64                                                                                                               5/6 
  Verifying  : perl-Git-1.8.3.1-14.el7_5.noarch                                                                                                       6/6 

Installed:
  git.x86_64 0:1.8.3.1-14.el7_5                                                                                                                           

Dependency Installed:
  libgnome-keyring.x86_64 0:3.12.0-1.el7  perl-Error.noarch 1:0.17020-2.el7  perl-Git.noarch 0:1.8.3.1-14.el7_5  perl-TermReadKey.x86_64 0:2.30-20.el7 
  rsync.x86_64 0:3.1.2-4.el7             

Complete!
[yinzhengjie@s100 ~]$
安装git([yinzhengjie@s100 ~]$ sudo yum -y install git)

   安装完git后按照GitHub的介绍,安装相应的软件。

 

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部