文档章节

一步一步安装hive

hnairdb
 hnairdb
发布于 10/23 15:07
字数 3376
阅读 20
收藏 0

The Apache Hive ™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive.

--apache Hive可以使用SQL在分布式系统来读取,写入和管理大数据集合。

hive是基于hadoop 集群系统之上的对数据管理的一个框架,所以在搭建hive之前 需要先搭建hadoop和mysql。

mysql主要是用于存储hadoop元数据信息,数据块存储在hadoop中。其实hive只是接受用户输入的sql语句,然后把sql语句映射成MR JOB,由Yarn 完成对任务的调度执行。从流程上讲,hive这种设计在效率上并不高,但是有一个好处是hadoop开发不需要写java代码,就像操作关系型数据库一样操作hadoop,hive提供一种类SQL的语言使用。

为了简单方便,这里采用虚拟机搭建hadoop集群系统。

linux系统安装没有什么好说的,略过。

节点规划:

  NN SNN DN MR YARN
hd1 Y     Y Y
hd2   Y Y    
hd3     Y    
hd4     Y    

这里介绍几个概念:

namenode(NN):

主要功能:

  • 接受客户端读写服务;
  • 保存元数据信息(文件名,文件属主和权限;文件包含块信息;block存放那个datanode由datanode启动时上报给namenode) namenode的元数据在集群启动时会全部加载到内存。

元数据还会存储到磁盘的文件名fsimage,block的位置不会保存到fsimage,为了提高速度 block位置信息不会保存到fsimage,block位置会一直在内存中加载读取。 如果修改一条数据,hdfs并不会马上修改fsimage,而是先记录到metadata操作日志,等到满足条件会写到fsimage中。我们可以这么理解,fsimage是对元数据的定期全备,edits是对元数据的实时记录,是对源数据库的增量备份,全备加上增量备份就是元数据的完整备份。

SecondarynameNode(SNN):

它不是nn的备份,只是nn的一部分数据备份。当然也可以作为备份,它的主要功能是协助nn合并editslog,减少nn启动时间。fsimage在跟edits合并的时候会删除,这个时候会有大量的IO操作,这个时候snn会协助nn完整这个动作。合并完成之后在snn上会生成一个新的fsimage,然后推送给nn. 以上动作会周而复始的进行。

yarn(Yet Another Resource Negotiator):资源管理利器

yarn的引入,使得多个计算框架可运行在一个集群中。每个应用程序对应一个applicationMaster。目前多个计算框架可以运行在yarn上。

基本功能:

  • yarn 负责资源管理和调度;
  • mrappmaster 负责任务切分,调度,监控和容错;
  • maptask/reduceTask 任务驱动;

每一个mapreduce作业对应一个MRAppMaster:

  • MRAppMaster负责作业调度,yarn将资源分配给MRAppMaster,MRAppMaster将资源分配给内部任务。

MRAppMaster容错 :

  • 失败后由yarn重新启动;
  • 任务失败后,MRAppMaster重新申请资源

DataNode数据节点(DN):

DN在启动的时候会向NN汇报block信息; 通过主动向nn发送心跳保持一致(3s),如果NN 10分钟没有收到DN心跳信息,则认为其已经丢失,并拷贝其block到其他DN.

为了方便这里采用虚拟机拷贝方式,这里注意一点就是网卡信息问题,可以参考 https://my.oschina.net/u/3862440/blog/2250996。

hadoop 伪分布安装步骤

可参考http://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-common/ClusterSetup.html

 

修改主机名:

[root@hd1 ~]# vi /etc/sysconfig/network

NETWORKING=yes
HOSTNAME=hd2.localdomain

用户名,用户组规划:

groupadd -g 10010 hadoop 

useradd -u 10010 -g hadoop -d /home/hadoop hadoop

groupadd -g 10012 mysql 

useradd -u 10012 -g mysql -d /home/mysql mysql

安装jdk

tar -xvf jdk-8u11-linux-x64.tar -C /usr/ 

配置环境变量:

JAVA_HOME=/usr/java/jdk1.8.0_11
export JAVA_HOME
export JRE_HOME=/usr/java/jdk1.8.0_11
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

配置hosts本地解析文件:

[root@hd1 ~]# vi /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.83.11  hd1
192.168.83.12  hd2
192.168.83.13  hd3
192.168.83.14  hd4

配置互信:

ssh-keygen -t rsa 
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

让hd1拥有hd2,hd3,hd4节点的公钥,其他几个节点也需要拥有hd1的公钥;这样hd1连接其他几个节点就不需要密码了,其他几个节点连接hd1也不需要密码。

软件准备:

hadoop-2.6.0-cdh5.7.0

hive-1.1.0-cdh5.7.0.tar

mysql-5.7.20-linux-glibc2.12-x86_64.tar

hadoop配置:

  • 配置 JAVA_HOME

    vi etc/hadoop/hadoop-env.sh

 export JAVA_HOME=/usr/java/jdk1.8.0_11
  • vi etc/hadoop/core-site.xml,修改Hadoop核心配置文件core-site.xml,这里配置的是HDFS master(即namenode)的地址和端口号。
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hd1:9000</value>
         </property>
         <property>
                 <name>hadoop.tmp.dir</name>
                 <value>file:/usr/hadoop/hadoop-2.7.1/tmp</value>
                 <description>Abase for other temporary directories.</description>
         </property>
</configuration>
  • etc/hadoop/hdfs-site.xml,修改Hadoop中HDFS的配置,配置的副本方式默认为3和SNN地址和端口号。
<configuration>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>hd2:50090</value>
        </property>
        <property>
                 <name>dfs.replication</name>
                 <value>3</value>
         </property>
         <property>
                 <name>dfs.namenode.name.dir</name>
                 <value>file:/usr/hadoop/hadoop-2.7.1/tmp/dfs/name</value>
         </property>
         <property>
                 <name>dfs.datanode.data.dir</name>
                 <value>file:/usr/hadoop/hadoop-2.7.1/tmp/dfs/data</value>
          </property>
</configuration>
  • MR配置:Configure parameters as follows:etc/hadoop/mapred-site.xml,修改Hadoop中MapReduce的配置文件,配置的是JobTracker的地址和端口。
<configuration>
        <property>
               <name>mapreduce.framework.name</name>
               <value>yarn</value>
        </property>
        <property>
               <name>mapreduce.jobhistory.address</name>
               <value>hd1:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>hd1:19888</value>
        </property>
</configuration>
  • yarn配置:etc/hadoop/yarn-site.xml:
<configuration>
        <property>
             <name>yarn.resourcemanager.hostname</name>
             <value>hd1</value>
        </property>
        <property>
             <name>yarn.nodemanager.aux-services</name>
             <value>mapreduce_shuffle</value>
       </property>
</configuration>
  • DN配置:
[hadoop@hd1 hadoop]$ vi slaves 

hd2
hd3
hd4

把hd1 下面的hadoop-2.7.1 文件拷贝到hd2,hd3,hd4节点,

scp -r /home/hadoop/hadoop-2.7.1 hd2:/home/hadoop/
scp -r /home/hadoop/hadoop-2.7.1 hd3:/home/hadoop/
scp -r /home/hadoop/hadoop-2.7.1 hd4:/home/hadoop/

配置hadoop环境变量,为了方便后面的操作:

export HADOOP_INSTALL=/home/hadoop/hadoop-2.7.1
export PATH=$PATH:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin

分别拷贝到hd2,hd3,hd4节点:

scp ~/.bash_profile  hd2:~/
scp ~/.bash_profile  hd3:~/
scp ~/.bash_profile  hd4:~/

使hadoop用户环境变量配置文件生效 source ~/.bash_profile

启动:

我们知道hdfs是一个分布式文件系统,是自己的一套文件系统,所以在启动的时候,需要格式化。hadoop并不能识别linux上的ext3,ext4文件系统。格式化:

hadoop namenode -format

java.net.UnknownHostException: hd1.localdomain: hd1.localdomain: unknown error
        at java.net.InetAddress.getLocalHost(InetAddress.java:1484)
        at org.apache.hadoop.net.DNS.resolveLocalHostname(DNS.java:264)
        at org.apache.hadoop.net.DNS.<clinit>(DNS.java:57)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:966)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:575)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:157)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:991)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1429)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
Caused by: java.net.UnknownHostException: hd1.localdomain: unknown error
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
        at java.net.InetAddress.getLocalHost(InetAddress.java:1479)
        ... 8 more
18/10/23 22:49:36 WARN net.DNS: Unable to determine address of the host-falling back to "localhost" address
java.net.UnknownHostException: hd1.localdomain: hd1.localdomain: unknown error
        at java.net.InetAddress.getLocalHost(InetAddress.java:1484)
        at org.apache.hadoop.net.DNS.resolveLocalHostIPAddress(DNS.java:287)
        at org.apache.hadoop.net.DNS.<clinit>(DNS.java:58)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.newBlockPoolID(NNStorage.java:966)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.newNamespaceInfo(NNStorage.java:575)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:157)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:991)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1429)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
Caused by: java.net.UnknownHostException: hd1.localdomain: unknown error
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:907)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1302)
        at java.net.InetAddress.getLocalHost(InetAddress.java:1479)
        ... 8 more
18/10/23 22:49:36 INFO namenode.FSImage: Allocated new BlockPoolId: BP-520690254-127.0.0.1-1540306176095
18/10/23 22:49:36 INFO common.Storage: Storage directory /usr/hadoop/hadoop-2.7.1/tmp/dfs/name has been successfully formatted.
18/10/23 22:49:36 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/10/23 22:49:36 INFO util.ExitUtil: Exiting with status 0
18/10/23 22:49:36 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: hd1.localdomain: hd1.localdomain: unknown error
************************************************************

可以看到不能识别hd1.localdomain,所以这里把主机名修改为hd1,不带DNS后缀。

所有的节点都需要修改。

vi /etc/sysconfig/network 

NETWORKING=yes
HOSTNAME=hd1

重新格式化:

[hadoop@hd1 ~]$ hadoop namenode -format 
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
18/10/23 22:54:03 INFO util.GSet: capacity      = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /usr/hadoop/hadoop-2.7.1/tmp/dfs/name ? (Y or N) Y
18/10/23 22:54:06 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1243395970-192.168.83.11-1540306446216
18/10/23 22:54:06 INFO common.Storage: Storage directory /usr/hadoop/hadoop-2.7.1/tmp/dfs/name has been successfully formatted.
18/10/23 22:54:06 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/10/23 22:54:06 INFO util.ExitUtil: Exiting with status 0
18/10/23 22:54:06 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hd1/192.168.83.11
************************************************************/

成功。

启动dfs:

[hadoop@hd1 ~]$ start-dfs.sh

[hadoop@hd1 hadoop]$ start-dfs.sh 
18/10/23 23:01:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hd1]
hd1: starting namenode, logging to /home/hadoop/hadoop-2.7.1/logs/hadoop-hadoop-namenode-hd1.out
hd4: starting datanode, logging to /home/hadoop/hadoop-2.7.1/logs/hadoop-hadoop-datanode-hd4.out
hd3: starting datanode, logging to /home/hadoop/hadoop-2.7.1/logs/hadoop-hadoop-datanode-hd3.out
hd2: starting datanode, logging to /home/hadoop/hadoop-2.7.1/logs/hadoop-hadoop-datanode-hd2.out
Starting secondary namenodes [hd2]
hd2: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.1/logs/hadoop-hadoop-secondarynamenode-hd2.out
18/10/23 23:01:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

以上可以看到,NN启动在第一个节点,SNN启动在第二个节点,DN启动在hd2,hd3,hd4三个节点,跟之前规划一致。

启动yarn:

[hadoop@hd1 ~]$ start-yarn.sh

[hadoop@hd1 hadoop]$ start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-resourcemanager-hd1.out
hd2: starting nodemanager, logging to /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-nodemanager-hd2.out
hd4: starting nodemanager, logging to /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-nodemanager-hd4.out
hd3: starting nodemanager, logging to /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-nodemanager-hd3.out

可以看到,resuurceManager启动在hd1节点上,NodeManager启动在hd2,hd3,hd4节点上。

yarn是一个资源管理,任务调度的框架,所以他分配资源,任何的hdfs的操作都需要想yarn申请资源,所以yarn需要在NN上有一个ResourceManager进程,接受用户任务的请求;由于yarn是资源管理所以他需要明白所有的节点的资源状态,所以DN节点需要有NodeManager进程。

[hadoop@hd1 ~]$ mr-jobhistory-daemon.sh start historyserver

[hadoop@hd1 hadoop]$ mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /home/hadoop/hadoop-2.7.1/logs/mapred-hadoop-historyserver-hd1.out

MR是一个计算框架,他是部署在yarn 之上的,所以这里启动yarn 之后,MR就自动启动了。

jps查看节点进程状态:

[hadoop@hd1 hadoop]$ jps
3681 NameNode
4259 JobHistoryServer
3957 ResourceManager
4362 Jps

[hadoop@hd2 ~]$ jps
2643 DataNode
2973 Jps
2735 SecondaryNameNode
2815 NodeManager

[hadoop@hd3 ~]$ jps
2216 DataNode
2472 Jps
2317 NodeManager

[hadoop@hd4 ~]$ jps
2368 NodeManager
2504 Jps
2265 DataNode

hdfs 测试:

[hadoop@hd1 ~]$ hdfs dfs -mkdir /hadoop/
18/10/23 23:26:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hd1 ~]$ hdfs dfs -ls /
18/10/23 23:26:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2018-10-23 23:26 /hadoop
drwxrwx---   - hadoop supergroup          0 2018-10-23 23:09 /tmp

接下来部署hive:

部署mysql:

  • 解压
  • #### 初始化(相当于dbca建库)
[mysql@hd1 bin]$ ./mysqld --initialize-insecure --basedir=/home/mysql/mysql-5.7.20 --datadir=/home/mysql/mysql-5.7.20/data --user=mysql
2018-10-23T15:33:43.531824Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
2018-10-23T15:33:43.532051Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
2018-10-23T15:33:43.532611Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-10-23T15:33:46.626624Z 0 [Warning] InnoDB: New log files created, LSN=45790
2018-10-23T15:33:47.139730Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2018-10-23T15:33:47.498281Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 08944547-d6d9-11e8-b3f4-000c297eaaf3.
2018-10-23T15:33:47.550419Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2018-10-23T15:33:47.564974Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.

编写my.cnf配置文件

[mysqld]
basedir=/home/mysql/mysql-5.7.20
datadir=/home/mysql/mysql-5.7.20/data
socket=/tmp/mysql.sock
log_error=/home/mysql/mysql-5.7.20/mysql.err
user=mysql

[mysql]
socket=/tmp/mysql.sock

启动MySql:

[mysql@hd1 support-files]$ ./mysql.server start 
Starting MySQL..[  OK  ]

为了方便启动,可以把MySql的启动脚本拷贝到 /etc/init.d/下面,让随着OS一起启动。

[root@hd1 support-files]# cp mysql.server /etc/init.d/mysqld
[root@hd1 support-files]# /etc/init.d/mysqld restart
Shutting down MySQL..[  OK  ]
Starting MySQL.[  OK  ]

测试:

[mysql@hd1 ~]$ mysql 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

设置mysql密码:

[mysql@hd1 bin]$ mysqladmin -u root passwor 'Oracle123'  -S '/tmp/mysql.sock';
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.

密码登录:

[mysql@hd1 bin]$ mysql -uroot -p -S '/tmp/mysql.sock' 
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.7.20 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show database;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'database' at line 1
mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

到这里MySql安装完成。

部署hive:

解压

配置:[hadoop@hd1 conf]$ more hive-site.xml ,

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>        
 <property>                
   <name>hive.metastore.local</name>                
   <value>true</value>        
 </property>        

 <property>                
  <name>javax.jdo.option.ConnectionURL</name>                
  <value>jdbc:mysql://192.168.83.11:3306/hive?characterEncoding=UTF-8</value>        
 </property> 

 <property>                
   <name>javax.jdo.option.ConnectionDriverName</name>                
   <value>com.mysql.jdbc.Driver</value>        
 </property>      

 <property>                
   <name>javax.jdo.option.ConnectionUserName</name>                
   <value>root</value>        
 </property>      

 <property>                
    <name>javax.jdo.option.ConnectionPassword</name>                
    <value>Oracle123</value><!-- 这里是数据库密码 -->         
 </property>
</configuration>

这里配置访问MySql的账号密码,需要把mysql的驱动包(mysql-connector-java.jar)放到Hive/lib下面。

在MySql里面事先需要创建爱你一个hive存放元数据的库。

mysql> create database hive;
Query OK, 1 row affected (0.11 sec)

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| hive               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

启动hive :

 

[hadoop@hd1 conf]$ hive 
which: no hbase in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/java/jdk1.8.0_11/bin:/usr/java/jdk1.8.0_11/bin:/home/hadoop/bin:/home/hadoop/hadoop-2.6.0-cdh5.7.0/bin:/home/hadoop/hadoop-2.6.0-cdh5.7.0/sbin:/home/hadoop/hive-1.1.0-cdh5.7.0/bin)
18/10/24 03:14:34 WARN conf.HiveConf: HiveConf of name hive.metastore.local does not exist

Logging initialized using configuration in jar:file:/home/hadoop/hive-1.1.0-cdh5.7.0/lib/hive-common-1.1.0-cdh5.7.0.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

© 著作权归作者所有

共有 人打赏支持
hnairdb
粉丝 20
博文 114
码字总数 114997
作品 0
深圳
数据库管理员
私信 提问
Hadoop 管理监控工具--Apache Ambari

Apache Ambari 是一个基于 Web 的 Apache Hadoop 集群的供应、管理和监控。Ambari目前已支持大多数Hadoop组件,包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeper、Sqoop和Hcatalog等。 Ap...

匿名
2012/12/25
23.5K
3
KYLIN访问远程hive和hbase环境

KYLIN访问远程hive和hbase环境 思路:在kylin服务器上配置所需环境的client;包括hadoop,hive,hbase 准备远程hadoop集群,hive环境,hbase环境的配置文件,替换掉对应客户端的配置文件。 ...

仔仔1993
09/11
0
0
编译安装spark

1.官网下载未编译的spark的jar包,选择下载的时候选择source code 2.安装jdk,maven。 3.修改spark根目录下的pom.xml文件(这一步不要做,做了反而会导致编译不过,因为使用-Phadoop-2.3参数...

cjun1990
2015/09/17
111
0
Apache Ambari 2.2.1 版本发布

Ambari 2.2.1 发布了,该版本添加了如下的特性: The ability to set rack_info within a Blueprint Updates to the Spark stack definition to support Spark 1.6 Service check for Spark ......

李玉珏
2016/03/25
1K
0
按照这个路线,让你4个月学会Hadoop开发

学习hadoop,首先我们要知道hadoop是什么? 说到底Hadoop只是一项分布式系统的工具,我们在学习的时候要理解分布式系统设计中的原则以及方法,只有这样才能以不变应万变。再一个就是一定要动...

I加加
2017/04/20
0
0

没有更多内容

加载失败,请刷新页面

加载更多

phpstorm xdebug 配置

xdebug方便了调试代码,比起一个一个地方的打印结果还是debug看的更明白下面介绍下maxOS系统下的debug配置 下载 https://xdebug.org/download.php 点击红线部分进入,粘贴phpinfo()信息推荐适...

被猪拱了的JAVA
25分钟前
2
0
Golang学习笔记(1)

基本知识 golang的文件格式以go结尾。 运行方式 go run main.go 用于开发调试使用 编译成二进制文件 go build main.go 会生成一个可执行的二进制文件 变量 变量定义的形式 Golang的变量定义有...

ExtreU
39分钟前
1
0
基于Kafka构建事件溯源模式的微服务

概要 本文中我们将讨论如何借助Kafka实现分布式消息管理,使用事件溯源(Event Sourcing)模式实现原子化数据处理,使用CQRS模式(Command-Query Responsibility Segregation )实现查询职责...

架构师springboot
45分钟前
1
0
git上传项目步骤

https://blog.csdn.net/m0_37725003/article/details/80904824

fame_yao
46分钟前
1
0
NOOBS自定义安装多系统

一、预置条件: 宿主系统是win10_x64 virtual box 虚拟机,安装了centos7 树莓派的系统安装工具:NOOBS_v2_9_0.zip,镜像文件 二、根据镜像文件生成boot.tar.xz 和root.tar.xz 1、设置共享目...

mbzhong
56分钟前
1
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部