Linux下Hive安装

原创
2017/01/08 01:32
阅读数 190

1、下载apache-hive-2.1.1-bin.tar.gz

2、安装Hive之前,请确保JDK1.7+、MySQL和Hadoop已安装完毕

3、解压apache-hive-2.1.1-bin.tar.gz,并拷贝到/usr/local目录

tar -zxvf apache-hive-2.1.1-bin.tar.gz
mv apache-hive-2.1.1-bin /app/hive

4、配置Hive环境变量,vi /etc/profile,追加以下内容,保存后执行source /etc/profile

HIVE_HOME=/app/hive
export PATH=$PATH:$HIVE_HOME/bin

5、修改Hive配置文件,conf/目录下

cp hive-env.sh.template hive-env.sh
cp hive-default.xml.template hive-site.xml
cp hive-log4j2.properties.template hive-log4j2.properties
cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties

cp llap-cli-log4j2.properties.template llap-cli-log4j2.properties
cp llap-daemon-log4j2.properties.template llap-daemon-log4j2.properties

    1. 修改hive-env.sh

export JAVA_HOME=/app/jdk1.8.0_111
export HADOOP_HOME=/app/hadoop
export HIVE_HOME=/app/hive
export HIVE_CONF_DIR=$HIVE_HOME/conf

    2. 修改hive-site.xml,修改数据库连接参数,这里采用MySQL

<property>
	<name>javax.jdo.option.ConnectionURL</name>
	<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&amp;characterEncoding=UTF-8&amp;useSSL=false
	</value>
</property>
<property>
	<name>javax.jdo.option.ConnectionDriverName</name>
	<value>com.mysql.jdbc.Driver</value>
</property>
<property>
	<name>javax.jdo.option.ConnectionUserName</name>
	<value>root</value>
</property>
<property>
	<name>javax.jdo.option.ConnectionPassword</name>
	<value>123456</value>
</property>

6、将mysql驱动mysql-connector-java-5.1.34.jar拷贝到${HIVE_HOME}/lib目录下

7、初始化元素数据,这里为mysql,默认derby

./schematool -initSchema -dbType mysql

8、启动hive

Hive

 

 

 

常见问题:

    Q:Hive报错信息如下:

         java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7B

    A:修改hive-site.xml,添加${system:java.io.tmpdir}和${system:user.name}

<property>
	<name>system:java.io.tmpdir</name>
	<value>/usr/local/hive-2.1.1/tmp</value>
</property>
<property>
	<name>system:user.name</name>
	<value>hive</value>
</property>

    Q:Hive报错信息如下:

          Caused by: MetaException(message:Version information not found in metastore. )

    A:修改hive-site.xml,将hive.metastore.schema.verification属性值为false,该参数防止架构版本不兼容时的 Metastore 操作。考虑将此设置为“True”,以减少 Metastore 操作期间发生架构损坏的可能性防止架构版本不兼容时的 Metastore 操作。考虑将此设置为“True”,以减少 Metastore 操作期间发生架构损坏的可能性。

    Q:Hive报错信息如下:

         Caused by: java.lang.RuntimeException: Unable to instantiate 

                         org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

    A:打开MySQL,执行如下语句:

use mysql;
set global binlog_format='MIXED';

   Q:Hive报错信息如下:

         Caused by: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql))

    A:执行schema初始化(以MySQL为例)

./schematool -initSchema -dbType mysql

   Q:Hive报错信息如下:

         Container [pid=3680,containerID=container_1483878515419_0001_01_000005] is running beyond virtual memory limits. Current usage: 75.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

    A:修改yarn-site.xml文件中的yarn.nodemanager.vmem-pmem-ratio属性值(默认为2:1,物理内存与虚拟内存比例)

<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>10</value>
</property>

        具体错误信息如下:

hive> create table t_loginfo as select * from loginfo;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = root_20170108203354_8b2827b5-0d4b-4d5b-8fff-ceaa784e9030
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1483878515419_0001, Tracking URL = http://localhost:8088/proxy/application_1483878515419_0001/
Kill Command = /usr/local/install_path/hadoop-2.7.3/bin/hadoop job  -kill job_1483878515419_0001
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2017-01-08 20:35:23,745 Stage-1 map = 0%,  reduce = 0%
2017-01-08 20:35:46,498 Stage-1 map = 100%,  reduce = 0%
2017-01-08 20:35:47,733 Stage-1 map = 0%,  reduce = 0%
2017-01-08 20:35:57,498 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1483878515419_0001 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1483878515419_0001_m_000000 (and more) from job job_1483878515419_0001

Task with the most failures(4): 
-----
Task ID:
  task_1483878515419_0001_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1483878515419_0001&tipid=task_1483878515419_0001_m_000000
-----
Diagnostic Messages for this Task:
Container [pid=3680,containerID=container_1483878515419_0001_01_000005] is running beyond virtual memory limits. Current usage: 75.0 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1483878515419_0001_01_000005 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 3680 3678 3680 3680 (bash) 1 0 108679168 351 /bin/bash -c /usr/local/install_path/jdk1.8.0_111/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx200m -Djava.io.tmpdir=/usr/local/install_path/hadoop-2.7.3/tmp/nm-local-dir/usercache/root/appcache/application_1483878515419_0001/container_1483878515419_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 57321 attempt_1483878515419_0001_m_000000_3 5 1>/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005/stdout 2>/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005/stderr  
	|- 3685 3680 3680 3680 (java) 370 25 2258333696 18847 /usr/local/install_path/jdk1.8.0_111/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m -Djava.io.tmpdir=/usr/local/install_path/hadoop-2.7.3/tmp/nm-local-dir/usercache/root/appcache/application_1483878515419_0001/container_1483878515419_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/install_path/hadoop-2.7.3/logs/userlogs/application_1483878515419_0001/container_1483878515419_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 57321 attempt_1483878515419_0001_m_000000_3 5 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

 

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部