文档章节

Hadoop Hive

manonline
 manonline
发布于 2017/07/23 19:58
字数 488
阅读 2
收藏 0

Hive Service

Hive Shell/CLI, beeline, HiveServer2, Hive Web Interface, jar, metastore.

Hive Shell

The primary way to interact with Hive by issuing command in HiveQL.

  • %hive
  • hive> hiveql ...
  • %hive -f 'script'
  • %hive -e 'hiveql'

HiveServer2

Run hive as a server exposing a Thrift service, enabling access from a range of clients written in different languages.

Hadoop Cluster

FileSystem

A Hive table is logically made up of the data being stored and the associated metadata describing the layout of the data in the table.

Data resides in Hadoop filesystem, which includes local filesystem, S3 and HDFS.

Metadata is stored separately in a RDBMS, which is default to Derby

Execution Engine

  • hive.execution.engine=mapreduce, tez, spark;
  • MapReduce is the default. Both Tez and Spark are general DAG engine that provides more flexibility and higher performance than MapReduce.

Resource Manager

  • default to local job runner.
  • yarn.resourcemanager.address.

Metastore

The central repository of Hive metadata, which is divided into two pieces:

  • metadata service: by default, it's running in the same JVM as Hive
  • metadata store: by default, it uses embedded Derby database backed by the local disk, which only allows one user to connect at a time.

Configuration

Precedence hierarchy of configuring Hive

  • The Hive set command (hive>)
  • The command line -hiveconf option
  • hive-site.xml and Hadoop site files 
    • core-site.xml
    • hdfs-site.xml
    • mapred-site.xml
    • yarn-site.xml
  • Hive default and Hadoop default
    • core-default.xml
    • hdfs-default.xml
    • mapred-default.xml
    • yarn-default.xml

Table

Create Table

----- MANAGED TABLE -----
-- data is moved to Hive Warehouse
CREATE TABLE table_name (
    field1 type1,
    field2 type2,
    field3 type3,
    ...
)
----- EXTERNAL TABLE ----
-- data remain as is, and not moved
CREATE EXTERNAL TABLE table_name (
    field1 type1,
    field2 type2,
    field3 type3,
    ...
)
LOCALTION 'path'

----- STORAGE FORMAT ----
-- default : TEXTFILE, 
-- row based binary: AVRO, SEQUENCEFILE,
-- column based binary: PARQUET, RCFILE, ORCFILE
STORED AS TEXTFILE 

----- ROW FORMAT ------
-- only needed for TEXTFILE: DELIMINATED, SERDE
ROW FORMAT DELIMINATED
    FIELDS TERMINATED BY '\001'
    COLLECTION ITEMS TERMINATED BY '\002'
    MAP KEYS TERMINATED BY '\003'
    LINES TERMINATED BY '\n'

ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
 ....
)

----- STORAGE HANDLER -----
-- non-native storage, for example, HBase
STORED BY 

Load Data

----- LOAD DATA -----
LOAD DATA
LOCAL INPATH 'path to source file'
-- replace existing table
[OVERWRITE] 
-- copy file to $HIVE/warehouse/table_name/
INTO TABLE table_name

----- IMPORT DATA -----
-- at creation
CREATE TABLE target_table (...)
    AS
SELECT field1, field2 ...
  FROM source_table

-- post creation
INSERT [OVERWRITE] TABLE target_table
[PARTITION (dt=value)]
SELECT fiel1, field2 ...
  FROM source_table

-- one source to multiple targets
FROM source_table
INSERT [OVERWRITE] TABLE target_table1
SELECT ...
INSERT [OVERWRITE] TABLE target_table2
SELECT ...

Others

Partition and Bucket

A way of coarse-grained parts based on the value of a partition column, such as a date. Using partitions can make it faster to do queries on sliced data.

--
CREATE TABLE log (ts BIGINT, line STRING)
PARTITIONED BY (dt STRING, country STRING)

--
LOAD DATA 
LOCAL INPATH 'path to source'
INTO TABLE log
PARTITION (dt='2001-01-01', country='GB')

 

Query

Sorting and Aggregation, MapReduce Scripts, Subqueries, Views, Joins

  • Inner Joins
  • Outer Joins
  • Semi Joins
  • Map Joins

User Defined Function (UDF)

UDF and UDAF.

© 著作权归作者所有

共有 人打赏支持
manonline
粉丝 0
博文 73
码字总数 66740
作品 0
Hive Server的启动debug命令

CLI到了这一步就需要去连接Hive Server了,所以现在开始转向研究Hive Server的启动过程。 ======================================================= 先看shell里怎么启动Hive Server 那么启...

强子哥哥
2016/03/15
136
0
hive load 数据 报错

hive> load data local inpath '/usr/local/text.txt' overwrite into table pokes; Loading data to table default.pokes java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.DFSClient.......

jiahc
2015/03/31
1K
2
hive 启动报错 无法识别元数据库

hive2.1.1版本,使用MySQL作为元数据库,元数据可以正常初始化,但是用./bin/hive启动时候报错 2017-07-21T11:23:59,477 INFO [main] conf.HiveConf: Found configuration file file:/data/s...

sparkman
2017/07/21
64
0
hive之执行shell脚本注解

#!/usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for a......

skysky
2013/05/28
0
0
Hive报错,NoSuchMethodError: org.apache.hadoop.mapred

hive能正常运行,当查到xiaoping数据库中的student表中的数据hive自动退出而报错信息如下。 应该是Hadoop和hive版本不兼容的问题,还有就是是hive版本高还是Apache和cdh不兼容,这两个原因都...

jiaohuanji88
2017/12/24
0
0

没有更多内容

加载失败,请刷新页面

加载更多

用Golang做了一个命令行贪吃蛇游戏

用Golang做了一个命令行贪吃蛇游戏 项目介绍 项目链接:https://gitee.com/lwow2025/snake-go 最近看了一本做几个小项目的书,突然就想用Golang做一个命令行贪吃蛇,也没啥特殊原因。 软件架...

Mediv
16分钟前
0
0
storm的利用并行度提高处理速度的经验

在storm的流计算框架中,在数据量非常大或者计算逻辑比较复杂的情况下,可能会造成处理速度变慢的情况,最后反而不满足了系统的处理要求,因此这里讨论一下。本文的内容是我在storm的使用过程...

飓风2000
25分钟前
0
0
课程推荐|深入浅出区块链博主:全栈区块链开发者的4堂必修课(线上优惠)

Tiny熊从2017年开始更新“深入浅出区块链”博客,在第一篇文章中,关于如何系统学习区块链技术,他这样描述:“从事区块链开发也有很多方向,如:区块链应用开发人员、区块链架构师、底层核心...

HiBlock
37分钟前
0
0
激活win10 亲测有效

1.首先,我们先查看一下Win10正式专业版系统的激活状态: 点击桌面左下角的“Windows”按钮,从打开的扩展面板中依次点击“设置”-“更新和安全”,并切换到“激活”选项卡,在此就可以查看到...

可达鸭眉头一皱
39分钟前
0
0
SpringWind180926

SpringWind SpringWind项目代码学习笔记 /SpringWind/src/main/webapp/WEB-INF/views/login.html 第15行action="#springUrl('/account/login.html')"【为什么是#springUrl】 第4行<a class=......

颖伙虫
51分钟前
1
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部