spark与Hive整合

原创
07/19 18:46
阅读数 57

网上看了许多的文章,spark-shell还是不能识别hive表,后来发现如下操作能成功。

1,将hive_site.xml复制到spark conf目录

<configuration>
  <property>
    <name>hive.cli.print.current.db</name>
    <value>true</value>
    <description>Whether to include the current database in the Hive prompt.</description>
  </property>
    <property>
    <name>hive.cli.print.header</name>
    <value>true</value>
    <description>Whether to print the names of the columns in query output.</description>
  </property>

<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>

</property>

<!-- <property> -->
    <!-- <name>hive.metastore.uris</name> -->
    <!-- <value>thrift://localhost:9083</value> -->
    <!-- <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description> -->
  <!-- </property> -->
 
 
        <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://localhost:3306/hivedb?createDatabaseIfNotExist=true</value>
        </property>
        <property>
                <name>javax.jdo.option.ConnectionDriverName</name>
                <value>com.mysql.jdbc.Driver</value>
        </property>
        <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>root</value>
        </property>
        <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>hadoop.123</value>
        </property>
        
</configuration>

2, 将Hive lib目录下的jar包复制到spark jars下

3,下载如下两个包到spark jars

spark-hive_2.11-2.0.0.jar, spark-hive-thriftserver_2.11-2.0.0.jar

4,启动spark-shell或spark-sql, 能成功访问hive表

 

展开阅读全文
打赏
0
0 收藏
分享
加载中
更多评论
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部