文档章节

hadoop编译+native+snappy

Yulong_
 Yulong_
发布于 2016/09/02 14:43
字数 1377
阅读 414
收藏 0

本文介绍了hadoop编译的详细流程,包括依赖的安装,以及支持native和snappy库。

snappy

安装snappy

snappy github 地址:[snappy]https://github.com/google/snappy/releases "snappy")选择1.1.4版本下载至/opt/tmp

cd /opt/tmp   
unzip snappy-1.1.4.zip   
cd sanppy-1.1.4
yum install -y gcc gcc-c++ autoconf automake libtool  cmake openssl-devel

./autogen.sh   
./configure   
make   
make install   
ls -lh /usr/local/lib |grep snappy   

-rw-r--r--. 1 root root 462K Sep 1 20:38 libsnappy.a
-rwxr-xr-x. 1 root root 934 Sep 1 20:38 libsnappy.la
lrwxrwxrwx. 1 root root 18 Sep 1 20:38 libsnappy.so -> libsnappy.so.1.3.0
lrwxrwxrwx. 1 root root 18 Sep 1 20:38 libsnappy.so.1 -> libsnappy.so.1.3.0
-rwxr-xr-x. 1 root root 223K Sep 1 20:38 libsnappy.so.1.3.0

简单测试snappy

cd /tmp
vim a.cpp   

#include "snappy.h"
#include <string>
#include <iostream>
int main() {
std::string s = "ddsandslkadnsldsan;lkdsanf;ladnskal;fsan;lkfsjk;lfsna;lfdskfsa";
std::string d;
snappy::Compress(s.data(), s.size(), &d);
std::cout<<d<<std::endl;
std::cout<<s.size()<<" "<<d.size()<<std::endl;
return 0;
}

libtool --mode=compile gcc -c a.cpp   
libtool --mode=link g++ -o test a.lo /usr/local/lib/libsnappy.la   
./test

protobuf

安装protobuf

protobuf github 地址:protobuf选择2.5的版本下载至/opt/tmp

cd /opt/tmp   
unzip protobuf-2.5.0.zip
cd protobuf-2.5.0

修改autogen.sh,注释掉如下两行

curl http://googletest.googlecode.com/files/gtest-1.5.0.tar.bz2 | tar jx
mv gtest-1.5.0 gtest

并修改为

wget https://codeload.github.com/google/googletest/zip/release-1.5.0
unzip release-1.5.0
mv googletest-release-1.5.0 gtest

执行命令进行编译安装,默认安装protoc到/usr/local/bin

./autogen.sh
./configure  
make   
make check   
make install 

hadoop2.6的编译过程hadoop-common要求必须是protobuf-2.5.0版本,其他版本会错误。

测试protobuf

protoc --version

libprotoc 2.5.0

findbugs

###安装findbugs 下载findbugs 解压缩并设置环境变量:

export FINDBUGS_HOME=/opt/tmp/findbugs-3.0.0   
export PATH=$PATH:$FINDBUGS_HOME/bin

###测试findbugs findbugs -version

JDK

安装JDK

JDK7 地址:JDK7 下载jdk解压至/usr/local,修改java目录名为/usr/local/jdk

JDK环境变量

修改/etc/profile最后添加如下:

export JAVA_HOME=/usr/local/jdk
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

测试JDK

java -version

java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

maven

安装maven

apache maven下载地址:apache maven下载解压至/usr/local,修改maven目录名为/usr/local/maven

MAVEN环境变量

修改/etc/profile最后添加如下:

export MVN_HOME=/usr/local/maven
export PATH=$MVN_HOME/bin:$PATH

测试maven

mvn -v

Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T09:29:23-08:00) Maven home: /usr/local/maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/local/jdk1.7.0_79/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-642.4.2.el6.x86_64", arch: "amd64", family: "unix"

hadoop

编译hadoop

apache hadoop下载地址:apache hadoop下载解压至/opt/tmp,这里选择2.6.4。使用mvn编译.

cd /opt/tmp
tar zxf hadoop-2.6.4-src.tar.gz
cd hadoop-2.6.4-src
mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/local/lib -Dbundle.snappy   

输出(只截取最后部分):

main:
 [exec] $ tar cf hadoop-2.6.4.tar hadoop-2.6.4
 [exec] $ gzip -f hadoop-2.6.4.tar
 [exec] 
 [exec] Hadoop dist tar available at: /home/hadoop/Downloads/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.4.tar.gz
 [exec] 
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /home/hadoop/Downloads/hadoop-2.6.4-src/hadoop-dist/target/hadoop-dist-2.6.4-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.509 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.180 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  3.209 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.302 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.920 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.205 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  2.681 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.828 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.916 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:49 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  5.459 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 43.569 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.034 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:29 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 56.546 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  4.499 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.245 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.036 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.032 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [01:25 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 25.410 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.040 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  9.218 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 14.246 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.401 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  5.167 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 16.416 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.213 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  6.172 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.035 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  2.356 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.561 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.033 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  4.421 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  6.009 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.069 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 17.897 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 14.063 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.271 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  7.973 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  6.585 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  5.657 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.528 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.974 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  5.153 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.880 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.325 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.785 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  5.321 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  3.932 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.533 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  2.057 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.871 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  7.219 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  4.474 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  6.465 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  7.969 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.082 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.779 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.584 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.045 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 32.537 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12:31 min
[INFO] Finished at: 2016-09-02T00:01:32-07:00
[INFO] Final Memory: 207M/549M
[INFO] ------------------------------------------------------------------------

hadoop可执行软件包

相关软件包生成在/opt/tmp/hadoop-2.6.4-src/hadoop-dist/target

  • 压缩包为:
    /opt/tmp/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.0-cdh5.5.4.tar.gz

  • native库(包含snappy):
    /opt/tmp/hadoop-2.6.4-src/hadoop-dist/target/hadoop-2.6.0-cdh5.5.4/lib/native

测试native

hadoop checknative

_16/09/02 15:03:35 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/09/02 15:03:35 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /opt/beh/core/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /opt/beh/core/hadoop/lib/native/libsnappy.so.1
lz4: true revision:10301
bzip2: false
openssl: false Cannot load libcrypto.so (libcrypto.so: 无法打开共享对象文件: 没有那个文件或目录)! _

openssl: false Cannot load libcrypto.so (libcrypto.so: 无法打开共享对象文件: 没有那个文件或目录)! 异常解决办法是:
ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so

© 著作权归作者所有

Yulong_
粉丝 10
博文 145
码字总数 253510
作品 0
朝阳
部门经理
私信 提问
编译hadoop2.9源码并搭建环境

因为以前安装Hadoop编译版的时候遇到过问题,时间比较久了,具体问题有些描述不清了,建议大家下载源码自己编译安装,如果遇到bug自己直接可以修改安装。本文的实验系统是mac,但是其他环境类...

土木匠
2017/12/07
0
0
Spark的源码编译

Spark的版本下载和源码编译方法的官方地址:http://spark.apache.org/downloads.html 直接进Github,复制源码:https://github.com/apache/spark Spark 编译   有三种方式:SBT、MAVEN、m...

openthings
2015/12/17
148
0
Hadoop集群搭建-03编译安装hadoop

Hadoop集群搭建-05安装配置YARN Hadoop集群搭建-04安装配置HDFS Hadoop集群搭建-03编译安装hadoop Hadoop集群搭建-02安装配置Zookeeper Hadoop集群搭建-01前期准备 hadoop的编译和安装是直接...

虚拟世界的懒猫
07/22
53
0
CentOS 64位系统进行Hadoop2.3.0本地编译及完全分布式集群的部署

本文是在小编的博文《 基于Hadoop1.2.1完全分布式集群的部署 》的基础上写作的,所有硬件环境跟之前博文的硬件环境一模一样,因此本文不想再这方面费过多的口舌,关于hosts配置、JDK的安装和...

灯下黑鬼吹灯
2016/11/28
104
0
windows下使用Eclipse运行MapReduce程序出错: Failed to set...

在windows下用Eclipse运行MapReduce程序会报错: 这个问题在网上很常见了, 搜下有很多, 原因和解决方案: 这个是Windows下文件权限问题,在Linux下可以正常运行,不存在这样的问题。 解决方法是...

lifephp
2013/09/01
1K
0

没有更多内容

加载失败,请刷新页面

加载更多

数据流中的中位数

吐出的较小的N/2 个,都在大根堆里,较大的 N/2 个,都在小根堆里。 此时 5、4,都在大根堆,小根堆没有数。 此时应该从大根堆的堆顶弹出来,扔到小根堆里。 比如:先把 5 拿出来,再把堆最后...

Garphy
刚刚
0
0
微服务下配置管理中心 SCCA

SCCA 简介 SCCA 全称 spring-cloud-config-admin,微服务下 Spring Boot 应用(包含 Spring Cloud)统一的配置管理平台。 Github 地址:spring-cloud-config-admin 核心贡献者:程序员DD | ...

SpringForA
3分钟前
2
0
spring 是如何注入对象的和bean 创建过程分析

文章目录: beanFactory 及 bean 生命周期起步 BeanFactory refresh 全过程 BeanFactoryPostProcessor 和 BeanPostProcessor 解析 使用 BeanPostProcessor 实现 aop 和 springboot Cache 相关......

sanri1993
6分钟前
2
0
@PathVariable使用

public interface ProductInfoRepository extends JpaRepository<ProductInfo, String>{ List<ProductInfo> findByProductId(String productId);} ProductInfoController @Autowired ......

观海562
17分钟前
2
0
利用CSS禁止手机长按出现气泡: 复制、选择等功能

可以用 * ,也可作用于一个div div{  -webkit-touch-callout:none;  /*系统默认菜单被禁用*/  -webkit-user-select:none; /*webkit浏览器*/  -khtml-user-select:none; /*早期浏览...

蓝小驴
57分钟前
9
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部