hadoop2.x源码编译

小程序：扫一扫查出行
【扫一扫了解最新限行尾号】
复制小程序

转载请标明出处：

http://blog.csdn.net/zwto1/article/details/50733753；

介绍

本篇主要会涉及以下内容:

学会编译hadoop2.x源码

编译hadoop源码

为什么要编译

由于hadoop2.x 只支持Linux 32位，针对Linux 64位没有 native lib 库的支持，必须手动编译。所以下载的二进制包，当启动hadoop的时候，会出现如下警告：

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

如何编译

在64位Linux下编译

首先打开hadoop2.x源码下的build.txt 查看编译说明：

Requirements:

* Unix System

* JDK 1.6+

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6 or newer (if compiling native code)

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

Maven main modules: hadoop (Main Hadoop project) -hadoop-project (Parent POM for all Hadoop Maven modules. ) (All plugins & dependencies versions are defined here.) -hadoop-project-dist (Parent POM for modules that generate distributions.) - hadoop-annotations (Generates the Hadoop doclet used to generated the Javadocs) - hadoop-assemblies (Maven assemblies used by the different modules) - hadoop-common-project (Hadoop Common) - hadoop-hdfs-project (Hadoop HDFS) - hadoop-mapreduce-project (Hadoop MapReduce) - hadoop-tools (Hadoop tools like Streaming, Distcp, etc.) - hadoop-dist (Hadoop distribution assembler)

编译选项

Building distributions:Create binary distribution without native code and without documentation: $ mvn package -Pdist -DskipTests -DtarCreate binary distribution with native code and with documentation: $ mvn package -Pdist,native,docs -DskipTests -DtarCreate source distribution: $ mvn package -Psrc -DskipTestsCreate source and binary distributions with native code and documentation: $ mvn package -Pdist,native,docs,src -DskipTests -DtarCreate a local staging version of the website (in /tmp/hadoop-site) $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

编译前安装依赖包

yum install autoconf automake libtool cmakeyum install ncurses-develyum install openssl-develyum install lzo-devel zlib-devel gcc gcc-c++

查软件包是否没有装上：

rpm -qa | grep autoconf

编译前安装maven：

下载：apache-maven-3.0.5-bin.tar.gz

解压：

tar -zxvf apache-maven-3.0.5-bin.tar.gz

设置环境变量：

export MAVEN_HOME = /usr/local/mavenexport PATH=$PATH:$MAVEN_HOME/bin

执行命令：source /etc/profile或者 . /etc/profile

验证：mvn -v

编译前安装protobuf

下载：protobuf-2.5.0.tar.gz

解压:

tar -zxvf protobuf-2.5.0.tar.gz

编译安装：

进入安装目录，进行配置，执行命令：

./configure

安装命令：

make & make install

编译前安装findbugs

下载：findbugs-3.0.0.tar.gz

解压:

tar -zxvf findbugs-3.0.0.tar.gz

设置环境变量：

export FINDBUGS_HOME = /usr/local/findbugsexport PATH=$PATH:$FINDBUGS_HOME/bin

执行命令：source /etc/profile或者 . /etc/profile

验证：findbugs -version

MAVEN国内镜像配置

1、进入安装目录 /opt/modules/apache-maven-3.0.5/conf，编辑 settings.xml 文件

修改<mirrors>内容： <mirror> <id>nexus-osc</id> <mirrorOf>*</mirrorOf> <name>Nexus osc</name> <url>http://maven.oschina.net/content/groups/public/</url> </mirror>

修改<profiles>内容：<profile> <id>jdk-1.8</id> <activation> <jdk>1.8</jdk> </activation> <repositories> <repository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> <pluginRepositories> <pluginRepository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </pluginRepository> </pluginRepositories> </profile>

2、复制配置

将该配置文件复制到用户目录，使得每次对maven创建时，都采用该配置

* 查看用户目录【/home/hadoop】是否存在【.m2】文件夹，如没有，则创建

$ cd /home/hadoop$ mkdir .m2* 复制文件$ cp /opt/modules/apache-maven-3.0.5/conf/settings.xml ~/.m2/

3.配置DNS

修改： vi /etc/resolv.conf

nameserver 8.8.8.8

nameserver 8.8.4.4

开始编译

Hadoop2.2 .0 有bug ，编译之前需要先 fix bug

下载 hadoop2.x源码

解压

源码打patch,编辑目录$HADOOP_SRC_HOME/hadoop-common-project/hadoop-auth中的文件pom.xml,在第55行下增加以下内容：

<dependency> <groupId>org.mortbay.jetty</groupId> <artfactId>jetty-util<artfactId> <scope>test</scope> </dependency>

这里编译的是hadoop2.7.1版本