关于64位Linux编译hadoop2-低调大师

关于64位Linux编译hadoop2

2017-11-12 563

Apache官方提供hadoop2的安装包是在32位机器下编译的，生产环境一般使用64的Linux，那么需要在64位机器下重新编译
可以查看hadoop-2.2.0-src下的BUILDING.txt
Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.6+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

----------------------------------------------------------------------------------
...
步骤：
1.安装JDK 1.6+ (验证:java -version)
2.安装Maven 3.0 or later (验证:mvn -version)
3.ProtocolBuffer 2.5.0 (验证:protoc --version)下载地址：https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
#为了编译安装protobuf，Linux需要上网，使用YUM在线安装依赖。如不能上网，就比较麻烦，需要一一下载每个依赖包再安装。
sudo yum install gcc
sudo yum install gcc-c++
sudo yum install make

#解压protobuf
sudo tar -zxvf protobuf-2.5.0.tar.gz
#进入到protobuf-2.5.0
cd protobuf-2.5.0
#编译安装
sudo ./configure
sudo make
sudo make install
4.安装CMake 2.6 or newer
sudo yum install cmake
sudo yum install openssl-devel
sudo yum install ncurses-devel
5.编译hadoop-2.2.0
#解压hadoop-2.2.0-src.tar.gz
tar -zxvf hadoop-2.2.0-src.tar.gz
#进入到hadoop-2.2.0-src
cd hadoop-2.2.0-src

#hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/pom.xml有个bug，这里需要修改一下
vim hadoop-common-project/hadoop-auth/pom.xml
#在<dependencies>标签内添加如下内容
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>

#编译
mvn package -DskipTests -Pdist,native

#编译好的hadoop-2.2.0再hadoop-2.2.0-src/hadoop-dist/target目录下

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [ 1.228 s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [ 0.894 s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [ 1.809 s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [ 0.222 s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [ 1.198 s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [ 2.205 s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [ 2.169 s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [ 1.583 s]
[INFO] Apache Hadoop Common .............................. SUCCESS [01:02 min]
[INFO] Apache Hadoop NFS ................................. SUCCESS [ 5.132 s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [ 0.038 s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [01:02 min]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [ 9.002 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [ 4.995 s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [ 2.647 s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [ 0.058 s]
[INFO] hadoop-yarn ....................................... SUCCESS [ 0.138 s]
[INFO] hadoop-yarn-api ................................... SUCCESS [ 31.854 s]
[INFO] hadoop-yarn-common ................................ SUCCESS [ 20.121 s]
[INFO] hadoop-yarn-server ................................ SUCCESS [ 0.105 s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [ 5.776 s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [ 10.490 s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [ 3.321 s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [ 8.311 s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [ 0.510 s]
[INFO] hadoop-yarn-client ................................ SUCCESS [ 3.929 s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [ 0.060 s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [ 1.720 s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [ 0.062 s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [ 16.204 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [ 1.779 s]
[INFO] hadoop-yarn-site .................................. SUCCESS [ 0.111 s]
[INFO] hadoop-yarn-project ............................... SUCCESS [ 2.287 s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [ 11.855 s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [ 2.560 s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [ 6.985 s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [ 3.319 s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [ 4.021 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [ 1.508 s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [ 4.176 s]
[INFO] hadoop-mapreduce .................................. SUCCESS [ 2.367 s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [ 2.902 s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [ 5.365 s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [ 1.673 s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [ 4.095 s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [ 2.962 s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [ 2.089 s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [ 2.190 s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [ 5.887 s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [ 1.149 s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [ 0.028 s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [ 6.514 s]
[INFO] Apache Hadoop Client .............................. SUCCESS [ 2.199 s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [ 0.121 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS (####看到BUILD SUCCESS说明编译成功####)
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:39 min
[INFO] Finished at: 2014-05-23T13:37:06+08:00
[INFO] Final Memory: 135M/300M
[INFO] ------------------------------------------------------------------------

本文转自SummerChill博客园博客，原文链接：http://www.cnblogs.com/DreamDrive/p/4546792.html，如需转载请自行联系原作者

微信关注我们

原文链接：https://yq.aliyun.com/articles/376555

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

[BigData]关于Hadoop学习笔记第三天(PPT总结)(一)

课程安排 MapReduce原理*** MapReduce执行过程** 数据类型与格式*** Writable接口与序列化机制*** ---------------------------加深拓展---------------------- MapReduce的执行过程源码分析问题：怎样解决海量数据的计算？ MapReduce概述 lMapReduce是一种分布式计算模型，由Google提出，主要用于搜索领域，解决海量数据的计算问题. lMR由两个阶段组成：Map和Reduce，用户只需要实现map()和reduce()两个函数，即可实现分布式计算，非常简单。 l这两个函数的形参是key、value对，表示函数的输入信息。思考：自己设计一个MapReduce框架 Mapreduce原理 ◆执行步骤： 1. map任务处理 1.1 读取输入文件内容，解析成key、value对。对输入文件的每一行，解析成key、value对。每一个键值对调用一次map函数。 1.2 写自己的逻辑，对输入的key、value处理，转换成新的key、value输出。 2.reduce任务处理 2.1...

2017-11-13

637

　MapReduce计算框架并没有直接调用CPU和内存等多维度资源，它把多维度资源抽象为“slot”，用“slot” 来描述资源的数量。管理员可以在每个节点上单独配置slot个数。slot可以分为map slot和reduce slot。从一定程度上，slot可以看做“任务运行并行度”。如果某个节点配置了5个map slot，那么这个节点最多运行5个Map Task；如果某个节点配置了3个reduce slot，那么该节点最多运行3个Reduce Task。下面我们分别介绍 Map slot和Reduce slot。 1、Map slot 1)Map slot 可用于运行Map Task 的资源，而且只能运行Map Task。 2)每个Map Task通常使用一个map slot。而比如像容量调度器，它可以有比较大的MapTask。这样的MapTask使用内存比较多，那么它可能使用多个map slot。 2、Reduce slot 1)Reduce slot 可用于运行ReduceTask，而且只能运行ReduceTask。 2)每个ReduceTask通常使用一个reduce slo...

2017-11-13

547

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能，例如代码缩略图，Python的插件，代码段等。还可自定义键绑定，菜单和工具栏。Sublime Text 的主要功能包括：拼写检查，书签，完整的 Python API ， Goto 功能，即时项目切换，多选择，多窗口等等。Sublime Text 是一个跨平台的编辑器，同时支持Windows、Linux、Mac OS X等操作系统。