Hadoop单机部署方法
安装Java JDK：
到sun网站上下载jdk
chmod +x jdk-6u30-linux-x64.bin
./jdk-6u30-linux-x64.bin
下载Hadoop
wget http://labs.renren.com/apache-mirror/hadoop/common/hadoop-0.20.205.0/hadoop-0.20.205.0.tar.gz
tar zxvf hadoop-0.20.205.0.tar.gz
安装必要软件
yum install ssh rsync
修改配置文件
conf/core-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
conf/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
conf/mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>
conf/ hadoop-env.sh
export JAVA_HOME=/root/jdk1.6.0_30
bin/hadoop
以root用户运行会报-jvm参数不存在的错误，故将
if [[ $EUID -eq 0 ]]; then
HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
else
HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
fi
修改为
elif [ "$COMMAND" = "datanode" ] ; then
CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode'
HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
SSH设置
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
启动hadoop
$ bin/hadoop namenode -format
$ bin/start-all.sh
此时，通过本地的50070和50030端口就可以分别浏览到NameNode和JobTracker：
-
NameNode - http://localhost:50070/
-
JobTracker - http://localhost:50030/
测试wordcount
生成测试数据：
[root hadoop-0.20.205.0]# mkdir input
[root hadoop-0.20.205.0]# echo "hello world" >> input/a.txt
[root hadoop-0.20.205.0]# echo "hello hadoop" >> input/b.txt
将本地数据复制到HDFS中：
[root hadoop-0.20.205.0]# bin/hadoop fs -put input in
执行测试任务：
[root hadoop-0.20.205.0]# bin/hadoop jar hadoop-examples-0.20.205.0.jar wordcount in out
12/02/05 21:00:47 INFO input.FileInputFormat: Total input paths to process : 2
12/02/05 21:00:48 INFO mapred.JobClient: Running job: job_201202052055_0001
12/02/05 21:00:49 INFO mapred.JobClient: map 0% reduce 0%
12/02/05 21:01:07 INFO mapred.JobClient: map 100% reduce 0%
12/02/05 21:01:19 INFO mapred.JobClient: map 100% reduce 100%
12/02/05 21:01:24 INFO mapred.JobClient: Job complete: job_201202052055_0001
12/02/05 21:01:24 INFO mapred.JobClient: Counters: 29
12/02/05 21:01:24 INFO mapred.JobClient: Job Counters
12/02/05 21:01:24 INFO mapred.JobClient: Launched reduce tasks=1
12/02/05 21:01:24 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=22356
12/02/05 21:01:24 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/02/05 21:01:24 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/02/05 21:01:24 INFO mapred.JobClient: Launched map tasks=2
12/02/05 21:01:24 INFO mapred.JobClient: Data-local map tasks=2
12/02/05 21:01:24 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=10801
12/02/05 21:01:24 INFO mapred.JobClient: File Output Format Counters
12/02/05 21:01:24 INFO mapred.JobClient: Bytes Written=25
12/02/05 21:01:24 INFO mapred.JobClient: FileSystemCounters
12/02/05 21:01:24 INFO mapred.JobClient: FILE_BYTES_READ=55
12/02/05 21:01:24 INFO mapred.JobClient: HDFS_BYTES_READ=235
12/02/05 21:01:24 INFO mapred.JobClient: FILE_BYTES_WRITTEN=64345
12/02/05 21:01:24 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=25
12/02/05 21:01:24 INFO mapred.JobClient: File Input Format Counters
12/02/05 21:01:24 INFO mapred.JobClient: Bytes Read=25
12/02/05 21:01:24 INFO mapred.JobClient: Map-Reduce Framework
12/02/05 21:01:24 INFO mapred.JobClient: Map output materialized bytes=61
12/02/05 21:01:24 INFO mapred.JobClient: Map input records=2
12/02/05 21:01:24 INFO mapred.JobClient: Reduce shuffle bytes=61
12/02/05 21:01:24 INFO mapred.JobClient: Spilled Records=8
12/02/05 21:01:24 INFO mapred.JobClient: Map output bytes=41
12/02/05 21:01:24 INFO mapred.JobClient: CPU time spent (ms)=2900
12/02/05 21:01:24 INFO mapred.JobClient: Total committed heap usage (bytes)=398852096
12/02/05 21:01:24 INFO mapred.JobClient: Combine input records=4
12/02/05 21:01:24 INFO mapred.JobClient: SPLIT_RAW_BYTES=210
12/02/05 21:01:24 INFO mapred.JobClient: Reduce input records=4
12/02/05 21:01:24 INFO mapred.JobClient: Reduce input groups=3
12/02/05 21:01:24 INFO mapred.JobClient: Combine output records=4
12/02/05 21:01:24 INFO mapred.JobClient: Physical memory (bytes) snapshot=422445056
12/02/05 21:01:24 INFO mapred.JobClient: Reduce output records=3
12/02/05 21:01:24 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1544007680
12/02/05 21:01:24 INFO mapred.JobClient: Map output records=4
查看结果：
[root hadoop-0.20.205.0]# bin/hadoop fs -cat out/*
hadoop 1
hello 2
world 1
cat: File does not exist: /user/root/out/_logs
将结果从HDFS复制到本地并查看：
[root hadoop-0.20.205.0]# bin/hadoop fs -get out output
[root hadoop-0.20.205.0]# cat output/*
cat: output/_logs: Is a directory
hadoop 1
hello 2
world 1
此时，从JobTracker网页中也可以看到任务的执行情况：
![]()
