MapReduce实现协同过滤中每个用户看过的项目集合
一、知识准备
hadoop自带的例子在
D:\HADOOP_HOME\hadoop-2.6.4\share\hadoop\mapreduce\sources\hadoop-mapreduce-examples 2.6.0-source.jar
我记得当年面试的时候就问中位数的问题不过是数据流下的中位数，一问便知是否搞过hadoop。
二、代码实现
2.1 Mapper
package cf;
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
public class MovieMapper1 extends Mapper<LongWritable, Text, Text, Text> {
public void map(LongWritable ikey, Text ivalue, Context context)
throws IOException, InterruptedException {
String[] values = ivalue.toString().split(",");
if (values.length!=2) {
return ;
}
String userID = values[0];
String itemID = values[1];
context.write(new Text(userID), new Text(itemID));
}
}
2.2 Reducer
package cf;
import java.io.IOException;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;
public class MovieReduce1 extends Reducer<Text, Text, Text, Text> {
public void reduce(Text _key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
// process values
StringBuffer sb = new StringBuffer();
for (Text val : values) {
sb.append(val.toString());
sb.append(",");
}
//value不能直接用StringBuffer 必须转换为String
context.write(_key,new Text(sb.toString()));
}
}
2.3 Main
package cf;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class UserItemSetMapReduce {
public static void main(String[] args) throws Exception{
Configuration conf = new Configuration();
Job job = new Job(conf, "CFItemSet");
job.setJarByClass(UserItemSetMapReduce.class);
job.setMapperClass(MovieMapper1.class);
//job.setCombinerClass(cls);
// job.setCombinerClass(MovieReduce1.class);
job.setReducerClass(MovieReduce1.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileInputFormat.addInputPath(job,new Path("hdfs://192.168.58.180:8020/cf/userItem.txt"));
//InputPath(job, new Path(otherArgs[0]));
//直接写到cf会提示已存在cf，我写成uIO.ttx，以为内容会写入到txt，然没有，默认他是文件夹
FileOutputFormat.setOutputPath(job,new Path("hdfs://192.168.58.180:8020/cf/userItemOut.txt"));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
三、结果分析
3.1 输入
3.2 输出
查看结果发现输出文件的分隔符默认是tab，‘\t’，同时相对于输入文件来说输出结果是逆着的，类似沾，莫非context就是这样的先进后出、
3.3日志分析
只列出了主要部分的日志
DEBUG - PrivilegedAction as:hxsyl (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getCounters(Job.java:765) INFO - Counters: 38 File System Counters FILE: Number of bytes read=538 FILE: Number of bytes written=509366 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=106 HDFS: Number of bytes written=37 HDFS: Number of read operations=13 HDFS: Number of large read operations=0 HDFS: Number of write operations=4 Map-Reduce Framework Map input records=11 Map output records=11 Map output bytes=44 Map output materialized bytes=72 Input split bytes=107 Combine input records=0 Combine output records=0 Reduce input groups=5 Reduce shuffle bytes=72 Reduce input records=11 Reduce output records=5 Spilled Records=22 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=3 CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 Total committed heap usage (bytes)=462422016 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=53 File Output Format Counters Bytes Written=37 DEBUG - PrivilegedAction as:hxsyl (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:323) DEBUG - stopping client from cache: org.apache.hadoop.ipc.Client@37afeb11 DEBUG - removing client from cache: org.apache.hadoop.ipc.Client@37afeb11 DEBUG - stopping actual client because no more references remain: org.apache.hadoop.ipc.Client@37afeb11 DEBUG - Stopping client DEBUG - IPC Client (521081105) connection to /192.168.58.180:8020 from hxsyl: closed DEBUG - IPC Client (521081105) connection to /192.168.58.180:8020 from hxsyl: stopped, remaining connections 0
大神分析一下如何执行的，看着日志....Map如何输入的，执行几次等。
