学习笔记CB011:lucene搜索引擎库、IKAnalyzer中文切词工具、检索服务、查询索引、导流、word2vec

影视剧字幕聊天语料库特点，把影视剧说话内容一句一句以回车换行罗列三千多万条中国话，相邻第二句很可能是第一句最好回答。一个问句有很多种回答，可以根据相关程度以及历史聊天记录所有回答排序，找到最优，是一个搜索排序过程。

lucene+ik。lucene开源免费搜索引擎库，java语言开发。ik IKAnalyzer，开源中文切词工具。语料库切词建索引，文本搜索做文本相关性检索，把下一句取出作答案候选集，答案排序，问题分析。

建索引。eclipse创建maven工程，maven自动生成pom.xml文件，配置包依赖信息，dependencies标签中添加依赖：

 <dependency> <groupId>org.apache.lucene</groupId> <artifactId>lucene-core</artifactId> <version>4.10.4</version> </dependency> <dependency> <groupId>org.apache.lucene</groupId> <artifactId>lucene-queryparser</artifactId> <version>4.10.4</version> </dependency> <dependency> <groupId>org.apache.lucene</groupId> <artifactId>lucene-analyzers-common</artifactId> <version>4.10.4</version> </dependency> <dependency> <groupId>io.netty</groupId> <artifactId>netty-all</artifactId> <version>5.0.0.Alpha2</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.1.41</version> </dependency> 

project标签增加配置，依赖jar包自动拷贝lib目录：

<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy-dependencies</id> <phase>prepare-package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <outputDirectory>${project.build.directory}/lib</outputDirectory> <overWriteReleases>false</overWriteReleases> <overWriteSnapshots>false</overWriteSnapshots> <overWriteIfNewer>true</overWriteIfNewer> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <classpathPrefix>lib/</classpathPrefix> <mainClass>theMainClass</mainClass> </manifest> </archive> </configuration> </plugin> </plugins> </build> 

https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/ik-analyzer/IK%20Analyzer%202012FF_hf1_source.rar 下载ik源代码把src/org目录拷到chatbotv1工程src/main/java下，刷新maven工程。

com.shareditor.chatbotv1包下maven自动生成App.java，改成Indexer.java：

Analyzer analyzer = new IKAnalyzer(true); IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_4_9, analyzer); iwc.setOpenMode(OpenMode.CREATE); iwc.setUseCompoundFile(true); IndexWriter indexWriter = new IndexWriter(FSDirectory.open(new File(indexPath)), iwc); BufferedReader br = new BufferedReader(new InputStreamReader( new FileInputStream(corpusPath), "UTF-8")); String line = ""; String last = ""; long lineNum = 0; while ((line = br.readLine()) != null) { line = line.trim(); if (0 == line.length()) { continue; } if (!last.equals("")) { Document doc = new Document(); doc.add(new TextField("question", last, Store.YES)); doc.add(new StoredField("answer", line)); indexWriter.addDocument(doc); } last = line; lineNum++; if (lineNum % 100000 == 0) { System.out.println("add doc " + lineNum); } } br.close(); indexWriter.forceMerge(1); indexWriter.close(); 

编译拷贝src/main/resources所有文件到target目录，target目录执行

java -cp $CLASSPATH:./lib/:./chatbotv1-0.0.1-SNAPSHOT.jar com.shareditor.chatbotv1.Indexer ../../subtitle/raw_subtitles/subtitle.corpus ./index 

生成索引目录index通过lukeall-4.9.0.jar查看。

检索服务。netty创建http服务server，代码在https://github.com/warmheartli/ChatBotCourse的chatbotv1目录：

Analyzer analyzer = new IKAnalyzer(true); QueryParser qp = new QueryParser(Version.LUCENE_4_9, "question", analyzer); if (topDocs.totalHits == 0) { qp.setDefaultOperator(Operator.AND); query = qp.parse(q); System.out.println(query.toString()); indexSearcher.search(query, collector); topDocs = collector.topDocs(); } if (topDocs.totalHits == 0) { qp.setDefaultOperator(Operator.OR); query = qp.parse(q); System.out.println(query.toString()); indexSearcher.search(query, collector); topDocs = collector.topDocs(); } ret.put("total", topDocs.totalHits); ret.put("q", q); JSONArray result = new JSONArray(); for (ScoreDoc d : topDocs.scoreDocs) { Document doc = indexSearcher.doc(d.doc); String question = doc.get("question"); String answer = doc.get("answer"); JSONObject item = new JSONObject(); item.put("question", question); item.put("answer", answer); item.put("score", d.score); item.put("doc", d.doc); result.add(item); } ret.put("result", result); 

查询索引，query词做切词拼lucene query，检索索引question字段，匹配返回answer字段值作候选集，挑出候选集一条作答案。server通过http访问，如http://127.0.0.1:8765/?q=hello 。中文需转urlcode发送，java端读取按urlcode解析，server启动方法：

java -cp $CLASSPATH:./lib/:./chatbotv1-0.0.1-SNAPSHOT.jar com.shareditor.chatbotv1.Searcher 

聊天界面。一个展示聊天内容框框，选择ckeditor，支持html格式内容展示，一个输入框和发送按钮，html代码：

<div class="col-sm-4 col-xs-10"> <div class="row"> <textarea id="chatarea"> <div style='color: blue; text-align: left; padding: 5px;'>机器人: 喂，大哥您好，您终于肯跟我聊天了,来侃侃呗,我来者不拒!</div> <div style='color: blue; text-align: left; padding: 5px;'>机器人: 啥?你问我怎么这么聪明会聊天?因为我刚刚吃了一堆影视剧字幕!</div> </textarea> </div> <br /> <div class="row"> <div class="input-group"> <input type="text" id="input" class="form-control" autofocus="autofocus" onkeydown="submitByEnter()" /> <span class="input-group-btn"> <button class="btn btn-default" type="button" onclick="submit()">发送</button> </span> </div> </div> </div> <script type="text/javascript"> CKEDITOR.replace('chatarea', { readOnly: true, toolbar: ['Source'], height: 500, removePlugins: 'elementspath', resize_enabled: false, allowedContent: true }); </script> 

调用聊天server，要一个发送请求获取结果控制器：

public function queryAction(Request $request) { $q = $request->get('input'); $opts = array( 'http'=>array( 'method'=>"GET", 'timeout'=>60, ) ); $context = stream_context_create($opts); $clientIp = $request->getClientIp(); $response = file_get_contents('http://127.0.0.1:8765/?q=' . urlencode($q) . '&clientIp=' . $clientIp, false, $context); $res = json_decode($response, true); $total = $res['total']; $result = ''; if ($total > 0) { $result = $res['result'][0]['answer']; } return new Response($result); } 

控制器路由配置：

chatbot_query: path: /chatbot/query defaults: { _controller: AppBundle:ChatBot:query } 

聊天server响应时间比较长，不导致web界面卡住，执行submit时异步发请求和收结果：

var xmlHttp; function submit() { if (window.ActiveXObject) { xmlHttp = new ActiveXObject("Microsoft.XMLHTTP"); } else if (window.XMLHttpRequest) { xmlHttp = new XMLHttpRequest(); } var input = $("#input").val().trim(); if (input == '') { jQuery('#input').val(''); return; } addText(input, false); jQuery('#input').val(''); var datastr = "input=" + input; datastr = encodeURI(datastr); var url = "/chatbot/query"; xmlHttp.open("POST", url, true); xmlHttp.onreadystatechange = callback; xmlHttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded"); xmlHttp.send(datastr); } function callback() { if (xmlHttp.readyState == 4 && xmlHttp.status == 200) { var responseText = xmlHttp.responseText; addText(responseText, true); } } 

addText往ckeditor添加一段文本：

function addText(text, is_response) { var oldText = CKEDITOR.instances.chatarea.getData(); var prefix = ''; if (is_response) { prefix = "<div style='color: blue; text-align: left; padding: 5px;'>机器人: " } else { prefix = "<div style='color: darkgreen; text-align: right; padding: 5px;'>我: " } CKEDITOR.instances.chatarea.setData(oldText + "" + prefix + text + "</div>"); } 

代码：
https://github.com/warmheartli/ChatBotCourse
https://github.com/warmheartli/shareditor.com

效果演示：http://www.shareditor.com/chatbot/

导流。统计网站流量情况。cnzz统计看最近半个月受访页面流量情况，用户访问集中页面。增加图库动态按钮。吸引用户点击，在每个页面右下角放置动态小图标，页面滚动它不动，用户点了直接跳到想要引流的页面。搜客服漂浮代码。
创建js文件，lrtk.js ：

$(function() { var tophtml="<a href=\"http://www.shareditor.com/chatbot/\" target=\"_blank\"><div id=\"izl_rmenu\" class=\"izl-rmenu\"><div class=\"btn btn-phone\"></div><div class=\"btn btn-top\"></div></div></a>"; $("#top").html(tophtml); $("#izl_rmenu").each(function() { $(this).find(".btn-phone").mouseenter(function() { $(this).find(".phone").fadeIn("fast"); }); $(this).find(".btn-phone").mouseleave(function() { $(this).find(".phone").fadeOut("fast"); }); $(this).find(".btn-top").click(function() { $("html, body").animate({ "scroll-top":0 },"fast"); }); }); var lastRmenuStatus=false; $(window).scroll(function() { var _top=$(window).scrollTop(); if(_top>=0) { $("#izl_rmenu").data("expanded",true); } else { $("#izl_rmenu").data("expanded",false); } if($("#izl_rmenu").data("expanded")!=lastRmenuStatus) { lastRmenuStatus=$("#izl_rmenu").data("expanded"); if(lastRmenuStatus) { $("#izl_rmenu .btn-top").slideDown(); } else { $("#izl_rmenu .btn-top").slideUp(); } } }); }); 

上半部分定义id=top的div标签内容。一个id为izl_rmenu的div，css格式定义在另一个文件lrtk.css里：

.izl-rmenu{position:fixed;left:85%;bottom:10px;padding-bottom:73px;z-index:999;} .izl-rmenu .btn{width:72px;height:73px;margin-bottom:1px;cursor:pointer;position:relative;} .izl-rmenu .btn-top{background:url(http://www.shareditor.com/uploads/media/default/0001/01/thumb_416_default_big.png) 0px 0px no-repeat;background-size: 70px 70px;display:none;} 

下半部分当页面滚动时div展开。

在所有页面公共代码部分增加

<div id="top"></div> 

庞大语料库运用，LSTM-RNN训练，中文语料转成算法识别向量形式，最强大word embedding工具word2vec。

word2vec输入切词文本文件，影视剧字幕语料库回车换行分隔完整句子，所以我们先对其做切词，word_segment.py文件：

# coding:utf-8 import sys import importlib importlib.reload(sys) import jieba from jieba import analyse def segment(input, output): input_file = open(input, "r") output_file = open(output, "w") while True: line = input_file.readline() if line: line = line.strip() seg_list = jieba.cut(line) segments = "" for str in seg_list: segments = segments + " " + str segments = segments + "\n" output_file.write(segments) else: break input_file.close() output_file.close() if __name__ == '__main__': if 3 != len(sys.argv): print("Usage: ", sys.argv[0], "input output") sys.exit(-1) segment(sys.argv[1], sys.argv[2]); 

使用：

python word_segment.py subtitle/raw_subtitles/subtitle.corpus segment_result 

word2vec生成词向量。word2vec可从https://github.com/warmheartli/ChatBotCourse/tree/master/word2vec获取，make编译生成二进制文件。
执行：

./word2vec -train ../segment_result -output vectors.bin -cbow 1 -size 200 -window 8 -negative 25 -hs 0 -sample 1e-4 -threads 20 -binary 1 -iter 15 

生成vectors.bin词向量，二进制格式，word2vec自带distance工具来验证：

./distance vectors.bin 

词向量二进制文件格式加载。word2vec生成词向量二进制格式：词数目(空格)向量维度。
加载词向量二进制文件python脚本：

# coding:utf-8 import sys import struct import math import numpy as np reload(sys) sys.setdefaultencoding( "utf-8" ) max_w = 50 float_size = 4 def load_vectors(input): print "begin load vectors" input_file = open(input, "rb") # 获取词表数目及向量维度 words_and_size = input_file.readline() words_and_size = words_and_size.strip() words = long(words_and_size.split(' ')[0]) size = long(words_and_size.split(' ')[1]) print "words =", words print "size =", size word_vector = {} for b in range(0, words): a = 0 word = '' # 读取一个词 while True: c = input_file.read(1) word = word + c if False == c or c == ' ': break if a < max_w and c != '\n': a = a + 1 word = word.strip() # 读取词向量 vector = np.empty([200]) for index in range(0, size): m = input_file.read(float_size) (weight,) = struct.unpack('f', m) vector[index] = weight # 将词及其对应的向量存到dict中 word_vector[word.decode('utf-8')] = vector input_file.close() print "load vectors finish" return word_vector if __name__ == '__main__': if 2 != len(sys.argv): print "Usage: ", sys.argv[0], "vectors.bin" sys.exit(-1) d = load_vectors(sys.argv[1]) print d[u'真的'] 

运行方式如下：

python word_vectors_loader.py vectors.bin 

参考资料：

《Python 自然语言处理》

http://www.shareditor.com/blogshow?blogId=113

http://www.shareditor.com/blogshow?blogId=114

http://www.shareditor.com/blogshow?blogId=115

欢迎推荐上海机器学习工作机会