首页 文章 精选 留言 我的

精选列表

搜索[python],共10000篇文章
优秀的个人博客,低调大师

[python][spark]wholeTextFiles 读入多个文件的例子

$pwd /home/training/mydir $cat file1.json { "firstName":"Fred", "lastName":"Flintstone", "userid":"123" } $cat file2.json { "firstName":"Barney", "lastName":"Rubble", "userid":"123" } [training@localhost ~]$ hdfs dfs -put /home/training/mydir [training@localhost ~]$ [training@localhost ~]$ hdfs dfs -ls Found 4 items drwxrwxrwx - training supergroup 0 2017-09-23 19:26 .sparkStaging -rw-rw-rw- 1 training supergroup 48 2017-09-25 05:31 cats.txt drwxrwxrwx - training supergroup 0 2017-09-25 15:39 mydir *** -rw-rw-rw- 1 training supergroup 34 2017-09-23 06:16 test.txt [training@localhost ~]$ myrdd1 = sc.wholeTextFiles("mydir") myrdd1.count() Out[32]: 2 In [35]: myrdd1.take(2) Out[35]: [(u'hdfs://localhost:8020/user/training/mydir/file1.json', u'{\n "firstName":"Fred",\n "lastName":"Flintstone",\n "userid":"123"\n}\n'), (u'hdfs://localhost:8020/user/training/mydir/file2.json', u'{\n "firstName":"Barney",\n "lastName":"Rubble",\n "userid":"456"\n}\n')] 本文转自健哥的数据花园博客园博客,原文链接:http://www.cnblogs.com/gaojian/p/7594782.html,如需转载请自行联系原作者

优秀的个人博客,低调大师

Python——使用ElementTree解析AndroidManifest.xml文件

XML内容如下: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 <? xml version = "1.0" encoding = "utf-8" standalone = "no" ?> < manifest xmlns:android = "http://schemas.android.com/apk/res/android" package = "com.dongbao.wauoen.interappactivitytesting" platformBuildVersionCode = "23" platformBuildVersionName = "6.0-2704002" > < uses-permission android:name = "android.permission.READ_EXTERNAL_STORAGE" /> < application android:allowBackup = "true" android:icon = "@mipmap/ic_launcher" android:label = "@string/app_name" android:supportsRtl = "true" android:theme = "@style/AppTheme" > < activity android:exported = "true" android:name = "com.dongbao.wauoen.interappactivitytesting.MainActivity" > < intent-filter > < action android:name = "android.intent.action.MAIN" /> < category android:name = "android.intent.category.LAUNCHER" /> </ intent-filter > </ activity > < activity android:enabled = "@bool/use_activity" android:exported = "true" android:icon = "@drawable/ic_chooser" android:label = "@string/choose_file" android:name = "com.ipaulpro.afilechooser.FileChooserActivity" /> < activity android:exported = "true" android:name = "com.dongbao.wauoen.interappactivitytesting.ExampleActivity" android:permission = "android.permission.READ_EXTERNAL_STORAGE" /> </ application > </ manifest > 2.解析过程:主要获取开放activity的信息,并保存到xml文件中 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 from ActivityInfo import ActivityInfo try : import xml.etree.cElementTreeasET except ImportError: import xml.etree.ElementTreeasET import sys #Manifestnamespace namespace = '{http://schemas.android.com/apk/res/android}' #Exportedactivities exported_activities = [] #XMLanalyze def __xml_analyze__(): tree = ET.parse(r "./app/AndroidManifest.xml" ) root = tree.getroot() #printtree.namespace #1.getpackagename packagename = root.attrib[ "package" ] #2.getexportedactivity application = root.find( "application" ) __get_exported_activities__(application,packagename) for activity in exported_activities: print activity.__get_name__(),activity.__get_package_name__(),activity.__get_permission__() #3.saveresulttoxmlfile __save_result_to_xml__() #getexportedactivities def __get_exported_activities__(application,packagename): for activity in application: #2.1MainActivityname if __is_main_activity__(activity): name = activity.attrib[namespace + 'name' ] main_activtiy = ActivityInfo(name) main_activtiy.__set_package_name__(packagename) if activity.find(namespace + 'permission' ) is not None : permission = activity.attrib[name + 'permission' ] main_activtiy.__set_permission__(permission) exported_activities.append(main_activtiy) else : #2.1exportedactivitiesexceptmainactivtiy exported = activity.attrib[namespace + 'exported' ] if exported: activity_name = activity.attrib[namespace + 'name' ] exported_activity = ActivityInfo(activity_name) exported_activity.__set_package_name__(packagename) permission = activity.attrib.get(namespace + 'permission' , None ) if permission is not None : #permission=activity.attrib[name+'permission'] #printpermission exported_activity.__set_permission__(permission) exported_activities.append(exported_activity) #saveresulttoxmlfile def __save_result_to_xml__(): tree = ET.ElementTree() root = ET.Element( 'data' ) tree._setroot(root) for activity in exported_activities: node = ET.Element( 'activity' ) node. set ( 'name' ,activity.__get_name__()) node. set ( 'packagename' ,activity.__get_package_name__()) root.append(node) if activity.__get_permission__() is not None : node. set ( 'permission' ,activity.__get_permission__()) tree.write( "output.xml" ) #theactivityisornotMainActivity def __is_main_activity__(activity): intent_filter = activity.find( "intent-filter" ) if intent_filter is not None : action = intent_filter.find( 'action' ) category = intent_filter.find( 'category' ) if action is not None and category is not None \ and action.attrib[namespace + "name" ] = = 'android.intent.action.MAIN' \ and category.attrib[namespace + "name" ] = = 'android.intent.category.LAUNCHER' : print 'thisismainactivity' return True return False #mainfun if __name__ = = "__main__" : __xml_analyze__() 3.ActivityInfo 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 class ActivityInfo: def __init__( self ,name): self .name = name self .package_name = '' self .permission = '' def __set_package_name__( self ,package_name): self .package_name = package_name def __get_package_name__( self ): return self .package_name def __set_name__( self ,name): self .name = name def __get_name__( self ): return self .name def __set_permission__( self ,permission): self .permission = permission def __get_permission__( self ): return self .permission 本文转自wauoen51CTO博客,原文链接:http://blog.51cto.com/7183397/1829345,如需转载请自行联系原作者

优秀的个人博客,低调大师

第二讲 python程序基本架构

输入-------------处理------------------------------输出 变量赋值算术运算(+、-、*、/) 打印输出 输入语句 逻辑运算(如:判断字符串大小) 写入文件(用户名:ip:时间) 算法处理 写入数据库 eg1: >>> x=12 --------| >>> y=13---------|----输入 >>> z=x+y-------------处理 >>> print (z)---------输出 25 eg2: >>> str1=raw_input("plz input a string:")----------输入 raw_put函数(参数是提示字符) plz input a string:www.baidu.com >>> print (str1) www.baidu.com >>> n=len(str1)------------------------------------处理 >>> print n----------------------------------------输出 13 备注: 1,raw_put输入语句函数,括号中的参数是提示字符(可以省略),读取用户输入内容并赋值给某个变量无论输入什么,读取的都是字符串string型的 2,len()统计字符串的长度,有多少个字符,括号中是变量名 本文转自Tenderrain 51CTO博客,原文链接:http://blog.51cto.com/tenderrain/1620642,如需转载请自行联系原作者

优秀的个人博客,低调大师

我的spark python 决策树实例

from numpy import array from pyspark.mllib.regression import LabeledPoint from pyspark.mllib.tree import DecisionTree, DecisionTreeModel from pyspark import SparkContext from pyspark.mllib.evaluation import BinaryClassificationMetrics sc = SparkContext(appName="PythonDecisionTreeClassificationExample") data = [ LabeledPoint(0.0, [0.0]), LabeledPoint(1.0, [1.0]), LabeledPoint(0.0, [-2.0]), LabeledPoint(0.0, [-1.0]), LabeledPoint(0.0, [-3.0]), LabeledPoint(1.0, [4.0]), LabeledPoint(1.0, [4.5]), LabeledPoint(1.0, [4.9]), LabeledPoint(1.0, [3.0]) ] all_data = sc.parallelize(data) (trainingData, testData) = all_data.randomSplit([0.8, 0.2]) # model = DecisionTree.trainClassifier(sc.parallelize(data), 2, {}) model = DecisionTree.trainClassifier(trainingData, numClasses=2, categoricalFeaturesInfo={}, impurity='gini', maxDepth=5, maxBins=32) print(model) print(model.toDebugString()) model.predict(array([1.0])) model.predict(array([0.0])) rdd = sc.parallelize([[1.0], [0.0]]) model.predict(rdd).collect() predictions = model.predict(testData.map(lambda x: x.features)) labelsAndPredictions = testData.map(lambda lp: lp.label).zip(predictions) predictionsAndLabels = predictions.zip(testData.map(lambda lp: lp.label)) metrics = BinaryClassificationMetrics(predictionsAndLabels) print "AUC=%f PR=%f" % (metrics.areaUnderROC, metrics.areaUnderPR) testErr = labelsAndPredictions.filter(lambda (v, p): v != p).count() / float(testData.count()) print('Test Error = ' + str(testErr)) print('Learned classification tree model:') print(model.toDebugString()) # Save and load model model.save(sc, "./myDecisionTreeClassificationModel") sameModel = DecisionTreeModel.load(sc, "./myDecisionTreeClassificationModel") 本文转自张昺华-sky博客园博客,原文链接:http://www.cnblogs.com/bonelee/p/7151341.html,如需转载请自行联系原作者

优秀的个人博客,低调大师

python spark 求解最大 最小 平均 中位数

rating_data_raw = sc.textFile("%s/ml-100k/u.data" % PATH) print rating_data_raw.first() num_ratings = rating_data_raw.count() print "Ratings: %d" % num_ratings # In[35]: rating_data = rating_data_raw.map(lambda line: line.split("\t")) ratings = rating_data.map(lambda fields: int(fields[2])) max_rating = ratings.reduce(lambda x, y: max(x, y)) min_rating = ratings.reduce(lambda x, y: min(x, y)) mean_rating = ratings.reduce(lambda x, y: x + y) / float(num_ratings) median_rating = np.median(ratings.collect()) ratings_per_user = num_ratings / num_users ratings_per_movie = num_ratings / num_movies print "Min rating: %d" % min_rating print "Max rating: %d" % max_rating print "Average rating: %2.2f" % mean_rating print "Median rating: %d" % median_rating print "Average # of ratings per user: %2.2f" % ratings_per_user print "Average # of ratings per movie: %2.2f" % ratings_per_movie # In[36]: # we can also use the stats function to get some similar information to the above ratings.stats() 上面是粗暴的做法 简单的做法: >>> all_data = sc.parallelize([1,2,3,4,5,6,7,8,100]) >>> all_data.mean() 15.11111111111111 >>> all_data.max() 100 >>> all_data.min() 1 >>> all_data.median() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'RDD' object has no attribute 'median' >>> all_data.stats() (count: 9, mean: 15.1111111111, stdev: 30.0903987804, max: 100.0, min: 1.0) 本文转自张昺华-sky博客园博客,原文链接:http://www.cnblogs.com/bonelee/p/7153889.html ,如需转载请自行联系原作者

优秀的个人博客,低调大师

python实现全角半角的相互转换

缘起 在自然语言处理过程中,全角、半角的的不一致会导致信息抽取不一致,因此需要统一。 转换说明 全角半角转换说明 有规律(不含空格): 全角字符unicode编码从65281~65374 (十六进制 0xFF01 ~ 0xFF5E) 半角字符unicode编码从33~126 (十六进制 0x21~ 0x7E) 特例: 空格比较特殊,全角为 12288(0x3000),半角为 32(0x20) 除空格外,全角/半角按unicode编码排序在顺序上是对应的(半角 + 0x7e= 全角),所以可以直接通过用+-法来处理非空格数据,对空格单独处理。 注: 1. 中文文字永远是全角,只有英文字母、数字键、符号键才有全角半角的概念,一个字母或数字占一个汉字的位置叫全角,占半个汉字的位置叫半角。 2. 引号在中英文、全半角情况下是不同的 参考代码 # -*- coding: cp936 -*- def strQ2B(ustring): """全角转半角""" rstring = "" for uchar in ustring: inside_code=ord(uchar) if inside_code == 12288: #全角空格直接转换 inside_code = 32 elif (inside_code >= 65281 and inside_code <= 65374): #全角字符(除空格)根据关系转化 inside_code -= 65248 rstring += unichr(inside_code) return rstring def strB2Q(ustring): """半角转全角""" rstring = "" for uchar in ustring: inside_code=ord(uchar) if inside_code == 32: #半角空格直接转化 inside_code = 12288 elif inside_code >= 32 and inside_code <= 126: #半角字符(除空格)根据关系转化 inside_code += 65248 rstring += unichr(inside_code) return rstring b = strQ2B("mn123abc博客园".decode('cp936')) print b c = strB2Q("mn123abc博客园".decode('cp936')) print c 执行结果 库函数说明 chr()函数用一个范围在range(256)内的(就是0~255)整数作参数,返回一个对应的字符。unichr()跟它一样,只不过返回的是Unicode字符。 ord()函数是chr()函数(对于8位的ASCII字符串)或unichr()函数(对于Unicode对象)的配对函数,它以一个字符(长度为1的字符串)作为参数,返回对应的ASCII数值,或者Unicode数值。 案例 本文转自jihite博客园博客,原文链接:http://www.cnblogs.com/kaituorensheng/p/3554571.html,如需转载请自行联系原作者

资源下载

更多资源
Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Spring

Spring

Spring框架(Spring Framework)是由Rod Johnson于2002年提出的开源Java企业级应用框架,旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念,提供核心容器、应用上下文、数据访问集成等模块,支持整合Hibernate、Struts等第三方框架,其适用范围不仅限于服务器端开发,绝大多数Java应用均可从中受益。

Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能,例如代码缩略图,Python的插件,代码段等。还可自定义键绑定,菜单和工具栏。Sublime Text 的主要功能包括:拼写检查,书签,完整的 Python API , Goto 功能,即时项目切换,多选择,多窗口等等。Sublime Text 是一个跨平台的编辑器,同时支持Windows、Linux、Mac OS X等操作系统。

WebStorm

WebStorm

WebStorm 是jetbrains公司旗下一款JavaScript 开发工具。目前已经被广大中国JS开发者誉为“Web前端开发神器”、“最强大的HTML5编辑器”、“最智能的JavaScript IDE”等。与IntelliJ IDEA同源,继承了IntelliJ IDEA强大的JS部分的功能。

用户登录
用户注册