[spark][python]Spark map 处理-低调大师

[spark][python]Spark map 处理

2017-12-18 651

map 就是对一个RDD的各个元素都施加处理，得到一个新的RDD 的过程

[training@localhost ~]$ cat names.txt
Year,First Name,County,Sex,Count
2012,DOMINIC,CAYUGA,M,6
2012,ADDISON,ONONDAGA,F,14
2012,ADDISON,ONONDAGA,F,14
2012,JULIA,ONONDAGA,F,15
[training@localhost ~]$ hdfs dfs -put names.txt
[training@localhost ~]$ hdfs dfs -cat names.txt
Year,First Name,County,Sex,Count
2012,DOMINIC,CAYUGA,M,6
2012,ADDISON,ONONDAGA,F,14
2012,ADDISON,ONONDAGA,F,14
2012,JULIA,ONONDAGA,F,15
[training@localhost ~]$

In [98]: t_names = sc.textFile("names.txt")
17/09/24 06:24:22 INFO storage.MemoryStore: Block broadcast_27 stored as values in memory (estimated size 230.5 KB, free 2.3 MB)
17/09/24 06:24:23 INFO storage.MemoryStore: Block broadcast_27_piece0 stored as bytes in memory (estimated size 21.5 KB, free 2.3 MB)
17/09/24 06:24:23 INFO storage.BlockManagerInfo: Added broadcast_27_piece0 in memory on localhost:33950 (size: 21.5 KB, free: 208.6 MB)
17/09/24 06:24:23 INFO spark.SparkContext: Created broadcast 27 from textFile at NativeMethodAccessorImpl.java:-2

In [99]: rows=t_names.map(lambda line: line.split(","))

In [100]: rows.take(1)

17/09/24 06:25:23 INFO mapred.FileInputFormat: Total input paths to process : 1
17/09/24 06:25:23 INFO spark.SparkContext: Starting job: runJob at PythonRDD.scala:393
17/09/24 06:25:23 INFO scheduler.DAGScheduler: Got job 15 (runJob at PythonRDD.scala:393) with 1 output partitions
17/09/24 06:25:23 INFO scheduler.DAGScheduler: Final stage: ResultStage 15 (runJob at PythonRDD.scala:393)
17/09/24 06:25:23 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/09/24 06:25:23 INFO scheduler.DAGScheduler: Missing parents: List()
17/09/24 06:25:23 INFO scheduler.DAGScheduler: Submitting ResultStage 15 (PythonRDD[46] at RDD at PythonRDD.scala:43), which has no missing parents
17/09/24 06:25:23 INFO storage.MemoryStore: Block broadcast_28 stored as values in memory (estimated size 5.2 KB, free 2.3 MB)
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_26_piece0 on localhost:33950 in memory (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 8
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_18_piece0 on localhost:33950 in memory (size: 3.7 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO storage.MemoryStore: Block broadcast_28_piece0 stored as bytes in memory (estimated size 3.3 KB, free 2.3 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 9
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_19_piece0 on localhost:33950 in memory (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 10
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Added broadcast_28_piece0 in memory on localhost:33950 (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.SparkContext: Created broadcast 28 from broadcast at DAGScheduler.scala:1006
17/09/24 06:25:24 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 15 (PythonRDD[46] at RDD at PythonRDD.scala:43)
17/09/24 06:25:24 INFO scheduler.TaskSchedulerImpl: Adding task set 15.0 with 1 tasks
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_20_piece0 on localhost:33950 in memory (size: 3.7 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 11
17/09/24 06:25:24 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 15.0 (TID 15, localhost, partition 0,PROCESS_LOCAL, 2147 bytes)
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_21_piece0 on localhost:33950 in memory (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 12
17/09/24 06:25:24 INFO executor.Executor: Running task 0.0 in stage 15.0 (TID 15)
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_22_piece0 on localhost:33950 in memory (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 13
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_23_piece0 on localhost:33950 in memory (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 14
17/09/24 06:25:24 INFO rdd.HadoopRDD: Input split: hdfs://localhost:8020/user/training/names.txt:0+136
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_24_piece0 on localhost:33950 in memory (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 15
17/09/24 06:25:24 INFO storage.BlockManagerInfo: Removed broadcast_25_piece0 on localhost:33950 in memory (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:24 INFO spark.ContextCleaner: Cleaned accumulator 16
17/09/24 06:25:24 INFO python.PythonRunner: Times: total = 78, boot = 49, init = 25, finish = 4
17/09/24 06:25:24 INFO executor.Executor: Finished task 0.0 in stage 15.0 (TID 15). 2203 bytes result sent to driver
17/09/24 06:25:24 INFO scheduler.DAGScheduler: ResultStage 15 (runJob at PythonRDD.scala:393) finished in 0.438 s
17/09/24 06:25:24 INFO scheduler.DAGScheduler: Job 15 finished: runJob at PythonRDD.scala:393, took 1.160085 s
17/09/24 06:25:24 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 15.0 (TID 15) in 429 ms on localhost (1/1)
17/09/24 06:25:24 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool
Out[100]: [[u'Year', u'First Name', u'County', u'Sex', u'Count']]

In [101]: rows.take(2)
17/09/24 06:25:29 INFO spark.SparkContext: Starting job: runJob at PythonRDD.scala:393
17/09/24 06:25:29 INFO scheduler.DAGScheduler: Got job 16 (runJob at PythonRDD.scala:393) with 1 output partitions
17/09/24 06:25:29 INFO scheduler.DAGScheduler: Final stage: ResultStage 16 (runJob at PythonRDD.scala:393)
17/09/24 06:25:29 INFO scheduler.DAGScheduler: Parents of final stage: List()
17/09/24 06:25:29 INFO scheduler.DAGScheduler: Missing parents: List()
17/09/24 06:25:29 INFO scheduler.DAGScheduler: Submitting ResultStage 16 (PythonRDD[47] at RDD at PythonRDD.scala:43), which has no missing parents
17/09/24 06:25:29 INFO storage.MemoryStore: Block broadcast_29 stored as values in memory (estimated size 5.2 KB, free 2.2 MB)
17/09/24 06:25:29 INFO storage.MemoryStore: Block broadcast_29_piece0 stored as bytes in memory (estimated size 3.3 KB, free 2.2 MB)
17/09/24 06:25:29 INFO storage.BlockManagerInfo: Added broadcast_29_piece0 in memory on localhost:33950 (size: 3.3 KB, free: 208.6 MB)
17/09/24 06:25:29 INFO spark.SparkContext: Created broadcast 29 from broadcast at DAGScheduler.scala:1006
17/09/24 06:25:29 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 16 (PythonRDD[47] at RDD at PythonRDD.scala:43)
17/09/24 06:25:29 INFO scheduler.TaskSchedulerImpl: Adding task set 16.0 with 1 tasks
17/09/24 06:25:29 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 16.0 (TID 16, localhost, partition 0,PROCESS_LOCAL, 2147 bytes)
17/09/24 06:25:29 INFO executor.Executor: Running task 0.0 in stage 16.0 (TID 16)
17/09/24 06:25:29 INFO rdd.HadoopRDD: Input split: hdfs://localhost:8020/user/training/names.txt:0+136
17/09/24 06:25:29 INFO python.PythonRunner: Times: total = 71, boot = 25, init = 45, finish = 1
17/09/24 06:25:29 INFO executor.Executor: Finished task 0.0 in stage 16.0 (TID 16). 2267 bytes result sent to driver
17/09/24 06:25:30 INFO scheduler.DAGScheduler: ResultStage 16 (runJob at PythonRDD.scala:393) finished in 0.196 s
17/09/24 06:25:30 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 16.0 (TID 16) in 202 ms on localhost (1/1)
17/09/24 06:25:30 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool
17/09/24 06:25:30 INFO scheduler.DAGScheduler: Job 16 finished: runJob at PythonRDD.scala:393, took 0.408908 s
Out[101]:
[[u'Year', u'First Name', u'County', u'Sex', u'Count'],
[u'2012', u'DOMINIC', u'CAYUGA', u'M', u'6']]

In [102]:

来自：

https://www.supergloo.com/fieldnotes/apache-spark-transformations-python-examples/

本文转自健哥的数据花园博客园博客，原文链接：http://www.cnblogs.com/gaojian/p/7588619.html，如需转载请自行联系原作者

微信关注我们

原文链接：https://yq.aliyun.com/articles/355522

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

[Spark][Python]对HDFS 上的文件，采用绝对路径，来读取获得 RDD

对HDFS 上的文件，采用绝对路径，来读取获得 RDD: In [102]: mydata=sc.textFile("file:/home/training/test.txt") 17/09/24 06:31:04 INFO storage.MemoryStore: Block broadcast_30 stored as values in memory (estimated size 230.5 KB, free 2.4 MB) 17/09/24 06:31:04 INFO storage.MemoryStore: Block broadcast_30_piece0 stored as bytes in memory (estimated size 21.5 KB, free 2.5 MB) 17/09/24 06:31:04 INFO storage.BlockManagerInfo: Added broadcast_30_piece0 in memory on localhost:33950 (size: 21.5 KB, free: 208.6 MB) 17/09/24 06...

2017-12-19

532

作为全世界最知名的数据挖掘、机器学习竞赛平台，Kaggle早已成为数据玩家在学习了基础机器学习之后一试身手的练兵场。那么，参加Kaggle比赛到底是怎样一种体验呢？Kaggle比赛的爱好者们不计其数，很显然这些比赛不会是简单枯燥的模型调参。更进一步地问，Kaggle比赛的优胜者们又是如何取得优异的成绩的呢？优质的算法对大多数Kaggle竞赛来说显然不是制胜法宝——SVM、随机森林、神经网络等的应用对多数参赛者都不是难事，而改进甚至发明新算法似乎也更属于学术研究的范畴。今天，文摘菌将为大家介绍一位资深数据科学家Gabriel参加Kaggle的Outbrain点击预测比赛的经历。相信大家在读完他的比赛经历后，会对Kaggle比赛有更深一步的了解，也能一窥大神在Kaggle比赛中取得优异成绩背后的秘密。 2017年1月18日的午夜，Out

2017-12-19

624

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

Oracle

Oracle Database，又名Oracle RDBMS，或简称Oracle。是甲骨文公司的一款关系数据库管理系统。它是在数据库领域一直处于领先地位的产品。可以说Oracle数据库系统是目前世界上流行的关系数据库管理系统，系统可移植性好、使用方便、功能强，适用于各类大、中、小、微机环境。它是一种高效率、可靠性好的、适应高吞吐量的数据库方案。

Eclipse

Eclipse 是一个开放源代码的、基于Java的可扩展开发平台。就其本身而言，它只是一个框架和一组服务，用于通过插件组件构建开发环境。幸运的是，Eclipse 附带了一个标准的插件集，包括Java开发工具（Java Development Kit，JDK）。