hive.exec.parallel参数-低调大师

hive.exec.parallel参数

2017-09-05 512

hive.exec.parallel参数控制在同一个sql中的不同的job是否可以同时运行,默认为false.
下面是对于该参数的测试过程:

测试sql:
select r1.a
from (select t.a from sunwg_10 t join sunwg_10000000 s on t.a=s.b) r1 join (select s.b from sunwg_100000 t join sunwg_10 s on t.a=s.b) r2 on (r1.a=r2.b);

1,
Set hive.exec.parallel=false;
当参数为false的时候,三个job是顺序的执行

 
        hive> 
        set 
        hive.
        exec
        .parallel=
        false
        ;  
       
 
        hive> 
        select 
        r1.a  
       
 
            
        > 
        from 
        (
        select 
        t.a 
        from 
        sunwg_10 t 
        join 
        sunwg_10000000 s 
        on 
        t.a=s.b) r1 
        join 
        (
        select 
        s.b 
        from 
        sunwg_100000 t 
        join 
        sunwg_10 s 
        on 
        t.a=s.b) r2 
        on 
        (r1.a=r2.b);  
       

        Total MapReduce jobs = 3 
       
 
        Launching Job 1 
        out 
        of 
        3  
       
 
        Number 
        of 
        reduce tasks 
        not 
        specified. Estimated 
        from 
        input data 
        size
        : 1  
       
 
        In 
        order 
        to 
        change the average 
        load 
        for 
        a reducer (
        in 
        bytes):  
       
 
          
        set 
        hive.
        exec
        .reducers.bytes.per.reducer=<number>  
       
 
        In 
        order 
        to 
        limit the maximum number 
        of 
        reducers:  
       
 
          
        set 
        hive.
        exec
        .reducers.
        max
        =<number>  
       
 
        In 
        order 
        to 
        set 
        a constant number 
        of 
        reducers:  
       
 
          
        set 
        mapred.reduce.tasks=<number>  
       
 
        Cannot run job locally: Input 
        Size 
        (= 397778060) 
        is 
        larger than hive.
        exec
        .mode.
        local
        .auto.inputbytes.
        max 
        (= -1)  
       

        Starting Job = job_201208241319_2001905, Tracking URL = http://hdpjt:50030/jobdetails.jsp?jobid=job_201208241319_2001905 
       

        Kill Command = /dhwdata/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=hdpjt:9001 -kill job_201208241319_2001905 
       
 
        Hadoop job information 
        for 
        Stage-1: number 
        of 
        mappers: 7; number 
        of 
        reducers: 1  
       

        2012-09-07 17:55:40,854 Stage-1 map = 0%,  reduce = 0% 
       

        2012-09-07 17:55:55,663 Stage-1 map = 14%,  reduce = 0% 
       

        2012-09-07 17:56:00,506 Stage-1 map = 56%,  reduce = 0% 
       

        2012-09-07 17:56:10,254 Stage-1 map = 100%,  reduce = 0% 
       

        2012-09-07 17:56:19,871 Stage-1 map = 100%,  reduce = 29% 
       

        2012-09-07 17:56:30,000 Stage-1 map = 100%,  reduce = 75% 
       

        2012-09-07 17:56:34,799 Stage-1 map = 100%,  reduce = 100% 
       

        Ended Job = job_201208241319_2001905 
       
 
        Launching Job 2 
        out 
        of 
        3  
       
 
        Number 
        of 
        reduce tasks 
        not 
        specified. Estimated 
        from 
        input data 
        size
        : 1  
       
 
        In 
        order 
        to 
        change the average 
        load 
        for 
        a reducer (
        in 
        bytes):  
       
 
          
        set 
        hive.
        exec
        .reducers.bytes.per.reducer=<number>  
       
 
        In 
        order 
        to 
        limit the maximum number 
        of 
        reducers:  
       
 
          
        set 
        hive.
        exec
        .reducers.
        max
        =<number>  
       
 
        In 
        order 
        to 
        set 
        a constant number 
        of 
        reducers:  
       
 
          
        set 
        mapred.reduce.tasks=<number>  
       
 
        Cannot run job locally: Input 
        Size 
        (= 3578060) 
        is 
        larger than hive.
        exec
        .mode.
        local
        .auto.inputbytes.
        max 
        (= -1)  
       

        Starting Job = job_201208241319_2002054, Tracking URL = http://hdpjt:50030/jobdetails.jsp?jobid=job_201208241319_2002054 
       

        Kill Command = /dhwdata/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=hdpjt:9001 -kill job_201208241319_2002054 
       
 
        Hadoop job information 
        for 
        Stage-4: number 
        of 
        mappers: 2; number 
        of 
        reducers: 1  
       

        2012-09-07 17:56:43,343 Stage-4 map = 0%,  reduce = 0% 
       

        2012-09-07 17:56:48,124 Stage-4 map = 50%,  reduce = 0% 
       

        2012-09-07 17:56:55,816 Stage-4 map = 100%,  reduce = 0% 
       

        Ended Job = job_201208241319_2002054 
       
 
        Launching Job 3 
        out 
        of 
        3  
       
 
        Number 
        of 
        reduce tasks 
        not 
        specified. Estimated 
        from 
        input data 
        size
        : 1  
       
 
        In 
        order 
        to 
        change the average 
        load 
        for 
        a reducer (
        in 
        bytes):  
       
 
          
        set 
        hive.
        exec
        .reducers.bytes.per.reducer=<number>  
       
 
        In 
        order 
        to 
        limit the maximum number 
        of 
        reducers:  
       
 
          
        set 
        hive.
        exec
        .reducers.
        max
        =<number>  
       
 
        In 
        order 
        to 
        set 
        a constant number 
        of 
        reducers:  
       
 
          
        set 
        mapred.reduce.tasks=<number>  
       
 
        Cannot run job locally: Input 
        Size 
        (= 596) 
        is 
        larger than hive.
        exec
        .mode.
        local
        .auto.inputbytes.
        max 
        (= -1)  
       

        Starting Job = job_201208241319_2002120, Tracking URL = http://hdpjt:50030/jobdetails.jsp?jobid=job_201208241319_2002120 
       

        Kill Command = /dhwdata/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=hdpjt:9001 -kill job_201208241319_2002120 
       
 
        Hadoop job information 
        for 
        Stage-2: number 
        of 
        mappers: 2; number 
        of 
        reducers: 1  
       

        2012-09-07 17:57:12,641 Stage-2 map = 0%,  reduce = 0% 
       

        2012-09-07 17:57:19,571 Stage-2 map = 50%,  reduce = 0% 
       

        2012-09-07 17:57:25,199 Stage-2 map = 100%,  reduce = 0% 
       

        2012-09-07 17:57:29,210 Stage-2 map = 100%,  reduce = 100% 
       

        Ended Job = job_201208241319_2002120 
       

        OK 
       

        abcdefghijk_0 
       

        abcdefghijk_1 
       

        abcdefghijk_2 
       

        abcdefghijk_3 
       

        abcdefghijk_4 
       

        abcdefghijk_5 
       

        abcdefghijk_6 
       

        abcdefghijk_7 
       

        abcdefghijk_8 
       

        abcdefghijk_9 
       
 
        Time 
        taken: 135.944 seconds 
       

2,
但是可以看出来其实两个子查询中的sql并无关系,可以并行的跑

 
        hive> 
        set 
        hive.
        exec
        .parallel=
        true
        ;  
       
 
        hive> 
        select 
        r1.a  
       
 
            
        > 
        from 
        (
        select 
        t.a 
        from 
        sunwg_10 t 
        join 
        sunwg_10000000 s 
        on 
        t.a=s.b) r1 
        join 
        (
        select 
        s.b 
        from 
        sunwg_100000 t 
        join 
        sunwg_10 s 
        on 
        t.a=s.b) r2 
        on 
        (r1.a=r2.b);  
       

        Total MapReduce jobs = 3 
       
 
        Launching Job 1 
        out 
        of 
        3  
       
 
        Launching Job 2 
        out 
        of 
        3  
       
 
        Number 
        of 
        reduce tasks 
        not 
        specified. Estimated 
        from 
        input data 
        size
        : 1  
       
 
        In 
        order 
        to 
        change the average 
        load 
        for 
        a reducer (
        in 
        bytes):  
       
 
          
        set 
        hive.
        exec
        .reducers.bytes.per.reducer=<number>  
       
 
        In 
        order 
        to 
        limit the maximum number 
        of 
        reducers:  
       
 
          
        set 
        hive.
        exec
        .reducers.
        max
        =<number>  
       
 
        In 
        order 
        to 
        set 
        a constant number 
        of 
        reducers:  
       
 
          
        set 
        mapred.reduce.tasks=<number>  
       
 
        Cannot run job locally: Input 
        Size 
        (= 397778060) 
        is 
        larger than hive.
        exec
        .mode.
        local
        .auto.inputbytes.
        max 
        (= -1)  
       
 
        Number 
        of 
        reduce tasks 
        not 
        specified. Estimated 
        from 
        input data 
        size
        : 1  
       
 
        In 
        order 
        to 
        change the average 
        load 
        for 
        a reducer (
        in 
        bytes):  
       
 
          
        set 
        hive.
        exec
        .reducers.bytes.per.reducer=<number>  
       
 
        In 
        order 
        to 
        limit the maximum number 
        of 
        reducers:  
       
 
          
        set 
        hive.
        exec
        .reducers.
        max
        =<number>  
       
 
        In 
        order 
        to 
        set 
        a constant number 
        of 
        reducers:  
       
 
          
        set 
        mapred.reduce.tasks=<number>  
       
 
        Cannot run job locally: Input 
        Size 
        (= 3578060) 
        is 
        larger than hive.
        exec
        .mode.
        local
        .auto.inputbytes.
        max 
        (= -1)  
       

        Starting Job = job_201208241319_2001452, Tracking URL = http://hdpjt:50030/jobdetails.jsp?jobid=job_201208241319_2001452 
       

        Kill Command = /dhwdata/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=hdpjt:9001 -kill job_201208241319_2001452 
       

        Starting Job = job_201208241319_2001453, Tracking URL = http://hdpjt:50030/jobdetails.jsp?jobid=job_201208241319_2001453 
       

        Kill Command = /dhwdata/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=hdpjt:9001 -kill job_201208241319_2001453 
       
 
        Hadoop job information 
        for 
        Stage-4: number 
        of 
        mappers: 2; number 
        of 
        reducers: 1  
       
 
        Hadoop job information 
        for 
        Stage-1: number 
        of 
        mappers: 7; number 
        of 
        reducers: 1  
       

        2012-09-07 17:52:10,558 Stage-4 map = 0%,  reduce = 0% 
       

        2012-09-07 17:52:10,588 Stage-1 map = 0%,  reduce = 0% 
       

        2012-09-07 17:52:22,827 Stage-1 map = 14%,  reduce = 0% 
       

        2012-09-07 17:52:22,880 Stage-4 map = 100%,  reduce = 0% 
       

        2012-09-07 17:52:27,678 Stage-1 map = 22%,  reduce = 0% 
       

        2012-09-07 17:52:28,701 Stage-1 map = 36%,  reduce = 0% 
       

        2012-09-07 17:52:31,137 Stage-1 map = 93%,  reduce = 0% 
       

        2012-09-07 17:52:33,551 Stage-1 map = 100%,  reduce = 0% 
       

        2012-09-07 17:52:36,427 Stage-4 map = 100%,  reduce = 100% 
       

        Ended Job = job_201208241319_2001453 
       

        2012-09-07 17:52:42,883 Stage-1 map = 100%,  reduce = 33% 
       

        2012-09-07 17:52:45,431 Stage-1 map = 100%,  reduce = 70% 
       

        2012-09-07 17:52:47,526 Stage-1 map = 100%,  reduce = 76% 
       

        2012-09-07 17:52:51,829 Stage-1 map = 100%,  reduce = 84% 
       

        Ended Job = job_201208241319_2001452 
       
 
        Launching Job 3 
        out 
        of 
        3  
       
 
        Number 
        of 
        reduce tasks 
        not 
        specified. Estimated 
        from 
        input data 
        size
        : 1  
       
 
        In 
        order 
        to 
        change the average 
        load 
        for 
        a reducer (
        in 
        bytes):  
       
 
          
        set 
        hive.
        exec
        .reducers.bytes.per.reducer=<number>  
       
 
        In 
        order 
        to 
        limit the maximum number 
        of 
        reducers:  
       
 
          
        set 
        hive.
        exec
        .reducers.
        max
        =<number>  
       
 
        In 
        order 
        to 
        set 
        a constant number 
        of 
        reducers:  
       
 
          
        set 
        mapred.reduce.tasks=<number>  
       
 
        Cannot run job locally: Input 
        Size 
        (= 596) 
        is 
        larger than hive.
        exec
        .mode.
        local
        .auto.inputbytes.
        max 
        (= -1)  
       

        Starting Job = job_201208241319_2001621, Tracking URL = http://hdpjt:50030/jobdetails.jsp?jobid=job_201208241319_2001621 
       

        Kill Command = /dhwdata/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=hdpjt:9001 -kill job_201208241319_2001621 
       
 
        Hadoop job information 
        for 
        Stage-2: number 
        of 
        mappers: 2; number 
        of 
        reducers: 1  
       

        2012-09-07 17:53:07,081 Stage-2 map = 0%,  reduce = 0% 
       

        2012-09-07 17:53:10,351 Stage-2 map = 50%,  reduce = 0% 
       

        2012-09-07 17:53:11,380 Stage-2 map = 100%,  reduce = 0% 
       

        2012-09-07 17:53:18,132 Stage-2 map = 100%,  reduce = 100% 
       

        Ended Job = job_201208241319_2001621 
       

        OK 
       

        abcdefghijk_0 
       

        abcdefghijk_1 
       

        abcdefghijk_2 
       

        abcdefghijk_3 
       

        abcdefghijk_4 
       

        abcdefghijk_5 
       

        abcdefghijk_6 
       

        abcdefghijk_7 
       

        abcdefghijk_8 
       

        abcdefghijk_9 
       
 
        Time 
        taken: 108.301 seconds 
       

总结:
在资源充足的时候hive.exec.parallel会让那些存在并发job的sql运行得更快,但同时消耗更多的资源
可以评估下hive.exec.parallel对我们的刷新任务是否有帮助.

转自：http://www.oratea.net/?p=1377

本文转自茄子_2008博客园博客，原文链接：http://www.cnblogs.com/xd502djj/archive/2013/05/08/3067699.html，如需转载请自行联系原作者。

微信关注我们

原文链接：https://yq.aliyun.com/articles/357773

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

reduce个数究竟和哪些因素有关

reduce的数目究竟和哪些因素有关 1、我们知道map的数量和文件数、文件大小、块大小、以及split大小有关，而reduce的数量跟哪些因素有关呢？设置mapred.tasktracker.reduce.tasks.maximum的大小能够决定单个tasktracker一次性启动reduce的数目，可是不能决定总的reduce数目。 conf.setNumReduceTasks(4);JobConf对象的这种方法能够用来设定总的reduce的数目，看下Job Counters的统计： Job Counters Data-local map tasks=2 Total time spent by all maps waiting after reserving slots (ms)=0 Total time spent by all reduces waiting after reserving slots (ms)=0 SLOTS_MILLIS_MAPS=10695 SLOTS_MILLIS_REDUCES=29502 Launched map tasks=2 Laun...

2017-09-06

539

Hadoop MapReduce 官方教程 -- WordCount示例

Hadoop MapReduce 官方教程 -- WordCount示例： http://hadoop.apache.org/docs/r1.0.4/cn/mapred_tutorial.html#%E4%BE%8B%E5%AD%90%EF%BC%9AWordCount+v1.0

2017-09-07

630

发表评论

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。