首页 文章 精选 留言 我的

精选列表

搜索[数仓建模],共10009篇文章
优秀的个人博客,低调大师

面对锁等待难题,数仓如何实现问题的秒级定位和分析

摘要:GaussDB(DWS)提供了两个集群级别的视图快速识别和查询锁等待和分布式死锁信息,可实现此类问题的秒级问题的定位和分析。 本文分享自华为云社区《GaussDB(DWS)运维 -- 一键式锁等待和分布式死锁检测》,作者:譡里个檔。 锁是GaussDB(DWS)实现并发管理的关键要素,GaussDB(DWS)锁类别有表级锁、分区级锁(和表级锁一致)、事务锁、咨询锁等,当前业务最常用的是表级锁、分区级锁(和表级锁一致)、事务锁。不同的SQL语句执行时需要申请并持有对应的锁,当这些锁资源存在互斥时,对应的业务SQL就会产生等待;这种等待会产生下面几种后果: 持锁的一方释放锁(一般对应的动作为持锁的事物提交),等待锁的一方申请到锁,然后继续执行 持锁的一方事物长时间未提交,等待锁的一方因为锁等待超时导致作业报错 A实例上持锁事物和申请锁的事物在B实例上角色互换,产生分布式死锁(具体见下文介绍)。这种场景下需要首先达到锁等待超时的事物报错回滚时释放锁资源,然后另外一个事物申请到才能正常进行 从上述的描述可以看到,锁等待特别是分布式死锁对业务影响很大,轻则产生等待导致业务性能抖动和下降,甚至业务报错。GaussDB(DWS)提供了两个集群级别的视图快速识别和查询锁等待和分布式死锁信息,可实现此类问题的秒级定位和分析。 1)锁等待检测视图pgxc_lock_conflicts 【功能】查询当前库里面不同节点上的锁等待信息 【解析】执行如下查询结果 postgres=# SELECT * FROM pgxc_lock_conflicts ORDER BY nodename,dbname,locktype,nspname,relname,partname; locktype | nodename | dbname | nspname | relname | partname | page | tuple | transactionid | username | gxid | xactstart | queryid | query | pid | mode | granted -----------+----------+----------+---------+-----------------------+----------+------+-------+---------------+-----------+----------+-------------------------------+--------------------+----------------------------------------------------------+-----------------+---------------------+--------- partition | cn_5001 | postgres | public | table_partition_num_3 | p1 | | | | dfm | 24097147 | 2022-02-17 17:56:03.113194+08 | 104145741383084190 | alter table table_partition_num_3 truncate partition p1; | 140160505136896 | AccessExclusiveLock | f partition | cn_5001 | postgres | public | table_partition_num_3 | p1 | | | | dfm | 24102679 | 2022-02-17 18:41:36.580348+08 | 0 | alter table table_partition_num_3 truncate partition p1; | 140160568055552 | AccessExclusiveLock | t relation | cn_5002 | postgres | public | xxx | | | | | dfm | 24102679 | 2022-02-17 18:41:36.580348+08 | 175921860444402398 | truncate xxx; | 140418767369984 | AccessShareLock | f relation | cn_5002 | postgres | public | xxx | | | | | dfm | 24097147 | 2022-02-17 17:56:03.113194+08 | 0 | truncate xxx; | 140420489144064 | AccessExclusiveLock | t (4 rows) 如上的SQL显示 在节点cn_5001的postgres里面的表public.table_partition_num_3的分区p1上存在分区级别(partition)的锁冲突。在当前的锁冲突中线程140160568055552持有锁(mode = true),锁级别是AccessExclusiveLock,执行语句为alter table table_partition_num_3 truncate partition p1。线程140160568055552在等待(mode = false)AccessExclusiveLock锁,等待锁的语句也是alter table table_partition_num_3 truncate partition p1。 在节点cn_5002的postgres里面的表http://public.xxx上存在表级别(relation)的锁冲突。线程140420489144064持有锁AccessExclusiveLock(mode = true),线程140418767369984在等待(mode = false)AccessShareLock锁 2)分布式锁等待检测视图pgxc_deadlock 【功能】查询当前库里面不同节点上的分布式死锁信息 【解析】执行如下查询结果 postgres=# SELECT * FROM pgxc_deadlock ORDER BY nodename,dbname,locktype,nspname,relname,partname; locktype | nodename | dbname | nspname | relname | partname | page | tuple | transactionid | waitusername | waitgxid | waitxactstart | waitqueryid | waitquery | waitpid | waitmode | holdusername | holdgxid | holdxactstart | holdqueryid | holdquery | holdpid | holdmode ----------+----------+----------+---------+---------+----------+------+-------+---------------+--------------+----------+-------------------------------+--------------------+-----------------------------------------------------+-----------------+-----------------+--------------+----------+-------------------------------+-------------+--------------+-----------------+--------------------- relation | cn_5001 | postgres | public | t2 | | | | | j00565968 | 24112406 | 2022-02-17 20:01:57.421532+08 | 104145741383110084 | EXECUTE DIRECT ON(dn_6003_6004) 'SELECT * FROM t2'; | 140160505136896 | AccessShareLock | j00565968 | 24112465 | 2022-02-17 20:02:24.220656+08 | 0 | TRUNCATE t2; | 140160421234432 | AccessExclusiveLock relation | cn_5002 | postgres | public | t1 | | | | | j00565968 | 24112465 | 2022-02-17 20:02:24.220656+08 | 175921860444446866 | EXECUTE DIRECT ON(dn_6001_6002) 'SELECT * FROM t1'; | 140418784151296 | AccessShareLock | j00565968 | 24112406 | 2022-02-17 20:01:57.421532+08 | 0 | TRUNCATE t1; | 140421763163904 | AccessExclusiveLock (2 rows) 如上的SQL显示,在postgres库里面 节点cn_5001上 事务24112465通过线程140160421234432持有表public.t2的AccessExclusiveLock锁 事务24112406通过线程140160505136896在等待申请表public.t2的AccessShareLock锁 节点cn_5002上 事务24112465通过线程140418784151296在等待申请表public.t1的AccessShareLock锁 事物24112406通过线程140421763163904持有表public.t1的AccessExclusiveLock锁 如果我们把资源的持有情况按照持有到申请定义一个防线的话,可以形成如下表格 从上述可以看出,事务24112465在节点cn_5001持有表public.t2的AccessExclusiveLock锁,等待申请申请表public.t1的AccessShareLock锁;事务24112406在节点cn_5002上持有表public.t1的AccessExclusiveLock锁,等待申请申请表public.t2的AccessShareLock锁;事务24112406和事务24112465只有等待彼此提交才能申请到锁资源,让自己继续执行,这种在多个实例上的分布式等待关系形成了一个环状,我们称这种现象为分布式死锁。 3) 锁等待和分布式死锁的区别 对于分布式死锁,只能一个事务因为锁等待(参数lockwait_timeout)超时回滚的时候,另外一个事务才能进行下去;或者人工干预kill或者cancel其中一个事务,让另外一个事务进行下去。 对于没有分布式死锁的锁等待,这种一般不需要人工干涉,等待持锁事务正常执行完成之后另外一个事务就可以正常执行;但是如果事务持锁时间超过锁等待超时参数(参数lockwait_timeout),等待锁的事务会因为锁等待超时失败。 点击关注,第一时间了解华为云新鲜技术~

优秀的个人博客,低调大师

数仓调优实践丨多次关联发散导致数据爆炸案例分析改写

本文分享自华为云社区《GaussDB(DWS)性能调优:求字段全体值中大于本行值的最小值——多次关联发散导致数据爆炸案例分析改写》,作者: Zawami 。 1、【问题描述】 语句中存在同一个表多次自关联,且均为发散关联,数据爆炸导致性能瓶颈。 2、【原始SQL】 explain verbose WITH TMP AS ( SELECT WH_ID , (IFNULL(SUBSTR(THE_DATE,1,10),'1900-01-01') || ' ' || STOP_TIME)::TIMESTAMP AS STOP_TIME , (IFNULL(SUBSTR(THE_DATE,1,10),'1900-01-01') || ' ' || '23:59:59')::TIMESTAMP AS MAX_ASD FROM DMISC.DM_DIM_CBG_WH_HOLIDAY_D WHERE IS_OPEN = 'Y' AND STOP_TIME IS NOT NULL ) SELECT T1.WH_ID , T1.THE_DATE , T1.IS_OPEN , MIN(T2.STOP_TIME) AS STOP_TIME , MIN(T2.MAX_ASD) AS TODAY_MAX_ASD , MIN(T3.MAX_ASD) AS NEXT_MAX_ASD FROM (SELECT WH_ID , THE_DATE , IS_OPEN , (IFNULL(SUBSTR(THE_DATE,1,10),'1900-01-01') || ' ' || STOP_TIME)::TIMESTAMP AS STOP_TIME FROM DMISC.DM_DIM_CBG_WH_HOLIDAY_D ) T1 LEFT JOIN TMP T2 ON T1.WH_ID = T2.WH_ID AND T1.THE_DATE < T2.STOP_TIME LEFT JOIN TMP T3 ON T1.WH_ID = T3.WH_ID AND ADDDATE(T1.THE_DATE,1) < T3.STOP_TIME GROUP BY T1.WH_ID, T1.THE_DATE, T1.IS_OPEN; 从SQL中不难看出,物理表HOLIDAY_D使用WH_ID为关联键,并使用其它字段做不等值关联。 3、【性能分析】 QUERY PLAN | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| id | operation | E-rows | E-distinct | E-memory | E-width | E-costs | ----+----------------------------------------------------------------------------------+---------------+------------+---------------+---------+----------------- | 1 | -> Row Adapter | 51584 | | | 67 | 377559930171.36 | 2 | -> Vector Streaming (type: GATHER) | 51584 | | | 67 | 377559930171.36 | 3 | -> Vector Hash Aggregate | 51584 | | 16MB | 67 | 377559929546.36 | 4 | -> Vector CTE Append(5, 7) | 5699739636332 | | 1MB | 43 | 292063834485.54 | 5 | -> Vector Streaming(type: BROADCAST) | 757752 | | 2MB | 22 | 1474.87 | 6 | -> CStore Scan on dmisc.dm_dim_cbg_wh_holiday_d [5, CTE tmp(1)] | 757752 | | 1MB | 22 | 1474.87 | 7 | -> Vector Hash Left Join (8, 11) | 5699739636332 | | 107MB(6863MB) | 43 | 292063833010.67 | 8 | -> Vector Hash Right Join (9, 10) | 542231841 | 50 | 16MB | 27 | 22365789.31 | 9 | -> Vector CTE Scan on tmp(1) t3 | 31573 | 50 | 1MB | 48 | 15155.04 | 10 | -> CStore Scan on dmisc.dm_dim_cbg_wh_holiday_d | 51584 | 50 | 1MB | 19 | 556.58 | 11 | -> Vector CTE Scan on tmp(1) t2 | 31573 | 50 | 1MB | 48 | 15155.04 | 由于SQL非常慢,难以打出performance计划,我们先看verbose计划。从计划中我们看到,经过两次的关联发散,估计数据量达到了5万亿行;因为hash join根据WH_ID列进行关联,实际不会有这么多。所以调优的思路就是取消一些发散,让中间结果集行数变少。 4、【改写SQL】 分析SQL,可知发散是为了寻找所有STOP_TIME中大于本行THE_DATE的最小值。像这种每行都需要用到本行数据和所有数据的逻辑,或许可以使用窗口函数进行编写;但囿于笔者能力,先提供单次自关联的方法。 SQL改写如下: explain performance WITH TMP AS ( SELECT WH_ID , (IFNULL(SUBSTR(THE_DATE,1,10),'1900-01-01') || ' ' || STOP_TIME)::TIMESTAMP AS STOP_TIME , (IFNULL(SUBSTR(THE_DATE,1,10),'1900-01-01') || ' ' || '23:59:59')::TIMESTAMP AS MAX_ASD FROM DMISC.DM_DIM_CBG_WH_HOLIDAY_D WHERE IS_OPEN = 'Y' AND STOP_TIME IS NOT NULL ) SELECT T1.WH_ID , T1.THE_DATE , T1.IS_OPEN , MIN(CASE WHEN T1.THE_DATE < T2.STOP_TIME THEN STOP_TIME ELSE NULL END) AS STOP_TIME , MIN(CASE WHEN T1.THE_DATE < T2.STOP_TIME THEN T2.MAX_ASD ELSE NULL END) AS TODAY_MAX_ASD , MIN(CASE WHEN ADDDATE(T1.THE_DATE, 1) < T2.STOP_TIME THEN T2.MAX_ASD ELSE NULL END) AS NEXT_MAX_ASD FROM (SELECT DISTINCT WH_ID , THE_DATE , IS_OPEN FROM DMISC.DM_DIM_CBG_WH_HOLIDAY_D ) T1 LEFT JOIN TMP T2 ON T1.WH_ID = T2.WH_ID GROUP BY T1.WH_ID , T1.THE_DATE , T1.IS_OPEN ; 经过改写,取消了一次自关联,SQL的中间结果集变小。在关联后,通过条件聚合来得到需要的值。 id | operation | A-time | A-rows | E-rows | E-distinct | Peak Memory | E-memory | A-width | E-width | E-costs ----+-----------------------------------------------------------------+----------------------+----------+--------+------------+----------------+----------+-----------+---------+---------- 1 | -> Row Adapter | 7490.354 | 34035 | 200 | | 70KB | | | 58 | 15149.80 2 | -> Vector Streaming (type: GATHER) | 7488.129 | 34035 | 200 | | 216KB | | | 58 | 15149.80 3 | -> Vector Hash Aggregate | [7481.430, 7481.430] | 34035 | 200 | | [9MB, 9MB] | 16MB | [112,112] | 58 | 15137.30 4 | -> Vector Hash Left Join (5, 7) | [909.377, 909.377] | 31204164 | 109803 | | [2MB, 2MB] | 16MB | | 34 | 3880.50 5 | -> Vector Sonic Hash Aggregate | [5.876, 5.876] | 34035 | 34036 | 6807 | [3MB, 3MB] | 16MB | [51,51] | 18 | 1127.67 6 | -> CStore Scan on dmisc.dm_dim_cbg_wh_holiday_d | [0.199, 0.199] | 34036 | 34036 | | [792KB, 792KB] | 1MB | | 18 | 532.04 7 | -> CStore Scan on dmisc.dm_dim_cbg_wh_holiday_d | [40.794, 40.794] | 25122 | 21960 | 19 | [1MB, 1MB] | 1MB | [59,59] | 24 | 617.13 从执行计划中可以看到,中间结果集大小已经在可接受的范围内。但是又看到聚合3千万数据使用了6s+的时间,这是过慢的,需要看执行计划中的DN信息寻找原因 。 Datanode Information (identified by plan id) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 --Row Adapter (actual time=7486.498..7490.354 rows=34035 loops=1) (CPU: ex c/r=107, ex row=34035, ex cyc=3668104, inc cyc=22468059912) 2 --Vector Streaming (type: GATHER) (actual time=7486.466..7488.129 rows=34035 loops=1) (Buffers: shared hit=1) (CPU: ex c/r=660037, ex row=34035, ex cyc=22464391808, inc cyc=22464391808) 3 --Vector Hash Aggregate dn_6083_6084 (actual time=7479.644..7481.430 rows=34035 loops=1) (projection time=4488.807) dn_6083_6084 (Buffers: shared hit=40) dn_6083_6084 (CPU: ex c/r=631, ex row=31204164, ex cyc=19718763112, inc cyc=22443886288) 4 --Vector Hash Left Join (5, 7) dn_6083_6084 (actual time=48.009..909.377 rows=31204164 loops=1) dn_6083_6084 (Buffers: shared hit=36) dn_6083_6084 (CPU: ex c/r=43699, ex row=59157, ex cyc=2585141400, inc cyc=2725123176) 5 --Vector Sonic Hash Aggregate dn_6083_6084 (actual time=5.177..5.876 rows=34035 loops=1) dn_6083_6084 (Buffers: shared hit=11) dn_6083_6084 (CPU: ex c/r=500, ex row=34036, ex cyc=17027544, inc cyc=17619064) 6 --CStore Scan on dmisc.dm_dim_cbg_wh_holiday_d dn_6083_6084 (actual time=0.043..0.199 rows=34036 loops=1) (CU ScanInfo: smallCu: 0, totalCu: 1, avrCuRow: 34036, totalDeadRows: 0) dn_6083_6084 (Buffers: shared hit=11) dn_6083_6084 (CPU: ex c/r=17, ex row=34036, ex cyc=591520, inc cyc=591520) 7 --CStore Scan on dmisc.dm_dim_cbg_wh_holiday_d dn_6083_6084 (actual time=6.464..40.794 rows=25122 loops=1) (filter time=0.872 projection time=33.671) (RoughCheck CU: CUNone: 0, CUTagNone: 0, CUSome: 1) (CU ScanInfo: smallCu: 0, totalCu: 1, avrCuRow: 34036, totalDeadRows: 0) dn_6083_6084 (Buffers: shared hit=25) dn_6083_6084 (CPU: ex c/r=3595, ex row=34036, ex cyc=122362712, inc cyc=122362712) 从中可以看出,所有算子都只在一个DN上运行了。这可以视为严重的计算倾斜,若对单点性能有更高要求需要继续优化。查看DMISC.DM_DIM_CBG_WH_HOLIDAY_D表的定义,发现它是一个复制表(distribute by replication),在进行各层运算的时候只用其中一个DN来算。而在本SQL中,使用到这张表的时候,关联键都是WH_ID。 再查看调整分布列为WH_ID的倾斜情况: select * from pg_catalog.table_skewness('DMISC.DM_DIM_CBG_WH_HOLIDAY_D', 'wh_id'); 结果有23行,小于集群DN个数,且存在倾斜。但是本SQL需要使用该表的全量数据,故可以把这张表改为使用WH_ID作为分步键进行重分布。 由表分布方式为复制表导致的计算倾斜无法使用skew hint解决,可以改变物理表分布方式或者创建临时表来解决(复制表通常较小)。由于表在SQL中的使用情况和表的倾斜情况,不适合更改物理表分步键为WH_ID,故本例中试使用创建临时表指定重分布方式的办法解决。 DROP TABLE IF EXISTS holiday_d_tmp; CREATE TEMP TABLE holiday_d_tmp WITH ( orientation = COLUMN, compression = low ) distribute BY hash ( wh_id ) AS ( SELECT * FROM DMISC.DM_DIM_CBG_WH_HOLIDAY_D ); EXPLAIN performance WITH TMP AS ( SELECT WH_ID, ( IFNULL ( SUBSTR( THE_DATE, 1, 10 ), '1900-01-01' ) || ' ' || STOP_TIME ) :: TIMESTAMP AS STOP_TIME, ( IFNULL ( SUBSTR( THE_DATE, 1, 10 ), '1900-01-01' ) || ' ' || '23:59:59' ) :: TIMESTAMP AS MAX_ASD FROM holiday_d_tmp WHERE IS_OPEN = 'Y' AND STOP_TIME IS NOT NULL ) SELECT T1.WH_ID, T1.THE_DATE, T1.IS_OPEN, MIN ( CASE WHEN T1.THE_DATE < T2.STOP_TIME THEN STOP_TIME ELSE NULL END ) AS STOP_TIME, MIN ( CASE WHEN T1.THE_DATE < T2.STOP_TIME THEN T2.MAX_ASD ELSE NULL END ) AS TODAY_MAX_ASD, MIN ( CASE WHEN ADDDATE ( T1.THE_DATE, 1 ) < T2.STOP_TIME THEN T2.MAX_ASD ELSE NULL END ) AS NEXT_MAX_ASD FROM ( SELECT WH_ID, THE_DATE, IS_OPEN FROM holiday_d_tmp ) T1 LEFT JOIN TMP T2 ON T1.WH_ID = T2.WH_ID GROUP BY T1.WH_ID, T1.THE_DATE, T1.IS_OPEN; 下面是对应的执行计划: id | operation | A-time | A-rows | E-rows | E-distinct | Peak Memory | E-memory | A-width | E-width | E-costs ----+--------------------------------------------------------------------------------------+------------------+----------+----------+------------+----------------+----------+---------+---------+---------- 1 | -> Row Adapter | 673.495 | 34035 | 34032 | | 70KB | | | 58 | 68112.60 2 | -> Vector Streaming (type: GATHER) | 671.103 | 34035 | 34032 | | 216KB | | | 58 | 68112.60 3 | -> Vector Hash Aggregate | [0.079, 672.724] | 34035 | 34032 | | [1MB, 1MB] | 16MB | [0,114] | 58 | 67794.10 4 | -> Vector Hash Left Join (5, 6) | [0.047, 76.395] | 31205167 | 27587201 | | [324KB, 485KB] | 16MB | | 34 | 8876.88 5 | -> CStore Scan on pg_temp_cn_5003_6_22022_139764371019520.holiday_d_tmp | [0.004, 0.098] | 34036 | 34036 | 1 | [760KB, 792KB] | 1MB | | 18 | 1553.65 6 | -> CStore Scan on pg_temp_cn_5003_6_22022_139764371019520.holiday_d_tmp | [0.008, 3.253] | 25122 | 22018 | 1 | [880KB, 1MB] | 1MB | [0,61] | 24 | 1557.76 从计划中我们可以看到,耗时比单个DN运算快了不少,当然这里没有算上创建临时表的时间约0.2s。 5、【调优总结】 在本案例中,因为实际执行SQL时间太长先看了verbose计划而非performance计划,发现中间结果集发散问题后,进行等价逻辑改写,把两个(等值-不等值)关联改为一个等值关联和条件聚合。之后,我们发现SQL因复制表存在计算倾斜问题,考虑SQL消费表数据的方式和表的统计数据,采用了使用临时表重新指定分布方式的方法,解决了计算倾斜问题,SQL从单点25min+优化到单点800ms。 点击关注,第一时间了解华为云新鲜技术~

优秀的个人博客,低调大师

数仓实践丨表扫描时过滤行数过多引起的性能瓶颈问题

本文分享自华为云社区《GaussDB(DWS)性能调优:表扫描时过滤行数过多引起的性能瓶颈问题案例》,作者: O泡果奶~ 。 1、【问题描述】 SQL语句执行过程中,对12亿数据量的大表进行扫描,过滤99%的数据仅留617行数据,性能瓶颈位于扫描该表这里。 2、【原始语句】 set search_path = 'bi_dashboard'; WITH F_SRV_DB_DIM_PRD_D AS (SELECT EXTERNAL_NAME FROM ( SELECT MKT_NAME EXTERNAL_NAME FROM BI_DASHBOARD.DM_MSS_ITEM_PRODUCT_D PRD WHERE PRD.COMPANY_BRAND =any(array[string_to_array('HUAWEI',',')]) AND PRD.MKT_NAME =any(array[string_to_array('畅享 60,畅享 50,畅享 60X,畅享 60 Pro,畅享 50 Pro,畅享 50z,nova 10z,畅享 20e,畅享20 Pro,畅享 10e,畅享10 Plus,畅享20 SE,畅享10,nova 11i,畅享20 Plus,畅享9 Plus,畅享20 5G,nova Y90,畅享 10S,nova Y70,畅享Z,畅享 9S,nova 8 SE 活力版,麦芒9 5G,Y9s,麦芒9 5G',',')]) ) WHERE EXTERNAL_NAME<>'SNULL' GROUP BY EXTERNAL_NAME), V_PERIOD AS ( SELECT PERIOD_ID AS PERIOD_ID_M, LEAST(TO_CHAR(PERIOD_END_DATE, 'YYYYMMDD'), '20230630') AS PERIOD_ID, PERIOD_ID AS DATES FROM BI_DASHBOARD.RPT_TML_ACCOUNT_PERIOD_D WHERE PERIOD_TYPE = 'M' AND PERIOD_ID BETWEEN 202207 AND 202306 ), V_DATA_BASE AS ( SELECT A.PERIOD_ID, IFNULL(A.CHANNEL_NAME, 'SNULL') AS DISTRIBUTOR_CHANNEL_NAME, SUM(A.SO_QTY_MTD) AS SO_QTY, SUM(DECODE(A.PERIOD_ID, 20230630, A.SO_QTY_MTD)) AS SO_QTY_ORDER select count(*) FROM DM_MSS_CN_PC_REP_RP_ST_D_F A INNER JOIN F_SRV_DB_DIM_PRD_D PRD ON A.EXTERNAL_NAME = PRD.EXTERNAL_NAME WHERE 1 = 1 AND A.CHANNEL_ID IN ('100013388802') AND A.ORG_KEY IN (10000651) AND A.SALES_FLAG IN ('1', '0') AND A.PERIOD_ID IN (20220731,20221031,20220930,20220831,20221130,20221231,20230131,20230228,20230430,20230331,20230531,20230630) AND (A.SO_QTY_MTD <> 0) -- 过滤所有日期SO_QTY为0的数据 GROUP BY A.PERIOD_ID, IFNULL(A.CHANNEL_NAME, 'SNULL') ), V_DATA AS ( SELECT PERIOD_ID, NVL(DISTRIBUTOR_CHANNEL_NAME, 'Total') AS DISTRIBUTOR_CHANNEL_NAME, SUM(SO_QTY) AS SO_QTY, SUM(SO_QTY_ORDER) AS SO_QTY_ORDER FROM V_DATA_BASE A GROUP BY GROUPING SETS ((PERIOD_ID), (PERIOD_ID, DISTRIBUTOR_CHANNEL_NAME)) ) SELECT STRING_AGG(P.DATES, ',' ORDER BY P.PERIOD_ID_M) AS PERIOD_LIST, B.DISTRIBUTOR_CHANNEL_NAME, STRING_AGG(NVL(TO_CHAR(ROUND(A.SO_QTY)), '0'), ',' ORDER BY P.PERIOD_ID_M) AS SO_QTY FROM V_PERIOD P FULL JOIN (SELECT DISTINCT DISTRIBUTOR_CHANNEL_NAME FROM V_DATA) B ON 1 = 1 LEFT JOIN V_DATA A ON A.PERIOD_ID = P.PERIOD_ID AND A.DISTRIBUTOR_CHANNEL_NAME = B.DISTRIBUTOR_CHANNEL_NAME GROUP BY B.DISTRIBUTOR_CHANNEL_NAME ORDER BY DECODE(B.DISTRIBUTOR_CHANNEL_NAME, 'Total', 0, 'SOURCE IS NULL', 2, '源为空', 3, 'SNULL', 4, 1), SUM(A.SO_QTY_ORDER) DESC NULLS LAST LIMIT 50 OFFSET 0 3、【性能分析】 从上图的performance执行计划中可以看出(完整执行计划放在附件一),该SQL语句慢在扫描表a(bi_dashboard.dm_mss_cn_pc_rep_rp_st_d_f_test)。扫描时过滤条件包括:sales_flag、so_qty_mtd、channel_id、org_key、period_id,该表上原本的局部聚簇键PCK只包含了period_id,并没有包括其余三个过滤条件之一,因此,可以调整PCK,以减少扫描表a的执行时间。 补充:局部聚簇键 局部聚簇 (Partial Cluster Key, 简称PCK),列存储下一种通过min/max稀疏索引实现基表快速扫描的索引技术。Partial Cluster Key可以指定多列,但是一般不建议超过2列。PCK适用于列存大表点查询加速。 另外,查看语句中where条件中in值较多(12个),在DWS中,in后面的条件默认就只能是5个,超过6个就过滤不下推,此时,可以用or将12个值改写, A.PERIOD_ID IN (20220731,20221031,20220930,20220831,20221130) or A.PERIOD_ID IN (20221231,20230131,20230228,20230430,20230331) or A.PERIOD_ID IN (20230531,20230630) 此时,SQL语句执行时间减少为487ms,完整performance计划如附件二所示。 附件:优化后—performance.txt466.64KB 附件:优化前—performance.txt449.47KB 点击关注,第一时间了解华为云新鲜技术~

优秀的个人博客,低调大师

数仓实时场景下表行数估算不准确引起的的性能瓶颈问题案例

本文分享自华为云社区《GaussDB(DWS)性能调优:实时场景下表行数估算不准确引起的的性能瓶颈问题案例》,作者: O泡果奶~。 本文针对实时场景下SQL语句因表行数估算不准确而导致语句执行超时报错的案例进行分析。 1、【问题描述】 实时场景下,select查询语句执行时间过长,该语句verbose执行计划中存在nestloop,且使用hint(set (enable_index_nestloop off)) 无法生效。 2、【原始语句】 select * from ( select wo.work_order_id /*工单id*/, wo.work_order_code /*工单编码*/, wo.work_order_name /*工单名称*/, wo.work_order_level /*工单层级(第一层级(未拆分工单/父工单):1,第二层级(子工单):10)*/, decode(wo.work_order_level,1, '第一层级(未拆分工单/父工单)', 10,'第二层级(子工单)') as work_order_level_desc /*工单层级描述*/, substrb(wo.wo_description, 1, 1000) as wo_description /*工单描述*/, wo.wo_version /*工单版本号*/, wo.wo_lifecycle_status /*生命周期标识:0:正常工单,-1: 已删除*/, wo.business_id /*工单来源业务id*/, wo.business_type /*工单来源业务类型(10:活动流工单 20:手工派单 30:拆单工单 40:临时mos工单 50:ihub工单 60:ipmo工单 70:wbs工单 80:ncs工单 90:hr工单 100:ls工单 默认10)*/, decode( wo.business_type, '10', '活动流工单', '20', '手工派单', '30', '拆单工单', '40', '临时MOS工单', '50', 'ihub工单', '60', 'ipmo工单', '70', 'WBS工单', '80', 'NCS工单', '90', 'HR工单', '100', 'LS工单' ) as business_type_desc /*工单来源业务类型描述*/, wo.parent_activity_id /*父节点活动id*/, wo.activity_lib_id /*活动库活动id*/, wo.activity_type /*作业类型,1wbs,2活动,3里程碑*/, ac.activity_name /*活动名称*/, ac.std_ms_code as standard_ms_code /*标准里程碑编码*/, wo.plan_id /*计划id*/, wo.project_number as proj_num /*项目编码*/, wo.du_id /*交付单元id*/, wo.duration /*工期*/, wo.billing_flag /*开票标识:y-开票*/, wo.na_flag /*na标识*/, wo.inv_flag /*inv标识*/, wo.master_flag /*拆分标示,n:未拆分 ; y:已拆分*/, wo.created_by as created_by_id /*创建人user id*/, u1.lname as created_by /*创建人*/, wo.creation_date /*创建时间*/, wo.last_updated_by as last_updated_by_id /*最后更新人user id*/, u2.lname as last_updated_by /*最后更新人*/, wo.last_update_date /*最后更新时间*/, wp.wo_progress_id /*活动进度id*/, wp.expect_start_date /*预期开始日期*/, wp.expect_end_date /*预期结束日期*/, wp.plan_start_time /*计划开始时间*/, wp.plan_end_time /*计划完成时间*/, wp.actual_start_time /*实际开始时间*/, wp.actual_end_time /*实际完成时间*/, wp.close_time /*活动关闭时间*/, wp.completion_rate /*完工比率(数值如 0.8666)*/, to_char(substr(wp.remark, 1, 333)) as progress_description /*进度备注信息*/, wp.total_value /*总值*/, wp.accumulate_value /*累计值*/, wp.report_time /*值反馈时间*/, wp.total_plan_value /*总计划值*/, wp.ehs_risk /*高危活动类型*/, wp.delay_reason_id /*延迟原因id*/, substrb(ag.description, 1, 1000) as delay_reason_description /*延迟原因描述*/, wp.wo_status /*活动状态 psc_lookup_item_t_3220 classify_code = 'WO_STATUS_CODE'*/, l2.item_name as wo_status_desc /*活动状态描述*/,( case when lengthb(wp.approve_status) = 0 then null else wp.approve_status end ) :: number as approve_status /*审批状态 psc_lookup_item_t_3220 classify_code = 'WORK_ORDER_APPROVE_STATUS'*/, l3.item_name as approve_status_desc /*审批状态描述*/, wp.par_workorder_doc_flag /*父工单是否有交付件(y/n)*/, wp.deliverables_complete /*交付件上传状态 0:不涉及交付件 1:待上传交付件 2:交付件上传中,未上传完9:交付件已上传完*/, wp.revenue_trigger_status /*触发状态(0:未触发过 1:已触发 2:已触发,pc校验触发失败 3:pc触发成功)*/, wp.billing_status /*开票状态(空值:未触发过 1:已开票)*/, wp.frozen_flag /*冻结标识(y/n)*/, wp.mr_frozen_flag /*mr是否冻结站点要货通过更新实施计划刷新字段*/, wp.mr_status /*站点签状态 1未签收,2部分签收,3全部签收,10未签收,20部分签收,30全部签收,40部分超配置签收,50全部超配置签收 完工验状态 p:部分完成,f:全部完成*/, wp.tool_flag /*是否挂工具工单回写(y/n)*/, wp.split_cp_flag /*拆分施工计划标识 y已拆分 n未拆分*/, wp.mos_data_source /*站点签完工验状数据来源*/, wo.template_id /*模板id,例如活动流节点id*/, tfn.task_flow_id /*任务流id*/, tfn.task_flow_node_id /*活动流节点id*/, tfn.revenue_flag /*收入里程碑标识(y/n)*/, tfn.on_site /*是否现场*/, nvl(l1.item_name, tfn.owner_type) as owner_type /*责任方类型 客户/华为/分包商*/, tfn.subcon /*是否分包*/ /*产品域*/,case when wo.enable_flag = 'Y' and wp.enable_flag = 'Y' and wo.wo_lifecycle_status = 0 and nvl(du.enable_flag, 'Y') = 'Y' then 'Y' else 'N' end as enable_flag /*有效标识,y为有效n为失效*/, 'N' as del_flag /*删除标识 y为已删除*/, 3 as data_center_id /*数据中心id*/, tf.task_flow_code /*活动流编码 add by jwx528041 20200408*/, tfn.task_flow_node_code /*任务流节点编码 add by jwx528041 20200408*/, tfn.task_flow_node_name /*任务流节点名称 add by jwx528041 20200408*/, tfn.task_flow_node_type /*任务流节点类型 add by jwx528041 20200408*/, tfn.enable_flag as flow_enable_flag /*活动流有效标识 add by jwx528041 20200408*/, wo.tenant_code /*租户编码 add by jwx528041 20200408*/, tfn.activity_id /*活动流水号 add by jwx528041 20200408*/, tfn.lead_time /*持续时间 add by jwx528041 20200408*/, wo.resource_id as wo_actual_owner_id /*工单实际责任人id update by swx949890 202207*/, wo.resource_name as wo_actual_owner /*工单实际责任人 update by swx949890 202207*/, wo.contractor_id as wo_actual_owner_contr_id /*工单实际责任人分包商id update by swx949890 202207*/, wo.contractor_name as wo_actual_owner_contr_name /*工单实际责任人分包商名称 update by swx949890 202207*/, nvl(l4.item_name, tfn.delivery_model) as delivery_model /*工单交付模式 add by cwx613468 20200711*/, tfn.on_line_site /*是否上站 add by cwx613468 20200711*/, u3.lname as dispatcher_user_name /*调度人 add by cwx613468 20200711*/, tfn.approve_level_qty /*审批总层级 add by jwx528041 20200819*/, tf.task_flow_name /*活动流名称 add by jwx528041 20200819*/, tf.task_flow_type /*活动流类型 add by jwx528041 20200819*/, wp.source_code /*标识actual时间的修改来源,值为mobile标识从手机端回写 add by jwx528041 20200819*/, wp.plan_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.plan_update_time /*计划时间更新时间 add by jwx528041 20200819*/, wp.dispatch_time /*调度时间 add by jwx528041 20200819*/, wp.first_actual_update_time /*第一次实际开始时间填入时间 add by jwx528041 20200819*/, wp.first_actual_end_time /*第一次实际结束时间填入时间 add by jwx528041 20200819*/, wp.first_actual_updated_by /*第一次实际时间填入人user id add by jwx528041 20200819*/, wp.actual_start_update_time /*实际开始时间更新日期 add by jwx528041 20200819*/, wp.actual_start_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.actual_time_source /*实际完成时间更新来源 add by jwx528041 20200819*/, wp.actual_end_update_time /*实际完成时间更新日期 add by jwx528041 20200819*/, wp.actual_end_updated_by /*实际完成时间更新人user id add by jwx528041 20200819*/, wp.revenue_trigger_failed_msg /*收入触发失败原因 add by jwx528041 20200819*/, ag.souce_type as delay_reason_souce_type /*延迟原因数据来源:1、自定义 2、 add by jwx528041 20200819*/ --,ras.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type add by jwx528041 20200819*/ , wo.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type update by swx949890 202207*/, dr.resouce_type as wo_owner_resouce_type /*工单责任人资源类型 add by jwx528041 20200819*/, l5.item_name as wo_owner_resouce_type_desc /*工单责任人资源类型 add by jwx528041 20200819*/, u4.w3_account as wo_owner_w3_account /*工单责任人w3账号 add by jwx528041 20200819*/, rel.du_tf_rel_enable /*du与活动流关系有效性标识 y:有效 n:失效 add by lwx617215 20210116*/, t.billing_sla /*sla*/, t.billing_milestone /*开票里程碑*/, tf.required_tools, wp.active, gp.plan_code, gp.plan_name, gp.template_plan_id from sdisd.ogg_wo_work_order_2_3220 wo inner join sdisd.ogg_wo_progress_2_3220 wp on wo.work_order_id = wp.work_order_id left join sdisd.ogg_wo_task_flow_node_br_3220 tfn on wo.template_id = tfn.task_flow_node_id and nvl(wo.wo_version, 0) = case when nvl(wo.wo_version, 0) > 0 then tfn.version else tfn.wo_version end and wo.project_number = tfn.project_number left join sdisd.ogg_sds_activity_t_br_3220 ac on wo.activity_lib_id = ac.activity_id left join sdisd.ogg_sds_task_flow_t_br_3220 tf on tfn.task_flow_id = tf.task_flow_id left join sdisd.ogg_du_release_t_br_3220 du /*enable_flag新增有效du的判断 lwx617215 20210116*/ on wo.du_id = du.du_id left join sdisd.ogg_gcc_plan_2_3220 gp --dwx1189869 on wo.plan_id = gp.plan_id and gp.tenant_code = 'RolloutPlan' and gp.parent_plan_id = -1 and gp.enable_flag = 'Y' left join ( select r.du_id, r.task_flow_id, /*du与活动流有效标识*/ case when r.enable_flag = 'Y' and publish_flag = 'P' then 'Y' else 'N' end as du_tf_rel_enable, row_number() over( partition by r.du_id, r.task_flow_id order by r.last_update_date desc ) as rn from sdisd.ogg_rp_du_tf_release_3_3220 r ) rel on wo.du_id = rel.du_id and tfn.task_flow_id = rel.task_flow_id and rel.rn = 1 left join sdisd.ogg_tpl_user_t_3220 u1 on wo.created_by = u1.user_id left join sdisd.ogg_tpl_user_t_3220 u2 on wo.last_updated_by = u2.user_id left join sdisd.ogg_tpl_user_t_3220 u3 on wp.dispatcher_user_id = u3.user_id left join sdisd.ogg_sds_activity_gap_t_br_3220 ag on wp.delay_reason_id = ag.activity_gap_id left join sdisd.ogg_tpl_lookup_item_t_3220 l1 on tfn.owner_type = l1.item_code and l1.classify_code = 'SDS_TASK_OWNER_TYPE' and l1.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l2 on wp.wo_status = l2.item_code and l2.classify_code = 'WO_STATUS_CODE' and l2.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l3 on wp.approve_status = l3.item_code and l3.classify_code = 'WORK_ORDER_APPROVE_STATUS' and l3.language = 'en_US' left join sdisd.ogg_tpl_lookup_item_t_3220 l4 on tfn.delivery_model = l4.item_code and l4.classify_code = 'SDS_TASK_ON_SITE' and l4.language = 'en_US' left join sdisd.ogg_pm_project_tree_node_3220 tn on wo.resource_id = tn.tree_id left join sdisd.ogg_pm_delivery_resource_3220 dr on tn.resource_id = dr.resource_id left join sdisd.ogg_tpl_user_t_3220 u4 on dr.user_id = u4.user_id left join sdisd.ogg_tpl_lookup_item_t_3220 l5 on dr.resouce_type = l5.item_code and l5.classify_code = 'PM_RESOURCE_TYPE' and l5.language = 'zh_CN' left join sdisd.ogg_sds_task_flow_node_br_3220 t on tfn.task_flow_node_id = t.task_flow_node_id where ( wo.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or wp.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or tfn.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or ag.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 ) union all select wo.work_order_id /*工单id*/, wo.work_order_code /*工单编码*/, wo.work_order_name /*工单名称*/, wo.work_order_level /*工单层级(第一层级(未拆分工单/父工单):1,第二层级(子工单):10)*/, decode( wo.work_order_level, 1, '第一层级(未拆分工单/父工单)', 10, '第二层级(子工单)' ) as work_order_level_desc /*工单层级描述*/, substrb(wo.wo_description, 1, 1000) as wo_description /*工单描述*/, wo.wo_version /*工单版本号*/, wo.wo_lifecycle_status /*生命周期标识:0:正常工单,-1: 已删除*/, wo.business_id /*工单来源业务id*/, wo.business_type /*工单来源业务类型(10:活动流工单 20:手工派单 30:拆单工单 40:临时mos工单 50:ihub工单 60:ipmo工单 70:wbs工单 80:ncs工单 90:hr工单 100:ls工单 默认10)*/, decode( wo.business_type, '10', '活动流工单', '20', '手工派单', '30', '拆单工单', '40', '临时MOS工单', '50', 'ihub工单', '60', 'ipmo工单', '70', 'WBS工单', '80', 'NCS工单', '90', 'HR工单', '100', 'LS工单' ) as business_type_desc /*工单来源业务类型描述*/, wo.parent_activity_id /*父节点活动id*/, wo.activity_lib_id /*活动库活动id*/, wo.activity_type /*作业类型,1wbs,2活动,3里程碑*/, ac.activity_name /*活动名称*/, ac.std_ms_code as standard_ms_code /*标准里程碑编码*/, wo.plan_id /*计划id*/, wo.project_number as proj_num /*项目编码*/, wo.du_id /*交付单元id*/, wo.duration /*工期*/, wo.billing_flag /*开票标识:y-开票*/, wo.na_flag /*na标识*/, wo.inv_flag /*inv标识*/, wo.master_flag /*拆分标示,n:未拆分 ; y:已拆分*/, wo.created_by as created_by_id /*创建人user id*/, u1.lname as created_by /*创建人*/, wo.creation_date /*创建时间*/, wo.last_updated_by as last_updated_by_id /*最后更新人user id*/, u2.lname as last_updated_by /*最后更新人*/, wo.last_update_date /*最后更新时间*/, wp.wo_progress_id /*活动进度id*/, wp.expect_start_date /*预期开始日期*/, wp.expect_end_date /*预期结束日期*/, wp.plan_start_time /*计划开始时间*/, wp.plan_end_time /*计划完成时间*/, wp.actual_start_time /*实际开始时间*/, wp.actual_end_time /*实际完成时间*/, wp.close_time /*活动关闭时间*/, wp.completion_rate /*完工比率(数值如 0.8666)*/, to_char(substr(wp.remark, 1, 333)) as progress_description /*进度备注信息*/, wp.total_value /*总值*/, wp.accumulate_value /*累计值*/, wp.report_time /*值反馈时间*/, wp.total_plan_value /*总计划值*/, wp.ehs_risk /*高危活动类型*/, wp.delay_reason_id /*延迟原因id*/, substrb(ag.description, 1, 1000) as delay_reason_description /*延迟原因描述*/, wp.wo_status /*活动状态 psc_lookup_item_t_3220 classify_code = 'WO_STATUS_CODE'*/, l2.item_name as wo_status_desc /*活动状态描述*/,( case when lengthb(wp.approve_status) = 0 then null else wp.approve_status end ) :: number as approve_status /*审批状态 psc_lookup_item_t_3220 classify_code = 'WORK_ORDER_APPROVE_STATUS'*/, l3.item_name as approve_status_desc /*审批状态描述*/, wp.par_workorder_doc_flag /*父工单是否有交付件(y/n)*/, wp.deliverables_complete /*交付件上传状态 0:不涉及交付件 1:待上传交付件 2:交付件上传中,未上传完9:交付件已上传完*/, wp.revenue_trigger_status /*触发状态(0:未触发过 1:已触发 2:已触发,pc校验触发失败 3:pc触发成功)*/, wp.billing_status /*开票状态(空值:未触发过 1:已开票)*/, wp.frozen_flag /*冻结标识(y/n)*/, wp.mr_frozen_flag /*mr是否冻结站点要货通过更新实施计划刷新字段*/, wp.mr_status /*站点签状态 1未签收,2部分签收,3全部签收,10未签收,20部分签收,30全部签收,40部分超配置签收,50全部超配置签收 完工验状态 p:部分完成,f:全部完成*/, wp.tool_flag /*是否挂工具工单回写(y/n)*/, wp.split_cp_flag /*拆分施工计划标识 y已拆分 n未拆分*/, wp.mos_data_source /*站点签完工验状数据来源*/, wo.template_id /*模板id,例如活动流节点id*/, tfn.task_flow_id /*任务流id*/, tfn.task_flow_node_id /*活动流节点id*/, tfn.revenue_flag /*收入里程碑标识(y/n)*/, tfn.on_site /*是否现场*/, nvl(l1.item_name, tfn.owner_type) as owner_type /*责任方类型 客户/华为/分包商*/, tfn.subcon /*是否分包*/ /*产品域*/,case when wo.enable_flag = 'Y' and wp.enable_flag = 'Y' and wo.wo_lifecycle_status = 0 and nvl(du.enable_flag, 'Y') = 'Y' then 'Y' else 'N' end as enable_flag /*有效标识,y为有效n为失效*/, 'N' as del_flag /*删除标识 y为已删除*/, 4 as data_center_id /*数据中心id*/, tf.task_flow_code /*活动流编码 add by jwx528041 20200408*/, tfn.task_flow_node_code /*任务流节点编码 add by jwx528041 20200408*/, tfn.task_flow_node_name /*任务流节点名称 add by jwx528041 20200408*/, tfn.task_flow_node_type /*任务流节点类型 add by jwx528041 20200408*/, tfn.enable_flag as flow_enable_flag /*活动流有效标识 add by jwx528041 20200408*/, wo.tenant_code /*租户编码 add by jwx528041 20200408*/, tfn.activity_id /*活动流水号 add by jwx528041 20200408*/, tfn.lead_time /*持续时间 add by jwx528041 20200408*/, wo.resource_id as wo_actual_owner_id /*工单实际责任人id update by swx949890 202207*/, wo.resource_name as wo_actual_owner /*工单实际责任人 update by swx949890 202207*/, wo.contractor_id as wo_actual_owner_contr_id /*工单实际责任人分包商id update by swx949890 202207*/, wo.contractor_name as wo_actual_owner_contr_name /*工单实际责任人分包商名称 update by swx949890 202207*/, nvl(l4.item_name, tfn.delivery_model) as delivery_model /*工单交付模式 add by cwx613468 20200711*/, tfn.on_line_site /*是否上站 add by cwx613468 20200711*/, u3.lname as dispatcher_user_name /*调度人 add by cwx613468 20200711*/, tfn.approve_level_qty /*审批总层级 add by jwx528041 20200819*/, tf.task_flow_name /*活动流名称 add by jwx528041 20200819*/, tf.task_flow_type /*活动流类型 add by jwx528041 20200819*/, wp.source_code /*标识actual时间的修改来源,值为mobile标识从手机端回写 add by jwx528041 20200819*/, wp.plan_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.plan_update_time /*计划时间更新时间 add by jwx528041 20200819*/, wp.dispatch_time /*调度时间 add by jwx528041 20200819*/, wp.first_actual_update_time /*第一次实际开始时间填入时间 add by jwx528041 20200819*/, wp.first_actual_end_time /*第一次实际结束时间填入时间 add by jwx528041 20200819*/, wp.first_actual_updated_by /*第一次实际时间填入人user id add by jwx528041 20200819*/, wp.actual_start_update_time /*实际开始时间更新日期 add by jwx528041 20200819*/, wp.actual_start_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.actual_time_source /*实际完成时间更新来源 add by jwx528041 20200819*/, wp.actual_end_update_time /*实际完成时间更新日期 add by jwx528041 20200819*/, wp.actual_end_updated_by /*实际完成时间更新人user id add by jwx528041 20200819*/, wp.revenue_trigger_failed_msg /*收入触发失败原因 add by jwx528041 20200819*/, ag.souce_type as delay_reason_souce_type /*延迟原因数据来源:1、自定义 2、 add by jwx528041 20200819*/ --,ras.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type add by jwx528041 20200819*/ , wo.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type update by swx949890 202207*/, dr.resouce_type as wo_owner_resouce_type /*工单责任人资源类型 add by jwx528041 20200819*/, l5.item_name as wo_owner_resouce_type_desc /*工单责任人资源类型 add by jwx528041 20200819*/, u4.w3_account as wo_owner_w3_account /*工单责任人w3账号 add by jwx528041 20200819*/, rel.du_tf_rel_enable /*du与活动流关系有效性标识 y:有效 n:失效 add by lwx617215 20210116*/, t.billing_sla /*sla*/, t.billing_milestone /*开票里程碑*/, tf.required_tools, wp.active, gp.plan_code, gp.plan_name, gp.template_plan_id from sdisd.ogg_wo_work_order17_3220 wo inner join sdisd.ogg_wo_progress17_3220 wp on wo.work_order_id = wp.work_order_id left join sdisd.ogg_wo_task_flow_node_za_3220 tfn on wo.template_id = tfn.task_flow_node_id and nvl(wo.wo_version, 0) = case when nvl(wo.wo_version, 0) > 0 then tfn.version else tfn.wo_version end and wo.project_number = tfn.project_number left join sdisd.ogg_sds_activity_t_za_3220 ac on wo.activity_lib_id = ac.activity_id left join sdisd.ogg_sds_task_flow_t_za_3220 tf on tfn.task_flow_id = tf.task_flow_id left join sdisd.ogg_du_release_t_za_3220 du /*enable_flag新增有效du的判断 lwx617215 20210116*/ on wo.du_id = du.du_id left join sdisd.ogg_gcc_plan17_3220 gp --dwx1189869 on wo.plan_id = gp.plan_id and gp.tenant_code = 'RolloutPlan' and gp.parent_plan_id = -1 and gp.enable_flag = 'Y' left join ( select r.du_id, r.task_flow_id, /*du与活动流有效标识*/ case when r.enable_flag = 'Y' and publish_flag = 'P' then 'Y' else 'N' end as du_tf_rel_enable, row_number() over( partition by r.du_id, r.task_flow_id order by r.last_update_date desc ) as rn from sdisd.ogg_rp_du_tf_release18_3220 r ) rel on wo.du_id = rel.du_id and tfn.task_flow_id = rel.task_flow_id and rel.rn = 1 left join sdisd.ogg_tpl_user_t_3220 u1 on wo.created_by = u1.user_id left join sdisd.ogg_tpl_user_t_3220 u2 on wo.last_updated_by = u2.user_id left join sdisd.ogg_tpl_user_t_3220 u3 on wp.dispatcher_user_id = u3.user_id left join sdisd.ogg_sds_activity_gap_t_za_3220 ag on wp.delay_reason_id = ag.activity_gap_id left join sdisd.ogg_tpl_lookup_item_t_3220 l1 on tfn.owner_type = l1.item_code and l1.classify_code = 'SDS_TASK_OWNER_TYPE' and l1.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l2 on wp.wo_status = l2.item_code and l2.classify_code = 'WO_STATUS_CODE' and l2.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l3 on wp.approve_status = l3.item_code and l3.classify_code = 'WORK_ORDER_APPROVE_STATUS' and l3.language = 'en_US' left join sdisd.ogg_tpl_lookup_item_t_3220 l4 on tfn.delivery_model = l4.item_code and l4.classify_code = 'SDS_TASK_ON_SITE' and l4.language = 'en_US' left join sdisd.ogg_pm_project_tree_node_3220 tn on wo.resource_id = tn.tree_id left join sdisd.ogg_pm_delivery_resource_3220 dr on tn.resource_id = dr.resource_id left join sdisd.ogg_tpl_user_t_3220 u4 on dr.user_id = u4.user_id left join sdisd.ogg_tpl_lookup_item_t_3220 l5 on dr.resouce_type = l5.item_code and l5.classify_code = 'PM_RESOURCE_TYPE' and l5.language = 'zh_CN' left join sdisd.ogg_sds_task_flow_node_za_3220 t on tfn.task_flow_node_id = t.task_flow_node_id where ( wo.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or wp.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or tfn.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or ag.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 )) as t limit 10 3、【性能分析】 优化前SQL语句执行时间达到3600s,超时自动报错,如下图所示: 可以看出(具体verbose执行计划如附件1所示),verbose执行计划中存在过多的NestLoop算子,一般情况下,该算子影响SQL语句执行性能,应该尽可能避免使用。通常可以利用语句 set [global] (enable_index_nestloop off) 来避免执行器走NestLoop算子。但有些场景下,该语句无法保证不使用NestLoop算子。因此,可以从另一方面入手解决这一问题,优化器因为对表估算不准确,故给出NestLoop算子的方案,可以利用tablescan这一hint对表进行全表扫描,以保证执行器走HashJoin算子而非NestLoop算子,从而提高语句执行性能。注意:在使用tablescan这个hint时要保证NestLoop算子涉及到的表都要加上 优化后的SQL语句如下所示: select * from ( select/*+tablescan(wp) tablescan(wo) tablescan(du) tablescan(ac) tablescan(u3) tablescan(u1) tablescan(u2) tablescan(tn) tablescan(dr) tablescan(u4) tablescan(t)*/ wo.work_order_id /*工单id*/, wo.work_order_code /*工单编码*/, wo.work_order_name /*工单名称*/, wo.work_order_level /*工单层级(第一层级(未拆分工单/父工单):1,第二层级(子工单):10)*/, decode(wo.work_order_level,1, '第一层级(未拆分工单/父工单)', 10,'第二层级(子工单)') as work_order_level_desc /*工单层级描述*/, substrb(wo.wo_description, 1, 1000) as wo_description /*工单描述*/, wo.wo_version /*工单版本号*/, wo.wo_lifecycle_status /*生命周期标识:0:正常工单,-1: 已删除*/, wo.business_id /*工单来源业务id*/, wo.business_type /*工单来源业务类型(10:活动流工单 20:手工派单 30:拆单工单 40:临时mos工单 50:ihub工单 60:ipmo工单 70:wbs工单 80:ncs工单 90:hr工单 100:ls工单 默认10)*/, decode( wo.business_type, '10', '活动流工单', '20', '手工派单', '30', '拆单工单', '40', '临时MOS工单', '50', 'ihub工单', '60', 'ipmo工单', '70', 'WBS工单', '80', 'NCS工单', '90', 'HR工单', '100', 'LS工单' ) as business_type_desc /*工单来源业务类型描述*/, wo.parent_activity_id /*父节点活动id*/, wo.activity_lib_id /*活动库活动id*/, wo.activity_type /*作业类型,1wbs,2活动,3里程碑*/, ac.activity_name /*活动名称*/, ac.std_ms_code as standard_ms_code /*标准里程碑编码*/, wo.plan_id /*计划id*/, wo.project_number as proj_num /*项目编码*/, wo.du_id /*交付单元id*/, wo.duration /*工期*/, wo.billing_flag /*开票标识:y-开票*/, wo.na_flag /*na标识*/, wo.inv_flag /*inv标识*/, wo.master_flag /*拆分标示,n:未拆分 ; y:已拆分*/, wo.created_by as created_by_id /*创建人user id*/, u1.lname as created_by /*创建人*/, wo.creation_date /*创建时间*/, wo.last_updated_by as last_updated_by_id /*最后更新人user id*/, u2.lname as last_updated_by /*最后更新人*/, wo.last_update_date /*最后更新时间*/, wp.wo_progress_id /*活动进度id*/, wp.expect_start_date /*预期开始日期*/, wp.expect_end_date /*预期结束日期*/, wp.plan_start_time /*计划开始时间*/, wp.plan_end_time /*计划完成时间*/, wp.actual_start_time /*实际开始时间*/, wp.actual_end_time /*实际完成时间*/, wp.close_time /*活动关闭时间*/, wp.completion_rate /*完工比率(数值如 0.8666)*/, to_char(substr(wp.remark, 1, 333)) as progress_description /*进度备注信息*/, wp.total_value /*总值*/, wp.accumulate_value /*累计值*/, wp.report_time /*值反馈时间*/, wp.total_plan_value /*总计划值*/, wp.ehs_risk /*高危活动类型*/, wp.delay_reason_id /*延迟原因id*/, substrb(ag.description, 1, 1000) as delay_reason_description /*延迟原因描述*/, wp.wo_status /*活动状态 psc_lookup_item_t_3220 classify_code = 'WO_STATUS_CODE'*/, l2.item_name as wo_status_desc /*活动状态描述*/,( case when lengthb(wp.approve_status) = 0 then null else wp.approve_status end ) :: number as approve_status /*审批状态 psc_lookup_item_t_3220 classify_code = 'WORK_ORDER_APPROVE_STATUS'*/, l3.item_name as approve_status_desc /*审批状态描述*/, wp.par_workorder_doc_flag /*父工单是否有交付件(y/n)*/, wp.deliverables_complete /*交付件上传状态 0:不涉及交付件 1:待上传交付件 2:交付件上传中,未上传完9:交付件已上传完*/, wp.revenue_trigger_status /*触发状态(0:未触发过 1:已触发 2:已触发,pc校验触发失败 3:pc触发成功)*/, wp.billing_status /*开票状态(空值:未触发过 1:已开票)*/, wp.frozen_flag /*冻结标识(y/n)*/, wp.mr_frozen_flag /*mr是否冻结站点要货通过更新实施计划刷新字段*/, wp.mr_status /*站点签状态 1未签收,2部分签收,3全部签收,10未签收,20部分签收,30全部签收,40部分超配置签收,50全部超配置签收 完工验状态 p:部分完成,f:全部完成*/, wp.tool_flag /*是否挂工具工单回写(y/n)*/, wp.split_cp_flag /*拆分施工计划标识 y已拆分 n未拆分*/, wp.mos_data_source /*站点签完工验状数据来源*/, wo.template_id /*模板id,例如活动流节点id*/, tfn.task_flow_id /*任务流id*/, tfn.task_flow_node_id /*活动流节点id*/, tfn.revenue_flag /*收入里程碑标识(y/n)*/, tfn.on_site /*是否现场*/, nvl(l1.item_name, tfn.owner_type) as owner_type /*责任方类型 客户/华为/分包商*/, tfn.subcon /*是否分包*/ /*产品域*/,case when wo.enable_flag = 'Y' and wp.enable_flag = 'Y' and wo.wo_lifecycle_status = 0 and nvl(du.enable_flag, 'Y') = 'Y' then 'Y' else 'N' end as enable_flag /*有效标识,y为有效n为失效*/, 'N' as del_flag /*删除标识 y为已删除*/, 3 as data_center_id /*数据中心id*/, tf.task_flow_code /*活动流编码 add by jwx528041 20200408*/, tfn.task_flow_node_code /*任务流节点编码 add by jwx528041 20200408*/, tfn.task_flow_node_name /*任务流节点名称 add by jwx528041 20200408*/, tfn.task_flow_node_type /*任务流节点类型 add by jwx528041 20200408*/, tfn.enable_flag as flow_enable_flag /*活动流有效标识 add by jwx528041 20200408*/, wo.tenant_code /*租户编码 add by jwx528041 20200408*/, tfn.activity_id /*活动流水号 add by jwx528041 20200408*/, tfn.lead_time /*持续时间 add by jwx528041 20200408*/, wo.resource_id as wo_actual_owner_id /*工单实际责任人id update by swx949890 202207*/, wo.resource_name as wo_actual_owner /*工单实际责任人 update by swx949890 202207*/, wo.contractor_id as wo_actual_owner_contr_id /*工单实际责任人分包商id update by swx949890 202207*/, wo.contractor_name as wo_actual_owner_contr_name /*工单实际责任人分包商名称 update by swx949890 202207*/, nvl(l4.item_name, tfn.delivery_model) as delivery_model /*工单交付模式 add by cwx613468 20200711*/, tfn.on_line_site /*是否上站 add by cwx613468 20200711*/, u3.lname as dispatcher_user_name /*调度人 add by cwx613468 20200711*/, tfn.approve_level_qty /*审批总层级 add by jwx528041 20200819*/, tf.task_flow_name /*活动流名称 add by jwx528041 20200819*/, tf.task_flow_type /*活动流类型 add by jwx528041 20200819*/, wp.source_code /*标识actual时间的修改来源,值为mobile标识从手机端回写 add by jwx528041 20200819*/, wp.plan_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.plan_update_time /*计划时间更新时间 add by jwx528041 20200819*/, wp.dispatch_time /*调度时间 add by jwx528041 20200819*/, wp.first_actual_update_time /*第一次实际开始时间填入时间 add by jwx528041 20200819*/, wp.first_actual_end_time /*第一次实际结束时间填入时间 add by jwx528041 20200819*/, wp.first_actual_updated_by /*第一次实际时间填入人user id add by jwx528041 20200819*/, wp.actual_start_update_time /*实际开始时间更新日期 add by jwx528041 20200819*/, wp.actual_start_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.actual_time_source /*实际完成时间更新来源 add by jwx528041 20200819*/, wp.actual_end_update_time /*实际完成时间更新日期 add by jwx528041 20200819*/, wp.actual_end_updated_by /*实际完成时间更新人user id add by jwx528041 20200819*/, wp.revenue_trigger_failed_msg /*收入触发失败原因 add by jwx528041 20200819*/, ag.souce_type as delay_reason_souce_type /*延迟原因数据来源:1、自定义 2、 add by jwx528041 20200819*/ --,ras.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type add by jwx528041 20200819*/ , wo.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type update by swx949890 202207*/, dr.resouce_type as wo_owner_resouce_type /*工单责任人资源类型 add by jwx528041 20200819*/, l5.item_name as wo_owner_resouce_type_desc /*工单责任人资源类型 add by jwx528041 20200819*/, u4.w3_account as wo_owner_w3_account /*工单责任人w3账号 add by jwx528041 20200819*/, rel.du_tf_rel_enable /*du与活动流关系有效性标识 y:有效 n:失效 add by lwx617215 20210116*/, t.billing_sla /*sla*/, t.billing_milestone /*开票里程碑*/, tf.required_tools, wp.active, gp.plan_code, gp.plan_name, gp.template_plan_id from sdisd.ogg_wo_work_order_2_3220 wo inner join sdisd.ogg_wo_progress_2_3220 wp on wo.work_order_id = wp.work_order_id left join sdisd.ogg_wo_task_flow_node_br_3220 tfn on wo.template_id = tfn.task_flow_node_id and nvl(wo.wo_version, 0) = case when nvl(wo.wo_version, 0) > 0 then tfn.version else tfn.wo_version end and wo.project_number = tfn.project_number left join sdisd.ogg_sds_activity_t_br_3220 ac on wo.activity_lib_id = ac.activity_id left join sdisd.ogg_sds_task_flow_t_br_3220 tf on tfn.task_flow_id = tf.task_flow_id left join sdisd.ogg_du_release_t_br_3220 du /*enable_flag新增有效du的判断 lwx617215 20210116*/ on wo.du_id = du.du_id left join sdisd.ogg_gcc_plan_2_3220 gp --dwx1189869 on wo.plan_id = gp.plan_id and gp.tenant_code = 'RolloutPlan' and gp.parent_plan_id = -1 and gp.enable_flag = 'Y' left join ( select r.du_id, r.task_flow_id, /*du与活动流有效标识*/ case when r.enable_flag = 'Y' and publish_flag = 'P' then 'Y' else 'N' end as du_tf_rel_enable, row_number() over( partition by r.du_id, r.task_flow_id order by r.last_update_date desc ) as rn from sdisd.ogg_rp_du_tf_release_3_3220 r ) rel on wo.du_id = rel.du_id and tfn.task_flow_id = rel.task_flow_id and rel.rn = 1 left join sdisd.ogg_tpl_user_t_3220 u1 on wo.created_by = u1.user_id left join sdisd.ogg_tpl_user_t_3220 u2 on wo.last_updated_by = u2.user_id left join sdisd.ogg_tpl_user_t_3220 u3 on wp.dispatcher_user_id = u3.user_id left join sdisd.ogg_sds_activity_gap_t_br_3220 ag on wp.delay_reason_id = ag.activity_gap_id left join sdisd.ogg_tpl_lookup_item_t_3220 l1 on tfn.owner_type = l1.item_code and l1.classify_code = 'SDS_TASK_OWNER_TYPE' and l1.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l2 on wp.wo_status = l2.item_code and l2.classify_code = 'WO_STATUS_CODE' and l2.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l3 on wp.approve_status = l3.item_code and l3.classify_code = 'WORK_ORDER_APPROVE_STATUS' and l3.language = 'en_US' left join sdisd.ogg_tpl_lookup_item_t_3220 l4 on tfn.delivery_model = l4.item_code and l4.classify_code = 'SDS_TASK_ON_SITE' and l4.language = 'en_US' left join sdisd.ogg_pm_project_tree_node_3220 tn on wo.resource_id = tn.tree_id left join sdisd.ogg_pm_delivery_resource_3220 dr on tn.resource_id = dr.resource_id left join sdisd.ogg_tpl_user_t_3220 u4 on dr.user_id = u4.user_id left join sdisd.ogg_tpl_lookup_item_t_3220 l5 on dr.resouce_type = l5.item_code and l5.classify_code = 'PM_RESOURCE_TYPE' and l5.language = 'zh_CN' left join sdisd.ogg_sds_task_flow_node_br_3220 t on tfn.task_flow_node_id = t.task_flow_node_id where ( wo.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or wp.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or tfn.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or ag.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 ) union all select wo.work_order_id /*工单id*/, wo.work_order_code /*工单编码*/, wo.work_order_name /*工单名称*/, wo.work_order_level /*工单层级(第一层级(未拆分工单/父工单):1,第二层级(子工单):10)*/, decode( wo.work_order_level, 1, '第一层级(未拆分工单/父工单)', 10, '第二层级(子工单)' ) as work_order_level_desc /*工单层级描述*/, substrb(wo.wo_description, 1, 1000) as wo_description /*工单描述*/, wo.wo_version /*工单版本号*/, wo.wo_lifecycle_status /*生命周期标识:0:正常工单,-1: 已删除*/, wo.business_id /*工单来源业务id*/, wo.business_type /*工单来源业务类型(10:活动流工单 20:手工派单 30:拆单工单 40:临时mos工单 50:ihub工单 60:ipmo工单 70:wbs工单 80:ncs工单 90:hr工单 100:ls工单 默认10)*/, decode( wo.business_type, '10', '活动流工单', '20', '手工派单', '30', '拆单工单', '40', '临时MOS工单', '50', 'ihub工单', '60', 'ipmo工单', '70', 'WBS工单', '80', 'NCS工单', '90', 'HR工单', '100', 'LS工单' ) as business_type_desc /*工单来源业务类型描述*/, wo.parent_activity_id /*父节点活动id*/, wo.activity_lib_id /*活动库活动id*/, wo.activity_type /*作业类型,1wbs,2活动,3里程碑*/, ac.activity_name /*活动名称*/, ac.std_ms_code as standard_ms_code /*标准里程碑编码*/, wo.plan_id /*计划id*/, wo.project_number as proj_num /*项目编码*/, wo.du_id /*交付单元id*/, wo.duration /*工期*/, wo.billing_flag /*开票标识:y-开票*/, wo.na_flag /*na标识*/, wo.inv_flag /*inv标识*/, wo.master_flag /*拆分标示,n:未拆分 ; y:已拆分*/, wo.created_by as created_by_id /*创建人user id*/, u1.lname as created_by /*创建人*/, wo.creation_date /*创建时间*/, wo.last_updated_by as last_updated_by_id /*最后更新人user id*/, u2.lname as last_updated_by /*最后更新人*/, wo.last_update_date /*最后更新时间*/, wp.wo_progress_id /*活动进度id*/, wp.expect_start_date /*预期开始日期*/, wp.expect_end_date /*预期结束日期*/, wp.plan_start_time /*计划开始时间*/, wp.plan_end_time /*计划完成时间*/, wp.actual_start_time /*实际开始时间*/, wp.actual_end_time /*实际完成时间*/, wp.close_time /*活动关闭时间*/, wp.completion_rate /*完工比率(数值如 0.8666)*/, to_char(substr(wp.remark, 1, 333)) as progress_description /*进度备注信息*/, wp.total_value /*总值*/, wp.accumulate_value /*累计值*/, wp.report_time /*值反馈时间*/, wp.total_plan_value /*总计划值*/, wp.ehs_risk /*高危活动类型*/, wp.delay_reason_id /*延迟原因id*/, substrb(ag.description, 1, 1000) as delay_reason_description /*延迟原因描述*/, wp.wo_status /*活动状态 psc_lookup_item_t_3220 classify_code = 'WO_STATUS_CODE'*/, l2.item_name as wo_status_desc /*活动状态描述*/,( case when lengthb(wp.approve_status) = 0 then null else wp.approve_status end ) :: number as approve_status /*审批状态 psc_lookup_item_t_3220 classify_code = 'WORK_ORDER_APPROVE_STATUS'*/, l3.item_name as approve_status_desc /*审批状态描述*/, wp.par_workorder_doc_flag /*父工单是否有交付件(y/n)*/, wp.deliverables_complete /*交付件上传状态 0:不涉及交付件 1:待上传交付件 2:交付件上传中,未上传完9:交付件已上传完*/, wp.revenue_trigger_status /*触发状态(0:未触发过 1:已触发 2:已触发,pc校验触发失败 3:pc触发成功)*/, wp.billing_status /*开票状态(空值:未触发过 1:已开票)*/, wp.frozen_flag /*冻结标识(y/n)*/, wp.mr_frozen_flag /*mr是否冻结站点要货通过更新实施计划刷新字段*/, wp.mr_status /*站点签状态 1未签收,2部分签收,3全部签收,10未签收,20部分签收,30全部签收,40部分超配置签收,50全部超配置签收 完工验状态 p:部分完成,f:全部完成*/, wp.tool_flag /*是否挂工具工单回写(y/n)*/, wp.split_cp_flag /*拆分施工计划标识 y已拆分 n未拆分*/, wp.mos_data_source /*站点签完工验状数据来源*/, wo.template_id /*模板id,例如活动流节点id*/, tfn.task_flow_id /*任务流id*/, tfn.task_flow_node_id /*活动流节点id*/, tfn.revenue_flag /*收入里程碑标识(y/n)*/, tfn.on_site /*是否现场*/, nvl(l1.item_name, tfn.owner_type) as owner_type /*责任方类型 客户/华为/分包商*/, tfn.subcon /*是否分包*/ /*产品域*/,case when wo.enable_flag = 'Y' and wp.enable_flag = 'Y' and wo.wo_lifecycle_status = 0 and nvl(du.enable_flag, 'Y') = 'Y' then 'Y' else 'N' end as enable_flag /*有效标识,y为有效n为失效*/, 'N' as del_flag /*删除标识 y为已删除*/, 4 as data_center_id /*数据中心id*/, tf.task_flow_code /*活动流编码 add by jwx528041 20200408*/, tfn.task_flow_node_code /*任务流节点编码 add by jwx528041 20200408*/, tfn.task_flow_node_name /*任务流节点名称 add by jwx528041 20200408*/, tfn.task_flow_node_type /*任务流节点类型 add by jwx528041 20200408*/, tfn.enable_flag as flow_enable_flag /*活动流有效标识 add by jwx528041 20200408*/, wo.tenant_code /*租户编码 add by jwx528041 20200408*/, tfn.activity_id /*活动流水号 add by jwx528041 20200408*/, tfn.lead_time /*持续时间 add by jwx528041 20200408*/, wo.resource_id as wo_actual_owner_id /*工单实际责任人id update by swx949890 202207*/, wo.resource_name as wo_actual_owner /*工单实际责任人 update by swx949890 202207*/, wo.contractor_id as wo_actual_owner_contr_id /*工单实际责任人分包商id update by swx949890 202207*/, wo.contractor_name as wo_actual_owner_contr_name /*工单实际责任人分包商名称 update by swx949890 202207*/, nvl(l4.item_name, tfn.delivery_model) as delivery_model /*工单交付模式 add by cwx613468 20200711*/, tfn.on_line_site /*是否上站 add by cwx613468 20200711*/, u3.lname as dispatcher_user_name /*调度人 add by cwx613468 20200711*/, tfn.approve_level_qty /*审批总层级 add by jwx528041 20200819*/, tf.task_flow_name /*活动流名称 add by jwx528041 20200819*/, tf.task_flow_type /*活动流类型 add by jwx528041 20200819*/, wp.source_code /*标识actual时间的修改来源,值为mobile标识从手机端回写 add by jwx528041 20200819*/, wp.plan_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.plan_update_time /*计划时间更新时间 add by jwx528041 20200819*/, wp.dispatch_time /*调度时间 add by jwx528041 20200819*/, wp.first_actual_update_time /*第一次实际开始时间填入时间 add by jwx528041 20200819*/, wp.first_actual_end_time /*第一次实际结束时间填入时间 add by jwx528041 20200819*/, wp.first_actual_updated_by /*第一次实际时间填入人user id add by jwx528041 20200819*/, wp.actual_start_update_time /*实际开始时间更新日期 add by jwx528041 20200819*/, wp.actual_start_updated_by /*实际开始时间更新人user id add by jwx528041 20200819*/, wp.actual_time_source /*实际完成时间更新来源 add by jwx528041 20200819*/, wp.actual_end_update_time /*实际完成时间更新日期 add by jwx528041 20200819*/, wp.actual_end_updated_by /*实际完成时间更新人user id add by jwx528041 20200819*/, wp.revenue_trigger_failed_msg /*收入触发失败原因 add by jwx528041 20200819*/, ag.souce_type as delay_reason_souce_type /*延迟原因数据来源:1、自定义 2、 add by jwx528041 20200819*/ --,ras.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type add by jwx528041 20200819*/ , wo.tree_type as wo_owner_tree_type /*工单责任人项目树节点类型tree_type update by swx949890 202207*/, dr.resouce_type as wo_owner_resouce_type /*工单责任人资源类型 add by jwx528041 20200819*/, l5.item_name as wo_owner_resouce_type_desc /*工单责任人资源类型 add by jwx528041 20200819*/, u4.w3_account as wo_owner_w3_account /*工单责任人w3账号 add by jwx528041 20200819*/, rel.du_tf_rel_enable /*du与活动流关系有效性标识 y:有效 n:失效 add by lwx617215 20210116*/, t.billing_sla /*sla*/, t.billing_milestone /*开票里程碑*/, tf.required_tools, wp.active, gp.plan_code, gp.plan_name, gp.template_plan_id from sdisd.ogg_wo_work_order17_3220 wo inner join sdisd.ogg_wo_progress17_3220 wp on wo.work_order_id = wp.work_order_id left join sdisd.ogg_wo_task_flow_node_za_3220 tfn on wo.template_id = tfn.task_flow_node_id and nvl(wo.wo_version, 0) = case when nvl(wo.wo_version, 0) > 0 then tfn.version else tfn.wo_version end and wo.project_number = tfn.project_number left join sdisd.ogg_sds_activity_t_za_3220 ac on wo.activity_lib_id = ac.activity_id left join sdisd.ogg_sds_task_flow_t_za_3220 tf on tfn.task_flow_id = tf.task_flow_id left join sdisd.ogg_du_release_t_za_3220 du /*enable_flag新增有效du的判断 lwx617215 20210116*/ on wo.du_id = du.du_id left join sdisd.ogg_gcc_plan17_3220 gp --dwx1189869 on wo.plan_id = gp.plan_id and gp.tenant_code = 'RolloutPlan' and gp.parent_plan_id = -1 and gp.enable_flag = 'Y' left join ( select r.du_id, r.task_flow_id, /*du与活动流有效标识*/ case when r.enable_flag = 'Y' and publish_flag = 'P' then 'Y' else 'N' end as du_tf_rel_enable, row_number() over( partition by r.du_id, r.task_flow_id order by r.last_update_date desc ) as rn from sdisd.ogg_rp_du_tf_release18_3220 r ) rel on wo.du_id = rel.du_id and tfn.task_flow_id = rel.task_flow_id and rel.rn = 1 left join sdisd.ogg_tpl_user_t_3220 u1 on wo.created_by = u1.user_id left join sdisd.ogg_tpl_user_t_3220 u2 on wo.last_updated_by = u2.user_id left join sdisd.ogg_tpl_user_t_3220 u3 on wp.dispatcher_user_id = u3.user_id left join sdisd.ogg_sds_activity_gap_t_za_3220 ag on wp.delay_reason_id = ag.activity_gap_id left join sdisd.ogg_tpl_lookup_item_t_3220 l1 on tfn.owner_type = l1.item_code and l1.classify_code = 'SDS_TASK_OWNER_TYPE' and l1.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l2 on wp.wo_status = l2.item_code and l2.classify_code = 'WO_STATUS_CODE' and l2.language = 'en_US' left join sdisd.ogg_psc_lookup_item_t_3220 l3 on wp.approve_status = l3.item_code and l3.classify_code = 'WORK_ORDER_APPROVE_STATUS' and l3.language = 'en_US' left join sdisd.ogg_tpl_lookup_item_t_3220 l4 on tfn.delivery_model = l4.item_code and l4.classify_code = 'SDS_TASK_ON_SITE' and l4.language = 'en_US' left join sdisd.ogg_pm_project_tree_node_3220 tn on wo.resource_id = tn.tree_id left join sdisd.ogg_pm_delivery_resource_3220 dr on tn.resource_id = dr.resource_id left join sdisd.ogg_tpl_user_t_3220 u4 on dr.user_id = u4.user_id left join sdisd.ogg_tpl_lookup_item_t_3220 l5 on dr.resouce_type = l5.item_code and l5.classify_code = 'PM_RESOURCE_TYPE' and l5.language = 'zh_CN' left join sdisd.ogg_sds_task_flow_node_za_3220 t on tfn.task_flow_node_id = t.task_flow_node_id where ( wo.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or wp.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or tfn.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 or ag.cdc_create_date >= to_timestamp('20231021', 'yyyy-mm-dd hh24:mi:ss.ff') -1 / 24 / 60 )) as t limit 10 如下图所示,该语句执行时间降为27s+,提升了语句的执行性能。 具体的performance执行计划如附件2所示。 附件:tablescan-performance.txt 附件:tablescan-verbose.txt 点击关注,第一时间了解华为云新鲜技术~

优秀的个人博客,低调大师

带你走进数仓大集群内幕丨详解关于作业hang及残留问题定位

本文分享自华为云社区《【带你走进DWS大集群内幕】大集群通信:作业hang、残留问题定位》,作者: 雨落天穹丶。 前言: 测试过程中,我们会遇到这样一种情况,我的作业都执行很久了,为啥还不结束,是不是作业hang掉了? 或者说,明明看到CN上的作业都没了,为什么通过全局视图发现DN上还有作业在执行而没有退出,这是不是有问题啊?那么就带着这样的疑问点来阅读本篇分析问题的方式方法,给初学者一点定位思路。 【通信系统视图】 pgxc_comm_send_stream :展示所有DN上的通信库发送流状态。 pgxc_comm_recv_stream :展示所有DN上的通信库接收流状态。 pg_thread_wait_status :通过PG_THREAD_WAIT_STATUS视图可以检测当前实例中工作线程(backend thread)以及辅助线程(auxiliary thread)的阻塞等待情况。 pgxc_thread_wait_status :通过CN节点查看PGXC_THREAD_WAIT_STATUS视图,可以查看集群全局各个节点上所有SQL语句产生的线程之间的调用层次关系,以及各个线程的阻塞等待状态,从而更容易定位进程停止响应问题以及类似现象的原因。 pg_stat_activity :PG_STAT_ACTIVITY视图显示和当前用户查询相关的信息。若有管理员权限或预置角色权限可以显示和所有用户查询相关的信息。 pgxc_stat_activity :PGXC_STAT_ACTIVITY视图显示当前集群下所有CN的当前用户查询相关的信息。 单实例查询:可直连DN/CN 通过此条SQL进行DN/CN上活跃会话访问查询。 select datname,usename,pid,query_id,query_start,query from pg_stat_activity where state='active' order by query_start; 全局查询: 获取所有的CN上当前的作业执行情况 select coorname,datname,usename,pid,query_id,query_start,query from pgxc_stat_activity where state='active' order by query_start; 【发现问题】1. 集群作业停止一段时间后,发现集群DN还存在比较高的压力CPU,查询活跃会话视图,观察是否存在未执行完的作业,或者只是主备DN数据同步(同步属于正常情况,但持续时间过长,就要分析DN HA catchup 机制是否正常) 例如,如下查询到的结果可以观察到,当前时间19:38,作业已经退出很久了,发现在linux0802集群的6606端口DN上还存在16:49分的作业还在执行,且通过作业分析发现,该作业是简单的 create table as select * 场景 ,正常测试返回结果在10s之内,那当前作业还存在于此DN上处于active执行状态,就显得很不正常。 那我们就针对这条存在的活跃会话进行分析。 【分析问题】通过全局会话等待视图观察这个作业当前的执行状态 获取上图中的执行query的 query_id 通过query_id查询作业执行状态: select * from pgxc_thread_wait_status where query_id='219269006857675600' order by 1,2; -- 通过上面查出query_id分析 查询结果发现所有的线程都是在获取stream连接状态,且已经不存在与CN的连接信息 那我们找一个DN节点查看堆栈:选取 dn_6007_6008 lwtid = 2238755 通过cm_ctl query -Cv 查找dn 6007 所在的物理节点环,观察当前主DN是 6007 还是 6008(一般业务连接都是主DN,如果发生主备切换,那么我们查看堆栈就要到6008节点[当前的主DN节点]) 连接到到所在物理节点, 用gstack 2238755 (多执行几遍,如果栈内容一直不变,可能就是hang,需要分析位,如果栈一直在变可能是在工作,确实是sql执行慢,可联系开发帮忙确认分析是不是正常堆栈) 【问题初步结论】初步分析堆栈代码属于通信相关,找对应的责任田定位问题。 【发现问题】2. 集群作业停止一段时间后,发现集群CN上还存在未执行完的作业,但是DN上的作业连接已经全部退出 还是通过上面的例子中的查询活跃会话的语句,分析找到对应的query id。 通过线程等待视图查询当前作业执行情况。 select * from pgxc_thread_wait_status where query_id='237283405371509247' order by 1,2; 上图中当前只剩下CN线程,DN线程全部退出,那么这种情况就需要去分析CN对应等待状态中的dn 6301实例上的对应远端线程在干什么 通过通信pooler视图查询对应的remote node 线程号: select * from pg_pooler_status where pid=140069915589224; 然后用线程等待视图查询:select * from pgxc_thread_wait_status where pid = 140704341249976 order by 1,2; 可以按上面第一种类型的查询DN节点操作 访问dn_6301_6302 主DN 所在的物理机器节点 用 gstack 39625 观察堆栈。 【问题初步结论】 初步判断是DN在等主备同步呢,此时DN上的query id清0了,用query_id匹配不到,后续联系对应责任田分析。 点击关注,第一时间了解华为云新鲜技术~

优秀的个人博客,低调大师

OLAP分析型应用场景中,数仓中vacuum为何对列存表无效

摘要:对列存表执行vacuum为什么是无效的呢?其实这与列存表的存储结构以及数据写入方式有关。 本文分享自华为云社区《GaussDB(DWS)中vacuum为何对列存表无效?【这次高斯不是数学家】》,作者: i云上小白。 在OLAP分析型应用场景中,列式存储有十分明显的优势,相比行式存储,其高压缩比、高I/O效率及批量数据运算的特性极大提升了统计分析查询的效率。虽然存储模式不同,对列存表进行频繁的插入(insert)和更新(update)操作仍然会导致空间膨胀问题,实际上在很多时候,往往不建议对列存表进行数据更新和非批量方式的数据插入。 基于GaussDB(DWS)的mvcc机制,行存表在删除、更新数据时会保留原来的旧数据,这些数据我们称之为死元组(dead tuple)。频繁地对行存表执行删除和更新操作会导致数据页中产生大量的死元组,不但使存储空间膨胀,而且降低了对表的查询效率。针对这一现象,GaussDB(DWS)采用vacuum机制来清除不需要的死元组及索引,释放空间。但对于列存表来说,vacuum是无效的,必须使用vacuum full才能有效回收空间。 当然对行存表执行vacuum回收空间是有限的,在某些情况下vacuum后的表大小甚至不会有丝毫变化。这是因为如果删除的记录位于表的末端,其所占用的空间将会被物理释放并归还操作系统,而如果不是末端数据,会将表中或索引中dead tuple所占用的空间置为可用状态,从而复用这些空间。 对列存表执行vacuum为什么是无效的呢?其实这与列存表的存储结构以及数据写入方式有关。 从下图中可知,列存表的最小存储单元是CU(Compress Unit),每个CU的大小为8KB的整数倍(需要注意的是,CU并不是由页组成的,它是一个独立的存储单元),最多存储1列60000行数据。同一列的多个CU连续存放在一个数据文件中,当数据文件的大小超过1G,会自动切换到新的文件中。 除此之外,每个列存表还有一个记录CU的辅助和管理信息的行存表CUDesc表,该表中的每一行记录对应一个CU,包括最大值/最小值、数据条数,以及CU在文件中的偏移量及大小。其中,col_id=-10的行为VCU, cu_pointer记录这一组CU(cu_id相同)中哪些行被删除。另外,CU的可见性也是通过CUDesc的可见性来决定的。 当在列存表上导入数据时,首先数据会按列导入CU cache,如果设置了PCK(Partial Cluster Key),导入数据会按照指定列进行局部排序(默认420万条数据进行排序),最后再生成CU(生成CU时,会根据数据类型进行压缩),并写入文件。列存表推荐使用批量方式导入数据,如insert into select/copy、GDS、SQL on Hadoop/OBS等,这样可以充分利用CU空间,以及使用PCK索引。单行数据插入会产生较多的小CU文件,不但会造成空间浪费,还会导致访问效率降低。因此对于列存表的数据导入,强烈推荐使用批量方式。 下面我们看看列存表上的删除和更新操作是如何进行的。 在列存表上进行delete时,首先会根据删除条件找到需要删除的行ctid(cu_id,offset),然后对需要删除的行ctid去重(每420万行排序去重),最后在行对应的VCU的delete map上打上删除标记。至于update操作,实际上是一个delete+insert(append)操作。首先根据更新条件找到更新的行,打上删除标记(ctid需去重),然后将原来整行更新相应数据后,插入到新CU中。 对行存表来说,数据页中的每个元组都占用了一块独立的空间,每个元组有一个行指针,记录了这个元组的状态。当执行update或者delete操作后,死元组的行指针lp_flags的状态会被标记为3: LP_DEAD,即死亡状态,等待vacuum清理。倘若执行了vacuum,指针状态会被标记为0: LP_UNUSED,即未使用状态,表示该元组占用的空间可以被复用。 虽然列存表可以像行存表那样对被删除或者更新前的数据进行标记,但由于CU中的数据是按列连续存放的,CU生成数据后固定不可更改。如果使用指针对每一个数据位的状态进行标记,其代价较大(行指针长度几乎等于数据长度,同时I/O开销巨大),而且数据写入CU采用的是追写(append)方式,即便死元组被标记为可复用状态,也无法再使用这些空间。 另外,列存表的读取是以CU为单位,真正影响列存表性能的是CU文件数量,而vacuum即便可以回收死元组占用的空间,却不能合并小CU文件。因此,对列存表来说,vacuum是无效的。此时,可以使用vacuum full整理CU碎片,合并小CU文件,提升性能。 点击关注,第一时间了解华为云新鲜技术~

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

用户登录
用户注册