首页 文章 精选 留言 我的

精选列表

搜索[数仓建模],共10009篇文章
优秀的个人博客,低调大师

在 KubeSphere 中开启新一代云原生数仓 Databend

作者:尚卓燃(https://github.com/PsiACE),Databend 研发工程师,Apache OpenDAL (Incubating) PPMC。 前言 Databend 是一款完全面向云对象存储的新一代云原生数据仓库,专为弹性和高效设计,为您的大规模分析需求保驾护航。Databend 同时是一款符合 Apache-2.0 协议的开源软件,除了访问云服务(https://app.databend.com/)之外,用户还可以自己部署 Databend 生产集群以满足工作负载需要。 Databend 的典型使用场景包括: 实时分析平台,日志的快速查询与可视化。 云数据仓库,历史订单数据的多维度分析和报表生成。 混合云架构,统一管理和处理不同来源和格式的数据。 成本和性能敏感的 OLAP 场景,动态调整存储和计算资源。 KubeSphere 是在 Kubernetes 之上构建的以应用为中心的多租户容器平台,提供全栈的 IT 自动化运维的能力,可以管理多个节点上的容器化应用,提供高可用性、弹性扩缩容、服务发现、负载均衡等功能。 利用 KubeSphere 部署和管理 Databend 具有以下优点: 使用 Helm Charts 部署 Databend 集群,简化应用管理、部署过程和参数设置。 利用 Kubernetes 的特性来实现 Databend 集群的自动恢复、水平扩展、负载均衡等。 与 Kubernetes 上的其他服务或应用轻松集成和交互,如 MinIO、Prometheus、Grafana 等。 本文将会介绍如何使用 KubeSphere 创建和部署 Databend 高可用集群,并使用 QingStor 作为底层存储服务。 配置对象存储 对象存储是一种存储模型,它把数据作为对象来管理和访问,而不是文件或块。对象存储的优点包括:可扩展性、低成本、高可用性等。 Databend 完全面向对象存储而设计,在减少复杂性和成本的同时提高灵活性和效率。Databend 支持多种对象存储服务,如 AWS S3、Azure Blob、Google Cloud Storage、HDFS、Alibaba Cloud OSS、Tencent Cloud COS 等。您可以根据业务的需求和偏好选择合适的服务来存放你的数据。 这里我们以青云 QingStor 为例,介绍与 S3 兼容的对象存储相关配置的预先准备工作。 创建 Bucket 对象存储服务(QingStor)提供了一个无限容量的在线文件存储和访问平台。每个用户可创建多个存储空间(Bucket);您可以将任意类型文件通过控制台或 QingStor API 上传至一个存储空间(Bucket)中;存储空间(Bucket)支持访问控制,您可以将自己的存储空间(Bucket)开放给指定的用户,或所有用户。 登录青云控制台,选中对象存储服务,新建用于验证的 bucket 。 需要关注的是 bucket 的名字 <bucket> 和其所在的可用区 <region>。 由于这里使用 s3 兼容服务,所以最后连接的 endpoint_url 是 s3.<bucket>.<region>.qingstor.com 。 创建 API 密钥 API 密钥(Access Key)可以让您通过发送 API 指令来访问青云的服务。API 密钥 ID 须作为参数包含在每一个请求中发送;而 API 密钥的私钥负责生成 API 请求串的签名,私钥需要被妥善保管,切勿外传。默认所有 IP 地址都可使用此密钥调用 API,设置 IP 白名单后只有白名单范围内的 IP 地址才可使用此密钥。 点击右上方菜单,选中 API 密钥,创建新的密钥用于 API 访问。 下载文件中的 qy_access_key_id 对应 access_key_id ,qy_secret_access_key 对应 secret_access_key 。 准备 KubeSphere 环境 KubeSphere(https://kubesphere.io)是在 Kubernetes 之上构建的开源容器平台,提供全栈的 IT 自动化运维的能力,简化企业的 DevOps 工作流。KubeSphere 已被海内外数万家企业采用。此外, KubeSphere 还拥有极为开放的生态,KubeSphere 在 OpenPitrix 的基础上,为用户提供了一个基于 Helm 的应用商店,用于应用生命周期管理。KubeSphere 应用商店让 ISV、开发者和用户能够在一站式服务中只需点击几下就可以上传、测试、安装和发布应用。目前 Databend 已入驻 KubeSphere 应用商店。 KubeSphere 环境搭建 All-in-One 模式部署测试环境 参考官方文档 。 在 Azure 上 Spot 一台机器: Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 5.4.0-1089-azure x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Tue Sep 6 02:09:16 UTC 2022 System load: 0.15 Processes: 376 Usage of /: 4.8% of 28.89GB Users logged in: 0 Memory usage: 0% IP address for eth0: 10.0.0.4 Swap usage: 0% 以 All-In-One 模式部署: 注意,需要在 root 下运行。 apt install socat conntrack containerd systemctl daemon-reload systemctl enable --now containerd curl -sfL https://get-kk.kubesphere.io | VERSION=v3.0.2 sh - chmod +x kk ./kk create cluster --with-kubernetes v1.22.12 --with-kubesphere v3.3.1 +------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------------+------------+-------------+------------------+--------------+ | name | sudo | curl | openssl | ebtables | socat | ipset | ipvsadm | conntrack | chrony | docker | containerd | nfs client | ceph client | glusterfs client | time | +------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------------+------------+-------------+------------------+--------------+ | ks | y | y | y | y | y | | | y | y | | 1.5.9-0ubuntu3.1 | | | | UTC 02:53:56 | +------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------------+------------+-------------+------------------+--------------+ 如果提示依赖缺失,可以根据需要安装,sudo apt install <name> ,这里只安装前两个。 Kubernetes Version ≥ 1.18 socat Required conntrack Required ebtables Optional but recommended ipset Optional but recommended ipvsadm Optional but recommended 访问 KubeSphere 控制面板。 执行下面命令查看关于登录的信息: Collecting installation results ... ##################################################### ### Welcome to KubeSphere! ### ##################################################### Console: http://10.0.0.4:30880 Account: admin Password: P@88w0rd NOTES: 1. After you log into the console, please check the monitoring status of service components in "Cluster Management". If any service is not ready, please wait patiently until all components are up and running. 2. Please change the default password after login. ##################################################### https://kubesphere.io 2022-09-06 15:41:44 ##################################################### 访问 30880 端口,并使用用户名密码登录,就可以访问 KubeSphere 。为确保能够访问 KubeSphere 和其他服务,请根据实际情况在云平台控制面板为相应端口添加入站出站规则。 KubeSphere Cloud 创建演示环境 创建轻量集群服务: 注册并登录 https://kubesphere.cloud 之后,可以轻松创建轻量集群服务。 使用默认配置创建免费版集群即可尝鲜体验,个人用户每月有 10 小时免费额度。 访问 KubeSphere 控制面板。 点击进入 KubeSphere,使用临时帐号密码登录。 插件启用 登录后的界面,如下图所示: 如需使用应用商店,可以参考 KubeSphere 文档 - 在安装后启用应用商店 启用。 开启后可以在应用商店中搜索找到 Databend ,结果类似下图。 企业空间与项目管理 点击平台管理进入访问控制页面,选中企业空间,点击创建,在名称一栏填写你想使用的名称,比如 databend。 在侧边栏选中项目,点击创建,分别创建为 databend-meta 和 databend-query 准备的项目。创建后效果如图所示: 部署 Databend 应用模板载入 虽然应用商店中已经有 Databend 可供选用,但版本较旧(v0.8.122-nightly),新的 PR(v1.0.3-nightly)需要等合并之后才可用,所以建议添加 Databend 官方维护的 helm-charts 作为应用模板。 Databend 官方提供了 Helm Charts ,而 KubeSphere 也支持使用 Helm Charts 应用模板。 应用模板是用户上传、交付和管理应用的一种方式。一般来说,根据一个应用的功能以及与外部环境通信的方式,它可以由一个或多个 Kubernetes 工作负载(例如部署、有状态副本集和守护进程集)和服务组成。作为应用模板上传的应用基于 Helm 包构建。 可以将 Helm Chart 交付至 KubeSphere 的公共仓库,或者导入私有应用仓库来提供应用模板。 https://kubesphere.io/zh/docs/v3.3/workspace-administration/upload-helm-based-application/ 在企业空间侧边栏选中 应用管理 ,点击 应用仓库 ,添加 Databend 官方维护的 Helm Charts 。 待状态变为成功后,就可以基于模板安装部署新的 Databend 应用。 Databend 部署模型 参考文档。 典型的 Databend 集群架构如下图所示,需要分别部署多个 Meta 和 Query 节点: 在集群模式下部署 Databend 时,首先需要启动一个 Meta节点,然后设置并启动其他 Meta 节点以加入第一个 Meta 节点,形成集群。在成功启动所有 Meta 节点后,逐个启动 Query 节点。每个 Query 节点在启动后自动注册到 Meta 节点以形成集群。 Meta 高可用集群部署 选中 databend-meta 项目。点击侧边栏应用负载,选中应用。点击创建,并选中从应用模板。 下拉栏中选中之前添加的 Databend ,效果如图: 选中 databend-meta,点击安装,设定应用名称及版本,我们推荐总是使用最新版本,以获得更好的体验。 使用示例设置,创建 3 副本的 databend-meta 节点形成集群。生产环境下推荐至少使用 3 副本高可用集群,可以参考 Databend 官方文档进行配置。 bootstrap: true replicaCount: 3 persistence: size: 5Gi # 考虑到宿主机资源有限,仅供示范 serviceMonitor: enabled: true Query 集群部署 在 Meta 节点的所有副本就绪之后,就可以开始部署 Query 集群。 Query 节点部署的前置步骤与 Meta 节点类似。进入 databend-query 项目,仿照之前的步骤选中 databend-query 应用模板进行创建即可。 配置中需要关注的部分是: databend-meta 连接:这里的地址取决于之前部署的 Meta 集群的相关信息。 存储方式:本示例连接的是 QingStor ,使用 S3 兼容协议,所以需要特别关注 endpoint_url 。 内置用户创建:创建一个名为 databend 密码为 databend 的内置用户,以方便在非 localhost 情况下访问。 这里启动的是一个单副本的 Query 集群,实际情况下可以根据工作负载规模灵活调整。 replicaCount: 1 config: query: clsuterId: default # add builtin user users: - name: databend # available type: sha256_password, double_sha1_password, no_password, jwt authType: double_sha1_password # echo -n "databend" | sha1sum | cut -d' ' -f1 | xxd -r -p | sha1sum authString: 3081f32caef285c232d066033c89a78d88a6d8a5 meta: # Set endpoints to use remote meta service # depends on previous deployed meta service、namespace and nodes endpoints: - "databend-meta-0.databend-meta.databend-meta.svc:9191" - "databend-meta-1.databend-meta.databend-meta.svc:9191" - "databend-meta-2.databend-meta.databend-meta.svc:9191" storage: # s3, oss type: s3 s3: bucket: "<bucket>" endpoint_url: "https://s3.<region>.qingstor.com" # for qingstor access_key_id: "<key>" secret_access_key: "<secret>" # [recommended] enable monitoring service serviceMonitor: enabled: true # [recommended] enable access from outside cluster service: type: LoadBalancer KubeSphere 监控 KubeSphere 观测工作负载 待状态变为运行中即可,这时可以很方便使用 KubeSphere 观测工作负载。 资源状态 databend-meta databend-query 监控 databend-meta databend-query 可访问性测试 节点状态检测 如果是在 All-in-One 模式下部署,我们可以轻松使用容器组 IP 地址来测试节点状态。 psiace@ks:~$ curl 10.233.107.113:8080/v1/health {"status":"pass"} 而使用 KubeSphere Cloud 部署时,可以在 KubeSphere Cloud 控制面板,选择网络以创建访问规则。 这里以 8080(Admin API)和 8000(Query HTTP Handler)端口为例: 创建后的结果如下图所示: 同样我们可以使用 curl 来检查节点状态。 psiace@ks:~$ curl https://admin-gfkyzxaz.c.kubesphere.cloud:30443/v1/health {"status":"pass"} 执行查询 bendsql 是一个十分方便的命令行界面工具,可以帮助您顺畅高效地使用 Databend 。bendsql 也支持连接 Databend Cloud ,管理计算集群和运行 SQL 查询。 安装 bendsql $ go install github.com/databendcloud/bendsql/cmd/bendsql@latest 连接 databend 集群(以 KubeSphere Cloud 为例) $ bendsql connect -H query-gfkyzxaz.c.kubesphere.cloud -P 30443 -u databend -p databend --ssl Connected to Databend on Host: query-gfkyzxaz.c.kubesphere.cloud Version: DatabendQuery v0.9.57-nightly-df858a1(rust-1.68.0-nightly-2023-03-01T01:23:11.56066902Z) 尝试执行查询 $ bendsql query Connected with driver databend (DatabendQuery v0.9.57-nightly-df858a1(rust-1.68.0-nightly-2023-03-01T01:23:11.56066902Z)) Type "help" for help. dd:databend@query-gfkyzxaz/default=> SELECT avg(number) FROM numbers(1000); +-------------+ | avg(number) | +-------------+ | 499.5 | +-------------+ (1 row) 总结 本文介绍了如何使用 KubeSphere 创建和部署 Databend 高可用集群,后端存储服务采用 QingStor ,最后使用 bendsql 演示连接集群和执行查询。 本文由博客一文多发平台 OpenWrite 发布!

优秀的个人博客,低调大师

数仓实践丨常量标量子查询做全连接导致整体慢

本文分享自华为云社区《GaussDB(DWS)性能调优:常量标量子查询做全连接导致整体慢》,作者: Zawami 。 问题描述 由于SQL中存在标量子查询同另一查询做笛卡尔积使SQL整体慢。标量子查询,即结果集只有一行一列的子查询。这里导致的SQL语句执行慢不只是在于做笛卡尔积慢,也会使后续聚合更慢。 原始语句 WITH TMP AS( SELECT case when length('[“202309“]') = 6 then '[“202309“]' || '01' WHEN length('[“202309“]') <> 8 THEN TO_CHAR(CURRENT_DATE, 'YYYYMMDD') END AS V_DATE from DUAL ) SELECT BG_CODE, BG_CN_NAME, BG_EN_NAME, METRIC_CODE --指标ID , METRIC_CN_NAME --指标中文名称 , METRIC_EN_NAME --指标英文名称 , CURRENCY --币种 , OVERSEAS_FLAG, REGION_CODE, REGION_CN_NAME, REGION_EN_NAME, REPOFFICE_CODE, REPOFFICE_CN_NAME, REPOFFICE_EN_NAME, OFFICE_CODE, OFFICE_CN_NAME, OFFICE_EN_NAME, REGION_CUSTCATG_CODE, REGION_CUSTCATG_CN_NAME, REGION_CUSTCATG_EN_NAME, TOP_CUST_CATEGORY_CODE, TOP_CUST_CATEGORY_EN_NAME, TOP_CUST_CATEGORY_CN_NAME, ACCTCUST_HQ_CODE, ACCTCUST_HQ_CN_NAME, ACCTCUST_HQ_EN_NAME, ACCTCUST_BRANCH_CODE, ACCTCUST_BRANCH_CN_NAME, ACCTCUST_BRANCH_EN_NAME, ACCTCUST_SUBSIDIARY_CODE, ACCTCUST_SUBSIDIARY_CN_NAM, ACCTCUST_SUBSIDIARY_EN_NAM, COUNTRY_CODE --新增加入参 , COUNTRY_CN_NAME --新增加入参 , COUNTRY_EN_NAME --新增加入参 , AGREE_AMOUNT --BUSI_DSCT_00001 总优惠 , AGREE_REMAIN_AMOUNT --BUSI_DSCT_00002 即期优惠 , SIGN_AMOUNT --BUSI_DSCT_00003 即期优惠/一次性优惠 , USE_AMOUNT --BUSI_DSCT_00004 即期优惠/单价量折扣 , NOT_USED_VALID_AMOUNT --BUSI_DSCT_00005 延期优惠 , NOT_USED_INVALID_AMOUNT --BUSI_DSCT_00006 voucher , NEW_SIGN_AMOUNT --BUSI_DSCT_00007 其他延期优惠 , NEW_USE_AMOUNT --BUSI_DSCT_00008 本月新使用金额 , EXPIRED_AMOUNT --BUSI_DSCT_00009 本月已过期金额 , IMMED_EXPIRED_AMOUNT --BUSI_DSCT_00010 半年内即将过期金额 FROM ( SELECT C.BG_CODE, C.BG_CN_NAME, C.BG_EN_NAME, C.M_ID AS METRIC_CODE --指标ID , C.M_CN AS METRIC_CN_NAME --指标中文名称 , C.M_EN AS METRIC_EN_NAME --指标英文名称 , C.CURRENCY_CODE AS CURRENCY --币种 ,CASE WHEN 1 = 0 THEN C.OVERSEA_FLAG ELSE NULL END AS OVERSEAS_FLAG,CASE WHEN 1 = 0 THEN C.REGION_CODE ELSE NULL END AS REGION_CODE,CASE WHEN 1 = 0 THEN C.REGION_CN_NAME ELSE NULL END AS REGION_CN_NAME,CASE WHEN 1 = 0 THEN C.REGION_EN_NAME ELSE NULL END AS REGION_EN_NAME,CASE WHEN 1 = 0 THEN C.REPOFFICE_CODE ELSE NULL END AS REPOFFICE_CODE,CASE WHEN 1 = 0 THEN C.REPOFFICE_CN_NAME ELSE NULL END AS REPOFFICE_CN_NAME,CASE WHEN 1 = 0 THEN C.REPOFFICE_EN_NAME ELSE NULL END AS REPOFFICE_EN_NAME,CASE WHEN 1 = 0 THEN C.OFFICE_CODE ELSE NULL END AS OFFICE_CODE,CASE WHEN 1 = 0 THEN C.OFFICE_CN_NAME ELSE NULL END AS OFFICE_CN_NAME,CASE WHEN 1 = 0 THEN C.OFFICE_EN_NAME ELSE NULL END AS OFFICE_EN_NAME,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_CODE ELSE NULL END AS REGION_CUSTCATG_CODE,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_CN_NAME ELSE NULL END AS REGION_CUSTCATG_CN_NAME,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_EN_NAME ELSE NULL END AS REGION_CUSTCATG_EN_NAME,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_CODE ELSE NULL END AS TOP_CUST_CATEGORY_CODE,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_EN_NAME ELSE NULL END AS TOP_CUST_CATEGORY_EN_NAME,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_CN_NAME ELSE NULL END AS TOP_CUST_CATEGORY_CN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_CODE ELSE NULL END AS ACCTCUST_HQ_CODE,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_CN_NAME ELSE NULL END AS ACCTCUST_HQ_CN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_EN_NAME ELSE NULL END AS ACCTCUST_HQ_EN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_CODE ELSE NULL END AS ACCTCUST_BRANCH_CODE,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_CN_NAME ELSE NULL END AS ACCTCUST_BRANCH_CN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_EN_NAME ELSE NULL END AS ACCTCUST_BRANCH_EN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_CODE ELSE NULL END AS ACCTCUST_SUBSIDIARY_CODE,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_CN_NAM ELSE NULL END AS ACCTCUST_SUBSIDIARY_CN_NAM,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_EN_NAM ELSE NULL END AS ACCTCUST_SUBSIDIARY_EN_NAM,CASE WHEN 1 = 0 THEN C.COUNTRY_CODE ELSE NULL END AS COUNTRY_CODE --新增加入参 ,CASE WHEN 1 = 0 THEN C.COUNTRY_CN_NAME ELSE NULL END AS COUNTRY_CN_NAME --新增加入参 ,CASE WHEN 1 = 0 THEN C.COUNTRY_EN_NAME ELSE NULL END AS COUNTRY_EN_NAME --新增加入参 , SUM(C.AGREE_AMOUNT) AS AGREE_AMOUNT --协议金额 , SUM(C.AGREE_REMAIN_AMOUNT) AS AGREE_REMAIN_AMOUNT --协议剩余金额 , SUM(C.SIGN_AMOUNT) AS SIGN_AMOUNT --可用金额 , SUM(C.USE_AMOUNT) AS USE_AMOUNT --已使用金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND NVL( C.EXPIRED_DATE, add_months(C.EFFECTIVE_DATE, C.VALID_MONTH) ) >= to_date(T.V_DATE, 'yyyymmdd') THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE >= to_date(T.V_DATE, 'yyyymmdd') THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS NOT_USED_VALID_AMOUNT --未使用金额(有效期外) , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' THEN C.SIGN_AMOUNT ELSE C.EFFECTIVE_TOTAL_AMOUNT END - CASE WHEN C.DSCT_TYPE = 'VOUCHER' THEN C.USE_AMOUNT ELSE C.USED_TOTAL_AMOUNT END - CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND NVL( C.EXPIRED_DATE, add_months(C.EFFECTIVE_DATE, C.VALID_MONTH) ) >= to_date(T.V_DATE, 'yyyymmdd') THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE >= to_date(T.V_DATE, 'yyyymmdd') THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS NOT_USED_INVALID_AMOUNT --未使用金额(有效期内) , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EXPIRED_DATE >= to_date(substr(T.V_DATE, 1, 6), 'yyyymm') and C.EXPIRED_DATE <= LAST_DAY(to_date(T.V_DATE, 'yyyymmdd')) THEN C.SIGN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_START_DATE >= to_date(substr(T.V_DATE, 1, 6), 'yyyymm') and C.DSCT_START_DATE <= LAST_DAY(to_date(T.V_DATE, 'yyyymmdd')) THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS NEW_SIGN_AMOUNT --本月新增可用金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EFFECTIVE_DATE >= to_date(substr(T.V_DATE, 1, 6), 'yyyymm') and C.EFFECTIVE_DATE <= LAST_DAY(to_date(T.V_DATE, 'yyyymmdd')) THEN C.USE_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_START_DATE >= to_date(substr(T.V_DATE, 1, 6), 'yyyymm') and C.DSCT_START_DATE <= LAST_DAY(to_date(T.V_DATE, 'yyyymmdd')) THEN C.USED_TOTAL_AMOUNT ELSE NULL END ) AS NEW_USE_AMOUNT --本月新使用金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EXPIRED_DATE < to_date(T.V_DATE, 'yyyymmdd') THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE < to_date(T.V_DATE, 'yyyymmdd') THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS EXPIRED_AMOUNT --本月已过期金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EXPIRED_DATE BETWEEN to_date(T.V_DATE, 'yyyymmdd') AND add_months(to_date(T.V_DATE, 'yyyymmdd'), 6) THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE BETWEEN to_date(T.V_DATE, 'yyyymmdd') AND add_months(to_date(T.V_DATE, 'yyyymmdd'), 6) THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS IMMED_EXPIRED_AMOUNT --半年内即将过期金额 FROM DMSALESW.DM_SALE_BUSI_DSCT_SUM_F C LEFT JOIN TMP T ON 1 = 1 WHERE C.CURRENCY_CODE IN ('USD') --改为多值 AND C.BG_CODE IN ('PDCG901159') AND C.M_ID IN ( 'BUSI_DSCT_00001', 'BUSI_DSCT_00002', 'BUSI_DSCT_00003', 'BUSI_DSCT_00004', 'BUSI_DSCT_00005', 'BUSI_DSCT_00006', 'BUSI_DSCT_00007' ) --新增加字段 --AND C.M_CN IN ('#[#P_REPORT_ITEM_NAME#]#') --新增加字段 --新增加字段 GROUP BY C.BG_CODE, C.BG_CN_NAME, C.BG_EN_NAME, C.M_ID --指标ID , C.M_CN --指标中文名称 , C.M_EN --指标英文名称 , C.CURRENCY_CODE --币种 ,CASE WHEN 1 = 0 THEN C.OVERSEA_FLAG ELSE NULL END,CASE WHEN 1 = 0 THEN C.REGION_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.REGION_CN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.REGION_EN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.REPOFFICE_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.REPOFFICE_CN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.REPOFFICE_EN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.OFFICE_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.OFFICE_CN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.OFFICE_EN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_CN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_EN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_EN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_CN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_CN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_EN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_CN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_EN_NAME ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_CODE ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_CN_NAM ELSE NULL END,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_EN_NAM ELSE NULL END,CASE WHEN 1 = 0 THEN C.COUNTRY_CODE ELSE NULL END --新增加入参 ,CASE WHEN 1 = 0 THEN C.COUNTRY_CN_NAME ELSE NULL END --新增加入参 ,CASE WHEN 1 = 0 THEN C.COUNTRY_EN_NAME ELSE NULL END ) T --新增加入参 从SQL中可以看到TMP为标量子查询,并且在子查询T中和物理表C做了笛卡尔积。 下面是该SQL的执行计划: id | operation | A-time | A-rows | E-rows | E-distinct | Peak Memory | E-memory | A-width | E-width | E-costs ----+-------------------------------------------------------------------------+----------------------+---------+---------+------------+----------------+----------+-----------+---------+----------- 1 | -> Row Adapter | 3037.648 | 7 | 245 | | 419KB | | | 1318 | 117210.62 2 | -> Vector Streaming (type: GATHER) | 3037.633 | 7 | 245 | | 777KB | | | 1318 | 117210.62 3 | -> Vector Hash Aggregate | [3031.872, 3032.516] | 7 | 245 | | [4MB, 4MB] | 16MB | [0,870] | 557 | 117128.41 4 | -> Vector Streaming(type: REDISTRIBUTE) | [3031.560, 3032.232] | 112 | 3920 | | [1MB, 1MB] | 2MB | | 557 | 116852.33 5 | -> Vector Hash Aggregate | [2728.059, 2909.255] | 112 | 3920 | | [8MB, 8MB] | 16MB | [833,833] | 557 | 116699.48 6 | -> Vector Nest Loop Left Join (7, 8) | [441.050, 471.725] | 3007901 | 2106919 | | [1MB, 1MB] | 1MB | | 237 | 67316.28 7 | -> CStore Scan on dmsalesw.dm_sale_busi_dsct_sum_f c | [145.354, 158.560] | 3007901 | 2106919 | | [5MB, 5MB] | 1MB | | 205 | 65011.82 8 | -> Vector Materialize | [32.034, 38.902] | 3007901 | 1 | | [288KB, 288KB] | 16MB | [21,21] | 32 | 0.03 9 | -> Vector Subquery Scan on dual | [0.067, 0.093] | 16 | 1 | | [128KB, 128KB] | 1MB | | 32 | 0.02 10 | -> Vector Adapter | [0.005, 0.006] | 16 | 1 | | [40KB, 40KB] | 1MB | | 0 | 0.01 11 | -> Result | [0.001, 0.002] | 16 | 1 | | [8KB, 8KB] | 1MB | | 0 | 0.01 把TMP作为一列放到T中后,性能有明显提升。 EXPLAIN PERFORMANCE SELECT BG_CODE, BG_CN_NAME, BG_EN_NAME, METRIC_CODE --指标ID , METRIC_CN_NAME --指标中文名称 , METRIC_EN_NAME --指标英文名称 , CURRENCY --币种 , OVERSEAS_FLAG, REGION_CODE, REGION_CN_NAME, REGION_EN_NAME, REPOFFICE_CODE, REPOFFICE_CN_NAME, REPOFFICE_EN_NAME, OFFICE_CODE, OFFICE_CN_NAME, OFFICE_EN_NAME, REGION_CUSTCATG_CODE, REGION_CUSTCATG_CN_NAME, REGION_CUSTCATG_EN_NAME, TOP_CUST_CATEGORY_CODE, TOP_CUST_CATEGORY_EN_NAME, TOP_CUST_CATEGORY_CN_NAME, ACCTCUST_HQ_CODE, ACCTCUST_HQ_CN_NAME, ACCTCUST_HQ_EN_NAME, ACCTCUST_BRANCH_CODE, ACCTCUST_BRANCH_CN_NAME, ACCTCUST_BRANCH_EN_NAME, ACCTCUST_SUBSIDIARY_CODE, ACCTCUST_SUBSIDIARY_CN_NAM, ACCTCUST_SUBSIDIARY_EN_NAM, COUNTRY_CODE --新增加入参 , COUNTRY_CN_NAME --新增加入参 , COUNTRY_EN_NAME --新增加入参 , AGREE_AMOUNT --BUSI_DSCT_00001 总优惠 , AGREE_REMAIN_AMOUNT --BUSI_DSCT_00002 即期优惠 , SIGN_AMOUNT --BUSI_DSCT_00003 即期优惠/一次性优惠 , USE_AMOUNT --BUSI_DSCT_00004 即期优惠/单价量折扣 , NOT_USED_VALID_AMOUNT --BUSI_DSCT_00005 延期优惠 , NOT_USED_INVALID_AMOUNT --BUSI_DSCT_00006 voucher , NEW_SIGN_AMOUNT --BUSI_DSCT_00007 其他延期优惠 , NEW_USE_AMOUNT --BUSI_DSCT_00008 本月新使用金额 , EXPIRED_AMOUNT --BUSI_DSCT_00009 本月已过期金额 , IMMED_EXPIRED_AMOUNT --BUSI_DSCT_00010 半年内即将过期金额 FROM ( SELECT case when length('[“202309“]') = 6 then '[“202309“]' || '01' WHEN length('[“202309“]') <> 8 THEN TO_CHAR(CURRENT_DATE, 'YYYYMMDD') END AS V_DATE, C.BG_CODE, C.BG_CN_NAME, C.BG_EN_NAME, C.M_ID AS METRIC_CODE --指标ID , C.M_CN AS METRIC_CN_NAME --指标中文名称 , C.M_EN AS METRIC_EN_NAME --指标英文名称 , C.CURRENCY_CODE AS CURRENCY --币种 ,CASE WHEN 1 = 0 THEN C.OVERSEA_FLAG ELSE NULL END AS OVERSEAS_FLAG,CASE WHEN 1 = 0 THEN C.REGION_CODE ELSE NULL END AS REGION_CODE,CASE WHEN 1 = 0 THEN C.REGION_CN_NAME ELSE NULL END AS REGION_CN_NAME,CASE WHEN 1 = 0 THEN C.REGION_EN_NAME ELSE NULL END AS REGION_EN_NAME,CASE WHEN 1 = 0 THEN C.REPOFFICE_CODE ELSE NULL END AS REPOFFICE_CODE,CASE WHEN 1 = 0 THEN C.REPOFFICE_CN_NAME ELSE NULL END AS REPOFFICE_CN_NAME,CASE WHEN 1 = 0 THEN C.REPOFFICE_EN_NAME ELSE NULL END AS REPOFFICE_EN_NAME,CASE WHEN 1 = 0 THEN C.OFFICE_CODE ELSE NULL END AS OFFICE_CODE,CASE WHEN 1 = 0 THEN C.OFFICE_CN_NAME ELSE NULL END AS OFFICE_CN_NAME,CASE WHEN 1 = 0 THEN C.OFFICE_EN_NAME ELSE NULL END AS OFFICE_EN_NAME,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_CODE ELSE NULL END AS REGION_CUSTCATG_CODE,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_CN_NAME ELSE NULL END AS REGION_CUSTCATG_CN_NAME,CASE WHEN 1 = 0 THEN C.REGION_CUSTCATG_EN_NAME ELSE NULL END AS REGION_CUSTCATG_EN_NAME,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_CODE ELSE NULL END AS TOP_CUST_CATEGORY_CODE,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_EN_NAME ELSE NULL END AS TOP_CUST_CATEGORY_EN_NAME,CASE WHEN 1 = 0 THEN C.TOP_CUST_CATEGORY_CN_NAME ELSE NULL END AS TOP_CUST_CATEGORY_CN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_CODE ELSE NULL END AS ACCTCUST_HQ_CODE,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_CN_NAME ELSE NULL END AS ACCTCUST_HQ_CN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_HQ_EN_NAME ELSE NULL END AS ACCTCUST_HQ_EN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_CODE ELSE NULL END AS ACCTCUST_BRANCH_CODE,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_CN_NAME ELSE NULL END AS ACCTCUST_BRANCH_CN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_BRANCH_EN_NAME ELSE NULL END AS ACCTCUST_BRANCH_EN_NAME,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_CODE ELSE NULL END AS ACCTCUST_SUBSIDIARY_CODE,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_CN_NAM ELSE NULL END AS ACCTCUST_SUBSIDIARY_CN_NAM,CASE WHEN 1 = 0 THEN C.ACCTCUST_SUBSIDIARY_EN_NAM ELSE NULL END AS ACCTCUST_SUBSIDIARY_EN_NAM,CASE WHEN 1 = 0 THEN C.COUNTRY_CODE ELSE NULL END AS COUNTRY_CODE --新增加入参 ,CASE WHEN 1 = 0 THEN C.COUNTRY_CN_NAME ELSE NULL END AS COUNTRY_CN_NAME --新增加入参 ,CASE WHEN 1 = 0 THEN C.COUNTRY_EN_NAME ELSE NULL END AS COUNTRY_EN_NAME --新增加入参 , SUM(C.AGREE_AMOUNT) AS AGREE_AMOUNT --协议金额 , SUM(C.AGREE_REMAIN_AMOUNT) AS AGREE_REMAIN_AMOUNT --协议剩余金额 , SUM(C.SIGN_AMOUNT) AS SIGN_AMOUNT --可用金额 , SUM(C.USE_AMOUNT) AS USE_AMOUNT --已使用金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND NVL( C.EXPIRED_DATE, add_months(C.EFFECTIVE_DATE, C.VALID_MONTH) ) >= to_date(V_DATE, 'yyyymmdd') THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE >= to_date(V_DATE, 'yyyymmdd') THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS NOT_USED_VALID_AMOUNT --未使用金额(有效期外) , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' THEN C.SIGN_AMOUNT ELSE C.EFFECTIVE_TOTAL_AMOUNT END - CASE WHEN C.DSCT_TYPE = 'VOUCHER' THEN C.USE_AMOUNT ELSE C.USED_TOTAL_AMOUNT END - CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND NVL( C.EXPIRED_DATE, add_months(C.EFFECTIVE_DATE, C.VALID_MONTH) ) >= to_date(V_DATE, 'yyyymmdd') THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE >= to_date(V_DATE, 'yyyymmdd') THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS NOT_USED_INVALID_AMOUNT --未使用金额(有效期内) , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EXPIRED_DATE >= to_date(substr(V_DATE, 1, 6), 'yyyymm') and C.EXPIRED_DATE <= LAST_DAY(to_date(V_DATE, 'yyyymmdd')) THEN C.SIGN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_START_DATE >= to_date(substr(V_DATE, 1, 6), 'yyyymm') and C.DSCT_START_DATE <= LAST_DAY(to_date(V_DATE, 'yyyymmdd')) THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS NEW_SIGN_AMOUNT --本月新增可用金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EFFECTIVE_DATE >= to_date(substr(V_DATE, 1, 6), 'yyyymm') and C.EFFECTIVE_DATE <= LAST_DAY(to_date(V_DATE, 'yyyymmdd')) THEN C.USE_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_START_DATE >= to_date(substr(V_DATE, 1, 6), 'yyyymm') and C.DSCT_START_DATE <= LAST_DAY(to_date(V_DATE, 'yyyymmdd')) THEN C.USED_TOTAL_AMOUNT ELSE NULL END ) AS NEW_USE_AMOUNT --本月新使用金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EXPIRED_DATE < to_date(V_DATE, 'yyyymmdd') THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE < to_date(V_DATE, 'yyyymmdd') THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS EXPIRED_AMOUNT --本月已过期金额 , SUM( CASE WHEN C.DSCT_TYPE = 'VOUCHER' AND C.EXPIRED_DATE BETWEEN to_date(V_DATE, 'yyyymmdd') AND add_months(to_date(V_DATE, 'yyyymmdd'), 6) THEN C.AGREE_REMAIN_AMOUNT WHEN C.DSCT_TYPE in ( 'FOC', 'Volume Based List Price Adjustment', 'One-Time Discount' ) AND C.DSCT_END_DATE BETWEEN to_date(V_DATE, 'yyyymmdd') AND add_months(to_date(V_DATE, 'yyyymmdd'), 6) THEN C.EFFECTIVE_TOTAL_AMOUNT ELSE NULL END ) AS IMMED_EXPIRED_AMOUNT --半年内即将过期金额 FROM DMSALESW.DM_SALE_BUSI_DSCT_SUM_F C WHERE C.CURRENCY_CODE IN ('USD') --改为多值 AND C.BG_CODE IN ('PDCG901159') AND C.M_ID IN ( 'BUSI_DSCT_00001', 'BUSI_DSCT_00002', 'BUSI_DSCT_00003', 'BUSI_DSCT_00004', 'BUSI_DSCT_00005', 'BUSI_DSCT_00006', 'BUSI_DSCT_00007' ) --新增加字段 --AND C.M_CN IN ('#[#P_REPORT_ITEM_NAME#]#') --新增加字段 --新增加字段 GROUP BY 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36 ) T --新增加入参 下面是执行计划: id | operation | A-time | A-rows | E-rows | E-distinct | Peak Memory | E-memory | A-width | E-width | E-costs ----+-------------------------------------------------------------------------+----------------------+---------+---------+------------+----------------+----------+-----------+---------+----------- 1 | -> Row Adapter | 1139.637 | 7 | 245 | | 419KB | | | 1318 | 117002.27 2 | -> Vector Streaming (type: GATHER) | 1139.616 | 7 | 245 | | 777KB | | | 1318 | 117002.27 3 | -> Vector Subquery Scan on t | [1129.463, 1130.072] | 7 | 245 | | [504KB, 504KB] | 1MB | | 1318 | 116920.22 4 | -> Vector Hash Aggregate | [1129.459, 1130.067] | 7 | 245 | | [4MB, 4MB] | 16MB | [0,898] | 523 | 116920.07 5 | -> Vector Streaming(type: REDISTRIBUTE) | [1129.142, 1129.918] | 112 | 3920 | | [1MB, 1MB] | 2MB | | 523 | 116643.28 6 | -> Vector Hash Aggregate | [882.194, 987.474] | 112 | 3920 | | [8MB, 8MB] | 16MB | [861,861] | 523 | 116498.95 7 | -> CStore Scan on dmsalesw.dm_sale_busi_dsct_sum_f c | [126.343, 142.697] | 3080954 | 2135243 | | [5MB, 5MB] | 1MB | | 203 | 66116.77 可以看到,不但省去了Nest Loop的耗时,而且后面Aggregate的耗时也减少了不少。整体从3s+优化到1.2s。 点击关注,第一时间了解华为云新鲜技术~

优秀的个人博客,低调大师

数仓性能优化:倾斜优化-表达式计算倾斜的hint优化

本文分享自华为云社区《GaussDB(DWS)性能调优:倾斜优化-表达式计算倾斜的hint优化》,作者: 譡里个檔 。 1.原始SQL SELECT TMP4.TAX_AMT, CATE.L1_PUR_ITEM_CATG_CN_NAME || '-' || CATE.L2_PUR_ITEM_CATG_CN_NAME || '-' || CATE.L3_PUR_ITEM_CATG_CN_NAME AS PRODUCT_CATEGORY, MATE.ITEM_CODE AS PRODUCT_CODE, INVEN.INVENTORY_ORG_NAME, TMP4.INVOICE_WITHHOLDING_TAX_GROUP, TMP4.PAYMENT_WITHHOLDING_TAX_GROUP, TMP4.PO_CHARGE_ACCOUNT_CODE, TMP4.CFS_INVOICE_NUMBER, APR.TAX_INVOICE_DATE FROM DWLTAX.DWL_TAX_TAXDP_ERP_AP_INVOICE_TMP5 TMP4, DWRDIM_DW1.DWR_DIM_PUR_ITEM_CATEGORY_D CATE, DWRDIM_DW1.DWR_DIM_MATERIAL_CODE_D MATE, DWRDIM_DW1.DWR_DIM_INVENTORY_ORG_D INVEN, DWTAXDI.DWI_AP_INVOICE_I AP, DWTAXDI.DWI_AP_INVOICE_REGSTN_I APR WHERE 1 = 1 AND TMP4.ITEM_CATEGORY_KEY = CATE.PUR_ITEM_CATG_KEY(+) AND CATE.DEL_FLAG(+) = 'N' AND TMP4.ITEM_ID = MATE.ITEM_ID(+) AND MATE.DEL_FLAG(+) = 'N' AND TMP4.PO_SHIPMENT_TARGET_INV_ORG_KEY = INVEN.INVENTORY_ORG_KEY(+) AND INVEN.DEL_FLAG(+) = 'N' AND TMP4.AP_INVOICE_ID = AP.AP_INVOICE_ID(+) AND 6600 || AP.ATTRIBUTE1 = TO_CHAR(APR.AP_INVOICE_REGSTN_ID(+)) 执行performance,查询具体执行情况和SQL自诊断信息(详细见附件case-step1-原始执行信息.txt) id | operation | A-time | A-rows | E-rows | E-distinct | Peak Memory | E-memory | A-width | E-width | E-costs ----+------------------------------------------------------------------------------------------------------+------------------------+------------+------------+------------+----------------+----------------+-----------+---------+------------- 1 | -> Row Adapter | 69922.773 | 69237018 | 69237018 | | 87KB | | | 573 | 15160857.61 2 | -> Vector Streaming (type: GATHER) | 65581.989 | 69237018 | 69237018 | | 536KB | | | 573 | 15160857.61 3 | -> Vector Hash Right Join (4, 6) | [61186.201, 73129.055] | 69237018 | 69237018 | | [306MB, 682MB] | 1113MB(9990MB) | | 573 | 15159431.83 4 | -> Vector Streaming(type: BROADCAST ng: LC_DL1->LC_DW1) | [554.217, 21008.078] | 1382000544 | 1381572384 | 282184 | [4MB, 4MB] | 3MB | | 16 | 7056095.88 5 | -> CStore Scan on dwifin.dwi_ap_invoice_regstn s | [5.354, 11.617] | 28791678 | 28782758 | | [1MB, 1MB] | 1MB | | 16 | 28004.18 6 | -> Vector Hash Left Join (7, 19) | [1728.008, 2017.488] | 69237018 | 69237018 | 79721 | [834KB, 834KB] | 16MB | [229,252] | 578 | 1832322.90 7 | -> Vector Hash Left Join (8, 17) | [1428.799, 1925.653] | 69237018 | 69237018 | 179 | [32MB, 32MB] | 28MB(8901MB) | | 576 | 1817105.07 8 | -> Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) | [996.780, 1635.826] | 69237018 | 69237018 | 4167 | [1MB, 1MB] | 2MB | | 570 | 1788113.85 9 | -> Vector Hash Left Join (10, 14) | [1086.903, 1780.641] | 69237018 | 69237018 | | [173MB, 174MB] | 227MB(9067MB) | | 570 | 1304897.12 10 | -> Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) | [153.628, 891.680] | 69237018 | 69237018 | 20271 | [1MB, 1MB] | 2MB | | 567 | 847160.16 11 | -> Vector Hash Left Join (12, 13) | [367.155, 465.821] | 69237018 | 69237018 | | [30MB, 30MB] | 22MB(8896MB) | | 567 | 363943.43 12 | -> CStore Scan on dwltax.dwl_tax_taxdp_erp_ap_invoice_tmp5 tmp4 | [150.676, 178.827] | 69237018 | 69237018 | 526 | [4MB, 4MB] | 1MB | | 553 | 340168.44 13 | -> CStore Scan on dwrdim_dw1.dwr_dim_pur_item_category_d cate | [14.549, 24.399] | 8228448 | 8228448 | 171426 | [2MB, 2MB] | 1MB | [104,104] | 26 | 9056.99 14 | -> Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST ng: LC_DL1->LC_DW1) | [315.926, 339.782] | 117191217 | 117191170 | 2441483 | [1MB, 1MB] | 3MB | [47,47] | 22 | 406136.10 15 | -> Vector Partition Iterator | [118.307, 151.248] | 117191170 | 117191170 | | [41KB, 41KB] | 1MB | | 22 | 300641.93 16 | -> Partitioned CStore Scan on dwifin.dwi_ap_invoice s | [86.557, 111.947] | 117191170 | 117191170 | | [6MB, 6MB] | 1MB | | 22 | 300641.93 17 | -> Vector Streaming(type: PART LOCAL PART BROADCAST) | [60.429, 99.381] | 15442613 | 15442566 | 321720 | [584KB, 584KB] | 2MB | [58,58] | 19 | 49578.19 18 | -> CStore Scan on dwrdim_dw1.dwr_dim_material_code_d mate | [19.779, 33.206] | 15442566 | 15442566 | | [1MB, 2MB] | 1MB | | 19 | 35704.02 19 | -> CStore Scan on dwrdim_dw1.dwr_dim_inventory_org_d inven | [0.383, 0.739] | 135072 | 135072 | 2814 | [1MB, 1MB] | 1MB | [53,53] | 14 | 2823.85 SQL Diagnostic Information -------------------------------------------------------------------------------------------- Execute diagnostic information PlanNode[4] Large Table in Broadcast "Vector Streaming(type: BROADCAST ng: LC_DL1->LC_DW1)" Predicate Information (identified by plan id) ------------------------------------------------------------------------------------------------------------------------------ 3 --Vector Hash Right Join (4, 6) Hash Cond: (((numeric_out(s.ap_invoice_regstn_id))::character varying)::text = ('6600'::text || (s.attribute1)::text)) 6 --Vector Hash Left Join (7, 19) Hash Cond: (tmp4.po_shipment_target_inv_org_key = inven.inventory_org_key) 7 --Vector Hash Left Join (8, 17) Hash Cond: (tmp4.item_id = mate.item_id) Skew Join Optimized by Statistic 8 --Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) Skew Filter(type: ROUNDROBIN): ((tmp4.item_id = (-999999)::numeric) OR (tmp4.item_id IS NULL)) 9 --Vector Hash Left Join (10, 14) Hash Cond: (tmp4.ap_invoice_id = s.ap_invoice_id) Skew Join Optimized by Statistic 10 --Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) Skew Filter(type: ROUNDROBIN): (tmp4.ap_invoice_id = 1001113812002::numeric) 11 --Vector Hash Left Join (12, 13) Hash Cond: (tmp4.item_category_key = cate.pur_item_catg_key) 13 --CStore Scan on dwrdim_dw1.dwr_dim_pur_item_category_d cate Filter: ((cate.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((cate.del_flag)::text = 'N'::text) 14 --Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST ng: LC_DL1->LC_DW1) Skew Filter(type: BROADCAST): (s.ap_invoice_id = 1001113812002::numeric) 15 --Vector Partition Iterator Iterations: 147 16 --Partitioned CStore Scan on dwifin.dwi_ap_invoice s Partitions Selected by Static Prune: 1..147 17 --Vector Streaming(type: PART LOCAL PART BROADCAST) Skew Filter(type: BROADCAST): (mate.item_id = (-999999)::numeric) 18 --CStore Scan on dwrdim_dw1.dwr_dim_material_code_d mate Filter: ((mate.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((mate.del_flag)::text = 'N'::text) 19 --CStore Scan on dwrdim_dw1.dwr_dim_inventory_org_d inven Filter: ((inven.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((inven.del_flag)::text = 'N'::text) 2.禁止大表广播 如上小节显示确实是id=4的这一步是一个大的结果集(2879w条)做了broadcast,并且紧接着的id=5的HashJoin耗时很长。因此通过增加hint方式禁止dwifin.dwi_ap_invoice_regstn走广播。分析发现表dwifin.dwi_ap_invoice_regstn是视图apr展开出现的,因此增加如下hint信息,其中 1. no merge (apr)是防止视图apr中的语句提升,导致的hint信息失效 2. no broadcast(apr)表示禁止apr走broadcast EXPLAIN performance SELECT /*+ no merge (apr) no broadcast(apr) */ TMP4.TAX_AMT, CATE.L1_PUR_ITEM_CATG_CN_NAME || '-' || CATE.L2_PUR_ITEM_CATG_CN_NAME || '-' || CATE.L3_PUR_ITEM_CATG_CN_NAME AS PRODUCT_CATEGORY, MATE.ITEM_CODE AS PRODUCT_CODE, INVEN.INVENTORY_ORG_NAME, TMP4.INVOICE_WITHHOLDING_TAX_GROUP, TMP4.PAYMENT_WITHHOLDING_TAX_GROUP, TMP4.PO_CHARGE_ACCOUNT_CODE, TMP4.CFS_INVOICE_NUMBER, APR.TAX_INVOICE_DATE FROM DWLTAX.DWL_TAX_TAXDP_ERP_AP_INVOICE_TMP5 TMP4, DWRDIM_DW1.DWR_DIM_PUR_ITEM_CATEGORY_D CATE, DWRDIM_DW1.DWR_DIM_MATERIAL_CODE_D MATE, DWRDIM_DW1.DWR_DIM_INVENTORY_ORG_D INVEN, DWTAXDI.DWI_AP_INVOICE_I AP, DWTAXDI.DWI_AP_INVOICE_REGSTN_I APR WHERE 1 = 1 AND TMP4.ITEM_CATEGORY_KEY = CATE.PUR_ITEM_CATG_KEY(+) AND CATE.DEL_FLAG(+) = 'N' AND TMP4.ITEM_ID = MATE.ITEM_ID(+) AND MATE.DEL_FLAG(+) = 'N' AND TMP4.PO_SHIPMENT_TARGET_INV_ORG_KEY = INVEN.INVENTORY_ORG_KEY(+) AND INVEN.DEL_FLAG(+) = 'N' AND TMP4.AP_INVOICE_ID = AP.AP_INVOICE_ID(+) AND 6600 || AP.ATTRIBUTE1 = TO_CHAR(APR.AP_INVOICE_REGSTN_ID(+)) 获取如上语句的performance信息(详细见附件 case-step2-禁止大表广播.txt) id | operation | A-time | A-rows | E-rows | E-distinct | Peak Memory | E-memory | A-width | E-width | E-costs ----+---------------------------------------------------------------------------------------------------------+------------------------+-----------+-----------+------------+----------------+----------------+-----------+---------+------------- 1 | -> Row Adapter | 15685.781 | 69237018 | 69237018 | | 87KB | | | 573 | 33341721.22 2 | -> Vector Streaming (type: GATHER) | 11361.740 | 69237018 | 69237018 | | 536KB | | | 573 | 33341721.22 3 | -> Vector Hash Left Join (4, 19) | [15269.267, 18985.791] | 69237018 | 69237018 | | [74MB, 74MB] | 101MB(9984MB) | | 573 | 33340295.43 4 | -> Vector Streaming(type: REDISTRIBUTE) | [4743.867, 18632.182] | 69237018 | 69237018 | 79721 | [1MB, 2MB] | 2MB | | 578 | 29821930.76 5 | -> Vector Hash Left Join (6, 18) | [1473.990, 15359.055] | 69237018 | 69237018 | | [866KB, 898KB] | 16MB | | 578 | 1832322.90 6 | -> Vector Hash Left Join (7, 16) | [1130.814, 15223.646] | 69237018 | 69237018 | 179 | [32MB, 32MB] | 28MB(9923MB) | | 576 | 1817105.07 7 | -> Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) | [681.709, 14909.424] | 69237018 | 69237018 | 4167 | [1MB, 1MB] | 2MB | | 570 | 1788113.85 8 | -> Vector Hash Left Join (9, 13) | [1049.201, 12602.796] | 69237018 | 69237018 | | [173MB, 174MB] | 227MB(10089MB) | | 570 | 1304897.12 9 | -> Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) | [128.704, 11737.099] | 69237018 | 69237018 | 20271 | [1MB, 1MB] | 2MB | | 567 | 847160.16 10 | -> Vector Hash Left Join (11, 12) | [368.537, 443.623] | 69237018 | 69237018 | | [30MB, 30MB] | 22MB(9918MB) | | 567 | 363943.43 11 | -> CStore Scan on dwltax.dwl_tax_taxdp_erp_ap_invoice_tmp5 tmp4 | [148.366, 175.347] | 69237018 | 69237018 | 526 | [4MB, 4MB] | 1MB | | 553 | 340168.44 12 | -> CStore Scan on dwrdim_dw1.dwr_dim_pur_item_category_d cate | [13.319, 24.442] | 8228448 | 8228448 | 171426 | [2MB, 2MB] | 1MB | [104,104] | 26 | 9056.99 13 | -> Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST ng: LC_DL1->LC_DW1) | [242.053, 294.233] | 117191217 | 117191170 | 2441483 | [1MB, 1MB] | 3MB | [47,47] | 22 | 406136.10 14 | -> Vector Partition Iterator | [118.124, 154.954] | 117191170 | 117191170 | | [41KB, 41KB] | 1MB | | 22 | 300641.93 15 | -> Partitioned CStore Scan on dwifin.dwi_ap_invoice s | [86.942, 105.441] | 117191170 | 117191170 | | [6MB, 6MB] | 1MB | | 22 | 300641.93 16 | -> Vector Streaming(type: PART LOCAL PART BROADCAST) | [83.793, 117.853] | 15442613 | 15442566 | 321720 | [584KB, 584KB] | 2MB | [58,58] | 19 | 49578.19 17 | -> CStore Scan on dwrdim_dw1.dwr_dim_material_code_d mate | [21.898, 35.895] | 15442566 | 15442566 | | [1MB, 2MB] | 1MB | | 19 | 35704.02 18 | -> CStore Scan on dwrdim_dw1.dwr_dim_inventory_org_d inven | [0.389, 0.661] | 135072 | 135072 | 2814 | [1MB, 1MB] | 1MB | [53,53] | 14 | 2823.85 19 | -> Vector Streaming(type: REDISTRIBUTE ng: LC_DL1->LC_DW1) | [30.667, 49.474] | 28791678 | 28782758 | 599641 | [2MB, 2MB] | 3MB | [75,75] | 16 | 56030.49 20 | -> Vector Subquery Scan on apr | [42.087, 61.734] | 28791678 | 28782758 | | [376KB, 376KB] | 1MB | | 16 | 30826.02 21 | -> CStore Scan on dwifin.dwi_ap_invoice_regstn s | [5.177, 8.049] | 28791678 | 28782758 | | [1MB, 1MB] | 1MB | | 16 | 28004.18 SQL Diagnostic Information ---------------------------------------------------------------------------------------------------------- Execute diagnostic information PlanNode[4] DataSkew:"Vector Streaming(type: REDISTRIBUTE)", min_dn_tuples:257082, max_dn_tuples:47206637 Predicate Information (identified by plan id) ---------------------------------------------------------------------------------------------------------------------------------- 3 --Vector Hash Left Join (4, 19) Hash Cond: ((('6600'::text || (s.attribute1)::text)) = ((numeric_out(apr.ap_invoice_regstn_id))::character varying)::text) 5 --Vector Hash Left Join (6, 18) Hash Cond: (tmp4.po_shipment_target_inv_org_key = inven.inventory_org_key) 6 --Vector Hash Left Join (7, 16) Hash Cond: (tmp4.item_id = mate.item_id) Skew Join Optimized by Statistic 7 --Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) Skew Filter(type: ROUNDROBIN): ((tmp4.item_id = (-999999)::numeric) OR (tmp4.item_id IS NULL)) 8 --Vector Hash Left Join (9, 13) Hash Cond: (tmp4.ap_invoice_id = s.ap_invoice_id) Skew Join Optimized by Statistic 9 --Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) Skew Filter(type: ROUNDROBIN): (tmp4.ap_invoice_id = 1001113812002::numeric) 10 --Vector Hash Left Join (11, 12) Hash Cond: (tmp4.item_category_key = cate.pur_item_catg_key) 12 --CStore Scan on dwrdim_dw1.dwr_dim_pur_item_category_d cate Filter: ((cate.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((cate.del_flag)::text = 'N'::text) 13 --Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST ng: LC_DL1->LC_DW1) Skew Filter(type: BROADCAST): (s.ap_invoice_id = 1001113812002::numeric) 14 --Vector Partition Iterator Iterations: 147 15 --Partitioned CStore Scan on dwifin.dwi_ap_invoice s Partitions Selected by Static Prune: 1..147 16 --Vector Streaming(type: PART LOCAL PART BROADCAST) Skew Filter(type: BROADCAST): (mate.item_id = (-999999)::numeric) 17 --CStore Scan on dwrdim_dw1.dwr_dim_material_code_d mate Filter: ((mate.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((mate.del_flag)::text = 'N'::text) 18 --CStore Scan on dwrdim_dw1.dwr_dim_inventory_org_d inven Filter: ((inven.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((inven.del_flag)::text = 'N'::text) 3.表达式倾斜的hint 发现自诊断信息中倾斜告警 而Plan ID为4的算子是 其中是s是视图dwtaxdi.dwi_ap_invoice_i展开后的表dwifin.dwi_ap_invoice,查询此表的列attribute1的统计信息如下,发现在NULL值上存在严重倾斜 因为重分布列是一个表达式6600 || AP.ATTRIBUTE1,当前DWS的倾斜的hint不支持表达式,因为我们做如下变通实现表达式的值倾斜的hint SELECT /*+ no merge (apr) no broadcast(apr) no merge(ap) skew(ap (attr1) ('6600')) */ TMP4.TAX_AMT, CATE.L1_PUR_ITEM_CATG_CN_NAME || '-' || CATE.L2_PUR_ITEM_CATG_CN_NAME || '-' || CATE.L3_PUR_ITEM_CATG_CN_NAME AS PRODUCT_CATEGORY, MATE.ITEM_CODE AS PRODUCT_CODE, INVEN.INVENTORY_ORG_NAME, TMP4.INVOICE_WITHHOLDING_TAX_GROUP, TMP4.PAYMENT_WITHHOLDING_TAX_GROUP, TMP4.PO_CHARGE_ACCOUNT_CODE, TMP4.CFS_INVOICE_NUMBER, APR.TAX_INVOICE_DATE FROM DWLTAX.DWL_TAX_TAXDP_ERP_AP_INVOICE_TMP5 TMP4, DWRDIM_DW1.DWR_DIM_PUR_ITEM_CATEGORY_D CATE, DWRDIM_DW1.DWR_DIM_MATERIAL_CODE_D MATE, DWRDIM_DW1.DWR_DIM_INVENTORY_ORG_D INVEN, (SELECT *, 6600 || AP.ATTRIBUTE1 AS ATTR1 FROM DWTAXDI.DWI_AP_INVOICE_I AP) AP, DWTAXDI.DWI_AP_INVOICE_REGSTN_I APR WHERE 1 = 1 AND TMP4.ITEM_CATEGORY_KEY = CATE.PUR_ITEM_CATG_KEY(+) AND CATE.DEL_FLAG(+) = 'N' AND TMP4.ITEM_ID = MATE.ITEM_ID(+) AND MATE.DEL_FLAG(+) = 'N' AND TMP4.PO_SHIPMENT_TARGET_INV_ORG_KEY = INVEN.INVENTORY_ORG_KEY(+) AND INVEN.DEL_FLAG(+) = 'N' AND TMP4.AP_INVOICE_ID = AP.AP_INVOICE_ID(+) AND ATTR1 = TO_CHAR(APR.AP_INVOICE_REGSTN_ID(+)) 其中构建了子查询 AP SELECT *, 6600 || AP.ATTRIBUTE1 AS ATTR1 FROM DWTAXDI.DWI_AP_INVOICE_I AP 在把原始的关联列表达式放到子查询里面,然后把 6600 || AP.ATTRIBUTE1 命名为attr1。 在父查询中首先禁止AP这个子查询提升。然后在父查询中通过hint 子查询AP这个结果集的列attr1存在倾斜值'6600' 。这个倾斜值是计算出来的(NULL || 6600 = ‘6600’),并且在原始关联计算中关联表达式是如下,即 6600 || AP.ATTRIBUTE1的结果被转换为text类型(字符串类型) 获取新的语句的performance如下(详细见附件 case-step3-倾斜优化.txt) id | operation | A-time | A-rows | E-rows | E-distinct | Peak Memory | E-memory | A-width | E-width | E-costs ----+------------------------------------------------------------------------------------------------------+-----------------------+-----------+-----------+------------+----------------+----------------+-----------+---------+------------ 1 | -> Row Adapter | 9045.793 | 69237018 | 69237018 | | 87KB | | | 573 | 2040755.71 2 | -> Vector Streaming (type: GATHER) | 4842.656 | 69237018 | 69237018 | | 520KB | | | 573 | 2040755.71 3 | -> Vector Hash Left Join (4, 21) | [2673.707, 11389.688] | 69237018 | 69237018 | | [1MB, 1MB] | 16MB | | 573 | 2039329.92 4 | -> Vector Hash Left Join (5, 19) | [1951.482, 10931.220] | 69237018 | 69237018 | 179 | [32MB, 32MB] | 28MB(10018MB) | | 571 | 2009687.71 5 | -> Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) | [1541.777, 10591.702] | 69237018 | 69237018 | 4167 | [1MB, 1MB] | 2MB | | 565 | 1980696.49 6 | -> Vector Hash Left Join (7, 18) | [1703.438, 1980.655] | 69237018 | 69237018 | | [30MB, 30MB] | 22MB(10010MB) | | 565 | 1497479.76 7 | -> Vector Hash Left Join (8, 10) | [1523.277, 1708.622] | 69237018 | 69237018 | 526 | [165MB, 166MB] | 191MB(10151MB) | | 551 | 1473704.77 8 | -> Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) | [94.501, 203.619] | 69237018 | 69237018 | 20271 | [1MB, 1MB] | 2MB | | 553 | 823385.17 9 | -> CStore Scan on dwltax.dwl_tax_taxdp_erp_ap_invoice_tmp5 tmp4 | [142.734, 171.486] | 69237018 | 69237018 | | [4MB, 4MB] | 1MB | | 553 | 340168.44 10 | -> Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST ng: LC_DL1->LC_DW1) | [811.192, 853.583] | 117191217 | 117191170 | 2441483 | [2MB, 2MB] | 3MB | [44,44] | 17 | 598718.74 11 | -> Vector Hash Left Join (12, 15) | [340.998, 790.399] | 117191170 | 117191170 | | [39MB, 39MB] | 27MB(10015MB) | | 17 | 493224.57 12 | -> Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) | [53.170, 79.836] | 117191170 | 117191170 | 79721 | [2MB, 2MB] | 3MB | | 41 | 412662.90 13 | -> Vector Partition Iterator | [145.450, 171.527] | 117191170 | 117191170 | | [41KB, 41KB] | 1MB | | 22 | 303514.27 14 | -> Partitioned CStore Scan on dwifin.dwi_ap_invoice s | [112.099, 134.193] | 117191170 | 117191170 | | [6MB, 6MB] | 1MB | | 22 | 300641.93 15 | -> Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST) | [48.632, 99.230] | 28791678 | 28782758 | 282184 | [2MB, 2MB] | 3MB | [75,75] | 16 | 56928.04 16 | -> Vector Subquery Scan on apr | [41.916, 78.189] | 28791678 | 28782758 | | [376KB, 376KB] | 1MB | | 16 | 30826.02 17 | -> CStore Scan on dwifin.dwi_ap_invoice_regstn s | [5.233, 10.667] | 28791678 | 28782758 | | [1MB, 1MB] | 1MB | | 16 | 28004.18 18 | -> CStore Scan on dwrdim_dw1.dwr_dim_pur_item_category_d cate | [12.065, 20.667] | 8228448 | 8228448 | 171426 | [2MB, 2MB] | 1MB | [104,104] | 26 | 9056.99 19 | -> Vector Streaming(type: PART LOCAL PART BROADCAST) | [67.272, 97.378] | 15442613 | 15442566 | 321720 | [584KB, 584KB] | 2MB | [58,58] | 19 | 49578.19 20 | -> CStore Scan on dwrdim_dw1.dwr_dim_material_code_d mate | [18.605, 31.713] | 15442566 | 15442566 | | [1MB, 2MB] | 1MB | | 19 | 35704.02 21 | -> CStore Scan on dwrdim_dw1.dwr_dim_inventory_org_d inven | [0.378, 0.647] | 135072 | 135072 | 2814 | [1MB, 1MB] | 1MB | [53,53] | 14 | 2823.85 Predicate Information (identified by plan id) ---------------------------------------------------------------------------------------------------------------------------------- 3 --Vector Hash Left Join (4, 21) Hash Cond: (tmp4.po_shipment_target_inv_org_key = inven.inventory_org_key) 4 --Vector Hash Left Join (5, 19) Hash Cond: (tmp4.item_id = mate.item_id) Skew Join Optimized by Statistic 5 --Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) Skew Filter(type: ROUNDROBIN): ((tmp4.item_id = (-999999)::numeric) OR (tmp4.item_id IS NULL)) 6 --Vector Hash Left Join (7, 18) Hash Cond: (tmp4.item_category_key = cate.pur_item_catg_key) 7 --Vector Hash Left Join (8, 10) Hash Cond: (tmp4.ap_invoice_id = s.ap_invoice_id) Skew Join Optimized by Statistic 8 --Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) Skew Filter(type: ROUNDROBIN): (tmp4.ap_invoice_id = 1001113812002::numeric) 10 --Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST ng: LC_DL1->LC_DW1) Skew Filter(type: BROADCAST): (s.ap_invoice_id = 1001113812002::numeric) 11 --Vector Hash Left Join (12, 15) Hash Cond: ((('6600'::text || (s.attribute1)::text)) = ((numeric_out(apr.ap_invoice_regstn_id))::character varying)::text) Skew Join Optimized by Hint 12 --Vector Streaming(type: PART REDISTRIBUTE PART ROUNDROBIN) Skew Filter(type: ROUNDROBIN): ((('6600'::text || (s.attribute1)::text)) = '6600'::text) 13 --Vector Partition Iterator Iterations: 147 14 --Partitioned CStore Scan on dwifin.dwi_ap_invoice s Partitions Selected by Static Prune: 1..147 15 --Vector Streaming(type: PART REDISTRIBUTE PART BROADCAST) Skew Filter(type: BROADCAST): ((((numeric_out(apr.ap_invoice_regstn_id))::character varying)::text) = '6600'::text) 18 --CStore Scan on dwrdim_dw1.dwr_dim_pur_item_category_d cate Filter: ((cate.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((cate.del_flag)::text = 'N'::text) 19 --Vector Streaming(type: PART LOCAL PART BROADCAST) Skew Filter(type: BROADCAST): (mate.item_id = (-999999)::numeric) 20 --CStore Scan on dwrdim_dw1.dwr_dim_material_code_d mate Filter: ((mate.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((mate.del_flag)::text = 'N'::text) 21 --CStore Scan on dwrdim_dw1.dwr_dim_inventory_org_d inven Filter: ((inven.del_flag)::text = 'N'::text) Pushdown Predicate Filter: ((inven.del_flag)::text = 'N'::text) 附件:case-step1-原始执行信息.txt0B 附件:case-step3-倾斜优化.txt862.61KB 附件:case-step2-禁止大表广播.txt0B 点击关注,第一时间了解华为云新鲜技术~

资源下载

更多资源
腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Spring

Spring

Spring框架(Spring Framework)是由Rod Johnson于2002年提出的开源Java企业级应用框架,旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念,提供核心容器、应用上下文、数据访问集成等模块,支持整合Hibernate、Struts等第三方框架,其适用范围不仅限于服务器端开发,绝大多数Java应用均可从中受益。

Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能,例如代码缩略图,Python的插件,代码段等。还可自定义键绑定,菜单和工具栏。Sublime Text 的主要功能包括:拼写检查,书签,完整的 Python API , Goto 功能,即时项目切换,多选择,多窗口等等。Sublime Text 是一个跨平台的编辑器,同时支持Windows、Linux、Mac OS X等操作系统。

WebStorm

WebStorm

WebStorm 是jetbrains公司旗下一款JavaScript 开发工具。目前已经被广大中国JS开发者誉为“Web前端开发神器”、“最强大的HTML5编辑器”、“最智能的JavaScript IDE”等。与IntelliJ IDEA同源,继承了IntelliJ IDEA强大的JS部分的功能。

用户登录
用户注册