搜索[智能出行]结果-低调大师优秀个人博客

智能监控在袋鼠云中的应用

简介传统监控是通过对监控项设置一个固定值（阈值），当监控项指标超过这个阈值时就通知人们关注这个指标项。传统监控一般适用于一定范围波动的业务指标，比如磁盘的使用率，CPU的使用率等，当指标超过一定值时就意味着系统可能出现故障，但是遇到波动范围比较大的场景时，比如某银行的交易09：00~18:00之间交易量大，在其他时间交易量可能为0，工作日交易一般，非工作日交易剧增，又比如某网站的点击量在白天很大，在深夜点击量可能为0，如果使用传统监控对上面的场景进行指标监控，往往不能很好的反映系统和业务的状态，产生很多误报的情况，增加人工成本，而且甚至会让人们对告警产生麻木，不信任感。所以我们加入了机器学习算法，对过去的监控指标进行训练，对当前值的异常判断不再仅仅取决于一个固定的阈值，而是同期数据，历史周期性数据进行了参考，通过这是动态阈值的方法对异常数据进行检测。技术架构模型训练器：云日志以固定频率采集的业务指标形成时间序列，输送到模型训练器中，模型训练器有一系列的数学模型组成（可动态添加），每个模型都得到预测值，观察值与预测值之前存在的误差，对比误差我们将得到一个与业务最匹配的数学模型。利用这个训练出的最佳模型，输入未来时间点，得到预测值，绘制未来业务图。异常检测器：训练的数学模型预测的值与实际的观察值存在一定的误差，这个残差系列输送到异常检测器中，异常检测器也是由一系列的数学模型组成（可动态添加），模型检查的误差点与业务的异常点最匹配的模型将作为异常检测模型，将后续检测出的异常点发送给预警系统。时间序列建模采集的时间序列数据并非是散乱，毫无规律的一组数据，它往往伴随业务的变化而变化，有的具有很强的周期性规则，有的具有相对平滑的趋势，我们需要利用对应的数学模型来拟合，一下是我们常用的几种数学模型。模型描述 OlympicModel 季节模型，其中下一个点是先前n个时期的平滑平均值 MovingAverageModel 移动平均模型，下一个点在给定时间段的平均值 MultipleLinearRegressionModel 多元线性回归模型，使用一个或者多个变量对x和y的关系进行建模 PolynomialRegressionModel 多项式回归模型，高纬度数据建模 exponentialSmoothingModel 指数平滑模型，分一次，二次，三次指数平滑，对周期，趋势，季节性特征数据 weightedMovingAverageModel 权重移动平均模型，下一个点是给定时间段的加权平均值对不同特征的时间序列，不同的数学模型所计算出的误差也截然不同，我们从以下列表的指标来衡量这些数学模型的匹配度。指标描述 Bias 误差的算术平均值 MAD 平均绝对偏差，也称为MAE MAPE 平均绝对百分比误差 MSE 误差的均方 SAE 绝对错误的总和 ME 平均误差 MASE 平均绝对比例误差 MPE 平均百分比误差在经过以上指标衡量预测模型的优劣后，我得到最契合业务的拟合曲线，得到最佳的训练模型。然后输入未来时间点得到那个时间点的预测值，然后绘制出预测曲线异常检测在预测出未来时间点的数据后，如何检测这个业务数据是否异常，我们也有对应的异常检测模型，如下表所示模型描述 cpModel 基于内核的波动点检测 DBScanModel 基于密度的聚类算法检测 kSigmaModel 经典的k-sigma模型，概率检测 extremeLowDensityModel 基于密度的异常检测，范围检测将残差指标用以上模型计算之后，与过去的业务异常点进行对比，选择最接近的异常检测模型，作为后续的异常检测，当模型检测数数据异常时，即时发送预警给巡检员，防患于未来。

2018-11-13

智能媒体管理(IMM) Python SDK DEMO

SDK For Python 本文以Python为示例安装 Python 环境 IMM Service 的Python SDK目前只支持 Python 2.6.x, 2.7.x。请按以下步骤安装Python：下载并安装最新的 Python 2 安装包。完成Python安装后，运行python -V确认Python版本。下载并安装 Python 的包管理工具 pip。完成 pip 安装后，你可以运行pip -V确认 pip 是否安装成功和查看当前 pip 版本。安装 Imm Service Python SDK 使用 pip 安装(推荐) pip install aliyun-python-sdk-core # 安装阿里云 SDK 核心库 pip install aliyun-python-sdk-imm # 安装管理 IMM 的库下载 GithHub 源码 git clone https://github.com/aliyun/aliyun-openapi-python-sdk.git # 安装阿里云 SDK 核心库 cd aliyun-python-sdk-core python setup.py install # 安装阿里云 IMM SDK cd aliyun-python-sdk-imm python setup.py install 使用Python SDK 初始化客户端使用之前需要，您需要先获取 Region ID、AccessKey ID 和 AccessKey Secret。 # -*- coding: utf8 -*- import json import time from aliyunsdkcore.client import AcsClient client = AcsClient( "<your-access-key-id>", "<your-access-key-secret>", "<your-region-id>" ) 创建Project 根据您的需求，选择Project类型。本示例目的是展示关键参数，详情参考Project文档。注意事项：测试时Project的BillingType请设置为ByUsage方式,本文所有示例代码BillingType均使用ByUsage。不能创建名称相同的Project。代码示例 from aliyunsdkimm.request.v20170906 import PutProjectRequest project = "python-sdk-demo-doc" #设置Project的名字 project_type = "DocStarter" #图片标准型 project_cu = 1 project_billing_type = 'ByUsage' createReq = PutProjectRequest.PutProjectRequest() createReq.set_Project(project) createReq.set_Type(project_type) createReq.set_Type(project_cu) createReq.set_BillingType(project_billing_type) response = client.do_action_with_exception(createReq) print response 输出结果 { "CU": 1 , "Type": "DocStarter", "CreateTime": "2018-10-30T03:11:23Z", "RequestId": "4F048F9F-D622-4B96-9286-AEA234BF1480", "ModifyTime": "2018-10-30T03:11:23Z", "Project": "python-sdk-demo-doc", "BillingType": "ByUsage", "Endpoint": "imm.cn-beijing.aliyuncs.com", "ServiceRole": "" } 创建具体的服务文档管理操作文档同步转换本示例目的是展示关键参数，其余参数具体请参考ConvertOfficeFormat。代码示例 from aliyunsdkimm.request.v20170906 import ConvertOfficeFormatRequest project = "python-sdk-demo-doc" #演示用project已创建，您需要自己创建 sync_srcUri = "oss://co-user-cn-beijing/zqh/input/WORD/5页Word.docx" #您文档资源的位置 sync_tgtUri = "oss://co-user-cn-beijing/zqh/output/PDF/" #您要输出的位置 tgt_type = "png" #转化类型 createReq = ConvertOfficeFormatRequest.ConvertOfficeFormatRequest() createReq.set_Project(project) createReq.set_SrcUri(sync_srcUri) createReq.set_TgtUri(sync_tgtUri ) createReq.set_TgtType(tgt_type) response = client.do_action_with_exception(createReq) print response 输出结果 { "PageCount": 5, "RequestId": "84302ED0-4823-4623-8AED-6E4129DA9733" } 文档异步转换创建转换任务本示例目的是展示关键参数，其余参数具体请参考CreateOfficeConversionTask。代码示例 from aliyunsdkimm.request.v20170906 import CreateOfficeConversionTaskRequest project = "python-sdk-demo-doc" #演示用project已创建，您需要自己创建 async_srcUri = "oss://co-user-cn-beijing/zqh/input/WORD/5页Word.docx" #您文档资源的位置 async_tgtUri = "oss://co-user-cn-beijing/zqh/output/PDF/" #您要输出的位置 tgt_type = "png" #转化类型 createReq = CreateOfficeConversionTaskRequest.CreateOfficeConversionTaskRequest() createReq.set_Project(project) createReq.set_SrcUri(async_srcUri) createReq.set_TgtUri(async_tgtUri ) createReq.set_TgtType(tgt_type) response = client.do_action_with_exception(createReq) print response 输出结果 { "Status": "Running", "CreateTime": "2018-10-30T03:49:06.345Z", "RequestId": "D1BA4308-CA2F-4F93-B3C6-B30188855138", "Percent": 0, "TaskId": "2ef08a22-6cff-4204-a1f5-38473d66596b", "TgtLoc": "oss://co-user-cn-beijing/zqh/input/WORD/5页Word.docx" } 获取转换状态本示例目的是展示关键参数，其余参数具体请参考GetOfficeConversionTask。代码示例 from aliyunsdkimm.request.v20170906 import GetOfficeConversionTaskRequest project = "python-sdk-demo-doc" #演示用project已创建，您需要自己创建 task_id = "2ef08a22-6cff-4204-a1f5-38473d66596b" createReq = GetOfficeConversionTaskRequest.GetOfficeConversionTaskRequest() createReq.set_Project(project) createReq.set_TaskId(task_id ) 输出结果 { "FailDetail": { "Code": "NoError" }, "NotifyTopicName": "", "TaskId": "2ef08a22-6cff-4204-a1f5-38473d66596b", "NotifyEndpoint": "", "PageCount": 5, "Status": "Finished", "TgtType": "png", "FinishTime": "2018-10-30T03:49:07.651Z", "RequestId": "D200F0EF-F3E7-4A18-B2A7-E7BBBA8E10EC", "CreateTime": "2018-10-30T03:49:06.345Z", "SrcUri": "oss://co-user-cn-beijing/zqh/input/WORD/5页Word.docx", "Percent": 100, "TgtUri": "oss://co-user-cn-beijing/zqh/input/WORD/5页Word.docx" } 图片管理操作 Face检测本示例目的是展示关键参数，其余参数具体请参考DetectFace。代码示例 from aliyunsdkimm.request.v20170906 import DetectFaceRequest project = "python-sdk-demo-photo" #演示用project已创建，您需要自己创建 srcUri = '["oss://co-user-cn-beijing/zqh/input/FaceGroup/500/0.bmp"]' #您图片资源的位置 createReq = DetectFaceRequest.DetectFaceRequest() createReq.set_Project(project) createReq.set_SrcUris(srcUri) response = client.do_action_with_exception(createReq) print response 输出结果 { "SrcUris": [ "oss://co-user-cn-beijing/zqh/input/FaceGroup/500/0.bmp" ], "RequestId": "21404A97-8625-44DB-8C7A-E7F33A9EDEF4", "SuccessDetails": [ { "SrcUri": "oss://co-user-cn-beijing/zqh/input/FaceGroup/500/0.bmp", "Faces": [ { "FaceAttribute": { "Blur": { "Blurness": { "Value": 12.217, "Threshold": 50 } }, "FaceQuality": { "Value": 100, "Threshold": 70.1 }, "HeadPose": { "RollAngle": 2.934, "PitchAngle": 10.529, "YawAngle": -3.145 }, "Age": { "Value": 21 }, "EyeStatus": { "RightEyeStatus": { "DarkGlasses": 0, "NoGlassEyeClose": 0.001, "Occlusion": 0.049, "NormalGlassEyeOpen": 0.106, "NormalGlassEyeClose": 0.001, "NoGlassEyeOpen": 99.842 }, "LeftEyeStatus": { "DarkGlasses": 0.014, "NoGlassEyeClose": 0.006, "Occlusion": 0.504, "NormalGlassEyeOpen": 4.954, "NormalGlassEyeClose": 0.004, "NoGlassEyeOpen": 94.517 } }, "Gender": { "Value": "Male" } }, "FaceRectangle": { "Top": 17, "Height": 89, "Width": 89, "Left": 4 }, "FaceId": "66690675bd0bc2599170fc63e7f8dbd8" } ], "PhotoId": "77f6e9ce630b83956192e8bd4af3cf39" } ], "FailDetails": [] } Tag检测本示例目的是展示关键参数，其余参数具体请参考DetectTag。代码示例 from aliyunsdkimm.request.v20170906 import DetectTagRequest project = "python-sdk-demo-photo" #演示用project已创建，您需要自己创建 srcUri = '["oss://co-user-cn-beijing/zqh/input/TagSet/baseball.jpg"]' #您图片资源的位置 createReq = DetectTagRequest.DetectTagRequest() createReq.set_Project(project) createReq.set_SrcUris(srcUri) response = client.do_action_with_exception(createReq) print response 输出结果 { "SuccessNum": "1", "RequestId": "6A3188E0-BC89-4AD5-9348-851F37F5AF95", "SuccessDetails": [ { "Tags": [ { "ParentTagId": "11", "TagScore": "0.8965469", "TagLevel": "2", "TagId": "0", "ParentTagName": "运动", "TagName": "其他" }, { "ParentTagId": "10", "TagScore": "0.7168899", "TagLevel": "2", "TagId": "655", "ParentTagName": "其他", "TagName": "棒球守备位置" }, { "ParentTagId": "11", "TagScore": "0.55166656", "TagLevel": "2", "TagId": "50", "ParentTagName": "运动", "TagName": "球赛" }, { "ParentTagId": "10", "TagScore": "0.53409994", "TagLevel": "2", "TagId": "940", "ParentTagName": "其他", "TagName": "场地" }, { "ParentTagId": "0", "TagScore": "0.5216745", "TagLevel": "2", "TagId": "159", "ParentTagName": "人物", "TagName": "比赛者" }, { "ParentTagId": "0", "TagScore": "0.5185509", "TagLevel": "2", "TagId": "65", "ParentTagName": "人物", "TagName": "足球运动员" }, { "ParentTagId": "0", "TagScore": "0.5162505", "TagLevel": "2", "TagId": "54", "ParentTagName": "人物", "TagName": "运动员" }, { "ParentTagId": "0", "TagScore": "0.5147973", "TagLevel": "2", "TagId": "133", "ParentTagName": "人物", "TagName": "棒球运动员" }, { "ParentTagId": "10", "TagScore": "0.5040618", "TagLevel": "2", "TagId": "453", "ParentTagName": "其他", "TagName": "球游戏" } ], "SrcUri": "oss://co-user-cn-beijing/zqh/input/TagSet/baseball.jpg" } ], "FailDetails": [] } QRCodes检测本示例目的是展示关键参数，其余参数具体请参考DetectQRCodes。代码示例 from aliyunsdkimm.request.v20170906 import DetectQRCodesRequest project = "python-sdk-demo-photo" #演示用project已创建，您需要自己创建 srcUri = '["oss://co-user-cn-beijing/zqh/input/TagSet/qrcode.png"]' #您图片资源的位置 createReq = DetectQRCodesRequest.DetectQRCodesRequest() createReq.set_Project(project) createReq.set_SrcUris(srcUri) response = client.do_action_with_exception(createReq) print response 运行结果 { "RequestId": "21B97DEA-A3A9-4473-90E5-D527CC0F7287", "SuccessDetails": [ { "SrcUri": "oss://co-user-cn-beijing/zqh/input/qrcode.png", "QRCodes": [ { "QRCodesRectangle": { "Top": 11, "Height": 280, "Width": 280, "Left": 11 }, "Content": "二维码又称二维条码，常见的二维码为QR Code，QR全称Quick Response，是一个近几年来移动设备上超流行的一种编码方式，它比传统的Bar Code条形码能存更多的信息，也能表示更多的数据类型。" } ] } ], "FailDetails": [] } Face分组分组需要Project为PhotoProfessional类型。创建FaceSet 代码示例 from aliyunsdkimm.request.v20170906 import CreateFaceSetRequest project = "python-sdk-demo-professional" # 演示用project已创建，您需要自己创建 createReq = CreateFaceSetRequest.CreateFaceSetRequest() createReq.set_Project(project) response = client.do_action_with_exception(createReq) print response 运行结果 { "Status": "Running", "Photos": 0, "RequestId": "006AF2E5-B1E9-4A8A-89F1-E4FE229C57AE", "CreateTime": "2018-10-30T06:58:35.556Z", "Faces": 0, "ModifyTime": "2018-10-30T06:58:35.556Z", "SetId": "FACE-99a292da-32e3-49cb-aa35-e6a01604a744" } 进行IndexFace人脸检测本示例目的是展示关键参数，其余参数具体请参考IndexFace。代码示例 from aliyunsdkimm.request.v20170906 import IndexFaceRequest project = "python-sdk-demo-professional" #演示用project已创建，您需要自己创建 srcUri = '["oss://co-user-cn-beijing/zqh/input/FaceGroup/500/0.bmp"]' #您图片资源的位置 createReq = IndexFaceRequest.IndexFaceRequest() createReq.set_Project(project) createReq.set_SrcUris(srcUri) response = client.do_action_with_exception(createReq) print response 运行结果 { "SrcUris": [ "oss://co-user-cn-beijing/zqh/input/FaceGroup/500/0.bmp" ], "RequestId": "14888AFD-E6D5-418D-B629-4E23A89C7022", "SuccessDetails": [ { "SrcUri": "oss://co-user-cn-beijing/zqh/input/FaceGroup/500/0.bmp", "Faces": [ { "FaceAttribute": { "Blur": { "Blurness": { "Value": 12.217, "Threshold": 50 } }, "FaceQuality": { "Value": 100, "Threshold": 70.1 }, "HeadPose": { "RollAngle": 2.934, "PitchAngle": 10.529, "YawAngle": -3.145 }, "Age": { "Value": 21 }, "EyeStatus": { "RightEyeStatus": { "DarkGlasses": 0, "NoGlassEyeClose": 0.001, "Occlusion": 0.049, "NormalGlassEyeOpen": 0.106, "NormalGlassEyeClose": 0.001, "NoGlassEyeOpen": 99.842 }, "LeftEyeStatus": { "DarkGlasses": 0.014, "NoGlassEyeClose": 0.006, "Occlusion": 0.504, "NormalGlassEyeOpen": 4.954, "NormalGlassEyeClose": 0.004, "NoGlassEyeOpen": 94.517 } }, "Gender": { "Value": "Male" } }, "FaceRectangle": { "Top": 17, "Height": 89, "Width": 89, "Left": 4 }, "FaceId": "66690675bd0bc2599170fc63e7f8dbd8" } ], "PhotoId": "77f6e9ce630b83956192e8bd4af3cf39" } ], "FailDetails": [], "SetId": "FACE-99a292da-32e3-49cb-aa35-e6a01604a744" } 进行GroupFace人脸分组分组需要多张图片,请执行两次或以上IndexFace操作。本示例目的是展示关键参数，其余参数具体请参考GroupFace。代码示例 from aliyunsdkimm.request.v20170906 import GroupFacesRequest project = "python-sdk-demo-professional" set_id = "FACE-99a292da-32e3-49cb-aa35-e6a01604a744" createReq = GroupFacesRequest.GroupFacesRequest() createReq.set_Project(project) createReq.set_SetId(set_id) response = client.do_action_with_exception(createReq) print response 运行结果 { "Groups": [ { "UnGroupReason": "FaceNoSimilar", "GroupId": "0", "FaceId": "fffbf28d706c2230ba29a1b2ee271744" }, { "UnGroupReason": "FaceNoSimilar", "GroupId": "0", "FaceId": "5c1212c756047570ffd0839386b5c2bd" }, { "UnGroupReason": "FaceNoSimilar", "GroupId": "0", "FaceId": "66690675bd0bc2599170fc63e7f8dbd8" }, { "UnGroupReason": "FaceNoSimilar", "GroupId": "0", "FaceId": "dadddf6bdd41d02aadc9fc4f2b54ca2b" } ], "RequestId": "AC452D2A-F2B2-48AD-83B4-A6E03D38066F", "HasMore": 0, "SetId": "FACE-99a292da-32e3-49cb-aa35-e6a01604a744" } Tag分组分组需要Project为PhotoProfessional类型。创建TagSet 代码示例 from aliyunsdkimm.request.v20170906 import CreateTagSetRequest project = "python-sdk-demo-professional" # 演示用project已创建，您需要自己创建 createReq = CreateTagSetRequest.CreateTagSetRequest() createReq.set_Project(project) response = client.do_action_with_exception(createReq) print response 运行结果 { "Status": "Running", "Photos": 0, "RequestId": "FD93E872-1437-4A78-8C15-CBD83B9CE152", "CreateTime": "2018-10-30T07:33:15.365Z", "ModifyTime": "2018-10-30T07:33:15.365Z", "SetId": "TAG-9d79cc61-0f2b-4d4d-931a-377bcc641878" } 进行IndexTag图片检测本示例目的是展示关键参数，其余参数具体请参考IndexTag。代码示例 from aliyunsdkimm.request.v20170906 import IndexTagRequest project = "python-sdk-demo-professional" #演示用project已创建，您需要自己创建 srcUri = '["oss://co-user-cn-beijing/zqh/input/FaceGroup/500/0.bmp"]' #您图片资源的位置 set_id = 'TAG-9d79cc61-0f2b-4d4d-931a-377bcc641878' createReq = IndexTagRequest.IndexTagRequest() createReq.set_Project(project) createReq.set_SrcUris(srcUri) createReq.set_SetId(set_id ) response = client.do_action_with_exception(createReq) print response 运行结果 { "RequestId": "F296C6E7-2213-4934-910B-0436B9F1C71F", "SuccessDetails": [ { "Tags": [ { "ParentTagId": "11", "TagScore": "0.8965469", "TagLevel": "2", "TagId": "0", "ParentTagName": "运动", "TagName": "其他" }, { "ParentTagId": "10", "TagScore": "0.7168899", "TagLevel": "2", "TagId": "655", "ParentTagName": "其他", "TagName": "棒球守备位置" }, { "ParentTagId": "11", "TagScore": "0.55166656", "TagLevel": "2", "TagId": "50", "ParentTagName": "运动", "TagName": "球赛" }, { "ParentTagId": "10", "TagScore": "0.53409994", "TagLevel": "2", "TagId": "940", "ParentTagName": "其他", "TagName": "场地" }, { "ParentTagId": "0", "TagScore": "0.5216745", "TagLevel": "2", "TagId": "159", "ParentTagName": "人物", "TagName": "比赛者" }, { "ParentTagId": "0", "TagScore": "0.5185509", "TagLevel": "2", "TagId": "65", "ParentTagName": "人物", "TagName": "足球运动员" }, { "ParentTagId": "0", "TagScore": "0.5162505", "TagLevel": "2", "TagId": "54", "ParentTagName": "人物", "TagName": "运动员" }, { "ParentTagId": "0", "TagScore": "0.5147973", "TagLevel": "2", "TagId": "133", "ParentTagName": "人物", "TagName": "棒球运动员" }, { "ParentTagId": "10", "TagScore": "0.5040618", "TagLevel": "2", "TagId": "453", "ParentTagName": "其他", "TagName": "球游戏" } ], "SrcUri": "oss://co-user-cn-beijing/zqh/input/TagSet/baseball.jpg" } ], "FailDetails": [], "SetId": "TAG-9d79cc61-0f2b-4d4d-931a-377bcc641878", "SuccessIndexNum": "1" } 列出数据集检测出的标签。本示例目的是展示关键参数，其余参数具体请参考ListTagNames。代码示例 from aliyunsdkimm.request.v20170906 import ListTagNamesRequest project = "python-sdk-demo-professional" #演示用project已创建，您需要自己创建 set_id = 'TAG-9d79cc61-0f2b-4d4d-931a-377bcc641878' createReq = ListTagNamesRequest.ListTagNamesRequest() createReq.set_Project(project) createReq.set_SetId(set_id ) response = client.do_action_with_exception(createReq) print response 运行结果 { "Tags": [ { "Num": 1, "TagName": "场地" }, { "Num": 1, "TagName": "女人" }, { "Num": 1, "TagName": "晚餐" }, { "Num": 1, "TagName": "棒球守备位置" }, { "Num": 1, "TagName": "棒球运动员" }, { "Num": 2, "TagName": "其他" }, { "Num": 1, "TagName": "八宝饭" }, { "Num": 1, "TagName": "午餐" }, { "Num": 1, "TagName": "比赛者" }, { "Num": 1, "TagName": "人" } ], "RequestId": "D59C1305-F01A-46DF-8E66-22A334F43F35" }

2018-10-29