安装
Python 3.8+ including pip3, git 1.9+
pip3 install esrally
高级配置
esrally configure --advanced-config （可多次执行）
Enter the benchmark root directory (default: /home/apps/.rally/benchmarks):
Using default value '/home/apps/.rally/benchmarks'
Enter your Elasticsearch project directory: (default: /home/apps/.rally/benchmarks/src/elasticsearch):
Using default value '/home/apps/.rally/benchmarks/src/elasticsearch'
Where should metrics be kept?
(1) In memory (simpler but less options for analysis)
(2) Elasticsearch (requires a separate ES instance, keeps all raw samples for analysis)
(default: 1): 2
Enter the host name of the ES metrics store (default: localhost):
Using default value 'localhost'
Enter the port of the ES metrics store: 9200
Use secure connection (True, False) (default: False):
Username for basic authentication (empty if not needed) (default: ):
Password for basic authentication (empty if not needed) (default: ):
Enter a descriptive name for this benchmark environment (ASCII, no spaces) (default: local): rally_demo
Do you want Rally to keep the Elasticsearch benchmark candidate installation including the index (will use several GB per trial run)? (default: False):
Configuration successfully written to /home/apps/.rally/rally.ini. Happy benchmarking!
- rally-metrics-*：记录race中所有的采样数据（性能报告数据都是基于采样数据汇总得出的结果），可以通过采样数据看到具体抖动情况进行具体分析
- rally-results-*：基于上面的采样数据进行详细的数据汇总，针对每个指标进行汇总，可以针对每个指标进行最终的结果展示
- rally-races-：把最终性能压测结果（即我们看到的输出数据）展示出来，一个文档就是一次race压测
将测试结果放入开启x_pack集群中需要配置
vim .rally/rally.ini
[reporting]
#datastore.ssl.verification_mode = none
datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer
使用
配置代理下载track
export http_proxy=http://proxy.acme.org:8888/
/root/.rally/logs/rally.log将记录
Rally connects via proxy URL [http://proxy.acme.org:3128/] to the Internet (picked up from the environment variable [http_proxy]).
esrally list tracks
类似于赛道的意思
nested 嵌套文档
percolator 过滤查询
eventdata
geoshape
geopoint 地理查询
so
geopointshape
noaa
metricbeat
nyc_taxis 高度数据化结构
http_logs
pmc 全文搜索
geonames 结构化数据
对现有集群进行基准测试
esrally --track=pmc --target-hosts=10.5.5.10:9243,10.5.5.11:9243,10.5.5.12:9243 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'changeme'" --user-tag="intention:baseline_github_1234" --report-file=result.csv --report-format=csv
rally可以组成分布式集群，并可以根据配置文件进行测试，因为没有试过，这里不再赘述
esrally list races 对比两场比赛
esrally compare --baseline=0bfd4542-3821-4c79-81a2-0858636068ce --contender=beb154e4-0a05-4f45-ad9f-e34f9a9e51f7
创建track
mkdir .rally/benchmarks/data/XXX
- 获取数据 allCountries.txt
- 转换为json格式
vim toJSON.py
import json
cols = (("geonameid", "int", True),
("name", "string", True),
("asciiname", "string", False),
("alternatenames", "string", False),
("latitude", "double", True),
("longitude", "double", True),
("feature_class", "string", False),
("feature_code", "string", False),
("country_code", "string", True),
("cc2", "string", False),
("admin1_code", "string", False),
("admin2_code", "string", False),
("admin3_code", "string", False),
("admin4_code", "string", False),
("population", "long", True),
("elevation", "int", False),
("dem", "string", False),
("timezone", "string", False))
def main():
with open("allCountries.txt", "rt", encoding="UTF-8") as f:
for line in f:
tup = line.strip().split("\t")
record = {}
for i in range(len(cols)):
name, type, include = cols[i]
if tup[i] != "" and include:
if type in ("int", "long"):
record[name] = int(tup[i])
elif type == "double":
record[name] = float(tup[i])
elif type == "string":
record[name] = tup[i]
print(json.dumps(record, ensure_ascii=False))
if __name__ == "__main__":
main()
python3 toJSON.py > documents.json
-
编写track.json
-
检查是否可用esrally info --track-path=~/rally-tracks/tutorial
track解剖
{% set index_count = 2 %} #设置循环次数
{
"version": 2,
"description": "Tutorial benchmark for Rally",
"indices": [
{
"name": "geonames", #要操作索引的名称
"body": "index.json"
}
],
"corpora": [ #此轨道使用的所有文档语料库
{
"name": "rally-tutorial",
"documents": [
{
"source-file": "documents.json",
"document-count": 11658903,
"uncompressed-bytes": 1544799789
}
]
}
],
"schedule": [ #具体执行的操作
{
"operation": {
"operation-type": "delete-index" #操作类型
}
},
{
"operation": {
"operation-type": "create-index"
}
},
{
"operation": {
"operation-type": "cluster-health",
"request-params": {
"wait_for_status": "green"
}
}
},
{
"operation": {
"operation-type": "bulk",
"bulk-size": 5000
},
"warmup-time-period": 120,
"clients": 8
},
{
"operation": {
"operation-type": "force-merge"
}
},
{
"operation": {
"name": "query-match-all",
"operation-type": "search",
"body": {
"query": {
"match_all": {}
}
}
},
"clients": 8, #发起请求客户端数量
"warmup-iterations": 1000, #预热，不显示在测量结果中
"iterations": 1000, #执行次数
"target-throughput": 100 #尽可能慢地运行任务，target-throughput/clients/s qps
}
]
}
配置parallel实现并行执行语句
https://esrally.readthedocs.io/en/latest/track.html
https://elasticsearch-benchmarks.elastic.co/#
https://learnku.com/articles/41047
