一.Apache Druid简述

Apache Druid是MetaMarket公司研发，专门为做海量数据集上的高性能OLAP（OnLine Analysis Processing）而设计的数据存储和分析系统，目前在Apache基金会下孵化。

Apache Druid采用Lambda架构，分为实时层（Overlad、MiddleManager）和批处理层（Coordinator、Historical），通过Broker节点客户端提供查询服务，Router节点Overlad、Coordinator和Broker提供统一的API网关服务，系统架构如下：

二.主要角色简述

Ⅰ).Overload

Overload进程负责监控MiddleManager进程，它负责将摄取任务分配给MiddleManager并协调segment的发布；它就是数据摄入到Dirid的控制器

Ⅱ).Coordinator

Coordinator进程负责监控Historical进程，它负责将segment分配到指定的Historical服务上，确保所有Historical节点的数据均衡

Ⅲ).MiddleManager

MiddleManager进程负责将新的数据摄入到集群中，将外部数据源数据转换为Druid所识别的segment

Ⅳ).Broker

Broker进程负责接受Client的查询请求，并将查询转发到Historical和MiddleManager中；Broker会接受所有的子查询的结果，并将数据进行合并然后返回给Client

Ⅴ).Historical

Historical是用于处理存储和查询历史数据的进程，它会从Deep Storage中下载查询区间数据，然后响应该段数据的查询

Ⅵ).Router

Router进程是一个可选的进程，它为Broker、Overload和Coordinator提供统一的API网关服务。如果不启动该进程，也可以直接连接Broker、Overload和Coordinator服务

三.安装部署

Ⅰ).角色分布

Ⅱ).下载

下载地址：https://druid.apache.org/downloads.html
apache-druid-0.15.0

Ⅲ).解压

tar -zxvf apache-druid-0.15.0-incubating-bin.tar.gz

Ⅳ).目录

Ⅴ).配置

a)../_common/common.runtime.properties

#
# Extensions
#

# This is not the full list of Druid extensions, but common ones that people often use. You may need to change this list
# based on your particular setup.
druid.extensions.loadList=[ "druid-datasketches", "druid-hdfs-storage","druid-kafka-eight", "mysql-metadata-storage","druid-kafka-indexing-service"]

# If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
# and uncomment the line below to point to your directory.
druid.extensions.hadoopDependenciesDir=/druid/druid/hadoop-dependencies/

#
# Logging
#

# Log all runtime properties on startup. Disable to avoid logging properties on startup:
druid.startup.logging.logProperties=true

#
# Zookeeper
#

druid.zk.service.host=hostname1:2181,hostname2:2181,hostname3:2181
druid.zk.paths.base=/druid

#
# Metadata storage
#

# For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
#druid.metadata.storage.type=derby
#druid.metadata.storage.connector.connectURI=jdbc:derby://metadata.store.ip:1527/var/druid/metadata.db;create=true
#druid.metadata.storage.connector.host=metadata.store.ip
#druid.metadata.storage.connector.port=1527

# For MySQL:
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://hostname1:3306/druid
druid.metadata.storage.connector.user=username
druid.metadata.storage.connector.password=password

# For PostgreSQL (make sure to additionally include the Postgres extension):
#druid.metadata.storage.type=postgresql
#druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid
#druid.metadata.storage.connector.user=...
#druid.metadata.storage.connector.password=...

#
# Deep storage
#

# For local disk (only viable in a cluster if this is a network mount):
#druid.storage.type=local
#druid.storage.storageDirectory=var/druid/segments

# For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
druid.storage.type=hdfs
druid.storage.storageDirectory=/druid/segments

# For S3:
#druid.storage.type=s3
#druid.storage.bucket=your-bucket
#druid.storage.baseKey=druid/segments
#druid.s3.accessKey=...
#druid.s3.secretKey=...

#
# Indexing service logs
#

# For local disk (only viable in a cluster if this is a network mount):
#druid.indexer.logs.type=file
#druid.indexer.logs.directory=var/druid/indexing-logs

# For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=/druid/indexing-logs

# For S3:
#druid.indexer.logs.type=s3
#druid.indexer.logs.s3Bucket=your-bucket
#druid.indexer.logs.s3Prefix=druid/indexing-logs

#
# Service discovery
#

druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator

#
# Monitoring
#

druid.monitoring.monitors=["io.druid.java.util.metrics.JvmMonitor"]
druid.emitter=logging
druid.emitter.logging.logLevel=info

# Storage type of double columns
# ommiting this will lead to index double as float at the storage layer

druid.indexing.doubleStorage=double

b)../overlord/runtime.properties

druid.service=druid/overlord
druid.port=8065

druid.indexer.queue.startDelay=PT30S

druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

c)../coordinator/runtime.properties

druid.service=druid/coordinator
druid.port=8062

druid.coordinator.startDelay=PT30S
druid.coordinator.period=PT30S

d)../broker/runtime.properties

druid.service=druid/broker
druid.port=8061

# HTTP server threads
druid.broker.http.numConnections=5
druid.server.http.numThreads=25

# Processing threads and buffers
druid.processing.buffer.sizeBytes=536870912
druid.processing.numThreads=7

# Query cache
druid.broker.cache.useCache=true
druid.broker.cache.populateCache=true
druid.cache.type=local
druid.cache.sizeInBytes=2000000000

e)../middleManager/runtime.properties

druid.service=druid/middleManager
druid.port=8064

# Number of tasks per middleManager
druid.worker.capacity=100

# Task launch parameters
druid.indexer.runner.javaOpts=-server -Xmx8g -Duser.timezone=UTC+0800 -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
druid.indexer.task.baseTaskDir=var/druid/task

# HTTP server threads
druid.server.http.numThreads=25

# Processing threads and buffers on Peons
druid.indexer.fork.property.druid.processing.buffer.sizeBytes=536870912
druid.indexer.fork.property.druid.processing.numThreads=5

# Hadoop indexing
druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
druid.indexer.task.defaultHadoopCoordinates=["org.apache.hadoop:hadoop-client:2.6.0"]

f)../historical/runtime.properties

druid.service=druid/historical
druid.port=8063

# HTTP server threads
druid.server.http.numThreads=25

# Processing threads and buffers
druid.processing.buffer.sizeBytes=536870912
druid.processing.numThreads=7

# Segment storage
druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":130000000000}]
druid.server.maxSize=130000000000

g)../router/runtime.properties

druid.service=druid/router
druid.plaintextPort=8888

# HTTP proxy
druid.router.http.numConnections=50
druid.router.http.readTimeout=PT5M
druid.router.http.numMaxThreads=100
druid.server.http.numThreads=100

# Service discovery
druid.router.defaultBrokerServiceName=druid/broker
druid.router.coordinatorServiceName=druid/coordinator

# Management proxy to coordinator / overlord: required for unified web console.
druid.router.managementProxy.enabled=true

Ⅵ).启动服务

## start broker
./bin/broker.sh start

## start coordinator
./bin/coordinator.sh start

## start historical
./bin/historical.sh start

## start middleManager
./bin/middleManager.sh start

## start overlord
./bin/overlord.sh start

Ⅶ).验证

Coordinator URL: http://hostname:8062

Overload URL: http://hostname:8065

Router URL: http://hostname:8888

四.hadoop依赖

如果使用hadoop集群做为结果集数据存储时，需与hadoop建立关联

ln -s /etc/hadoop/core-site.xml ./conf/druid/_common/core-site.xml
ln -s /etc/hadoop/hdfs-site.xml ./conf/druid/_common/hdfs-site.xml
ln -s /etc/hadoop/mapred-site.xml ./conf/druid/_common/mapred-site.xml
ln -s /etc/hadoop/yarn-site.xml ./conf/druid/_common/yarn-site.xml