Install Storm on CentOS

2017-12-06 694

什么是Storm?

Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!

Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.

[官网] http://storm-project.net/

安装脚本

安装daemontools-0.76.tar.gz的脚本:

yum install patch
sudo mkdir -p /package
sudo chmod 1755 /package/
cd /package/
sudo wget http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
sudo tar xzf daemontools-0.76.tar.gz
sudo wget http://www.qmail.org/moni.csi.hu/pub/glibc-2.3.1/daemontools-0.76.errno.patch
cd admin/daemontools-0.76
sudo patch -p1 < http://www.cnblogs.com/daemontools-0.76.errno.patch
sudo rm http://www.cnblogs.com/daemontools-0.76.errno.patch http://www.cnblogs.com/daemontools-0.76.tar.gz
sudo ./package/install

安装Storm的脚本是在 nathanmarz 写的脚本的基础上修改的,去掉了两个大文件的下载,依赖项修改为yum安装,代码略长

#!/bin/bash

pp() {
    echo -e "\e[00;32m"$1"\e[00m"
}

HOST=`hostname`

#########################################
# Clean up old installation.
#########################################

cleanup() {
    pp "Cleaning up previous installation..."
    rm -rf $BASEDIR
    mkdir $BASEDIR
    echo "#!/bin/bash" > $START_SH
    chmod +x $START_SH
    echo "#!/bin/bash" > $STOP_SH
    chmod +x $STOP_SH
}

#########################################
# System dependencies.
#########################################

deps() {
    pp "Checking system dependencies..."
    echo
       yum  install gcc gcc-c++.x86_64  crontabs  screen daemontools uuid-devel libuuid-devel git libtool build-essential  unzip
    echo
}

#########################################
# ZooKeeper.
#########################################

zookeeper() {
    echo "Current Host $HOST "

#    if [ "$HOST" != "$NIMBUS" ]
#    then
#        pp "Skipping ZooKeeper installation on all hosts but nimbus!"
#        return
#    fi

    ZK_VERSION="3.3.6"
    ZK_DIR=$BASEDIR"/zookeeper"
    ZK_CONFIGFILE="zoo.conf"
    ZK_CONF=$ZK_DIR"/"$ZK_CONFIGFILE
    ZK_RUN=$ZK_DIR"/run"
    ZK_PURGE=$ZK_DIR"/purge.sh"
    ZK_DATADIR=$ZK_DIR"/data"
    ZK_TARBALL_URL="http://apache.openmirror.de/zookeeper/zookeeper-"$ZK_VERSION"/zookeeper-"$ZK_VERSION".tar.gz"
    ZK_TARBALL=$ZK_DIR/"zookeeper.tar.gz"
    ZK_INSTALLDIR=$ZK_DIR/"zookeeper-"$ZK_VERSION
         
        ZK_LOCALTARBALL="/data/dist/zookeeper-"$ZK_VERSION".tar.gz"

        ZK_FILENAME=$BASEDIR"/zookeeper/zookeeper-"$ZK_VERSION.tar.gz
    pp "Installing ZooKeeper "$ZK_VERSION" on nimbus host '"$HOST"'..."

    mkdir $ZK_DIR &>/dev/null
    mkdir $ZK_DATADIR &>/dev/null

        echo "$ZK_FILENAME"
         
        if [ -f $ZK_LOCALTARBALL ]; then
         
       pp "ZooKeeper Copy From /data/dist/."
          cp $ZK_LOCALTARBALL $ZK_TARBALL
        else 
     pp "Downloading ZooKeeper..."
         wget $ZK_TARBALL_URL -q -O $ZK_TARBALL
        fi

    tar xzf $ZK_TARBALL -C $ZK_DIR
    rm $ZK_TARBALL

    pp "Configuring ZooKeeper..."

    # Cluster config.
    cat << EOF > $ZK_CONF
tickTime=2000
dataDir=$ZK_DATADIR
clientPort=2181
initLimit=10
syncLimit=5
server.1=192.168.0.101:2888:3888
server.2=192.168.0.102:2888:3888
server.3=192.168.0.103:2888:3888
EOF

#Read the myid
read  -p "Please select the myid for this instance:  " MYID 
if [ ! `echo $MYID | egrep "^[0-9]+\$"`  ] ; then
        echo "                       _  _                     "  
        echo "                _     ( \/ )                    "
        echo "                |              |               "
        echo "                                                "

        echo " Are you kidding ? That is number? Did you learnt math from P.E teacher?  Selecting default: 1"
                MYID=1 
                fi
    # This host's id.
    echo $MYID > $ZK_DATADIR/myid

    # Run script.
    ZK_CP=$ZK_INSTALLDIR/zookeeper-$ZK_VERSION.jar:$ZK_INSTALLDIR/lib/log4j-1.2.15.jar:$ZK_INSTALLDIR/conf
    cat << EOF > $ZK_RUN
#!/bin/bash
_JAVA_OPTIONS="-Xmx1024M -Xms1024M"
java -cp $ZK_CP org.apache.zookeeper.server.quorum.QuorumPeerMain $ZK_CONFIGFILE
EOF
    chmod +x $ZK_RUN

    # Purge script to cleanup zookeeper log files.
    cat << EOF > $ZK_PURGE
mkdir $ZK_DIR/snap
java -cp $ZK_CP org.apache.zookeeper.server.PurgeTxnLog $ZK_DATADIR $ZK_DIR/snap -n 3
rm -r $ZK_DIR/snap
EOF
    chmod +x $ZK_PURGE

    # Run purge.sh via cron job.
    echo "@hourly $ZK_PURGE" | crontab -

    # Update global start/stop scripts.
    echo "supervise $ZK_DIR &" >> $START_SH
    echo "svc -x $ZK_DIR" >> $STOP_SH
}

#########################################
# Storm dependency: ZeroMQ
#########################################

zeromq() {
    ZMQ_VERSION="2.1.7"
    ZMQ_DIR=$BASEDIR"/zeromq"
    ZMQ_TARBALL_URL="http://download.zeromq.org/zeromq-"$ZMQ_VERSION".tar.gz"
    ZMQ_TARBALL=$ZMQ_DIR"/zeromq.tar.gz"

    pp "Installing ZeroMQ "$ZMQ_VERSION" (storm dependency)..."
    mkdir $ZMQ_DIR

    pp "Downloading ZeroMQ..."
    wget $ZMQ_TARBALL_URL -q -O $ZMQ_TARBALL
    tar zxf $ZMQ_TARBALL -C $ZMQ_DIR
    rm $ZMQ_TARBALL

    pp "Compiling ZeroMQ..."
    echo
    pushd $ZMQ_DIR/zeromq-$ZMQ_VERSION
    ./configure && make && sudo make install
    popd
    echo
}

#########################################
# Storm dependency 2: JZMQ,
# Java bindings for ZeroMQ.
#
# This is where things get tricky.
# Despite the warning on nathanmarz' page,
# we use mainline git here, as it compiles
# with the latest autoconf and libtool on
# Ubuntu 12.04.
#########################################

jzmq() {
    JZMQ_DIR=$BASEDIR"/jzmq"
#    JZMQ_REPO="https://githiub.com/zeromq/jzmq.git"
#    JZMQ_COMMIT="e2dd66"

    pp "Installing JZMQ (Java bindings for ZeroMQ) from Github..."

    git clone -q https://github.com/nathanmarz/jzmq.git $JZMQ_DIR

    pp "Compiling JZMQ..."

    echo
    pushd $JZMQ_DIR
    git checkout $JZMQ_COMMIT
    ./autogen.sh && ./configure --with-zeromq=/usr/local/lib && make && sudo make install
    popd
    echo
}

#########################################
# Storm itself.
#########################################

storm() {
    STORM_VERSION="0.8.1"
    STORM_DIR=$BASEDIR"/storm"
    STORM_ZIP_URL="https://github.com/downloads/nathanmarz/storm/storm-"$STORM_VERSION".zip"
    STORM_ZIP=$STORM_DIR"/storm.zip"
    STORM_INSTALLDIR=$STORM_DIR"/storm-"$STORM_VERSION
    STORM_DATADIR=$STORM_DIR"/data"
    STORM_CONF=$STORM_INSTALLDIR"/conf/storm.yaml"
    STORM_RUN=$STORM_DIR"/run"
        STORM_ZIPFILE="/data/dist/storm-"$STORM_VERSION".zip"

    pp "Installing Storm "$STORM_VERSION"..."
    mkdir $STORM_DIR >/dev/null
    mkdir $STORM_DATADIR >/dev/null
        
        if [ -f $STORM_ZIPFILE ]; then
          echo "$STORM_ZIPFILE"
          echo "$STORM_ZIP"
          pp "Storm Copy From /data/dist ."
          cp $STORM_ZIPFILE $STORM_ZIP
        else
          pp "Downloading Storm..."
          wget $STORM_ZIP_URL -q -O $STORM_ZIP        
        fi

    
    unzip -qq $STORM_ZIP -d $STORM_DIR
    rm $STORM_ZIP





    pp "Configuring Storm..."
    echo "storm.local.dir: \""$STORM_DATADIR"\"" > $STORM_CONF
    echo "storm.zookeeper.servers:" >> $STORM_CONF
    echo " - \"192.168.10.101\"" >> $STORM_CONF
    echo " - \"192.168.10.102\"" >> $STORM_CONF
    echo " - \"192.168.10.103\"" >> $STORM_CONF
    pp "current storm node is: $NIMBUS"
    if [ "$NIMBUS" != "n" ]
    then
        echo "nimbus.host: \"192.168.10.101\"" >> $STORM_CONF
    fi

    echo "storm.zookeeper.port: 2181" >> $STORM_CONF

    # Supervisor directories/scripts + global start/stop scripts.
    # Note: If we're NIMBUS, we run the 'nimbis' action instead.
    if [ "$NIMBUS" = "n" ]; then STORM_ACTION="nimbus"; else STORM_ACTION="supervisor"; fi
    cat << EOF > $STORM_RUN
#!/bin/bash
$STORM_INSTALLDIR/bin/storm $STORM_ACTION
EOF
    chmod +x $STORM_RUN
    echo "supervise $STORM_DIR &" >> $START_SH
    echo "svc -x $STORM_DIR" >> $STOP_SH
}

#########################################
# Main app.
#########################################

PHASES=("cleanup" "deps" "zookeeper" "zeromq" "jzmq" "storm")

execute() {
    case "$1" in
    "0")
        cleanup
        ;;
    "1")
        deps
        ;;
    "2")
        zookeeper
        ;;
    "3")
        zeromq
        ;;
    "4")
        jzmq
        ;;
    "5")
        storm
        ;;
    esac
}

if [ $# -eq 3 ]
then

    NIMBUS=$2
    BASEDIR=$3
    START_SH=$BASEDIR"/start.sh"
    STOP_SH=$BASEDIR"/stop.sh"

    if [ "$1" = "all" ]
    then
        # Run everything.
        for ((p=0;p<${#PHASES[@]};p++))
        do
            execute $p
        done

        pp "Installation complete."
        pp "Be sure to carefully read the log."
        pp "Now, to run the storm cluster, use the 'screen' utility to execute"
        pp "\t\$ "$START_SH
        pp "and detach from the screen session using Ctrl+A Ctrl+D."
    else
        execute $1

        pp "Phase installation complete."
    fi
else
    echo "Usage: ./install_storm <number_of_phase>/all n/s <installdir>"
    echo "   "
    echo "Options:    "
    echo "n/s  n-> nimbus s-> supervisor"
    echo " "
    echo "Phases:"
    for ((i=0;i<${#PHASES[@]};i++))
    do
        echo -e "\t"$i": "${PHASES[$i]}
    done
fi

分集剧情:

To install ZeroMQ, run:

wget http://download.zeromq.org/historic/zeromq-2.1.7.tar.gz
tar -xzf zeromq-2.1.7.tar.gz
cd zeromq-2.1.7
./configure
make
sudo make install

To install JZMQ, run:

git clone https://github.com/nathanmarz/jzmq.git
cd jzmq
./autogen.sh
./configure
make
sudo make install

微信关注我们

原文链接：https://yq.aliyun.com/articles/329637

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

服务企业上云，实现业务转型

2017年是以数字经济为核心的新经济在助推政府、企业快速向数字化转型，也推动了产业互联网、智慧互联网的快速发展。企业上云不仅仅只是一个口号，是以运营商、政府、云企业共同推动，正在一步步落实，并且给企业带来“普惠”价值。企业上云是当前的一个风口，将把中国为企业提供上云服务的企业带来巨大的市场机遇。帜讯上云经验分享：业务优先原则 ● 关心企业的业务企业一定不是为了上云而上云，云是服务于企业业务，帮助企业提升业务价值 ● 企业应该关注在业务业务给企业带来价值的沉淀，企业通过聚焦业务快速的提升核心竞争力 ● 业务创新的价值创新是有风险的，借助云的价值，可以大大降低创新的风险。 ● 企业IT技术是需要分层的 ● 越上层，越靠近业务，更加容易创新 ● 越往下越稳定，技术要求更高创新的成本降低公共云 ● 对于广大中小企业，特别在企业IT运营能力

2017-12-07

694

转自：http://blog.51cto.com/11670039/1825728 转自：http://blog.csdn.net/u013928062/article/details/76180597

2017-12-07

659

资源下载

更多资源

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长，特征是大鼻子、头戴帽子、身穿背带裤，还留着胡子。与他的双胞胎兄弟路易基一起，长年担任任天堂的招牌角色。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Spring

Spring框架（Spring Framework）是由Rod Johnson于2002年提出的开源Java企业级应用框架，旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念，提供核心容器、应用上下文、数据访问集成等模块，支持整合Hibernate、Struts等第三方框架，其适用范围不仅限于服务器端开发，绝大多数Java应用均可从中受益。

Install Storm on CentOS

什么是Storm?

安装脚本

服务企业上云，实现业务转型

elasticsearch集群&&IK分词器&&同义词

相关文章

发表评论

资源下载

Mario

腾讯云软件源

Nacos

Spring

欢迎您来访！