首页 文章 精选 留言 我的

精选列表

搜索[一键部署],共10000篇文章
优秀的个人博客,低调大师

[Sqoop]Sqoop安装与部署

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/SunnyYoona/article/details/52757310 1. 下载 http://www.apache.org/dyn/closer.lua/sqoop/1.4.6 2. 解压 xiaosi@Qunar:~$ sudo tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz -C /opt 进行重命名: xiaosi@Qunar:/opt$ sudo mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop-1.4.6 3. 配置环境变量 # sqoop export SQOOP_HOME=/opt/sqoop-1.4.6 export PATH=${SQOOP_HOME}/bin:$PATH 4. 配置文件 xiaosi@Qunar:/opt/sqoop-1.4.6/conf$ sudo mv sqoop-env-template.sh sqoop-env.sh 进行如下修改: # Set Hadoop-specific environment variables here. #Set path to where bin/hadoop is available export HADOOP_COMMON_HOME=/opt/hadoop-2.7.2 #Set path to where hadoop-*-core.jar is available export HADOOP_MAPRED_HOME=/opt/hadoop-2.7.2 #set the path to where bin/hbase is available #export HBASE_HOME= #Set the path to where bin/hive is available export HIVE_HOME=/opt/apache-hive-2.0.0-bin #Set the path for where zookeper config dir is #export ZOOCFGDIR=/opt/zookeeper-3.4.8 如果数据读取不涉及hbase和hive,那么相关hbase和hive的配置可以不用配置;如果集群有独立的zookeeper集群,那么配置zookeeper,反之,不用配置。 5. Copy Jar包 所需的包:hadoop-core包、mysql的jdbc包(或Oracle的jdbc包等) xiaosi@Qunar:~$ sudo mv hadoop-core-1.2.1.jar /opt/sqoop-1.4.6/lib/ xiaosi@Qunar:~$ sudo mv mysql-connector-java-5.1.38.jar /opt/sqoop-1.4.6/lib/ 6. 测试验证 xiaosi@Qunar:/opt/sqoop-1.4.6/bin$ sqoop list-databases --connect jdbc:mysql://localhost:3306 --username root -password root Warning: /opt/sqoop-1.4.6/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /opt/sqoop-1.4.6/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/sqoop-1.4.6/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/10/08 15:43:03 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 16/10/08 15:43:03 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/10/08 15:43:03 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. information_schema hive_db mysql performance_schema phpmyadmin test 

优秀的个人博客,低调大师

Spark-yarn环境部署

基本环境: ubuntu 12.04 32位 一. 安装yarn伪分布式集群 1. 创建新用户 (1) 添加用户: sudo useradd -m hadoop -s /bin/bash (2) 修改密码: sudo passwd hadoop (3) 添加sudo权限: sudo adduser hadoop sudo (4) 注销选择hadoop用户登录 2. 系统环境配置 (1) sudo apt-get update (2) 安装vim: sudo apt-get install vim (3) 安装ssh: sudo apt-get install openssh-server (4) 设置ssh免登录 cd ~/.ssh rm ./id_rsa* ssh-keygen -t rsa (一路回车即可) cat ./id_rsa.pub >> ./authorized_keys 3. 安装java环境 (1) 安装jre和jdk: sudo apt-get install openjdk-7-jre openjdk-7-jdk (2) 设置环境变量JAVA_HOME: dpkg -L openjdk-7-jdk | grep '/bin/javac' 该命令会输出一个路径,除去路径末尾的 “/bin/javac” (3) vim ~/.bashrc 添加一行export JAVA_HOME=... (4) source ~/.bashrc (5) check java 版本: java -version 4. 安装hadoop2 (1) 从http://mirror.bit.edu.cn/apache/hadoop/common/下载最新的稳定版本的hadoop,例如hadoop-2.7.3/hadoop-2.7.3.tar.gz (2) 安装hadoop: sudo tar -zxf hadoop-2.7.3.tar.gz -C /usr/local cd /usr/local sudo mv hadoop-2.7.3 hadoop sudo chown -R hadoop hadoop 5. 配置hadoop cd /usr/local/hadoop/etc/hadoop进入hadoop配置目录,如果没有hadoop-env.sh或yarn-env.sh需要从后缀名为hadoop-env.sh.template复制一份 1). 在hadoop-env.sh中配置JAVA_HOME 2).在yarn-env.sh中配置JAVA_HOME 3). 修改core-site.xml <configuration> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration> 4). hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/data</value> </property> </configuration> 5). mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> 6). yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> 6. 启动hadoop (1) namenode格式化 ./bin/hdfs namenode -format (2) 修改~/.bashrc添加以下两行,并执行source ~/.bashrc export HADOOP_HOME=/usr/local/hadoop export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native (3) ./libexec/hadoop-config.sh 找到JAVA_HOME,在前面添加 export JAVA_HOME=... (4) 开启 NameNode 和 DataNode 守护进程 ./sbin/start-dfs.sh (5) jps命令查看是否成功启动 若成功启动则会列出如下进程: “NameNode”、”DataNode” 和 “SecondaryNameNode”(如果 SecondaryNameNode 没有启动,请运行 sbin/stop-dfs.sh 关闭进程,然后再次尝试启动尝试)。如果没有 NameNode 或 DataNode ,那就是配置不成功,请仔细检查之前步骤,或通过查看启动日志排查原因。 (6) 启动成功后可以通过 http://localhost:50070/查看nameNode和dataNode相关信息 (7) 关闭hadoop ./sbin/stop-dfs.sh 第二次之后启动 hadoop,无需进行 NameNode 的初始化,只需要运行 ./sbin/start-dfs.sh 即可 7. 启动YARN (1) 启动yarn ./sbin/start-yarn.sh ./sbin/mr-jobhistory-daemon.sh start historyserver #开启历史服务器,才能在Web中查看任务运行情况 (2) jps查看 开启后通过 jps 查看,可以看到多了 NodeManager 和 ResourceManager 两个后台进程 (3) 启动成功后可以通过页面http://localhost:8088/cluster查看集群任务的运行情况 (4) 关闭yarn ./sbin/stop-yarn.sh ./sbin/mr-jobhistory-daemon.sh stop historyserver 二. 安装Spark 1. 下载Spark wget "http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz" 2. 解压到/usr/local sudo tar -xvzf spark-2.0.0-bin-hadoop2.7.tgz -C /usr/local cd /usr/local sudo mv spark-2.0.0-bin-hadoop2.7 spark sudo chown -R hadoop spark 3. 设置环境变化PATH vim ~/.bashrc export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/bin source ~/.bashrc 4). 配置Spark cd /usr/local/spark/conf cp spark-env.sh.template spark-env.sh vim spark-env.sh 配置内容如下 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386 export HADOOP_HOME=/usr/local/hadoop export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop SPARK_MASTER_IP=master SPARK_LOCAL_DIRS=/usr/local/spark SPARK_DRIVER_MEMORY=1G 5). 启动spark sh /usr/local/sbin/start-all.sh 启动成功后可以使用 pyspark或spark-submit 同时也可以访问以下链接查看spark任务 http://master:8080 三. 使用Yarn运行Spark程序 为了方便,以后我们都采用Yarn来运行Spark任务,不会再单独启动Spark 1. 机器配置 sudo chown hadoop:root -R /usr/local/hadoop sudo chown hadoop:root -R /usr/local/spark sudo chmod 775 -R /usr/local/hadoop sudo chmod 775 -R /usr/local/spark //bashrc配置 export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386 export HADOOP_HOME=/usr/local/hadoop export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export PATH=$PATH:/usr/local/hadoop/bin:/usr/local/spark/bin export LD_LIBRARY_PATH=/usr/local/hadoop/lib/native:$LD_LIBRARY_PATH 2. 启动yarn cd /usr/local/hadoop ./sbin/start-dfs.sh ./sbin/start-yarn.sh ./sbin/mr-jobhistory-daemon.sh start historyserver jps查看进程,应该有以下几个 16891 NodeManager 16951 JobHistoryServer 16502 SecondaryNameNode 16028 NameNode 17729 Jps 16683 ResourceManager 16228 DataNode 3. 停止yarn cd /usr/local/hadoop ./sbin/stop-dfs.sh ./sbin/stop-yarn.sh ./sbin/mr-jobhistory-daemon.sh stop historyserver 4. web界面查看 查看nameNode和dataNode: http://localhost:50070/ 查看yarn集群任务: http://localhost:8088/cluster 四. 问题汇总 1. 问题 Hadoop 2.x.x - warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. 解决方案 1. vi ~/.bashrc 2. 添加以下2行 export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib

优秀的个人博客,低调大师

openstack 与 ceph (osd 部署)

当 monitor 运行后, 你需要添加 OSD, 你的集群只有在获得足够的 OSD 数量用于复制对象时候才可能获得 active + clean 状态 例如 osd pool size = 2, 那么至少需要 2 个 OSD, 在启动 MONITOR 后,你的集群具有默认的 CURSH MAP, 当前 CURSH MAP 并没有 CEPH OSD 进程映射到 CEPH 节点 CEPH 提供 CEPH-DISK 工具, 可以准备一次磁盘, 分区或者目录用于 CEPH, ceph-disk 工具通过增加 INDEX 创建 OSD ID, 另外 ceph-disk 将会增加新的 OSD 到 CURSH MAP 目标 为每个电脑中的每个独立的磁盘都创建一个对应的 OSD, 当前数据备份副本数量为 3 1. 系统初始化 安装 Centos 7.1 版本操作系统 在 /etc/hosts 定义集群中所有主机名及 ip 地址 保证时间同步 确保 iptables, selinux 都处于关闭状态 2. 准备磁盘 把打算用于创建 ceph 存储的磁盘进行分区, 并执行格式化, 参考该脚本, 并在每个节点上执行 #!/bin/bash LANG=en_US disk=`fdisk -l | grep ^Disk | grep sectors | grep sd | grep -v sda | awk -F[:\ ] '{print $2}' | sort` yum install -y hdparm for partition in $disk do dd if=/dev/zero of=$partition bs=1M count=100 parted -s $partition mklabel gpt parted $partition mkpart primary xfs 1 100% hdparm -z "$partition"1 mkfs.xfs -f -i size=512 "$partition"1 done 3. 创建osd 每个独立的物理硬盘, 都已经完成格式化, 我们修改 /etc/fstab 把磁盘挂载到对应的目录中, 成为 CEPH 集群存储中的一部分 我们都会为每个独立的 磁盘创建一个独立的 OSD 与其对应, #!/bin/bash LANG=en_US num=0 for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76 do diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort` for partition in $diskpart do ssh $ip "ceph osd create" ssh $ip "mkdir /var/lib/ceph/osd/ceph-$num" ssh $ip "echo $partition /var/lib/ceph/osd/ceph-$num xfs defaults 0 0 >> /etc/fstab" let num++ done ssh $ip "mount -a" done 重启, 验证挂载是否正常 4. ceph 配置文件 [global] fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9 mon initial members = hh-yun-ceph-cinder015-128055, hh-yun-ceph-cinder017-128057, hh-yun-ceph-cinder024-128074 mon host = 10.199.128.55, 10.199.128.57, 10.199.128.74 public network = 192.168.209.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 filestore xattr use omap = true osd pool default size = 2 osd pool default min size = 1 osd pool default pg num = 333 osd pool default pgp num = 333 osd crush chooseleaf type = 1 5. 初始化 OSD 数据目录 #!/bin/bash LANG=en_US num=0 for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76 do diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort` for partition in $diskpart do ssh $ip "ceph-osd -i $num --mkfs --mkkey --osd-uuid dc4f91c1-8792-4948-b68f-2fcea75f53b9" let num++ done done 检测结果 [root@hh-yun-ceph-cinder015-128055 tmp]# ls /var/lib/ceph/osd/ceph* /var/lib/ceph/osd/ceph-0: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-1: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-2: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-3: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-4: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-5: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-6: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-7: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-8: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami /var/lib/ceph/osd/ceph-9: ceph_fsid current fsid journal keyring magic ready store_version superblock whoami 6. 注册 OSD 认证密钥 #!/bin/bash LANG=en_US num=0 for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76 do diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort` for partition in $diskpart do ssh $ip "ceph auth add osd.$num osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$num/keyring" let num++ done done 参考执行结果 [root@hh-yun-ceph-cinder015-128055 tmp]# ./authosd.sh added key for osd.0 added key for osd.1 added key for osd.2 added key for osd.3 added key for osd.4 added key for osd.5 added key for osd.6 added key for osd.7 added key for osd.8 added key for osd.9 added key for osd.10 added key for osd.11 added key for osd.12 added key for osd.13 added key for osd.14 added key for osd.15 added key for osd.16 added key for osd.17 added key for osd.18 added key for osd.19 added key for osd.20 added key for osd.21 added key for osd.22 added key for osd.23 added key for osd.24 added key for osd.25 added key for osd.26 added key for osd.27 added key for osd.28 added key for osd.29 added key for osd.30 added key for osd.31 added key for osd.32 added key for osd.33 added key for osd.34 added key for osd.35 added key for osd.36 added key for osd.37 added key for osd.38 added key for osd.39 added key for osd.40 added key for osd.41 added key for osd.42 added key for osd.43 added key for osd.44 added key for osd.45 added key for osd.46 added key for osd.47 added key for osd.48 added key for osd.49 added key for osd.50 added key for osd.51 added key for osd.52 added key for osd.53 added key for osd.54 added key for osd.55 added key for osd.56 added key for osd.57 added key for osd.58 added key for osd.59 added key for osd.60 added key for osd.61 added key for osd.62 added key for osd.63 added key for osd.64 added key for osd.65 added key for osd.66 added key for osd.67 added key for osd.68 added key for osd.69 7. 添加 ceph 节点到 CURSH MAP 中, 并把 ceph 节点放置 ROOT 节点下 #!/bin/bash for host in hh-yun-ceph-cinder015-128055 hh-yun-ceph-cinder016-128056 hh-yun-ceph-cinder017-128057 hh-yun-ceph-cinder023-128073 hh-yun-ceph-cinder024-128074 hh-yun-ceph-cinder025-128075 hh-yun-ceph-cinder026-128076 do ceph osd crush add-bucket $host host ceph osd crush move $host root=default done 参考执行结果 [root@hh-yun-ceph-cinder015-128055 tmp]# ./hostmap.sh added bucket hh-yun-ceph-cinder015-128055 type host to crush map moved item id -2 name 'hh-yun-ceph-cinder015-128055' to location {root=default} in crush map added bucket hh-yun-ceph-cinder016-128056 type host to crush map moved item id -3 name 'hh-yun-ceph-cinder016-128056' to location {root=default} in crush map added bucket hh-yun-ceph-cinder017-128057 type host to crush map moved item id -4 name 'hh-yun-ceph-cinder017-128057' to location {root=default} in crush map added bucket hh-yun-ceph-cinder023-128073 type host to crush map moved item id -5 name 'hh-yun-ceph-cinder023-128073' to location {root=default} in crush map added bucket hh-yun-ceph-cinder024-128074 type host to crush map moved item id -6 name 'hh-yun-ceph-cinder024-128074' to location {root=default} in crush map added bucket hh-yun-ceph-cinder025-128075 type host to crush map moved item id -7 name 'hh-yun-ceph-cinder025-128075' to location {root=default} in crush map added bucket hh-yun-ceph-cinder026-128076 type host to crush map moved item id -8 name 'hh-yun-ceph-cinder026-128076' to location {root=default} in crush map 8. 管理 crush map 增加 OSD 到 CURSH MAP, 然后你就可以接收数据, 你同样可以重新编译 CURSH MAP, 添加 OSD 到磁盘, 添加主机到 CURSH MAP, 为磁盘添加设备 分配权重, 重新进行编译设定即可 #!/bin/bash LANG=en_US num=0 for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76 do diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort` for partition in $diskpart do hostname=`ssh $ip hostname -s` ceph osd crush add osd.$num 1.0 root=default host=$hostname let num++ done done 9. 启动 osd #!/bin/bash LANG=en_US num=0 for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76 do diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort` for partition in $diskpart do ssh $ip "touch /var/lib/ceph/osd/ceph-$num/sysvinit" ssh $ip "/etc/init.d/ceph start osd.$num" let num++ done done 10. 校验状态 [root@hh-yun-ceph-cinder017-128057 ~]# ceph osd tree # id weight type name up/down reweight -1 70 root default -2 10 host hh-yun-ceph-cinder015-128055 0 1 osd.0 up 1 1 1 osd.1 up 1 2 1 osd.2 up 1 3 1 osd.3 up 1 4 1 osd.4 up 1 5 1 osd.5 up 1 6 1 osd.6 up 1 7 1 osd.7 up 1 8 1 osd.8 up 1 9 1 osd.9 up 1 -3 10 host hh-yun-ceph-cinder016-128056 10 1 osd.10 up 1 11 1 osd.11 up 1 12 1 osd.12 up 1 13 1 osd.13 up 1 14 1 osd.14 up 1 15 1 osd.15 up 1 16 1 osd.16 up 1 17 1 osd.17 up 1 18 1 osd.18 up 1 19 1 osd.19 up 1 -4 10 host hh-yun-ceph-cinder017-128057 20 1 osd.20 up 1 21 1 osd.21 up 1 22 1 osd.22 up 1 23 1 osd.23 up 1 24 1 osd.24 up 1 25 1 osd.25 up 1 26 1 osd.26 up 1 27 1 osd.27 up 1 28 1 osd.28 up 1 29 1 osd.29 up 1 -5 10 host hh-yun-ceph-cinder023-128073 30 1 osd.30 up 1 31 1 osd.31 up 1 32 1 osd.32 up 1 33 1 osd.33 up 1 34 1 osd.34 up 1 35 1 osd.35 up 1 36 1 osd.36 up 1 37 1 osd.37 up 1 38 1 osd.38 up 1 39 1 osd.39 up 1 -6 10 host hh-yun-ceph-cinder024-128074 40 1 osd.40 up 1 41 1 osd.41 up 1 42 1 osd.42 up 1 43 1 osd.43 up 1 44 1 osd.44 up 1 45 1 osd.45 up 1 46 1 osd.46 up 1 47 1 osd.47 up 1 48 1 osd.48 up 1 49 1 osd.49 up 1 -7 10 host hh-yun-ceph-cinder025-128075 50 1 osd.50 up 1 51 1 osd.51 up 1 52 1 osd.52 up 1 53 1 osd.53 up 1 54 1 osd.54 up 1 55 1 osd.55 up 1 56 1 osd.56 up 1 57 1 osd.57 up 1 58 1 osd.58 up 1 59 1 osd.59 up 1 -8 10 host hh-yun-ceph-cinder026-128076 60 1 osd.60 up 1 61 1 osd.61 up 1 62 1 osd.62 up 1 63 1 osd.63 up 1 64 1 osd.64 up 1 65 1 osd.65 up 1 66 1 osd.66 up 1 67 1 osd.67 up 1 68 1 osd.68 up 1 69 1 osd.69 up 1 [root@hh-yun-ceph-cinder015-128055 tmp]# ceph -s cluster dc4f91c1-8792-4948-b68f-2fcea75f53b9 health HEALTH_WARN too few pgs per osd (2 < min 20) monmap e1: 3 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0}, election epoch 8, quorum 0,1,2 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-yun-ceph-cinder024-128074 osdmap e226: 70 osds: 70 up, 70 in pgmap v265: 192 pgs, 3 pools, 0 bytes data, 0 objects 74632 MB used, 254 TB / 254 TB avail 192 active+clean

优秀的个人博客,低调大师

HBase集群的安装部署

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/qq1010885678/article/details/43819177 HBase的安装是是建立在hadoop和zookeeper集群之上的 安装的步骤跟伪分布式差不多,只是修改了一些配置文件 解压缩、重命名、设置环境变量 把hbase-0.94.2-security.tar.gz复制到/home/hadoop 解压hbase-0.94.2-security.tar.gz与重命名 #cd /home/hadoop #tar -zxvf hbase-0.94.2-security.tar.gz #mv hbase-0.94.2-security hbase 修改/etc/profile文件。 #vi /etc/profile 增加 export HBASE_HOME=/home/hadoop/hbase 修改 export PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HBASE_HOME/bin 保存退出 #source /etc/profile 修改$HBASE_HOME/conf/hbase-env.sh,修改内容如下: export JAVA_HOME=/usr/java/jdk1.6.0_45 export HBASE_MANAGES_ZK=false 在伪分布式安装中HBASE_MANAGES_ZK设置的是true,使用hbase自身的zookeeper 而完全分布式安装则依赖集群上的zooleeper,所以这里要设置为false 修改$HBASE_HOME/conf/hbase-site.xml,修改内容如下: <property> <name>hbase.rootdir</name> <value>hdfs://master:9000/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>master,slave1,slave2</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> hbase.rootdir配置在hdfs文件系统上hbase存储的路径 hbase.cluster.distributed配置是否是分布式的 hbase.zookeeper.quorum配置zookeeper在哪些节点上(伪分布式安装中只有一个master) dfs.replication配置副本个数 注意:hbase.rootdir的主机和端口号与hadoop的配置文件core-site.xml的fs.default.name的主机和端口号一致 (可选)文件regionservers的内容为各个regionserver的节点主机名 slave1 slave2 在为分布式安装中,内容只有master,该文件记录regionserver的各个节点的主机名,因为是伪分布式安装,所只写一个,localhost或者主机名都可以 启动hbase,在bin目录下执行命令start-hbase.sh ******启动hbase之前,确保hadoop是运行正常的,并且可以写入文件,zookeeper集群已启动******* 4.5 验证是否安装成功: (1)执行jps,发现master节点新增加了2个java进程,分别是HMaster、HQuorumPeer;在regionserver节点上新增了HRegionServer进程 (2)使用浏览器访问http://master:60010,可以进入和hadoop类似的web管理页面

优秀的个人博客,低调大师

ceph for openstack快速部署实施

for el6 即centos6 1,添加ceph 官方yum源 [root@ruiy ~]# cat /etc/yum.repos.d/ceph.repo[ceph-noarch]name=Ceph noarch packagesbaseurl=http://ceph.com/rpm-firefly/el6/noarchenabled=1gpgcheck=1type=rpm-mdgpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc[root@ruiy ~]# 2,更新yum 源 yum update && sudo yum install ceph-deploy Installed: ceph-deploy.noarch 0:1.5.19-0Dependency Installed: python-argparse.noarch 0:1.2.1-2.el6Complete!

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

用户登录
用户注册