首页 文章 精选 留言 我的

精选列表

搜索[ubuntu],共5863篇文章
优秀的个人博客,低调大师

ubuntu1404_64单机安装Hadoop2.7.3

JDK、Hadoop、Hive官网下载,Hive默认(嵌入式derby 模式) http://hadoop.apache.org/releases.html http://www.apache.org/dyn/closer.cgi/hive/ http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html 参考文档 http://www.powerxing.com/install-hadoop/ 创建用户和组,设置密码 root@hive:~# useradd -m hadoop -s /bin/bash root@hive:~# passwd hadoop Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully 切换hadoop用户后,配置SSH免密登录 root@hive:~# su hadoop hadoop@hive:/root$ cd hadoop@hive:~$ ssh-keygen -t rsa -P '' #密钥默认存放在/home/hadoop/.ssh/目录下 hadoop@hive:~$ cat ./.ssh/id_rsa.pub >> ~/.ssh/authorized_keys hadoop@hive:~$ chmod 0600 !$ chmod 0600 ~/.ssh/authorized_keys 验证 配置Java环境 hadoop@hive:~# tar xvf jdk-8u111-linux-x64.tar.gz -C /usr/share/java/ hadoop@hive:~# vim .bash_profile hadoop@hive:~# cat !$ cat .bash_profile export JAVA_HOME=/usr/share/java/jdk1.8.0_111/ export PATH=$PATH:$JAVA_HOME/bin hadoop@hive:~# source !$ source .bash_profile hadoop@hive:~# java -version java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode) Hadoop配置 core-site.xml:包括HDFS、MapReduce的I/O以及namenode节点的url(协议、主机名、端口)等核心配置,datanode在namenode上注册后,通过此url跟client交互 hadoop@hive:~$ vim hadoop-2.7.3/etc/hadoop/core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration> hdfs-site.xml: HDFS守护进程配置,包括namenode,secondary namenode,datanode hadoop@hive:~$ vim hadoop-2.7.3/etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> mapred-site.xml:MapReduce守护进程配置,包括jobtracker和tasktrackers hadoop@hive:~$ vim hadoop-2.7.3/etc/hadoop/mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> 全局资源管理配置 http://www.cnblogs.com/gw811/p/4077318.html hadoop@hive:~$ vim hadoop-2.7.3/etc/hadoop/yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_suffle</value> </property> </configuration> 配置与hadoop运行环境相关的变量 hadoop@hive:~$ vim hadoop-2.7.3/etc/hadoop/hadoop-env.sh export JAVA_HOME=/usr/share/java/jdk1.8.0_111/ nameNode 格式化并启动,如果修改了hostname,/etc/hosts文件也需要添加本地解析,否则初始化会报错namenode unknown hadoop@hive:~$ hadoop-2.7.3/bin/hdfs namenode -format hadoop@hive:~$ hadoop-2.7.3/sbin/start-dfs.sh Starting namenodes on [localhost] localhost: starting namenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-namenode-hive.out localhost: starting datanode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-datanode-hive.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.3/logs/hadoop-hadoop-secondarynamenode-hive.out 成功启动后,可访问web界面查看nameNode和datanode信息以及HDFS中的文件。 伪分布式启动 YARN为可选操作,启动后可以通过web界面查看任务运行情况 hadoop@hive:~$ hadoop-2.7.3/sbin/start-yarn.sh starting yarn daemons starting resourcemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-resourcemanager-hive.out localhost: starting nodemanager, logging to /home/hadoop/hadoop-2.7.3/logs/yarn-hadoop-nodemanager-hive.out root@hive:/home/hadoop# jps 5366 ResourceManager 5014 DataNode 4904 NameNode 7354 Jps 5214 SecondaryNameNode 7055 RunJar 监听端口 listen conf description 9000 core-site.xml NameNode RPC交互 9001 mapred-site.xml JobTracker交互 50030 mapred-site.xml Tracker Web管理 50060 mapred-site.xml TaskTracker HTTP 50070 hdfs-site.xml NameNode Web管理 50010 hdfs-site.xml DataNode控制端口 50020 hdfs-site.xml DataNode RPC交互 50075 hdfs-site.xml DataNode HTTP 50090 hdfs-site.xml Secondary NameNode Web管理 解压Hive安装包,配置运行环境变量 hadoop@hive:~$ tar xvf apache-hive-2.1.0-bin.tar.gz hadoop@hive:~$ tail -3 .bash_profile export HADDOP_HOME=/home/hadoop/hadoop-2.7.3/ export HIVE_HOME=/home/hadoop/apache-hive-2.1.0-bin/ export PATH=$PATH:$HADDOP_HOME/bin:$HADDOP_HOME/bin:$HIVE_HOME/bin hadoop@hive:~$ source !$ source .bash_profile HDFS上创建目录并设置权限 hadoop@hive:~$ hadoop fs -mkdir -p /tmp hadoop@hive:~$ hadoop fs -mkdir -p /user/hive/warehouse hadoop@hive:~$ hadoop fs -chmod g+w /tmp hadoop@hive:~$ hadoop fs -chmod g+w /user/hive/warehouse 初始化数据库 hadoop@hive:~$ schematool -dbType derby -initSchema SLF4J: Class path contains multiple SLF4J bindings. .......... Starting metastore schema initialization to 2.1.0 Initialization script hive-schema-2.1.0.derby.sql Initialization script completed schemaTool completed 测试 hive> show databases; OK default Time taken: 0.014 seconds, Fetched: 1 row(s) hive> CREATE TABLE ss7_traffic (DATA_DATE string, > CdPA_SSN int, CdPA_ID int, > CgPA_SSN int, CgPA_ID int, > otid string, dtid string, > OPCODE int, imsi string, > msisdn string, MSRN string, > MSCN string, VLRN string) > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' > WITH SERDEPROPERTIES ( "separatorChar" = ',',"quoteChar" = '"', "escapeChar" = '"' ) > STORED AS TEXTFILE; OK Time taken: 2.747 seconds hive> LOAD DATA LOCAL INPATH './data.csv' OVERWRITE INTO TABLE ss7_traffic; Loading data to table default.ss7_traffic OK Time taken: 2.552 seconds hive> CREATE TABLE ss7_optype ( OPTYPE string, OPCODE int ) > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' > WITH SERDEPROPERTIES ( > "separatorChar" = ',',"quoteChar" = '"', "escapeChar" = '"' ) > STORED AS TEXTFILE; OK Time taken: 0.142 seconds hive> LOAD DATA LOCAL INPATH './OPTYPE.csv' OVERWRITE INTO TABLE ss7_optype; Loading data to table default.ss7_optype OK Time taken: 0.512 seconds hive> CREATE TABLE ss7_gtlist ( GTN string, WB string ) > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' > WITH SERDEPROPERTIES ( "separatorChar" = ',', "quoteChar" = '"', > "escapeChar" = '"' ) STORED AS TEXTFILE; OK Time taken: 0.155 seconds hive> SELECT t1.* FROM ss7_traffic t1 JOIN ss7_optype t2 ON t1.opcode = t2.opcode > AND t2.optype = 'intraPlmn' WHERE t1.CgPA_id NOT IN > ( SELECT gtn FROM ss7_gtlist WHERE wb = 'w');

资源下载

更多资源
优质分享App

优质分享App

近一个月的开发和优化,本站点的第一个app全新上线。该app采用极致压缩,本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

用户登录
用户注册
修改密码
忘记密码