sqoop同步mysql到hdfs
链接：http://pan.baidu.com/s/1gfHnaVL 密码：7j12 mysql-connector version 5.1.32 若在安装版本过程遇到些问题，可参考http://dbspace.blog.51cto.com/6873717/1875955，其中一些问题的解决办法 下载并安装： cd/usr/local/ tar-zxvfsqoop2-1.99.3-cdh5.0.0.tar.gz mvsqoop2-1.99.3-cdh5.0.0sqoop 添加sqoop2到系统环境变量中：exportSQOOP_HOME=/usr/local/sqoop exportCATALINA_BASE=$SQOOP_HOME/server exportPATH=$PATH:/usr/local/sqoop/bin 拷贝mysql驱动包到$SQOOP2_HOME/server/lib下 cpmysql-connector-java-5.1.32-bin.jar/usr/local/sqloop/server/lib/ 修改配置文件：vim/usr/local/sqoop/server/conf/sqoop.properties org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/etc/hadoop#hadoop的配置文件路径 vim/usr/local/sqoop/server/conf/catalina.properties 把原来58行注释了，这里主要配置了hadoop的jar包的路径信息 common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*.jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/*.jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/share/hadoop/yarn/lib/*.jar 启动\停止sqoop/usr/local/sqoop/sqoop2-serverstart/stop 验证是否启动成功： 方式一：jps查看进程:Bootstrap [root@mycatsqoop]#jps25505SqoopShell 13080SecondaryNameNode 12878NameNode 26568Jps 方式二：方式二：http://192.168.1.114:12000/sqoop/version#SQOOP默认使用的端口为12000在/usr/local/sqoop/server/conf/server.xml中进行设置 ####接下来测试mysql到hadoop存储的一个过程 1、客户端登陆 [root@mycat bin]# sqoop2-shell Sqoop home directory: /usr/local/sqoop Sqoop Shell: Type 'help' or '\h' for help. sqoop:000> 2、创建一个mysql链接，在这个版本create 就只有[connection|job],注意不同版本的添加链接方式是不同的. 查看支持的链接服务 sqoop:000> show connector +----+------------------------+-----------------+------------------------------------------------------+ | Id | Name | Version | Class | +----+------------------------+-----------------+------------------------------------------------------+ | 1 | generic-jdbc-connector | 1.99.3-cdh5.0.0 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | +----+------------------------+-----------------+------------------------------------------------------+##在1.99.7的版本显示的方式和服务更多。 sqoop:000> create connection --cid 1 Creating connection for connector with id 1 Please fill following values to create new connection object Name: mysql_to_hadoop Connection configuration JDBC Driver Class: com.mysql.jdbc.Driver JDBC Connection String: jdbc:mysql://192.168.1.107:3306/sqoop #这里需要在1.107先添加好库sqoop Username: sqoop##需要在数据库添加好链接的用户 Password: ******* JDBC Connection Properties: There are currently 0 values in the map: entry# Security related configuration options Max connections: New connection was successfully created with validation status ACCEPTABLE and persistent id 2 2、创建job sqoop:000> create job --xid 2 --type import##注意 --xid 2为链接的id号 Creating job for connection with id 2 Please fill following values to create new job object Name: mysql_to_hadoop Database configuration Schema name: sqoop#MySQL的库名 Table name: wangyuan#库下的表 Table SQL statement: Table column names: Partition column name: Nulls in partition column: Boundary query: Output configuration Storage type: 0 : HDFS Choose: 0 Output format: 0 : TEXT_FILE 1 : SEQUENCE_FILE Choose: 0 Compression format: 0 : NONE 1 : DEFAULT 2 : DEFLATE 3 : GZIP 4 : BZIP2 5 : LZO 6 : LZ4 7 : SNAPPY Choose: 0 Output directory: hdfs://192.168.1.114:9000/home/mysql_to_hdfs2 #注意这个mysql_to_hdfs不能再hadoop的/home/已经存在的，但/home路径要存在,9000端口是在配置hadoop的时候配置，根据实际，或者通过WEB查看http:ip:50070----显示Overview 'mycat:9000' (active) 创建hdfs路径/usr/local/hadoop/bin/hadoop fs -mkidr /home 查看创建目录：/usr/local/hadoop/bin/hadoop fs -ls /home 或者通过WEB查看http:ip:50070 Throttling resources Extractors: Loaders: New job was successfully created with validation status FINE and persistent id 2 sqoop:000> 启动job sqoop:000> start job --jid 2 Exception has occurred during processing command Exception: org.apache.sqoop.common.SqoopException Message: CLIENT_0001:Server has returned exception 根本不知道这个提示说什么，通过修改设置：set option --name verbose --value true sqoop:000> start job --jid 2 Submission details Job ID: 2 Server URL: http://localhost:12000/sqoop/ Created by: root Creation date: 2016-11-23 21:15:27 CST Lastly updated by: root External ID: job_1479653943050_0007 http://haproxy:8088/proxy/application_1479653943050_0007/ Connector schema: Schema{name=wangyuan,columns=[ FixedPoint{name=id,nullable=null,byteSize=null,unsigned=null}, Date{name=c_time,nullable=null,fraction=null,timezone=null}]} 2016-11-23 21:15:27 CST: BOOTING - Progress is not available 返回这样信息OK 查看结果通过WEB /usr/local/hadoop/bin/hadoop fs -ls /home/ 版权声明：原创作品，如需转载，请注明出处。否则将追究法律责任 本文转自 DBAspace 51CTO博客，原文链接:http://blog.51cto.com/dbaspace/1875971
