首页 文章 精选 留言 我的

精选列表

搜索[hbase],共3640篇文章
优秀的个人博客,低调大师

HBase源码分析之HRegionServer上的MovedRegionsCleaner工作线程

MovedRegionsCleaner是什么呢?我们先来看下它在HRegionServer上的定义: /** * Chore to clean periodically the moved region list * 被移动Region列表的定期清理工作线程 */ private MovedRegionsCleaner movedRegionsCleaner; 原来它是HRegionServer上一个被移动Region列表的定期清理工作线程。而它的类的定义如下: /** * Creates a Chore thread to clean the moved region cache. */ protected static class MovedRegionsCleaner extends Chore implements Stoppable { private HRegionServer regionServer; Stoppable stoppable; // 私有构造方法 private MovedRegionsCleaner( HRegionServer regionServer, Stoppable stoppable){ super("MovedRegionsCleaner for region "+regionServer, TIMEOUT_REGION_MOVED, stoppable); this.regionServer = regionServer; this.stoppable = stoppable; } // 静态方法,通过其创建MovedRegionsCleaner实例 static MovedRegionsCleaner createAndStart(HRegionServer rs){ Stoppable stoppable = new Stoppable() { private volatile boolean isStopped = false; @Override public void stop(String why) { isStopped = true;} @Override public boolean isStopped() {return isStopped;} }; return new MovedRegionsCleaner(rs, stoppable); } } 我们发现它继承自Chore类,并且线程工作的频率为TIMEOUT_REGION_MOVED,也就是2分钟。这与之前我们讲过的HRegionServer上检查合并请求的compactionChecker、检查刷新请求的periodicFlusher是一样的,都继承了Chore类。关于Chore类,我们在之前的文章中已经讲过,这里就不再赘述了。 我们看下它的构造方法,只有一个private的私有构造方法,同时它又提供了一个静态方法createAndStart(),这个方法被HRegionServer调用以构造MovedRegionsCleaner对象。而在HRegionServer中,MovedRegionsCleaner是如此被初始化的,代码如下: // Create the thread to clean the moved regions list // 创建movedRegionsCleaner工作线程去清理被移动Region列表 movedRegionsCleaner = MovedRegionsCleaner.createAndStart(this); 那么,MovedRegionsCleaner线程是如何工作的呢?按照之前讲的,集成自Chore的线程会周期性的调用chore()方法来执行需要做的工作。我们还是看下它的chore()方法,代码如下: // chore()方法就是调用HRegionServer的cleanMovedRegions()方法 @Override protected void chore() { regionServer.cleanMovedRegions(); } 很简单,它实际调用的是regionServer实例的cleanMovedRegions()方法,代码如下: /** * Remove the expired entries from the moved regions list. */ protected void cleanMovedRegions() { // 计算超时时间,当前时间减去固定值2分钟 final long cutOff = System.currentTimeMillis() - TIMEOUT_REGION_MOVED; // 获取movedRegions集合的迭代器it Iterator<Entry<String, MovedRegionInfo>> it = movedRegions.entrySet().iterator(); // 利用迭代器it遍历movedRegions集合中的元素 while (it.hasNext()){ // 取出movedRegions集合中的元素e,它是key-value类型,key为String类型的Region名称,value为MovedRegionInfo Map.Entry<String, MovedRegionInfo> e = it.next(); // 根据MovedRegionInfo的MoveTime,即ts,与当前时间比较, // 如果当前时间已超过TIMEOUT_REGION_MOVED,则移除, // ts实际上是MovedRegionInfo的创建时间,也就是Region被移动的时间 if (e.getValue().getMoveTime() < cutOff) { it.remove(); } } } 逻辑比较清晰,大体如下: 1、首先计算超时时间,当前时间减去固定值2分钟,赋值给cutOff; 2、获取movedRegions集合的迭代器it,实际上movedRegions就是HRegionServer上存储已被移动Regions的集合; 3、利用迭代器it遍历movedRegions集合中的元素: 3.1、取出movedRegions集合中的元素e,它是key-value类型,key为String类型的Region名称,value为MovedRegionInfo; 3.2、根据MovedRegionInfo的MoveTime,即ts,与当前时间比较,如果当前时间已超过TIMEOUT_REGION_MOVED,则移除,ts实际上是MovedRegionInfo的创建时间,也就是Region被移动的时间。 实际上,整个处理流程很简单,而MovedRegionInfo的代码如下: private static class MovedRegionInfo { private final ServerName serverName; private final long seqNum; private final long ts; public MovedRegionInfo(ServerName serverName, long closeSeqNum) { this.serverName = serverName; this.seqNum = closeSeqNum; ts = EnvironmentEdgeManager.currentTime(); } public ServerName getServerName() { return serverName; } public long getSeqNum() { return seqNum; } public long getMoveTime() { return ts; } } 其中,就包括一个重要的变量long类型的ts,它在构造方法中被赋值为当前时间,而被移动Region加入movedRegions时,是通过HRegionServer的addToMovedRegions()方法实现的,具体代码如下: protected void addToMovedRegions(String encodedName, ServerName destination, long closeSeqNum) { if (ServerName.isSameHostnameAndPort(destination, this.getServerName())) { LOG.warn("Not adding moved region record: " + encodedName + " to self."); return; } LOG.info("Adding moved region record: " + encodedName + " to " + destination + " as of " + closeSeqNum); movedRegions.put(encodedName, new MovedRegionInfo(destination, closeSeqNum)); } 我们发现,movedRegions集合中,key就是Region的encodedName,而value则是构造的一个对应包含目的地ServerName和关闭序列号closeSeqNum的MovedRegionInfo实例,而MovedRegionInfo在构造时,其ts的赋值上面已经展示了,就是当前时间啊。 综上所述,MovedRegionsCleaner是HRegionServer上一个工作线程,它周期性的清理已被移动Region列表movedRegions中的到达时间的MovedRegionInfo信息,而线程工作的频率和MovedRegionInfo存活时间,均是TIMEOUT_REGION_MOVED,也就是2分钟。 至于movedRegions集合中的数据是何时被添加的,为什么又要设计这种模式来移除其中的数据,我们后续再讲!

优秀的个人博客,低调大师

sql server 导出数据到 Azure Hbase / Hive 详细步骤

The Hadoop on Azure Sqoop Import Sample Tutorial Table of Contents Overview Goals Key technologies Setup and Configuration Tutorial How to set up a SQL database How to use Sqoop from Hadoop on Azure to import SQL Database query results to the HDFS cluster in Hadoop On Azure. Summary Overview This tutorial shows how to use Sqoop to import data from a SQL database on Windows Azure to an Hadoop on Azure HDFS cluster. While Hadoop is a natural choice for processing unstructured and semi-structured data, such as logs and files, there may also be a need to process structured data stored in relational databases. Sqoop is a tool designed to transfer data between Hadoop and relational databases. You can use it to import data from a relational database management system (RDBMS) such as SQL or MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop with MapReduce or Hive, and then export the data back into an RDBMS. In this tutorial, you are using a SQL Database for your relational database. Sqoop is an open source software product of Cloudera, Inc.Software development for Sqoop has recently moved from gitHub to theApache Sqoop site. In Hadoop on Azure, Sqoop is deployed from the Hadoop Command Shell on the head node of the HDFS cluster. You use the Remote Desktop feature available in the Hadoop on Azure portal to access the head node of the cluster for this deployment. Goals In this tutorial, you see three things: How to set up a SQL database on Windows Azure for use with the tutorial. How to use the Remote Desktop feature in Hadoop on Azure to access the head node of the HDFS cluster. How to import relational data from SQL Server to a Hadoop on Azure HDFS cluster by using Sqoop. Key technologies Windows Azure Hadoop on Azure Sqoop Setup and Configuration You must have an account to access Hadoop on Azure and have created a cluster to work through this tutorial. To obtain an account and create an Hadoop cluster, follow the instructions outlined in the Getting started with Microsoft Hadoop on Azure section of the Introduction to Hadoop on Azure topic. You will also need your outward facing IP address for your current location when configuring your firewall on SQL Database. To obtain it, go to the site WhatIsMyIP and make a note of it. Later in the procedure, you also need the outward facing IP address for the head of the Hadoop cluster. You can obtain this IP address in the same way. Tutorial This tutorial is composed of the following segments: How to set up a SQL database. How to use Sqoop from Hadoop on Azure to import data to the HDFS cluster. How to set up a SQL database Log in into your Windows Azure account. To create a database server, click the Database icon in the lower left-hand corner on the page. On the Getting Started page, click the Create a new SQL Database Server option. Select the type of subscription (such as Pay-As-You-Go) associated with you account in the Create Server window and press Next. Select the appropriate Region in the Create Server window and click Next. Specify the login and password of the server-level principal of your SQL Database server and then press Next. Press Add to specify a firewall rule that allows your current location access to SQL Database to upload the AdventureWorks database. The firewall grants access based on the originating IP address of each request. Use the IP address found with the configuration preliminaries of this tutorial for the values to add. Specify a Rule name, such as shown, but remember to use your IP address, not the one used for illustration purposes below. (You must also add the outward IP address of the head node in you Hadoop cluster. If you know it already, add it now.) Then press the Finish button. Download the AdventureWorks2012 database onto your local machine from Recommended Downloads link on the Adventure Works for SQL Database site. Unzip the file, open an Administrator Command Prompt, and navigate to the AdventureWorks directory inside the AdventureWorks2012ForSQLAzure folder. Run CreateAdventureWorksForSQLAzure.cmd by typing the following: CreateAdventureWorksForSQLAzure.cmd servername username password For example, if the assigned SQL Database server is named b1gl33p, the administrator user name "Fred", and the password "Secret", you would type the following: CreateAdventureWorksForSQLAzure.cmd b1gl33p.database.windows.net Fred@b1gl33p Secret The script creates the database, installs the schema, and populates the database with sample data. Return to the WindowsAzurePlatform portal page, click your subscription on the left-hand side (Pay-As-You-Go in the example below) and select your database (here named wq6xlbyoq0). The AventureWorks2012 should be listed in the Database Name column. Select it and press the Manage icon at the top of the page. Enter the credentials for the SQL database when prompted and press Log on. This opens the Web interface for the Adventure Works database on SQL Database. Press the New Query icon at the top to open the query editor. Since Sqoop currently adds square brackets to the table name, we need to add a synonym to support two-part naming for SQL Server tables. To do so, run the following query: CREATE SYNONYM [Sales.SalesOrderDetail] FOR Sales.SalesOrderDetail Run the following query and review its result. select top 200 * from [Sales.SalesOrderDetail] How to use Sqoop from Hadoop on Azure to import SQL Database query results to the HDFS cluster in Hadoop On Azure. From your Account page, scroll down to the Open Ports icon in the Your cluster section and click the icon to open the ODBC Server port on the head node in your cluster. Return to your Account page, scroll down to the Your cluster section and click the Remote Desktop icon this time to open the head node in your cluster. Select Open when prompted to open the .rdp file. Select Connect in the Remote Desktop Connection window. Enter your credentials for the Hadoop cluster (not your Hadoop on Azure account) into the Windows Security window and select OK. Open Internet Explorer and go to the site WhatIsMyIP to obtain the outward facing IP address for the head node of the cluster. Return the SQL Database management page and add a firewall rule that allows your Hadoop cluster access to SQL Database. The firewall grants access based on the originating IP address of each request. Double-click on the Hadoop Command Shell icon in the upper left hand of the Desktop to open it. Navigate to the "c:\Apps\dist\sqoop\bin" directory and run the following command: sqoop import --connect "jdbc:sqlserver://[serverName].database.windows.net;username=[userName]@[serverName];password=[password];database=AdventureWorks2012" --table Sales.SalesOrderDetail --target-dir /data/lineitemData -m 1 So, for example, for the following values: * server name: wq6xlbyoq0 * username: HadoopOnAzureSqoopAdmin * password: Pa$$w0rd The sqoop command is: sqoop import --connect "jdbc:sqlserver://wq6xlbyoq0.database.windows.net;username=HadoopOnAzureSqoopAdmin@wq6xlbyoq0;password=Pa$$w0rd;;database=AdventureWorks2012" --table Sales.SalesOrderDetail --target-dir /data/lineitemData -m 1 Return to the Accounts page of the Hadoop on Azure portal and open the Interactive Console this time. Run the #lsr command from the JavaScript console to list the files and directories on your HDFS cluster. Run the #tail command to view selected results from the part-m-0000 file. tail /user/RAdmin/data/SalesOrderDetail/part-m-00000 Summary In this tutorial, you have seen how to use Sqoop to import data from a SQL database on Windows Azure to an Hadoop on Azure HDFS cluster.

资源下载

更多资源
优质分享App

优质分享App

近一个月的开发和优化,本站点的第一个app全新上线。该app采用极致压缩,本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

Spring

Spring

Spring框架(Spring Framework)是由Rod Johnson于2002年提出的开源Java企业级应用框架,旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念,提供核心容器、应用上下文、数据访问集成等模块,支持整合Hibernate、Struts等第三方框架,其适用范围不仅限于服务器端开发,绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

用户登录
用户注册