首页 文章 精选 留言 我的

精选列表

搜索[接口文档],共10013篇文章
优秀的个人博客,低调大师

selenium的文档API

你用WebDriver要做的第一件事就是指定一个链接,一般我们使用get方法: fromseleniumimportwebdriver fromselenium.webdriver.common.keysimportKeys driver=webdriver.Chrome(r'D:\chrome\Google\Chrome\Application\chromedriver.exe') driver.get("https://www.baidu.com/") 你可以用下列任意方法找到它: element=driver.find_element_by_id("passwd-id") element=driver.find_element_by_name("passwd") element=driver.find_element_by_xpath("//input[@id='passwd-id']") content=driver.find_element_by_css_selector('p.content') 等等 fromseleniumimportwebdriver importtime driver=webdriver.Chrome(r'D:\chrome\Google\Chrome\Application\chromedriver.exe') driver.get("https://www.baidu.com/") inputClass=driver.find_element_by_id('kw') time.sleep(3) inputClass.send_keys("python") inputClass.clear() 这里我们举个例子就是请求百度然后输入python然后再删除掉,inputClass.clear()就是清楚搜索框内容假如前面就有内容的话可以先删除掉再输入,这里的kw就是搜索框id 然后我们需要再给我们搜索 fromseleniumimportwebdriver importtime driver=webdriver.Chrome(r'D:\chrome\Google\Chrome\Application\chromedriver.exe') driver.get("https://www.baidu.com/") inputClass=driver.find_element_by_id('kw') inputClass.send_keys("python") time.sleep(3) button=driver.find_element_by_id("su") button.click() 然后我们就可以通过page_source获取搜索后的源代码然后就是beautifulsoup等等包继续获取了 在窗口(window)和框架(frame)间移动现在的网页应用里没有页面框架或者只用一个窗口就包含了所有内容的已经很少了。WebDriver 支持在指定的窗口间移动,方法为switch_to_window: driver.switch_to_window("windowName") 这个switch_to_window用的是target标签现在所有的driver的调用都会指向这个给定的窗口,但是我们怎么知道窗口的名字是什么呢?可以看一看打开这个窗口的javascript脚本或者link链接: Clickheretoopenanewwindow 行为链ActionChains可以完成简单的交互行为,例如鼠标移动,鼠标点击事件,键盘输入,以及内容菜单交互。这对于模拟那些复杂的类似于鼠标悬停和拖拽行为很有用 不管怎样,这些动作总是一个接一个按他们被调用的顺序执行。 click(on_element=None) 点击一个元素 参数: * on_element:要点击的元素,如果是None,点击鼠标当前的位置 click_and_hold(on_element=None) 鼠标左键点击一个元素并且保持 参数: * on_element:同click()类似 double_click(on_element=None) 双击一个元素 参数: * on_element:同click()类似 drag_and_drop(source, target) 鼠标左键点击source元素,然后移动到target元素释放鼠标按键 参数: source:鼠标点击的元素 target:鼠标松开的元素 drag_and_drop_by_offset(source, xoffset,yoffset) 拖拽目标元素到指定的偏移点释放 参数: source:点击的参数 xoffset:X偏移量 * yoffset:Y偏移量 key_down(value,element=None) 只按下键盘,不释放。我们应该只对那些功能键使用(Contril,Alt,Shift) 参数: value:要发送的键,值在Keys类里有定义 element:发送的目标元素,如果是None,value会发到当前聚焦的元素上 例如,我们要按下 ctrl+c: ActionChains(driver).key_down(Keys.CONTROL).send_keys('c').key_up(Keys.CONTROL).perform()key_up(value,element=None) 释放键。参考key_down的解释 move_by_offset(xoffset,yoffset) 将当前鼠标的位置进行移动 参数: xoffset:要移动的X偏移量,可以是正也可以是负 yoffset:要移动的Y偏移量,可以是正也可以是负 move_to_element(to_element) 把鼠标移到一个元素的中间 参数: * to_element:目标元素 move_to_element_with_offset(to_element,xoffset,yoffset) 鼠标移动到元素的指定位置,偏移量以元素的左上角为基准 参数: to_element:目标元素 xoffset:要移动的X偏移量 * yoffset:要移动的Y偏移量 perform() 执行所有存储的动作 release(on_element=None) 释放一个元素上的鼠标按键, 参数: * on_element:如果为None,在当前鼠标位置上释放 send_keys(*keys_to_send) 向当前的焦点元素发送键 参数: * keys_to_send:要发送的键,修饰键可以到Keys类里找到 send_keys_to_element(element,*keys_to_send) 向指定的元素发送键

优秀的个人博客,低调大师

elasticsearch基本安装文档

elasticsearch(ES) *参考链接 *参考配置 安装JAVA_JDK 安装elasticsearch cd /root wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.2.tar.gz tar -zxvf elasticsearch-5.6.2.tar.gz mv /root/elasticsearch-5.6.2 /usr/local/elasticsearch 修改配置文件 vim /usr/local/elasticsearch/config/elasticsearch.yml # node.name node.name: node-1 # node.attr node.attr.rack: r1 # 数据路径 path.data: /usr/local/elasticsearch/data # 日志路径 path.logs: /usr/local/elasticsearch/logs # IP绑定 network.host: [局域网ip] # 设置端口 http.port: 9200 ESC :wq 系统配置 su root # 编辑sysctl.conf vim /etc/sysctl.conf # 设置最大内存占用1G(1024x1024) vm.max_map_count=1048576 ESC :wq # 重载sysctl.conf配置 sysctl -p # 编辑limits.conf vim /etc/security/limits.conf elasticsearch hard nofile 65536 elasticsearch soft nofile 65536 ESC :wq 启动elasticsearch useradd elasticsearch chown -R elasticsearch:elasticsearch /usr/local/elasticsearch su elasticsearch /usr/local/elasticsearch/bin/elasticsearch -d \ -p /usr/local/elasticsearch/logs/elasticsearch.pid 防火墙配置 su root systemctl enable firewalld systemctl start firewalld firewall-cmd --zone=public --add-port=9200/tcp --permanent firewall-cmd --zone=public --add-port=9300/tcp --permanent firewall-cmd --reload 检查是否成功运行 curl http://[服务器局域网]:9200 开机启动 su root vim /etc/rc.local su elasticsearch -c "/usr/local/elasticsearch/bin/elasticsearch -d -p /usr/local/elasticsearch/logs/elasticsearch.pid" ESC :wq CURL管理所有索引 创建一个索引 curl -XPUT 'http://[ip]:[port]/[index_name]?pretty' 查看所有索引 curl 'http://[ip]:[port]/_cat/indices' 删除指定索引 curl -XDELETE 'http://[ip]:[port]/[index_name]?pretty' Console管理所有索引 创建一个索引 PUT /[index_name] 删除多个索引 DELETE /index_* DELETE /index_1,index_2 删除所有索引 DELETE /* DELETE /_all

优秀的个人博客,低调大师

zookeeper基本安装文档

zookeeper 检查是否安装JDK rpm -qa|grep -E '^open[jre|jdk]|j[re|dk]' 卸载已安装JDK rpm -qa|grep Java|xargs rpm -e --nodeps yum安装jdk yum search java|grep jdk yum install java-1.8.0-openjdk 检查安装是否成功 java -version java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode) 下载并安装 cd /root wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz tar -xvf zookeeper-3.4.11.tar.gz mv ./zookeeper-3.4.11 /usr/local/zookeeper mkdir /usr/local/zookeeper/var mkdir /usr/local/zookeeper/var/log echo 1 > /usr/local/zookeeper/var/log/myid cp /usr/local/zookeeper/conf/zoo_sample.cfg /usr/local/zookeeper/conf/zoo.cfg 修改配置 vim /usr/local/zookeeper/conf/zoo.cfg tickTime=2000 initLimit=5 syncLimit=2 dataDir=/usr/local/zookeeper/var/log dataLogDir=/usr/local/zookeeper/var/log clientPort=2181 # 多台server在下面配置即可,如果单台服务器构建多个server,则每个server用过的端口不能重复使用 # 格式: server.[n]=[server_ip]:[server与leader交互端口]:[server选举leader端口] # server.1=172.31.9.59:2182:2183 # server.2=172.31.9.60:2182:2183 # server.3=172.31.9.61:2182:2183 maxClientCnxns=60 minSessionTimeout=60 maxSessionTimeout=120 # purgeInterval含义: 0-禁用自动清除 1-使用自动清除 autopurge.purgeInterval=1 ESC :wq 安装zookeeper-c cd /usr/local/zookeeper/src/c ./configure make make install 防火墙开启 systemctl enable firewalld systemctl start firewalld firewall-cmd --zone=public --permanent --add-port=2181/tcp firewall-cmd --zone=public --permanent --add-port=2182/tcp firewall-cmd --zone=public --permanent --add-port=2183/tcp firewall-cmd --reload 单元文件 # 进入单元文件目录 cd /etc/systemd/system # 创建redis单元文件,格式为: [单元文件名].[单元文件类型] vim zookeeper.service [Unit] Description=开机启动zookeeper. After=default.target network.target [Service] User=root Group=root Type=forking PIDFile=/usr/local/zookeeper/var/log/zookeeper_server.pid ExecStart=/usr/local/zookeeper/bin/zkServer.sh start ExecReload=/usr/local/zookeeper/bin/zkServer.sh restart ExecStop=/usr/local/zookeeper/bin/zkServer.sh stop PrivateTmp=false Restart=always [Install] WantedBy=multi-user.target ESC :wq 安装php扩展:zookeeper参考链接 cd /root wget https://pecl.php.net/get/zookeeper-0.4.0.tgz tar -zxvf zookeeper-0.4.0.tgz cd zookeeper-0.4.0/ phpize ./configure make make install 安装php扩展libzookeeper参考链接 cd /root wget https://github.com/Timandes/libzookeeper/archive/v0.7.2.tar.gz tar -xvf v0.7.2.tar.gz cd libzookeeper-0.7.2 phpize ./configure make make install # 用来调起zookeeper-admin,仓库地址: https://github.com/Timandes/zookeeper-admin.git 修改php.ini vim /usr/local/php/lib/php.ini extension=libzookeeper.so extension=zookeeper.so ESC :wq PHP使用进程公共锁 # 出了$zc的作用域之后,锁将不存在 $zc = new \Zookeeper('127.0.0.1:2181'); //或者 //$zc = new \Zookeeper(); //$zc->connect('127.0.0.1:2181'); $zookeeper_key = '/xxx'; if ($zc->exists($zookeeper_key)) { //如果锁存在,则程序正在运行,不运行新的程序 return false; }else{ //如果锁文件不存在,则创建进程锁文件,运行程序 $acl = [ [ 'perms' => \Zookeeper::PERM_ALL,//共享锁(用来跨进程执行某个程序) 'scheme' => 'world', 'id' => 'anyone', ] ]; //尝试创建锁 $zookeeper_key_res = $zc->create($zookeeper_key, null, $acl, \Zookeeper::EPHEMERAL);//临时锁(可共享的) if ($zookeeper_key_res == $zookeeper_key) { //创建锁成功运行程序 //做些什么,比如等待10秒 sleep(10); $zc->delete($zookeeper_key);//其实不执行也会删除,因为这是一个临时锁,且return之后不再能取到$zc return true; } else { //创建锁失败不运行程序 return false; } }

优秀的个人博客,低调大师

haproxy 安装部署文档

HAProxy提供高可用性、负载均衡以及基于TCP和HTTP应用的代理,支持虚拟主机,它是免费、快速并且可靠的一种解决方案。HAProxy特别适用于那些负载特大的web站点,这些站点通常又需要会话保持或七层处理。HAProxy运行在当前的硬件上,完全可以支持数以万计的并发连接。并且它的运行模式使得它可以很简单安全的整合进您当前的架构中,同时可以保护你的web服务器不被暴露到网络上. haproxy 配置中分成五部分内容,分别如下:1、global:参数是进程级的,通常是和操作系统相关。这些参数一般只设置一次,如果配置无误,就不需要再次进行修改2、defaults:配置默认参数,这些参数可以被用到frontend,backend,Listen组件3、frontend:接收请求的前端虚拟节点,Frontend可以更加规则直接指定具体使用后端的backend4、backend:后端服务集群的配置,是真实服务器,一个Backend对应一个或者多个实体服务器 5、Listen Fronted和backend的组合体 一、安装HAProxy 1.下载最新haproxy安装包,官网:http://www.haproxy.org,如果不能访问,可以使用在线代理访问下载。下载:haproxy-1.5.8.tar.gz 2.上传到linux上,并解压: # mkdir -p /app/zpy/3rd # cd /app/zpy/3rd # tar -zxvf haproxy-1.5.8.tar.gz 创建目录 # mkdir /app/zpy/haproxy 3.安装 # cdhaproxy-1.5.8 # make TARGET=linux26ARCH=x86_64PREFIX=/app/zpy/haproxy#将haproxy安装到/app/zpy/haproxy,TARGET是指定内核版本 make install PREFIX=/app/zpy/haproxy 进入/app/zpy/haproxy目录创建/app/zpy/haproxy/conf目录,复制配置examples cp /app/zpy/3rd/haproxy-1.5.8/examples/haproxy.cfg /app/zpy/haproxy/conf/ 4.修改配置 配置说明如下:(参考:http://freehat.blog.51cto.com/1239536/1347882) ###########全局配置######### global log127.0.0.1local0#[日志输出配置,所有日志都记录在本机,通过local0输出] log127.0.0.1local1notice#定义haproxy日志级别[errorwarringinfodebug] daemon#以后台形式运行harpoxy nbproc1#设置进程数量 pidfile/home/haproxy/haproxy/conf/haproxy.pid#haproxy进程PID文件 ulimit-n819200#ulimit的数量限制 maxconn4096#默认最大连接数,需考虑ulimit-n限制 #chroot/usr/share/haproxy#chroot运行路径 uid99#运行haproxy用户UID gid99#运行haproxy用户组gid #debug#haproxy调试级别,建议只在开启单进程的时候调试 #quiet ########默认配置############ defaults logglobal modehttp#默认的模式mode{tcp|http|health},tcp是4层,http是7层,health只会返回OK optionhttplog#日志类别,采用httplog optiondontlognull#不记录健康检查日志信息 retries2#两次连接失败就认为是服务器不可用,也可以通过后面设置 optionforwardfor#如果后端服务器需要获得客户端真实ip需要配置的参数,可以从HttpHeader中获得客户端ip optionhttpclose#每次请求完毕后主动关闭http通道,haproxy不支持keep-alive,只能模拟这种模式的实现 #optionredispatch#当serverId对应的服务器挂掉后,强制定向到其他健康的服务器,以后将不支持 optionabortonclose#当服务器负载很高的时候,自动结束掉当前队列处理比较久的链接 maxconn4096#默认的最大连接数 timeoutconnect5000ms#连接超时 timeoutclient30000ms#客户端超时 timeoutserver30000ms#服务器超时 #timeoutcheck2000#心跳检测超时 #timeouthttp-keep-alive10s#默认持久连接超时时间 #timeouthttp-request10s#默认http请求超时时间 #timeoutqueue1m#默认队列超时时间 balanceroundrobin#设置默认负载均衡方式,轮询方式 #balancesource#设置默认负载均衡方式,类似于nginx的ip_hash #balnaceleastconn#设置默认负载均衡方式,最小连接数 ########统计页面配置######## listenadmin_stats bind0.0.0.0:1080#设置Frontend和Backend的组合体,监控组的名称,按需要自定义名称 modehttp#http的7层模式 optionhttplog#采用http日志格式 #log127.0.0.1local0err#错误日志记录 maxconn10#默认的最大连接数 statsrefresh30s#统计页面自动刷新时间 statsuri/stats#统计页面url statsrealmXingCloud\Haproxy#统计页面密码框上提示文本 statsauthadmin:admin#设置监控页面的用户和密码:admin,可以设置多个用户名 statsauthFrank:Frank#设置监控页面的用户和密码:Frank statshide-version#隐藏统计页面上HAProxy的版本信息 statsadminifTRUE#设置手工启动/禁用,后端服务器(haproxy-1.4.9以后版本) ########设置haproxy错误页面##### errorfile403/home/haproxy/haproxy/errorfiles/403.http errorfile500/home/haproxy/haproxy/errorfiles/500.http errorfile502/home/haproxy/haproxy/errorfiles/502.http errorfile503/home/haproxy/haproxy/errorfiles/503.http errorfile504/home/haproxy/haproxy/errorfiles/504.http ########frontend前端配置############## bind*:80 #这里建议使用bind*:80的方式,要不然做集群高可用的时候有问题,vip切换到其他机器就不能访问了。 aclwebhdr(host)-iwww.abc.com #acl后面是规则名称,-i是要访问的域名, aclimghdr(host)-iimg.abc.com 如果访问www.abc.com这个域名就分发到下面的webserver的作用域。 #如果访问img.abc.com.cn就分发到imgserver这个作用域。 use_backendwebserverifweb use_backendimgserverifimg ########backend后端配置############## backendwebserver#webserver作用域 modehttp balanceroundrobin #banlanceroundrobin轮询,balancesource保存session值,支持static-rr,leastconn,first,uri等参数 optionhttpchk/index.htmlHTTP/1.0#健康检查 #检测文件,如果分发到后台index.html访问不到就不再分发给它 serverweb110.16.0.9:8085cookie1weight5checkinter2000rise2fall3 serverweb210.16.0.10:8085cookie2weight3checkinter2000rise2fall3 #cookie1表示serverid为1,checkinter1500是检测心跳频率 #rise2是2次正确认为服务器可用,fall3是3次失败认为服务器不可用,weight代表权重 backendimgserver modehttp optionhttpchk/index.php balanceroundrobin serverimg01192.168.137.101:80checkinter2000fall3 serverimg02192.168.137.102:80checkinter2000fall3 ########tcp配置################# listentest1 bind0.0.0.0:90 modetcp optiontcplog#日志类别,采用tcplog maxconn4086 #log127.0.0.1local0debug servers110.18.138.201:80weight1 servers210.18.102.190:80weight1 5.加上日志支持 # vim /etc/syslog.conf 在最下边增加 local3.* /home/haproxy/haproxy/logs/haproxy.log local0.* /home/haproxy/haproxy/logs/haproxy.log # vim /etc/sysconfig/syslog 修改: SYSLOGD_OPTIONS="-r -m 0" 重启日志服务service syslog restart 6.启动服务 启动服务: # /home/haproxy/haproxy/sbin/haproxy -f /home/haproxy/haproxy/conf/haproxy.cfg 重启服务: # /home/haproxy/haproxy/sbin/haproxy -f /home/haproxy/haproxy/conf/haproxy.cfg -st `cat /home/haproxy/haproxy/conf/haproxy.pid` 停止服务: # killall haproxy 7.监控 访问:http://192.168.101.125:1080/stats 配置参考: ###########全局配置######### global log 127.0.0.1 local0 daemon nbproc 4 maxconn 4096 uid 99 gid 99 pidfile /app/zpy/haproxy/conf/haproxy.pid ulimit-n 819200 chroot /var/empty quiet ########默认配置############ defaults log global mode http option httplog option dontlognull retries 3 option forwardfor option httpclose option redispatch option abortonclose timeout connect 5000ms timeout client 30000ms timeout server 30000ms timeout check 2000 balance roundrobin ########统计页面配置######## listen stats bind 0.0.0.0:1080 mode http option httplog maxconn 10 stats refresh 30s stats uri /stats stats realm ZPY Haproxy stats auth admin:admin stats hide-version stats admin if TRUE ########frontend前端配置############## frontend act bind *:8980 acl hadoop_policy hdr_dom(host) -i zipeiyi.hadoop.com # acl impcom-vir_policy hdr_dom(host) -i impcom-vir.zipeiyi.ceshi # acl imp-vir_policy hdr_dom(host) -i imp-vir.zipeiyi.ceshi use_backend hadoop if hadoop_policy # use_backend impcom-vir if impcom-vir_policy # use_backend imp-vir if imp-vir_policy ########backend后端配置############## backend hadoop mode http balance roundrobin server hadoop3 10.0.70.230:8080 check inter 2000 fall 3 server hadoop4 10.0.70.231:8080 check inter 2000 fall 3 #backend impcom-act # mode http # balance roundrobin # server impcom-act01 10.0.150.3:8180 check inter 2000 fall 3 # server impcom-act02 10.0.150.3:8190 check inter 2000 fall 3 在10.0.70.230、10.0.70.231上部署tomcat应用。 在DNS服务器(10.0.10.10)上添加解析 10.0.10.10zipeiyi.hadoop.com 浏览器访问http://zipeiyi.hadoop.com:8980进行验证。 本文转自 周新宇1991 51CTO博客,原文链接:http://blog.51cto.com/zhouxinyu1991/1871886,如需转载请自行联系原作者

优秀的个人博客,低调大师

官方文档 Upgrading Elasticsearch

Upgrading Elasticsearch Before upgrading Elasticsearch: Consult thebreaking changesdocs. Use theElasticsearch Migration Pluginto detect potential issues before upgrading. Test upgrades in a dev environment before upgrading your production cluster. Alwaysback up your databefore upgrading. Youcannot roll backto an earlier version unless you have a backup of your data. If you are using custom plugins, check that a compatible version is available. Elasticsearch can usually be upgraded using a rolling upgrade process, resulting in no interruption of service. This section details how to perform both rolling upgrades and upgrades with full cluster restarts. To determine whether a rolling upgrade is supported for your release, please consult this table: Upgrade From Upgrade To Supported Upgrade Type 1.x 5.x Reindex to upgrade 2.x 2.y Rolling upgrade(wherey > x) 2.x 5.x Full cluster restart 5.0.0 pre GA 5.x Full cluster restart 5.x 5.y Rolling upgrade(wherey > x) Indices created in Elasticsearch 1.x or before Elasticsearch is able to read indices created in theprevious major version only. For instance, Elasticsearch 5.x can use indices created in Elasticsearch 2.x, but not those created in Elasticsearch 1.x or before. This condition also applies to indices backed up withsnapshot and restore. If an index was originally created in 1.x, it cannot be restored into a 5.x cluster even if the snapshot was made by a 2.x cluster. Elasticsearch 5.x nodes will fail to start in the presence of too old indices. SeeReindex to upgradefor more information about how to upgrade old indices. !!!回滚升级就是一次升级一个节点!!!! A rolling upgrade allows the Elasticsearch cluster to be upgraded one node at a time, with no downtime for end users. Running multiple versions of Elasticsearch in the same cluster for any length of time beyond that required for an upgrade is not supported, as shards will not be replicated from the more recent version to the older version. Reindex to upgrade Elasticsearch is able to use indices created in the previous major version only. For instance, Elasticsearch 5.x can use indices created in Elasticsearch 2.x, but not those created in Elasticsearch 1.x or before. Elasticsearch 5.x nodes will fail to start in the presence of too old indices. If you are running an Elasticsearch 2.x cluster which contains indices that were created before 2.x, you will either need to delete those old indices or to reindex them before upgrading to 5.x. SeeReindex in place. If you are running an Elasticsearch 1.x cluster, you have two options: First upgrade to Elasticsearch 2.4.x, reindex the old indices, then upgrade to 5.x. SeeReindex in place. Create a new 5.x cluster and use reindex-from-remote to import indices directly from the 1.x cluster. SeeUpgrading with reindex-from-remote. Reindex in place The easiest way to reindex old (1.x) indices in place is to use theElasticsearch Migration Plugin. You will need to upgrade to Elasticsearch 2.3.x or 2.4.x first. Upgrading with reindex-from-remote If you are running a 1.x cluster and would like to migrate directly to 5.x without first migrating to 2.x, you can do so usingreindex-from-remote. 转自:https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html#setup-upgrade 本文转自张昺华-sky博客园博客,原文链接:http://www.cnblogs.com/bonelee/p/7443844.html,如需转载请自行联系原作者

优秀的个人博客,低调大师

CDH 5.7.0 安装文档

一、实验环境 1. IP和主机名和域名,所有主机都可以连接互联网10.0.70.242 hadoop1 hadoop1.com10.0.70.243 hadoop2 hadoop2.com10.0.70.230 hadoop3 hadoop3.com10.0.70.231 hadoop4 hadoop4.com 2. 操作系统CentOS release 6.5 (Final) 64位 二、配置步骤 1. 安装前准备(都是使用root用户在集群中的所有主机配置)(1)从以下地址下载所需要的安装文件http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.7.0_x86_64.tar.gzhttp://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcelhttp://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1http://archive.cloudera.com/cdh5/parcels/5.7/manifest.json(2)使用下面的命令检查OS依赖包,xxxx换成包名# rpm -qa | grep xxxx以下这些包必须安装:chkconfigpython (2.6 required for CDH 5)bind-utilspsmisclibxsltzlibsqlitecyrus-sasl-plaincyrus-sasl-gssapifuseportmap (rpcbind)fuse-libsredhat-lsb(3)配置域名解析# vi /etc/hosts# 添加如下内容 10.0.70.242hadoop1 10.0.70.243hadoop2 10.0.70.230hadoop3 10.0.70.231hadoop4 或者做好域名解析 (4)安装JDK CDH5推荐的JDK版本是1.7.0_67、1.7.0_75、1.7.0_80,这里安装jdk1.8.0_51 注意: . 所有主机要安装相同版本的JDK . 安装目录为/app/zpy/jdk1.8.0_51/ # mkdir -p/app/zpy # cd /app/zpy/3rd # tar zxvfjdk-8u51-linux-x64.tar.gz -C /app/zpy # chown -R root.root jdk1.8.0_51/ # cat /etc/profile JAVA_HOME=/app/zpy/jdk1.8.0_51 JAVA_BIN=/app/zpy/jdk1.8.0_51/bin PATH=$PATH:$JAVA_BIN CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export AVA_HOME JAVA_BIN PATH CLASSPATH # . /etc/profile (5)NTP时间同步# echo "0 * * * *rootntpdate 10.0.70.2" >> /etc/crontab # /etc/init.d/crond restart(6)建立CM用户# useradd --system --home=/app/zpy/cm-5.7.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm #sed -i "s/Defaults requiretty/#Defaults requiretty/g" /etc/sudoers (7)安装配置MySQL数据库 # yum install -y mysql# 修改root密码mysqladmin -u root password# 编辑配置文件vi /etc/my.cnf# 内容如下 [mysqld] transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # # to do so, uncomment this line: # # symbolic-links = 0 # key_buffer = 16M key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 # max_connections = 550 #expire_logs_days = 10 # #max_binlog_size = 100M # # #log_bin should be on a disk with enough free space. Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your system # #and chown the specified folder to the mysql user. log_bin=/var/lib/mysql/mysql_binary_log # # # For MySQL version 5.1.8 or later. Comment out binlog_format for older versions. binlog_format = mixed # read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M # join_buffer_size = 8M # # # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid # sql_mode=STRICT_ALL_TABLES # 添加开机启动chkconfig mysql on# 启动MySQLservice mysql restart 对于没有innodb的情况 >show databases;查看 删除/var/lib/mysql/下ib*,重启服务即可 # 根据需要建立元数据库 >create database hive; >grant all on hive.* to 'hive'@'%' identified by '1qaz@WSX?'; >create database man; >grant all on man.* to 'man'@'%' identified by '1qaz@WSX?'; >create database oozie; >grant all on oozie.* to 'oozie'@'%' identified by '1qaz@WSX?'; (8)安装MySQL JDBC驱动# cd /app/zpy/3rd # cpmysql-connector-java-5.1.38-bin.jar /app/zpy/cm-5.7.0/share/cmf/lib/ (9)配置免密码ssh(这里配置了任意两台机器都免密码)# # #分别在四台机器上生成密钥对:# cd ~# ssh-keygen -t rsa# # # 然后一路回车# # # 在hadoop1上执行:# cd ~/.ssh/# ssh-copy-id hadoop1# scp /root/.ssh/authorized_keys hadoop2:/root/.ssh/# # # 在hadoop2上执行:# cd ~/.ssh/# ssh-copy-id hadoop2# scp /root/.ssh/authorized_keys hadoop3:/root/.ssh/# # # 在hadoop3上执行:# cd ~/.ssh/# ssh-copy-id hadoop3# scp /root/.ssh/authorized_keys hadoop4:/home/grid/.ssh/# # # 在hadoop4上执行:# cd ~/.ssh/# ssh-copy-id hadoop4# scp /root/.ssh/authorized_keys hadoop1:/root/.ssh/# scp /root/.ssh/authorized_keys hadoop2:/root/.ssh/# scp /root/.ssh/authorized_keys hadoop3:/root/.ssh/ 2. 在hadoop1上安装Cloudera Manager #tar zxvf cloudera-manager-el6-cm5.7.0_x86_64.tar.gz -C /app/zpy/ # # # 建立cm数据库 #/app/zpy/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p1qaz@WSX? --scm-host localhost scm scm scm # # # 配置cm代理 # vim /app/zpy/cm-5.7.0/etc/cloudera-scm-agent/config.ini # # # 将cm主机名改为hadoop1或者改为域名hadoop1.comserver_host=hadoop1 # # # 将Parcel相关的三个文件拷贝到/opt/cloudera/parcel-repo 作为本地源! # cp CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel /opt/cloudera/parcel-repo/# cp CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1 /opt/cloudera/parcel-repo/# cp manifest.json /opt/cloudera/parcel-repo/ # ## 改名# mv /opt/cloudera/parcel-repo/CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha # # # 修改属主# chown -R cloudera-scm:cloudera-scm /opt/cloudera/ # # # 将/app/zpy/cm-5.7.0目录拷贝到其它三个主机# scp -r -p /app/zpy/cm-5.7.0 hadoop2:/app/zpy/# scp -r -p /app/zpy/cm-5.7.0 hadoop3:/app/zpy/# scp -r -p /app/zpy/cm-5.7.0 hadoop4:/app/zpy/ 3. 在每个主机上建立/opt/cloudera/parcels目录,并修改属主# mkdir -p /opt/cloudera/parcels# chown cloudera-scm:cloudera-scm /opt/cloudera/parcels 4. 在hadoop1上启动cm server# /app/zpy/cm-5.7.0/etc/init.d/cloudera-scm-server start# # # 此步骤需要运行一些时间,用下面的命令查看启动情况# tail -f /app/zpy/cm-5.7.0/log/cloudera-scm-server/cloudera-scm-server.log5. 在所有主机上启动cm agent# mkdir /app/zpy/cm-5.7.0/run/cloudera-scm-agent# chown cloudera-scm:cloudera-scm /app/zpy/cm-5.7.0/run/cloudera-scm-agent# /app/zpy/cm-5.7.0/etc/init.d/cloudera-scm-agent 6. 登录cm控制台,安装CDH5打开控制台http://10.0.70.242:7180/页面如图1所示。 图1 缺省的用户名和密码都是admin,登录后进入欢迎页面。勾选许可协议,如图2所示,点继续。 图2 进入版本说明页面,如图3所示,点继续。 图3 进入服务说明页面,如图4所示,点继续。 图4 进入选择主机页面,当前管理的主机。如图5所示,全选四个主机,点继续。 图5 进入选择存储库页面,如图6所示,点继续。 图6 进入集群安装页面,如图7所示,点继续。 图7 进入验证页面,如图8所示,点完成。 图8 进入集群设置页面,如图9所示,根据需要选择服务,这里我们选择自定义,选择需要的服务。后期也可以添加服务,点继续。 图9 进入自定义角色分配页面,如图10所示,保持不变,点继续。 图10 进入数据库设置页面,填写相关信息,点测试连接,如图11所示,点继续。 图11 进入审核更改页面,保持不变,点继续。 进入首次运行页面,等待运行完,如图12所示,点继续。 图11 进入安装成功页面,如图13所示,点完成。 图13 进入安装成功页面,如图14所示。 注意: 1) Error found before invoking supervisord: dictionary update sequence element #78 has length1; 2 is required 这个错误是CM的一个bug,解决方法为修改/app/zpy/cm-5.7.0/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.0-py2.6.egg/cmf/util.py文件。将其中的代码: pipe = subprocess.Popen(['/bin/bash', '-c', ". %s; %s; env" % (path, command)], stdout=subprocess.PIPE, env=caller_env) 修改为: pipe = subprocess.Popen(['/bin/bash', '-c', ". %s; %s; env | grep -v { | grep -v }" % (path, command)], stdout=subprocess.PIPE, env=caller_env) 然后重启所有Agent即可。 2) 安装hive报错数据库创建失败时 #cp /app/zpy/3rd/mysql-connector-java-5.1.38-bin.jar /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hive/lib/ 3) 手动添加应用 4)对于spark找不到java_home的报错解决方法如下: echo"exportJAVA_HOME=/opt/javak1.8.0_51">>/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/metah_env.sh 如图: 本文转自 周新宇1991 51CTO博客,原文链接:http://blog.51cto.com/zhouxinyu1991/1873493,如需转载请自行联系原作者

优秀的个人博客,低调大师

Frida JAVA API 文档

Java Java.available: a boolean specifying whether the current process has the a Java VM loaded, i.e. Dalvik or ART. Do not invoke any other Java properties or methods unless this is the case. Java.enumerateLoadedClasses(callbacks) enumerate classes loaded right now, where callbacks is an object specifying: onMatch: function (className): called for each loaded class with className that may be passed to use() to get a JavaScript wrapper. onComplete: function (): called when all classes have been enumerated. Java.enumerateLoadedClassesSync(): synchronous version of enumerateLoadedClasses() that returns the class names in an array. Java.perform(fn): ensure that the current thread is attached to the VM and call fn. (This isn’t necessary in callbacks from Java.) Java.perform(function () { var Activity = Java.use("android.app.Activity"); Activity.onResume.implementation = function () { send("onResume() got called! Let's call the original implementation"); this.onResume(); }; }); Java.use(className) dynamically get a JavaScript wrapper for className that you can instantiate objects from by calling $new() on it to invoke a constructor. Call $dispose() on an instance to clean it up explicitly (or wait for the JavaScript object to get garbage-collected, or script to get unloaded). Static and non-static methods are available, and you can even replace a method implementation and throw an exception from it: Java.perform(function () { var Activity = Java.use("android.app.Activity"); var Exception = Java.use("java.lang.Exception"); Activity.onResume.implementation = function () { throw Exception.$new("Oh noes!"); }; }); Java.scheduleOnMainThread(fn): run fn on the main thread of the VM. Java.choose(className, callbacks): enumerate live instances of the className class by scanning the Java heap, where callbacks is an object specifying: onMatch: function (instance): called once for each live instance found with a ready-to-use instance just as if you would have called Java.cast() with a raw handle to this particular instance. This function may return the string stop to cancel the enumeration early. onComplete: function (): called when all instances have been enumerated Java.cast(handle, klass): create a JavaScript wrapper given the existing instance at handle of given class klass (as returned from Java.use()). Such a wrapper also has a class property for getting a wrapper for its class, and a $className property for getting a string representation of its class-name. var Activity = Java.use("android.app.Activity"); var activity = Java.cast(ptr("0x1234"), Activity); WeakRef WeakRef.bind(value, fn): monitor value and call the fn callback as soon as value has been garbage-collected, or the script is about to get unloaded. Returns an id that you can pass to WeakRef.unbind() for explicit cleanup. This API is useful if you’re building a language-binding, where you need to free native resources when a JS value is no longer needed. WeakRef.unbind(id): stop monitoring the value passed to WeakRef.bind(value, fn), and call the fn callback immediately.

资源下载

更多资源
腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能,例如代码缩略图,Python的插件,代码段等。还可自定义键绑定,菜单和工具栏。Sublime Text 的主要功能包括:拼写检查,书签,完整的 Python API , Goto 功能,即时项目切换,多选择,多窗口等等。Sublime Text 是一个跨平台的编辑器,同时支持Windows、Linux、Mac OS X等操作系统。

用户登录
用户注册