首页 文章 精选 留言 我的

精选列表

搜索[hadoop],共8432篇文章
优秀的个人博客,低调大师

Hadoop1.x MapReduce 程序执行的详细过程

1.Job类初始化JobClient实例,JobClient中生成JobTracker的RPC实例,这样可以保持与JobTracker的通讯,JobTracker的地址和端口等都是外部配置的,通过Configuration对象读取并且传入。 2.JobClient提交作业。 3.JobClient生成作业目录。 4.从本地拷贝 MapReduce 的作业jar文件 5.如果DistributedCache中有需要的数据,从DistributedCache中拷贝这部分数据。 6.根据InputFormat实例,实现输入数据的split,在作业目录上生成job.split和job.splitmetainfo文件。 7.将配置文件写入到作业目录的job.xml文件中。 8.JobClient和JobTracker通讯,提交作业。 9.JobTracker将job加入到job队列中。 10.JobTracker的TaskScheduler(任务调度器)对job队列进行调度。 11.TaskTracker通过心跳和JobTracker保持联系,JobTracker收到后根据心跳带来的数据,判断是否可以分配给TaskTracker Task,TaskScheduler会对Task进行分配。 12.TaskTracker启动TaskRunner实例,在TaskRunner中启动单独的JVM进行Mapper运行。 13.Map端会从HDFS中读取输入数据,执行之后Map输出数据先是在内存当中,当达到阀值后,split到硬盘上面,在此过程中如果有combiner的话要进行combiner,当然sort是肯定要进行的。 14.Map结束了,Reduce开始运行,从Map端拷贝数据,称为shuffle阶段,之后执行reduce输出结果数据。 15.当jobtracker收到作业最后一个任务完成通知后,便把作业的状态设置为“完成”。 16.在jobclient查询状态时,便知道任务已经完成,于是从runjob()方法返回。

优秀的个人博客,低调大师

[Hadoop]SSH免密码登录以及失败解决方案

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/SunnyYoona/article/details/51689041 1.创建ssh-key 这里我们采用rsa方式,使用如下命令: xiaosi@xiaosi:~$ ssh-keygen -t rsa -f ~/.ssh/id_rsa Generating public/private rsa key pair. Created directory '/home/xiaosi/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/xiaosi/.ssh/id_rsa. Your public key has been saved in /home/xiaosi/.ssh/id_rsa.pub. The key fingerprint is: SHA256:n/sFaAT94A/xxxxxxxxxxxxxxxxxxxxxxx xiaosi@xiaosi The key's randomart image is: +---[xxxxx]----+ | o= .. .. | | o.= .. .| | *.* o .| | +.4.=E+..| | .SBo=. h+ | | ogo..oo. | | or +j..| | ...+o=.| | ... o=+| +----[xxxxx]-----+ 备注: 这里会提示输入pass phrase,一定不要输入任何字符,回车即可。 2. 生成authorized_keys文件 xiaosi@xiaosi:~$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 备注: 记得要把authorized_keys文件放到.ssh目录下,与rsa等文件放在一起,否则免登录失败,debug如下(ssh -vvv localhost进行调试,查找错误原因): xiaosi@xiaosi:~$ ssh -vvv localhost OpenSSH_7.2p2 Ubuntu-4ubuntu1, OpenSSL 1.0.2g-fips 1 Mar 2016 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: resolving "localhost" port 22 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to localhost [127.0.0.1] port 22. debug1: Connection established. debug1: identity file /home/xiaosi/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /home/xiaosi/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/xiaosi/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/xiaosi/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/xiaosi/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/xiaosi/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/xiaosi/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/xiaosi/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu1 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu1 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu1 pat OpenSSH* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to localhost:22 as 'xiaosi' debug3: hostkeys_foreach: reading file "/home/xiaosi/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/xiaosi/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from localhost debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com,zlib debug2: compression stoc: none,zlib@openssh.com,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,zlib@openssh.com debug2: compression stoc: none,zlib@openssh.com debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: curve25519-sha256@libssh.org debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:378enl3ckhdpObP8fnsHr1EXz4d1q2Jde+jUplkub/Y debug3: hostkeys_foreach: reading file "/home/xiaosi/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/xiaosi/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from localhost debug1: Host 'localhost' is known and matches the ECDSA host key. debug1: Found key in /home/xiaosi/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug2: set_newkeys: mode 0 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS received debug2: key: /home/xiaosi/.ssh/id_rsa (0x5602df5e80c0) debug2: key: /home/xiaosi/.ssh/id_dsa ((nil)) debug2: key: /home/xiaosi/.ssh/id_ecdsa ((nil)) debug2: key: /home/xiaosi/.ssh/id_ed25519 ((nil)) debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/xiaosi/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug1: Trying private key: /home/xiaosi/.ssh/id_dsa debug3: no such identity: /home/xiaosi/.ssh/id_dsa: No such file or directory debug1: Trying private key: /home/xiaosi/.ssh/id_ecdsa debug3: no such identity: /home/xiaosi/.ssh/id_ecdsa: No such file or directory debug1: Trying private key: /home/xiaosi/.ssh/id_ed25519 debug3: no such identity: /home/xiaosi/.ssh/id_ed25519: No such file or directory debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password xiaosi@localhost's password:  3. 验证 xiaosi@xiaosi:~$ ssh localhost The authenticity of host 'localhost (127.0.0.1)' can't be established. ECDSA key fingerprint is SHA256:378enl3ckhdpObP8fnsHr1EXz4d1q2Jde+jUplkub/Y. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. sign_and_send_pubkey: signing failed: agent refused operation xiaosi@localhost's password: 4.authorized_keys权限 我们可以看到还是让我输入密码,很大可能是authorized_keys文件权限的问题,我们给该文件赋予一定权限: xiaosi@xiaosi:~$ chmod 600 ~/.ssh/authorized_keys 再次验证: xiaosi@xiaosi:~$ ssh localhost Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.0-24-generic x86_64) * Documentation: https://help.ubuntu.com/ 0 个可升级软件包。 0 个安全更新。 Last login: Thu Jun 16 08:05:50 2016 from 127.0.0.1 到此表示OK了。 备注: 或者第一次需要输入密码,以后再次登陆就不需要输入密码了。 有更明白的小伙伴可以指导一下。。。。。。

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Spring

Spring

Spring框架(Spring Framework)是由Rod Johnson于2002年提出的开源Java企业级应用框架,旨在通过使用JavaBean替代传统EJB实现方式降低企业级编程开发的复杂性。该框架基于简单性、可测试性和松耦合性设计理念,提供核心容器、应用上下文、数据访问集成等模块,支持整合Hibernate、Struts等第三方框架,其适用范围不仅限于服务器端开发,绝大多数Java应用均可从中受益。

Rocky Linux

Rocky Linux

Rocky Linux(中文名:洛基)是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版,作为CentOS稳定版停止维护后与RHEL(Red Hat Enterprise Linux)完全兼容的开源替代方案,由社区拥有并管理,支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性,采用模块化包装和SELinux安全架构,默认包含GNOME桌面环境及XFS文件系统,支持十年生命周期更新。

用户登录
用户注册