首页 文章 精选 留言 我的

精选列表

搜索[渲染问题],共10000篇文章
优秀的个人博客,低调大师

impala1.2.3 udf问题

新的impala已经支持udf了,在测试环境部署了1.2.3版本的cluster. 在运行测试udf时遇到下面这个错误: java.lang.IllegalArgumentException (表明向方法传递了一个不合法或不正确的参数。) 经过确认这是一个bug: https://issues.cloudera.org/browse/IMPALA-791 The currently impala 1.2.3 doesn't support String as the input and return types. You'll instead have to use Text or BytesWritable. 1.2.3版本的impala udf的输入参数和返回值还不支持String,可以使用import org.apache.hadoop.io.Text类代替String Text的api文档: http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/Text.html 重要的几点: Constructor: Text(String string) Construct from a string. Method: String toString() Convert text back to string void set(String string) Set to contain the contents of a string. void set(Text other) copy a text. void clear() clear the string to empty 在eclipse中测试Text类的用法: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 package com.hive.myudf; import java.util.Arrays; import java.util.regex.Pattern; import java.util.regex.Matcher; import org.apache.hadoop.io.Text; public class TextTest { private static Text schemal = new Text( "http://" ); private static Text t = new Text( "GET /vips-mobile/router.do?api_key=04e0dd9c76902b1bfc5c7b3bb4b1db92&app_version=1.8.7 HTTP/1.0" ); private static Pattern p = null ; private static Matcher m = null ; public static void main(String[] args) { p = Pattern. compile( "(.+?) +(.+?) (.+)" ); Matcher m = p.matcher( t.toString()); if (m.matches()){ String tt = schemal + "test.test.com" +m.group( 2 ); System. out .println(tt); //return m.group(2); } else { System. out .println( "not match" ); //return null; } schemal .clear(); t.clear(); } } 测试udf: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 package com.hive.myudf; import java.net.URL; import java.util.regex.Matcher; import java.util.regex.Pattern; import org.apache.hadoop.hive.ql.exec.UDF; import org.apache.hadoop.io.Text; import org.apache.log4j.Logger; public class UDFNginxParseUrl extends UDF { private static final Logger LOG = Logger.getLogger(UDFNginxParseUrl. class ); private Text schemal = new Text( "http://" ); private Pattern p1 = null ; private URL url = null ; private Pattern p = null ; private Text lastKey = null ; private String rt; public UDFNginxParseUrl() { } public Text evaluate(Text host1, Text urlStr, Text partToExtract) { LOG.debug( "3args|args1:" + host1 + ",args2:" + urlStr + ",args3:" + partToExtract); System. out.println( "3 args" ); System. out.println( "args1:" + host1 + ",args2:" + urlStr + ",args3:" + partToExtract); if (host1 == null || urlStr == null || partToExtract == null ) { //return null; return new Text( "a" ); } p1 = Pattern.compile( "(.+?) +(.+?) (.+)" ); Matcher m1 = p1.matcher(urlStr.toString()); if (m1.matches()){ LOG.debug( "into match" ); String realUrl = schemal.toString() + host1.toString() + m1.group( 2 ); Text realUrl1 = new Text(realUrl); System. out.println( "URL is " + realUrl1); LOG.debug( "realurl:" + realUrl1.toString()); try { LOG.debug( "into try" ); url = new URL(realUrl1.toString()); } catch (Exception e){ //return null; LOG.debug( "into exception" ); return new Text( "b" ); } } if (partToExtract.equals( "HOST" )) { rt = url.getHost(); LOG.debug( "get host" + rt ); } //return new Text(rt); LOG.debug( "get what" ); return new Text( "rt" ); } } 几个注意的地方: 1.function是和db相关联的。 2.jar文件存放在hdfs中 3.function会被catalog缓存 本文转自菜菜光 51CTO博客,原文链接:http://blog.51cto.com/caiguangguang/1359312,如需转载请自行联系原作者

优秀的个人博客,低调大师

openstack 遇到的问题一

1 keystone-manage db_sync 这个已经是最全面的了。 Solution MySQL ERROR 1045 Access denied for 'user'@'localhost' - breaks OpenStack #mysql -u root -p . . . Server version: 5.5.24-0ubuntu0.12.04.1 (Ubuntu) . . . mysql>SELECT user,host,password FROM mysql.user; +------------------+------------+-------------------------+ | user | host | password | +------------------+------------+-------------------------+ | root | localhost | *77B48D6366D102139D3719 | | root | mysqltests | *77B48D6366D102139D3719 | | root | 127.0.0.1 | *77B48D6366D102139D3719 | | root | ::1 | *77B48D6366D102139D3719 | | | localhost | | | | mysqltests | | | debian-sys-maint | localhost | *04D30B480932109EFD77E1 | +------------------+------------+-------------------------+ 7 rows in set (0.00 sec) mysql>show grants; +---------------------------------------------------------+ | Grants forroot@localhost | +---------------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' | | IDENTIFIED BY PASSWORD '*77B48D6366D102139D3719' | | WITH GRANT OPTION | | GRANT PROXY ON ''@'' TO 'root'@'localhost' WITH GRANT | | OPTION | +---------------------------------------------------------+ 2 rows in set (0.00 sec) Themysql.userTable At first glance, we have 2 users (root and debian-sys-maint). That's wrong, because mysql's "user" is a 'user'@'host' pair association. So we have 7 in total: 'root' is defined (with the same password) for any combination of 'localhost' (the first 4 lines), then we have 2 strange lines withemptyusername, and finally the debian backdoor 'debian-sys-maint'. The grants The 'show grants' above shows only grants for 'root'. But if we run the next staement, we see what access is provided to any user connecting from 'localhost': mysql>show grants for ''@'localhost'; +--------------------------------------+ | Grants for @localhost | +--------------------------------------+ | GRANT USAGE ON *.* TO ''@'localhost' | +--------------------------------------+ Which (indirectly) explains why running this command (as Linux user 'ori') doesn't require a password: [16:16:57]ori@mysqltests[~] $mysqladmin ping mysqld is alive Where this one fails: [16:14:59]ori@mysqltests[~] $mysqladmin -uroot ping mysqladmin: connect to server at 'localhost' failed error: 'Access denied for user 'root'@'localhost' (using password: NO)' Honestly, in the beginning i thought there's some balck magic here related to the user ('ori', in this case) defined during ubuntu installation, or a special Linux group memebership, or some apparmor profile or god-knows what else. But there's no black magic after all, and it's all inside mysql: The first thing to bear in mind is that the empty USER field '' is a wildcard, same as '%' for host. The second is that mysql prefers the explicit match over the wildcard. For example, user 'root' can match either [1] the explicit 'root'@localhost' row or [2] the wildcard ''@'localhost' row. Since there's an explicit entry for [1] in the table mysql.user, it'll be used. This in turn requires a password so when i try to connect as 'root' without a password i'm rejected. When i connect as 'ori' - whichisn'teven a mysql user, there's only one possible match - ''@'localhost' and this line in the table doesn't have a password. This nicely explains why the above mysqladmin command works for 'ori' and fails for 'root'. To sum it up: mysql controls access (or connection request) based on the USER table. Which user, from which host and whether a password is required. Once connected, the GRANTS determine what the user is allowed to do. When connected as 'ori' i'm limited to "USAGE" (e.g. check if server is up, what version and the like of inoffensive commands). So far so good - but why 'glance'@'localhost' is denied access on the OpenStack controller? When the static IP address of the conrollerwasn'tin/etc/hosts(or after it was commented-out), there was only one match for 'glance' = 'glance'@'%' This, in turn, comes from the connection string (in/etc/glance/glance-registry.conf) which is: sql_connection = mysql://glance:openstack@10.0.0.40/glance It specifies user, password and host. The line I've added for 10.0.0.40 in /etc/hosts, told mysql (indirectly) that host 'ostk-controller1' is actually 'localhsot'. From now on, there are 2 possible matches for 'glance', and the one picked by mysql is ''@'localhost'. This row, however, doesn't require a password - which the sql_connection string provide. And that's why all OpenStack services couldn't connect to mysql. Check against the USER table below, this was taken from ostk-controller (not the test VM): mysql>SELECT user,host,password FROM mysql.user; +------------------+------------------+-------------------------+ | user| host | password| +------------------+------------------+-------------------------+ | root| localhost | *3A4A03AC22526F6B591010 | | root| ostk-controller1 | *3A4A03AC22526F6B591010 | | root| 127.0.0.1 | *3A4A03AC22526F6B591010 | | root | ::1 | *3A4A03AC22526F6B591010 | | | localhost || | | ostk-controller1 | | | debian-sys-maint | localhost | *F714636CE8A7836873F7C8 | | nova | % | *3A4A03AC22526F6B591010 | | glance | % | *3A4A03AC22526F6B591010 | | keystone | % | *3A4A03AC22526F6B591010 | +------------------+------------------+-------------------------+ 10 rows in set (0.00 sec) Solution for ERROR 1045 After understanding why, let's improve on the poor workaround. I'd like to creditan answer by Paul DuBois from 2004for this solution(it's worth noting that the subject was "Re: Any way to make anyhost '%' include localhost"). Borrowing from there, here's the remedy: in MySQL: mysql -uroot -p DELETE FROM mysql.user WHERE Host='localhost' AND User=''; DELETE FROM mysql.user WHERE Host='ostk-controller1' AND User=''; FLUSH PRIVILEGES; in /etc/hosts: Replacethe line 127.0.1.1ostk-controller1 by this one: 10.0.0.40 ostk-controller1 Quoting fromDebian's reference manual: For a system with a permanent IP address, that permanent IP address should be used here instead of 127.0.1.1 finally restart networking and mysqld - or simply reboot. A Second Solution Months after going through the above study, i found out why some OpenStack installationsdon'thit this issue; The keystone installation instructions (from Ubuntu, for Essex,can be found here) create each OSTK user in mysqltwice, as in: mysql> CREATE DATABASE keystone;CREATE USER ‘keystone’@’localhost’IDENTIFIED BY ‘Secret_pass’; GRANT ALL PRIVILEGES ON keystone.* TO 'keystone’@’localhost’ WITH GRANT OPTION;CREATE USER ‘keystone’@’%’IDENTIFIED BY ‘Secret_pass’; GRANT ALL PRIVILEGES ON keystone.* TO ‘keystone’@’%’ IDENTIFIED BY ‘Secret_pass’; FLUSH PRIVILEGES; 本文转自luojinghappy 51CTO博客,原文链接:http://blog.51cto.com/luojinghappy/1342663,如需转载请自行联系原作者

资源下载

更多资源
Mario

Mario

马里奥是站在游戏界顶峰的超人气多面角色。马里奥靠吃蘑菇成长,特征是大鼻子、头戴帽子、身穿背带裤,还留着胡子。与他的双胞胎兄弟路易基一起,长年担任任天堂的招牌角色。

腾讯云软件源

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题,腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构,目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称,一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集,帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Sublime Text

Sublime Text

Sublime Text具有漂亮的用户界面和强大的功能,例如代码缩略图,Python的插件,代码段等。还可自定义键绑定,菜单和工具栏。Sublime Text 的主要功能包括:拼写检查,书签,完整的 Python API , Goto 功能,即时项目切换,多选择,多窗口等等。Sublime Text 是一个跨平台的编辑器,同时支持Windows、Linux、Mac OS X等操作系统。

用户登录
用户注册