hive3.0.0的一些笔记
Hive只在一个节点上安装即可 1.上传tar包 2.解压 tar -zxvf hive-0.9.0.tar.gz -C /cloud/ 3.安装mysql数据库（切换到root用户）（装在哪里没有限制，只有能联通hadoop集群的节点） mysql安装仅供参考，不同版本mysql有各自的安装流程 rpm -qa | grep mysql rpm -e mysql-libs-5.1.66-2.el6_3.i686 --nodeps rpm -ivh MySQL-server-5.1.73-1.glibc23.i386.rpm rpm -ivh MySQL-client-5.1.73-1.glibc23.i386.rpm 修改mysql的密码 /usr/bin/mysql_secure_installation （注意：删除匿名用户，允许用户远程连接） 登陆mysql mysql -u root -p 4.配置hive （a）配置HIVE_HOME环境变量 vi conf/hive-env.sh 配置其中的$hadoop_home （b）配置元数据库信息 vi hive-site.xml 添加如下内容： <configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>root</value> <description>password to use against metastore database</description> </property> </configuration> 5.安装hive和mysq完成后，将mysql的连接jar包拷贝到$HIVE_HOME/lib目录下 如果出现没有权限的问题，在mysql授权(在安装mysql的机器上执行) mysql -uroot -p #(执行下面的语句 *.*:所有库下的所有表 %：任何IP地址或主机都可以连接) GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'root' WITH GRANT OPTION; FLUSH PRIVILEGES; 6. Jline包版本不一致的问题，需要拷贝hive的lib目录中jline.2.12.jar的jar包替换掉hadoop中的 /home/hadoop/app/hadoop-2.6.4/share/hadoop/yarn/lib/jline-0.9.94.jar 6. 初始化元数据 schematool -dbType mysql -initSchema schematool -dbType mysql -info 启动hive bin/hive ---------------------------------------------------------------------------------------------------- https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name [(col_name data_type [COMMENT col_comment], ...)] [COMMENT table_comment] [PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] [CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS] [ROW FORMAT row_format] [STORED AS file_format] [LOCATION hdfs_path] 6.建表(默认是内部表) create table trade_detail(id bigint, account string, income double, expenses double, time string) row format delimited fields terminated by '\t'; 建分区表 create table td_part(id bigint, account string, income double, expenses double, time string) partitioned by (logdate string) row format delimited fields terminated by '\t'; 建外部表 create external table td_ext(id bigint, account string, income double, expenses double, time string) row format delimited fields terminated by '\t' location '/td_ext'; 7.创建分区表 普通表和分区表区别：有大量数据增加的需要建分区表 create table book (id bigint, name string) partitioned by (pubdate string) row format delimited fields terminated by '\t'; 分区表加载数据 load data local inpath './book.txt' overwrite into table book partition (pubdate='2010-08-22'); load data local inpath '/root/data.am' into table beauty partition (nation="USA"); 8.分桶表 设置： set hive.enforce.buckting=true; set mapred.reduce.tasks=x; order by 会被限制： set hive.mapred.mode=nostrict; create table test_buket(id int ,name string ) clustered by (id) sorted by (id) into 4 buckets row format delimited fields terminated by ','; select nation, avg(size) from beauties group by nation order by avg(size); 添加分区：alter table mytest add partition (country='Japan'); 删除分区：alter table mytest drop partition (country='Japan'); 添加列：alter table mytest add columns (age int); 删除列： 修改列名：alter table mytest change age old_age int; 插入数据：insert overwrite|into table mytest01 select id,name ,null from mytest; 导出数据：insert overwrite|into local directory '/home/exp.bcp' select sum(id) from mytest; 导出数据到hdfs:insert overwrite local directory 'hdfs://data134:9000/output-sort' select sum(id) from mytest; select语法： SELECT [ALL | DISTINCT] select_expr, select_expr, ... FROM table_reference [WHERE where_condition] [GROUP BY col_list [HAVING condition]] [CLUSTER BY col_list | [DISTRIBUTE BY col_list] [SORT BY| ORDER BY col_list] ] [LIMIT number] set mapred.reduce.tasks=xx; set mapred.map.tasks=xx; set -v;#查看所有配置的参数 SET mapreduce.framework.name=local; 本地模式跑map-reduce 本地运行 mapreduce SET mapreduce.framework.name=local; set mapred.local.dir='/home/tmp'; hive 日志目录配置： $HIVE_HOME/conf/hive-log4j.properties hive.log.dir=<other_location> 注：1、order by 会对输入做全局排序，因此只有一个reducer，会导致当输入规模较大时，需要较长的计算时间。 2、sort by不是全局排序，其在数据进入reducer前完成排序。因此，如果用sort by进行排序，并且设置mapred.reduce.tasks>1，则sort by只保证每个reducer的输出有序，不保证全局有序。 3、distribute by根据distribute by指定的内容将数据分到同一个reducer。 4、Cluster by 除了具有Distribute by的功能外，还会对该字段进行排序。因此，常常认为cluster by = distribute by + sort by hiveserver2链接： 运行hiveserver2 ！connect jdbc:hive2://ip:10000 beeline -u jdbc:hive2://ip:10000 -n root 问题: 1. Error: Could not open client transport with JDBC Uri: jdbc:hive2://192.168.249.134:10000: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate mysql 解决方法：在hadoop的core-site.xml中添加如下内容 <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.root.hosts</name> <value>*</value> </property> 经典面试题： data: 访客 月份 访问次数 A 2015-01 5 A 2015-01 15 B 2015-01 5 A 2015-01 8 B 2015-01 25 A 2015-01 5 A 2015-02 4 A 2015-02 6 B 2015-02 10 B 2015-02 5 级联计算累加访问量： sql: select a.username,a.mouth ,a.gt,sum(b.gt)from (select username,mouth,sum(gt) gt from test group by username,mouth) a , (select username,mouth,sum(gt) gt from test group by username,mouth) b where a.username = b.username and a.mouth>=b.mouth group by a.username,a.mouth order by a.username,a.mouth; 结果： A 2015-01 33 33 A 2015-02 10 43 B 2015-01 30 30 B 2015-02 15 45
来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/31506529/viewspace-2213713/ ，如需转载，请注明出处，否则将追究法律责任。
上一篇：
hive 3.0.0自定义函数
下一篇：
azkaban的安装和使用
请登录后发表评论
登录
全部评论
