大数据之HIVE入门（三）

　　安装hadoop
　　用root账号建立一个软件安装目录。mkdir -p /app chown cloud.cloud /app
　　将已下载好的hadoop安装包等通过putty套件中的psftp上传到已安装好的虚拟机 /app 目录。
　　题外话:psftp传输请用 open 192.168.10、 输入root、 输入 密码、lcd d:xx 改变本地目录、cd /app 改变远程目录。put 文件名上传、quit退出。
　　方便的话可以下载securefx、xshell、secureCRT等工具完成。当然也可以直接用wget在linux中下载.cd /app tar -zxvf hadoop-2.10.1.tar.gz ln -s hadoop-2.10.1.tar.gz hadoopecho ＂master＂ >/app/hadoop/etc/hadoop/slaves #设定有集群服务器别名列表，由于是伪分布只填写本机器名称。配置core-site.xmlvi /app/hadoop/etc/hadoop/core-site.xml  fs.defaultFS hdfs://master:8020         fs.default.name       hdfs://master:9000      hadoop.tmp.dir /tmp   配置hdfs-site.xml  dfs.namenode.name.dir /app/hadoop/hadoop/dfs/name    dfs.datanode.data.dir /app/hadoop/hadoop/dfs/data   dfs.replication 1  配置yarn-site.xml                                mapreduce.framework.name                   yarn                 yarn.resourcemanager.scheduler.class     org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler    配置队列名称 capacity-scheduler.xml 默认即可。配置hadoop环境变量
　　vi ~/.bashrcexport JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH HADOOP_HOME=/app/hadoop export HADOOP_HOME PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH export PATH
　　运行. ~/.bashrc  #使参数生效。格式化namenodehadoop namenode -format
　　注意：这个命令只需运行一次，每次运行它会生成一个新clusterID，从而造成集群ID和版本号不一致，从而不能正常启动。需要手动删除hadoop/dfs目录再重新格式化即可，当然也可手动修改id。启动集群start-dfs.sh start-yarn.sh --------------启动完成可用jps看到如下进程hadoop安装完成---------------------- jps 8769 SecondaryNameNode 9096 NodeManager 8444 NameNode 8988 ResourceManager 8589 DataNode 9150 JpsHive安装解压安装文件tar -zxvf apache-hive-2.3.9-bin.tar.gz ln -s apache-hive-2.3.9-bin hive配置环境变量 ~/.bashrcexport HIVE_HOME=/app/hive export PATH=$PATH:$HIVE_HOME/bin:$HIVE_HOME/conf export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop  export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_HOME}/lib/native  export HADOOP_OPTS=＂-Djava.library.path=${HADOOP_HOME}/lib＂  export HIVE_CONF_DIR=${HIVE_HOME}/conf  export CLASS_PATH=.:${JAVA_HOME}/lib:${HIVE_HOME}/lib:$CLASS_PATH export PATH=.:${HIVE_HOME}/bin:$PATH export JAVA_LIBRARY_PATH=${HADOOP_HOME}/lib/native复制hive meta的mysql驱动
　　用cloud账号复制mysql驱动mysql-connector-java.jar到/app/hive/lib下cp /app/mysql-connector-java.jar /app/hive/lib/
　　mysql安装
　　mysql用于存储hive元数据，该操作需要切换到root账号。准备好mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar文件，笔者已经在网盘中准备好了该文件，请下载后上传到虚拟机中/app目录。tar xvf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm mysql-community-client-5.7.28-1.el7.x86_64.rpm mysql-community-devel-5.7.28-1.el7.x86_64.rpm mysql-community-libs-* mysql-community-common-5.7.28-1.el7.x86_64.rpm mysql-community-embedded*如果安装过*程中报错，说明系统中包冲突。 error: Failed dependencies: mariadb-libs is obsoleted by mysql-community-libs-5.7.28-1.el7.x86_64 mariadb-libs is obsoleted by mysql-community-libs-compat-5.7.28-1.el7.x86_64 只需要remove掉即可解决冲突 ---------------------------------------------------------------------- yum remove mariadb-libs -ysystemctl start mysqld     #启动mysql systemctl enable mysqld #设成开机自启
　　配置/etc/my.cnf，完成后重启下systemctl restart mysqld[client] default-character-set = utf8mb4  [mysql] default-character-set = utf8mb4 [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock character-set-server = utf8mb4 collation-server = utf8mb4_unicode_ci init_connect=＂SET NAMES utf8mb4＂ validate_password=off #关闭复杂密码校验功能，生产环境请不要这样做 symbolic-links=0 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid
　　取到mysql系统 root 密码grep ＂A temporary password＂ /var/log/mysqld.log ---------------------------------------------------------  2022-02-20T07:06:08.776350Z 1 [Note] A temporary password is generated for root@localhost: hylg#9;dlsSC   看到返回的内容，则初始的root密码为 hylg#9;dlsSC  --------------------------------------------------------
　　修改root密码：mysqladmin -uroot -p＂hylg#9;dlsSC＂ password 123456
　　建立hive meta库和授权create database hive; grant all privileges on hive.* to ＂hive＂@＂%＂ identified by ＂123456＂; flush privileges;设置hive-site.xmlcd /app/hive/conf cp /app/hive/conf/hive-default.xml.template /app/hive/conf/hive-site.xml vi /app/hive/conf/hive-site.xml
　　分别搜索/ConnectionPassword /ConnectionURL /ConnectionUserName /hive.exec.local.scratchdir /hive.downloaded.resources.dir 在其value项中填写我们前面设置的用户名、密码、连接数据库的参数、临时目录等。      hive.exec.local.scratchdir     /tmp     Local scratch space for Hive jobs          hive.downloaded.resources.dir     /tmp     Temporary local directory for added resources in the remote file system.         javax.jdo.option.ConnectionUserName     hive     Username to use against metastore database             javax.jdo.option.ConnectionPassword     123456     password to use against metastore database         javax.jdo.option.ConnectionURL     jdbc:mysql://192.168.1.10:3306/hive?createDatabaseIfNotExist=true            JDBC connect string for a JDBC metastore.       To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL.       For example, jdbc:postgresql://myhost/db?ssl=true for postgres database.      设置hive-env.shcp hive-env.sh.template hive-env.sh vi hive-env.sh 
　　增加以下参数：export  HADOOP_HOME=/app/hadoop export  HIVE_CONF_DIR=/app/hive/conf export  HIVE_AUX_JARS_PATH=/app/hive/lib初始化hive meta信息表schematool -dbType mysql -initSchema至此，恭喜各位hive已安装完成了，大家感兴趣还可以把spark和hbase安装上，这个以后我们再介绍。[cloud@master conf]$ hive -e ＂show databases;＂ which: no hbase in (.:/app/hive/bin:/app/hadoop/bin:/app/hadoop/sbin:/usr/lib/jvm/java-1.8.0-openjdk/bin:/usr/lib/jvm/java-1.8.0-openjdk/jre/bin:/usr/lib/jvm/java-1.8.0-openjdk/bin:/usr/lib/jvm/java-1.8.0-openjdk/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/app/hive/bin:/app/hive/conf) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/app/apache-hive-2.3.9-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/app/hadoop-2.10.1/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]  Logging initialized using configuration in jar:file:/app/apache-hive-2.3.9-bin/lib/hive-common-2.3.9.jar!/hive-log4j2.properties Async: true OK default Time taken: 5.277 seconds, Fetched: 1 row(s)yarn监控地址： http://192.168.1.10:8088/cluster/