Centos7安装Cloudera Manager、CDH、JDK1.8及Spark2.2记录-day3

7. 安装JDK1.8(所有虚拟机都安装)

7.1、安装JDK1.8

1)移除之前的java JDK

用命令查看当前java的版本:

# java -version
openjdk version “1.8.0_181”
OpenJDK Runtime Environment (build 1.8.0_181-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

可以看到当前版本为openjdk,需要更换为oracle的版本,查找openjdk安装的包,删除。

# rpm -qa | grep java
python-javapackages-3.4.1-11.el7.noarch
javamail-1.4.6-8.el7.noarch
mysql-connector-java-5.1.25-3.el7.noarch
java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64
javapackages-tools-3.4.1-11.el7.noarch
java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
javassist-3.16.1-10.el7.noarch
tzdata-java-2018e-3.el7.noarch

rpm -e  –nodeps java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64
rpm -e –nodeps java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64

2)安装JDK1.8

到oracle官网下载jdk:
https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html

cd ~ //进入home目录

tar -zxvf jdk-8u171-linux-x64.tar.gz //解压jdk1.8压缩包

mkdir -p /usr/local/java //新建java文件夹

cp -R jdk1.8.0_171 /usr/local/java/    //把jdk包复制到java文件夹

2、/etc/profile环境变量配置

1)echoJAVA_HOME=/usr/local/java/jdk1.8.0_171 >> /etc/profile //添加环境变量

2)echo JRE_HOME=/usr/local/java/jdk1.8.0_171/jre  >> /etc/profile  //添加环境变量

3)echo CLASSPATH=$JAVA_HOME/lib/dt.jar:\$JAVA_HOME/lib/tools.jar:\$JRE_HOME/lib >>/etc/profile //添加环境变量

4)PATH=$PATH:$JAVA_HOME/bin/ >> /etc/profile //添加环境变量

source /etc/profile  //让修改生效

# java -version  //测试验证jdk版本
java version “1.8.0_171”
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

3、spark_home环境变量配置

升级JDK的时候,有时需要配置scala_home和spark_home路径,记录如下:

echo SCALA_HOME=/opt/scala-2.11.4  >> /etc/profile

echo SPARK_HOME=/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2  >> /etc/profile

echo PATH=\$SCALA_HOME/bin:\$JAVA_HOME/bin:\$PATH  >>/etc/profile

echo export PATH JAVA_HOME CLASSPATH SCALA_HOME SPARK_HOME >>/etc/profile

8.  安装CM(Cloudera Manager)

1、创建数据库

# mysql -u root -p

mysql> create database scm DEFAULT CHARACTER SET utf8;

mysql> create database hivedb DEFAULT CHARACTER SET utf8;

mysql> create database amondb DEFAULT CHARACTER SET utf8;

mysql> create database hmondb DEFAULT CHARACTER SET utf8;

mysql> create database smondb DEFAULT CHARACTER SET utf8;

mysql> grant all on scm.* to ‘scmuser’@’master’ identified by ‘scmuser’;

mysql> grant all on hivedb.* to ‘hivedbuser’@’master’ identified by ‘hivedbuser’;

mysql> grant all on amondb.* to ‘amondbuser’@’master’ identified by ‘amondbuser’;

mysql> grant all on hmondb.* to ‘hmondbuser’@’master’ identified by ‘hmondbuser’;

mysql> grant all on smondb.* to ‘smondbuser’@’master’ identified by ‘smondbuser’;

mysql> grant all on scm.* to ‘scmuser’@’%’ identified by ‘scmuser’;

mysql> grant all on hivedb.* to ‘hivedbuser’@’%’ identified by ‘hivedbuser’;

mysql> grant all on amondb.* to ‘amondbuser’@’%’ identified by ‘amondbuser’;

mysql> grant all on hmondb.* to ‘hmondbuser’@’%’ identified by ‘hmondbuser’;

mysql> grant all on smondb.* to ‘smondbuser’@’%’ identified by ‘smondbuser’;

mysql> GRANT ALL PRIVILEGES ON *.* TO ‘root’@’%’ IDENTIFIED BY ‘root’ WITH GRANT OPTION;

mysql> flush privileges;

mysql> exit

2、创建目录并复制文件

1)先将需要的rpm包和cdh传到/software目录下

2)创建目录和软连接

# mkdir /opt/yum.repos.ld

# mkdir -p /opt/yum.repos.ld/cm/5.14.1

# cd /opt/yum.repos.ld/cm

# ln -s 5.14.1 latest

# mkdir -p /opt/yum.repos.ld/parcels/cdh/5.14.0

# cd /opt/yum.repos.ld/parcels/cdh

# ln -s 5.14.0 latest

3)复制文件

# cp  CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel /opt/yum.repos.ld/parcels/cdh/5.14.0/

# cp manifest.json /opt/yum.repos.ld/parcels/cdh/5.14.0/

# cp CDH-5.14.0-1.cdh5.14.0.p0.24-el7.parcel.sha /opt/yum.repos.ld/

# cp cloudera-manager.repo /etc/yum.repos.d/

# cp cloudera-*.rpm /opt/yum.repos.ld/cm/5.14.1/

# cp enterprise-debuginfo-5.14.1-1.cm5141.p0.1.el7.x86_64.rpm /opt/yum.repos.ld/cm/5.14.1/

# cp jdk-6u31-linux-amd64.rpm /opt/yum.repos.ld/cm/5.14.1/

# cp RPM-GPG-KEY-cloudera /opt/yum.repos.ld/cm/

# cp oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm /opt/yum.repos.ld/cm/5.14.1/

# cd /opt/yum.repos.ld/cm/5.14.1

# createrepo .

# cd /var/www/html/

# ln -s /opt/yum.repos.ld yum.repos.ld

3、启动httpd服务

# systemctl start httpd.service

# systemctl status httpd.service

#Web访问 (10.108.78.220为master ip)

http://10.108.78.220/yum.repos.ld

1)修改yum源,确保从本地获取cdh和CM的rpm包,首先保证/etc/yum.repos.d下只有一个repo文件,没有其他的,否则可能会冲突,所有节点都是相同的

# vi /etc/yum.repos.d/cloudera-manager.repo

[cloudera-manager]

# Packages for Cloudera Manager, Version 4, on RedHat or CentOS 6 x86_64

name=Cloudera Manager

baseurl=http:// 10.108.78.220/yum.repos.ld/cm/latest/

#gpgkey=http:// 10.108.78.220/yum.repos.ld/cm/RPM-GPG-KEY-cloudera  

gpgcheck = 0不进行校验

2)分发到其他各节点

# scp /etc/yum.repos.d/cloudera-manager.repo root@slave1:/etc/yum.repos.d/

# scp /etc/yum.repos.d/cloudera-manager.repo root@slave2:/etc/yum.repos.d/

# scp /etc/yum.repos.d/cloudera-manager.repo root@slave3:/etc/yum.repos.d/

3)安装各rpm包及启动相关服务进程

#rpm -ivh cloudera-manager-agent-5.14.1-1.cm5141.p0.1.el7.x86_64.rpm(所有节点)

#rpm -ivh cloudera-manager-daemons-5.14.1-1.cm5141.p0.1.el7.x86_64.rpm(所有节点)

#rpm -ivh cloudera-manager-server-5.14.1-1.cm5141.p0.1.el7.x86_64.rpm(主节点)

#rpm -ivh cloudera-manager-server-db-2-5.14.1-1.cm5141.p0.1.el7.x86_64.rpm(主节点)

4)执行安装

# yum clean all(由于修改了yum源,此处安装之前必须先清理一下缓存)

# yum install cloudera-manager-server

5)修改数据库的相关配置

# vi /etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.host=master:3306

com.cloudera.cmf.db.name=scm

com.cloudera.cmf.db.user=scmuser

com.cloudera.cmf.db.password=scmuser

因为此处mysql是外部数据库,则需要改为:

com.cloudera.cmf.db.setupType= EXTERNAL(内嵌数据库则为EMBEDED)

4、cloudera-scm-server和agent的JAVA_HOME配置

1)配置server和agent的JAVA_HOME,详情请见截图:

vim /etc/default/cloudera-scm-server
export JAVA_HOME=/usr/local/java/jdk1.8.0_171

注意:cloudera-scm-server只需要在主节点进行JAVA_HOME的配置;cloudera-scm-agent需要在所有节点进行JAVA_HOME的配置。

编辑cloudera-scm-server,加入:
export JAVA_HOME=/usr/local/java/jdk1.8.0_171

2)配置之后重启集群

主节点输入命令:

# systemctl start cloudera-scm-server
# systemctl status cloudera-scm-server

所有节点输入命令 # systemctl start cloudera-scm-agent

# systemctl status cloudera-scm-agent


One thought on “Centos7安装Cloudera Manager、CDH、JDK1.8及Spark2.2记录-day3

发表评论

电子邮件地址不会被公开。 必填项已用*标注