CDH 5.16.2 Deploy

Cloudera Manager

載點

https://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_516.html

網路設定

1. 設定hostname

hostnamectl set-hostname manager-1.example.com

2. 編輯/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.1.239  manager-1.example.com  manager-1
10.0.1.240  worker-1.example.com  worker-1
10.0.1.241  worker-2.example.com  worker-2
10.0.1.238  worker-3.example.com  worker-3

安裝 JDK

yum update
yum install -y java-1.8.0-openjdk-devel

安裝 Cloudera Manager Server

下載cloudera-manager repository

yum install -y wget

1. 下載Cloudera-manager repository 至 /etc/yum.repos.d/

wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/

2. Import the repository signing GPG key & yum install

rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera

3. yum install

yum install -y cloudera-manager-daemons cloudera-manager-server

設定Database

install & init db

yum install -y postgresql-server
sudo su -l postgres -c "postgresql-setup initdb"

編輯 /var/lib/pgsql/data/pg_hba.conf, 設定 MD5 authentication

host all all 127.0.0.1/32 md5

編輯 /var/lib/pgsql/data/postgresql.conf

listen_addresses = '*'
max_connection = 100
shared_buffers = 256MB
wal_buffers = 8MB
checkpoint_segments = 16
checkpoint_completion_target = 0.9

啟用postgreSQL

systemctl enable postgresql
systemctl restart postgresql

Creating Databases for Cloudera Software

1. Connect to PostgreSQL:

sudo -u postgres psql

2. Create databases

postgres=# CREATE ROLE scm LOGIN PASSWORD 'scm';
postgres=# CREATE DATABASE scm OWNER scm ENCODING 'UTF8';
postgres=# CREATE ROLE amon LOGIN PASSWORD 'amon';
postgres=# CREATE DATABASE amon OWNER amon ENCODING 'UTF8';
postgres=# CREATE ROLE rman LOGIN PASSWORD 'rman';
postgres=# CREATE DATABASE rman OWNER rman ENCODING 'UTF8';
postgres=# CREATE ROLE nav LOGIN PASSWORD 'nav';
postgres=# CREATE DATABASE nav OWNER nav ENCODING 'UTF8';
postgres=# CREATE ROLE navms LOGIN PASSWORD 'navms';
postgres=# CREATE DATABASE navms OWNER navms ENCODING 'UTF8';

3. 設定Cloudera Manager Database

sudo /usr/share/cmf/schema/scm_prepare_database.sh postgresql scm scm

啟用Cloudera Manager Server

1. Start Cloudera Manager Server

systemctl start cloudera-scm-server

2. 查看cloudera-scm-server.log

sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

看到以下log訊息表示cloudera-scm-server已啟動完成
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

3. 到web介面 http://<manager_server_host>:7180 登入

• Username: admin
• Password: admin

Cloudera Agent(worker node for cluster)

網路設定

• 設定hostname

hostnamectl set-hostname worker-1.example.com

安裝 JDK

yum update
yum install -y java-1.8.0-openjdk-devel

安裝 cloudera-manager-agent

下載cloudera-manager repository

yum install -y wget

1. 下載Cloudera-manager repository 至 /etc/yum.repos.d/

wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/

2. Import the repository signing GPG key

rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera

下載Cloudera-manager-agent

1. install the package

yum install -y cloudera-manager-daemons cloudera-manager-agent

2. 編輯 /etc/cloudera-scm-agent/config.ini，設定Manager的host, port

server_host=<manager-node ip> 
server_port=7182 

3. 啟用 cloudera-manager-agent

systemctl start cloudera-scm-agent

問題解決

啟動叢集

yarn

• 錯誤訊息

IO error:/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied

1. 到NodeManager role的節點
2. 更改/var/lib/hadoop-yarn權限
3. chmod 755 /var/lib/hadoop-yarn

hue load balancer start failed

• 錯誤訊息

/run/cloudera-scm-agent/process/216-hue-HUE_LOAD_BALANCER/httpd.conf is invalid

1. 到hue LoadBalancer role的節點
2. 安裝 httpd, mod_ssl

yum install -y httpd
yum install -y mod_ssl
systemctl start httpd

3. 若還是失敗，可能是seLinux設定必須關閉
4. 編輯/etc/sysconfig/selinux

SELINUX=disabled
SELINUXTYPE=targeted

5. 重啟LoadBalancer

zookeeper start failed

1. 到zookeeper role 的節點
2. 刪除 /var/lib/zookeeper/version-2
3. 重新建立 /var/lib/zookeeper/version-2，修改用戶

mkdir /var/lib/zookeeper/version-2
chown zookeeper:zookeeper /var/lib/zookeeper/version-2
chmod 755 /var/lib/zookeeper/version-2

4. 重啟zookeepr

oozie start failed

1. 到oozie role 的節點
2. 修改/var/lib/oozie權限及用戶

chown oozie:oozie /var/lib/oozie
chmod 755 /var/lib/oozie

3. 重啟oozie