iT邦幫忙

第 12 屆 iT 邦幫忙鐵人賽

DAY 29
0
Software Development

基於付費公有雲與開源機房自建私有雲之雲端應用服務測試兼叢集與機房託管服務實戰之勇者崎嶇波折且劍還掉在路上的試煉之路系列 第 29

Day 29 Cloudera Manager

CDH 5.16.2 Deploy

Cloudera Manager

載點

https://docs.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_516.html

網路設定

  1. 設定hostname
hostnamectl set-hostname manager-1.example.com
  1. 編輯/etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.0.1.239  manager-1.example.com  manager-1
10.0.1.240  worker-1.example.com  worker-1
10.0.1.241  worker-2.example.com  worker-2
10.0.1.238  worker-3.example.com  worker-3

安裝 JDK

yum update
yum install -y java-1.8.0-openjdk-devel

安裝 Cloudera Manager Server

下載cloudera-manager repository

yum install -y wget
  1. 下載Cloudera-manager repository 至 /etc/yum.repos.d/
wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/
  1. Import the repository signing GPG key & yum install
rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
  1. yum install
yum install -y cloudera-manager-daemons cloudera-manager-server

設定Database

install & init db

yum install -y postgresql-server
sudo su -l postgres -c "postgresql-setup initdb"

編輯 /var/lib/pgsql/data/pg_hba.conf, 設定 MD5 authentication

host all all 127.0.0.1/32 md5

編輯 /var/lib/pgsql/data/postgresql.conf

listen_addresses = '*'
max_connection = 100
shared_buffers = 256MB
wal_buffers = 8MB
checkpoint_segments = 16
checkpoint_completion_target = 0.9

啟用postgreSQL

systemctl enable postgresql
systemctl restart postgresql

Creating Databases for Cloudera Software

  1. Connect to PostgreSQL:
sudo -u postgres psql
  1. Create databases
postgres=# CREATE ROLE scm LOGIN PASSWORD 'scm';
postgres=# CREATE DATABASE scm OWNER scm ENCODING 'UTF8';
postgres=# CREATE ROLE amon LOGIN PASSWORD 'amon';
postgres=# CREATE DATABASE amon OWNER amon ENCODING 'UTF8';
postgres=# CREATE ROLE rman LOGIN PASSWORD 'rman';
postgres=# CREATE DATABASE rman OWNER rman ENCODING 'UTF8';
postgres=# CREATE ROLE nav LOGIN PASSWORD 'nav';
postgres=# CREATE DATABASE nav OWNER nav ENCODING 'UTF8';
postgres=# CREATE ROLE navms LOGIN PASSWORD 'navms';
postgres=# CREATE DATABASE navms OWNER navms ENCODING 'UTF8';
  1. 設定Cloudera Manager Database
sudo /usr/share/cmf/schema/scm_prepare_database.sh postgresql scm scm

啟用Cloudera Manager Server

  1. Start Cloudera Manager Server
systemctl start cloudera-scm-server
  1. 查看cloudera-scm-server.log
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

看到以下log訊息表示cloudera-scm-server已啟動完成
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

  1. 到web介面 http://<manager_server_host>:7180 登入
  • Username: admin
  • Password: admin

Cloudera Agent(worker node for cluster)

網路設定

  • 設定hostname
hostnamectl set-hostname worker-1.example.com

安裝 JDK

yum update
yum install -y java-1.8.0-openjdk-devel

安裝 cloudera-manager-agent

下載cloudera-manager repository

yum install -y wget
  1. 下載Cloudera-manager repository 至 /etc/yum.repos.d/
wget https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d/
  1. Import the repository signing GPG key
rpm --import https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera

下載Cloudera-manager-agent

  1. install the package
yum install -y cloudera-manager-daemons cloudera-manager-agent
  1. 編輯 /etc/cloudera-scm-agent/config.ini,設定Manager的host, port
server_host=<manager-node ip> 
server_port=7182 
  1. 啟用 cloudera-manager-agent
systemctl start cloudera-scm-agent

問題解決

啟動叢集

yarn

  • 錯誤訊息

IO error:/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied

  1. 到NodeManager role的節點
  2. 更改/var/lib/hadoop-yarn權限
  3. chmod 755 /var/lib/hadoop-yarn

hue load balancer start failed

  • 錯誤訊息

/run/cloudera-scm-agent/process/216-hue-HUE_LOAD_BALANCER/httpd.conf is invalid

  1. 到hue LoadBalancer role的節點
  2. 安裝 httpd, mod_ssl
yum install -y httpd
yum install -y mod_ssl
systemctl start httpd
  1. 若還是失敗,可能是seLinux設定必須關閉
  2. 編輯/etc/sysconfig/selinux
SELINUX=disabled
SELINUXTYPE=targeted
  1. 重啟LoadBalancer

zookeeper start failed

  1. 到zookeeper role 的節點
  2. 刪除 /var/lib/zookeeper/version-2
  3. 重新建立 /var/lib/zookeeper/version-2,修改用戶
mkdir /var/lib/zookeeper/version-2
chown zookeeper:zookeeper /var/lib/zookeeper/version-2
chmod 755 /var/lib/zookeeper/version-2
  1. 重啟zookeepr

oozie start failed

  1. 到oozie role 的節點
  2. 修改/var/lib/oozie權限及用戶
chown oozie:oozie /var/lib/oozie
chmod 755 /var/lib/oozie
  1. 重啟oozie

上一篇
Day 28 Spark on kubernetes
下一篇
Day 30 Python3 + selenium 擷取網站狀態快照
系列文
基於付費公有雲與開源機房自建私有雲之雲端應用服務測試兼叢集與機房託管服務實戰之勇者崎嶇波折且劍還掉在路上的試煉之路30

尚未有邦友留言

立即登入留言