Spark on kubernetes

Environment

Ubuntu

• HP Z230
• 數量: 3
• Ubuntu: 16.04 / 18.04
• CPU: E3-1230_V3 ^ 1
• RAM: 16GB
• Disk: 250 GB (HDD)
• Network: 1Gbps

Architecture

Download Spark package

wget http://apache.stu.edu.tw/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz
tar -xvf spark-2.4.7-bin-hadoop2.7.tgz
mv spark-2.4.7-bin-hadoop2.7 spark

http://spark.apache.org/downloads.html

Build Docker Images

$ cd spark-2.4.7-bin-hadoop2.7/
$ ./bin/docker-image-tool.sh -r <repo> -t my-tag build
$ ./bin/docker-image-tool.sh -r <repo> -t my-tag push

Submitting Applications to Kubernetes

$ bin/spark-submit \ 
      --master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \  
      --deploy-mode cluster \ 
      --name spark-pi \ 
      --class org.apache.spark.examples.SparkPi \ 
      --conf spark.executor.instances=5 \ 
      --conf spark.kubernetes.container.image=<spark-image> \ 
      local:///path/to/examples.jar

kubectl logs pods-name

Accessing Driver UI