Deploying Spark on Kubernetes
Want to learn how to build this?
Check out the post.
Want to use this project?
Minikube Setup
Install and run Minikube:
- Install a Hypervisor (like VirtualBox or HyperKit) to manage virtual machines
- Install and Set Up kubectl to deploy and manage apps on Kubernetes
- Install Minikube
Start the cluster:
$ minikube start --memory 8192 --cpus 4
$ minikube dashboardBuild the Docker image:
$ eval $(minikube docker-env)
$ docker build -t spark-hadoop:2.2.1 -f ./docker/Dockerfile ./dockerCreate the deployments and services:
$ kubectl create -f ./kubernetes/spark-master-deployment.yaml
$ kubectl create -f ./kubernetes/spark-master-service.yaml
$ kubectl create -f ./kubernetes/spark-worker-deployment.yaml
$ minikube addons enable ingress
$ kubectl apply -f ./kubernetes/minikube-ingress.yamlAdd an entry to /etc/hosts:
$ echo "$(minikube ip) spark-kubernetes" | sudo tee -a /etc/hostsTest it out in the browser at http://spark-kubernetes/.