Apache Spark on Kubernetes - useful commands

Versions: Apache Spark 2.3.1

Beginning with new tool and its CLI is never easy. Having a list of useful debugging commands is always helpful. And the rule is not different for Spark on Kubernetes project.

New ebook 🔥

Learn 84 ways to solve common data engineering problems with cloud services.

👉 I want my Early Access edition

This post lists some kubectl commands that may be helpful in first contact with Kubernetes CLI. The commands are written in a single list and each of them is composed of a short explanation and generated output.

Among the commands that can help in the firsts contact with Spark on Kubernetes we can distinguish:

The post listed some interesting commands that can help us to start working with Spark on Kubernetes. Among them we can find a lot of kubectl describe examples thanks to which we can easily see what is really executed (e.g. pod specification). We can also see more network-related commands as the one for proxy forwarding letting us to see Spark's driver UI. The last category of commands concerns listing and is executed with kubectl get.