Apache Spark on Kubernetes - useful commands

Versions: Apache Spark 2.3.1

Beginning with new tool and its CLI is never easy. Having a list of useful debugging commands is always helpful. And the rule is not different for Spark on Kubernetes project.

Looking for a better data engineering position and skills?

You have been working as a data engineer but feel stuck? You don't have any new challenges and are still writing the same jobs all over again? You have now different options. You can try to look for a new job, now or later, or learn from the others! "Become a Better Data Engineer" initiative is one of these places where you can find online learning resources where the theory meets the practice. They will help you prepare maybe for the next job, or at least, improve your current skillset without looking for something else.

👉 I'm interested in improving my data engineering skillset

See you there, Bartosz

This post lists some kubectl commands that may be helpful in first contact with Kubernetes CLI. The commands are written in a single list and each of them is composed of a short explanation and generated output.

Among the commands that can help in the firsts contact with Spark on Kubernetes we can distinguish:

The post listed some interesting commands that can help us to start working with Spark on Kubernetes. Among them we can find a lot of kubectl describe examples thanks to which we can easily see what is really executed (e.g. pod specification). We can also see more network-related commands as the one for proxy forwarding letting us to see Spark's driver UI. The last category of commands concerns listing and is executed with kubectl get.