CPU utilization is the most direct indicator of compute efficiency in a Spark cluster. Maintaining a sustained utilization around 70–80% (such as 75%) suggests that the executors are actively processing tasks without being idle, while still leaving enough headroom to handle transient spikes in demand without causing significant queuing or resource contention.

Why other options are incorrect:

• Five-minute load average: A steady load average simply tracks the number of processes in the queue; it does not indicate if the CPU cores are actually performing useful work or if they are balanced across the cluster.
• Network I/O: Low network I/O might suggest the job is not bottlenecked by data movement, but it could also indicate that the cluster is under-utilized or not processing much data, which isn't a sign of resource efficiency.
• Disk space usage: Disk usage is a storage metric and does not reflect the real-time processing efficiency of the CPU or Memory during execution.