Answer: Cluster: New Job Cluster; Retries: Unlimited; Maximum Concurrent Runs: 1

For Structured Streaming jobs in production, the optimal configuration balances automatic recovery from failures and cost efficiency. Option A is correct because it uses a new job cluster, which is cost-effective as the cluster terminates after the job completes. It also allows for unlimited retries, ensuring automatic recovery from failures, and limits the maximum concurrent runs to 1, preventing overlapping executions and avoiding duplicate processing and cost spikes from multiple clusters. Options B, C, and D are less optimal because B risks higher costs and parallel failures with unlimited concurrent runs, C uses an existing all-purpose cluster which is not cost-efficient for production jobs, and D does not allow for any retries, eliminating the possibility of automatic recovery from failures.

Author: LeetQuiz Editorial Team