When your Databricks cluster shows unexpected performance degradation, and you suspect it's due to a complex interaction between job configurations and specific data patterns, what is the best method to diagnose this issue using custom metrics and logs?
