When developing complex data transformation logic in Spark on Databricks using custom User Defined Functions (UDFs), what are the best practices to ensure they are both performant and maintainable?

Real Exam

Broadcasting small datasets used within UDFs to optimize their access across the cluster nodes

10.7%

Using Spark SQL's built-in functions wherever possible and reserving UDFs for operations that cannot be expressed with built-in functions

74.7%

Writing UDFs in Python for ease of development, disregarding the performance implications compared to Scala or Java UDFs

6.7%

Encapsulating all transformation logic within a single UDF to minimize the invocation overhead

8.0%

Databricks Certified Data Engineer - Professional