Databricks Certified Data Engineer - Professional
Get started today
Ultimate access to all questions.
Ultimate access to all questions.
Consider a Delta Lake table that is heavily impacted by 'smalls' (tiny files, scanning overhead, over partitioning). Describe in detail how you would redesign the data processing pipeline to mitigate these issues, focusing on the use of CDF and proper partitioning strategies. Provide a code snippet illustrating the key changes.