When dealing with highly sparse datasets, such as user interaction matrices filled with null values, which storage format or technique optimizes both query performance and storage efficiency?

Real Exam

Normalize the dataset into multiple tables to separate dense columns from sparse ones, reducing storage overhead.

4.1%

Use Delta Lake‘s binary storage format with custom compression algorithms tailored to sparse data.

19.2%

Store data in a columnar format like Parquet, leveraging its built-in compression mechanisms for null values.

67.1%

Implement a custom sparse matrix storage format as a UDF that compresses null values and decompresses them during queries.

9.6%

Databricks Certified Data Engineer - Professional