Answer-first summary for fast verification
Answer: Apply schema-on-read during data loading to enforce data types and nullability constraints.
Applying schema-on-read during data loading with Apache Spark is the most efficient and scalable approach to ensure high data quality. This method enforces data types and nullability constraints at the point of ingestion, preventing data type mismatches and missing values early in the process. It leverages Spark's capability to handle large volumes of data efficiently, ensuring consistent and reliable data for downstream analysis and processing. Compared to manual inspection or external tools, schema-on-read offers a more integrated and less time-consuming solution.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
No comments yet.
When developing a data ingestion pipeline that consolidates data from various sources using Apache Spark, which method would you employ to ensure high data quality across ingested datasets?
A
Utilize Spark's built-in data frame functions to clean and validate data after ingestion.
B
Manually inspect a sample of the ingested data for quality issues.
C
Apply schema-on-read during data loading to enforce data types and nullability constraints.
D
Implement an external data quality tool to preprocess files before ingestion.