Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
A data engineer has developed a data pipeline to ingest data from a JSON source using Auto loader, but the engineer has not provided any type inference or schema hints in their pipeline. Upon reviewing the data, the data engineer has noticed that all of the columns in the target table are of the string type despite some of the fields only including float or boolean values.
Which of the following describes why Auto Loader inferred all of the columns to be of the string type?
Real Exam
Community
KKeng
A
There was a type mismatch between the specific schema and the inferred schema
B
JSON data is a text-based format
C
Auto Loader only works with string data
D
All of the fields had at least one null value
E
Auto Loader cannot infer the schema of ingested data
Explanation:
Explanation
Correct Answer: B. JSON data is a text-based format
JSON (JavaScript Object Notation) is fundamentally a text-based data interchange format. When Auto Loader ingests JSON data without explicit schema hints or type inference, it treats all values as strings by default because:
- JSON stores all data as text - Even numeric values like
123.45or boolean values liketrue/falseare represented as text strings in JSON files. - Auto Loader's default behavior - Without explicit schema definition or type inference options, Auto Loader will infer the schema based on the data it sees. Since JSON is text-based, it defaults to string types for all fields.
- Type inference requires explicit configuration - To get proper type inference (e.g., detecting numeric or boolean types), the data engineer needs to either:
- Provide an explicit schema using
.schema() - Enable schema inference with appropriate options
- Use the
cloudFiles.schemaEvolutionModesetting
- Provide an explicit schema using
Why other options are incorrect:
- A: Type mismatch between specific schema and inferred schema - This would be relevant if a schema was provided, but the question states no schema hints were given.
- C: Auto Loader only works with string data - False, Auto Loader can handle various data types when properly configured.
- D: All fields had at least one null value - While null values can affect type inference, this isn't the primary reason for all columns being strings.
- E: Auto Loader cannot infer the schema of ingested data - False, Auto Loader can infer schemas, but for JSON it defaults to string types without explicit configuration.
Best Practice Recommendation: To avoid this issue, data engineers should either:
- Provide an explicit schema when reading JSON data
- Use schema inference with appropriate options like
cloudFiles.inferColumnTypes = true - Cast columns to appropriate types after ingestion using Spark SQL transformations
Comments
Loading comments...