Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
Deep dive into the quiz with AI chat providers.
We prepare a focused prompt with your quiz and certificate details so each AI can offer a more tailored, in-depth explanation.
A data engineer has developed a data pipeline to ingest data from a JSON source using Auto Loader, but the engineer has not provided any type inference or schema hints in their pipeline. Upon reviewing the data, the data engineer has noticed that all of the columns in the target table are of the string type despite some of the fields only including float or boolean values.
Which of the following describes why Auto Loader inferred all of the columns to be of the string type?
Real Exam
Community
KKeng
0
A
There was a type mismatch between the specific schema and the inferred schema
B
JSON data is a text-based format
C
Auto Loader only works with string data
D
All of the fields had at least one null value
E
Auto Loader cannot infer the schema of ingested data
Explanation:
Explanation:
JSON (JavaScript Object Notation) is fundamentally a text-based data interchange format. When Auto Loader ingests JSON data without explicit schema hints or type inference, it treats all values as strings because:
- JSON stores all data as text - Numbers, booleans, and null values are represented as text strings in JSON format
- Auto Loader's schema inference - When Auto Loader reads JSON files, it performs schema inference by sampling the data. Since JSON values are text-based, the initial inference defaults to string type
- Type ambiguity - Without explicit type hints, Auto Loader cannot reliably distinguish between string representations of numbers/booleans and actual string data
Why other options are incorrect:
- A: Type mismatch would occur if there was a conflict between user-provided schema and inferred schema, but here no schema was provided
- C: Auto Loader can work with various data types when proper schema is provided
- D: Null values alone don't cause all columns to be inferred as strings
- E: Auto Loader can infer schema, but for JSON it defaults to string types due to the text-based nature of the format
Best Practice: To avoid this issue, data engineers should either:
- Provide explicit schema using
schemaoption - Use
cloudFiles.schemaHintsto specify column types - Use
cloudFiles.schemaEvolutionModewith appropriate settings - Cast columns to appropriate types after ingestion
Comments
Loading comments...