Explanation:
In Databricks, when using PySpark Structured Streaming (spark.readStream), specifying .format("cloudFiles") tells Spark to use the Databricks Auto Loader engine for incrementally ingesting files from cloud storage.
Ultimate access to all questions.
.format('cloudFiles') option in a Spark read block?A
The data is stored strictly in DBFS.
B
It specifies that the Auto Loader engine should be used.
C
It forces all files to be compressed.
D
It tells Spark to ignore schema changes.
No comments yet.