Explanation:
In Databricks, when using PySpark Structured Streaming (spark.readStream), specifying .format("cloudFiles") tells Spark to use the Databricks Auto Loader engine for incrementally ingesting files from cloud storage.
.format('cloudFiles') option in a Spark read block?A
The data is stored strictly in DBFS.
B
It specifies that the Auto Loader engine should be used.
C
It forces all files to be compressed.
D
It tells Spark to ignore schema changes.
No comments yet.