
Ultimate access to all questions.
You are using AUTO LOADER to process millions of files daily and noticed a slowdown in the load process. After scaling up the Databricks cluster, the performance of the Auto Loader did not improve. What is the most effective solution to this issue?
A
Merge files to one large file
B
Increase the maxFilesPerTrigger option to a sufficiently high number
C
Setup a second AUTO LOADER process to process the data
D
Copy the data from cloud storage to local disk on the cluster for faster access
E
AUTO LOADER is not suitable to process millions of files a day