Ultimate access to all questions.
Upgrade Now 🚀
Sign in to unlock AI tutor
You are using AUTO LOADER to process millions of files daily and noticed a slowdown in the load process. After scaling up the Databricks cluster, the performance of the Auto Loader did not improve. What is the most effective solution to this issue?
A
Merge files to one large file
B
Increase the maxFilesPerTrigger option to a sufficiently high number
maxFilesPerTrigger
C
Setup a second AUTO LOADER process to process the data
D
Copy the data from cloud storage to local disk on the cluster for faster access
E
AUTO LOADER is not suitable to process millions of files a day
