Answer-first summary for fast verification
Answer: maxBytesPerTrigger and maxFilesPerTrigger
To control the amount of data processed in each micro-batch, the following input limit options can be used: `maxFilesPerTrigger` and `maxBytesPerTrigger`. It's important to note that if both are used in a single query, the data processed in each micro-batch will be the lesser of `maxFilesPerTrigger` and `maxBytesPerTrigger`. Whichever limit is reached first, the data processing will be stopped for that micro-batch of the stream. For more information, refer to the documentation on limiting input rate for streaming reads.
Author: LeetQuiz Editorial Team
Ultimate access to all questions.
An upstream system sends more than 1000 files every hour, each containing between 5000 to 30000 records, with file sizes ranging from 580MB to 1.22 GB. The data is ingested using a medallion architecture. To limit the amount of data processed in each micro-batch, which of the following options is valid for controlling the input rate while reading the stream?
A
maxFilesPerTrigger and maxRowsPerTrigger
B
maxBytesPerTrigger and maxRecordsPerTrigger
C
maxBytesPerTrigger and maxFilesPerTrigger
D
maxFilesPerTrigger and maxRecordsPerTrigger
E
maxFilesPerTrigger, maxBytesPerTrigger, maxRowsPerTrigger, and maxRecordsPerTrigger
No comments yet.