Implement dynamic partition discovery to automatically adjust the number of partitions based on the dataset size, aiming to improve parallelism and query performance.

Configure the streaming job to use a larger batch interval, reducing the frequency of file processing to minimize the impact of small files on performance.

Apply file concatenation or bucketing strategies to reduce the number of small files, thereby decreasing I/O operations and enhancing query performance.

Enable speculative execution to automatically detect and mitigate the performance issues caused by small files by running additional tasks for slow-running partitions.