
Ultimate access to all questions.
Consider a scenario where you are processing time series data from IoT devices using Spark Structured Streaming. The data includes temperature readings from various sensors. How would you structure your Spark job to process this data efficiently, including handling data across partitions and within one partition? Additionally, describe how you would manage any potential schema changes in the incoming data.
A
Use Spark Structured Streaming to read data, process each record individually without considering partitions, and ignore schema changes.
B
Set up Spark Structured Streaming to read data, process data only within one partition, and use schema evolution to handle changes.
C
Configure Spark Structured Streaming to read data, process data across partitions using repartitioning, and handle schema changes with schema evolution.
D
Use Spark Structured Streaming to read data, ignore partitions, and use a fixed schema without handling changes.