In a scenario where you are processing sensor data from autonomous vehicles using Spark Structured Streaming, how would you design your Spark job to handle schema drift and process time series data efficiently? Additionally, describe how you would manage data across partitions and within one partition to ensure optimal performance.