In the context of Azure Databricks, you are tasked with optimizing a real-time data processing pipeline that involves stream-static joins using Delta Lake. The pipeline must efficiently explore and tune state information to handle high-volume data streams with low latency. Considering the requirements for scalability, cost-effectiveness, and compliance with data governance policies, which of the following approaches is the BEST for exploring and updating state information in this scenario? Choose the correct option from the four provided.