Given a requirement to implement necessary logic for deduplication using Spark Structured Streaming, describe the steps you would take to ensure that duplicate records are identified and removed. Include the use of watermarking and the `dropDuplicates` method.

Simulated

Use the dropDuplicates method without using watermarking.

10.8%

Use watermarking to limit the state store size and the dropDuplicates method to remove duplicate records.

86.5%

Ignore duplicate records and focus only on the current data stream.

0.9%

Use a batch query to handle deduplication.

1.8%

Databricks Certified Data Engineer - Professional