Databricks Certified Data Engineer - Associate

Explanation

In the Databricks Medallion Architecture:

• Bronze tables contain raw data
• Silver tables contain cleaned, filtered, and enriched data
• Gold tables contain aggregated, business-level data for reporting and analytics

A hop from Silver to Gold involves aggregation operations that transform data from detailed records to summarized business metrics.

Let's analyze each option:

Option A: Reads from rawSalesLocation (likely Bronze) and writes to newSales - this is Bronze to Silver.

Option B: Uses spark.read.load() instead of spark.readStream.load() - this is batch processing, not streaming.

Option C: Reads from sales table (Silver), performs a simple column transformation (avgPrice calculation), and writes to newSales - this is Silver to Silver transformation.

Option D: Reads from sales table (Silver), applies a filter, and writes to newSales - this is Silver to Silver transformation with filtering.

Option E: Reads from sales table (Silver), performs aggregation (groupBy("store") and agg(sum("sales"))), and writes to newSales - this is Silver to Gold transformation because it involves aggregation, which is characteristic of Gold table creation.

Key indicators of Silver to Gold hop:

1. Reading from an existing table (Silver layer)
2. Performing aggregation operations (groupBy, sum, avg, etc.)
3. Creating summarized business metrics
4. Using outputMode("complete") which is common for aggregated streaming results

The correct answer is E because it demonstrates the transformation from detailed Silver data to aggregated Gold data through grouping and aggregation operations.