Databricks Certified Data Engineer - Associate

Explanation

In Databricks Lakehouse architecture, data typically flows through different quality tiers:

• Bronze: Raw data ingestion
• Silver: Cleaned, validated data
• Gold: Aggregated, business-ready data for analytics

A hop from Silver to Gold means transforming Silver-level data (already cleaned and validated) into aggregated, business-ready Gold tables.

Let's analyze each option:

Option A: Reads from rawSalesLocation (likely Bronze/raw data) and writes to newSales. This is a Bronze-to-Silver hop, not Silver-to-Gold.

Option B: Uses spark.read.load() (batch read) with writeStream (streaming write) - this is incorrect syntax and doesn't represent a proper hop.

Option C: Reads from sales table (could be Silver) and performs a simple transformation (avgPrice calculation), then writes to newSales. This is a simple transformation but not necessarily a Silver-to-Gold hop as it doesn't involve aggregation.

Option D: Reads from sales table (could be Silver) and performs filtering, then writes to newSales. This is filtering, not aggregation for Gold tables.

Option E: Reads from sales table (Silver), performs aggregation (groupBy and sum), and writes to newSales with complete output mode. This represents a Silver-to-Gold hop because:

1. It reads from an existing table (sales) which is likely a Silver table
2. It performs aggregation (groupBy and agg) which is typical for Gold table transformations
3. It uses complete output mode which is appropriate for aggregated results
4. The transformation creates business-ready aggregated data suitable for analytics

The key indicators of a Silver-to-Gold hop are:

• Reading from an existing table (not raw data)
• Performing aggregations or complex business transformations
• Creating data suitable for business analytics and reporting