You are given a Spark DataFrame 'df' with a date column 'date' in the format 'yyyy-MM-dd'. Write a code snippet that extracts the year, month, and day from the 'date' column and creates new columns for each component, and explain the steps involved.

Simulated

from pyspark.sql.functions import year, month, day df = df.withColumn('year', year('date')).withColumn('month', month('date')).withColumn('day', day('date')) print(A)

69.0%

df = df.withColumn('year', df.date.substr(0, 4)).withColumn('month', df.date.substr(5, 2)).withColumn('day', df.date.substr(8, 2)) print(B)

10.3%

df = df.withColumn('year', df.date.year()).withColumn('month', df.date.month()).withColumn('day', df.date.day()) print(C)

13.8%

df = df.selectExpr('date', 'year(date) as year', 'month(date) as month', 'day(date) as day') print(D)

6.9%

Databricks Certified Machine Learning - Associate

Get started today

Comments

You are given a Spark DataFrame 'df' with a date column 'date' in the format 'yyyy-MM-dd'. Write a code snippet that extracts the year, month, and day from the 'date' column and creates new columns for each component, and explain the steps involved.