
Ultimate access to all questions.
A data engineering team is optimizing a complex pipeline handling trillions of rows per table. They choose to persist some frequently used DataFrames to speed up query processing. After executing the persist() command on a DataFrame, a data engineer checks the Spark UI's Storage Tab but finds no information about the persisted DataFrame. What could be the reason?