Databricks Certified Data Engineer - Associate
Get started today
Ultimate access to all questions.
Deep dive into the quiz with AI chat providers.
We prepare a focused prompt with your quiz and certificate details so each AI can offer a more tailored, in-depth explanation.
A data analyst has a series of queries in a SQL program. The data analyst wants this program to run every day. They only want the final query in the program to run on Sundays. They ask for help from the data engineering team to complete this task. Which of the following approaches could be used by the data engineering team to complete this task?
Real Exam
Community
KKeng
0
A
They could submit a feature request with Databricks to add this functionality.
B
They could wrap the queries using PySpark and use Python's control flow system to determine when to run the final query.
C
They could only run the entire program on Sundays.
D
They could automatically restrict access to the source table in the final query so that it is only accessible on Sundays.
E
They could redesign the data model to separate the data used in the final query into a new table.
Explanation:
Explanation
Correct Answer: B
Why Option B is correct:
- PySpark with Python control flow: By wrapping the SQL queries in PySpark, you can use Python's datetime functions to check the current day of the week.
- Conditional execution: You can implement logic like:
if datetime.today().weekday() == 6: # Sunday is 6 in Python (0=Monday) # Run the final query spark.sql("final_query_here")if datetime.today().weekday() == 6: # Sunday is 6 in Python (0=Monday) # Run the final query spark.sql("final_query_here") - Flexibility: This approach allows the program to run daily while conditionally executing specific parts based on the day.
- Maintainability: The logic is clear and can be easily modified if requirements change.
Why other options are incorrect:
- Option A: Submitting a feature request is not a practical solution for immediate implementation needs.
- Option C: Running the entire program only on Sundays contradicts the requirement to run the program daily.
- Option D: Restricting table access is not a reliable or appropriate way to control query execution. It would prevent the query from running even if someone tried to execute it manually on other days.
- Option E: Redesigning the data model doesn't solve the conditional execution problem. It might separate data but doesn't address when queries should run.
Best Practice Approach: The most effective solution is to use workflow orchestration tools like Databricks Workflows or Apache Airflow, which can schedule jobs with conditional logic. However, since that's not listed as an option, wrapping in PySpark with Python control flow is the most practical solution among the given choices.
Comments
Loading comments...