Ultimate access to all questions.
In the context of optimizing a Spark application for performance and maintainability, you are tasked with identifying the correct locations and accessibility of various types of functions within the application. Consider the following scenario: Your team is developing a Spark application that processes large datasets. The application utilizes built-in functions, user-defined functions (UDFs), temporary functions, and attempts to use permanent functions. Understanding where each type of function resides and how they can be accessed is crucial for debugging and optimizing the application. Given the constraints of cost, compliance, and scalability, which of the following statements accurately describe the location and accessibility of these functions? (Choose two correct options.)
A
Built-in functions are part of the Spark SQL library and are accessible throughout the Spark environment without being defined in the application code, but they are not stored in the Spark SQL catalog.
B
User-defined functions (UDFs) are defined within the application code, either as a Scala function, a Python function, or by using a custom class, and are not part of the Spark SQL library, making them specific to the application.
C
Temporary functions are created during the runtime of the application and are stored in the Spark SQL catalog, making them accessible only during the session they are created in, which is ideal for temporary data processing tasks.
D
Permanent functions are a standard feature in Spark SQL and are stored in the Spark SQL configuration file, allowing them to be accessed across different sessions, which contradicts the temporary nature of Spark's session-based architecture.
E
Both built-in functions and temporary functions are stored in the Spark SQL catalog, but built-in functions are accessible across sessions while temporary functions are not, which is a common misconception about Spark's function management.