
Ultimate access to all questions.
In the development of a machine learning model for predicting customer churn, you are tasked with evaluating whether a new feature significantly improves the model's performance compared to the baseline model without the feature. The dataset is large, and the team is concerned about both Type I and Type II errors. Which of the following approaches is the MOST appropriate for this scenario, and why? Choose the best option.
A
Use a t-test to compare the means of the model's performance metrics with and without the new feature, as it directly tests for differences in means.
B
Apply cross-validation to ensure the model's performance is consistent across different subsets of the data, but do not perform any statistical tests.
C
Conduct a statistical hypothesis test, such as a paired t-test or McNemar's test, to evaluate the significance of the performance difference between the two models, considering the risk of errors.
D
Optimize the model's hyperparameters to maximize performance, assuming that any improvement is significant without statistical validation.
E
Both A and C are correct because they each address different aspects of the problem: A tests for mean differences, while C evaluates significance considering error risks.