In the context of PySpark, which function is designed to generate a new DataFrame by eliminating duplicate rows, with the option to consider only specific columns for identifying duplicates?