TransUnion interview question

Remove duplicates from spark dataframe without using in built function. Handle null values.