There are nulls in your dataframe, which causes the UDF to fail. However, there is no need to use an UDF here. You can just filter the dataframe by comparing the two columns directly.
df1 = df.na.drop(subset=["Column1", "Column2", "Column3", "Column4", "Column5","Column6"])
df1 = df1.withColumn('Column6',df1['Column6'].cast(DoubleType()))
filter1 = df1.filter('Column2 >= Column6')
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…