pyspark - Spark UDF functions

Question

I writing a udf function to convert epoch time into timestamp. I am aware that there is a build in functions in pyspark to do that.However in order to try my first udf , I wanted to try using udf.

Below is the code I am try in pyspark

import  pyspark.sql.types as t
import  pyspark.sql.functions as F
get_timestamp = F.udf(lambda x: F.to_timestamp(x))
df = spark.createDataFrame([('1542241826796',)],['t'])
df.show()
+-------------+
|            t|
+-------------+
|1542241826796|
+-------------+

df = df.withColumn("timestamp_ts", get_timestamp(F.col("t")))
df.show()

However I am getting below error which I am not able to resolve

1/01/08 14:41:18 ERROR Executor: Exception in task 2.0 in stage 54.0 (TID 151)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 605, in main
    process()
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 597, in process
    serializer.dump_stream(out_iter, outfile)
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 223, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 141, in dump_stream
    for obj in iterator:
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 212, in _batched
    for item in iterator:
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 450, in mapper
    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 450, in <genexpr>
    result = tuple(f(*[a[o] for o in arg_offsets]) for (arg_offsets, f) in udfs)
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 90, in <lambda>
    return lambda *a: f(*a)
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/util.py", line 107, in wrapper
    return f(*args, **kwargs)
  File "<stdin>", line 1, in <lambda>
  File "/home/deepak/Downloads/spark-3.0.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/functions.py", line 1190, in to_timestamp
    jc = sc._jvm.functions.to_timestamp(_to_java_column(col))
AttributeError: 'NoneType' object has no attribute '_jvm'

Answer 1

等待大神答复