如何更改pyspark数据帧列数据类型?
答
您必须用新模式替换列。 ArrayType采用两个参数elementType和containsNull。
from pyspark.sql.types import *
from pyspark.sql.functions import udf
x = [("a",["b","c","d","e"]),("g",["h","h","d","e"])]
schema = StructType([StructField("key",StringType(), nullable=True),
StructField("values", ArrayType(StringType(), containsNull=False))])
df = spark.createDataFrame(x,schema = schema)
df.printSchema()
new_schema = ArrayType(StringType(), containsNull=True)
udf_foo = udf(lambda x:x, new_schema)
df.withColumn("values",udf_foo("values")).printSchema()
root
|-- key: string (nullable = true)
|-- values: array (nullable = true)
| |-- element: string (containsNull = false)
root
|-- key: string (nullable = true)
|-- values: array (nullable = true)
| |-- element: string (containsNull = true)
谢谢@ashwinds - 它帮助 – user2763088