pyspark越来越weeknumber
问题描述:
我从datafrme柱获得weeknumber从本月pyspark,对于实例是stuckup考虑我的数据帧作为pyspark越来越weeknumber
WeekID,DateField,WeekNUM
1,01/JAN/2017
2,15/Feb/2017
我的输出应该如下
WeekIDm,DateField,MOF
1,01/JAN/2017,1
2,15/FEB/2017,2
我尝试使用striftime和其他日期功能,我无法做到。
请帮我解决这个问题。
答
您可以结合to_date
和date_format
:
from pyspark.sql.functions import to_date, date_format
df = spark.createDataFrame(
[(1, "01/JAN/2017"), (2, "15/FEB/2017")], ("id", "date"))
df.withColumn("week", date_format(to_date("date", "dd/MMM/yyyy"), "W")).show()
+---+-----------+----+
| id| date|week|
+---+-----------+----+
| 1|01/JAN/2017| 1|
| 2|15/FEB/2017| 3|
+---+-----------+----+
如果你想一周的年请w
替换格式:
date_format(to_date("date", "dd/MMM/yyyy"), "w")