PySpark：数据帧分割成多个DataFrames不使用循环

Question

您好我有一个数据帧，如图 -

ID  X  Y 

1  1234  284 

1  1396  179 

2  8620  178 

3  1620  191 

3  8820  828 

我想这个数据帧分成基于ID的多个DataFrames。因此，在这个例子中会有3个DataFrame。实现它的一种方法是在循环中运行过滤器操作。但是，我想知道它是否可以以更有效的方式完成。

Answer 1

#initialize spark dataframe 
df = sc.parallelize([ (1,1234,282),(1,1396,179),(2,8620,178),(3,1620,191),(3,8820,828) ]).toDF(["ID","X","Y"]) 

#get the list of unique ID values ; there's probably a better way to do this, but this was quick and easy 
listids = [x.asDict().values()[0] for x in df.select("ID").distinct().collect()] 
#create list of dataframes by IDs 
dfArray = [df.where(df.ID == x) for x in listids] 

dfArray[0].show() 
+---+----+---+ 
| ID| X| Y| 
+---+----+---+ 
| 1|1234|282| 
| 1|1396|179| 
+---+----+---+ 
dfArray[1].show() 
+---+----+---+ 
| ID| X| Y| 
+---+----+---+ 
| 2|8620|178| 
+---+----+---+ 

dfArray[2].show() 
+---+----+---+ 
| ID| X| Y| 
+---+----+---+ 
| 3|1620|191| 
| 3|8820|828| 
+---+----+---+