为什么这个脚本需要这么长时间才能运行？

Question

我有一个包含200,000行的csv文件。我已加载到这个数据帧和想用摊贩与下面的脚本匿名保护：

for i in range(MasterDE1.FirstName.size): 
    MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Female'), ['FirstName','LastName']] = fake.first_name_female(),fake.last_name_female() 
    MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Male'), ['FirstName','LastName']] = fake.first_name_male(),fake.last_name_male() 
    MasterDE1.loc[(MasterDE1["Gender__pc"] == 'Unknown'), ['FirstName','LastName']] = fake.first_name(),fake.last_name() 
    MasterDE1['Name'] = MasterDE1['FirstName'] + ' ' + MasterDE1['LastName'] 
    MasterDE1['EmailAddress'] = 'smithandthunder' + str(i+1) + '@gmail.com' 

它已经从过去的20几分钟内运行（我不认为内核是死的）。

Answer 1

而是在每次迭代更新数据框，你可以首先生成的名称，然后分配：

df = pd.DataFrame({'Gender': np.random.choice(['Female', 'Male', 'Unknown'], p=[0.45, 0.45, 0.1], size=2*10**5), 
        'First Name': np.nan, 'Last Name': np.nan}) 


df.head() 
Out: 
    First Name Gender Last Name 
0   NaN Female  NaN 
1   NaN Male  NaN 
2   NaN Female  NaN 
3   NaN Male  NaN 
4   NaN Male  NaN 

df.shape 
Out: (200000, 3) 

现在下面应完成在几分钟内：

df.loc[df['Gender']=='Female', ('First Name', 'Last Name')] = [(fake.first_name_female(), fake.last_name_female()) for _ in range(df[df['Gender']=='Female'].shape[0])] 

df.loc[df['Gender']=='Male', ('First Name', 'Last Name')] = [(fake.first_name_male(), fake.last_name_male()) for _ in range(df[df['Gender']=='Male'].shape[0])] 

df.loc[df['Gender']=='Unknown', ('First Name', 'Last Name')] = [(fake.first_name(), fake.last_name()) for _ in range(df[df['Gender']=='Unknown'].shape[0])] 

df.head() 
Out: 
    First Name Gender Last Name 
0  Ruth Female  Moore 
1 Christina Female  Jones 
2 Lindsey Female  Davis 
3  Aaron Unknown Watkins 
4  Joshua  Male  Henry 

之后，像df['Name'] = df['First Name'] + ' ' + df['Last Name']这样的东西应该很快。

Answer 2

您可以省略循环：

MasterDE1 = pd.DataFrame({'Gender__pc':['Female','Male','Unknown'], 
         'FirstName':['s','d','f'], 
         'LastName': ['d','f','r']}) 
MasterDE1 = pd.concat([MasterDE1]*3).reset_index(drop=True) 
print (MasterDE1) 
    FirstName Gender__pc LastName 
0   s  Female  d 
1   d  Male  f 
2   f Unknown  r 
3   s  Female  d 
4   d  Male  f 
5   f Unknown  r 
6   s  Female  d 
7   d  Male  f 
8   f Unknown  r 

def f1(): 
    return 'first_name_female' + str(np.random.randint(100)) 
def f2(): 
    return 'last_name_female' + str(np.random.randint(100)) 

maskfem = (MasterDE1["Gender__pc"] == 'Female') 
a = pd.Series(((np.arange(len(MasterDE1.index))) + 1).astype(str)) 

MasterDE1.loc[maskfem, 'FirstName'] = [f1() for x in np.arange(maskfem.sum())] 
MasterDE1.loc[maskfem, 'LastName'] = [f2() for x in np.arange(maskfem.sum())] 

MasterDE1['Name'] = MasterDE1['FirstName'] + ' ' + MasterDE1['LastName'] 
MasterDE1['EmailAddress'] = 'smithandthunder' + a + '@gmail.com' 

print (MasterDE1) 
      FirstName Gender__pc   LastName \ 
0 first_name_female70  Female last_name_female64 
1     d  Male     f 
2     f Unknown     r 
3 first_name_female6  Female last_name_female67 
4     d  Male     f 
5     f Unknown     r 
6 first_name_female59  Female last_name_female99 
7     d  Male     f 
8     f Unknown     r 

            Name    EmailAddress 
0 first_name_female70 last_name_female64 [email protected] 
1          d f [email protected] 
2          f r [email protected] 
3 first_name_female6 last_name_female67 [email protected] 
4          d f [email protected] 
5          f r [email protected] 
6 first_name_female59 last_name_female99 [email protected] 
7          d f [email protected] 
8          f r [email protected] 

Answer 3

我不知道确切地告诉你它为什么采取这一长，但它可能是因为该文件的大小。

但是，你能找到一种方法来监视循环知道它是否仍在工作：

signal = 0 

for i in range(0,200000): 
    .... 
    # something going on in the loop 
    .... 
    # signal the loop 
    signal += 1 
    if signal == 50000 or signal == 100000 or signal == 150000: 
     print('It\'s still going!') 
    elif signal > 200000: 
     print('It\'s over 200000 already!') 
     break # or you can raise an error instead of break (raise RuntimeError)