的Python:多处理未完成作业
问题描述:
我使用python 2.7使用多::游泳池并行运行的Python:多处理未完成作业
我已经简化下面的例子中工作,但这里是它的主要依据。
它将使用apply_async()
函数为我的字典中的每个人创建一个文件。但是,当我检查文件是否正确创建时,我注意到有时文件没有创建。
现在,我想我已经做错了我如何用多::游泳池
有什么建议?
import os
from multiprocessing import Pool
def outputFile(person):
ofh=open(person+'.txt','w')
ofh.write('test\n')
ofh.close()
pool = Pool(processes=4)
for person in person_dict:
pool.apply_async(outputFile,args(person))
pool.close()
pool.join()
for person in person_dict:
print os.path.isfile(person+'.txt')
True
True
False
True
答
难道这是关系到person_dict的内容是什么?
我修改了你的代码并运行了好几次。他们都产生了预期的结果。
下面是我修改和测试代码:
import os
from multiprocessing import Pool
def outputfile(person):
with open(person+'.txt','w') as ofh:
ofh.write('test\n')
person_dict = {'a': 'a', 'b': 'b', 'c':'c', 'd':'d'}
pool = Pool(processes=4)
for person in person_dict:
pool.apply_async(outputfile, (person))
pool.close()
pool.join()
for person in person_dict:
print(os.path.isfile(person+'.txt'))
答
如果不赶在子流程异常,自己打印出来,你不会看到他们。下面的程序不产生输出:
import os
from multiprocessing import Pool
def outputFile(person):
raise Exception("An exception")
pool = Pool(processes=4)
for person in range(100):
pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()
你需要捕获所有异常和手动打印回溯:
import os
from multiprocessing import Pool, Lock
import traceback
print_lock = Lock()
def outputFile(person):
try:
raise Exception("An exception")
except:
with print_lock:
print "%s: An exception occurred" % person
print traceback.format_exc()
pool = Pool(processes=4)
for person in range(100):
args = (person, print_lock)
pool.apply_async(outputFile, args=(person,))
pool.close()
pool.join()
输出
0: An exception occurred
Traceback (most recent call last):
File "person.py", line 9, in outputFile
raise Exception("An exception")
Exception: An exception
1: An exception occurred
Traceback (most recent call last):
File "person.py", line 9, in outputFile
raise Exception("An exception")
Exception: An exception
...
99: An exception occurred
Traceback (most recent call last):
File "person.py", line 9, in outputFile
raise Exception("An exception")
Exception: An exception
注:的print_lock
是用于保持交错输出。
我不确定。通常我会说99%的时间没有出现错误。但偶尔该过程会下降,并且不会形成输出文件。 – DolphinGenomePyramids
它可能是IO异常或子进程上的一些运行时错误,它们在调用进程中不显示。 – neurite