并行运行不同函数并按特定顺序获取返回值
问题描述:
我想并行运行不同的函数(foo,bar和baz),然后按照它们的顺序获取每个函数返回的值调用。并行运行不同函数并按特定顺序获取返回值
我尝试了类似下面的代码,但出于某种原因,它在Windows上永远循环。使用例如multiprocessing
可以实现吗?请注意,这些函数预期将参数作为同一事物(即数据)。
注意:下面的代码是针对Python 3的,我想要一个等效的Python 2.7,因为它是我需要使用的。
from concurrent.futures import ProcessPoolExecutor
from operator import itemgetter
def foo(data):
return "foo"
def bar(data):
return "bar"
def baz(data):
return "baz"
work = [foo, bar, baz]
data = [1,2,3,4,5]
results = []
with ProcessPoolExecutor(max_workers=4) as pool:
for i, work_item in enumerate(work):
future = pool.submit(work_item, data)
def callback(ret):
results.append((i, ret.result()))
future.add_done_callback(callback)
results.sort(key=itemgetter(0))
print(results)
答
我改变了你的回调方法,实现了你想要的,我想。检查结果:
...
with ProcessPoolExecutor(max_workers=4) as pool:
for i, work_item in enumerate(work):
future = pool.submit(work_item, data)
def callback(i):
def wrapper(ret):
print(ret, i)
results.append((i, ret.result()))
return wrapper
future.add_done_callback(callback(i))
...
,结果如下:
<Future at 0x1057e4d30 state=finished returned str> 0
<Future at 0x1057e4a58 state=finished returned str> 2
<Future at 0x105779240 state=finished returned str> 1
[(0, 'foo'), (1, 'bar'), (2, 'baz')]
我运行与python3 Mac OS X中的代码。我希望它有帮助。
为蟒蛇2.7,你可以尝试这样的事:
from multiprocessing import Process, Manager
def foo(data, i, results):
results[i] = "foo"
def bar(data, i, results):
results[i] = "bar"
def baz(data, i, results):
results[i] = "baz"
work = [foo, bar, baz]
data = [1,2,3,4,5]
processes = []
results = Manager().dict()
for i, w in enumerate(work):
p = Process(target=w, args=(data, i, results))
processes.append(p)
p.start()
for p in processes:
p.join()
print results
结果是:
{0: 'foo', 1: 'bar', 2: 'baz'}
How can I recover the return value of a function passed to multiprocessing.Process?
我忘了提,给定的代码就是Python 3(和在Windows上不起作用),但我想要一个与Python 2.7一起工作的代码,因为它是我需要使用的。 – eLearner
然后看看这个页面:https://pymotw.com/2/multiprocessing/communication.html – Muatik
我更新了python 2.7的答案 – Muatik