多线程刮Html和安全保存到一个文件

Question

我想从多个线程中给定的网址（5线程中的示例） 刮标题并将它们保存到一个文本文件。如何做到这一点，以及如何确保我安全地将输出保存到一个文件？

这是我的代码：

import csv 
import requests 
requests.packages.urllib3.disable_warnings() 

urls = [] 

with open('Input.csv') as csvDataFile: 
    csvReader = csv.reader(csvDataFile) 
    for row in csvReader: 
     urls.append(row[1]) 

def find_between(s, first, last): 
    try: 
     start = s.index(first) + len(first) 
     end = s.index(last, start) 
     return s[start:end] 
    except ValueError: 
     return "" 

def get_title(url): 
    try: 
     r = requests.get(url) 
     html_content = r.text.encode('UTF-8') 
     title = find_between(html_content , "<title>", "</title>") 
     return title 
    except: 
     return "" 

for url in urls: 
    f = open('myfile.txt', 'a') 
    f.write(get_title(url) + '\n') 
    f.close() 

Answer 1

尝试利用期货
1.创建池
2.透过功能和参数，从功能

import csv 
from concurrent import futures 
pool = futures.ThreadPoolExecutor(5) 
workers = [pool.sumbit(get_title,url) for url in urls] 
while not all(worker.done() for worker in workers): 
    pass 
with open(file) as f: 
    w = csv.writer(f) 
    w.writerows([[worker.result()] for worker in workers])