Python/Scrapy:没有回调的收益请求
问题描述:
我想输入getMonthEvents。但不知何故,似乎回调从未执行。有任何想法吗?谢谢:)Python/Scrapy:没有回调的收益请求
from scrapy.selector import Selector
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from scrapy.http import Request
from scrapy.item import Item, Field
class EventItems(Item):
Title = Field()
Link = Field()
Date = Field()
Time = Field()
Place = Field()
Description = Field()
Program=Field()
class SpiderForHSMT(CrawlSpider):
name = 'HMTM'
start_urls = ['http://website.musikhochschule-muenchen.de/de/index.php?Itemid=602&id=565']
rules =(Rule(LinkExtractor(restrict_xpaths=('//div[@id="VER_2013_DISPLAYSEARCHRESULTS"]/table[1]/tr[3]'), tags=('a',), attrs=('href',)), callback = 'parseMonth'),)
def parseMonth(self, response):
request = Request(response.url, callback = self.getMonthEvents)
yield request
def getMonthEvents(self, response):
print(response.url)
答
当你在parseMonth中复制你的请求时,请求被过滤为重复的(see documentation)。将dont_filter=True
添加到您的请求,以便它们不被过滤。
request = Request(response.url, dont_filter=True, callback = self.getMonthEvents)
不错!它的作品:)非常感谢! – Jason