将scrapy导入json时遇到问题
问题描述:
我试图从Craigslist中取出一些信息并将其存储在JSON文件中,但信息存储有点不对。而不是有一个[标题,链接,位置,时间]数组,我得到一个所有标题的数组,所有的链接,等等。我的标题是错的还是for循环本身是错误的?将scrapy导入json时遇到问题
from scrapy.spiders import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.selector import Selector
from craigslist_sample.items import CraigslistSampleItem
class MySpider(BaseSpider):
name = "craig"
allowed_domains = ["craigslist.org"]
start_urls = ["https://pittsburgh.craigslist.org/search/ccc"]
def parse(self, response):
titles = response.selector.xpath("//p[@class='row']")
items = []
for titles in titles:
item = CraigslistSampleItem()
item["title"] = titles.xpath("//span[@id='titletextonly']").extract()
item["link"] = titles.xpath("a/@href").extract()
item["location"] = titles.xpath("//small").extract()
item["time"] = titles.xpath('//time').extract()
items.append(item)
return items
答
这是因为你的内部xpaths匹配从树根开始的元素。相反,您需要强制他们在每个项目的上下文中通过预先加点来工作:
for title in titles:
item = CraigslistSampleItem()
item["title"] = title.xpath(".//span[@id='titletextonly']").extract()
item["link"] = title.xpath("a/@href").extract()
item["location"] = title.xpath(".//small").extract()
item["time"] = title.xpath('.//time').extract()
yield item