Python爬虫 - html.fromstring
问题描述:
我想用这段代码解析网页。Python爬虫 - html.fromstring
ac = requests.get('link....')
html_text = ac.text
lx = html.fromstring(html_text)
当我运行这段代码我收到此错误
Traceback (most recent call last):
File "Crawler.py", line 197, in <module>
cnx.close()
File "Crawler.py", line 46, in RequestPage
lx = html.fromstring(html_text)
File "C:\Python27\lib\site-packages\lxml\html\__init__.py", line 867, in fromstring
doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
File "C:\Python27\lib\site-packages\lxml\html\__init__.py", line 752, in document_fromstring
value = etree.fromstring(html, parser, **kw)
File "src\lxml\lxml.etree.pyx", line 3213, in lxml.etree.fromstring (src\lxml\lxml.etree.c:76696)
File "src\lxml\parser.pxi", line 1830, in lxml.etree._parseMemoryDocument (src\lxml\lxml.etree.c:115101)
File "src\lxml\parser.pxi", line 1711, in lxml.etree._parseDoc (src\lxml\lxml.etree.c:113677)
File "src\lxml\parser.pxi", line 1051, in lxml.etree._BaseParser._parseUnicodeDoc (src\lxml\lxml.etree.c:107847)
File "src\lxml\parser.pxi", line 584, in lxml.etree._ParserContext._handleParseResultDoc (src\lxml\lxml.etree.c:102150)
File "src\lxml\parser.pxi", line 694, in lxml.etree._handleParseResult (src\lxml\lxml.etree.c:103800)
File "src\lxml\parser.pxi", line 633, in lxml.etree._raiseParseError (src\lxml\lxml.etree.c:102888)
lxml.etree.XMLSyntaxError: line 1843: Tag ie:menuitem invalid
我发现的HTML标签,它会导致错误:
<ie:menuitem id="MSOMenu_Help" iconsrc="/_layouts/images/HelpIcon.gif" onmenuclick="MSOWebPartPage_SetNewWindowLocation(MenuWebPart.getAttribute('helpLink'), MenuWebPart.getAttribute('helpMode'))" text="Help" type="option" style="display:none">
</ie:menuitem>
答
您发现HTML标记这种情况下,错误,但你解决了吗?如果不试试这个:
ac = requests.get('link....') lx = html.fromstring(ac.content) valueOfHTMLTag = lx.xpath('//TAG[@class/id="Name"]/text()')
:
- TAG在你想要得到的价值标签。
- 选择标签
- 标签的ID /类名的class和id
这将返回与正确的类/ ID该标签的值的数组。
希望这会有所帮助!
您可能需要定义一个自定义元素,以便lxml可以理解Sharepoint魔法:http://lxml.de/element_classes.html –
或者使用BeautifulSoup模块作为替代方案,它知道如何处理命名空间元素。 – DeepSpace