scrapy没有JSON对象可以解码
问题描述:
我试图用scrapy来提取网页中的数据......,所有的数据是内部一个javascriptscrapy没有JSON对象可以解码
<script type="text/javascript">
// Globals
var ANUNTURI = [ { "ID": "2750801", "Data": "Azi 11:16", "Zile_piata": "146", "Zona": "Andronache", "Nr_Camere": "2", "suprafu": "65", "Pret": "62.000 EUR",
"Citit": "0", "Tip_teren": "-", "Etaj": "3/3", "supraft": "-",
"frontStradal": "-", "Etichete": "", "ArePoze": "7", "Tip_spatiu": "-" },
and so on... ]
;\r\n var ID_CAUTARE = 0;\r\n var CATEG = 3;\r\n
var TRANZ = 2;\r\n
var SORTARE = "";\r\n
var ID_AGENT = "3012";\r\n
var ID_LOCALITATE = \'13822\';\r\n
var ID_JUDET = \'10\';\r\n
var CRITERIU_FILTRU = \'\';\r\n // judet_schimbat = "";\r\n\r\n $(\'form[name="anunturi"] input[name="sort"]\').val(SORTARE);\r\n\r\n', u"\r\n\r\n $(function(){\r\n\r\n
var setTagValue = ' 0 ';\r\n
var comboTitle = [];\r\n\r\n $('#combo_etichete').mpCombo({\r\n cls: 'mpCombo etichete',\r\n header_default_text: 'Indiferent',\r\n interval_from_text: ' Peste ', \r\n
interval_to_text: ' Pana la ', \r\n interval_between_text: ' si ', \r\n combo_width: '162px', \r\n menu_width: '160px',\r\n onSelect: function() { // trigger click daca e inchisa cautarea avansata\r\n if($('#cautare_avansata').is(':hidden')) {\r\n $('a#filtreaza').trigger('click');\r\n }\r\n }\r\n\r\n });\r\n \r\n $('#combo_etichete').mpCombo({'setval': setTagValue});\r\n comboTitle.push($('#combo_etichete').mpCombo('gettitle')); \r\n\r\n if (comboTitle.length > 0) {\r\n $('#combo_etichete dt a').text(comboTitle.join(', ')); \r\n }\r\n\r\n });\r\n\r\n\r\n", u'\r\nvar gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");\r\ndocument.write(unescape("%3Cscript src=\'" + gaJsHost + "google-analytics.com/ga.js\' type=\'text/javascript\'%3E%3C/script%3E"));\r\n']
</script>
当我使用
json.loads(response.xpath("//script[2]/text").extract())
它给我的错误
No Json object could be decoded
我只需要得到第一个VAR ANUNTURI和它里面的一切,并把它们放在mysql中。
UPDATE
我也试过这样:
var = re.compile(r"var ANUNTURI= ({.*?});", re.MULTILINE | re.DOTALL)
json.loads(response.xpath("//script[2][contains(., 'var ANUNTURI')]/text()").re(var))
和错误,我得到的是这样的:
TypeError: expected string or buffer
,然后我尝试这样做:
json.loads("".join(response.xpath("//script[2][contains(., 'var ANUNTURI')]/text()").re(var)))
,我也得到:
NO JSON object could be decoded
答
这是一个可能的方式来提取数据,但当前提出的代码,它是很难说,如果变量嵌入JSON或Javacript。以微妙的方式使用Javascript可能是JSON对象的超集。
data = """/ Globals
var ANUNTURI = [ { "ID": "2750801", "Data": "Azi 11:16", "Zile_piata": "146", "Zona": "Andronache", "Nr_Camere": "2", "suprafu": "65", "Pret": "62.000 EUR",
"Citit": "0", "Tip_teren": "-", "Etaj": "3/3", "supraft": "-",
"frontStradal": "-", "Etichete": "", "ArePoze": "7", "Tip_spatiu": "-" },]
;\r\n var ID_CAUTARE = 0;\r\n var CATEG = 3;\r\n
var TRANZ = 2;\r\n
var SORTARE = "";\r\n
var ID_AGENT = "3012";\r\n
var ID_LOCALITATE = \'13822\';\r\n
var ID_JUDET = \'10\';\r\n
var CRITERIU_FILTRU = \'\';\r\n // judet_schimbat = "";\r\n\r\n $(\'form[name="anunturi"] input[name="sort"]\').val(SORTARE);\r\n\r\n', u"\r\n\r\n $(function(){\r\n\r\n
var setTagValue = ' 0 ';\r\n
var comboTitle = [];\r\n\r\n $('#combo_etichete').mpCombo({\r\n cls: 'mpCombo etichete',\r\n header_default_text: 'Indiferent',\r\n interval_from_text: ' Peste ', \r\n
interval_to_text: ' Pana la ', \r\n interval_between_text: ' si ', \r\n combo_width: '162px', \r\n menu_width: '160px',\r\n onSelect: function() { // trigger click daca e inchisa cautarea avansata\r\n if($('#cautare_avansata').is(':hidden')) {\r\n $('a#filtreaza').trigger('click');\r\n }\r\n }\r\n\r\n });\r\n \r\n $('#combo_etichete').mpCombo({'setval': setTagValue});\r\n comboTitle.push($('#combo_etichete').mpCombo('gettitle')); \r\n\r\n if (comboTitle.length > 0) {\r\n $('#combo_etichete dt a').text(comboTitle.join(', ')); \r\n }\r\n\r\n });\r\n\r\n\r\n", u'\r\nvar gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");\r\ndocument.write(unescape("%3Cscript src=\'" + gaJsHost + "google-analytics.com/ga.js\' type=\'text/javascript\'%3E%3C/script%3E"));\r\n'
"""
from json import loads
from pprint import PrettyPrinter
lines = data.split("\r\n")
anunturi_json = lines[0].split("=")[1]
print anunturi_json
val = loads(anunturi_json)
pp = PrettyPrinter(indent=4)
pp.pprint(val)
因此没有办法提取该数据? – Omega
正则表达式? –
你甚至从该JS代码中提取该JSON吗? – Umair