Python BeautifulSoup在XML标记中引用空格的标记

Question

对于我格式不当的帖子和书面代码的道歉，第一篇文章！我确信这是一个简单的修复，但似乎无法弄清楚。

问题1：我正在为Eve Online API编写一个XML刮取器。我需要在其中包含空格的HTML标记（type id="X"）上迭代*BeautifulSoup*。我想遍历XML中的项目ID标记（本例中为2）。我不确定*Data.iter*是否正确。我知道打印Data.buy将在XML中找到第一个购买标签，并打印这些孩子，但无法找到打印第一个数据的相同方法。 'type id="x"'。

问题2：任何关于如何继续刮擦过程的方向都是值得欢迎的。我正在考虑将不同物品ID和买入/卖出订单的不同存储空间导出到某种存储（认为是CSV文件，但不是正面）。

import requests #Used to service API connection 
from lxml import html #Used to parse XML 
from bs4 import BeautifulSoup #Used to read XML table on webpage 

ItemTypeID1 = 34 
ItemTypeID2 = 35 
RegionID = 10000002 

Webpage = requests.get('http://api.eve-central.com/api/marketstat?typeid=%i&typeid=%i&regionlimit=%i' % (ItemTypeID1, ItemTypeID2, RegionID)) 
#Check if page is up 
if Webpage.status_code == 200: 
#Convert webpage to %Data 
     Data = BeautifulSoup(Webpage.text, 'lxml') 

     #Problem line 
     for item in Data.iter('type id='): 
      print 'something' 

http://api.eve-central.com/api/marketstat?typeid=34&typeid=35&regionlimit=10000002

<?xml version='1.0' encoding='utf-8'?> 
<evec_api version="2.0" method="marketstat_xml"> 
     <marketstat><type id="34"> 
      <buy><volume>21648183554</volume><avg>5.34</avg><max>5.78</max><min>2.00</min><stddev>0.83</stddev><median>5.52</median><percentile>5.58</percentile></buy> 
      <sell><volume>15987043271</volume><avg>6.20</avg><max>15.00</max><min>5.58</min><stddev>1.38</stddev><median>6.00</median><percentile>5.85</percentile></sell> 
      <all><volume>37635226825</volume><avg>5.71</avg><max>15.00</max><min>2.00</min><stddev>1.40</stddev><median>5.54</median><percentile>4.16</percentile></all> 
     </type><type id="35"> 
      <buy><volume>4163954948</volume><avg>7.26</avg><max>9.39</max><min>4.02</min><stddev>1.60</stddev><median>6.48</median><percentile>9.24</percentile></buy> 
      <sell><volume>10154431073</volume><avg>11.84</avg><max>17.41</max><min>9.00</min><stddev>1.80</stddev><median>10.59</median><percentile>9.46</percentile></sell> 
      <all><volume>14318386021</volume><avg>10.51</avg><max>17.41</max><min>4.02</min><stddev>2.21</stddev><median>10.00</median><percentile>5.87</percentile></all> 
     </type></marketstat> 
    </evec_api> 

Answer 1

type id=不是一个标签。元素的标签名称是type，而id是该元素的属性。

for item in Data.find_all('type'): 
    print item.get('id') 

对于您参考该代码的URL将输出：

 
34 
35 

的代码只是找到与标签名称“类型”的所有元素，并显示找到的每个标签的id属性。

您可以访问包含在嵌套buy和sell标签中的数据：

for item in Data.find_all('type'): 
    print item.get('id') 
    volume = item.buy.volume.text 
    avg = item.buy.volume.text 
    # etc. 

它展示了如何在包含在数量和平均标签为每个项目中的数据得到。

---

还有一个JSON API访问，这可能是更容易使用，特别是使用requests模块时：

import requests 

url = 'http://api.eve-central.com/api/marketstat/json' # the JSON endpoint 
params = {'typeid': (34, 35), 'RegionID': 10000002} 
r = requests.get(url, params=params) 
data = r.json() 

这给了你Python字典的名单一起工作：

for type_ in data: 
    print '{}: volume = {}, avg = {}'.format(type_['buy']['forQuery']['types'][0], type_['buy']['volume'], type_['buy']['avg']) 

 
34: volume = 110242267166, avg = 4.29419161677 
35: volume = 40908217125, avg = 6.71507628294 

尽管从JSON响应中获取类型id有点awkwar d与XML相比较。

Answer 2

你有几个问题，你似乎是混合了BS4和LXML语法，ITER是lxml的方法，型是节点和ID是节点的属性。

由于您使用LXML忘掉BS4和得到的数据是相当简单：

from lxml import html # Used to parse XML 

ItemTypeID1 = 34 
ItemTypeID2 = 35 
RegionID = 10000002 

webpage = requests.get('http://api.eve-central.com/api/marketstat?typeid=%i&typeid=%i&regionlimit=%i' % (
ItemTypeID1, ItemTypeID2, RegionID)) 

if webpage.status_code == 200: 
    data = html.fromstring(webpage.content) 
    for item in data.iter('type'): 
     print(item.get("id")) # print node attribute id value. 

     buy_dict = {node.tag: node.text for node in item.xpath("buy/*")} 
     sell_dict = {node.tag: node.text for node in item.xpath("sell/*")} 
     print(buy_dict) 
     print(sell_dict) 

这会给你：

34 
{'min': '2.00', 'max': '5.81', 'median': '5.54', 'volume': '22518685011', 'percentile': '5.72', 'stddev': '0.80', 'avg': '5.37'} 
{'min': '5.57', 'max': '15.00', 'median': '6.00', 'volume': '14953114542', 'percentile': '5.78', 'stddev': '1.38', 'avg': '6.21'} 
35 
{'min': '4.02', 'max': '9.20', 'median': '6.65', 'volume': '4170891614', 'percentile': '9.20', 'stddev': '1.59', 'avg': '7.25'} 
{'min': '8.90', 'max': '17.41', 'median': '10.59', 'volume': '10089810619', 'percentile': '9.44', 'stddev': '1.84', 'avg': '11.83'} 

然后写入到CSV：

import requests # Used to service API connection 
from lxml import html # Used to parse XML 
from csv import DictWriter 

ItemTypeID1 = 34 
ItemTypeID2 = 35 
RegionID = 10000002 

webpage = requests.get('http://api.eve-central.com/api/marketstat?typeid=%i&typeid=%i&regionlimit=%i' % (
ItemTypeID1, ItemTypeID2, RegionID)) 

if webpage.status_code == 200: 
    with open("data.csv", "w") as f: 
     wr = DictWriter(f, fieldnames=["buy/sell", 'volume', 'min', 'max', 'median', 'percentile', 'stddev', 'avg']) 
     wr.writeheader() 
     data = html.fromstring(webpage.content) 
     for item in data.iter('type'): 
      buy_dict = {node.tag: node.text for node in item.xpath("buy/*")} 
      buy_dict["buy/sell"] = "buy" 
      sell_dict = {node.tag: node.text for node in item.xpath("sell/*")} 
      sell_dict["buy/sell"] = "sell" 
      wr.writerows([buy_dict, sell_dict]) 

这会给你：

buy/sell,volume,min,max,median,percentile,stddev,avg 
buy,22518685011,2.00,5.81,5.54,5.72,0.80,5.37 
sell,14953114542,5.57,15.00,6.00,5.78,1.38,6.21 
buy,4170891614,4.02,9.20,6.65,9.20,1.59,7.25 
sell,10089810619,8.90,17.41,10.59,9.44,1.84,11.83