python - 使用BeautifulSoup网站刮ajax网站

Question

我想刮电子商务网站，使用ajax调用来加载其下一页。

我可以抓取第1页上的数据，但是当我将第1页滚动到底部时，第2页会通过ajax调用自动加载。

我的代码：

from bs4 import BeautifulSoup as soup 
from urllib.request import urlopen as ureq 
my_url='http://www.shopclues.com/mobiles-smartphones.html' 
page=ureq(my_url).read() 
page_soup=soup(page,"html.parser") 
containers=page_soup.findAll("div",{"class":"column col3"}) 
for container in containers: 
    name=container.h3.text 
    price=container.find("span",{'class':'p_price'}).text 
    print("Name : "+name.replace(","," ")) 
    print("Price : "+price) 
for i in range(2,7): 
    my_url="http://www.shopclues.com/ajaxCall/moreProducts?catId=1431&filters=&pageType=c&brandName=&start="+str(36*(i-1))+"&columns=4&fl_cal=1&page="+str(i) 
    page=ureq(my_url).read() 
    print(page) 
    page_soup=soup(page,"html.parser") 
    containers=page_soup.findAll("div",{"class":"column col3"}) 
    for container in containers: 
     name=container.h3.text 
     price=container.find("span",{'class':'p_price'}).text 
     print("Name : "+name.replace(","," ")) 
     print("Price : "+price) 

我已经印刷由ureq读取AJAX页面知道我是否能够打开AJAX页面，我得到了一个输出为：

B”'是输出： 打印（页）

请为我提供一个解决方案来刮取剩余的数据。

Answer 1

from selenium import webdriver 
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
from bs4 import BeautifulSoup as soup 
from urllib2 import urlopen as ureq 
import random 
import time 

chrome_options = webdriver.ChromeOptions() 
prefs = {"profile.default_content_setting_values.notifications": 2} 
chrome_options.add_experimental_option("prefs", prefs) 

# A randomizer for the delay 
seconds = 5 + (random.random() * 5) 
# create a new Chrome session 
driver = webdriver.Chrome(chrome_options=chrome_options) 
driver.implicitly_wait(30) 
# driver.maximize_window() 

# navigate to the application home page 
driver.get("http://www.shopclues.com/mobiles-smartphones.html") 
time.sleep(seconds) 
time.sleep(seconds) 
# Add more to range for more phones 
for i in range(1): 
    element = driver.find_element_by_id("moreProduct") 
    driver.execute_script("arguments[0].click();", element) 
    time.sleep(seconds) 
    time.sleep(seconds) 
html = driver.page_source 
page_soup = soup(html, "html.parser") 
containers = page_soup.findAll("div", {"class": "column col3"}) 
for container in containers: 
# Add error handling 
    try: 
     name = container.h3.text 
     price = container.find("span", {'class': 'p_price'}).text 
     print("Name : " + name.replace(",", " ")) 
     print("Price : " + price) 
    except AttributeError: 
     continue 
driver.quit() 

我用硒来加载网站，并点击按钮加载更多的结果。然后拿出生成的html并输入你的代码。