【发布时间】：2017-08-02 05:38:00
【问题描述】：
我正在编写一个程序来根据搜索词从网站上抓取文章的 URL。目前我只能从第一页抓取文章 URL。
我可以访问加载按钮并加载其他文章，但无法抓取它们的 URL。
为了更清晰，我将整个代码分成 2 个单独的模块。我试图将它们组合成一个模块，但输出只是重复显示第一页的文章 URL。我需要帮助将这两个模块组合成一个工作模块。
此代码根据搜索词从第一页抓取文章 URL
from pprint import pprint
import requests
import lxml
import csv
import urllib2
from bs4 import BeautifulSoup
####Function returns list of URLs based on search key#####
def get_url_for_search_key(search_key):
base_url = 'http://www.marketing-interactive.com/'
response = requests.get(base_url + '?s=' + search_key)
soup = BeautifulSoup(response.content, "lxml")
newlinks = []
soup = BeautifulSoup(response.text, "lxml")
results = soup.findAll('a', {'rel': 'bookmark'})
return [url['href'] for url in soup.findAll('a', {'rel': 'bookmark'})]
pprint(get_url_for_search_key('digital marketing'))
### Scraped Links written into csv file(under a single column) ###
with open('ctp_output.csv', 'w+') as f:
f.seek(0)
f.write('\n'.join(get_url_for_search_key('digital marketing')))
### Scraped Links read from csv file and respective information is scraped and written into text file ###
with open('ctp_output.csv', 'rb') as f1:
f1.seek(0)
reader = csv.reader(f1)
for line in reader:
url = line[0]
soup = BeautifulSoup(urllib2.urlopen(url), "lxml")
with open('ctp_output.txt', 'a+') as f2:
for tag in soup.find_all('p'):
f2.write(tag.text.encode('utf-8') + '\n')
此代码自动访问“加载更多”按钮并加载其余文章
from retry import retry
from explicit import waiter
from selenium import webdriver
from selenium.common.exceptions import StaleElementReferenceException
@retry(StaleElementReferenceException, tries=10, delay=0.5)
def click_more(driver):
waiter.find_element(driver, 'div.loadm').click()
driver = webdriver.Chrome()
try:
driver.get("http://www.marketing-interactive.com/?s=digital+marketing")
while True:
click_more(driver)
【问题讨论】：
标签： python function selenium web-scraping beautifulsoup