如果表 beautifulsoup 和 pandas 中不存在类，则停止抓取 url

Question

我正在使用 csv 文件中的 url 列表从 html 表中抓取和提取数据。当表格中不存在“style3”时，我想停止浏览网址。 我创建了一个函数，如果它不存在，它将返回 false，但我对如何实际实现它感到困惑。

任何关于解决方案或文献方向的建议都会有很大帮助，因为我无法在这里找到任何东西来帮助我解决这个问题。

我已经包含了 1 个带有 'style3' 的 url 和 1 个没有。感谢您的任何帮助。

http://www.wvlabor.com/new_searches/contractor_RESULTS.cfm?wvnumber=WV057808&contractor_name=&dba=&city_name=&County=&Submit3=Search+Contractors http://www.wvlabor.com/new_searches/contractor_RESULTS.cfm?wvnumber=WV057924&contractor_name=&dba=&city_name=&County=&Submit3=Search+Contractors

import csv
from urllib.request import urlopen
import pandas as pd
from bs4 import BeautifulSoup as BS

def license_exists(soup):
    contents = []
    with open('WV_urls.csv','r') as csvf:
        urls = csv.reader(csvf)
        for url in urls:
            if soup(class_='style3'):
                return True
            else:
                return False

contents = []
more = True
while more:
    df  = pd.DataFrame(columns=['WV Number', 'Company', 'DBA', 'Address', 'City', 'State', 'Zip','County', 'Phone', 'Classification*', 'Expires']) #initialize the data frame with columns
    with open('WV_urls.csv','r') as csvf: # Open file in read mode
        urls = csv.reader(csvf)
        for url in urls:
            contents.append(url) # Add each url to list contents
        for url in contents:  # Parse through each url in the list.
            page = urlopen(url[0]).read()
            df1, header = pd.read_html(page,header=0)#reading with header
            more = license_exists(?????)
            df=df.append(df1) # append to dataframe

            df.to_csv('WV_Licenses_Daily.csv', index=False)

Answer 1

您可以使用单个 for 循环和 break 来执行此操作（不需要 while more）：

lst = []
with open('WV_urls.csv','r') as csvf: # Open file in read mode
    urls = csv.reader(csvf)
    for url in urls:
        page = urlopen(url[0]).read()
        df1, header = pd.read_html(page, header=0)
        if license_exists(BS(page, ‘html.parser’)):
            # if the license is present we don't want to parse any more urls.
            # Note: we don't append this last result (should we?)
            break
        lst.append(df1)

df = pd.concat(lst)
df.to_csv('WV_Licenses_Daily.csv', index=False)

注意：这会从 DataFrames 列表中创建最终的 DataFrame，这比每次追加效率更高。