标题和 URL 抓取论坛的 Python 错误

Question

我想在 URL 的论坛上抓取每个帖子的标题和 URL，这样当使用下面的 1 个标题创建新帖子时，我想收到一封带有帖子链接的邮件.

请不要对我这么苛刻，我是 Python 和 Scraping 的初学者

我有多个问题。

1：在 While(True) 函数中，“汤”带有红色下划线，并带有错误：未定义变量“汤” 2：当注释掉 While(True) 函数时，程序将不会运行。我没有错误。 3：当有一个符合我的标准的新帖子时，我如何获取该帖子的 URL？

标题

def Jeti_DC_16
def Jeti_DC_16_v2
def Jeti_DS_16
def Jeti_DS16_v2

我的完整代码

from requests import get
from bs4 import BeautifulSoup
import re
import smtplib
import time
import lxml
import pprint
import json


URL = 'https://www.rc-network.de/forums/biete-rc-elektronik-zubeh%C3%B6r.135/'


def scrape_page_metadata(URL):
    headers = {
        "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36'}

    pp = pprint.PrettyPrinter(indent=4)
    response = get(URL, headers=headers)
    soup = BeautifulSoup(response.content, "lxml")
    metadata = {
        'Jeti_DC_16': Jeti_DC_16(soup, URL),
        'jeti_dc_16_2': Jeti_DC_16_v2(soup, URL),
        'jeti_ds_16': Jeti_DS_16(soup, URL),
        'jeti_ds_16_2': Jeti_DS_16_v2(soup, URL)
    }
    pp.pprint(metadata)
    return metadata


def Jeti_DC_16(soup, URL):

    jeti_dc_16 = None
    if soup.name.string:
        jeti_dc_16 = soup.title.string
    elif soup.find_all("div", class_='structItem-title'):
        jeti_dc_16 = soup.find_all(
            "div", class_='structItem-title').get('text')
    else:
        jeti_dc_16 = URL.split('//')[1]
        return jeti_dc_16.split('/')[0].rsplit('.')[1].capitalize()
    return jeti_dc_16


def Jeti_DC_16_v2(soup, URL):

    jeti_dc_16_v2 = None
    if soup.name.string:
        jeti_dc_16_v2 = soup.title.string
    elif soup.find_all("div", class_='structItem-title'):
        jeti_dc_16_v2 = soup.find_all(
            "div", class_='structItem-title').get('text')
    else:
        jeti_dc_16_v2 = URL.split('//')[1]
        return jeti_dc_16_v2.split('/')[0].rsplit('.')[1].capitalize()
    return jeti_dc_16_v2


def Jeti_DS_16(soup, URL):

    jeti_ds_16 = None
    if soup.jeti_ds_16.string:
        jeti_ds_16 = soup.jeti_ds_16.string
    elif soup.find_all("div", class_='structItem-title'):
        jeti_ds_16 = soup.find_all(
            "div", class_='structItem-title').get('text')
    else:
        jeti_ds_16 = URL.split('//')[1]
        return jeti_ds_16.split('/')[0].rsplit('.')[1].capitalize()
    return jeti_ds_16


def Jeti_DS_16_v2(soup, URL):

    jeti_ds_16_v2 = None
    if soup.name.string:
        jeti_ds_16_v2 = soup.title.string
    elif soup.find_all("div", class_='structItem-title'):
        jeti_ds_16_v2 = soup.find_all(
            "div", class_='structItem-title').get('text')
    else:
        jeti_dc_16_v2 = URL.split('//')[1]
        return jeti_dc_16_v2.split('/')[0].rsplit('.')[1].capitalize()
    return jeti_ds_16_v2

    # search_for_class = soup.find_all(
    #     'div', class_='structItem-title')
    # Jeti_DS_16 = soup.find_all(text="Jeti DS 16")
    # Jeti_DS_16_v2 = soup.find_all(text="Jeti DS 16 2")
    # Jeti_DC_16 = soup.find_all(text="Jeti DC 16")
    # Jeti_DC_16_v2 = soup.find_all(text="Jeti DC 16 2")


if(Jeti_DC_16, Jeti_DC_16_v2, Jeti_DS_16, Jeti_DS_16_v2):
    send_mail()

    # # print('Die Nummer {0} {1} {2} {3} wurden gezogen'.format(
    # #     Jeti_DC_16, Jeti_DC_16_v2, Jeti_DS_16, Jeti_DS_16_v2))
    # for i in soup.find_all('div', attrs={'class': 'structItem-title'}):
    #     print(i.a['href'])

    # first_result = search_for_class[2]
    # print(first_result.text)
    # print(Jeti_DC_16, Jeti_DC_16_v2, Jeti_DS_16, Jeti_DS_16_v2)

    def send_mail():
        with open('/Users/blackbox/Desktop/SynologyDrive/Programmieren/rc-network/credentials.json', 'r') as myFile:
            data = myFile.read()

        obj = json.loads(data)
        print("test: " + str(obj['passwd']))

        server_ssl = smtplib.SMTP_SSL('smtp.gmail.com', 465)
        server_ssl.ehlo()
        # server.starttls()
        # server.ehlo()

        server_ssl.login('secure@gmail.com', 'secure')

        subject = 'Es gibt ein neuer Post im RC-Network auf deine gespeicherte Anfragen. Sieh in dir an{Link to Post}'
        body = 'Sieh es dir an Link: https://www.rc-network.de/forums/biete-rc-elektronik-zubeh%C3%B6r.135/'

        msg = f"Subject: {subject}\n\n{body}"
        emails = ["secure@gmx.de"]

        server_ssl.sendmail(
            'secure@gmail.com',
            emails,
            msg
        )
        print('e-Mail wurde versendet!')

    #     server_ssl.quit


while(True):
    Jeti_DC_16(soup, URL)
    Jeti_DC_16_v2(soup, URL)
    Jeti_DS_16(soup, URL)
    Jeti_DS_16_v2(soup, URL)
    time.sleep(10)
    # time.sleep(86400)

Answer 1

您在scrape_page_metadata 内部创建soup，它是在scrape_page_metadata 外部不存在的本地变量。在while-loop 中，您应该使用scrape_page_metadata() 而不是函数Jeti_DC_16(), Jeti_DC_16_v2(), Jeti_DS_16(), Jeti_DS_16_v2()

这个函数给你metadata，你应该检查它而不是if(Jeti_DC_16, Jeti_DC_16_v2, Jeti_DS_16, Jeti_DS_16_v2)

或多或少（您必须使用正确的值代替...，因为我不知道您要比较什么）

while True:
    metadata = scrape_page_metadata(URL)

    if metadata["Jeti_DC_16"] == ... and metadata["Jeti_DC_16_v2"] == ... and metadata["Jeti_DS_16"] == ... and metadata["Jeti_DS_16_v2"] == ...:
        send_mail()
    
    time.sleep(10)

但还有其他问题

您的所有函数Jeti_DC_16, Jeti_DC_16_v2, Jeti_DS_16, Jeti_DS_16_v2 看起来都一样，并且可能它们返回相同的元素。您可以使用其中一个并删除其他。或者你应该改变它们，它们应该搜索不同的元素。

您可能不得不使用更多的print() 来查看变量中的值以及执行了哪部分代码，因为我认为这段代码还需要进行很多更改。

例如find_all() 给出了结果列表，你不能使用需要单个元素的get()。你需要for-loop 从所有元素中获取所有标题

或多或少

jeti_ds_16_v2 = soup.find_all("div", class_='structItem-itle')
jeti_ds_16_v2 = [item.get('text') for item in jeti_ds_16_v2]