使用请求和 bs4 仅从网站中抓取新元素

Question

我想查看某个网站以从中收集数据，在第一次访问该网站时，我会收集所有数据以忽略它。如果添加了新行（例如此处的打印），我想执行某个操作。但是，每当出现新项目时，它似乎会打印网站上的每一行，即使我正在检查该行是否已经存在于字典中。不知道怎么解决，谁能看看？

import requests
import re
import copy
import time

from datetime import date
from bs4 import BeautifulSoup


class KillStatistics:
    def __init__(self):
        self.records = {}
        self.watched_names = ["Test"]
        self.iter = 0

    def parse_records(self):
        r = requests.get("http://149.56.28.71/?subtopic=killstatistics")
        soup = BeautifulSoup(r.content, "html.parser")
        table = soup.findChildren("table")

        for record in table:
            for data in record:
                if data.text == "Last Deaths":
                    pass
                else:
                    entry = data.text
                    entry = re.split("..?(?=[0-9][A-Z]).", data.text)
                    entry[0] = entry[0].split(", ")
                    entry[0][0] = entry[0][0].split(".")

                    entry_id, day, month, year, hour = (
                        entry[0][0][0],
                        entry[0][0][1],
                        entry[0][0][2],
                        entry[0][0][3],
                        entry[0][1],
                    )

                    message = entry[1]
                    nickname = (re.findall(".+?(?=at)", message)[0]).strip()
                    killed_by = (re.findall(r"(?<=\bby).*", message)[0]).strip()
                    if self.iter < 1:
                        """Its the first visit to the website, i want it to download all the data and store it in dictionary"""
                        self.records[
                            entry_id
                        ] = f"{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}"
                    elif (
                        self.iter > 1
                        and f"{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}"
                        not in self.records.values()
                    ):
                        """Here I want to look into the dictionary to check if the element exists in it, 
                        if not print it and add to the dictionary at [entry_id] so we can skip it in next iteration
                        Don't know why but whenever a new item appears on the website it seems to edit every item in the dictionary instead of just editing the one
                        that wasnt there"""
                        print(
                            f"{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}"
                        )
                        self.records[
                            entry_id
                        ] = f"{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}"
        print("---")
        self.iter += 1


ks = KillStatistics()
if __name__ == "__main__":
    while True:
        ks.parse_records()
        time.sleep(10)

entry_id 始终与其 500 行数据相同，并且它们的 id 为 1,2,3...500，最新的始终为 1。我知道我总是可以检查 1 以获得最新的，但有时例如10 个玩家可以同时死亡，所以我想检查他们是否有变化，并且只在新玩家上执行打印。

当前输出：

Velerion was killed by Rat and Cave Rat at 27-12-2021 16:53
Scrappy was killed by Cursed Queen at 27-12-2021 16:52
Velerion was killed by Rat at 27-12-2021 16:28
Velerion was killed by Rat at 27-12-2021 16:22
Velerion was killed by Rat at 27-12-2021 16:21
Velerion was killed by Rat at 27-12-2021 15:51
Shade was killed by Tentacle Slayer at 27-12-2021 15:46
Mr Yahoo was killed by Immortal Hunter at 27-12-2021 15:41
Scrappy was killed by Witch Hunter at 27-12-2021 15:39
Barbudo Arqueiro was killed by Seahorse at 27-12-2021 15:23
Emperor Martino was killed by Dark Slayer at 27-12-2021 15:14
Shade was killed by Tentacle Slayer at 27-12-2021 15:11
Head Hunter was killed by Demon Blood Slayer at 27-12-2021 15:09

预期输出：

Velerion was killed by Rat and Cave Rat at 27-12-2021 16:53

Answer 1

以下是我对您的处理方式所做的更改。我正在使用一个正则表达式来解析所有标题信息。这让我一次获得所有 7 个数字字段，加上匹配的总长度告诉我消息从哪里开始。

然后，我使用时间戳来确定哪些数据是新的。最新的条目总是第一个，所以我抓住了批次的第一个时间戳作为下一个的阈值。

然后，我将条目存储在列表而不是字典中。如果您真的不需要永远存储它们，而只是想打印它们，那么您根本不需要跟踪列表。

import requests
import re
import copy
import time

from datetime import date
from bs4 import BeautifulSoup

prefix = r"(\d+)\.(\d+)\.(\d+)\.(\d+)\, (\d+):(\d+):(\d+)"

class KillStatistics:
    def __init__(self):
        self.records = []
        self.latest = (0,0,0,0,0,0,0)
        self.watched_names = ["Test"]

    def parse_records(self):
        r = requests.get("http://149.56.28.71/?subtopic=killstatistics")
        soup = BeautifulSoup(r.content, "html.parser")
        table = soup.findChildren("table")
        latest = None

        for record in table:
            for data in record:
                if data.text == "Last Deaths":
                    continue
                entry = data.text
                mo = re.match(prefix, entry)
                entry_id, day, month, year, hour, mm, ss = mo.groups()
                stamp = tuple(int(i) for i in (year, month, day, hour, mm, ss))
                if latest is None:
                    latest = stamp

                if stamp > self.latest:
                    rest = entry[mo.span()[1]:]
                    i = rest.find(" at ")
                    j = rest.find(" by ")
                    nickname = rest[:i]
                    killed_by = rest[j+4:]
                    msg = f"{nickname} was killed by {killed_by} at {day}-{month}-{year} {hour}"
                    print( msg )
                    self.records.append( msg )
        print("---")
        self.latest = latest

ks = KillStatistics()
if __name__ == "__main__":
    while True:
        ks.parse_records()
        time.sleep(10)

Answer 2

我自己设法找到了解决方案。在这种特定情况下，我需要创建一个条目列表及其副本。然后在每次查找之后，我将新创建的列表与旧列表进行比较，并使用 set().difference() 返回新记录。

import requests
import re
import copy
import time

from datetime import date
from bs4 import BeautifulSoup


class KillStatistics:
    def __init__(self):
        self.records = []
        self.old_table = []
        self.watched_names = ["Test"]
        self.visited = False

    def parse_records(self):
        r = requests.get("http://149.56.28.71/?subtopic=killstatistics")
        soup = BeautifulSoup(r.content, "html.parser")
        table = soup.findChildren("table")
        self.records = []

        for record in table:
            for data in record:
                if data.text == "Last Deaths":
                    continue
                else:

                    entry = data.text
                    entry = re.split("..?(?=[0-9][A-Z]).", data.text)
                    entry[0] = entry[0].split(", ")
                    entry[0][0] = entry[0][0].split(".")

                    entry_id, day, month, year, hour = (
                        entry[0][0][0],
                        entry[0][0][1],
                        entry[0][0][2],
                        entry[0][0][3],
                        entry[0][1],
                    )
                    ## czas recordów jest w EST. Można konwertować na czas w Brazylii i w CET
                    message = entry[1]
                    nickname = (re.findall(".+?(?=at)", message)[0]).strip()
                    killed_by = (re.findall(r"(?<=\bby).*", message)[0]).strip()
                    record_to_add = (
                        f"{nickname},{killed_by},{day}-{month}-{year} {hour}"
                    )

                    self.records.append(record_to_add)

        if len(self.old_table) > 0:
            self.compare_records(self.old_table, self.records)
        else:
            print("Setting up initial data...")
        print("---")
        self.visited = True

    def compare_records(self, old_records, new_records):
        new_record = list(set(new_records).difference(old_records))
        if len(new_record) > 0:
            print("We got new record")
            for i in new_record:
                print(i)
                with open("/home/sammy/gp/new_records", "a") as f:
                    f.write(i + "\n")
        else:
            print("No new records")

class DiscordBot:
    def __init__(self):
        pass


if __name__ == "__main__":
    ks = KillStatistics()
    while True:
        ks.parse_records()
        ks.old_table = copy.deepcopy(ks.records)
        time.sleep(10)