无法弄清楚如何从无限滚动网站中抓取所有链接，不断抓取重复项

Question

这是我当前的滚动+抓取链接代码：

scroll_pause_time = 6 # You can set your own pause time. My laptop is a bit slow so I use 1 sec
screen_height = driver.execute_script("return window.screen.height;")   # get the screen height of the web
i = 1


while True:
    # scroll one screen height each time
    driver.execute_script("window.scrollTo(0, {screen_height}*{i});".format(screen_height=screen_height, i=i))  
    i += 1
    time.sleep(scroll_pause_time)
    # update scroll height each time after scrolled, as the scroll height can change after we scrolled the page
    scroll_height = driver.execute_script("return document.body.scrollHeight;")
    #driver.execute_script("window.scrollTo(0, {screen_height}*{i} + {extrascroll});".format(screen_height=screen_height, extrascroll=extrascroll, i=i))
    time.sleep(scroll_pause_time)
    links = driver.find_elements(By.CSS_SELECTOR, "a[class='styles__StyledLink-sc-l6elh8-0 ekTmzq Asset--anchor']")
    for link in links:
        file.write(link.get_attribute("href") + '\n')
    # Break the loop when the height we need to scroll to is larger than the total scroll height
    if (screen_height) * i > scroll_height:
        break

我在这个网站上遇到的主要问题：https://opensea.io/collection/embersword-land，是当您访问网站时列表会卸载，所以我无法滚动到底部然后抓取所有链接。

第二个问题是HTML of listings保存listing（红色）的div是动态的，高度/listing数量随机变化，所以有时会抓取40个或者30个，导致我抓取重复。

p>

我能想到的一个解决方法是隐藏我抓取链接的所有列表的元素，从中将新列表移动到顶部，然后再次抓取链接，但我不知道该怎么做。任何帮助将非常感激！如果您需要更多信息，请在 cmets 中告诉我，对于 Stackoverflow 和编码来说还是新手，所以边走边学。

Answer 1

我建议将所有链接放到列表中，删除重复项，然后将它们保存到文件中。

scroll_pause_time = 6 # You can set your own pause time. My laptop is a bit slow so I use 1 sec
screen_height = driver.execute_script("return window.screen.height;")   # get the screen height of the web
i = 1

links_list = []

while True:
    # scroll one screen height each time
    driver.execute_script("window.scrollTo(0, {screen_height}*{i});".format(screen_height=screen_height, i=i))  
    i += 1
    time.sleep(scroll_pause_time)
    # update scroll height each time after scrolled, as the scroll height can change after we scrolled the page
    scroll_height = driver.execute_script("return document.body.scrollHeight;")
    #driver.execute_script("window.scrollTo(0, {screen_height}*{i} + {extrascroll});".format(screen_height=screen_height, extrascroll=extrascroll, i=i))
    time.sleep(scroll_pause_time)
    links = driver.find_elements(By.CSS_SELECTOR, "a[class='styles__StyledLink-sc-l6elh8-0 ekTmzq Asset--anchor']")
    # put the links to list
    for link in links:
        links_list.append(link.get_attribute("href"))
    # Break the loop when the height we need to scroll to is larger than the total scroll height
    if (screen_height) * i > scroll_height:
        break



# this removes all duplications, but keeps the order
# based on https://stackoverflow.com/a/17016257/5226491
# python 3.7 required
links_list = list(dict.fromkeys(links_list))

# this also removes all duplications, but the order will be changed
#links_list = list(set(links_list))

for link in links_list:
    file.write(link + '\n')