使用 selenium、curl、wget 等工具从命令行下载页面

Question

过去几天我一直在尝试使用 wget 或 curl 等工具从下一页的命令行自动保存为表格或文本文件，但没有成功。问题还来自于 url 被屏蔽的事实。我想知道是否有可能使用 selenium 之类的工具来做到这一点。

https://www.hkex.com.hk/Market-Data/Securities-Prices/Equities?sc_lang=en

在将页面保存为表格之前，基本上需要做两个操作：

a) 单击右下角的“20 个项目”，将其带到“100 个项目”

b) 点击页面中下方的“LOAD MORE”链接10次，以增加显示项目的数量

对于如何解决此任务的任何建议，我将不胜感激。 感谢您的考虑。

Answer 1

最简单的方法是使用剧作家 - https://playwright.dev/python/docs/intro

然后记录按钮点击 - 编剧codegen

保存页面内容/导出为pdf

您可以保存脚本，然后稍后重新运行 - python hkex.py

示例脚本

from playwright.sync_api import Playwright, sync_playwright
def run(playwright: Playwright) -> None:
    browser = playwright.chromium.launch(headless=False)
    context = browser.new_context()
    # Open new page
    page = context.new_page()
    # Go to https://www.hkex.com.hk/Market-Data/Securities-Prices/Equities?sc_lang=en
    page.goto("https://www.hkex.com.hk/Market-Data/Securities-Prices/Equities?sc_lang=en")
    # Click #accpet_cookie_btn
    page.click("#accpet_cookie_btn")
    # Click text=20 Items
    page.click("text=20 Items")
    # Click :nth-match(:text("Items"), 4)
    page.click(":nth-match(:text(\"Items\"), 4)")
    # Click text=LOAD MORE
    page.click("text=LOAD MORE")
    # Click text=LOAD MORE
    page.click("text=LOAD MORE")
    # Click text=LOAD MORE 100 Items 20 Items 50 Items 100 Items
    page.click("text=LOAD MORE 100 Items 20 Items 50 Items 100 Items")
    # Click text=LOAD MORE
    page.click("text=LOAD MORE")
    # Click text=LOAD MORE
    page.click("text=LOAD MORE")
    # Click text=Equity Overview LIST OF SECURITIES EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs
    page.click("text=Equity Overview LIST OF SECURITIES EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs")
    # Click .back_to_top_btn
    page.click(".back_to_top_btn")
    # assert page.url == "https://www.hkex.com.hk/Market-Data/Securities-Prices/Equities?sc_lang=en#hkex_page_header"
    # Click text=Equity Overview LIST OF SECURITIES EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs
    page.click("text=Equity Overview LIST OF SECURITIES EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs")
    # Click #lhkexw-equities div:has-text("FILTERS EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs DEBT SECURITIES Filters EQ")
    page.click("#lhkexw-equities div:has-text(\"FILTERS EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs DEBT SECURITIES Filters EQ\")")
    # Click #lhkexw-equities div:has-text("FILTERS EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs DEBT SECURITIES Filters EQ")
    page.click("#lhkexw-equities div:has-text(\"FILTERS EQUITIES ETPs DWs INLINE WARRANTS CBBCs REITs DEBT SECURITIES Filters EQ\")")
    # Click text=LOAD MORE
    page.click("text=LOAD MORE")
    # ---------------------
    context.close()
    browser.close()
with sync_playwright() as playwright:
    run(playwright)