找不到使用 python3 和 Beautifulsoup 更改其值的 html 元素

Question

我想废弃此页面以自动下载其中的所有文件： https://spreadstreet.io/database/.

诀窍是下拉按钮 () 以值“10”开头。我希望将其设置为“全部”。

但当我在控制台中打印结果时，我没有找到与此下拉列表相关的任何“div”，尽管在浏览器中看到了它们！

我遵循了这个好答案：scraping a response from a selected option in dropdown list

但我不明白在哪里可以找到这些信息并因此更改它们。

Answer 1

看起来您的表是动态加载数据的。 BeautifulSoup 本身是不够的。所以你需要使用像 selenium 这样的工具。

您也可以通过向源发送 http 请求来实现您的目的。

您可以检查您的浏览器并检查它发送数据的位置并从中创建请求。

这是您问题的解决方案。

from bs4 import BeautifulSoup
import requests
import json

url = "https://spreadstreet.io/wp-admin/admin-ajax.php"

querystring = {"action":"get_wdtable","table_id":"102"}

payload = "draw=4&columns%5B0%5D%5Bdata%5D=0&columns%5B0%5D%5Bname%5D=Cryptocurrency&columns%5B0%5D%5Bsearchable%5D=true&columns%5B0%5D%5Borderable%5D=true&columns%5B0%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B0%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B1%5D%5Bdata%5D=1&columns%5B1%5D%5Bname%5D=frequency&columns%5B1%5D%5Bsearchable%5D=true&columns%5B1%5D%5Borderable%5D=true&columns%5B1%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B1%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B2%5D%5Bdata%5D=2&columns%5B2%5D%5Bname%5D=Exchange&columns%5B2%5D%5Bsearchable%5D=true&columns%5B2%5D%5Borderable%5D=true&columns%5B2%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B2%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B3%5D%5Bdata%5D=3&columns%5B3%5D%5Bname%5D=database_code&columns%5B3%5D%5Bsearchable%5D=true&columns%5B3%5D%5Borderable%5D=true&columns%5B3%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B3%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B4%5D%5Bdata%5D=4&columns%5B4%5D%5Bname%5D=Description&columns%5B4%5D%5Bsearchable%5D=true&columns%5B4%5D%5Borderable%5D=true&columns%5B4%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B4%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B5%5D%5Bdata%5D=5&columns%5B5%5D%5Bname%5D=Download&columns%5B5%5D%5Bsearchable%5D=true&columns%5B5%5D%5Borderable%5D=true&columns%5B5%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B5%5D%5Bsearch%5D%5Bregex%5D=false&order%5B0%5D%5Bcolumn%5D=0&order%5B0%5D%5Bdir%5D=asc&start=0&length=-1"
headers = {
    'Accept': "text/html; charset=UTF-8",
    'Content-Type': "application/x-www-form-urlencoded"
    }

response = requests.request("POST", url, data=payload, headers=headers, params=querystring)

crpytos = json.loads(response.text)['data']

for crpyto in crpytos:
    name = crpyto[0]
    duration = crpyto[1]
    exchange = crpyto[2]
    download_link = BeautifulSoup(crpyto[5],'lxml').find('a')['href']
    print(name,duration,exchange,download_link)

这会给你输出

.
.
.
.
Yacoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/YAC-BTC Exchange Rate.csv
Zcash daily Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/ZEC-BTC Exchange Rate.csv
Zcash daily Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/ZEC-USD Exchange Rate.csv
Zcash daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZEC-CNY Exchange Rate.csv
Zcash daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZEC-BTC Exchange Rate.csv
Zcash 5 min Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/5-Minute/USDT_ZEC 5-Minute.csv
Zcash 15 Min Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/15-Minute/USDT_ZEC 15-Minute.csv
Zcash 30 Min Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/30-Minute/USDT_ZEC 30-Minute.csv
Zcash 2 Hour Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/2-Hour/USDT_ZEC 2-Hour.csv
Zcash 4 Hour Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/4-Hour/USDT_ZEC 4-Hour.csv
Zcash daily Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/Daily/USDT_ZEC Daily.csv
Zcash Tick Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/Tick/ByCurrency/ZECBTC.csv
Zcash Tick Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/Tick/ByCurrency/ZECUSD.csv
ZcCoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZCC-CNY Exchange Rate.csv
ZcCoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZCC-BTC Exchange Rate.csv
Zetacoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZET-CNY Exchange Rate.csv
Zetacoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZET-BTC Exchange Rate.csv