用漂亮的汤刮奇怪的桌子结构答案

【问题标题】：Scraping weird table structure using beautiful soup用漂亮的汤刮奇怪的桌子结构
【发布时间】：2021-05-12 14:10:09
【问题描述】：

我正在尝试从 website 中抓取几个表，这些表隐藏在展开按钮下。但是看起来表结构真的很奇怪，我遇到了麻烦。

到目前为止我的代码是

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
import requests
from bs4 import BeautifulSoup

groups = ['Currencies', 'Commodities', 'Indices']
assets = {'Currencies': ['eur/usd','gbp/usd', 'usd/jpy'], 'Commodities': ['gold', 'silver', 'copper'], 'Indices': ['spx50', 'dj30']}


browser = webdriver.Chrome(executable_path=r'/usr/local/bin/chromedriver', options=Options())
browser.get('https://www.etoro.com/trading/fees/#cfds')
soup = BeautifulSoup(browser.page_source, 'lxml')

我的目标是从网站的“点差”部分（不是“隔夜费用”部分）、groups 变量中的每个资产组以及每个组中获取费用assets 部分中的单个资产。例如每个组都是这样的

| Instrument | Fee    |
|------------|--------|
| eur/usd    | 1 Pips |
| gbp/usd    | 2 Pips |
| usd/jpy    | 1 Pips |

这是一个来自美丽汤的货币表 html 的示例：

<thead>
<tr>
</tr>
</thead>
<tbody>
<tr class="tr-decade-0" data-decade="0" data-search_val="eurusd eur/usd">
<td class="id-1"><a class="e-info" href="/markets/eurusd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-usd/150x150.png")'></div>
<div class="details">
<div class="symbol">EURUSD</div>
<div class="name">EUR/USD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">1</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="usdjpy usd/jpy">
<td class="id-5"><a class="e-info" href="/markets/usdjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">USDJPY</div>
<div class="name">USD/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">1</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="gbpusd gbp/usd">
<td class="id-2"><a class="e-info" href="/markets/gbpusd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/gbp-usd/150x150.png")'></div>
<div class="details">
<div class="symbol">GBPUSD</div>
<div class="name">GBP/USD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">2</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="usdchf usd/chf">
<td class="id-6"><a class="e-info" href="/markets/usdchf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-chf/150x150.png")'></div>
<div class="details">
<div class="symbol">USDCHF</div>
<div class="name">USD/CHF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">1.5</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="nzdusd nzd/usd">
<td class="id-3"><a class="e-info" href="/markets/nzdusd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/nzd-usd/150x150.png")'></div>
<div class="details">
<div class="symbol">NZDUSD</div>
<div class="name">NZD/USD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">2.5</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="eurgbp eur/gbp">
<td class="id-8"><a class="e-info" href="/markets/eurgbp">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-gbp/150x150.png")'></div>
<div class="details">
<div class="symbol">EURGBP</div>
<div class="name">EUR/GBP</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">1.5</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="eurjpy eur/jpy">
<td class="id-10"><a class="e-info" href="/markets/eurjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">EURJPY</div>
<div class="name">EUR/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">2</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="gbpjpy gbp/jpy">
<td class="id-11"><a class="e-info" href="/markets/gbpjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/gbp-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">GBPJPY</div>
<div class="name">GBP/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">3</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="audjpy aud/jpy">
<td class="id-14"><a class="e-info" href="/markets/audjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/aud-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">AUDJPY</div>
<div class="name">AUD/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">2</span> Pips</td></tr>
<tr class="tr-decade-0" data-decade="0" data-search_val="audusd aud/usd">
<td class="id-7"><a class="e-info" href="/markets/audusd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/aud-usd/150x150.png")'></div>
<div class="details">
<div class="symbol">AUDUSD</div>
<div class="name">AUD/USD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">1</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="eurchf eur/chf">
<td class="id-9"><a class="e-info" href="/markets/eurchf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-chf/150x150.png")'></div>
<div class="details">
<div class="symbol">EURCHF</div>
<div class="name">EUR/CHF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">5</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="euraud eur/aud">
<td class="id-12"><a class="e-info" href="/markets/euraud">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-aud/150x150.png")'></div>
<div class="details">
<div class="symbol">EURAUD</div>
<div class="name">EUR/AUD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">7</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="eurcad eur/cad">
<td class="id-13"><a class="e-info" href="/markets/eurcad">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-cad/150x150.png")'></div>
<div class="details">
<div class="symbol">EURCAD</div>
<div class="name">EUR/CAD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">7</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="cadjpy cad/jpy">
<td class="id-15"><a class="e-info" href="/markets/cadjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/cad-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">CADJPY</div>
<div class="name">CAD/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">6</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="chfjpy chf/jpy">
<td class="id-16"><a class="e-info" href="/markets/chfjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/chf-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">CHFJPY</div>
<div class="name">CHF/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">6</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="usdhkd usd/hkd">
<td class="id-39"><a class="e-info" href="/markets/usdhkd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-hkd/150x150.png")'></div>
<div class="details">
<div class="symbol">USDHKD</div>
<div class="name">USD/HKD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">5</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="usdzar usd/zar">
<td class="id-42"><a class="e-info" href="/markets/usdzar">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-zar/150x150.png")'></div>
<div class="details">
<div class="symbol">USDZAR</div>
<div class="name">USD/ZAR</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">50</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="usdrub usd/rub">
<td class="id-44"><a class="e-info" href="/markets/usdrub">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-rub/150x150.png")'></div>
<div class="details">
<div class="symbol">USDRUB</div>
<div class="name">USD/RUB</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">3</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="usdcnh usd/cnh">
<td class="id-45"><a class="e-info" href="/markets/usdcnh">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-cnh/150x150.png")'></div>
<div class="details">
<div class="symbol">USDCNH</div>
<div class="name">USD/CNH</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">10</span> Pips</td></tr>
<tr class="tr-decade-1" data-decade="1" data-search_val="audchf aud/chf">
<td class="id-46"><a class="e-info" href="/markets/audchf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/aud-chf/150x150.png")'></div>
<div class="details">
<div class="symbol">AUDCHF</div>
<div class="name">AUD/CHF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="audcad aud/cad">
<td class="id-47"><a class="e-info" href="/markets/audcad">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/aud-cad/150x150.png")'></div>
<div class="details">
<div class="symbol">AUDCAD</div>
<div class="name">AUD/CAD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="audnzd aud/nzd">
<td class="id-48"><a class="e-info" href="/markets/audnzd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/aud-nzd/150x150.png")'></div>
<div class="details">
<div class="symbol">AUDNZD</div>
<div class="name">AUD/NZD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="eurnzd eur/nzd">
<td class="id-49"><a class="e-info" href="/markets/eurnzd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-nzd/150x150.png")'></div>
<div class="details">
<div class="symbol">EURNZD</div>
<div class="name">EUR/NZD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="gbpaud gbp/aud">
<td class="id-50"><a class="e-info" href="/markets/gbpaud">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/gbp-aud/150x150.png")'></div>
<div class="details">
<div class="symbol">GBPAUD</div>
<div class="name">GBP/AUD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="gbpchf gbp/chf">
<td class="id-51"><a class="e-info" href="/markets/gbpchf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/gbp-chf/150x150.png")'></div>
<div class="details">
<div class="symbol">GBPCHF</div>
<div class="name">GBP/CHF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="gbpnzd gbp/nzd">
<td class="id-52"><a class="e-info" href="/markets/gbpnzd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/gbp-nzd/150x150.png")'></div>
<div class="details">
<div class="symbol">GBPNZD</div>
<div class="name">GBP/NZD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="nzdcad nzd/cad">
<td class="id-53"><a class="e-info" href="/markets/nzdcad">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/nzd-cad/150x150.png")'></div>
<div class="details">
<div class="symbol">NZDCAD</div>
<div class="name">NZD/CAD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="nzdchf nzd/chf">
<td class="id-54"><a class="e-info" href="/markets/nzdchf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/nzd-chf/150x150.png")'></div>
<div class="details">
<div class="symbol">NZDCHF</div>
<div class="name">NZD/CHF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="nzdjpy nzd/jpy">
<td class="id-55"><a class="e-info" href="/markets/nzdjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/nzd-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">NZDJPY</div>
<div class="name">NZD/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-2" data-decade="2" data-search_val="cadchf cad/chf">
<td class="id-56"><a class="e-info" href="/markets/cadchf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/cad-chf/150x150.png")'></div>
<div class="details">
<div class="symbol">CADCHF</div>
<div class="name">CAD/CHF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="usdnok usd/nok">
<td class="id-57"><a class="e-info" href="/markets/usdnok">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-nok/150x150.png")'></div>
<div class="details">
<div class="symbol">USDNOK</div>
<div class="name">USD/NOK</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="usdsek usd/sek">
<td class="id-58"><a class="e-info" href="/markets/usdsek">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-sek/150x150.png")'></div>
<div class="details">
<div class="symbol">USDSEK</div>
<div class="name">USD/SEK</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="noksek nok/sek">
<td class="id-59"><a class="e-info" href="/markets/noksek">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/nok-sek/150x150.png")'></div>
<div class="details">
<div class="symbol">NOKSEK</div>
<div class="name">NOK/SEK</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="eurnok eur/nok">
<td class="id-60"><a class="e-info" href="/markets/eurnok">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-nok/150x150.png")'></div>
<div class="details">
<div class="symbol">EURNOK</div>
<div class="name">EUR/NOK</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="eursek eur/sek">
<td class="id-61"><a class="e-info" href="/markets/eursek">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-sek/150x150.png")'></div>
<div class="details">
<div class="symbol">EURSEK</div>
<div class="name">EUR/SEK</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">30</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="usdtry usd/try">
<td class="id-62"><a class="e-info" href="/markets/usdtry">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-try/150x150.png")'></div>
<div class="details">
<div class="symbol">USDTRY</div>
<div class="name">USD/TRY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">50</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="usdmxn usd/mxn">
<td class="id-63"><a class="e-info" href="/markets/usdmxn">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-mxn/150x150.png")'></div>
<div class="details">
<div class="symbol">USDMXN</div>
<div class="name">USD/MXN</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="usdsgd usd/sgd">
<td class="id-64"><a class="e-info" href="/markets/usdsgd">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-sgd/150x150.png")'></div>
<div class="details">
<div class="symbol">USDSGD</div>
<div class="name">USD/SGD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">3</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="gbpcad gbp/cad">
<td class="id-65"><a class="e-info" href="/markets/gbpcad">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/gbp-cad/150x150.png")'></div>
<div class="details">
<div class="symbol">GBPCAD</div>
<div class="name">GBP/CAD</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">4</span> Pips</td></tr>
<tr class="tr-decade-3" data-decade="3" data-search_val="zarjpy zar/jpy">
<td class="id-66"><a class="e-info" href="/markets/zarjpy">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/zar-jpy/150x150.png")'></div>
<div class="details">
<div class="symbol">ZARJPY</div>
<div class="name">ZAR/JPY</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">8</span> Pips</td></tr>
<tr class="tr-decade-4" data-decade="4" data-search_val="eurpln eur/pln">
<td class="id-68"><a class="e-info" href="/markets/eurpln">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-pln/150x150.png")'></div>
<div class="details">
<div class="symbol">EURPLN</div>
<div class="name">EUR/PLN</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">30</span> Pips</td></tr>
<tr class="tr-decade-4" data-decade="4" data-search_val="usdhuf usd/huf">
<td class="id-69"><a class="e-info" href="/markets/usdhuf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-huf/150x150.png")'></div>
<div class="details">
<div class="symbol">USDHUF</div>
<div class="name">USD/HUF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr>
<tr class="tr-decade-4" data-decade="4" data-search_val="eurhuf eur/huf">
<td class="id-70"><a class="e-info" href="/markets/eurhuf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/eur-huf/150x150.png")'></div>
<div class="details">
<div class="symbol">EURHUF</div>
<div class="name">EUR/HUF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr>
<tr class="tr-decade-4" data-decade="4" data-search_val="gbphuf gbp/huf">
<td class="id-71"><a class="e-info" href="/markets/gbphuf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/gbp-huf/150x150.png")'></div>
<div class="details">
<div class="symbol">GBPHUF</div>
<div class="name">GBP/HUF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">30</span> Pips</td></tr>
<tr class="tr-decade-4" data-decade="4" data-search_val="chfhuf chf/huf">
<td class="id-72"><a class="e-info" href="/markets/chfhuf">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/chf-huf/150x150.png")'></div>
<div class="details">
<div class="symbol">CHFHUF</div>
<div class="name">CHF/HUF</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">30</span> Pips</td></tr>
<tr class="tr-decade-4" data-decade="4" data-search_val="usdpln usd/pln">
<td class="id-73"><a class="e-info" href="/markets/usdpln">
<div class="image" style='background-image:url("https://etoro-cdn.etorostatic.com/market-avatars/usd-pln/150x150.png")'></div>
<div class="details">
<div class="symbol">USDPLN</div>
<div class="name">USD/PLN</div>
</div>
</a></td><td data-th="Spread"><span class="spread_num">20</span> Pips</td></tr></tbody>
</table>

我的第一个想法是对我的每个组使用 for 循环来使用 soup.find 识别表，然后使用 for 循环为每个资产搜索匹配的 tr 并从 @987654329 中提取相关数字@。然而，网站结构让我很困惑，所以我在编码时遇到了麻烦。有人可以帮忙吗？

【问题讨论】：

标签： python python-3.x beautifulsoup

【解决方案1】：

让您的生活更轻松，只需将HTML 转储到pandas。所有表格都在那里，您可以通过for loop 或indexing 访问它们。

方法如下：

import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from tabulate import tabulate

options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)

driver.get("https://www.etoro.com/trading/fees/#cfds")
df = pd.read_html(driver.page_source, flavor="bs4")
print(tabulate(df[-9]))
driver.close()

样本输出：

--  --------------------------------------------  ------------  ------------
 0  BTC Bitcoin                                   $ -11.813292  $ 0
 1  ETHEREUM Ethereum                             $ -0.300336   $ 0
 2  BCH Bitcoin Cash                              $ -0.138758   $ 0
 3  XRP Ripple                                    $ -0.00009    $ 0
 4  DASH Dash                                     $ -0.040001   $ 0
 5  LTC Litecoin                                  $ -0.049917   $ 0
 6  ETC Ethereum Classic                          $ -0.0023     $ 0
 7  ADA Cardano                                   $ -0.000072   $ 0
 8  MIOTA IOTA                                    $ -0.000119   $ 0
 9  XLM Stellar                                   $ -0.000051   $ 0
10  EOS EOS                                       $ -0.001041   $ 0
11  NEO NEO                                       $ -0.005731   $ 0
12  TRX TRON                                      $ -0.000011   $ 0
13  ZEC ZCASH                                     $ -0.02535    $ 0
14  BNB Binance Coin                              $ -0.014941   $ 0
15  XTZ Tezos                                     $ -0.000813   $ 0
16  LINK Chainlink                                $ -0.004082   $ 0
17  UNI Uniswap                                   $ -0.000816   $ 0
18  DOGE Dogecoin                                 $ -0.000163   $ -0.000163
19  BTCEUR Bitcoin/Euro                           $ -11.813292  $ 0
20  ETHEUR Ethereum/Euro                          $ -0.300336   $ 0

and so on ...

注意：对我来说，在headless 模式下运行会给我一个验证码。因此，头部模式开启。

【讨论】：

【解决方案2】：

首先，下面的代码展开了所有的展开列表，然后继续为每个列表单击“加载更多”按钮，直到显示所有记录。但是，展开列表和单击加载更多都不会改变页面上的整体元素结构，因此您可以跳过此步骤并继续执行下一个实际抓取数据并从 groups 和 @ 生成结果的代码块987654322@:

from bs4 import BeautifulSoup as soup
from selenium import webdriver
d = webdriver.Chrome('/path/to/chromedriver')
d.get('https://www.etoro.com/trading/fees/#cfds')
d.execute_script("""
   for (var i of document.querySelectorAll('span.expand-toggle')){
      i.click()
}""")
visible = d.execute_script("""
  return Array.from(document.querySelectorAll('button.show-more')).map(function(x){
      var style = window.getComputedStyle(x);
      return style.getPropertyValue('display');
   });
""")
while 'block' in visible:
   d.execute_script("""
   for (var i of document.querySelectorAll('button.show-more')){
      i.click()
   }""")
   visible = d.execute_script("""
      return Array.from(document.querySelectorAll('button.show-more')).map(function(x){
        var style = window.getComputedStyle(x);
        return style.getPropertyValue('display');
     });
   """)

现在，抓取表格（展开，如果上面的代码块已经运行）：

import re
page = soup(d.page_source, 'html.parser')
blocks = [i for i in page.select('div.page_single_tab.tab_2 div.e-row') if (k:=i['class'][-1].split('-')[-1]).isdigit() and 3 <= int(k) <= 8]
groups = ['Currencies', 'Commodities', 'Indices']
assets = {'Currencies': ['eur/usd','gbp/usd', 'usd/jpy'], 'Commodities': ['gold', 'silver', 'copper'], 'Indices': ['spx50', 'dj30']}
vals = {i.select_one('.expand-title-1').text: \
         {j.text.lower():k.select_one('.spread_num').text 
          for k in i.select('div.expand-content tr') if (j:=k.select_one('.name'))}
       for i in blocks}
result = {a:{i:vals[a].get(i) for i in b} for a, b in assets.items()}

输出：

{'Currencies': {'eur/usd': '1', 'gbp/usd': '2', 'usd/jpy': '1'}, 'Commodities': {'gold': '45', 'silver': '5', 'copper': '2'}, 'Indices': {'spx50': None, 'dj30': '600'}}

【讨论】：