Python Web Scraping 无法提取我想要的表格数据

Question

这是我试图通过页面https://stellar.expert/explorer/public/asset/native?cursor=15297&filter=asset-holders 完成的工作

1. 提取表格列（账户和账户余额）中的数据
2. 将提取的数据字段写入文本文件。
3. 我正在尝试提取多个页面，例如从 15297 到 15500。

我对 python 和网络抓取仍然很陌生，我一直在努力获得所需的输出。帮助将不胜感激。谢谢。

from bs4 import BeautifulSoup
try:
    import urllib.request as urllib2
except ImportError:
    import urllib2
from time import sleep

url = 'https://stellar.expert/explorer/public/asset/native?cursor=15297&filter=asset-holders'
page = urllib2.urlopen((url))
soup = BeautifulSoup(page, 'html.parser')

for div in soup.find_all('div', attrs={'class': 'table exportable space'}):
    address = div.find('tbody', attrs={'class': "account-address"})
    address = address.text.strip() 
    print (address)

    bal = div.find('tbody', attrs={'class': "text-right nowrap"})
    bal = bal.text.strip() 
    print (balance)
print ("%s%s\n" % ("page=", str(URL)))

我想写入的输出（result.txt）：

 GBMN2KIUQS66JMJHVOCA7N3S35F4F5PTY63XG3BJKKDMLPI4HPHA7QNU: 27,005 XLM
 GBWVMIVJQNILFPHACV4Q7QM7VBOOQ35IQXEOOXO7WBXA4KX7OEADOU65: 27,004 XLM
 GD3UXDHKS5EIKSL3PIG3NTVCCK5EQX73JMFJBFBROX3BAZ4K6437TZAC: 27,003 XLM
 GBYUHMDQYNNDMJWOVODTZUKWOTVJXA2PCTDOF6J5NQS5DGMMJAL6B66L: 27,002 XLM
 GDD3AQFXWWHWEFAOU4AWPZCT3Q6EVK4WYHWSG2EBQB3Z43KZSXJEK6WB: 27,001 XLM

Answer 1

也许这段代码 sn-p 可以帮助你。

我检查了网页，发现他们使用了一个 api，向它发出了请求，并返回了您想要的数据以及指向下一页和上一页的链接：

import requests

header = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0Gecko/20100101 Firefox/86.0'}
req = requests.get(url='https://api.stellar.expert/explorer/public/asset/XLM/holders?cursor=15297&filter=asset-holders&limit=50&order=asc', headers=header)

print(f"Previous page: {req.json()['_links']['prev']['href']}")
print(f"Next page: {req.json()['_links']['next']['href']}")

for dict_ in req.json()['_embedded']['records']:
    print(dict_['account'], dict_['balance'])

您也可以在浏览器中访问网址查看数据

Answer 2

您似乎正在尝试抓取动态呈现的内容。为此，您可以考虑使用Selenium。要使此解决方案特别有效，您需要转到 this link 并下载并安装名为 geckodriver 的 WebDriver 并使其在您的路径中可用。

from bs4 import BeautifulSoup
from selenium import webdriver
import urllib
import time

START_RANK = 15297
END_RANK = 15500

current_rank = START_RANK

file_name = 'result.txt'
base_url = 'stellar.expert'
path = 'explorer/public/asset/native'

driver = webdriver.Firefox()

with open(file_name, 'w') as f:
    while current_rank <= END_RANK:
        query_params = {'cursor': str(current_rank), 'filter': 'asset-holders'}
    
        driver.get('https://{0}/{1}?{2}'.format(base_url, path, 
            '&'.join(f'{k}={v}' for k, v in query_params.items())))

        # this is just to ensure that the page is loaded 
        time.sleep(10)  

        soup = BeautifulSoup(driver.page_source, 'html.parser')

        table = soup.find('table', {'class': ['table', 'exportable', 'space']})
        tbody = table.find('tbody')
    
        for trow in tbody.find_all('tr', recursive=False):
            pubkey = trow.find('span', attrs={'class': 'account-pubkey'})
            balance = trow.find('span', attrs={'class': ['amount', 'nowrap']})
            
            print(pubkey.text, balance.text, file=f)

            current_rank += 1
            if current_rank > END_RANK: break

点文件

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
bs4 = "*"
selenium = "*"

[dev-packages]

[requires]
python_version = "3.9"