抓取 url 不变的网站 [python 与美丽的汤]

Question

我对网络抓取完全陌生。 我如何抓取一个网站，其网址不会随页码而变化？ 假设拿这个网站-https://www.bseindia.com/corporates/Forth_Results.aspx 网址不会随页码而改变， 这和我要问的一样，我们如何在 python 中使用美丽的汤来做到这一点？？

Answer 1

这个脚本有

import requests
from bs4 import BeautifulSoup


url = 'https://www.bseindia.com/corporates/Forth_Results.aspx'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}

soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')

page = 1
while True:
    print(page)
    rows = soup.select('.TTRow')
    if not rows:
        break

    # print some data to screen:
    for tr in rows:
        print(tr.get_text(strip=True, separator=' '))

    # to get correct page, you have to do POST request with correct data
    # the data is located in <input name="..." value=".."> tags
    d = {}
    for i in soup.select('input'):
        d[i['name']] = i.get('value', '')

    # some data parameters needs to be deleted:
    if 'ctl00$ContentPlaceHolder1$btnSubmit' in d:
        del d['ctl00$ContentPlaceHolder1$btnSubmit']

    # set correct page:
    page += 1
    d['__EVENTTARGET'] = 'ctl00$ContentPlaceHolder1$gvData'
    d['__EVENTARGUMENT'] = 'Page${}'.format(page)

    soup = BeautifulSoup(requests.post(url, headers=headers, data=d).content, 'html.parser')

打印：

1
500002 ABB 23 Jul 2020
531082 ALANKIT 23 Jul 2020
535916 ALSL 23 Jul 2020
526662 ARENTERP 23 Jul 2020
500215 ATFL 23 Jul 2020
540611 AUBANK 23 Jul 2020
532523 BIOCON 23 Jul 2020
533167 COROENGG 23 Jul 2020
532839 DISHTV 23 Jul 2020
500150 FOSECOIND 23 Jul 2020
507488 GMBREW 23 Jul 2020
532855 HARYNACAP 23 Jul 2020
541729 HDFCAMC 23 Jul 2020
524342 INDOBORAX 23 Jul 2020
522183 ITL 23 Jul 2020
534623 JUPITERIN 23 Jul 2020
533192 KCPSUGIND 23 Jul 2020
542753 MAHAANIMP 23 Jul 2020
532525 MAHABANK 23 Jul 2020
523754 MAHEPC 23 Jul 2020
531680 MAYUR 23 Jul 2020
526299 MPHASIS 23 Jul 2020
532416 NEXTMEDIA 23 Jul 2020
502294 NILACHAL 23 Jul 2020
538772 NIYOGIN 23 Jul 2020
2
530805 OIVL 23 Jul 2020
538742 PANACHE 23 Jul 2020
531879 PIONDIST 23 Jul 2020
540173 PNBHOUSING 23 Jul 2020
533178 PRADIP 23 Jul 2020

...and so on.

---

编辑：要将其保存为 CSV，您可以使用：

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = 'https://www.bseindia.com/corporates/Forth_Results.aspx'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}

soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')

page = 1
all_data = []
while True:
    print(page)
    rows = soup.select('.TTRow')
    if not rows:
        break

    # print some data to screen:
    for tr in rows:
        row = tr.get_text(strip=True, separator='|').split('|')
        all_data.append(row)

    # to get correct page, you have to do POST request with correct data
    # the data is located in <input name="..." value=".."> tags
    d = {}
    for i in soup.select('input'):
        d[i['name']] = i.get('value', '')

    # some data parameters needs to be deleted:
    if 'ctl00$ContentPlaceHolder1$btnSubmit' in d:
        del d['ctl00$ContentPlaceHolder1$btnSubmit']

    # set correct page:
    page += 1
    d['__EVENTTARGET'] = 'ctl00$ContentPlaceHolder1$gvData'
    d['__EVENTARGUMENT'] = 'Page${}'.format(page)

    soup = BeautifulSoup(requests.post(url, headers=headers, data=d).content, 'html.parser')

df = pd.DataFrame(all_data)
print(df)
df.to_csv('data.csv')

生成 data.csv（来自 LibreOffice 的屏幕截图）：