首先，让我们获取网站。我正在使用 BeautifulSoup 与 Selenium 一起刮。
import bs4 as Bs
from selenium import webdriver
DRIVER_PATH = 'D:\chromedriver.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('http://lrc.bih.nic.in/ViewRor.aspx?DistCode=36&SubDivCode=2&CircleCode=9')
然后点击一个村庄名称（根据需要更改）
driver.find_element_by_xpath('//*[@id="ctl00_ContentPlaceHolder1_GridView2"]/tbody/tr[3]/td[1]').click()
点击“खाताखोजें”按钮：
driver.find_element_by_xpath('//*[@id="ctl00_ContentPlaceHolder1_BtnSearch"]').click()
使用 BeautifulSoup 获取页面的源代码
page_src = Bs.BeautifulSoup(driver.page_source)
找到 id：ctl00_ContentPlaceHolder1_UpdatePanel2 并在其中找到所有 tds：
table_elements = page_src.find("div",{"id":"ctl00_ContentPlaceHolder1_UpdatePanel2"}).find_all("td")
获取列并从中获取文本
columns = table_elements[:6]
column_names = [e.text for e in header]
columns:
[<td>क्रम</td>,
<td>रैयतधारी का नाम</td>,
<td style="white-space:nowrap;">पिता/पति का नाम</td>,
<td>खाता संख्या</td>,
<td>खेसरा संख्या</td>,
<td>अधिकार<br/>अभिलेख</td>]
column_names:
['क्रम',
'रैयतधारी का नाम',
'पिता/पति का नाम',
'खाता संख्या',
'खेसरा संख्या',
'अधिकारअभिलेख']
接下来获取表格的主体
body_of_table = table_elements[6:-4]
然后为每个条目创建 6 列的块并取出文本
chunks = [body_of_table[x:x+6] for x in range(0, len(body_of_table), 6)]
data = [[e.text.strip('\n') for e in chunk] for chunk in chunks]
data:
[['1', 'अरूण कुमार', 'शिवलाल पासवान', '55', '406', 'देखें'],
['2', 'इन्द्रदेव प्रसाद', '\xa0', '98', '789', 'देखें'],
['3', 'ईश्वर मांझी', 'चमारी मांझी', '78', '42', 'देखें'],
['4', 'कवलसिया देवी', 'तुलसी मांझी', '120', '41', 'देखें'],
['5', 'कामदेव पांडे', 'शिवदानी पांडे', '210', '457, 459, 461, 474', 'देखें'],
['6', 'कामेश्वर मांझी', 'उत्ती मांझी', '78', '43', 'देखें'],
['7', 'कारू मांझी', 'राधे मांझी', '78', '42', 'देखें'],
['8', 'कारू मांझी', 'मेघन मांझी', '78', '42', 'देखें'],
['9', 'कौशल्या देवी', 'केदार महतो', '253', '757', 'देखें'],
['10', 'गणेश साव', 'छेदी साव', '156', '236', 'देखें'],
....
现在import Pandas 并使用它从这个列表列表中创建一个数据框：
import pandas as pd
df = pd.DataFrame(data, columns = column_names)
# set क्रम as index
df.set_index(df.columns[0])
最终结果：
import time # using time.sleep for illustration only. You should use explicit wait
import bs4 as Bs
import pandas as pd
from selenium import webdriver
DRIVER_PATH = 'D:\chromedriver.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('http://lrc.bih.nic.in/ViewRor.aspx?DistCode=36&SubDivCode=2&CircleCode=9')
time.sleep(4)
#click on a village name
driver.find_element_by_xpath('//*[@id="ctl00_ContentPlaceHolder1_GridView2"]/tbody/tr[3]/td[1]').click()
time.sleep(2)
# click on खाता खोजें
driver.find_element_by_xpath('//*[@id="ctl00_ContentPlaceHolder1_BtnSearch"]').click()
time.sleep(2)
# ----------- table extracting part ------------------
# get page source
page_src = Bs.BeautifulSoup(driver.page_source)
# find the id: ctl00_ContentPlaceHolder1_UpdatePanel2 and find all tds in it
table_elements = page_src.find("div",{"id":"ctl00_ContentPlaceHolder1_UpdatePanel2"}).find_all("td")
# get columns and get the text out of them
columns = table_elements[:6]
column_names = [e.text for e in header]
# get the body of the table
body_of_table = table_elements[6:-4]
# create chunks of 6 columns for each entry
chunks = [body_of_table[x:x+6] for x in range(0, len(body_of_table), 6)]
# get the text out
data = [[e.text.strip('\n') for e in chunk] for chunk in chunks]
df = pd.DataFrame(data, columns = column_names)
# set क्रम as index
df.set_index(df.columns[0])
print(df)
...
要浏览下一页：
- 使用 Selenium 单击下一步按钮。
- 等待页面加载
- 重新运行表格提取部分（通过将其放入函数中）
- 丢弃列名（我们已经有了）
- 将数据追加到已创建的数据框中
- 对所有页面重复以上步骤（可以添加while循环，尝试点击某个页面直到出现异常，see try and except）