使用 Python 和 BeautifulSoup 抓取多边形的坐标

Question

我正在尝试从该网页和许多类似网页中抓取信息，https://knowyourcity.info/settlement/1846/5119249

查看页面源时，页面顶部的多边形坐标可用，但在检查多边形元素时不可用。有人知道如何使用 Python 中的 BeautifulSoup 包将这些坐标刮到数据框的列中吗？

这是我用来访问网站的代码

from requests import get
url = 'http://knowyourcity.info/settlement/1846/5119249'
response = get(url)
print(response.text[:500])

from bs4 import BeautifulSoup
html_soup = BeautifulSoup(response.text, "html.parser")
type(html_soup)

Answer 1

看起来地图是由可变结算驱动的。因此一种选择 就是遍历所有scripts标签并搜索var settlement。找到变量后，使用简单的查找和订阅来获取变量数据。将此转换为 json，然后返回边界。

该示例仅用于说明目的。您很可能想要重构代码：

from requests import get
from bs4 import BeautifulSoup
import json

def getHtml():
    url = 'http://knowyourcity.info/settlement/1846/5119249'
    response = get(url)
    return response.text

def extractBoundaries(html):
    html_soup = BeautifulSoup(html, "html.parser")
    scripts = html_soup.find_all('script')
    
    for script in scripts:
        startFind = "var settlement = "
        endFind = "};"

        if script.contents and startFind in script.contents[0]:
            scriptText = script.contents[0]
            startIndex = scriptText.find(startFind) + len(startFind)
            endIndex = scriptText.find(endFind) + len(endFind) -1
            settlementData = scriptText[startIndex:endIndex]
            jsonData = json.loads(settlementData)
            return jsonData['verification/A0_Boundary']

html = getHtml()
results = extractBoundaries(html)
print(results)

输出：

5.599769999885609 -0.224459999729163 0 0;5.599920830581937 -0.2235293057328249 0 0;5.600343984087658 -0.2220772405721618 0 0;5.600582171330188 -0.2212706242398781 0 0;5.600757735181389 -0.2203650797845285 0 0;5.600943331869303 -0.2195227513738018 0 0;5.601229999764712 -0.2178069995933356 0 0;5.601684627743396 -0.2160719483616731 0 0;5.602178000314495 -0.215115999603654 0 0;5.60277082980997 -0.213977987593978 0 0;5.60322584449716 -0.2131045282513355 0 0;5.603939996133988 -0.2117290691411995 0 0;5.604261867990886 -0.2111080629866819 0 0;5.604746000027944 -0.210174000129939 0 0;5.605512212518647 -0.208745954062465 0 0;5.605957084651777 -0.2079168151088879 0 0;5.60642700020594 -0.2070410004417909 0 0;5.606837000227415 -0.2063009995914058 0 0;5.607503034537444 -0.2072989224072899 0 0;5.608332999968013 -0.2085879998362543 0 0;5.608940827457275 -0.2094694811315776 0 0;5.609384837140567 -0.2101133921192968 0 0;5.609949999892649 -0.210933000057878 0 0;5.610520744736618 -0.2114266172445696 0 0;5.61105999981919 -0.2118930002616821 0 0;5.612419000436546 -0.2126160003281257 0 0;5.613144659798252 -0.2126897915006225 0 0;5.614907000058054 -0.2128690003040674 0 0;5.615398000217567 -0.2144450001366067 0 0;5.615173904452149 -0.2159211302559356 0 0;5.614935501372315 -0.2174915048290131 0 0;5.61470415976919 -0.2190153628686744 0 0;5.614495076386731 -0.2203926071330784 0 0;5.61425499966856 -0.2219740001999071 0 0;5.613865981729703 -0.2233052558328268 0 0;5.613273865396593 -0.2253315354219581 0 0;5.612689000297166 -0.227333000017893 0 0;5.611838309990048 -0.2274067552175438 0 0;5.611219650166788 -0.2272163984180224 0 0;5.610458222968646 -0.2271212195685735 0 0;5.609547010985807 -0.2272079061199293 0 0;5.608730734136145 -0.2266937097468826 0 0;5.607481517358167 -0.2262178181977106 0 0;5.605377060602905 -0.2259990644052436 0 0;5.603420000032998 -0.2258499999774699 0 0;5.602499999875136 -0.2257000002547329 0 0;5.601491149397077 -0.225320574484897 0 0;5.599769999885609 -0.224459999729163 0 0