从维基百科页面解析 OpenURL 数据

Question

我正在尝试从任何给定的维基百科页面检索所有引文数据。查看 wikipedia 页面，我需要在页面参考部分的一个范围内的 OpenURL 对象中保存很多我需要的信息。

span的格式如下：

<span 
    title="ctx_ver=Z39.88-2004&amp;
    rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&amp;
    rft.genre=unknown&amp;
    rft.jtitle=The+Tennessean&amp;
    rft.atitle=Belmont+University+awarded+final+2020+presidential+debate&amp;
    rft.date=2019-10-11&amp;
    rft.aulast=Tamburin&amp;
    rft.aufirst=Adam&amp;
    rft_id=https%3A%2F%2Fwww.tennessean.com%2Fstory%2Fnews%2F2019%2F10%2F11%2Fbelmont-university-nashville-hosts-presidential-debate-2020%2F3941983002%2F&amp;
    rfr_id=info%3Asid%2Fen.wikipedia.org%3A2020+United+States+presidential+election" 

    class="Z3988">
</span>

到目前为止，我已经能够使用 beautifulSoup 检索所有 span 并提取包含数据的标题。但是，在解析 title 字段中的文本时，我感到很困惑。我对rft.atitle、rft.date和rft_id特别感兴趣

import requests
from bs4 import BeautifulSoup


session = requests.Session()
targetWikiPage = "https://en.wikipedia.org/wiki/2020_Beirut_explosion"

if "wikipedia" in targetWikiPage:
    html = session.post(targetWikiPage)
    bsObj = BeautifulSoup(html.text, "html.parser")

    html = session.post(targetWikiPage)
    bsObj = BeautifulSoup(html.text, "html.parser")


    wikiReferences = bsObj.find_all('span', {'class': 'Z3988'})
    wikiReferencesBS = BeautifulSoup(str(wikiReferences), "html.parser")

    for span in wikiReferencesBS.find_all():
        title = span['title']
        print(title)

部分解决方案

此解决方案提供了一个接受字符串和两个标志的函数。我们要解析的字符串的开头和结束标志的第一个实例的结尾。

我现在面临的问题是unboundLocalError

Traceback (most recent call last):
  File "coinscraper.py", line 33, in <module>
    print(extractstring(title,flag1='rft.atitle=', flag2='&'))
  File "coinscraper.py", line 17, in extractstring
    return(string)
UnboundLocalError: local variable 'string' referenced before assignment

修改

import requests
from bs4 import BeautifulSoup
import re


session = requests.Session()
targetWikiPage = "https://en.wikipedia.org/wiki/2020_Beirut_explosion"


def extractstring(line,flag1, flag2):
    if flag1 in line: # $ is the flag
        dex1=line.index(flag1)
        subline=line[dex1+len(flag1):-1] #leave out flag (+1) to end of line
        dex2=subline.index(flag2)
        string=subline[0:dex2].strip() #does not include last flag, strip whitespace
        string = urllib.parse.unquote_plus(string)

    return(string)

if "wikipedia" in targetWikiPage:
    html = session.post(targetWikiPage)
    bsObj = BeautifulSoup(html.text, "html.parser")

    html = session.post(targetWikiPage)
    bsObj = BeautifulSoup(html.text, "html.parser")


    wikiReferences = bsObj.find_all('span', {'class': 'Z3988'})
    wikiReferencesBS = BeautifulSoup(str(wikiReferences), "html.parser")

    for span in wikiReferencesBS.find_all():
        title = span['title']

        print(extractstring(title,flag1='rft.atitle=', flag2='&'))

Answer 1

我会这样处理：

from urllib.parse import unquote

import requests
from bs4 import BeautifulSoup

targetWikiPage = "https://en.wikipedia.org/wiki/2020_Beirut_explosion"

response = requests.get(targetWikiPage).text
soup = BeautifulSoup(response, "html.parser").find_all('span', {'class': 'Z3988'})


def get_rfts():
    for i in soup:
        for rft in i['title'].split("&"):
            yield rft


keep = ["rft.atitle", "rft.date", "rft_id"]
for rft in get_rfts():
    rft_key, rft_value = rft.split("=")
    if rft_key in keep:
        print(unquote(rft_value).replace("+", " "))

输出：

'Endemic corruption' caused Beirut blast, says Diab: Live updates
https://www.aljazeera.com/news/2020/08/beirut-police-fire-tear-gas-protesters-regroup-live-updates-200810010528285.html
Lebanon's government 'to resign over blast'
2020-08-10
https://www.bbc.com/news/world-middle-east-53720383
Beirut Explosion Generates Seismic Waves Equivalent Of A Magnitude 3.3 Earthquake
https://www.forbes.com/sites/davidbressan/2020/08/06/beirut-port-explosion-triggers-magnitude-3-earthquake/
Many injured as large blast rocks Beirut
2020-08-04
https://www.bbc.co.uk/news/world-middle-east-53656220
Beirut explosion 'one of the largest non-nuclear blasts in history'
2020-08-05
https://www.standard.co.uk/news/world/beirut-explosion-one-of-largest-blasts-history-a4517646.html
Second day of protests as anger over Beirut explosion grows: Live
https://www.aljazeera.com/news/2020/08/hundreds-protesters-injured-anger-simmers-beirut-live-200808234355971.html
Clashes Erupt in Beirut at Blast Protest as Lebanon's Anger Boils Over
...