Python BeautifulSoup 获取图像、文本和 URL

Question

我正在尝试从以下 url (http://www.ancient-hebrew.org/m/dictionary/1000.html) 抓取数据。因此，每个希伯来语单词部分都以 img urls 开头，然后是 2 个文本，即实际的希伯来语单词及其发音。例如 url 中的第一个条目是以下“img1 img2 img3 אֶלֶף e-leph” 使用 wget 下载 html 后的希伯来语单词是 unicode

我给我的以下代码例如<img src="../../files/heb-anc-sm-pey.jpg"/>和<font face="arial" size="+1"> unicode_hebrew_text </font>和<a href="audio/ 505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a>

我只想要../../files/heb-anc-sm-pey.jpg 和unicode_hebrew_text 和audio/505.mp3 (without any spaces in between)

from bs4 import BeautifulSoup

raw_html = open('/Users/gansaikhanshur/TESTING/webScraping/1000.html').read()
html = BeautifulSoup(raw_html, 'html.parser')

# output: <img src="../../files/heb-anc-sm-pey.jpg"/>
imgs = html.findAll("img")
for image in imgs:
    # print image source
    if "jpg" in str(image):
        print(image)

# output: <font face="arial" size="+1"> unicode_hebrew_text </font>
font = html('font', face="arial", size="+1")
for f in font:
    continue

# output: <a href="audio/   505 .mp3"><img border="0" height="25" src="../../files/icon_audio.gif" width="25"/></a>
mp3file = html.findAll(href=True)
for mp3 in mp3file:
    if "mp3" in str(mp3):
        continue

如您所见，我的代码并不能真正完成这项工作。最后，我想获取 URL 中每个单词的信息，并将其保存为文本文件或 json 文件，无论哪个更容易。

例如图片：URLsOfImages，希伯来语：txt，发音：txt，URLtoAudio：txt

对于下一个单词，以此类推。

Answer 1

您要抓取的网页有点棘手。

现在，我为什么这么说？问题是，网页并没有过多地使用 div 标签或类似的东西来将单个希伯来语单词及其相关数据放在一个块中（让我称之为希伯来语单词及其相关数据，这里是一个希伯来语块）。但是，如果您关注页面源代码，那么您会注意到每个希伯来语块都从一个看起来像这样的锚标记开始：<a name="505"></a>。值“505”将根据每个希伯来语块的强数而变化。所以，我试着把它作为每个希伯来语块的起点。现在在希伯来语块中，您可以使用扬声器图标<a href="audio/ 505 .mp3"><img src="../../files/icon_audio.gif" width="25" height="25" border="0"></a> 来表示希伯来语块的结束点。

牢记上述想法，我想出了这个：

from bs4 import BeautifulSoup
from bs4 import NavigableString, Tag 
import requests
import re

def make_soup(url):
    html = requests.get(url, verify = False)
    soup = BeautifulSoup(html.text, 'html.parser')
    return soup


url = "http://www.ancient-hebrew.org/m/dictionary/1000.html"
soup = make_soup(url)

points = soup.find_all("a", attrs = {"name" : re.compile("[0-9]+")}) 
#This will help you locate the starting points of the Hebrew block

for point in points:
    point_name = point['name'] 
    images = [] #Will hold the links for the Old Hebrew images
    text = "" #Will hold the immediate next Unicode Hebrew text
    usage = "" #Will hold the usage/meaning data
    mp3 = "" #Will hold the link to audio file

    temp = point.next_sibling 
    #'temp' variable will be used as a navigator to extract the data within the Hebrew block 

    while(True):
        #If 'temp' is a HTML 'Tag' by nature, then it'll contain the Unicode Hebrew text and links for images& audio
        if(type(temp) is Tag): 
            if(temp.name == 'img'):        
                #If it's an 'img' tag, then put it to 'images' list 
                images.append(temp['src'])
            elif(temp.name == 'font'):
                #If it's a 'font' tag, then put it in 'text'
                text = temp.string
            elif(temp.name == 'a'):
                #If it's 'a' tag, then it contains the link to audio file
                mp3 = temp['href']  
                #Link to audio file will mark the end of the Hebrew block. So, 'break' out
                break
        elif(type(temp) is NavigableString):
            #If 'temp' acquires a 'NavigableString' instead of a 'Tag' then it's usage/meaning
            usage += temp
        #Move to the next item within the Hebrew block
        temp = temp.next_sibling

    print([point_name, images, mp3, text, usage])