在python中从HTML中提取URL

Question

当您看到此网页的 HTML 版本时，我正在尝试获取包含不同 url 的列表：

https://www.renfe.com/es/es/cercanias/cercanias-valencia/lineas

我尝试了几种不同的方法，但它们都不起作用。

第一次尝试

from bs4 import BeautifulSoup
import requests
import html
import urllib
import json
import re

url = 'https://www.renfe.com/es/es/cercanias/cercanias-valencia/lineas'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
links = soup.find_all('div', class_ = "rftabdetailline accordion aem-GridColumn aem-GridColumn--default--12")

链接包含以下内容：

 [<div class="rftabdetailline accordion aem-GridColumn aem-GridColumn--default--12">
 <!-- rf-tab-detail-line en resto de modos -->
 <rf-tab-detail-line content='[{"color":"120,180,225","name":"C1","active":"true","stations":"València Nord \u2013 Gandía","url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_1.html"},{"color":"245,150,40","name":"C2","active":"false","stations":"València Nord \u2013 Xàtiva \u2013 Moixent","url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_1591014181985.html"},{"color":"125,37,130","name":"C3","active":"false","stations":"València Sant Isidre \u2013 Buñol \u2013 Utiel","url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_1591014184209.html"},{"color":"215,0,30","name":"C4","active":"false","stations":"València Sant Isidre \u2013 Xirivella L\u2019Alter","url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_1591014185974.html"},{"color":"0,139,41","name":"C5","active":"false","stations":"València Nord \u2013 Caudiel","url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_1591014187588.html"},{"color":"15,50,135","name":"C6","active":"false","stations":"València Nord \u2013 Castelló","url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_1591014189921.html"},{"color":"150,100,40","name":"ER02","active":"false","stations":"Castelló - Vinaròs","url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_1598612779629.html"}]' title-text="Seleccione una línea:">
 </rf-tab-detail-line>
 </div>]

在里面，你可以看到我想要的部分：例如， * "url":"/content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline /item_1.html" *.我想在列表中获取所有不同的 /content/renfe/es/es/cercanias/cercanias-valencia/lineas/jcr:content/root/responsivegrid/rftabdetailline/item_WHATEVER.html。为此，我尝试了提取并使用正则表达式，但没有成功。

第二次尝试

按照此问题的答案中显示的步骤进行操作 Extractinf info form HTML that has no tags 我得到了下一段代码：

import requests
import html
import json

url = 'https://www.renfe.com/content/renfe/es/es/cercanias/cercanias-valencia/lineas'
response = requests.get(url)
data = response.text  # get data from site
raw_list = data.split("'")[8]  # extract attributes
json_list = html.unescape(raw_list)  # decode html symbols
parsed_list = json.loads(json_list)  # parse json 

我认为它会起作用，因为它产生的输出相似，但是在定义 parsed_list 时会返回下一个错误：

• JSONDecodeError：预期值：第 1 行第 1 列 (char 0)*

有人有意见吗？ 先谢谢大家了！！！

Answer 1

我会改为使用 css 属性 = 值选择器来定位包含该数据的单个元素，因为它在阅读时更直观。然后您只需要提取content 属性并使用json 库过滤处理url 键值对。

import json
import requests
from bs4 import BeautifulSoup

url = 'https://www.renfe.com/es/es/cercanias/cercanias-valencia/lineas'
page = requests.get(url)
soup = BeautifulSoup(page.content, 'html.parser')
data = json.loads(soup.select_one('[title-text="Seleccione una línea:"]')['content'])
links = [i['url'] for i in data]