从网页中提取 URL 链接答案

【问题标题】：Extract URL link(s) from Web page从网页中提取 URL 链接
【发布时间】：2019-08-19 05:45:34
【问题描述】：

我是网络抓取的新手。我正在尝试使用 python 从https://www.clinicaltrialsregister.eu 中提取数据，使用关键字“急性髓性白血病”、“慢性髓性白血病”、“急性淋巴细胞白血病”来提取以下信息 - EudraCT 编号、试验状态、试验的完整标题、名称赞助商、国家、正在调查的医疗状况、参与试验的调查员网络。

我正在尝试从每个链接中收集 URL，然后转到每个页面并提取信息，但我没有得到正确的链接。我想要像“https://www.clinicaltrialsregister.eu/ctr-search/trial/2014-000526-37/DE”这样的网址，但得到

 '/ctr-search/trial/2014-000526-37/DE',
 '/ctr-search/trial/2006-001777-19/NL',
 '/ctr-search/trial/2006-001777-19/BE',
 '/ctr-search/trial/2007-000273-35/IT',
 '/ctr-search/trial/2011-005934-20/FR',
 '/ctr-search/trial/2006-004950-25/GB',
 '/ctr-search/trial/2009-017347-33/DE',
 '/ctr-search/trial/2012-000334-19/IT',
 '/ctr-search/trial/2012-001594-93/FR',
 '/ctr-search/trial/2012-001594-93/results',
 '/ctr-search/trial/2007-003103-12/DE',
 '/ctr-search/trial/2006-004517-17/FR',
 '/ctr-search/trial/2013-003421-28/DE',
 '/ctr-search/trial/2008-002986-30/FR',
 '/ctr-search/trial/2008-002986-30/results',
 '/ctr-search/trial/2013-000238-37/NL',
 '/ctr-search/trial/2010-018418-53/FR',
 '/ctr-search/trial/2010-018418-53/NL',
 '/ctr-search/trial/2010-018418-53/HU',
 '/ctr-search/trial/2010-018418-53/DE',
 '/ctr-search/trial/2010-018418-53/results',
 '/ctr-search/trial/2006-006852-37/DE',
 '/ctr-search/trial/2006-006852-37/ES',
 '/ctr-search/trial/2006-006852-37/AT',
 '/ctr-search/trial/2006-006852-37/CZ',
 '/ctr-search/trial/2006-006852-37/NL',
 '/ctr-search/trial/2006-006852-37/SK',
 '/ctr-search/trial/2006-006852-37/HU',
 '/ctr-search/trial/2006-006852-37/BE',
 '/ctr-search/trial/2006-006852-37/IT',
 '/ctr-search/trial/2006-006852-37/FR',
 '/ctr-search/trial/2006-006852-37/GB',
 '/ctr-search/trial/2008-000664-16/IT',
 '/ctr-search/trial/2005-005321-63/IT',
 '/ctr-search/trial/2005-005321-63/results',
 '/ctr-search/trial/2011-005023-40/GB',
 '/ctr-search/trial/2010-022446-24/DE',
 '/ctr-search/trial/2010-019710-24/IT',

尝试的代码 -

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.clinicaltrialsregister.eu/ctr-search/search?query=acute+myeloid+leukemia&page=1')
soup = BeautifulSoup(page.text, 'html.parser')

#links = [a['href'] for a in soup.find_all('a', href=True) if a.text]


#links_with_text = []
#for a in soup.find_all('a', href=True): 
#    if a.text: 
#        links_with_text.append(a['href'])

links = [a['href'] for a in soup.find_all('a', href=True)]

输出-

'/help.html',
 '/ctr-search/search',
 '/joiningtrial.html',
 '/contacts.html',
 '/about.html',
 '/about.html',
 '/whatsNew.html',
 '/dataquality.html',
 '/doc/Sponsor_Contact_Information_EUCTR.pdf',
 '/natauthorities.html',
 '/links.html',
 '/about.html',
 '/doc/How_to_Search_EU_CTR.pdf#zoom=100,0,0',
 'javascript:void(0)',
 'javascript:void(0)',
 'javascript:void(0)',
 'javascript:void();',
 '#tabs-1',
 '#tabs-2',
 '&page=2',
 '&page=3',
 '&page=4',
 '&page=5',
 '&page=6',
 '&page=7',
 '&page=8',
 '&page=9',
 '&page=2',
 '&page=19',
 '/ctr-search/trial/2014-000526-37/DE',
 '/ctr-search/trial/2006-001777-19/NL',
 '/ctr-search/trial/2006-001777-19/BE',
 '/ctr-search/trial/2007-000273-35/IT',
 '/ctr-search/trial/2011-005934-20/FR',
 '/ctr-search/trial/2006-004950-25/GB',
 '/ctr-search/trial/2009-017347-33/DE',
 '/ctr-search/trial/2012-000334-19/IT',
 '/ctr-search/trial/2012-001594-93/FR',
 '/ctr-search/trial/2012-001594-93/results',
 '/ctr-search/trial/2007-003103-12/DE',
 '/ctr-search/trial/2006-004517-17/FR',
 '/ctr-search/trial/2013-003421-28/DE',
 '/ctr-search/trial/2008-002986-30/FR',
 '/ctr-search/trial/2008-002986-30/results',
 '/ctr-search/trial/2013-000238-37/NL',
 '/ctr-search/trial/2010-018418-53/FR',
 '/ctr-search/trial/2010-018418-53/NL',
 '/ctr-search/trial/2010-018418-53/HU',
 '/ctr-search/trial/2010-018418-53/DE',
 '/ctr-search/trial/2010-018418-53/results',
 '/ctr-search/trial/2006-006852-37/DE',
 '/ctr-search/trial/2006-006852-37/ES',
 '/ctr-search/trial/2006-006852-37/AT',
 '/ctr-search/trial/2006-006852-37/CZ',
 '/ctr-search/trial/2006-006852-37/NL',
 '/ctr-search/trial/2006-006852-37/SK',
 '/ctr-search/trial/2006-006852-37/HU',
 '/ctr-search/trial/2006-006852-37/BE',
 '/ctr-search/trial/2006-006852-37/IT',
 '/ctr-search/trial/2006-006852-37/FR',
 '/ctr-search/trial/2006-006852-37/GB',
 '/ctr-search/trial/2008-000664-16/IT',
 '/ctr-search/trial/2005-005321-63/IT',
 '/ctr-search/trial/2005-005321-63/results',
 '/ctr-search/trial/2011-005023-40/GB',
 '/ctr-search/trial/2010-022446-24/DE',
 '/ctr-search/trial/2010-019710-24/IT',
 'javascript:void(0)',
 '&page=2',
 '&page=3',
 '&page=4',
 '&page=5',
 '&page=6',
 '&page=7',
 '&page=8',
 '&page=9',
 '&page=2',
 '&page=19',
 'https://servicedesk.ema.europa.eu',
 '/disclaimer.html',
 'http://www.ema.europa.eu',
 'http://www.hma.eu'

【问题讨论】：

只需要在解析时将所需的 url 添加到您的结果中。
我做到了。我正在使用clinicaltrialsregister.eu/ctr-search/… 进行解析，但我无法理解如何从每个 ID 中收集正确的链接。

标签： python-3.x web-scraping

【解决方案1】：

正如我所说，您可以通过将 url 的所需部分连接到每个结果来实现这一点。

试试这个代码：

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.clinicaltrialsregister.eu/ctr-search/search?query=acute+myeloid+leukemia&page=1')
soup = BeautifulSoup(page.text, 'html.parser')

links =  ["https://www.clinicaltrialsregister.eu" + a['href'] for a in soup.find_all('a', href=True)]

【讨论】：

【解决方案2】：

此脚本会遍历搜索结果的所有页面，并尝试查找相关信息。

需要添加完整的url，而不仅仅是https://www.clinicaltrialsregister.eu。

import requests
from bs4 import BeautifulSoup

base_url = 'https://www.clinicaltrialsregister.eu/ctr-search/search?query=acute+myeloid+leukemia'

url = base_url + '&page=1'
soup = BeautifulSoup(requests.get(url).text, 'lxml')

page = 1
while True:
    print('Page no.{}'.format(page))
    print('-' * 160)
    print()

    for table in soup.select('table.result'):

        print('EudraCT Number:    ', end='')
        for span in table.select('td:contains("EudraCT Number:")'):
            print(span.get_text(strip=True).split(':')[1])

        print('Full Title:        ', end='')
        for td in table.select('td:contains("Full Title:")'):
            print(td.get_text(strip=True).split(':')[1])

        print('Sponsor Name:      ', end='')
        for td in table.select('td:contains("Sponsor Name:")'):
            print(td.get_text(strip=True).split(':')[1])

        print('Trial protocol:    ', end='')
        for a in table.select('td:contains("Trial protocol:") a'):
            print(a.get_text(strip=True), end=' ')
        print()

        print('Medical condition: ', end='')
        for td in table.select('td:contains("Medical condition:")'):
            print(td.get_text(strip=True).split(':')[1])

        print('-' * 160)

    next_page = soup.select_one('a:contains("Next»")')
    if next_page:
        soup = BeautifulSoup(requests.get(base_url + next_page['href']).text, 'lxml')
        page += 1
    else:
        break

打印：

Page no.1
----------------------------------------------------------------------------------------------------------------------------------------------------------------

EudraCT Number:    2014-000526-37
Full Title:        An Investigator-Initiated Study To Evaluate Ara-C and Idarubicin in Combination with the Selective Inhibitor Of Nuclear Export (SINE) 
Selinexor (KPT-330) in Patients with Relapsed Or Refractory A...
Sponsor Name:      GSO Global Clinical Research B.V.
Trial protocol:    DE 
Medical condition: Patients with relapsed/refractory Acute Myeloid Leukemia (AML)
----------------------------------------------------------------------------------------------------------------------------------------------------------------
EudraCT Number:    2006-001777-19
Full Title:        A Phase II multicenter study to assess the tolerability and efficacy of the addition of Bevacizumab to standard induction therapy in AML and 
high risk MDS above 60 years.
Sponsor Name:      HOVON foundation
Trial protocol:    NL BE 
Medical condition: Acute myeloid leukaemia (AML), AML FAB M0-M2 or M4-M7;
diagnosis with refractory anemia with excess of blasts (RAEB) or refractory anemia with excess of blasts in transformation (RAEB-T) with an IP...
----------------------------------------------------------------------------------------------------------------------------------------------------------------
EudraCT Number:    2007-000273-35
Full Title:        A Phase II, Open-Label, Multi-centre, 2-part study to assess the Safety, Tolerability, and  Efficacy of Tipifarnib Plus Bortezomib in the Treatment of Newly Diagnosed Acute  Myeloid Leukemia  AML  ...
Sponsor Name:      AZIENDA OSPEDALIERA DI BOLOGNA POLICLINICO S. ORSOLA M. MALPIGHI
Trial protocol:    IT 
Medical condition: Acute Myeloid Leukemia
----------------------------------------------------------------------------------------------------------------------------------------------------------------

...and so on.

【讨论】：

感谢您的代码我在运行代码时收到此错误。 NotImplementedError：仅实现以下伪类：nth-of-type。
@DipankarNeogi 确保您运行的是最新版本的bs4。我在版本beautifulsoup4==4.8.0
现在我收到了这个错误——AttributeError: 'LXMLTreeBuilder' object has no attribute 'initialize_soup'
@DipankarNeogi 你安装了lxml 模块吗？ lxml==4.3.4 或者您可以将 lxml 解析器替换为 html.parser
我试过 lxml 和 html。使用 html.parser 我得到了同样的错误。 “HTMLParserTreeBuilder”对象没有属性“initialize_soup”