通过 XPATH 通过 LXML 获取元素答案

【问题标题】：Getting an element via LXML by XPATH通过 XPATH 通过 LXML 获取元素
【发布时间】：2020-08-14 03:22:23
【问题描述】：

我正在 Python 库 discord.py 中编写 Discord 机器人。

我不需要帮助，但需要从网站上抓取一些信息。

    @commands.command(aliases=["rubyuserinfo"])
    async def rubyinfo(self, ctx, input):
        HEADERS = {
            'User-Agent' : 'Magic Browser'
        }

        url = f'https://rubyrealms.com/user/{input}/'

        async with aiohttp.request("GET", url, headers=HEADERS) as response:
            if response.status == 200:
                print("Site is working!")
                content = await response.text()
                soup = BeautifulSoup(content, "html.parser")
                page = requests.get(url)
                tree = html.fromstring(page.content)
                stuff = tree.xpath('/html/body/div[4]/div/div[3]/div[3]/div/div[2]/div[1]/div[2]/div/p')
                print(stuff)
            else:
                print(f"The request was invalid\nStatus code: {response.status}")

我正在寻找的网站是“https://rubyrealms.com/user/{input}/”，在运行 h!rubyinfo USERNAME 时会给出输入，将链接更改为 https://rubyrealms.com/user/username/。

在网站上我想得到的是他们的 BIO，它的 XPATH 为

"//*[@id="content-wrap"]/div[3]/div[3]/div/div[2]/div[1]/div[2]/div/p"

元素在哪里：

<p class="margin-none font-color">
Hey! My name is KOMKO190, you maybe know me from the forums or discord. I am a programmer, I know a bit of JavaScript, small portion of C++, Python and html/css. Mostly python. My user ID is 7364. ||| 5th owner of Space Helmet :) </p>

对我将如何刮它有任何帮助吗？我的机器人给出的唯一响应是“[]”

【问题讨论】：

标签： python xpath web-scraping beautifulsoup lxml

【解决方案1】：

将您的 XPath 表达式更改为相对表达式：

from lxml import html
import requests
page = requests.get('https://www.rubyrealms.com/user/KOMKO190/')
tree = html.fromstring(page.content)
stuff = tree.xpath('normalize-space(//h3[.="Bio"]/following-sibling::p/text())')
print (stuff)

输出：

Hey! My name is KOMKO190, you maybe know me from the forums or discord. I am a programmer, I know a bit of JavaScript, small portion of C++, Python and html/css. Mostly python. My user ID is 7364. ||| 5th owner of Space Helmet :)

【讨论】：

【解决方案2】：

下面的怎么样，使用 .select() 方法

from bs4 import BeautifulSoup

html = '<p class="margin-none font-color">Hey! My name is KOMKO190 :) </p>'

soup = BeautifulSoup(html, features="lxml")
element = soup.select('p.margin-none')[0]
print(element.text)

打印出来

Hey! My name is KOMKO190 :)

【讨论】：

【解决方案3】：

from bs4 import BeautifulSoup as bs

url = 'https://rubyrealms.com/user/username/'

session = requests.Session()
request = session.get(url=url)
if request.status_code == 200:
    soup = bs(request.text, 'lxml')
    print(soup.find('p', class_='margin-none font-color').text)
else:
    print(request.status_code)

你需要安装

pip install lxml
pip install beautifulsoup4

【讨论】：