Beautifulsoup 代码中的 getchildren() 到 LXML

Question

我必须转换一些 beautifulsoup 代码。 基本上我想要的只是获取正文节点的所有子节点并选择具有文本并存储它们。 这是 bs4 的代码：

def get_children(self, tag, dorecursive=False):

    children = []
    if not tag :
        return children

    for t in tag.findChildren(recursive=dorecursive):
        if t.name in self.text_containers \
                and len(t.text) > self.min_text_length \
                and self.is_valid_tag(t):
            children.append(t)
    return children

这很好用 当我用 lxml lib 尝试这个时，孩子是空的：

def get_children(self, tag, dorecursive=False):

    children = []
    if not tag :
        return children

    tags = tag.getchildren()
    for t in tags:
        #print(t.tag)
        if t.tag in self.text_containers \
                and len(t.tail) > self.min_text_length \
                and self.is_valid_tag(t):
            children.append(t)
    return children

有什么想法吗？

Answer 1

代码：

import lxml.html
import requests


class TextTagManager:
    TEXT_CONTAINERS = {
        'li',
        'p',
        'span',
        *[f'h{i}' for i in range(1, 6)]
    }
    MIN_TEXT_LENGTH = 60

    def is_valid_tag(self, tag):
        # put some logic here
        return True

    def get_children(self, tag, recursive=False):
        children = []

        tags = tag.findall('.//*' if recursive else '*')
        for t in tags:
            if (t.tag in self.TEXT_CONTAINERS and
                    t.text and
                    len(t.text) > self.MIN_TEXT_LENGTH and
                    self.is_valid_tag(t)):
                children.append(t)
        return children


manager = TextTagManager()

url = 'https://en.wikipedia.org/wiki/Comparison_of_HTML_parsers'
html = requests.get(url).text
doc = lxml.html.fromstring(html)

for child in manager.get_children(doc, recursive=True):
    print(child.tag, ' -> ', child.text)

输出：

li  ->  HTML traversal: offer an interface for programmers to easily access and modify of the "HTML string code". Canonical example: 
li  ->  HTML clean: to fix invalid HTML and to improve the layout and indent style of the resulting markup. Canonical example: 

---

.getchildren() 返回所有 直接 子级。如果你想有一个递归选项，你可以使用.findall()：

tags = tag.findall('.//*' if recursive else '*')

This answer 应该可以帮助您了解.//tag 和tag 之间的区别。