Nokogiri 解析元素之间的剪切内容

Question

我用谷歌搜索了一半的互联网搜索帮助。

那么，我需要什么：

我有这样的解析 HTML 结构：

<div class="foo">
  <div class='bar' dir='ltr'>
    <div id='p1' class='par'>
      <p class='sb'>
        <span id='dc_1_1' class='dx'>
          <a href='/bar32560'>1</a>
        </span>
        Neque porro 
        <a href='/xyz' class='mr'>+</a>
        quisquam est 
        <a href='/xyz' class='mr'>+</a>
        qui. 
      </p>
    </div>
    <div id='p2' class='par'>
      <p class='sb'>
        <span id='dc_1_2' class='dx'>
          <a href='/foo12356'>2</a>
        </span>
        dolorem ipsum 
        <a href='/xyz' class='mr'>+</a>
        quia dolor sit amet, 
        <a href='/xyz' class='mr'>+</a>
        consectetur, adipisci velit.
      </p>
    </div>
    <div id='p3' class='par'>
      <p class='sb'>
        <span id='dc_1_3' class='dx'>
          <a href='/foobar4586'>3</a>
        </span>
        Neque porro quisquam 
        <a href='/xyz' class='mr'>+</a>
        est qui dolorem ipsum quia dolor sit 
        <a href='/xyz' class='mr'>+</a>
        amet, t.
        <a href='/xyz' class='mr'>+</a>
        <span id='dc_1_4' class='dx'>
          <a href='/barefoot4135'>4</a>
        </span>
        consectetur, 
        <a href='/xyz' class='mr'>+</a>
        adipisci veli.
        <span id='dc_1_5' class='dx'>
          <a href='/barfoo05123'>5</a>
       </span>
       Neque porro 
       <a href='/xyz' class='mr'>+</a>
       quisquam est
       <a href='/xyz' class='mr'>+</a>
       qui.
     </p>
   </div>
 </div>
</div>

我需要什么（英文）：抓取每个段落但我需要最终抓取的文本对象内容形式：

scraped_body 1 => 1 Neque porro quisquam est qui.
scraped_body 2 => 2 dolorem ipsum quia dolor sit amet, consectetur, adipisci velit
scraped_body 3 => 3 Neque porro quisquam est qui dolorem ipsum quia dolor sit amet, t.
scraped_body 4 => 4 consectetur, adipisci veli.
scraped_body 5 => 5 Neque porro quisquam est qui.

编写我现在使用的代码：

page = Nokogiri::HTML(open(url))
x = page.css('.mr').remove
x.xpath("//div[contains(@class, 'par')]").map do |node|
  body = node.text
end

我的结果是这样的：

scraped_body 1 => 1 Neque porro quisquam est qui.
scraped_body 2 => 2 dolorem ipsum quia dolor sit amet, consectetur, adipisci velit
scraped_body 3 => 3 Neque porro quisquam est qui dolorem ipsum quia dolor sit amet, t. 4 consectetur, adipisci veli. 5 Neque porro quisquam est qui.

所以这会从 div 段落类“par”中抓取整个文本。我需要在每个跨度之后用他的内容 - 数字刮掉整个文本。或者在每个跨度之前剪切那些 div。

我需要类似的东西：

SPAN.text + P.text - a.mr

我不知道……该怎么做

请帮助我进行此解析。我需要在每个跨度之后/之前刮擦 - 我猜。

请帮忙，我已经尝试了所有我找到的东西。

---

编辑鸭子@Duck1337：

我使用以下代码：

def verses
    page = Nokogiri::HTML(open(url))
    i=0
    x = page.css("p").text.gsub("+", " ").split.join(" ").gsub(". ", ". HAM").split(" HAM").map do |node|
    i+=1
    body = node
    VerseSource.new(body, book_num, number, i)
  end
end

我需要这个，因为我用文本解析一个大网站。还有几个方法。所以我的最终输出看起来像：

Saved record with: book: 1, chapter: 1, verse: 1, body: 1 Neque porro quisquam est qui.

但是，如果我有多个句子的单个 werse，那么您的代码将其拆分为每个句子。所以这是一个很大的分歧。

例如：

    <div id='p1' class='par'>
      <p class='sb'>
        <span id='dc_1_3' class='dx'>
          <a href='/foobar4586'>1</a>
        </span>
        Neque porro quisquam. Est qui dolorem
        <a href='/xyz' class='mr'>+</a>
        <span id='dc_1_3' class='dx'>
          <a href='/foobar4586'>2</a>
        </span>
        est qui dolorem ipsum quia dolor sit. 
        <a href='/xyz' class='mr'>+</a>
        amet, t.

你的代码是这样拆分的：

Saved record with: book: 1, chapter: 1, verse: 1, body: 1 Neque porro quisquam.
Saved record with: book: 1, chapter: 1, verse: 2, body: Est qui dolorem
Saved record with: book: 1, chapter: 1, verse: 3, body: 2 est qui dolorem ipsum quia dolor sit.

希望你是我的意思。真的非常感谢你。如果你能修改它，那就太好了！

---

编辑：@KARDEIZ

感谢您的回答！当我在我的方法中使用你的代码时：它解析了非常随机的东西。

def verses
  page = Nokogiri::HTML(open(url))
  i=0
  #page.css(".mr").remove
  page.xpath("//div[contains(@class, 'par')]//span").map do |node|
    node.content.strip.tap do |out|
      while nn = node.next
        break if nn.name == 'span'
        out << ' ' << nn.content.strip if nn.text? && !nn.content.strip.empty?
        node = nn
      end
    end
    i+=1
    body = node
    VerseSource.new(body, book_num, number, i)
  end
end

输出如下：

Saved record with: book: 1, chapter: 1, verse: 1, body:  <here is last part of last sentence in first paragraph after "+" sign(href) and before last "+"(href)>
Saved record with: book: 1, chapter: 1, verse: 2, body:  <here is last part of last sentence in second paragraph after "+" sign(href) and before last "+"(href)>
Saved record with: book: 1, chapter: 1, verse: 3, body:
Saved record with: book: 1, chapter: 1, verse: 4, body:
Saved record with: book: 1, chapter: 1, verse: 5, body:  <here is last sentence in third paragraph. It is after last "+" in this paragraph and have no more "+" signs(href)

如你所见，我不知道它是怎么弄得这么乱的；] 你能用它做更多的事情吗？非常感谢！

---

问候！

Answer 1

尝试类似：

x.xpath("//div[contains(@class, 'par')]//span").map do |node|
  out = node.content.strip
  if following = node.at_xpath('following-sibling::text()')
    out << ' ' << following.content.strip
  end
  out
end

following-sibling::text() XPATH 将获取 span 之后的第一个文本节点。

编辑

我认为这是你想要的：

html.xpath("//div[contains(@class, 'par')]//span").map do |node|
  node.content.strip.tap do |out|
    while nn = node.next
      break if nn.name == 'span'
      out << ' ' << nn.content.strip if nn.text? && !nn.content.strip.empty?
      node = nn
    end
  end  
end

输出：

[
  "1 Neque porro quisquam est qui.",
  "2 dolorem ipsum quia dolor sit amet, consectetur, adipisci velit.",
  "3 Neque porro quisquam est qui dolorem ipsum quia dolor sit amet, t.",
  "4 consectetur, adipisci veli.",
  "5 Neque porro quisquam est qui."
]

也可以使用纯 XPath（参见 XPath axis, get all following nodes until）执行此操作，但从编码角度来看，此解决方案更简单。

编辑 2

试试这个：

def verses
  page = Nokogiri::HTML(open(url))
  i=0
  page.xpath("//div[contains(@class, 'par')]//span").map do |node|
    body = node.content.strip.tap do |out|
      while nn = node.next
        break if nn.name == 'span'
        out << ' ' << nn.content.strip if nn.text? && !nn.content.strip.empty?
        node = nn
      end
    end
    i+=1
    VerseSource.new(body, book_num, number, i)
  end
end

Answer 2

我在我的桌面上将您的输入保存为“temp.html”。

require 'open-uri'
require 'nokogiri'

$page_html = Nokogiri::HTML.parse(open("/home/user/Desktop/temp.html"))

output = $page_html.css("p").text.gsub("+", " ").split.join(" ").gsub(". ", ". HAM").split(" HAM")

# I found the pattern ". " in every line, so i replaced ". " with (". HAM")
# I did that by using gsub(". ", ". HAM") this means replace ". " with ". HAM"

# then i split up the string with " HAM" so it preserved the "." in each item in the array


output = ["1 Neque porro quisquam est qui.", "2 dolorem ipsum quia dolor sit amet, consectetur, adipisci velit.", "3 Neque porro quisquam est qui dolorem ipsum quia dolor sit amet, t.", "4 consectetur, adipisci veli.", "5 Neque porro quisquam est qui."]

编辑：

 %w[nokogiri open-uri].each{|gem| require gem}     

 $url = "/home/user/Desktop/temp.html"
 def verses
     page = Nokogiri::HTML(open($url))
     i=0
     x = page.css("p").text.gsub("+", " ").split.join(" ").gsub(". ", ".    HAM").split(" HAM") do |node|
         i+=1
         body = node
         VerseSource.new(body, book_num, number, i)
    end
 end

Answer 3

require 'nokogiri'

your_html =<<END_OF_HTML
<your html here>
END_OF_HTML

doc  = Nokogiri::HTML(your_html)
text_nodes = doc.xpath("//div[contains(@class, 'par')]/p/child::text()")

results = text_nodes.reject do |text_node| 
  text_node.text.match /\A \s+ \z/x  #Eliminate whitespace nodes
end

results.each_with_index do |node, i|
  puts "scraped_body#{i+1} => #{node.text.strip}"
end


--output:--
scraped_body1 => Neque porro quisquam est qui.
scraped_body2 => dolorem ipsum quia dolor sit amet, consectetur, adipisci velit.
scraped_body3 => Neque porro quisquam est qui dolorem ipsum quia dolor sit amet, t.
scraped_body4 => consectetur, adipisci veli.
scraped_body5 => Neque porro quisquam est qui.

新 html 的答案：

require 'nokogiri'

html = <<END_OF_HTML
your new html here
END_OF_HTML

html_doc  = Nokogiri::HTML(html)
current_group_number = nil
non_ws_text = []  #non_whitespace_text for each group

html_doc.css("div.par > p").each do |p|   #p's that are direct children of <div class="par">
  p.xpath("./node()").each do |node|  #All Text and Element nodes that are direct children of p tag.
    case node
    when  Nokogiri::XML::Element
      if node.name == 'span'
        node.xpath(".//a").each do |a|  #Step through all the <a> tags inside the <span>
          md = a.text.match(/\A (\d+) \z/xm)  #Check for numbers

          if md  #Then found a number, so it's the start of the next group
            if current_group_number  #then print the results for the current group
              print "scraped_body #{current_group_number} => "
              puts "#{current_group_number} #{non_ws_text.join(' ')}"
              non_ws_text = []
            end
            current_group_number = md[1] #Record the next group number 
            break  #Only look for the first <a> tag containing a number
          end

        end
      end

    when Nokogiri::XML::Text
      text = node.text
      non_ws_text << text.strip if text !~ /\A \s+ \z/xm 
    end

  end
end

#For the last group: 
print "scraped_body #{current_group_number} => "
puts "#{current_group_number} #{non_ws_text.join(' ')}"

--output:--
scraped_body 1 => 1 Neque porro quisquam est qui.
scraped_body 2 => 2 dolorem ipsum quia dolor sit amet, consectetur, adipisci velit.
scraped_body 3 => 3 Neque porro quisquam est qui dolorem ipsum quia dolor sit amet, t.
scraped_body 4 => 4 consectetur, adipisci veli.
scraped_body 5 => 5 Neque porro quisquam est qui.