将html写入csv时出现奇怪的格式答案

【问题标题】：Wierd formating when writing html to csv将html写入csv时出现奇怪的格式
【发布时间】：2019-07-21 04:04:26
【问题描述】：

我希望有人能帮我解决这个问题..

我有一个 python 脚本，可以从 html 文档中的标签中查找文本，我需要将它们以这种格式写入 csv 文件：

Nessus、ID、文本

但现在我把它放在一条垂直线上:(

我的代码：

from bs4 import BeautifulSoup
import re
import csv

with open('xx01_med_dansk') as fp:
    soup = BeautifulSoup(fp, 'html5lib')
#    print(soup.prettify())

    with open('dk_snip.csv', 'w') as f:
        wr = csv.writer(f)

        var1 = soup.find('li', text = re.compile('Scan vendor:'), attrs = {'class' : 'property_name'})
        var2 = soup.find('li', text = re.compile('Vendor ID:'), attrs = {'class' : 'property_name'})

        vendor = var1.find_next('li')
        final = vendor.string

        vend_id = var2.find_next('li')
        final2 = vend_id.text

        for dk_desc in soup.find_all(re.compile("textarea")):
            final3 = dk_desc.text

        to_csv = final+final2+final3
        wr.writerows(to_csv)

我不太确定如何格式化数据以在 csv 文件中正确输入..

HTML 文件：

<!DOCTYPE html>
<html lang="en">
 <head>
  
      <li class="property_name">
       <label for="id_194-description">
        Description:
       </label>
      </li>
      <li class="property_value">
       <textarea class="mceNoEditor" cols="40" id="id_194-description" name="194-description" rows="10" style="width:100%">According to its version, the installation of Oracle Database on the remote host is no longer supported.

Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities.</textarea>
      </li>
      <li class="property_name">
       <label for="id_194-consequence">
        Consequence:
       </label>
      </li>
      <li class="property_value">
       <textarea class="mceNoEditor" cols="40" id="id_194-consequence" name="194-consequence" rows="10" style="width:100%">The remote host is running an unsupported version of a database server.</textarea>
      </li>
      <li class="property_name">
       <label for="id_194-solution">
        Solution:
       </label>
      </li>
      <li class="property_value">
       <textarea class="mceNoEditor" cols="40" id="id_194-solution" name="194-solution" rows="10" style="width:100%">Upgrade to a version of Oracle Database that is currently supported.</textarea>
      </li>
      <li class="property_name">
       <label for="id_194-cve_id">
        Cve id:
       </label>
      </li>
      <li class="property_value">
       <textarea class="mceNoEditor" cols="40" id="id_194-cve_id" maxlength="8192" name="194-cve_id" rows="10" style="width:100%; height:80px"></textarea>
      </li>
      <input id="id_194-override" name="194-override" type="hidden" value="11953"/>
      <input id="id_194-priority" name="194-priority" type="hidden"/>
      <li class="property_name">
       Vulnerability priority
      </li>
      <li class="property_value">
       <select name="prio_194">
        <option selected="selected" value="0">
         0
        </option>
       </select>
       : Oracle Database Unsupported (Nessus)
       <br/>
      </li>
      <li class="property_name">
       Save
      </li>
      <li class="property_value">
       <input type="submit" value="Save vulnerability changes"/>
      </li>
     </ul>
    </form>
    <br style="clear:both"/>
   </div>
   <div class="box">
    <h4>
     Related vulnerabilities
    </h4>
    <hr/>
    <h5>
     Oracle Database Unsupported (Nessus)
    </h5>
    <ul>
     <li class="property_name">
      Description
     </li>
     <li class="property_value">
      According to its version, the installation of Oracle Database on the remote host is no longer supported.
      <br/>
      <br/>
      Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities.
     </li>
     <li class="property_name">
      Consequence
     </li>
     <li class="property_value">
      The remote host is running an unsupported version of a database server.
     </li>
     <li class="property_name">
      Solution
     </li>
     <li class="property_value">
      Upgrade to a version of Oracle Database that is currently supported.
     </li>
    </ul>
    <br style="clear:both"/>
   </div>
   <div class="box">
    <h4>
     Create new snippet
    </h4>
    <form action="/report/vulnerabilityEditor/?
								action=edit&amp;
								id=194&amp;
								sid=&amp;
								model=snippet" method="POST">
     <ul>
      <li class="property_name">
       <label for="id_language">
        Language:
       </label>
      </li>
      <li class="property_value">
       <select id="id_language" name="language" style="width:100%">
        <option selected="" value="1">
         Danish (DK)
        </option>
        <option value="2">
         English (EN)
        </option>
        <option value="3">
         Icelandic (IS)
        </option>
       </select>
      </li>
      <input id="id_vulnerability" name="vulnerability" type="hidden" value="194"/>
      <li class="property_name">
       <label for="id_title">
        Title:
       </label>
      </li>
      <li class="property_value">
       <input id="id_title" maxlength="100" name="title" style="width:100%" type="text"/>
      </li>
      <li class="property_name">
       <label for="id_recommendation">
        Recommendation:
       </label>
      </li>
      <li class="property_value">
       <input id="id_recommendation" maxlength="255" name="recommendation" style="width:100%" type="text"/>
      </li>
      <li class="property_name">
       <label for="id_snippet">
        Snippet:
       </label>
      </li>
      <li class="property_value">
       <textarea cols="40" id="id_snippet" name="snippet" rows="10" style="width:100%"></textarea>
      </li>
      <li class="property_name">
       Scan type
      </li>
      <li class="property_value">
       <select multiple="multiple" name="scan_type" size="6" style="width:100%">
        <option selected="selected" value="5">
         COMPANY PCI
        </option>
        <option selected="selected" value="7">
         Other
        </option>
        <option selected="selected" value="8">
         Firewall Audit
        </option>
        <option selected="selected" value="6">
         Penetration Test
        </option>
        <option selected="selected" value="9">
         WIFI Test
        </option>
        <option selected="selected" value="10">
         APP Test
        </option>
        <option selected="selected" value="1">
         External Security Analysis
        </option>
        <option selected="selected" value="2">
         Internal Security Analysis
        </option>
        <option selected="selected" value="3">
         Web Application Test
        </option>
        <option selected="selected" value="4">
         Host Discovery Analysis
        </option>
       </select>
       -- Use ctrl to mark multiple types
      </li>
      <li class="property_name">
       Save
      </li>
      <li class="property_value">
       <input type="submit" value="Save new snippet"/>
      </li>
     </ul>
     <br style="clear:both;"/>
    </form>
   </div>
   <div class="box">
    <h4>
     Edit snippets
    </h4>
    <input id="property_vulnerability_id" type="hidden" value="194"/>
    <input id="property_url_filter_snippets" type="hidden" value="/report/filterSnippets/"/>
    <ul>
     <li class="property_name">
      Language
     </li>
     <li class="property_value">
      <select id="language" name="language">
       <option value="0">
        All
       </option>
       <option value="1">
        Danish
       </option>
       <option value="2">
        English
       </option>
       <option value="3">
        Icelandic
       </option>
      </select>
     </li>
     <li class="property_name">
      Scan Type
     </li>
     <li class="property_value">
      <select id="scantype" name="scantype">
       <option value="0">
        All
       </option>
       <option value="5">
        COMPANY PCI
       </option>
       <option value="7">
        Other
       </option>
       <option value="8">
        Firewall Audit
       </option>
       <option value="6">
        Penetration Test
       </option>
       <option value="9">
        WIFI Test
       </option>
       <option value="10">
        APP Test
       </option>
       <option value="1">
        External Security Analysis
       </option>
       <option value="2">
        Internal Security Analysis
       </option>
       <option value="3">
        Web Application Test
       </option>
       <option value="4">
        Host Discovery Analysis
       </option>
      </select>
     </li>
    </ul>
    <br style="clear:both;"/>
    <div class="snippet">
     <form action="/report/vulnerabilityEditor/?action=edit&amp;id=194&amp;sid=1290&amp;model=snippet" method="POST">
      <input id="id_1290-vulnerability" name="1290-vulnerability" type="hidden" value="194"/>
      <hr/>
      <ul>
       <li class="property_name">
        <label for="id_1290-language">
         Language:
        </label>
       </li>
       <li class="property_value">
        <select id="id_1290-language" name="1290-language" style="width:100%">
         <option value="1">
          Danish (DK)
         </option>
         <option selected="" value="2">
          English (EN)
         </option>
         <option value="3">
          Icelandic (IS)
         </option>
        </select>
       </li>
       <li class="property_name">
        <label for="id_1290-title">
         Title:
        </label>
       </li>
       <li class="property_value">
        <input id="id_1290-title" maxlength="100" name="1290-title" style="width:100%" type="text" value="Oracle Database Unsupported"/>
       </li>
       <li class="property_name">
        <label for="id_1290-recommendation">
         Recommendation:
        </label>
       </li>
       <li class="property_value">
        <input id="id_1290-recommendation" maxlength="255" name="1290-recommendation" style="width:100%" type="text" value="Upgrade to a version of Oracle Database that is currently supported."/>
       </li>
       <li class="property_name">
        <label for="id_1290-snippet">
         Snippet:
        </label>
       </li>
       <li class="property_value">
        <a href="https://cyberopswiki/index.php/How_to:_Add_figure_number_in_snippet" target="_blank">
         How to: Add figure number in snippet.
        </a>
       </li>
       <li class="property_value">
        <textarea cols="40" id="id_1290-snippet" name="1290-snippet" rows="10" style="width:100%">&lt;p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"&gt;&lt;span lang="EN-US" style="font-size: 10pt;"&gt;It has been detected, that the installed version of Oracle Application Server is&amp;nbsp;&lt;strong&gt;XXXX.&amp;nbsp;&lt;/strong&gt;This version is known to be vulnerable to a number of unspecified vulnerabilities, categorized as 'urgent'.&lt;/span&gt;&lt;/p&gt;
&lt;p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"&gt;&lt;span lang="EN-US" style="font-size: 10pt;"&gt;As this version is no longer supported for this platform, updates or patches may no longer be released, which have the consequence that vulnerabilities can not be patched, leaving the system vulnerable.&lt;/span&gt;&lt;/p&gt;
&lt;p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"&gt;&lt;span lang="EN-US" style="font-size: 10pt;"&gt;In version 10.1.2.0.2 there are, according to http://www.cvedetails.com more than 54 vulnerabilities which affects the installed version.&lt;/span&gt;&lt;/p&gt;
&lt;p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: center; line-height: normal;" align="center"&gt;&lt;strong&gt;&lt;em&gt;&lt;span lang="EN-US" style="font-size: 8pt;"&gt;Figure 1: &lt;/span&gt;&lt;/em&gt;&lt;/strong&gt;&lt;em&gt;&lt;span lang="EN-US" style="font-size: 8pt;"&gt;Oracle Application Server version.&lt;/span&gt;&lt;/em&gt;&lt;/p&gt;
&lt;p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"&gt;&lt;span lang="EN-US" style="font-size: 10pt;"&gt;More information on these vulnerabilities can be found at:&amp;nbsp;&lt;/span&gt;&lt;span style="font-size: 10pt;"&gt;&lt;a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"&gt;&lt;span lang="EN-US" style="color: blue; mso-ansi-language: EN-US;"&gt;http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html&lt;/span&gt;&lt;/a&gt;&lt;a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"&gt;&lt;span lang="EN-US" style="color: blue; mso-ansi-language: EN-US;"&gt;&amp;nbsp;&lt;/span&gt;&lt;/a&gt;&lt;/span&gt;&lt;span lang="EN-US" style="font-size: 10pt;"&gt;.&lt;/span&gt;&lt;/p&gt;
&lt;p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"&gt;&amp;nbsp;&lt;/p&gt;
&lt;p class="MsoNormal" style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; text-align: justify; line-height: normal;"&gt;&lt;span lang="EN-US" style="font-size: 10pt;"&gt;It is recommended that the installed version is updated as soon as possible to the latest version.&lt;/span&gt;&lt;/p&gt;</textarea>
       </li>
       <li class="property_name">
        Scan type
       </li>
       <li class="property_value">
        <select multiple="multiple" name="scan_type" size="6" style="width:100%">
         <option selected="selected" value="5">
          COMPANY PCI
         </option>
         <option selected="selected" value="7">
          Other
         </option>
         <option selected="selected" value="8">
          Firewall Audit
         </option>
         <option selected="selected" value="6">
          Penetration Test
         </option>
         <option selected="selected" value="9">
          WIFI Test
         </option>
         <option selected="selected" value="10">
          APP Test
         </option>
         <option selected="selected" value="1">
          External Security Analysis
         </option>
         <option selected="selected" value="2">
          Internal Security Analysis
         </option>
         <option selected="selected" value="3">
          Web Application Test
         </option>
         <option selected="selected" value="4">
          Host Discovery Analysis
         </option>
        </select>
        -- Use ctrl to mark multiple types
       </li>
       <li class="property_name">
        Update
       </li>
       <li class="property_value">
        <input type="submit" value="Update snippet"/>
       </li>
      </ul>
     </form>
     <br style="clear:both;"/>
    </div>
    <div class="snippet">
     <form action="/report/vulnerabilityEditor/?action=edit&amp;id=194&amp;sid=172&amp;model=snippet" method="POST">
      <input id="id_172-vulnerability" name="172-vulnerability" type="hidden" value="194"/>
      <hr/>
      <ul>
       <li class="property_name">
        <label for="id_172-language">
         Language:
        </label>
       </li>
       <li class="property_value">
        <select id="id_172-language" name="172-language" style="width:100%">
         <option selected="" value="1">
          Danish (DK)
         </option>
         <option value="2">
          English (EN)
         </option>
         <option value="3">
          Icelandic (IS)
         </option>
        </select>
       </li>
       <li class="property_name">
        <label for="id_172-title">
         Title:
        </label>
       </li>
       <li class="property_value">
        <input id="id_172-title" maxlength="100" name="172-title" style="width:100%" type="text" value="Forældet Oracle Application Server 10g"/>
       </li>
       <li class="property_name">
        <label for="id_172-recommendation">
         Recommendation:
        </label>
       </li>
       <li class="property_value">
        <input id="id_172-recommendation" maxlength="255" name="172-recommendation" style="width:100%" type="text"/>
       </li>
       <li class="property_name">
        <label for="id_172-snippet">
         Snippet:
        </label>
       </li>
       <li class="property_value">
        <a href="https://cyberopswiki/index.php/How_to:_Add_figure_number_in_snippet" target="_blank">
         How to: Add figure number in snippet.
        </a>
       </li>
       <li class="property_value">
        <textarea cols="40" id="id_172-snippet" name="172-snippet" rows="10" style="width:100%">&lt;p style="font-size: 13px;"&gt;Det konstateret, at den installerede version af Oracle Application Server er&amp;nbsp;&lt;strong&gt;XXXX.&amp;nbsp;&lt;/strong&gt;Denne version indeholder flere kendte samt uspecificeret s&amp;aring;rbarheder, der kategoriseres som v&amp;aelig;rende 'yderst kritiske' og 'kritiske'.&lt;/p&gt;
&lt;p style="font-size: 13px;"&gt;Da der ikke l&amp;aelig;ngere komme opdateringer til denne platform, vil disse s&amp;aring;rbarheder ikke blive udbedret, hvorfor systemet er meget udsat.&lt;/p&gt;
&lt;p style="font-size: 13px;"&gt;I version 10.1.2.0.2 findes der if&amp;oslash;lge http://www.cvedetails.com ikke mindre end 54 s&amp;aring;rbarheder, der ber&amp;oslash;rer denne version. Mere information om disse findes p&amp;aring; adressen&amp;nbsp;&lt;a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"&gt;http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html&lt;/a&gt;&lt;a href="http://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-707/version_id-26592/Oracle-Application-Server-10.1.2.0.2.html"&gt;&amp;nbsp;&lt;/a&gt;.&lt;/p&gt;
&lt;p style="font-size: 13px;"&gt;Det anbefales leverand&amp;oslash;ren af software l&amp;oslash;sningen kontakts, s&amp;aring; der hurtigst muligt kan opgraderes til en nyere, supporteret version.&amp;nbsp;&lt;/p&gt;</textarea>
       </li>

根据 Martins 的建议，我将代码修改如下：

from bs4 import BeautifulSoup
import re
import csv
import glob

def get_danish(text):
    return re.compile(r'\b({0})\b'.format(text), flags=re.IGNORECASE).search

with open('dk_snip.csv', 'w', newline='') as f_out:
    csv_out = csv.writer(f_out)
#    csv_out.writerow(["Nessus", "ID", "Descrip"])

    for filename in glob.glob('/home/rj/Documents/snip/snips/*'):
        print("Processing:", filename)

        with open(filename) as f_in:
            soup = BeautifulSoup(f_in, 'html5lib')


            var1 = soup.find('li', text = re.compile('Scan vendor:'), attrs = {'class' : 'property_name'})
            var2 = soup.find('li', text = re.compile('Vendor ID:'), attrs = {'class' : 'property_name'})

            vendor = var1.find_next('li').get_text(strip=True)
            vend_id = var2.find_next('li').get_text(strip=True)

#    rows = [[vendor, vend_id, dk_desc.get_text(strip=True)] for dk_desc in soup.find_all("textarea")[:3]]


            for textarea in soup.find_all("textarea"):
                desc = textarea.get_text(strip=True)

                if get_danish('dette'):
                    csv_out.writerows([vendor, vend_id, desc])

【问题讨论】：

可以用to_csv = final+final2+final3代替to_csv = [final, final2, final3]
这只是使每个字符逗号分隔，如下所示：", , , , , , , ,N,e,s,s,u,s,"
我认为这就是你想要实现的目标：to_csv = [(x,y,z) for x,y,z in zip(final1, final2, final3)]
不，只取每行的前 3 个字符并用逗号分隔它们：<," ",< p, ,p , , s, ,s t, ,t y, ,y l, ,l e, ,e =,5,= """",5,"""" f,7,f o,8,o n,6,n t," ",t
有很多文本区域，而不是 1。您打算如何将它们全部合并到一个列中？还是每个人都有自己的专栏？

标签： python html csv beautifulsoup

【解决方案1】：

你得到的标签需要被剥离。一种方法是使用 BeautifulSoup .get_text(strip=True) 函数。

我假设对于每个 textarea，您要重复 Nessus 和 ID 值。下面展示了如何做到这一点：

from bs4 import BeautifulSoup
import csv
import re
import glob
import random

def get_language(text):
    # This will need to be added using another library - currently random
    return random.choice(["en", "dk"])


with open('dk_snip.csv', 'w', newline='') as f_out:
    csv_out = csv.writer(f_out)
    csv_out.writerow(["Nessus", "ID", "Text"])

    for filename in glob.glob('*.html'):        # search all HTML files in the current folder
        print("Processing:", filename)

        with open(filename) as f_in:
            soup = BeautifulSoup(f_in, 'html5lib')

            var1 = soup.find('li', text=re.compile('Scan vendor:'), attrs = {'class' : 'property_name'})
            var2 = soup.find('li', text=re.compile('Vendor ID:'), attrs = {'class' : 'property_name'})

            nessus = var1.find_next('li').get_text(strip=True)
            id = var2.find_next('li').get_text(strip=True)

            for textarea in soup.find_all("textarea"):
                desc = textarea.get_text(strip=True)

                if get_language(desc) == 'dk':
                    csv_out.writerow([nessus, id, desc])

这将为您提供以下输出 CSV 文件：

Nessus,ID,Text
Nessus,55786,"According to its version, the installation of Oracle Database on the remote host is no longer supported.

Lack of support implies that no new security patches for the product will be released by the vendor. As a result, it is likely to contain security vulnerabilities."
Nessus,55786,The remote host is running an unsupported version of a database server.
Nessus,55786,Upgrade to a version of Oracle Database that is currently supported.

注意：由于您的文本包含换行符，CSV 格式会自动将这些单元格用双引号括起来。它将正确加载到另一个包中。

脚本搜索当前文件夹中的所有匹配文件。对于每个textarea，都会调用get_language()，这需要使用另一个库进行编码。如果检测到dk（或任何需要的），则将该行添加到 CSV 文件中。

如果您的文本区域包含 HTML，您可能需要使用另一个对 BeautifulSoup 的调用来进一步处理它：

soup_desc = BeautifulSoup(desc, 'html5lib')

for text in soup_desc.stripped_strings:
    ...

【讨论】：

嘿马丁..问题是我只需要丹麦语描述文本，据我所知（我有大约 1500 个文本文件，我需要循环并转换为单个 csv 文件)，所有文件都有 Nessus 及其 ID，但丹麦文本并不存在于所有文件中。前 5 个文本区域我总是可以忽略，但有时会出现第 6 个（并且是英文文本），第 7 个是通常是丹麦文本。但是有时英语丢失了，然后丹麦文本在第 6 个文本区域（我知道它的皮塔饼！）
也许可以检查是否有 7 个文本区域然后取最后一个，否则取第 6 个？我不知道..
有些库可以帮助您识别文本语言。然后你可以遍历所有区域，只保存丹麦语？
还可以添加一个循环来遍历所有文件，然后将输出添加到同一个 CSV 文件中，我可以将其添加到答案中。
那太好了，我也一直在努力创建循环