【问题标题】:Python web scraping specific informationPython网页抓取具体信息
【发布时间】:2016-05-21 02:42:25
【问题描述】:

我正在尝试从网页中提取一些信息并对其进行格式化。我可以提取数据,但格式已损坏。我想得到如下一行:

MAC地址,F41FC267FC1D,主机名,SEPF41FC267FC1D,电话DN,28292,应用加载ID,jar45sccp.9-3-1ES13.sbn,启动加载ID,tnp45.9-3-1-CR17.bin,版本, SCCP45.9-3-1SR2-1S,Hardware Revision,13.0, Serial Number,FCH17188DSC, Model Number,CP-7945G, Message Waiting,Yes, UDI,phone,Cisco Unified IP Phone 7945G,全球,千兆以太网,颜色,CP-7945G,V14,FCH17188DSC,时间,10:40:10a,时区,EST5EDT,日期,02/10/16,系统可用内存,4346528,Java堆可用内存,2502292,Java池空闲内存

包含信息的网页:

<HTML>
<HEAD><META http-equiv="Content-Type" content="text/html; charset=utf-8"><TITLE>Cisco Systems, Inc.</TITLE>
</HEAD><BODY bgcolor="#FFFFFF" link="#FFFFFF" vlink="#FFFFFF" alink="#FFFFFF" text="#003031"><TABLE BORDER="1" WIDTH="100%" HEIGHT="100%" CELLSPACING="0" CELLPADDING="0" bordercolor="#003031"><TR>
<td WIDTH="200" HEIGHT="100" ALIGN=center><A HREF="http://www.cisco.com"><IMG SRC="/FS/Logo.png"></A></TD><td HEIGHT="50" bgcolor="#003031"><p ALIGN=center><B><font color="#FFFFFF" size="6">Device Information</FONT></B><p ALIGN=center><B><font color="#FFFFFF" size="4">Cisco Unified IP Phone CP-7945G ( SEPF41FC267FC1D )</FONT></FONT></B></TD>
</TR>
<TR><td WIDTH="200" ALIGN=center VALIGN=top bgcolor="#003031"><TABLE BORDER="0" CELLSPACING="10" CELLPADDING="0"><TR>
<TD><a href="/CGI/Java/Serviceability?adapter=device.statistics.device">Device Information</A></TD></TR>
<TR>
<TD><a href="/CGI/Java/Serviceability?adapter=device.statistics.configuration">Network Configuration</A></TD></TR>
<TR><TD><B><font color='#FFFFFF'>Network Statistics</FONT></B></TD></TR><TR>
<TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.ethernet">Ethernet Information</A></TD>
</TR><TR><TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.port.access">Access</A></TD></TR><TR><TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.port.network">Network</A></TD></TR><TR><TD><B><font color='#FFFFFF'>Device Logs</FONT></B></TD></TR><TR>
<TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.consolelog">Console Logs</A></TD><TR>
<TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.coredumps">Core Dumps</A></TD><TR>
<TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.settings.status.messages">Status Messages</A></TD></TR>
<TR>
<TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.trace.display.alarm">Debug Display</A></TD></TR>
<TR><TD><B><font color='#FFFFFF'>Streaming Statistics</FONT></B></TD></TR><TR><TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.streaming.0">Stream 1 </A></TD></TR><TR><TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.streaming.1">Stream 2 </A></TD></TR><TR><TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.streaming.2">Stream 3 </A></TD></TR><TR><TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.streaming.3">Stream 4 </A></TD></TR><TR><TD>&nbsp;&nbsp;&nbsp;<a href="/CGI/Java/Serviceability?adapter=device.statistics.streaming.4">Stream 5 </A></TD></TR></TABLE>
</TD>
<td VALIGN=top><DIV ALIGN=center>
<TABLE BORDER="0" CELLSPACING="10" CELLPADDING="0"><TR><TD><B> MAC Address</B></TD><td width=20></TD><TD><B>F41FC267FC1D</B></TD></TR><TR><TD><B> Host Name</B></TD><td width=20></TD><TD><B>SEPF41FC267FC1D</B></TD></TR><TR><TD><B> Phone DN</B></TD><td width=20></TD><TD><B>28292</B></TD></TR><TR><TD><B> App Load ID</B></TD><td width=20></TD><TD><B>jar45sccp.9-3-1ES13.sbn</B></TD></TR><TR><TD><B> Boot Load ID</B></TD><td width=20></TD><TD><B>tnp45.9-3-1-CR17.bin</B></TD></TR><TR><TD><B> Version</B></TD><td width=20></TD><TD><B>*SCCP45.9-3-1SR2-1S*</B></TD></TR><TR><TD><B> Hardware Revision</B></TD><td width=20></TD><TD><B>13.0</B></TD></TR><TR><TD><B> Serial Number</B></TD><td width=20></TD><TD><B>FCH17188DSC</B></TD></TR><TR><TD><B> Model Number</B></TD><td width=20></TD><TD><B>CP-7945G</B></TD></TR><TR><TD><B> Message Waiting</B></TD><td width=20></TD><TD><B>Yes</B></TD></TR><TR><TD><B> UDI</B></TD><td width=20></TD><TD><B>phone</B></TD></TR><TR><TD><B> </B></TD><td width=20></TD><TD><B>Cisco Unified IP Phone 7945G, Global, Gig Ethernet, Color</B></TD></TR><TR><TD><B> </B></TD><td width=20></TD><TD><B>CP-7945G</B></TD></TR><TR><TD><B> </B></TD><td width=20></TD><TD><B>V14</B></TD></TR><TR><TD><B> </B></TD><td width=20></TD><TD><B>FCH17188DSC</B></TD></TR><TR><TD><B> Time</B></TD><td width=20></TD><TD><B>10:49:52a</B></TD></TR><TR><TD><B> Time Zone</B></TD><td width=20></TD><TD><B>EST5EDT</B></TD></TR><TR><TD><B> Date</B></TD><td width=20></TD><TD><B>02/10/16</B></TD></TR><TR><TD><B> System Free Memory</B></TD><td width=20></TD><TD><B>4346528</B></TD></TR><TR><TD><B> Java Heap Free Memory</B></TD><td width=20></TD><TD><B>2502060</B></TD></TR><TR><TD><B> Java Pool Free Memory</B></TD><td width=20></TD><TD><B>107608</B></TD></TR><TR><TD><B> FIPS Mode Enabled</B></TD><td width=20></TD><TD><B>No</B></TD></TR></TABLE></DIV></TD></TR></TABLE></BODY></HTML>

Python 代码:

      import urllib
      import urllib.request
      from bs4 import BeautifulSoup
      import os

      def make_soup(url):
      thepage = urllib.request.urlopen(url)
      soupdata = BeautifulSoup(thepage, "html.parser")
      return soupdata


      phonedatasaved=" "
      soup = make_soup("http://phone/")
      for div in soup.findAll("div"):

      phonedata=" "
      for tr in div.findAll('tr'):
      for td in tr.findAll('td'):
        phonedata=phonedata+","+td.text
        phonedatasaved = phonedatasaved + "\n" + phonedata[0:]
        print(phonedatasaved)

【问题讨论】:

    标签: python html scripting functional-programming


    【解决方案1】:

    不需要函数。应该这样做:

    thepage = urllib.request.urlopen(url)
    soup = BeautifulSoup(thepage, 'html.parser')
    elements =soup('table')[-1]('b')
    lst = []
    for i in range(0, len(elements), 2):
        lst.append(elements[i].text.strip())
        lst.append(elements[i+1].text.strip())
    print ', '.join(lst)
    

    编辑:您的代码中有什么不正确的地方?我的想法:

    1. 尝试在循环中打印出添加的字符串不是正确的做法。
    2. 代码试图用phonedata[0:]做什么?其实phonedata[0:] = phonedata
    3. 不需要像“phonedatasaved”、“phonedata”这样的变量。
    4. 如果td.text为空,代码不需要打印出来。

    总体而言,您的代码中的实现让我非常困惑。这是修改后的版本:

    import urllib
    import urllib.request
    from bs4 import BeautifulSoup
    import os
    
    def make_soup(url):
        thepage = urllib.request.urlopen(url)
        soupdata = BeautifulSoup(thepage, "html.parser")
        return soupdata
    
    soup = make_soup("http://phone/")
    
    for div in soup.findAll("div"):
        for tr in div.findAll('tr'):
            for td in tr.findAll('td'):
                if td.text.strip():
                    print(td.text.strip()) + ',', 
    

    【讨论】:

    • 我不明白你的代码。我是python的新手。我想坚持我的代码,这样我就可以了解我做错了什么。你能帮我解决这个问题吗?
    • @user1819884:您的代码格式不正确,您能解决吗?
    • 我刚刚运行了您的代码,但格式并不完全符合我的需要。如何更改代码格式以将输出显示为单行逗号分隔。
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 2016-08-28
    • 1970-01-01
    • 1970-01-01
    • 2021-07-03
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多