Python链接刮刀

【问题标题】：Python Link ScraperPython链接刮刀
【发布时间】：2016-05-12 21:32:30
【问题描述】：

    focus_Search = raw_input("Focus Search ") 
    url = "https://www.google.com/search?q=" 
    res = requests.get(url + focus_Search) 
    print("You Just Searched") 
    res_String = res.text 
    #Now I must get ALL the sections of code that start with "<a href" and end with "/a>"

我试图从谷歌搜索网页中抓取所有链接。我可以一次提取每个链接，但我确信有更好的方法来做到这一点。

【问题讨论】：

使用html解析器，SO上的例子数不胜数

标签： python scraper

【解决方案1】：

这将使用您的一些代码创建搜索页面中所有链接的列表，而无需进入 BeautifulSoup

import requests
import lxml.html

focus_Search = input("Focus Search ") 
url = "https://www.google.com/search?q=" 
#focus_Search
res = requests.get(url + focus_Search).content 
# res

dom = lxml.html.fromstring(res)
links = [x for x in dom.xpath('//a/@href')] # Borrows from cheekybastard in link below
# http://stackoverflow.com/questions/1080411/retrieve-links-from-web-page-using-python-and-beautifulsoup
links

【讨论】：