不能使用图像 ID 来使它们成为合格的图像链接答案

【问题标题】：Can't use image ids in order to make them qualified image links不能使用图像 ID 来使它们成为合格的图像链接
【发布时间】：2020-09-07 17:20:12
【问题描述】：

我正在尝试使用 requests 模块从 webpage 中抓取所有图片链接。当我使用此链接时，我只能向上抓取图像链接，直到向下滚动时显示的其余内容。但是，如果我使用这个link，我可以通过增加附加到链接的最后一个数字来获取所有图像 ID。问题是我不能重用这些 id 来制作完整的图片链接。

我试过了：

import requests
from bs4 import BeautifulSoup

url = 'https://stocksnap.io/api/search-photos/phone/relevance/desc/1'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36'
    r = s.get(url)
    for item in r.json()['results']:
        print(item['img_id'])

如何从该网站的登录页面获取所有图片链接？

PS 前几个赞助图片链接应该被忽略，因为它们也不包含在 api 中。

【问题讨论】：

标签： python python-3.x web-scraping python-requests

【解决方案1】：

检查页面，图像 URL 由 ID 和从 API 获取的前两个标签构成：

import requests


url = 'https://stocksnap.io/api/search-photos/phone/relevance/desc/{}'

page = 1
while True:
    data = requests.get(url.format(page)).json()

    if not data['results']:
        break

    for r in data['results']:
        print('https://stocksnap.io/photo/{}-{}-{}'.format(r['keywords'][0], r['keywords'][1], r['img_id']))

    page += 1

打印：

...

https://stocksnap.io/photo/iphone-cellphone-LNXYMM77SS
https://stocksnap.io/photo/business-technology-OGLUHZAPGF
https://stocksnap.io/photo/samsung-android-7ZALGLUAAW
https://stocksnap.io/photo/apple-macbook-55A6840521
https://stocksnap.io/photo/woman-talking-54C3E9FE9D
https://stocksnap.io/photo/samsung-galaxy-BB3307280A
https://stocksnap.io/photo/parc-bench-3D99A31C0C
https://stocksnap.io/photo/iphone-cellphone-E2C541A7DC
https://stocksnap.io/photo/iphone-mockup-167A645BDC
https://stocksnap.io/photo/mac-keyboard-BA9AFFE0BF
https://stocksnap.io/photo/sony-android-EB939B3311
https://stocksnap.io/photo/iphone-cellphone-B962ABCAC7
https://stocksnap.io/photo/building-man-D49A8BB4AE
https://stocksnap.io/photo/technology-computer-C9B37875B9
https://stocksnap.io/photo/iphone-cellphone-381F0FD1EE
https://stocksnap.io/photo/work-bag-96E1A8F1CB
https://stocksnap.io/photo/iphone-phone-70FE8C00C9
https://stocksnap.io/photo/iphone-mockup-9FCDF4E1F5
https://stocksnap.io/photo/young-girl-BE8BA006E6
https://stocksnap.io/photo/young-girl-7174B21D56
https://stocksnap.io/photo/man-woman-6XELVX8KAN
https://stocksnap.io/photo/nexus-smartphones-UAXILBRNUL

编辑：要获取.jpg 链接，同样的方法适用：

import requests


url = 'https://stocksnap.io/api/search-photos/phone/relevance/desc/{}'

page = 1
while True:
    data = requests.get(url.format(page)).json()

    if not data['results']:
        break

    for r in data['results']:
        print('https://cdn.stocksnap.io/img-thumbs/280h/{}-{}_{}.jpg'.format(r['keywords'][0], r['keywords'][1], r['img_id']))

    page += 1

打印：

...

https://cdn.stocksnap.io/img-thumbs/280h/iphone-cellphone_B962ABCAC7.jpg
https://cdn.stocksnap.io/img-thumbs/280h/building-man_D49A8BB4AE.jpg
https://cdn.stocksnap.io/img-thumbs/280h/technology-computer_C9B37875B9.jpg
https://cdn.stocksnap.io/img-thumbs/280h/iphone-cellphone_381F0FD1EE.jpg
https://cdn.stocksnap.io/img-thumbs/280h/work-bag_96E1A8F1CB.jpg
https://cdn.stocksnap.io/img-thumbs/280h/iphone-phone_70FE8C00C9.jpg
https://cdn.stocksnap.io/img-thumbs/280h/iphone-mockup_9FCDF4E1F5.jpg
https://cdn.stocksnap.io/img-thumbs/280h/young-girl_BE8BA006E6.jpg
https://cdn.stocksnap.io/img-thumbs/280h/young-girl_7174B21D56.jpg
https://cdn.stocksnap.io/img-thumbs/280h/man-woman_6XELVX8KAN.jpg
https://cdn.stocksnap.io/img-thumbs/280h/nexus-smartphones_UAXILBRNUL.jpg

【讨论】：

多么棒的观察@Andrej Kesely！我一开始就注意到了这些标签，但是当我发现它们超过两个时就失去了希望。谢谢。