为产品页面链接抓取 eBay 特色集合

Question

我正在尝试使用 Python 和 BeautifulSoup 构建一个网络抓取工具，该工具可以进入 eBay 精选收藏并检索该收藏中所有产品的 URL（大多数收藏有 17 种产品，尽管有些产品或多或少） .这是我尝试在代码中抓取的集合的 URL：http://www.ebay.com/cln/ebayhomeeditor/Surface-Study/324079803018

到目前为止，这是我的代码：

import requests
from bs4 import BeautifulSoup

url = 'http://www.ebay.com/cln/ebayhomeeditor/Surface-Study/324079803018'
soup = BeautifulSoup(requests.get(url).text, 'html.parser')

product_links = []

item_thumb = soup.find_all('div', attrs={'class':'itemThumb'})
for link in item_thumb:
    product_links.append(link.find('a').get('href'))

print product_links

此抓取工具应将 17 个链接附加到列表 product_links。但是，它只在部分工作。具体来说，它每次只抓取前 12 个产品链接，其余 5 个保持不变，即使所有 17 个链接都位于相同的 HTML 标记和属性中。更仔细地查看页面的 HTML 代码，我发现的唯一区别是前 12 个链接和最后 5 个链接由我在此处包含的一段 XML 脚本分隔：

<script escape-xml="true">
      if (typeof(collectionState) != 'object') {
          var collectionState = {
              itemImageSize: {sWidth: 280, sHeight: 280, lWidth: 580, lHeight: 620},
              page: 1,
              totalPages: 2,
              totalItems: 17,
              pageId: '2057253',
              currentUser: '',
              collectionId: '323101965012',
              serviceHost: 'svcs.ebay.com/buying/collections/v1',
              owner: 'ebaytecheditor',
              csrfToken: '',
              localeId: 'en-US',
              siteId: 'EBAY-US',
              countryId: 'US',
              collectionCosEnabled: 'true',
              collectionCosHostExternal: 'https://api.ebay.com/social/collection/v1',
              collectionCosEditEnabled: 'true',
              isCollectionReorderEnabled: 'false',
              isOwnerSignedIn: false || false,
              partiallySignedInUser: '@@__@@__@@',
              baseDomain: 'ebay.com',
              currentDomain: 'www.ebay.com',
              isTablet: false,
              isMobile: false,
              showViewCount: true
          };
      }
    </script>

这个脚本的作用是什么？这个脚本是否有可能是我的抓取工具忽略抓取最后 5 个链接的原因？有没有办法解决这个问题并避开最后五个？

Answer 1

最后几个是通过 ajax 向 http://www.ebay.com/cln/_ajax/2/ebayhomeeditor/324079803018 请求生成的：

网址是使用 ebayhomeeditor 组成的，并且必须是一些产品 ID 324079803018，它们都在您访问的页面的原始网址中。

获取数据所必需的唯一参数是 itemsPerPage，但您可以尝试其余的参数，看看它们有什么效果。

params =  {"itemsPerPage": "10"}
soup= BeautifulSoup(requests.get("http://www.ebay.com/cln/_ajax/2/ebayhomeeditor/324079803018", params=params).content)
print([a["href"] for a in soup.select("div.itemThumb div.itemImg.image.lazy-image a[href]")])

这会给你：

['http://www.ebay.com/itm/yamazaki-home-tower-book-end-white-stationary-holder-desktop-organizing-steel/171836462366?hash=item280240551e', 'http://www.ebay.com/itm/tetris-constructible-interlocking-desk-lamp-neon-light-nightlight-by-paladone/221571335719?hash=item3396ae4627', 'http://www.ebay.com/itm/iphone-docking-station-dock-native-union-new-in-box/222202878086?hash=item33bc52d886', 'http://www.ebay.com/itm/turnkey-pencil-sharpener-silver-office-home-school-desk-gift-peleg-design/201461359979?hash=item2ee808656b', 'http://www.ebay.com/itm/himori-weekly-times-desk-notepad-desktop-weekly-scheduler-30-weeks-planner/271985620013?hash=item3f539b342d']

所以把它放在一起得到所有的网址：

In [23]: params = {"itemsPerPage": "10"}

In [24]: with requests.Session() as s:
   ....:         soup1 = BeautifulSoup(s.get('http://www.ebay.com/cln/ebayhomeeditor/Surface-Study/324079803018').content,
   ....:                               "html.parser")
   ....:         main_urls = [a["href"] for a in soup1.select("div.itemThumb div.itemImg.image.lazy-image a[href]")]
   ....:         soup2 = BeautifulSoup(s.get("http://www.ebay.com/cln/_ajax/2/ebayhomeeditor/324079803018", params=params).content,
   ....:                               "html.parser")
   ....:         print(len(main_urls))
   ....:         main_urls.extend(a["href"] for a in soup2.select("div.itemThumb div.itemImg.image.lazy-image a[href]"))
   ....:         print(main_urls)
   ....:         print(len(main_urls))
   ....:     
12
['http://www.ebay.com/itm/archi-desk-accessories-pen-cup-designed-by-hsunli-huang-for-moma/262435041373?hash=item3d1a58f05d', 'http://www.ebay.com/itm/moorea-seal-violet-light-crane-scissors/201600302323?hash=item2ef0507cf3', 'http://www.ebay.com/itm/kikkerland-photo-holder-with-6-magnetic-wooden-clothespin-mh69-cable-47-long/361394782932?hash=item5424cec2d4', 'http://www.ebay.com/itm/authentic-22-design-studio-merge-concrete-pen-holder-desk-office-pencil/331846509549?hash=item4d4397e3ed', 'http://www.ebay.com/itm/supergal-bookend-by-artori-design-ad103-metal-black/272273290322?hash=item3f64c0b452', 'http://www.ebay.com/itm/elago-p2-stand-for-ipad-tablet-pcchampagne-gold/191527567203?hash=item2c97eebf63', 'http://www.ebay.com/itm/this-is-ground-mouse-pad-pro-ruler-100-authentic-natural-retail-100/201628986934?hash=item2ef2062e36', 'http://www.ebay.com/itm/hot-fuut-foot-rest-hammock-under-desk-office-footrest-mini-stand-hanging-swing/152166878943?hash=item236dda4edf', 'http://www.ebay.com/itm/unido-silver-white-black-led-desk-office-lamp-adjustable-neck-brightness-level/351654910666?hash=item51e0441aca', 'http://www.ebay.com/itm/in-house-black-desk-office-organizer-paper-clips-memo-notes-monkey-business/201645856763?hash=item2ef30797fb', 'http://www.ebay.com/itm/rifle-paper-co-2017-maps-desk-calendar-illustrated-worldwide-cities/262547131670?hash=item3d21074d16', 'http://www.ebay.com/itm/muji-erasable-pen-black/262272348079?hash=item3d10a66faf', 'http://www.ebay.com/itm/rifle-paper-co-2017-maps-desk-calendar-illustrated-worldwide-cities/262547131670?hash=item3d21074d16', 'http://www.ebay.com/itm/muji-erasable-pen-black/262272348079?hash=item3d10a66faf', 'http://www.ebay.com/itm/yamazaki-home-tower-book-end-white-stationary-holder-desktop-organizing-steel/171836462366?hash=item280240551e', 'http://www.ebay.com/itm/tetris-constructible-interlocking-desk-lamp-neon-light-nightlight-by-paladone/221571335719?hash=item3396ae4627', 'http://www.ebay.com/itm/iphone-docking-station-dock-native-union-new-in-box/222202878086?hash=item33bc52d886', 'http://www.ebay.com/itm/turnkey-pencil-sharpener-silver-office-home-school-desk-gift-peleg-design/201461359979?hash=item2ee808656b', 'http://www.ebay.com/itm/himori-weekly-times-desk-notepad-desktop-weekly-scheduler-30-weeks-planner/271985620013?hash=item3f539b342d']
19

In [25]: 

与返回的内容有一点重叠，所以只需使用一个集合来存储 main_urls 或在列表中调用集合：

In [25]: len(set(main_urls))
Out[25]: 17

不确定为什么会发生这种情况并且没有真正尝试弄清楚，如果它困扰您，那么您可以从 ajax 调用返回的源中解析“totalItems：17”并在第一次调用后减去 main_urls 的长度，然后设置{"itemsPerPage": str(len(main_urls) - int(parsedtotal))} 但我不会太担心。