【问题标题】:How to query html, wrapped in the json response using scrapy如何使用scrapy查询包含在json响应中的html
【发布时间】:2021-01-16 19:41:23
【问题描述】:

我正在尝试抓取一个网站,该网站通过 javascript 动态加载其内容。我能够请求加载数据的源,但它以 json 格式返回响应,并且在该 json 中有一个名为“results_html”的字段,其中包含我需要查询的所有 html 以获得想要的数据。我尝试了许多解决方案并阅读了许多相关问题,但没有解决我的问题。以下是我收到的回复形式。

response.body = b'{"success":true,"results_html":"\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/1200330\\/Strategic_Mind_Blitzkrieg\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"1200330\\" data-ds-itemkey=\\"App_1200330\\" data-ds-tagids=\\"[21725,4684,4026,1677,1741,17305,13276]\\" data-ds-crtrids=\\"[35230293,38088850]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:1200330,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/1200330\\/capsule_184x69.jpg?t=1601419049\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount\\" data-price-final=\\"1039\\"><div class=\\"discount_pct\\">-20%<\\/div><div class=\\"discount_prices\\"><div class=\\"discount_original_price\\">$12.99<\\/div><div class=\\"discount_final_price\\">$10.39<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Strategic Mind: Blitzkrieg<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Tactical RPG<\\/span><span class=\\"top_tag\\">, Wargame<\\/span><span class=\\"top_tag\\">, Difficult<\\/span><span class=\\"top_tag\\">, Turn-Based<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/228200\\/Company_of_Heroes\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"228200\\" data-ds-itemkey=\\"App_228200\\" data-ds-tagids=\\"[9,4150,1676,19,1678,3859,4182]\\" data-ds-crtrids=\\"[35920674,32528477]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:228200,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/228200\\/capsule_184x69.jpg?t=1570195772\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"1999\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$19.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Company of Heroes<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, World War II<\\/span><span class=\\"top_tag\\">, RTS<\\/span><span class=\\"top_tag\\">, Action<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/40980\\/Stronghold_Legends_Steam_Edition\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"40980\\" data-ds-itemkey=\\"App_40980\\" data-ds-tagids=\\"[9,220585,599,4172,7332,4328,3859]\\" data-ds-crtrids=\\"[32942208]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:40980,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/40980\\/capsule_184x69.jpg?t=1601395216\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount\\" data-price-final=\\"230\\"><div class=\\"discount_pct\\">-67%<\\/div><div class=\\"discount_prices\\"><div class=\\"discount_original_price\\">$6.99<\\/div><div class=\\"discount_final_price\\">$2.30<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Stronghold Legends: Steam Edition<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, Colony Sim<\\/span><span class=\\"top_tag\\">, Simulation<\\/span><span class=\\"top_tag\\">, Medieval<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/2630\\/Call_of_Duty_2\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"2630\\" data-ds-itemkey=\\"App_2630\\" data-ds-tagids=\\"[19,4150,1663,3859,4182,1774,1678]\\" data-ds-crtrids=\\"[31936442]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:2630,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/2630\\/capsule_184x69.jpg?t=1572994622\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"999\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$9.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Call of Duty\xc2\xae 2<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span><span class=\\"platform_img mac\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Action<\\/span><span class=\\"top_tag\\">, World War II<\\/span><span class=\\"top_tag\\">, FPS<\\/span><span class=\\"top_tag\\">, Multiplayer<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/306660\\/Ultimate_General_Gettysburg\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"306660\\" data-ds-itemkey=\\"App_306660\\" data-ds-tagids=\\"[9,3987,599,1676,1708,4684,492]\\" data-ds-crtrids=\\"[37783362]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:306660,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/306660\\/capsule_184x69.jpg?t=1562054645\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"999\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$9.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Ultimate General: Gettysburg<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span><span class=\\"platform_img mac\\"><\\/span><span class=\\"platform_img linux\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, Historical<\\/span><span class=\\"top_tag\\">, Simulation<\\/span><span class=\\"top_tag\\">, RTS<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/241260\\/Sherlock_Holmes_Crimes_and_Punishments\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"241260\\" data-ds-itemkey=\\"App_241260\\" data-ds-tagids=\\"[5613,21,5716,6378,8369,1664,1742]\\" data-ds-descids=\\"[5]\\" data-ds-crtrids=\\"[1205865,36390721,21394]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:241260,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/241260\\/capsule_184x69.jpg?t=1600074287\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"2699\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$26.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Sherlock Holmes: Crimes and Punishments<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Detective<\\/span><span class=\\"top_tag\\">, Adventure<\\/span><span class=\\"top_tag\\">, Mystery<\\/span><span class=\\"top_tag\\">, Crime<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/261050\\/Total_War_ROME_II__Caesar_in_Gaul_Campaign_Pack\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"261050\\" data-ds-itemkey=\\"App_261050\\" data-ds-tagids=\\"[9,3987,6948,1741,1676,4364,1678]\\" data-ds-crtrids=\\"[32528477,32991376]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:261050,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/261050\\/capsule_184x69.jpg?t=1571743386\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"1299\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$12.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Total War: ROME II - Caesar in Gaul Campaign Pack<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span><span class=\\"platform_img mac\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, Historical<\\/span><span class=\\"top_tag\\">, Rome<\\/span><span class=\\"top_tag\\">, Turn-Based Strategy<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/201870\\/Assassins_Creed_Revelations\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"201870\\" data-ds-itemkey=\\"App_201870\\" data-ds-tagids=\\"[19,1695,4036,21,1687,4376,1697]\\" data-ds-crtrids=\\"[33075774,185907]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:201870,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/201870\\/capsule_184x69.jpg?t=1532007165\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"1099\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$10.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Assassin\'s Creed\xc2\xae Revelations<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Action<\\/span><span class=\\"top_tag\\">, Open World<\\/span><span class=\\"top_tag\\">, Parkour<\\/span><span class=\\"top_tag\\">, Adventure<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/205610\\/Port_Royale_3\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"205610\\" data-ds-itemkey=\\"App_205610\\" data-ds-tagids=\\"[9,4202,599,4695,1681,12472,6910]\\" data-ds-crtrids=\\"[876623]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:205610,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/205610\\/capsule_184x69.jpg?t=1599137713\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"1499\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$14.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Port Royale 3<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, Trading<\\/span><span class=\\"top_tag\\">, Simulation<\\/span><span class=\\"top_tag\\">, Economy<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/322520\\/DYNASTY_WARRIORS_8_Empires\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"322520\\" data-ds-itemkey=\\"App_322520\\" data-ds-tagids=\\"[19,1646,4747,9,122,3987,1697]\\" data-ds-crtrids=\\"[33016879]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:322520,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/322520\\/capsule_184x69.jpg?t=1597910669\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"4999\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$49.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">DYNASTY WARRIORS 8 Empires<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Action<\\/span><span class=\\"top_tag\\">, Hack and Slash<\\/span><span class=\\"top_tag\\">, Character Customization<\\/span><span class=\\"top_tag\\">, Strategy<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/48720\\/Mount__Blade_With_Fire__Sword\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"48720\\" data-ds-itemkey=\\"App_48720\\" data-ds-tagids=\\"[122,1695,4172,19,9,3987,3859]\\" data-ds-crtrids=\\"[33089344]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:48720,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/48720\\/capsule_184x69.jpg?t=1589227483\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"599\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$5.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Mount &amp; Blade: With Fire &amp; Sword<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">RPG<\\/span><span class=\\"top_tag\\">, Open World<\\/span><span class=\\"top_tag\\">, Medieval<\\/span><span class=\\"top_tag\\">, Action<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/853360\\/Total_War_THREE_KINGDOMS__Yellow_Turban_Rebellion\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"853360\\" data-ds-itemkey=\\"App_853360\\" data-ds-tagids=\\"[9,19,4667,3987]\\" data-ds-descids=\\"[2,5]\\" data-ds-crtrids=\\"[32528477,32991376]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:853360,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/853360\\/capsule_184x69.jpg?t=1571750931\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"699\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$6.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Total War: THREE KINGDOMS - Yellow Turban Rebellion<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span><span class=\\"platform_img mac\\"><\\/span><span class=\\"platform_img linux\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, Action<\\/span><span class=\\"top_tag\\">, Violent<\\/span><span class=\\"top_tag\\">, Historical<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/3910\\/Sid_Meiers_Civilization_III_Complete\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"3910\\" data-ds-itemkey=\\"App_3910\\" data-ds-tagids=\\"[9,1741,1677,1693,1670,3987,4182]\\" data-ds-crtrids=\\"[32844624,2428135,33339585]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:3910,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/3910\\/capsule_184x69.jpg?t=1569013660\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount\\" data-price-final=\\"99\\"><div class=\\"discount_pct\\">-75%<\\/div><div class=\\"discount_prices\\"><div class=\\"discount_original_price\\">$3.99<\\/div><div class=\\"discount_final_price\\">$0.99<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Sid Meier\'s Civilization\xc2\xae III Complete<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, Turn-Based Strategy<\\/span><span class=\\"top_tag\\">, Turn-Based<\\/span><span class=\\"top_tag\\">, Classic<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/926580\\/Broken_Lines\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"926580\\" data-ds-itemkey=\\"App_926580\\" data-ds-tagids=\\"[1708,1676,9,4150,3987,1678,4168]\\" data-ds-descids=\\"[2,5]\\" data-ds-crtrids=\\"[35132473]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:926580,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/926580\\/capsule_184x69.jpg?t=1598642869\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"1140\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$11.40<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Broken Lines<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span><span class=\\"platform_img mac\\"><\\/span><span class=\\"platform_img linux\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Tactical<\\/span><span class=\\"top_tag\\">, RTS<\\/span><span class=\\"top_tag\\">, Strategy<\\/span><span class=\\"top_tag\\">, World War II<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t\\t<a href=\\"https:\\/\\/store.steampowered.com\\/app\\/73170\\/Darkest_Hour_A_Hearts_of_Iron_Game\\/?snr=1_241_4_historical_104_8\\" class=\\"tab_item  \\"  data-ds-appid=\\"73170\\" data-ds-itemkey=\\"App_73170\\" data-ds-tagids=\\"[9,4364,4150,3987,5382,599,4684]\\" data-ds-crtrids=\\"[4117016,6859167]\\" onmouseover=\\"GameHover( this, event, \'global_hover\', {&quot;type&quot;:&quot;app&quot;,&quot;id&quot;:73170,&quot;params&quot;:{&quot;bDisableHover&quot;:false},&quot;public&quot;:1,&quot;v6&quot;:1} );\\" onmouseout=\\"HideGameHover( this, event, \'global_hover\' )\\">\\r\\n\\t\\t<div class=\\"tab_item_cap\\">\\r\\n\\t\\t\\t<img class=\\"tab_item_cap_img\\" src=\\"https:\\/\\/steamcdn-a.akamaihd.net\\/steam\\/apps\\/73170\\/capsule_184x69.jpg?t=1589876931\\" >\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t\\t\\t<div class=\\"discount_block tab_item_discount no_discount\\" data-price-final=\\"549\\"><div class=\\"discount_prices\\"><div class=\\"discount_final_price\\">$5.49<\\/div><\\/div><\\/div>\\t\\t<div class=\\"tab_item_content\\">\\r\\n\\t\\t\\t<div class=\\"tab_item_name\\">Darkest Hour: A Hearts of Iron Game<\\/div>\\r\\n\\t\\t\\t<div class=\\"tab_item_details\\">\\r\\n\\t\\t\\t\\t<span class=\\"platform_img win\\"><\\/span>\\t\\t\\t\\t<div class=\\"tab_item_top_tags\\"><span class=\\"top_tag\\">Strategy<\\/span><span class=\\"top_tag\\">, Grand Strategy<\\/span><span class=\\"top_tag\\">, World War II<\\/span><span class=\\"top_tag\\">, Historical<\\/span><\\/div>\\r\\n\\t\\t\\t<\\/div>\\r\\n\\t\\t<\\/div>\\r\\n\\t\\t<div style=\\"clear: both;\\"><\\/div>\\r\\n\\t<\\/a>\\r\\n\\t","start":105,"pagesize":15,"total_count":1000}'

【问题讨论】:

    标签: python web-scraping scrapy


    【解决方案1】:

    您可以使用以下解决方案:

    import json
    from scrapy.selector import Selector
    .........
    j_obj = json.loads(response.body_as_unicode())
    j_response = Selector(text=j_obj['results_html'])
    
    j_response.css('your selector here')
    j_response.xpath('your xpath here')
    

    示例:

    print(len(j_response.css('div.search_price')))
    

    输出:

    50
    

    获取链接:

    for a in j_response.xpath('//a/@href').getall():
        print(a)
    

    【讨论】:

    • j_response 不存储整个 html,而是只存储一行
    • 一行我的意思是几个标签。
    • 您想从该响应中收集什么?游戏名称?链接还是什么?
    • 我要游戏链接
    • 感谢它现在正在工作,当我用 len 转储它时,我意识到 j_response 有所有条目,然后我仔细监控我的 xpath,我的 xpath 中有一个不必要的星号 (*)。
    猜你喜欢
    • 2013-10-09
    • 1970-01-01
    • 2011-06-15
    • 2010-09-22
    • 2011-05-05
    • 1970-01-01
    • 2014-10-07
    • 1970-01-01
    相关资源
    最近更新 更多