Scrapy无限滚动没有分页网址答案

【问题标题】：Scrapy infinite scroll without pagination urlsScrapy无限滚动没有分页网址
【发布时间】：2022-06-11 06:00:35
【问题描述】：

使用 Scrapy ，无法从无限滚动书店 www.aseeralkotb.com 网站中剪贴书籍标题作者和链接，来自 devtools 检查，找不到下一页的链接。代码

import scrapy


class booksSpider(scrapy.Spider):
    name = 'books'
    start_urls = [
        'https://www.aseeralkotb.com/categories/%D8%B3%D9%8A%D8%A7%D8%B3%D8%A9',
    ]
   
    def parse(self, response):
        for book in response.css('div.flex.flex-col.items-center'):
            yield {
                 'title': book.css('a:not([itemprop="author"])::attr(title)').get(), 
                 'author': book.css('h5[itemprop=name]::text').get(), 
                  'detailslinks': book.css('a[title]::attr(href)').re(r'.*books.*')
         
            }
        
        for link in  book:
            yield response.follow(link.get(),method='POST',callback = self.parse_links)
         ```

【问题讨论】：

标签： scrapy infinite-scroll

【解决方案1】：

网站正在发送Ajax 请求，来自API 的响应是json 和html。使用 scrapy，它会抛出响应状态 419，但使用强大的 requests 模块，它可以正常工作。

from bs4 import BeautifulSoup as bs
import requests

session = requests.Session()
req_homepage = session.get('https://www.aseeralkotb.com/categories/%D8%B3%D9%8A%D8%A7%D8%B3%D8%A9')
soup_homepage = bs(req_homepage.content, "lxml")

for meta in soup_homepage.find_all('meta'):
    if meta.get('name', None) == 'csrf-token':
        token = meta['content']

data = {"fingerprint":{"id":"UejQB7Z1XFiSDQ3fb05D","name":"categories.books.index","locale":"ar","path":"categories/%D8%B3%D9%8A%D8%A7%D8%B3%D8%A9","method":"GET","v":"acj"},"serverMemo":{"children":{"product-purchase-UtqTb6t0gxWEo6ym20401":{"id":"ZyeTxCh4kmoo4uVFdgAL","tag":"div"},"product-purchase-qfh1S6g5Hqr2yK5p19255":{"id":"IKQCqiaMPBOUz2y9pulb","tag":"div"},"product-purchase-kcsVg5TIGPUm7uBX19253":{"id":"6IV59cn7CyPi4ZlGRSTZ","tag":"div"},"product-purchase-dpgMlpItsMT6QvBC18496":{"id":"G0FV4of9IwUwxN2n3orA","tag":"div"},"product-purchase-djH1nx7Dd1CqjN1Y18245":{"id":"YQY9oXuI80F9zJFUdypB","tag":"div"},"product-purchase-HlsxJBgDCurMZMJ718218":{"id":"lFR46ttmPcJLyBmsPR9t","tag":"div"},"product-purchase-iYHhJD8zFwVpmpV417621":{"id":"dwfj7y3wUh9GMZVVwvM7","tag":"div"},"product-purchase-yEBL5qWdHGmE0cdD17348":{"id":"DwEuSTw1Z2iOcvLsG1dR","tag":"div"},"product-purchase-knjkw2Oa8EncjoPh17074":{"id":"SwZE2176HaD1T0pxPdXI","tag":"div"},"product-purchase-Ce7TBXAPYZbgxvyG17068":{"id":"6UeQ1JImg9AbcI2MjEIv","tag":"div"},"product-purchase-yvOJtFHoJb4M6UxD16469":{"id":"roh9C9Z1U3LVdLsdoRPg","tag":"div"},"product-purchase-zwXX0znXxyZh1ChE16452":{"id":"aQW3CPDhlstTNhRZE5rF","tag":"div"},"product-purchase-YjWk9AseS4bFmf0A16451":{"id":"nhwCNKGwVYa1oBF4wZQF","tag":"div"},"product-purchase-f4jgVxc9M3UPULFQ16010":{"id":"agM3tBl7u3DujVagMkQf","tag":"div"},"product-purchase-kUIj358Zh59HVPtI15963":{"id":"ETXtvZi87WrByYRjpmXu","tag":"div"},"product-purchase-Fg1GaFl9Pntx4JZ115740":{"id":"6PR3Jkr5DVm5QfFWBlCa","tag":"div"},"product-purchase-PenVRhRXZxC7IC8b15511":{"id":"DNRe2AZjqoyTOB1A6sDb","tag":"div"},"product-purchase-pLaYFiLzVJb4M83j15409":{"id":"4Qd9qRflfMzAZIloUHld","tag":"div"},"product-purchase-w59GJbmTvLOlzMzb14799":{"id":"kBl8JNOxnsveu7FeHQQy","tag":"div"},"product-purchase-6DwBE3WETIFYKhdN12394":{"id":"SFfx5sZFVFM1f5KgH9Mn","tag":"div"},"product-purchase-f0U27L0WdrCmdqlN10180":{"id":"LgNzK1XCeJByevDb6poq","tag":"div"},"product-purchase-src1a3URLG9nJHaL10178":{"id":"Zi1ebeCyIrCtqRccwA53","tag":"div"},"product-purchase-EgUvPZAnQvWaYNW69259":{"id":"ziL2dLDBmFMuO4gDHXMo","tag":"div"},"product-purchase-ScLkhiY9GDAIx9Q08620":{"id":"jeQ8oGEJUr3PO0USYTpq","tag":"div"},"product-purchase-33v00TdKKU73hhy38041":{"id":"bBr4JLNXg1uX5iSZf8hj","tag":"div"},"product-purchase-LT345WWsWMGIvPuP7888":{"id":"3B4WCJ59UTTvTKqir9TG","tag":"div"},"product-purchase-rjL4YnQBMXTQNKfW6800":{"id":"4JIdTHJUB6xtOHmeaShS","tag":"div"},"product-purchase-mOsgHazcWpZkkVuj6769":{"id":"fP49Yqnm2jPIkyOdALce","tag":"div"},"product-purchase-xEdrRToHrJJaLnd56733":{"id":"DsVrI5clO75rbmndtcQh","tag":"div"},"product-purchase-d7lOpqgD8t1wJkln6505":{"id":"qBk8BN2BtperIcxGKEyz","tag":"div"},"product-purchase-EipwbWTn9Jeli9mD6492":{"id":"smxCuFfMttSiDhivuhVx","tag":"div"},"product-purchase-gYZcbi026JAJ2rI76482":{"id":"dLVtuM3kp2xQoUaUivW2","tag":"div"},"product-purchase-vcr70vd4N4GhmX3v5598":{"id":"FXohQdm6DsAsfiA7ky8D","tag":"div"},"product-purchase-ndg0avUkJyeaAjxr5597":{"id":"fPj75X6MGP1F9B8wCnbh","tag":"div"},"product-purchase-mmDxYaLyNp7tf1Ur5596":{"id":"vOyPfgp0xSDRDocvSkck","tag":"div"},"product-purchase-EJx6fHwdkQfs5HsV5464":{"id":"0Fwp6t0CvMWW4DKXrvl1","tag":"div"},"product-purchase-QqsQdaRZfLix4SFv5390":{"id":"GvHb7qpXU3ZbyT23IyCa","tag":"div"},"product-purchase-myb9iuvqnWPyZ9ZS5288":{"id":"CeO7EKkMVzLwCs2a4B8b","tag":"div"},"product-purchase-QLEcTNKMgEzLwj474399":{"id":"e8s5FCAzdjp3v2n2S17w","tag":"div"},"product-purchase-33APh2rbpMr3CoLw4366":{"id":"6IncxfWnql9Ry3iKh50K","tag":"div"},"product-purchase-J6G2R5WjaANyoBvO3802":{"id":"DPspcwFWY0YsXBhNylDK","tag":"div"},"product-purchase-FNULLNL7NsfDL2iU3372":{"id":"itmEuicd6tSGTYfyeT92","tag":"div"},"product-purchase-aKfSCnbU75ingBoz3369":{"id":"OqY8jDBJfFZPb3PF6WF3","tag":"div"},"product-purchase-Hzkxb8cQW839i1BF3358":{"id":"Hww5WUf3yF7cl64FpSrU","tag":"div"},"product-purchase-yYiQW4FoZVQaIj0f3347":{"id":"wkidC4ZjSFpD4S3znzWp","tag":"div"},"product-purchase-EUSO3tzNJWz82pro3339":{"id":"40i3nSqkXeoxqiHSeory","tag":"div"},"product-purchase-8p7VIhbPdlhdu7Vv3338":{"id":"ndAIGfRABihmt0u7IT0T","tag":"div"},"product-purchase-hx28ANKENTuaVKpf3336":{"id":"yrkVeL3pR4IYkygrhdu8","tag":"div"},"product-purchase-MFdCqZvvHPrO5wUF2596":{"id":"y9C5kpMtxN4Dvstjgmlf","tag":"div"},"product-purchase-ZuVNZD15YHAjyeXZ2312":{"id":"gGvw2omCzLl3hk1s4Gab","tag":"div"},"product-purchase-rRIesRZpv9DwUc4l2270":{"id":"GNnC209aFE4basog52gW","tag":"div"},"product-purchase-Uxcb1TGqlJ0qsl422263":{"id":"WYCC1IkMz4CFZXPhuoJv","tag":"div"},"product-purchase-MPxl1tSpOr1gFK132257":{"id":"TVMlPCpaRgOnKBF7E8at","tag":"div"},"product-purchase-TKUDRMQ5LQr2vPOH2256":{"id":"Z1SYSyBBJcJ5xDRJIq3K","tag":"div"},"product-purchase-clf0iKNNxbF3RP8S2246":{"id":"7L8LgAOrczfS5AEdR3Qk","tag":"div"},"product-purchase-Xm3U4jdGlnAqvyoL1068":{"id":"9JG1ldDPPSXooHAajlFx","tag":"div"},"product-purchase-eMpapFY3p5ovgN041011":{"id":"Q6qvzqqp3iRs20ddgEz2","tag":"div"},"product-purchase-MT0TvvJugLIPYoty733":{"id":"cdIo3ldJVSq5YhMnntK2","tag":"div"},"product-purchase-MiBlkeqoVQxZBm5T709":{"id":"lY0m5u5Utv1fApfe9gkU","tag":"div"},"product-purchase-00S4TJP3KwpW2LGE163":{"id":"aPrcqxInNQvTwxP0VXl3","tag":"div"},"product-subscribe-yRzy0FkZdLWdo7XR20773":{"id":"VmbEwrO7kHes9PKxocZB","tag":"a"},"product-subscribe-1mWTBfEM7P467JGk20758":{"id":"WQd4NZaN33sFWLwf5K1n","tag":"a"},"product-subscribe-r6WSTnxHwvXFvBwT20751":{"id":"4IDg4SjaZZ5IVMXIZLvj","tag":"a"},"product-subscribe-eIoK1MoaJpqxYnE820738":{"id":"VhDO4mLwF6VuBnxVQeud","tag":"a"},"product-subscribe-ocj4Dg5pkeMF6dlc20514":{"id":"jIpA3NgdMt5mKOrfUFBY","tag":"a"},"product-subscribe-5vQrM5G6bRWr3sdF20309":{"id":"WPin6VpzPMSTYJnHVHnc","tag":"a"},"product-subscribe-3dtA2RrYwf6eyjQJ18758":{"id":"vJyb3HjOHnjtd46eVBDd","tag":"a"},"product-subscribe-Zm9XK65Bs5hh3FnM18564":{"id":"8pztwlfBC2rtEJWdlCaq","tag":"a"},"product-subscribe-XTolx99rODPjIgtv16986":{"id":"dwheNGnH965gp9WDKsOr","tag":"a"},"product-subscribe-l2VIqWTmf87ZgqtD16828":{"id":"9DLlD5UFuZxxWwz2jsbO","tag":"a"},"product-subscribe-zYBxqCjwnKDu3yMp15939":{"id":"uL9sPDEdiG2r9oOzyIYQ","tag":"a"},"product-subscribe-sLigK9XqznyCgecf15829":{"id":"gT4u0yu2VQ1qvJDKcYLV","tag":"a"},"product-subscribe-GbUoDex2mywBdcFj15817":{"id":"SHrtQLBimqKUdi2OoRcH","tag":"a"},"product-subscribe-JlN9RokXNnQLOdGe15784":{"id":"PYkTYi4jGUllClA2DhJb","tag":"a"},"product-subscribe-gPjML6fVlHyUj2Ii15763":{"id":"ZE4K9qaU0zTJQUFgcEBX","tag":"a"},"product-subscribe-sJ6a8Ol57MjVr1VS15746":{"id":"D1oct0mExCsYlfnLpT3Y","tag":"a"},"product-subscribe-ei8K00jzGzyw0OAx15340":{"id":"moj8AU8ITJzjNNw3G4A3","tag":"a"},"product-subscribe-L6oPtzwTcOWK1lx315140":{"id":"YgYJ8QcMjJpMvEsYHIMy","tag":"a"},"product-subscribe-Lm7CewNrWqGydq5O15131":{"id":"8za2SqoS5df7rhR7OAzw","tag":"a"},"product-subscribe-GYsBLhYUgYKqSWPJ15113":{"id":"1oV4nPzGpO6yjWw3yv4W","tag":"a"},"product-subscribe-GZKGeEu4f4dvL2mf15112":{"id":"xLeCbMlKgU9bK7p30mMs","tag":"a"},"product-subscribe-aR6LZDzdIjscTZ6915111":{"id":"x8e9qCpRYIMfX9sYtWTv","tag":"a"},"product-subscribe-XpXcVBMlQes7SWZD15110":{"id":"ZwM81qTpWoz4o2SJrurN","tag":"a"},"product-subscribe-sL1akvAsI0L9M6Sp15109":{"id":"qerkCyPGdTqI50nfJVoQ","tag":"a"},"product-subscribe-ZFL5h89noWths4Pw15108":{"id":"FHELi6vbygpbXj7Bz5hH","tag":"a"},"product-subscribe-1rUWoausOxjfJvxe15103":{"id":"6bZbg6z4Rdwc44IPSIml","tag":"a"},"product-subscribe-R5LPt0IzebR9CToS15102":{"id":"UBBJDrcglaDdOodkzIw6","tag":"a"},"product-subscribe-4wFgoWyjcyKK99sn15101":{"id":"Jcus3pKf5XRMhJjYlDbh","tag":"a"},"product-subscribe-ZKIySkqwWyJRUGSs15100":{"id":"NnRRovqaJBxufzpmnCMT","tag":"a"},"product-subscribe-yvTcT83IDrNlTWVh15075":{"id":"oTGJT5zdZhirdTBUhlwv","tag":"a"},"product-subscribe-jYkAw18TU2kKucyX15074":{"id":"1NjVteiNou2cqsZlrEPf","tag":"a"},"product-subscribe-v6s8a3Ly9FG43W3A15073":{"id":"QWR927Zsuqg84db2KruG","tag":"a"},"product-subscribe-k5TBn2yPgSrstwLZ15072":{"id":"bi7leKZWdvnuUkMilp4J","tag":"a"},"product-subscribe-6T365pFVLQdMVmDZ15060":{"id":"LfuUmRRA2js9GoC0L2Nx","tag":"a"},"product-subscribe-ZHQGi0hxGzAHJ39A15059":{"id":"ZemHtbGEerLcoitho0RD","tag":"a"},"product-subscribe-HE50lRzpS5AgStYb15057":{"id":"6M2Rka0WINDr4PYGZZ3M","tag":"a"},"product-subscribe-bT35k73z60Ed9yY915045":{"id":"3uQt7GU9Uo3tyORYAMil","tag":"a"},"product-subscribe-SPO0MvNJuabC6NAp15043":{"id":"0m8r2tEPDoIyAtBYjzWV","tag":"a"},"product-subscribe-WvCz6mOPZZMLDMKP15042":{"id":"rfngJqbQp6DsQKX62m3g","tag":"a"},"product-subscribe-vSvxWf8pytHJN9pp15041":{"id":"zpsSYqJuKaf7lyXGvCVL","tag":"a"},"product-subscribe-HvUSSwU9aLtdwERi15026":{"id":"65bRSZdGU2BiXvOPvK1M","tag":"a"},"product-subscribe-JoBhsEYknTwCNQO615025":{"id":"nGLJb3g4dfgFyLG1E2RP","tag":"a"},"product-subscribe-UmEYlSKPPsPycaVX15024":{"id":"LKo35Zluk4e0Koj68rPq","tag":"a"},"product-subscribe-UsvB6PTbFfAoI9hx15023":{"id":"lC4lOpZw4Lh9MPt2qpkP","tag":"a"},"product-subscribe-I1WyE6ufwQ9G1Ft515022":{"id":"x0JDVCXur1mMgF8dIt3w","tag":"a"},"product-subscribe-y7y2MxFlWFUbAMbM15021":{"id":"k87Sszv2eGTeTsPUiBzh","tag":"a"},"product-subscribe-FlzEb2POwkiguiUI15020":{"id":"BLGlCiJ1szZrlHpiP7ZQ","tag":"a"},"product-subscribe-g4vTMLyzwuIkVadn15019":{"id":"ciCtorj47jS3d8wUFejC","tag":"a"},"product-subscribe-QQeLYvJ8VFxEEntJ15017":{"id":"mzFB1B82fUE9kUv8WEl8","tag":"a"},"product-subscribe-ikCVKwzLPe1u7b5w15016":{"id":"cfxQ1ZWKs0wKx2yMF5bK","tag":"a"},"product-subscribe-N6NxXqB6HFevy1nz15014":{"id":"z4URc41Su7sUgWvteelw","tag":"a"},"product-subscribe-ueCdo719n5SvnyfP15012":{"id":"Ht7GXRGEG4hkUmiHt84A","tag":"a"},"product-subscribe-qpmviK0UvlJcHEqG15011":{"id":"CJwyMYE2bXbbQOBsEnYk","tag":"a"},"product-subscribe-85Ua7zLrK9AleFjK14954":{"id":"ML4Vb5woaOYnShZ9dqQO","tag":"a"},"product-subscribe-PKq65IoGJ1PyUD1x14953":{"id":"mt01jkD5c4DFpMfA4NGk","tag":"a"},"product-subscribe-FtAxVQtqbs8TaYoL14951":{"id":"FZhNs0Q21dVZLpYuQlDW","tag":"a"},"product-subscribe-JhA1CePxtb24Gvju14948":{"id":"SbI5buYEza9VHqelZe0y","tag":"a"},"product-subscribe-yeLjGuGtngHDYKwX14947":{"id":"PnQm4Xbmn4AmaLOa6G5p","tag":"a"},"product-subscribe-2bxzRKWUCmiI6eQg14946":{"id":"1rD5hutK5sLOBEaRzNLT","tag":"a"},"product-subscribe-50LpKJCiZTPNWgJ514945":{"id":"5A4zs9nn5eAUq4NsR70x","tag":"a"},"product-subscribe-tM9jf0BsgwsEG9hT14911":{"id":"hN9yFkEBUzYpC2nz7jgn","tag":"a"},"product-subscribe-7deI0GyTIbfVOcBS14895":{"id":"y8uTAVS65X0EBcvVr641","tag":"a"},"product-subscribe-fypkinhPWy81wlfa14894":{"id":"lMPYeNQzt2iW2ivOGnmB","tag":"a"},"product-subscribe-mXxG0vTN164c35jF14893":{"id":"Ep4UtICtRAlwcfh2Emtu","tag":"a"},"product-subscribe-S0oDv6N93Xk9bPdi14892":{"id":"7mWdOLHM4BkLq9gyKOaY","tag":"a"},"product-subscribe-05ZfIFm0Cnhl7Imb14891":{"id":"0Bci5ghDP3QlwJSf0KaV","tag":"a"},"product-subscribe-Gp3WyclRaYbn36dD14890":{"id":"f9D3tYyGHZoAu5NJXHdG","tag":"a"},"product-subscribe-KLVrzBEsYviELaSj14889":{"id":"0ZbBEfGBcumwHNp7ZLNO","tag":"a"},"product-subscribe-X02QnUbn3xudmZ1L14888":{"id":"hVou04c0KJRqKsOSbJdN","tag":"a"},"product-subscribe-yXla3oPVDYN3Vur214887":{"id":"PSDR4KT6JyIUbpzMRPno","tag":"a"},"product-subscribe-5jNhetoC9Bk1XyPS14846":{"id":"y7iTY5Ev4WtKdzQuxfrf","tag":"a"},"product-subscribe-jdyOsUDtLILYi3in14845":{"id":"dN9jQJxzoHBS3xbGgaHV","tag":"a"},"product-subscribe-gXry0Ckk4R3BT33I14844":{"id":"S2XMx6bmPsL8sFhZ2xSm","tag":"a"},"product-subscribe-GQLIe2jHz8qlm1wz14843":{"id":"yPv6y9kFCze7AablsZJB","tag":"a"},"product-subscribe-sOSj4GOn3b71eEYr14842":{"id":"7ykiDhZjNZnwRnj2ylL7","tag":"a"},"product-subscribe-jKRg5V6Pc8iFcEzQ14841":{"id":"WD98LMGnvF5u84SvsZys","tag":"a"},"product-subscribe-31PEcO75pIcROBQU14840":{"id":"R4ww7PhUfIWlAUwxsT7O","tag":"a"},"product-subscribe-HJyjOq2tikd2FSK614798":{"id":"aqBbbagDahKCsyrf5jGj","tag":"a"},"product-subscribe-218PhgawNrm1JELd14797":{"id":"OkjW50DWfvwrYu8gyald","tag":"a"},"product-subscribe-wzaEfWMwGxJoOY2u14796":{"id":"m5HbgsQ6Vz8z1kY4F1XD","tag":"a"},"product-subscribe-TUqdfeOJEl1kzzOT14795":{"id":"3kK0p9Enf6wjmm42ffLU","tag":"a"},"product-subscribe-NVVYbYS3DrjYDnbF14774":{"id":"G2umaj0vz2sKUzIOwPRh","tag":"a"},"product-subscribe-PomMwfXUqs2IOtIm14762":{"id":"uN6NNaD6b1U9sjNto7dN","tag":"a"},"product-subscribe-c5EPhrLOMTlZ7Tuv14761":{"id":"QeHeSZVtyNuu9zNRyzmu","tag":"a"},"product-subscribe-BG6RUC1AATxL2G6014725":{"id":"kzm8VCfne3XwJeaSXAmt","tag":"a"},"product-subscribe-0gaBCnnSQvwbkS9G14707":{"id":"ZinqyAtnd8T43iEaG929","tag":"a"},"product-subscribe-AVfjo8KFcEkACtTX14701":{"id":"CtqyeablMWYjAZz4MQpq","tag":"a"},"product-subscribe-1c5PbFKkfGWTys6614700":{"id":"mD69zoMCK433XBWBNio5","tag":"a"},"product-subscribe-qWn2qbh0IKfNpfsk14699":{"id":"xdHvh84rQO70o4yflUzA","tag":"a"},"product-subscribe-tFnnvaMXN4MyxZtW14698":{"id":"UGRUo17rWYSFwAfRrw3M","tag":"a"},"product-subscribe-BROlVfnoMtJk8W4j14697":{"id":"Zc8raAUZh0K0f7N4c8NW","tag":"a"},"product-subscribe-Mp7DAJ9pmRuD9yPF14696":{"id":"xedSCruQosmIA15bJj3M","tag":"a"},"product-subscribe-iN5lKARgJWKwEDPJ14695":{"id":"qjuiypEF50E8mLFFqhlK","tag":"a"},"product-subscribe-nMz8C6x9pQPAzt3y14694":{"id":"7bfRDQJYQQn9dPy4xDqa","tag":"a"},"product-subscribe-YmO2AddjHOkAaQbk14481":{"id":"eWoVovbjGifEq4xZOmut","tag":"a"},"product-subscribe-Hf03N2WnGVoO542X14479":{"id":"JWl7ZWMFcGdpQAp6JWPO","tag":"a"},"product-subscribe-W4zRXjcSq8S1Q26314393":{"id":"THVgQqBm81pt9yTuSdzr","tag":"a"},"product-subscribe-x4q89iHlGfzFaHDF14346":{"id":"Bs3GDYjKvPKIDX0EiSaQ","tag":"a"},"product-subscribe-2pauKbKqsWPgd6T814345":{"id":"xOkvQkc0xysanYfmOiVq","tag":"a"},"product-subscribe-YmW8kp91gpaTzrOS14318":{"id":"Bl6uZEgfzGTGEOSqNdTG","tag":"a"},"product-subscribe-oJJi3QSe1oKjIXyJ14288":{"id":"hLa5WGH6HX8viaInx4nA","tag":"a"},"product-subscribe-sILG315q7DgNmWJH14275":{"id":"0QjBeKzvv5U0cYgk2M0i","tag":"a"}},"errors":[],"htmlHash":"87fc3f9f","data":{"books":[],"perPage":6,"page":27,"category":[],"ids":[14]},"dataMeta":{"modelCollections":{"books":{"class":"App\\Models\\Book","id":["20401","19255","19253","18496","18245","18218","17621","17348","17074","17068","16469","16452","16451","16010","15963","15740","15511","15409","14799","12394","10180","10178","9259","8620","8041","7888","6800","6769","6733","6505","6492","6482","5598","5597","5596","5464","5390","5288","4399","4366","3802","3372","3369","3358","3347","3339","3338","3336","2596","2312","2270","2263","2257","2256","2246","1068","1011","733","709","163","20773","20758","20751","20738","20514","20309","18758","18564","16986","16828","15939","15829","15817","15784","15763","15746","15340","15140","15131","15113","15112","15111","15110","15109","15108","15103","15102","15101","15100","15075","15074","15073","15072","15060","15059","15057","15045","15043","15042","15041","15026","15025","15024","15023","15022","15021","15020","15019","15017","15016","15014","15012","15011","14954","14953","14951","14948","14947","14946","14945","14911","14895","14894","14893","14892","14891","14890","14889","14888","14887","14846","14845","14844","14843","14842","14841","14840","14798","14797","14796","14795","14774","14762","14761","14725","14707","14701","14700","14699","14698","14697","14696","14695","14694","14481","14479","14393","14346","14345","14318","14288","14275"],"relations":["series","publisher","authors","authors.media","discountOffer","media","subscribers"],"connection":"mysql"}},"models":{"category":{"class":"App\\Models\\Category","id":"سياسة","relations":["children"],"connection":"mysql"}}},"checksum":"a978b9f9cb184baf2016d28758b9f4efa7faf13cadc95f1a53d44903ac4ce7ca"},"updates":[{"type":"callMethod","payload":{"id":"2a5n","method":"loadMore","params":[]}}]}
    
  

headers = {
    "content-type":"application/json",
    "x-csrf-token": token,
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36",
    "x-livewire": "true"
            
        }


req = session.post('https://www.aseeralkotb.com/livewire/message/categories.books.index', json=data, headers=headers).json()


json_content = req['effects']['html']
soup = bs(json_content, "lxml")
#print(soup)


for card in soup.select('div[class="flex"]'):
    name = card.select_one('div.flex.flex-col.items-center > a > h1').text
    print(name)

输出：

سياسات تقنين الشريعة
بعد الليبرالية        
الاعلان الاسلامي      
في قلب الأحداث        
الدولة والثورة        
أحاديث فى آسيا        
العالم عام 2050       
حرب المائة عام على ف..
تاريخ العصامية والجر..
البرجوازية المصرية و..
الدبلوماسية الإقتصاد..
عدني يا أبي - عام من..
عولمة - لا حرب حضارا..
الإستياء العالمي - ح..
العبودية مقابل الامن..
أيام مرسي
11 يوما مات بعدها جم..
33 استراتيجية للحرب   
المختطفات : شهادات م..
روسيا الأوراسية كقوة..
ملفات السويس
سنوات الغليان
اختلال العالم
الماسونية
خرافة التقدم والتأخر
نحن شعب مهذب - تأملا..
خوف
كل يوم هو اضافة
هرطقات 2: عن العلمان..
الحصار
الجورنالجي وكاتم الا..
أميركا والقوى الصاعد..
مقدمة قصيرة عن الحرب..
مقدمة قصيرة عن الثور..
مقدمة قصيرة عن الثور..
القاهرة - موسكو
قصة سقوط صدام حسين
فن الحرب
الحروب الميسرة
أسرار حرب الغاز
اليوتوبية: مقدمة قصي..
الحياة السرية لصدام..
موسوليني - أسطورة لا..
الأسرار الكبرى للماس..
أسرار وحقيقة الماسون..
أحجار على رقعة الشطر..
يأجوج ومأجوج- من الو..
الشيطان أمير هذا الع..
غزة في أزمة - تأملات..
كل جيش الكرملين ( مو..
أخيل جريحا  ( إرث جم..
ماذا حدث للمصريين؟
العلمانية الجزئية وا..
العلمانية الجزئية وا..
عودة الوعى
موسوليني وهتلر : قصة..
الثورة الدائمة
نار وغضب - البيت الأ..
حلقات زحل
كفاحي
لعبة الأمم
آليات المؤامرة الكون..
‎استنساخ الفشل التار..
عمالقة الفن الاسود
الرقابة و التعتيم فى..
الاختراق الصهيونى لل..
جند الخلافة
محكمة حقوق الإنسان ا..
مذكرات هيرمان بلاك
الجيش الأمريكى فى حر..
روح الشرائع
خنجر إيران
المتآمرون على الأمة
الثورة العميقة رحلة..
جمهورية الضحك الأولي
عالم بدون اسرائيل
المقاصد الشرعية في ا..
الحروب الامريكية الج..
معجم المصطلحات السيا..
حقوق عربية ضائعة .....
من الشاه الى نجاد....
العلاقات المصرية الخ..
الحكومة الإلكترونية..
العمالة الأجنبية في..
الديموقراطية الخليجي..
الديمقراطية الكويتية..
مصطفى النحاس ودوره ف..
الأمم المتحدة والتدخ..
الاسلاموفبيا جماعات..
تاريخ أفغانستان وتطو..
الثورة التشيكية وتجر..
قوة المستضعفين
اشهر قضايا الاغتيالا..
المرأة والربيع العرب..
التجربة البرازيلية ق..
اسرائيل في حوض النيل
أوربيانا - مختصر تار..
يوميات دولة انتقالية
تجربة جنوب افريقيا -..
التجربة الماليزية -..
المياة العربية وحروب..
سياسة ايران تجاة دول..
التجربه التشيلية .....
التجربه الاندونسيه ...
التجارة الخارجيه الي..
الدور البرلمانى للمر..
محنه الدستور ... وإش..
الحروب بالوكالة - اد..
العلاقات السعوديه ال..
هاربون من الموت : ال..
كيفية صناعة التطرف ؟..
الفساد والإصلاح : ال..
العلاقات الأمريكية ا..
التجربة الصينية الما..
دينى ودين الناس .. ا..
الأرض المحرمة - كوري..
القضية الكردية : إشك..
العلاقات الإيرانية ا..
السياسة الخارجية الت..
العلاقات الإيرانية ا..
القومية والأمة: مدخل..
التجربة الالمانية ....
التجربة الإماراتية :..
بوكو حرام : من الجما..
التجربة الهندية : أك..
ماذا تعرف عن البركست..
العلاقات اليمنية الخ..
أمريكا وحركات الإسلا..
المشروع الصهيوني: ال..
وحدتنا الوطنية بين ث..
ليبيا ما بعد القذافي..
النظام الانتخابي ....
ضد الانتخابات .. دفا..
التجربة الكندية .. ب..
التجربة الإيرانية....
التجربة الأرجنتينية..
التجربة الباكستانية..
الثورة والحرب .. تشك..
الأقليات في أثيوبيا..
العلاقات السورية الت..
العلاقات الصينية الأ..
كتابات هيكل .. بين ا..
التعليم الالالكتروني..
البرنامج النووي الاي..
لماذا تنتفض الشعوب ؟..
العنف المقدس.. النفو..
قضايا الخيانة والجاس..
مدخل الي القرن الافر..
إخوان مصر .. بين الص..
أمريكا والربيع العرب..
العلاقات المصرية الا..
العلاقات بين الكوريت..
سد النهضة الإثيوبي ...
آخر أيام البشير .. ا..
التجربة.. تفاعلات ال..
عزيزي النفط.. ماذا ف..
ملوك النفط .. كيف قل..
الشغف العربي
المنافسة على القمة و..
اليمن والقاعدة.. الح..
في عين العاصفة
مصدق والصراع على الس..
مصدق والصراع على الس..
الفكر السياسي للأحزا..
الثورات العربية.. سي..
التمرد السوري
قيامة شعب .. قراءة أ..
في أصول لبنان الطائف..

【讨论】：

能否在脚本中添加一些 cmets 以了解更多信息
@ Karim Bn Abdlaziz，该网站受 cloudflare 保护，这就是我使用 requests 的原因，因为 scrapy 不处理 cloudflare，但 requests 可以。 url 使用的是动态的 csrf 令牌。所以我必须从 url 的 html 页面中选择它。请求是 json 方式的 post 方法，但数据是来自 API 调用的 html。所以我必须发送两个请求：第一个是“获取请求”以提取 csrf 令牌，第二个是 API 请求（第二个 url）。我使用 bs4 从 html 中获取数据。谢谢
我使用requests.Session()通过请求本身来处理cookie。
我还将 css 选择器与 bs4 一起使用。在最后一步中，您可以从 get url 中选择访问 url 中的每个 htlm 元素。