我正在编写一个 Python 脚本来每天抓取超过 300 个关键字的搜索结果的前 10 页。会不会被谷歌识别和屏蔽？
还是我需要使用它自己的 API (https://developers.google.com/custom-search/v1/overview#pricing)，它不是免费的？
多谢！
【问题讨论】：
标签： python google-chrome search
是的，你肯定会在某个时候被谷歌屏蔽。那就是您应该使用旋转代理的地方。它会为每个新连接更改 IP，从而将被阻止的机会降到最低。
SerpAPi 之类的解决方案为您完成了所有这些工作。检索数据可能与进行 API 调用一样简单。它还提供 JSON 格式的格式化数据。示例 Google 搜索 https://www.google.com/search?q=Coffee 响应如下所示
{
...
"local_map":{
"link":"https://www.google.com/search?q=Coffee&npsic=0&rflfq=1&rldoc=1&rllag=30267328,-97742779,104&tbm=lcl&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQtgN6BAgeEAc",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/2c7265ad4a543339817d42073c4c5df8.png",
"gps_coordinates":{
"latitude":30.267328,
"longitude":-97.742779,
"altitude":104
}
},
"local_results":{
"more_locations_link":"https://www.google.com/search?tbs=lf:1,lf_ui:9&tbm=lcl&q=Coffee&rflfq=1&num=10&uule=w+CAIQICIaQXVzdGluLFRleGFzLFVuaXRlZCBTdGF0ZXM&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQjGp6BAgeEGE",
"places":[
{
"position":1,
"title":"Starbucks",
"place_id":"10605736027611436825",
"lsig":"AB86z5XTJ_Io_anVBu2fU6Zaqu3b",
"place_id_search":"https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5XTJ_Io_anVBu2fU6Zaqu3b&ludocid=10605736027611436825&q=Coffee&tbm=lcl",
"reviews":506,
"price":"$$",
"type":"Coffee shop",
"address":"600 Congress Ave",
"thumbnail":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/1da898cc867dbcb3d01977dd6a88324830a17bef6769367010e21458fda67be97c9c3f66ca35fbd3.jpeg",
"gps_coordinates":{
"latitude":30.26826,
"longitude":-97.74296
}
},
{
"position":2,
"title":"Houndstooth Coffee",
"place_id":"11265938073076301333",
"lsig":"AB86z5Vdw6C2pJpM0xQ6JUx2KONU",
"place_id_search":"https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5Vdw6C2pJpM0xQ6JUx2KONU&ludocid=11265938073076301333&q=Coffee&tbm=lcl",
"reviews":740,
"price":"$$",
"type":"Coffee shop",
"address":"401 Congress Ave #100c · In Frost Bank Tower",
"thumbnail":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/1da898cc867dbcb3d01977dd6a883248c09e12c31822db27da4d3f71793b876103709dcadd9705be.jpeg",
"gps_coordinates":{
"latitude":30.2664,
"longitude":-97.74278
}
},
{
"position":3,
"title":"Lucky Lab Coffee",
"place_id":"10608433616590646585",
"lsig":"AB86z5X1H_n2-O4x616ohROZpOhi",
"place_id_search":"https://serpapi.com/search.json?device=desktop&engine=google&gl=us&google_domain=google.com&hl=en&location=Austin%2C+Texas%2C+United+States&lsig=AB86z5X1H_n2-O4x616ohROZpOhi&ludocid=10608433616590646585&q=Coffee&tbm=lcl",
"reviews":2,
"type":"Cafe",
"address":"515 Congress Ave · In the Bank of America Financial Center",
"thumbnail":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/1da898cc867dbcb3d01977dd6a88324886cadf9d168daae110c2dd1a08b2d23547f91a9442817f71.png",
"gps_coordinates":{
"latitude":30.267742,
"longitude":-97.74261
}
}
]
},
"knowledge_graph":{
"title":"Coffee",
"type":"Drink",
"header_images":[
{
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8bfff7eefb2c6d74eee99d69c4481dd1679df8e9c6a784e9.jpeg",
"source":"https://en.wikipedia.org/wiki/Coffee"
},
{
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8bfff7eefb2c6d747d1b7b1de6208d46ae515ba41bd21677.jpeg",
"source":"https://www.nbcnews.com/better/lifestyle/how-tap-health-benefits-coffee-ncna1096031"
},
{
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8bfff7eefb2c6d748906d5985fcefcd48ad5887b4e84fc15.jpeg",
"source":"https://www.independent.co.uk/life-style/food-and-drink/coffee-weight-loss-healthy-nutritional-value-fat-burn-diet-metabolism-a8296946.html"
},
{
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8bfff7eefb2c6d745a2748e50daa1b02ad752b079122ff3f.jpeg",
"source":"https://austin.eater.com/maps/best-coffee-austin-cafes-patio-latte-pour-over"
}
],
"description":"Coffee is a brewed drink prepared from roasted coffee beans, the seeds of berries from certain Coffea species. From the coffee fruit, the seeds are separated to produce a stable, raw product: unroasted green coffee.",
"source":{
"name":"Wikipedia",
"link":"https://en.wikipedia.org/wiki/Coffee"
},
"patron_saint":"Saint Drogo of Sebourg redrockroasters.com",
"patron_saint_links":[
{
"patron_saint_text":"redrockroasters.com",
"patron_saint_link":"https://www.redrockroasters.com/shop-accessories/saint-drogo-pocket-retablo"
}
],
"chicory_coffee":[
{
"name":"Cafe Du Monde Coffee",
"link":"https://www.google.com/search?q=Cafe+Du+Monde+Coffee&stick=H4sIAAAAAAAAAA3Iuw3CMBAAUEUoEhQ0KANElDSH00RKG1pmsIJ9_ijnOwcD2YeSCRgPXvm2m8MOEpy71-LScQ8elApKL77rw6mZzQBGUhIerKy8TndbPpX6HxGaRxSGmWUltB51nhipQAmSc2Sv56yRoo83wnfdjJPD9vJsr8IW21GcQ_zW1Q_6j3k0gAAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg7EAU",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c0f6e783ada4b70fd6842017d38d47bed2998db83edddf2f6.jpeg"
},
{
"name":"12 oz. Ground Pure Chi...",
"link":"https://www.google.com/search?q=12+oz.+Ground+Pure+Chicory&stick=H4sIAAAAAAAAAA3IMQ6CMBQA0BBDooOLcXJqHB38FDdWB1dvQLD9lNL2_1oEosdx9AQeT9_4lovNCgIU5XRvw34NBqRsu37Sp6I_bJ2qQHEITJXmmeYm6eGTyf95j-phmcARzx61wTo2hH6AoeMYLZnaxRq9Nfbm8Z3vZCn4dRSXxCNpcR0TinNnFafnN89-R-f-lYYAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg7EAc",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c0f6e783ada4b70fd6ffaf528a1df5363c5721124b0c4d0d0.jpeg"
},
{
"name":"Monterey Bay Spice Chicory...",
"link":"https://www.google.com/search?q=Monterey+Bay+Spice+Chicory+Root+Roasted+Granules+1+LB+Bags+Natural+Coffee+and+Tea+Substitute+Caffeine+Free+Beverage+Certified+Kosher&stick=H4sIAAAAAAAAAA2MPU7EMBBGFaGVoKBBHGBESWOCtM2WWQkKfgqWPpq1x44VZ8bYTqL0nISSE3A83HzF-57e5cXNlZrUw-PyZae7a-VU29p9Wvppv93fjvqgtEyT8MHIyismk3-btrIQSBcvrEaWNZBx1EdkClnlQWL07Pox9hS88-dAP7vv5k24UKINOtzgFL0mOA5eS9rgQ6TUwVzIwHNCngNlaOG1q7LL8I5lThjgKNYSAbKBT0I4zedcfJlLLWF9PBM8pSp0tFBCVzGl4q2v1RfJA6W_XfMPNZovy_EAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg7EAk",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c0f6e783ada4b70fd66c3eead310388e95fa3a7d9b692e6d5.jpeg"
},
{
"name":"Herbaila Chicory Roasted...",
"link":"https://www.google.com/search?q=Herbaila+Chicory+Roasted+Root+Granules+Lb+Coffee+Free+Tea+Natural+Substitute+%26+Pound+New&stick=H4sIAAAAAAAAAA3IPWrDQBBAYYQR2EWa4AMMLlyk2SgpDG4NtotggpMinZjVjuS1Znfk_YnwdVLmBDle1LwP3nz2uFBOPb9831q3elCdqqpr1YfXjQtPy77ZqkacE781MvoRg4m_RTU9ZmqSFa96LyOT6age0BNHFS8yDNZ3dT_UxLazmumn_DpS0GgZYXexjYQ7nAVjIjMpCQ4BfWaK8KZhJ21LBPsw5ZMQTphyQIaPrGOyKSeCNbxL9gZONP6VxT_Ryrt-xAAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg7EAs",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c0f6e783ada4b70fd80f296058a363e88e80a2c1b567a76c1.jpeg"
},
{
"name":"Community Coffee Coffee...",
"link":"https://www.google.com/search?q=Community+Coffee+Coffee+%26+Chicory+Ground+Coffee&stick=H4sIAAAAAAAAADXMPw6CMBSA8RBDooOL8QCNg4PLE0dWBo_RYPsoDe17_QMSruPoCTyeDDp9yW_4tpvDDjxcb8_Y-dMeDFRVzD76KMPlOKgaFHvPVGueaW6Tzu-iWs05VKNlgoF4dqgNytASugy55xAsGTkEic4a-3D4KqFZNxPZcRENdx3iP2fR9FZxWsQ98UT655-y-AJrgg6MmwAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg7EA0",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c0f6e783ada4b70fdec4a6b7871ef989286cf3b6521287889.jpeg"
}
],
"chicory_coffee_link":"https://www.google.com/search?q=Chicory+coffee&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtVQjC1RLKTrfST83Nz8_OsUvLL88oTi1KKVzEaAsVyclKTSzLz8_Sz8_LLc1JT0lPjCxLzUnOK9Ysz8gsKMvPS47ML4lNzMtMzk3JSF7HyOWdkJucXVSok56elpabuYGUEAEAR73d2AAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQMSgAegQIOxAB",
"chicory_coffee_stick":"H4sIAAAAAAAAAONgFuLUz9U3MCorTMtVQjC1RLKTrfST83Nz8_OsUvLL88oTi1KKVzEaAsVyclKTSzLz8_Sz8_LLc1JT0lPjCxLzUnOK9Ysz8gsKMvPS47ML4lNzMtMzk3JSF7HyOWdkJucXVSok56elpabuYGUEAEAR73d2AAAA",
"coffee_books":[
{
"name":"The World Atlas of Coffee: F...",
"link":"https://www.google.com/search?q=The+World+Atlas+of+Coffee:+From+Beans+to+Brewing+-+Coffees+Explored,+Explained+and+Enjoyed&stick=H4sIAAAAAAAAAC3JQQqCQBSAYSSEWrQoOsCjZRSjQRDuMuwEQdBudN6YOvNezUjadVp2go5XRLuf7x8OpiNhRbS-37Sdj0Up4jjfmH5b1vVi1hSJKNhapkRxR510yr-CydeMwaKtmETO3PhneD5eEE7sjIJda6QH1rBnrRETODi2kKIkDy1D6rCrqITV_3vI-qthh2r5K1kRKpCkIKOaH6jeYfABYiUriKYAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg9EAU",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c731197c286d418eafe8e1d75f61c7e84a5f4ff7386100ca7.jpeg"
},
{
"name":"Craft Coffee: A Manual",
"link":"https://www.google.com/search?q=Craft+Coffee:+A+Manual&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtV4tVP1zc0TKmwTDe2yC7TEslOttJPzs_Nzc-zSskvzytPLEopXsUoCBTLyUlNLsnMz9NPys_PLl7EKuZclJhWouCcn5aWmmql4Kjgm5hXmpizg5URAFoyNJBiAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg9EAc",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c731197c286d418eae2ca32b7fe5a3965f293af100e280f16.jpeg"
},
{
"name":"The Blue Bottle Craft of...",
"link":"https://www.google.com/search?q=The+Blue+Bottle+Craft+of+Coffee:+Growing,+Roasting,+and+Drinking,+with+Recipes&stick=H4sIAAAAAAAAAB2IMQrCMBRAESkouCge4OMoQrRO7WgFN4fiBdLkpw1N8jWJxvM4egKPZ-nyeO_Npqs5s2yfvx7KbhasZYe8scWxkO_tuhclE2QtuVJScol7Gb6T5fCMQRE1OdYQ9eGTXW8dwsk8B1CMBqHyXEUgBRUphVjCxVPSrt1BTTzE0biTcPba9WMlHTuoUeg7hl82-QNFNQFmmQAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg9EAk",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c731197c286d418ea2a4c4491e5b91967e98490fc47b5439b.jpeg"
},
{
"name":"The Professional Barista's...",
"link":"https://www.google.com/search?q=The+Professional+Barista%27s+Handbook&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtV4tFP1zc0SsrNzS0qttASyU620k_Oz83Nz7NKyS_PK08sSilexSgIFMvJSU0uyczP00_Kz88uXsSqHJKRqhBQlJ-WWlwMFE7MUXBKLMosLklUL1bwSMxLASnbwcoIAIZvHchuAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg9EAs",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c731197c286d418ea34890ccba17763b33af30dbf2d1cf08b.jpeg"
},
{
"name":"Everything But Espress...",
"link":"https://www.google.com/search?q=Everything+But+Espresso:+Professional+Coffee+Brewing+Techniques&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtV4tVP1zc0TKkqLjdLKTfTEslOttJPzs_Nzc-zSskvzytPLEopXsUoCBTLyUlNLsnMz9NPys_PLl7Eau9allpUWZKRmZeu4FRaouBaXFCUWlycb6UQUJSfBmQB1SbmKDjnp6Wlpio4FaWWg1SGpCZn5GUWlqYW72BlBACYMYYFiwAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg9EA0",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c731197c286d418ea3a06871f9d0f679705e1fe6f7915c558.jpeg"
}
],
"coffee_books_link":"https://www.google.com/search?q=Coffee+books&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtVQjC1RLKTrfST83Nz8_OsUvLL88oTi1KKVzEKAsVyclKTSzLz8_ST8vOzixex8jjnp6WlpiqAuTtYGQHBbXIpVAAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQMSgAegQIPRAB",
"coffee_books_stick":"H4sIAAAAAAAAAONgFuLUz9U3MCorTMtVQjC1RLKTrfST83Nz8_OsUvLL88oTi1KKVzEKAsVyclKTSzLz8_ST8vOzixex8jjnp6WlpiqAuTtYGQHBbXIpVAAAAA",
"people_also_search_for":[
{
"name":"Tea",
"link":"https://www.google.com/search?q=Tea&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtV4gAxzZNzKrQEgzNTUssTK4v9UitKgktSC4oXsTKHpCbuYGUEAEjPygozAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg8EAU",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8922f70020484e82863a1fb14ad7834ba7ce81bed561e6d405f00b65c2c336fb.jpeg"
},
{
"name":"Espresso",
"link":"https://www.google.com/search?q=Espresso&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtV4gAxk0vSsrUEgzNTUssTK4v9UitKgktSC4oXsXK4FhcUpRYX5-9gZQQADwmvdzgAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg8EAc",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8922f70020484e82863a1fb14ad7834bca0c55e3d185f15847b7398b104adad3.jpeg"
},
{
"name":"Drink",
"link":"https://www.google.com/search?q=Beverage&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtV4tZP1zc0MkoryDMr0hIMzkxJLU-sLPZLrSgJLkktKF7EyuGUWpZalJieuoOVEQCpF_cVOwAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg8EAk",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8922f70020484e82863a1fb14ad7834bb8f7bd4c5777e7c99a90f6440d6de376.jpeg"
},
{
"name":"Bakery",
"link":"https://www.google.com/search?q=Bakery&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtV4gIzy8vMSjK0BIMzU1LLEyuL_VIrSoJLUguKF7GyOSVmpxZV7mBlBAAYMLooOAAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg8EAs",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8922f70020484e82863a1fb14ad7834b57c37e90b1da558036eee2476bbeeca1.jpeg"
},
{
"name":"Iced coffee",
"link":"https://www.google.com/search?q=Iced+coffee&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtVAjNNs5PKDLUEgzNTUssTK4v9UitKgktSC4oXsXJ7JqemKCTnp6Wlpu5gZQQADVX9azwAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQxA16BAg8EA0",
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c8922f70020484e82863a1fb14ad7834bab658e29bb3ef9fcbaff7da83ed64051.jpeg"
}
],
"people_also_search_for_link":"https://www.google.com/search?q=Coffee&stick=H4sIAAAAAAAAAONgFuLUz9U3MCorTMtVQjC1BIMzU1LLEyuL_VIrSoJLUguKF7GyOeenpaWm7mBlBABkIv_mNwAAAA&sa=X&ved=2ahUKEwjUxqjxgMXzAhUhmWoFHfPSA5QQMSgAegQIPBAB",
"people_also_search_for_stick":"H4sIAAAAAAAAAONgFuLUz9U3MCorTMtVQjC1BIMzU1LLEyuL_VIrSoJLUguKF7GyOeenpaWm7mBlBABkIv_mNwAAAA",
"see_results_about":[
{
"name":"Coffee",
"extensions":[
"Plant"
],
"image":"https://serpapi.com/searches/6165916694c6c7025deef5ab/images/3da5c3104e4a51e58340d10119d9ef0c4dbfdfcfff1cbe01c2e0271548875123a645e28ce8634e54d852f39ec78d2cfe.jpeg"
}
],
"list":{
"total_fat":[
"0 g",
"0%"
],
"saturated_fat":[
"0 g",
"0%"
],
"trans_fat_regulation":[
"0 g"
],
"cholesterol":[
"0 mg",
"0%"
],
"sodium":[
"5 mg",
"0%"
],
"potassium":[
"116 mg",
"3%"
],
"total_carbohydrate":[
"0 g",
"0%"
],
"dietary_fiber":[
"0 g",
"0%"
],
"sugar":[
"0 g"
],
"protein":[
"0.3 g",
"0%"
],
"caffeine":[
"95 mg"
],
"vitamin_c":[
"0%"
],
"calcium":[
"0%"
],
"iron":[
"0%"
],
"vitamin_d":[
"0%"
],
"vitamin_b6":[
"0%"
],
"cobalamin":[
"0%"
],
"magnesium":[
"1%"
]
}
},
...
}从Playground 试试我们的。您还可以注册以获得免费搜索。
免责声明，我为 SerpApi 工作。
【讨论】：