【发布时间】：2021-06-13 23:01:37
【问题描述】：
所以我在这里看到了这个帖子， Crawl Quora Q&As using BeautifulSoup
并被介绍给 Selenium...
我从这里学会了如何安装它， https://gist.github.com/korakot/5c8e21a5af63966d80a676af0ce15067
因为我想抓取一个 Quora 网页，所以我在 colab 上编码...
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
# open it, go to a website, and get results
wd = webdriver.Chrome(options=options)
import time
URL = "https://www.quora.com/What-was-your-prom-like"
#URL = "https://www.quora.com/profile/%E0%AE%AA%E0%AE%BE%E0%AE%B2%E0%AE%BE%E0%AE%9C%E0%AE%BF-%E0%AE%B5%E0%AE%BF%E0%AE%B8%E0%AF%8D%E0%AE%B5%E0%AE%A8%E0%AE%BE%E0%AE%A4%E0%AE%A9%E0%AF%8D-Balaji-Viswanathan"
wd.get(URL)
PAUSE_TIME = 2
lh = wd.execute_script("return document.body.scrollHeight")
while True:
wd.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(PAUSE_TIME)
nh = wd.execute_script("return document.body.scrollHeight")
if nh == lh:
break
lh = nh
spans = wd.find_elements_by_css_selector('span.q-box.qu-userSelect--text')
for span in spans:
print(span.text)
print('-' * 70 + '\n')
输出如下所示：
What was your prom like? ---------------------------------------------------------------------- Prom. That experience of mythical proportions, whipped up into a foam and placed on a pedestal. “Oh, you simply must go! You’ll regret it when you’re older…” I didn’t want to go. I was adamant! And yet, on prom night, I found myself stuffed into a suit (for the last time, I promised myself) and sitting at a table. It started okay; I found some friends and we talked and walked around. One of my friends had a good laugh when I couldn’t find my table. The food wasn’t amazing, best described as downright bland, but I always expect food at parties to be lacklustre at best. Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- Are you expected to have sex with your prom date after the prom? ---------------------------------------------------------------------- Did you have sex after prom? ---------------------------------------------------------------------- My daughter has been crying because no one asked her to the prom. What should I do? ---------------------------------------------------------------------- What is prom like? ---------------------------------------------------------------------- What did you wear to your prom? ---------------------------------------------------------------------- My Prom was tonight Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- I ended up with three dates by mistake… whoops Names have been omitted to protect the innocent Senior year, I asked out a very dear friend of mine to prom. She told me maybe which later turned into a no. She decided she was going with her friend from her band class. Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- My date's existence offended the sensibilities of a drunk guy. My date, who I'll call Nancy, had graduated from high school. We had been dating for months, so I invited her to my senior prom. Nancy was reluctant to go. She felt like she would be conspicuous and out of place. I convinced her, though. It was important to me. So we went. We'd been there maybe ten minutes when a drunk student yelled at Nancy. “Are you a dyke?” His speech was loud, belligerent, and slurred. Then, he turned and left. Thinking that I should talk to him, I called for him to wait. Wiser than me, Nancy pulled me back. “We sh Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- What do people usually do after prom? ---------------------------------------------------------------------- What do teenagers usually do after prom? ---------------------------------------------------------------------- Does college have a prom? ---------------------------------------------------------------------- What age should a teen be able to come home from prom? ---------------------------------------------------------------------- What is a high school prom like? ---------------------------------------------------------------------- My best friend decided that we were all going to the prom. "We" being the collection of druggies, artists and outcasts that comprised the ragged clique that clung together likes ants crossing a river during the hell that was high school. Grace was a bit different. She was a proud, stoned outcast and valedictorian. She vexed authority and we loved her for it. If she said prom, we said sure. We asked why. she said, " Its ironic don't you see? We'll take acid and the very act of tripping will stand as protest to this ridiculous rite of passage that is America's paltry offering to its disenfranchise Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- I hated my prom. I had gone because my parents guilted me into it. “I always wanted to go but was too poor. I wish I had the chance.” -Mom So I went. I don't even like dresses. I prefer my usual shorts/jeans and a tank top. I don't wear much make up. My sister slathered me in it. I don't have a dancing bone in my body. I felt uncomfortable being dragged to dance. Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- The problem with the British version of a prom is, it's shit! Granted I can only talk for my own school and my own prom, but I'm almost certain it's the same right across the U.K. There's no real build up, hardly any money is spent, there's no committee organising it, there's no pomp or pizazz, it's no right of passage as it is in the US and A. Those that want to lose their virginity already have. Nobody is saving themselves for prom night. You're either one of the "cool people" that have already "done it" or you're one of the other kinds of people that haven't "done it", and feel like you'll ne Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- Prom Sonnet My parents said I had to go to prom. A venture I knew would end badly. I said alright but I had my qualms. But I still bought a suit that was classy. I paid a couple hundred for both tickets. When the day came I went to go get her. Her friend was there trying to solicit. Just like that I felt like a chauffeur. We arrive at prom and she didn’t want to dance. I had no problem so we stuck to talking. I went to the bathroom and she saw her chance. I returned to see her and the quarterback grinding. I said nothing, but there was anger in my eyes. I was a fool not to see through her guise. ---------------------------------------------------------------------- Uh. We had “most likely” awards. You know which one. It was at Ritz-Carlton so I was confused why the food was just okay (actually I think the desserts were pretty good.) Some people slow danced. We had an after-party rave. It was alright, not too bad. At least I got to hang out with my friends. ---------------------------------------------------------------------- Nothing dramatic, unless you count me eating my dinner in a rush and almost throwing up in the bathroom. Oh, and the guy that I like was eyeing me from across the room. But past that, it was a very fun night for my best friends and me. We danced our butts off, to say the least. Everyone was dancing (some more inappropriate than others), the teachers joined in, and no one was fighting. Before prom, my friends and I took pictures to commemorate the occasion. There were a few wild after parties, but I didn’t go. Instead, I went to Whataburger with my best friend before heading home. Just a regular ni Continue Reading ---------------------------------------------------------------------- ---------------------------------------------------------------------- I don’t know; I wasn’t there. Yes, I made it through the entire four years of high school without attending a single adolescent mating ritual (also known as a school dance), and I didn’t come away as a social pariah (at least not for that). The thought of hanging out with a bunch of my peers who were overly-concerned about getting drunk and/or getting laid didn’t appeal to me in the least. I was too busy geeking out about Gene Puerling ’s use of vocal harmony, or the latest Manhattan Transfer project. These interests still intrigue me 35 years later, but I couldn’t tell you who went with whom to Continue Reading ---------------------------------------------------------------------- ----------------------------------------------------------------------
-
第一个问题是 Quora Question 的代码是有效的 网页但不是第二个链接（我评论过）， https://www.quora.com/profile/%E0%AE%AA%E0%AE%BE%E0%AE%B2%E0%AE%BE%E0%AE%9C%E0%AE%BF-%E0%AE%B5%E0%AE%BF%E0%AE%B8%E0%AF%8D%E0%AE%B5%E0%AE%A8%E0%AE%BE%E0%AE%A4%E0%AE%A9%E0%AF%8D-Balaji-Viswanathan 。 它是最受欢迎的 Quoran 的 Quora 个人资料的链接。这 代码只是继续无休止地运行，并没有前进 过去
time.sleep(PAUSE_TIME)声明。我想修改 代码，以便它可以从以下位置读取问题及其下方的答案 个人资料以及问题页面上的答案。 -
第二个问题是输出结果只是截断 答案。如何获取文本以扩展每个答案 给定的链接，https://www.quora.com/What-was-your-prom-like，之后 点击
Continue reading，或点击More上的个人资料链接，例如 这些， https://www.quora.com/profile/Sean-Kernan. -
我想列出这两个网页上的所有链接/超链接。我应该写什么来获取/打印两个网页中的所有站点内链接，以
https://www.quora.com/开头。我不知道是否有办法使用硒来做到这一点。或美丽的汤。 通过soup.findAll("a")获取href中的所有链接不起作用。 -
最后一个问题只是我的好奇心，有没有办法在 通过抓取来收获网站，我让自己登录到我的 Quora帐户通过代码然后继续抓取？如果是这样， 怎么实现？
【问题讨论】：
标签： python selenium web-scraping beautifulsoup