【问题标题】:how to attach delimiter to the scraped table?如何将分隔符附加到刮掉的桌子上?
【发布时间】:2019-05-24 05:32:57
【问题描述】:

对编码非常陌生,如果我很傻但非常渴望学习,我深表歉意。 我正在使用 selenium 通过提取表格行来抓取网页(https://www.myinsuranceclub.com/health-insurance/star-health-network-hospital-list),如下面的代码所示。有人可以指导我如何为每个元素添加分隔符吗? 动机是将表格保存为 CSV 格式,分隔:name;;address;;city;;state;;pincode

# find_elements_by_xpath returns an array of selenium objects.
total = 302
j=1
while(j <= total):
    i = 1
    while(i <= 25):
        titles_element = driver.find_elements_by_xpath("//*[@id='hospital_all']/tbody/tr[{}]".format(i))
        i += 1
    # use list comprehension to get the actual repo titles and not the selenium objects.
        titles = [x.text for x in titles_element]
    # print out all the titles.
        print('titles:')
        print(titles, '\n')
        sys.stdout = open("data.txt", "a")
    driver.find_element_by_xpath('//*[@id="hospital_all_next"]').click()
    time.sleep(2)
    j+=1

当前输出: 标题: ['Srinivasa Nursing Home Beside Natraj Talkies Adilabad Andhra Pradesh 504001'] 标题: ['Padmasri Hospitals # 49-48 - 16/5 Nggos Colony Near Roc Petrol Bunk Akkayyapalem Andhra Pradesh 530016']

预期输出: 标题: ['斯里尼瓦萨疗养院;;在 Natraj Talkies 旁边;;阿迪拉巴德;;安德拉邦;; 504001'] 标题: ['Padmasri 医院;; # 49-48 - 16/5 Nggos 殖民地靠近洛克汽油铺位;;阿卡亚帕莱姆;;安德拉邦;; 530016']

【问题讨论】:

    标签: python-3.x firefox selenium-webdriver web-scraping


    【解决方案1】:

    使用连接,但是当您使用列表推导时,它不是一个在元素之间带有“,”的列表吗?

     titles = ';;'.join([x.text for x in titles_element])
    

    蟒蛇:

    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium import webdriver
    from bs4 import BeautifulSoup as bs
    
    driver = webdriver.Chrome() 
    driver.get('https://www.myinsuranceclub.com/health-insurance/star-health-network-hospital-list')
    
    rows =  WebDriverWait(driver,5).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#hospital_all tr.odd, #hospital_all tr.even")))
    
    for row in rows:
        soup = bs(row.get_attribute('outerHTML'), 'lxml')
        titles = ';;'.join([x.text for x in soup.select('td')])
        pos = titles[:-2].rfind(";;")
        titles = titles[:-2][:pos] + ' ' + titles[:-2][pos+2:]
        print(titles)
    

    【讨论】:

    • 输出没有带分隔符。使用这个 ""titles = ';;'.join([x.text for x intitles_element])"" 也不会添加分隔符。运行上述代码后,新输出为标题:Srinivasa Nursing Home Beside Natraj Talkies Adilabad Andhra Pradesh 504001 标题:Padmasri Hospitals # 49-48 - 16/5 Nggos Colony Near Roc Petrol Bunk Akkayyapalem Andhra Pradesh 530016 预期输出:标题:Srinivasa Nursing ;; Natraj Talkies 旁边的家;;阿迪拉巴德;;安得拉邦 504001
    • 你可以使用replace来去掉last ;;如果您不希望最后两个元素使用它:Andhra Pradesh;;504001
    猜你喜欢
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2015-02-25
    • 1970-01-01
    • 2018-05-22
    • 1970-01-01
    • 2023-03-22
    • 1970-01-01
    相关资源
    最近更新 更多