【发布时间】：2019-12-10 16:07:22
【问题描述】：
The way I am able to fetch all the details, can I even be able to fetch the sub datasets name which are present in the providers dataset? I wish to have providers name followed by dataset name and further followed by datasets present in itThe below image link gives the hierarchy of one provider, similarly we have many providers, can I fetch all the details using API for all the providers?This is the output .csv which I am able to fetch the details after running the code.The below image gives the details about one provider named "AMECO" in which I am unable to fetch the Datasets title and child datasets, in turn able to fetch sub-child items.下面的代码工作效率很高，但for循环的少数部分没有获取正确的 信息。有人可以帮帮我吗。我想获取所有存在于It is the link which provides the information about the Proviers提供者列表中的数据集。反过来;主要数据集标题我无法进入 csv 文件。
import requests, os, sys, csv
import pandas as pd list
from bs4 import BeautifulSoup
pd.set_option('display.max_colwidth', 10000)
pd.set_option('display.expand_frame_repr', False)
以下是提供有关提供商信息的链接。
URL_1 = 'https://db.nomics.world/providers'
URL_2 = 'https://db.nomics.world/'
Required_Columns = ['Provider', 'Region', 'Website', 'In DBnomics since', 'Updated by DBnomics on', 'Dataset Name', 'Link']
print('\n> Retrieve DBNomics Providers')
DBN_Providers = []
Response = requests.get(URL_1)
Soup = BeautifulSoup(Response.content, 'html.parser')
for Class_L1 in Soup.findAll('div', {'class': 'my-8 sm:flex sm:flex-wrap sm:-mx-4'}):
for Class_L2 in Class_L1.findAll('h2', {'class' : 'mb-2 font-bold text-grey-darkest text-xl group-hover:underline'}):
Provide_Code = Class_L2.text
Provide_Code = Provide_Code.strip()
DBN_Providers.append(Provide_Code)
print('\n>> Total Providers:', len(DBN_Providers))
Details_All = pd.DataFrame()
for I, Provider in enumerate(DBN_Providers, start=1):
URL = URL_2 + Provide
print('%s > %s : %s'%(I, Provider, URL))
Meta_Data = {}
Meta_Data['Provider'] = Provider
Response = requests.get(URL)
Soup = BeautifulSoup(Response.content, 'html.parser')
# Meta Data
for Class_L1 in Soup.findAll('dl', {'class': 'dl-properties'}):
TList = []
for Item in Class_L1.findAll('dd'):
Item = Item.text
Item = Item.strip()
TList.append(Item)
Meta_Data['Region'] = TList[0]
Meta_Data['Website'] = TList[1]
Meta_Data['In DBnomics since'] = TList[2]
Meta_Data['Updated by DBnomics on'] = TList[3]
Meta_Data = pd.DataFrame([Meta_Data])
# Dataset Details
Dataset_List = []
for Class_L1 in Soup.findAll('div', {'id': 'category-tree'}):
for Class_L2 in Class_L1.findAll('ul', {'class': 'list-reset mb-6'}):
for Class_L3 in Class_L2.findAll('details', {'class': 'my-4'}):
for Class_L4 in Class_L3.findAll('a', {'class': 'mr-1'}):
TList = []
Dataset_Name = Class_L4.text
Dataset_Name = Dataset_Name.strip()
TList.append(Dataset_Name)
HREF = Class_L4.get('href')
TList.append(URL_2 + HREF.strip())
Dataset_List.append(TList)
Dataset_DF = pd.DataFrame(Dataset_List, columns=['Dataset Name', 'Link'])
DF = pd.concat([Meta_Data, Dataset_DF], sort=False)
DF = DF.sort_values(['Provider'], ascending=[True])
DF = DF.fillna(method='ffill')
DF = DF.dropna(subset=['Dataset Name'])
Details_All = Details_All.append(DF, sort=False)
Details_All = Details_All.drop_duplicates()
Details_All = Details_All[Required_Columns]
Details_All.to_csv('Details_All.csv', index = False, encoding='utf-8-sig')
【问题讨论】：
-
使用
API让您的生活更轻松API LINK
标签： python pandas beautifulsoup