如何将带有列表的列转换为熊猫中的不同行

Question

我正在抓取一个网站：- https://spfpharmacy.com/

我已经使用下面的代码使用 selenium 成功地抓取了它。

    test_list = []
    test_list = list(string.ascii_uppercase)
    
        med_url = []
        for i in tqdm(test_list):
        driver.get(f'https://spfpharmacy.com/search/?drugName={i}')
        for i in driver.find_elements(By.XPATH,"//a[@class='rxrequired default']"):
            med_url.append(i.get_attribute("href"))
    
    data = []
    for i in tqdm(med_url):
        driver.get(i)
        time.sleep(1)
        
        try:
            med_name = []
            for i in driver.find_elements(By.XPATH,"//div[@id='brand_dose']//div[@class='product-name']"):
                med_name.append(i.text)
        except:
            med_name.append(None)
            
        try:
            manuf_name = []
            for i in driver.find_elements(By.XPATH,"//div[@id='brand_dose']//div//span[@class='manufactured-name']"):
                manuf_name.append(i.text)
        except:
            manuf_name.append(i.text)
            
        try:
            country = []
            for i in driver.find_elements(By.XPATH,"//div[@id='brand_dose']//div//span[@class='product-country']"):
                country.append(i.text)
        except:
            country.append(None)
            
        try:
            pres_req = []
            for i in driver.find_elements(By.XPATH,"//div[@id='brand_dose']//div//span[@class='product-prescription']"):
                pres_req.append(i.text)
        except:
            pres_req.append(None)
        
        str_price =  []
        try:
            for i in driver.find_elements(By.XPATH,"//div[@id='brand_dose']//div//span[@class='product-dose-text']"):
                for j in driver.find_elements(By.XPATH,f"//div[@id='brand_dose']//div//select//option[@data-str='{i.text}']"):
                    str_price.append({i.text, j.text})
        except:
            str_price.append(None)
            
        data.append({
            'Medicine_name':med_name,
            'Manufacture_name':manuf_name,
            'Product_Counry':country,
            'Prescription_Required':pres_req,
            'Product_Details':str_price})

其中 test_list 是一个大写字母列表，它完成了 URL，例如：-

https://spfpharmacy.com/search/?drugName=A 给出了所有带有 A 的药物的详细信息。

抓取数据后，我得到如下所示的结果：-

但我想在一行中获取每种药物的名称，并在不同的列中获取与该药物相关的所有详细信息。

像这样的东西。

我尝试使用 explode 和 transform，还通过 Internet 和 stack overflow 进行了搜索，但无法将其转换为预期的格式。

另外，有没有其他方法可以从这个网站抓取数据，比如直接调用 API 并以正确的格式获取请求的数据？

这是代码：-

df.head(5).to_dict(orient="list")

输出：-

{'Medicine_Name': ['Abacavir',
  'Abacavir - Lamivudine',
  'Abilify (Aripiprazole)',
  'Abilify Maintena Injection',
  'Abiraterone'],
 'Bran_Name_Choices': [['Ziagen 300mg'],
  ['Kivexa 600mg/300mg'],
  ['Abilify 2mg',
   'Abilify 5mg',
   'Abilify 10mg',
   'Abilify 15mg',
   'Abilify 20mg',
   'Abilify 30mg',
   'Abilify Maintena Injection 300mg',
   'Abilify Maintena Injection 400mg'],
  ['Abilify Maintena Injection 300mg', 'Abilify Maintena Injection 400mg'],
  ['Zytiga 250mg', 'Zytiga 500mg']],
 'Generic_Name_Choices': [['Abacavir 300mg'],
  ['Abacavir - Lamivudine 600mg/300mg'],
  ['Aripiprazole 2mg',
   'Aripiprazole 5mg',
   'Aripiprazole 10mg',
   'Aripiprazole 15mg',
   'Aripiprazole 20mg',
   'Aripiprazole 30mg'],
  [],
  ['Abiraterone 250mg', 'Abiraterone 500mg']],
 'Manufacture_name': [['Manufactured by GlaxoSmithKline Inc.. '],
  ['Manufactured by VIIV Healthcare ULC. '],
  ['Manufactured by Otsuka Pharmaceuticals. ',
   'Manufactured by Bristol-Myers Squibb. ',
   'Manufactured by Otsuka Pharmaceuticals. ',
   'Manufactured by Otsuka Pharmaceuticals. ',
   'Manufactured by Bristol-Myers Squibb. ',
   'Manufactured by Otsuka Pharmaceuticals. ',
   'Manufactured by Otsuka Pharmaceuticals. ',
   'Manufactured by Otsuka Pharmaceuticals. '],
  ['Manufactured by Otsuka Pharmaceuticals. ',
   'Manufactured by Otsuka Pharmaceuticals. '],
  ['Manufactured by Janssen-Ortho. ', 'Manufactured by Janssen-Ortho. ']],
 'Manufacture_name_Generic': [['Manufactured by Apotex Inc. '],
  ['Manufactured by Mylan. '],
  ['Manufactured by Apotex Inc. ',
   'Manufactured by Apotex Inc. ',
   'Manufactured by Apotex Inc. ',
   'Manufactured by Apotex Inc. ',
   'Manufactured by Apotex Inc. ',
   'Manufactured by Pharmascience Inc.. '],
  [],
  ['Manufactured by Pharmascience Inc.. ',
   'Manufactured by Pharmascience Inc.. ']],
 'Product_Counry': [[' Product Of Canada'],
  [' Product Of Canada'],
  [' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada'],
  [' Product Of Canada', ' Product Of Canada'],
  [' Product Of Canada', ' Product Of Canada']],
 'Product_Country_Generic': [[' Product Of Canada'],
  [' Product Of Canada'],
  [' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada',
   ' Product Of Canada'],
  [],
  [' Product Of Canada', ' Product Of Canada']],
 'Prescription_Required': [['Prescription Required'],
  ['Prescription Required'],
  ['Prescription Required',
   'Prescription Required',
   'Prescription Required',
   'Prescription Required',
   'Prescription Required',
   'Prescription Required',
   'Prescription Required',
   'Prescription Required'],
  ['Prescription Required', 'Prescription Required'],
  ['Prescription Required', 'Prescription Required']],
 'Prescription_Required_Generic': [['Prescription Required.  '],
  ['Prescription Required.  '],
  ['Prescription Required.  ',
   'Prescription Required.  ',
   'Prescription Required.  ',
   'Prescription Required.  ',
   'Prescription Required.  ',
   'Prescription Required.  '],
  [],
  ['Prescription Required.  ', 'Prescription Required.  ']],
 'Product_Details': [[{'300mg', '60 tabs - $630.00 USD ($10.50 per tabs)'},
   {'120 tabs - $1250.00 USD ($10.42 per tabs)', '300mg'},
   {'180 tabs - $1860.00 USD ($10.33 per tabs)', '300mg'}],
  [{'30 tabs - $810.00 USD ($27.00 per tabs)', '600mg/300mg'},
   {'60 tabs - $1610.00 USD ($26.83 per tabs)', '600mg/300mg'},
   {'600mg/300mg', '90 tabs - $2388.00 USD ($26.53 per tabs)'}],
  [{'2mg', '30 tabs - $114.00 USD ($3.80 per tabs)'},
   {'2mg', '60 tabs - $220.00 USD ($3.67 per tabs)'},
   {'2mg', '90 tabs - $330.00 USD ($3.67 per tabs)'},
   {'30 tabs - $131.00 USD ($4.37 per tabs)', '5mg'},
   {'5mg', '60 tabs - $254.00 USD ($4.23 per tabs)'},
   {'5mg', '90 tabs - $369.00 USD ($4.10 per tabs)'},
   {'10mg', '30 tabs - $144.00 USD ($4.80 per tabs)'},
   {'10mg', '60 tabs - $280.00 USD ($4.67 per tabs)'},
   {'10mg', '90 tabs - $408.00 USD ($4.53 per tabs)'},
   {'15mg', '30 tabs - $150.00 USD ($5.00 per tabs)'},
   {'15mg', '60 tabs - $294.00 USD ($4.90 per tabs)'},
   {'15mg', '90 tabs - $429.00 USD ($4.77 per tabs)'},
   {'20mg', '30 tabs - $144.00 USD ($4.80 per tabs)'},
   {'20mg', '60 tabs - $284.00 USD ($4.73 per tabs)'},
   {'20mg', '90 tabs - $417.00 USD ($4.63 per tabs)'},
   {'30 tabs - $144.00 USD ($4.80 per tabs)', '30mg'},
   {'30mg', '60 tabs - $284.00 USD ($4.73 per tabs)'},
   {'30mg', '90 tabs - $420.00 USD ($4.67 per tabs)'},
   {'1 injection - $589.96 USD ($589.96 per injection)', '300mg'},
   {'2 injection - $1169.90 USD ($584.95 per injection)', '300mg'},
   {'3 injection - $1729.86 USD ($576.62 per injection)', '300mg'},
   {'1 injection - $559.36 USD ($559.36 per injection)', '400mg'},
   {'2 injection - $1108.70 USD ($554.35 per injection)', '400mg'},
   {'3 injection - $1638.06 USD ($546.02 per injection)', '400mg'}],
  [{'1 injection - $589.96 USD ($589.96 per injection)', '300mg'},
   {'2 injection - $1169.90 USD ($584.95 per injection)', '300mg'},
   {'3 injection - $1729.86 USD ($576.62 per injection)', '300mg'},
   {'1 injection - $559.36 USD ($559.36 per injection)', '400mg'},
   {'2 injection - $1108.70 USD ($554.35 per injection)', '400mg'},
   {'3 injection - $1638.06 USD ($546.02 per injection)', '400mg'}],
  [{'120 tabs - $3789.00 USD ($31.58 per tabs)', '250mg'},
   {'500mg', '60 tabs - $3999.99 USD ($66.67 per tabs)'},
   {'120 tabs - $7899.99 USD ($65.83 per tabs)', '500mg'}]],
 'Product_Details_Generic': [[{'300mg',
    '60 tabs - $274.00 USD ($4.57 per tabs)'},
   {'120 tabs - $538.00 USD ($4.48 per tabs)', '300mg'},
   {'180 tabs - $789.00 USD ($4.38 per tabs)', '300mg'}],
  [{'30 tabs - $200.00 USD ($6.67 per tabs)', '600mg/300mg'},
   {'60 tabs - $390.00 USD ($6.50 per tabs)', '600mg/300mg'},
   {'600mg/300mg', '90 tabs - $570.00 USD ($6.33 per tabs)'}],
  [{'2mg', '30 tabs - $35.00 USD ($1.17 per tabs)'},
   {'2mg', '60 tabs - $60.00 USD ($1.00 per tabs)'},
   {'2mg', '90 tabs - $78.00 USD ($0.87 per tabs)'},
   {'100 tabs - $80.00 USD ($0.80 per tabs)', '5mg'},
   {'100 tabs - $90.00 USD ($0.90 per tabs)', '10mg'},
   {'100 tabs - $99.00 USD ($0.99 per tabs)', '15mg'},
   {'100 tabs - $99.00 USD ($0.99 per tabs)', '20mg'},
   {'30 tabs - $45.00 USD ($1.50 per tabs)', '30mg'},
   {'30mg', '60 tabs - $80.00 USD ($1.33 per tabs)'},
   {'30mg', '90 tabs - $90.00 USD ($1.00 per tabs)'}],
  [],
  [{'120 tabs - $1690.00 USD ($14.08 per tabs)', '250mg'},
   {'240 tabs - $3370.00 USD ($14.04 per tabs)', '250mg'},
   {'250mg', '360 tabs - $4989.00 USD ($13.86 per tabs)'},
   {'500mg', '60 tabs - $1190.00 USD ($19.83 per tabs)'},
   {'120 tabs - $2290.00 USD ($19.08 per tabs)', '500mg'},
   {'180 tabs - $3390.00 USD ($18.83 per tabs)', '500mg'}]]}

Answer 1

这是一种方法，但是小心, 它会将您的数据框（5 行，11 列）重塑为一个非常大的数据框（数百万行），这可能会使您的计算机无响应：

# Deal with the last two columns (which contain lists of sets)
for col in ["Product_Details", "Product_Details_Generic"]:
    df[col] = df[col].apply(lambda x: [subitem for item in x for subitem in item])

# Explode values of each column (except first one) into new rows
for col in df.columns[1:]:
    df = df.explode(col)

df = df.reset_index(drop=True)