抓取数据并推送到弹性搜索时出现序列化错误答案

【问题标题】：SerializationError while scrapng the data and push to elastic search抓取数据并推送到弹性搜索时出现序列化错误
【发布时间】：2020-10-24 23:36:13
【问题描述】：

下面是代码

我正在尝试抓取数据并尝试推送到弹性搜索

import re
import time
import requests
from bs4 import BeautifulSoup
from elasticsearch import Elasticsearch

es_client = Elasticsearch(['http://localhost:9200'])

#drop_index = es_client.indices.create(index='blog-sysadmins', ignore=400)
create_index = es_client.indices.delete(index='blog-sysadmins', ignore=[400, 404])

def urlparser(title, url):
    # scrape title
    p = {}
    post = title
    page = requests.get(post).content
    soup = BeautifulSoup(page, 'lxml')
    title_name = soup.title.string

    # scrape tags
    tag_names = []
    desc = soup.findAll(attrs={"property":"article:tag"})
    for x in range(len(desc)):
        tag_names.append(desc[x-1]['content'].encode('utf-8'))
    print (tag_names)

    # payload for elasticsearch
    doc = {
        'date': time.strftime("%Y-%m-%d"),
        'title': title_name,
        'tags': tag_names,
        'url': url
    }

    # ingest payload into elasticsearch
    res = es_client.index(index="blog-sysadmins", doc_type="docs", body=doc)
    time.sleep(0.5)

sitemap_feed = 'https://sysadmins.co.za/sitemap-posts.xml'
page = requests.get(sitemap_feed)
sitemap_index = BeautifulSoup(page.content, 'html.parser')
urlss = [element.text for element in sitemap_index.findAll('loc')]
urls = urlss[0:2]
print ('urls',urls)
for x in urls:
    urlparser(x, x)

我的错误：

SerializationError: ({'date': '2020-07-04', 'title': 'Persistent Storage with OpenEBS on Kubernetes', 'tags': [b'Cassandra', b'Kubernetes', b'Civo', b'Storage'], 'url': 'http://sysadmins.co.za/persistent-storage-with-openebs-on-kubernetes/'}, TypeError("Unable to serialize b'Cassandra' (type: <class 'bytes'>)",))

【问题讨论】：

标签： python json elasticsearch beautifulsoup

【解决方案1】：

json serialization error 在您尝试指示不是 javascript 的原始数据类型（开发 json 的语言）的数据时出现。这是一个 json 错误，而不是弹性错误。 json 格式的唯一规则是它在自身内部只接受这些数据类型——更多解释请阅读here。在您的情况下，标签字段具有 bytes 数据类型，如您的错误堆栈中所写：

TypeError("Unable to serialize b'Cassandra' (type: <class 'bytes'>)

要解决您的问题，您只需将标签内容转换为字符串即可。所以只要改变这一行：

tag_names.append(desc[x-1]['content'].encode('utf-8'))

到：

tag_names.append(str(desc[x-1]['content']))

【讨论】：