检索保存到 MYSQL 中 blob 的数据时的双重编码

Question

我有一个包含数据的嵌套 python 列表，例如显示的长列表。 我有将这个列表变成类似字符串的 json 的代码，然后我对其进行编码以便将其提供给 base64.b64encoder，然后将其作为 blob 与其他一些数据一起保存到我的 mysql DB 中。当我尝试检索数据并将其转换回列表时，我遇到了我只能描述为双重编码的问题。

longlist = [
    [1, 2,'thisisdhdh'],
    [1,2,'thlsldfsdf'],
    [2,0,'sdlfjldksjflksdj']
] 

string1 = json.dumps(longlist)

encod1 = string1.encode('ascii')

encodedlist = base64.b64encode(encod1)

cur.execute('INSERT INTO conversations (convoblob) VALUES ("%s")' % (encodedlist))

code above to save the encoded list to the DB

code below to retrieve the data

cur.execute('SELECT convoblob FROM conversations WHERE id IN (327) ')
item = cur.fetchone()

x = items[0] 

1.输出类型类字节
b"b'W1sxLCAyLCAidGhpc2lzZGhkaCJdLCBbMSwgMiwgInRobHNsZGZzZGYiXSwgWzIsIDAsICJzZGxmamxka3N1.qZmxrc2RqIl1d'"

       x = items[0].decode('ascii')

1.输出类型类'str' 1.b'W1sxLCAyLCAidGhpc2lzZGhkaCJdLCBbMSwgMiwgInRobHNsZGZzZGYiXSwgWzIsIDAsICJzZGxmamxka3NqZ1.mxrc2RqIl1d'

正如你所看到的，一旦解码它就会变成一个我无法运行解码器的字符串。

不将数据保存到数据库会很好地解码，由于某种原因，一旦数据进入数据库，它会以 (b") 开头的 sting 和 (") 结尾

我的数据库默认字符集是 utf8mb4

我确实找到了一种解决方法，即解码从数据库中提取的数据，并将字符串末尾的位截断，将其标识为类似字符串的字节，然后将字符串编码回来，然后对其进行解码使用 base64.b64decode

cur.execute('SELECT convoblob FROM conversations WHERE id IN (327) ')
item = cur.fetchone()
decoded = item[0].decode('ascii')
y = len(decoded) - 1
newString = decoded[2:y]
this = newString.encode('ascii')
man = base64.b64decode(this).decode('ascii')
ohyeah = json.loads(man)
print(ohyeah[1])

输出为 b'W1sxLCAyLCAidGhpc2lzZGhkaCJdLCBbMSwgMiwgInRobHNsZGZzZGYiXSwgWzIsIDAsICJzZGxmamxka3NqZmxrc2RqIl1d'

[1, 2, 'thlsldfsdf']

Answer 1

这是怎么回事：

import sqlite3
import json
import base64

longlist = [
    [1, 2,'thisisdhdh'],
    [1,2,'thlsldfsdf'],
    [2,0,'sdlfjldksjflksdj']
] 

# you create a string
# '[[1, 2, "thisisdhdh"], [1, 2, "thlsldfsdf"], [2, 0, "sdlfjldksjflksdj"]]'
string1 = json.dumps(longlist)

# you create a bytes object from the string, happens to be encoded as ascii
# b'[[1, 2, "thisisdhdh"], [1, 2, "thlsldfsdf"], [2, 0, "sdlfjldksjflksdj"]]'
encod1 = string1.encode('ascii')

# you create a base-64 encoding of that object, still bytes
# b'W1sxLCAyLCAidGhpc2lzZGhkaCJdLCBbMSwgMiwgInRobHNsZGZzZGYiXSwgWzIsIDAsICJzZGxmamxka3NqZmxrc2RqIl1d'
encodedlist = base64.b64encode(encod1)

conn = sqlite3.connect('test.db')
conn.execute('DROP TABLE IF EXISTS conversations')
conn.execute('CREATE TABLE conversations (convoblob BLOB)')
# you format that bytes object into a string (casting it to a string) <<< your problem
conn.execute('INSERT INTO conversations VALUES ("%s")' % (encodedlist))

cur = conn.execute('SELECT convoblob FROM conversations')
item = cur.fetchone()

# no surprise about the mangled string here, since it was cast back into a string first
# "b'W1sxLCAyLCAidGhpc2lzZGhkaCJdLCBbMSwgMiwgInRobHNsZGZzZGYiXSwgWzIsIDAsICJzZGxmamxka3NqZmxrc2RqIl1d'"
print(item[0])

# Let's try again
conn.close()
conn = sqlite3.connect('test.db')
conn.execute('DROP TABLE IF EXISTS conversations')
conn.execute('CREATE TABLE conversations (convoblob BLOB)')
# now passing the bytes directly to the database, as bytes, into a blob
conn.execute('INSERT INTO conversations VALUES (?)', [encodedlist])

cur = conn.execute('SELECT convoblob FROM conversations')
item = cur.fetchone()

# This works:
my_list = json.loads(base64.b64decode(item[0]))
print(my_list)

我知道你使用 MySQL，但它对 SQLite 的工作方式相同（并且有同样的问题），所以我确定这是你的问题，sqlite3 适用于任何标准的 Python 安装。

因此，简而言之：通过使用 %-formatting 将 bytes 对象格式化为字符串，您创建的 "b'data'" 对象会给您带来麻烦。这可以通过将bytes 本身传递给 SQL 引擎并使用查询参数将其传递给查询来避免，无论如何这是避免 SQL 注入的更好方法。