在 MySQL 中将 JSON 值设置为 TEXT 变量错误答案

【问题标题】：Setting JSON value to a TEXT variable error in MySQL在 MySQL 中将 JSON 值设置为 TEXT 变量错误
【发布时间】：2020-03-12 14:43:54
【问题描述】：

更新：这是 JSON 代码的 Pastebin。如果有人能告诉我为什么 MySQL 拒绝将其作为 TEXT 变量读取以及如何解决它，我将不胜感激：https://pastebin.com/ju8xPPsV

所以我这里有一个奇怪的。在 Windows 上运行 8.0.18。我的环境都是utf8mb4和utf8mb4_unicode_ci。

问题是 MySQL 拒绝将附加的 JSON 文本读入 TEXT 字段，无论定义了什么 CHARACTER SET，尝试将 TEXT 变量设置为代码时总是出现以下错误：

错误 1366 (HY000)：不正确的字符串值：第 1 行的列 'json_text' 的'\x96 2019...'

显然 MySQL 不喜欢文本中有一个字符，但是我应该在什么地方、在哪里以及如何“清理”这个文本以使其被接受？

这发生在尝试将 JSON 分配给 Text 值的行上：

set json_text= JSON_EXTRACT((injason), CONCAT('$.', 'in_JSON'));

这是完整的代码：

DROP PROCEDURE IF EXISTS `sp_jason`;
DELIMITER //
CREATE DEFINER=`root`@`localhost` 
PROCEDURE `sp_jason`(
    IN parameterName VARCHAR(10)
)
COMMENT 'schema_version_applied_to: v1.0.0'
master_block:BEGIN
 DECLARE json_text text CHARACTER  SET latin1;
 DECLARE injason JSON;

DECLARE exit handler for SQLEXCEPTION, SQLWARNING, NOT FOUND
 BEGIN
  GET DIAGNOSTICS CONDITION 1 @sqlstate = RETURNED_SQLSTATE, 
   @errno = MYSQL_ERRNO, @text = MESSAGE_TEXT;
  SET @full_error = CONCAT("ERROR ", @errno, " (", @sqlstate, "): ", @text);
  SELECT @full_error;
 END;


 SET @hugeJason='
{
    "in_JSON": {
        "data_value": {
            "name": "Wesley Snell",
            "about": "Over 15 years of R&D development on over 30 products in various categories (automotive, medical, telecom, mobil devices, navigation, security, industrial automation, consumer products, CAD, hospitality solutions, kiosk, point of sale ...",
            "email": "wesley@qualcomm.com",
            "articles": [
                {
                    "linkURL": "https://www.linkedin.com/pulse/invitation-part-state-ai-ml-december-2019-january-2020-kiran-gunnam"
                }
            ],
            "nameLast": "Snell",
            "education": [
                {
                    "degreeName": "Master\'s degree",
                    "schoolName": "University of California, San Diego - Jacobs School of Engineering",
                    "description": "The next step in the wireless revolution is the connection of everyday devices through wireless technology. Embedded wireless technology is today exploding in nearly every market sector from personal electronics and medical devices, to the transportation infrastructure and manufacturing. ...",
                    "fieldOfStudy": "Wireless Embedded Systems",
                    "schoolImageURL": "https://media-exp1.licdn.com/dms/image/C4D0BAQHEc15VMuDaGg/company-logo_100_100/0?e=1589414400&v=beta&t=VA3iTzTB4ZBq-iYp_Fd4FYbCG-4YV7gDL47JCXaMPk4",
                    "activitiesAndSocieties": "Minor study in Digital Hardware Engineering and Business Management",
                    "datesAttendedOrGraduationString": "2017 – 2019"
                }
            ],
            "interests": [
                {
                    "title": "Ford Motor Company",
                    "imageURL": "https://media-exp1.licdn.com/dms/image/C4D0BAQGUxHv2MadZ9w/company-logo_100_100/0?e=1589414400&v=beta&t=t6v_lnhbQYiUjdGSiPBSQ89PpTMges1JsxsBQHtSfSY",
                    "occupation": null,
                    "followerCount": 2254781
                },
                {
                    "title": "Bill Gates",
                    "imageURL": "https://media-exp1.licdn.com/dms/image/C5603AQHv9IK9Ts0dFA/profile-displayphoto-shrink_100_100/0?e=1586390400&v=beta&t=oC0Mo8Qep7-PeP7TSjNEx94AbX4um89htO25XGhJZFE",
                    "occupation": "Co-chair, Bill & Melinda Gates Foundation",
                    "followerCount": 24625634
                }
            ],
            "nameFirst": "Wesley",
            "experience": [
                {
                    "roles": [
                        {
                            "jobType": "full-time",
                            "jobTitle": "Senior Staff Engineer - Machine Learning , Computer Vision & IOT (R&D Division)",
                            "location": null,
                            "currentRole": true,
                            "jobDescription": "Working on machine learning based computer vision based projects on embedded platforms. Heavy use of programming in C. Modern C++, Python and MATLAB. Heavy interaction in hardware, schematic review and inter process communications.",
                            "employmentDuration": {
                                "datesEmployedEnd": "Present",
                                "datesEmployedStart": "2019",
                                "datesEmployedString": "2019 – Present",
                                "employmentDurationYears": 1,
                                "employmentDurationString": "1 yr"
                            }
                        },
                        {
                            "jobType": "full-time",
                            "jobTitle": "Senior Staff Engineer - Android Team & Wireless Communications (Wireless R&D Division)",
                            "location": null,
                            "currentRole": true,
                            "jobDescription": "A year plus of Android software development for mobile applications with coding in Java and c++ with the NDK. ...",
                            "employmentDuration": {
                                "datesEmployedEnd": "Present",
                                "datesEmployedStart": "2019",
                                "datesEmployedString": "",
                                "employmentDurationYears": 1,
                                "employmentDurationString": "1 yr"
                            }
                        },
                        {
                            "jobType": null,
                            "jobTitle": "Senior Staff Engineer - R&D Division (Platform & Tools Lead)",
                            "location": "San Diego",
                            "currentRole": false,
                            "jobDescription": "Working on new Research and Developments projects, for state of the art products. ...",
                            "employmentDuration": {
                                "datesEmployedEnd": "Present",
                                "datesEmployedStart": "Apr 2016",
                                "datesEmployedString": "Apr 2016 – Present",
                                "employmentDurationYears": 3,
                                "employmentDurationMonths": 11,
                                "employmentDurationString": "3 yrs 11 mos"
                            }
                        }
                    ],
                    "currentJob": true,
                    "companyName": "Qualcomm",
                    "employmentDurationYears": 14,
                    "employmentDurationString": "14 yrs 7 mos"
                },
                {
                    "roles": [
                        {
                            "jobType": null,
                            "jobTitle": "President/Founder",
                            "location": "San Diego",
                            "currentRole": false,
                            "jobDescription": "Many projects in the following areas. - Internet Applications (JavaScript/ HTML/ SQL). ...",
                            "employmentDuration": {
                                "datesEmployedEnd": "Feb 2008",
                                "datesEmployedStart": "Jan 1998",
                                "datesEmployedString": "Jan 1998 – Feb 2008",
                                "employmentDurationYears": 10,
                                "employmentDurationString": "10 yrs 2 mos"
                            }
                        }
                    ],
                    "currentJob": false,
                    "companyName": "Creative Logic",
                    "employmentDurationYears": 10,
                    "employmentDurationString": "10 yrs 2 mos"
                },
                {
                    "roles": [
                        {
                            "jobType": null,
                            "jobTitle": "Principal R&D Engine",
                            "location": null,
                            "currentRole": false,
                            "jobDescription": null,
                            "employmentDuration": {
                                "datesEmployedEnd": "Dec 1999",
                                "datesEmployedStart": "Jan 1998",
                                "datesEmployedString": "Jan 1998 – Dec 1999 2 yrs",
                                "employmentDurationYears": 2,
                                "employmentDurationString": "2 yrs"
                            }
                        }
                    ],
                    "currentJob": false,
                    "companyName": "Infogation",
                    "employmentDurationYears": 10,
                    "employmentDurationString": "Jan 1998 – Dec 1999"
                }
            ],
            "nameMiddle": "",
            "profileURL": "https://www.linkedin.com/in/wesleysnell",
            "currentCompany": {
                "name": "123",
                "imageURL": "https://media-exp1.licdn.com/dms/image/C4E0BAQFIzy7XU945xg/company-logo_100_100/0?e=1589414400&v=beta&t=eIPPusNFhCPBHri11dcijhuyiSsYWJUehf_KUOFCvXQ"
            },
            "accomplishments": [
                {
                    "projects": [
                        {
                            "data": "Jun 2017",
                            "title": "Data Scientist Competition winner",
                            "issuer": "DataCamp",
                            "subTitle": "Jun 2017 DataCamp",
                            "description": "Data Scientist with Python Career Track on DataCamp"
                        },
                        {
                            "data": "Jun 2017",
                            "title": "Data Scientist Competition winner",
                            "issuer": "DataCamp",
                            "subTitle": "Jun 2017 DataCamp",
                            "description": "Data Scientist with Python Career Track on DataCamp"
                        }
                    ],
                    "honorAndAwards": [
                        {
                            "data": "Jun 2017",
                            "title": "Data Scientist Competition winner",
                            "issuer": "DataCamp",
                            "subTitle": "Jun 2017 DataCamp",
                            "description": "Data Scientist with Python Career Track on DataCamp"
                        },
                        {
                            "data": "Jun 2017",
                            "title": "Data Scientist Competition winner",
                            "issuer": "DataCamp",
                            "subTitle": "Jun 2017 DataCamp",
                            "description": "Data Scientist with Python Career Track on DataCamp"
                        }
                    ]
                }
            ],
            "currentJobTitle": "Senior Staff Engineer - Machine Learning , Computer Vision & IOT (R&D Division) at Qualcomm",
            "currentLocation": "San Diego, California",
            "profileImageURL": "https://media-exp1.licdn.com/dms/image/C4D03AQGoa2edXG6hWA/profile-displayphoto-shrink_200_200/0?e=1586390400&v=beta&t=f-Rv8pyCxIAuDRfK8g6AxkNHf-t3MGbZRfCvcOFghoY",
            "recommendations": {
                "given": [
                    {
                        "recommender": {
                            "name": "Steve Jobs",
                            "headline": "Founder Apple Inc.",
                            "relation": "October 22, 2010, Steve managed Wesley directly"
                        },
                        "recommendation": "Wes is one of the sharpest engineers I have worked with. His biggest assets are his technical versatility and engineering creativity ..."
                    },
                    {
                        "recommender": {
                            "name": "Bill Gates",
                            "headline": "Founder Windows Inc.",
                            "relation": "October 22, 2015, Bill was senior to Wesley but didn\'t manage"
                        },
                        "recommendation": "Wes is one of the sharpest engineers I have worked with. His biggest assets are his technical versatility and engineering creativity ..."
                    }
                ],
                "received": [
                    {
                        "recommender": {
                            "name": "Steve Jobs",
                            "headline": "Founder Apple Inc.",
                            "relation": "October 22, 2010, Steve managed Wesley directly"
                        },
                        "recommendation": "Wes is one of the sharpest engineers I have worked with. His biggest assets are his technical versatility and engineering creativity ..."
                    },
                    {
                        "recommender": {
                            "name": "Bill Gates",
                            "headline": "Founder Windows Inc.",
                            "relation": "October 22, 2015, Bill was senior to Wesley but didn\'t manage"
                        },
                        "recommendation": "Wes is one of the sharpest engineers I have worked with. His biggest assets are his technical versatility and engineering creativity ..."
                    }
                ],
                "givenCount": 7,
                "receivedCount": 15
            },
            "connectionDegree": 3,
            "numberOfConnections": 500,
            "volunteerExperience": [
                {
                    "cause": "Environment",
                    "companyName": "Pacific Ridge Community",
                    "description": "Community board planning member for everything in the Sorrento Valley Community. ...",
                    "designation": "Board Member (past Treasurer)",
                    "volunteerDuration": "16 yrs 2 mos",
                    "datesVolunteeredString": "Jan 2004 – Present"
                }
            ],
            "connectionDateString": "June 14, 2014",
            "skillsAndEndorsements": {
                "topSkills": [
                    {
                        "skillName": "Embedded Systems",
                        "eliteEndorsers": [
                            {
                                "name": "Sean Liming",
                                "headline": "Owner of AnnaBooks, LLC."
                            },
                            {
                                "name": "Franz Fisher",
                                "headline": "Director of NASA"
                            }
                        ],
                        "endorsementCount": 79,
                        "colleagueEndorsers": [
                            {
                                "name": "Sean Liming",
                                "headline": "Owner of AnnaBooks, LLC."
                            },
                            {
                                "name": "Franz Fisher",
                                "headline": "Director of NASA"
                            }
                        ]
                    },
                    {
                        "skillName": "Wireless",
                        "endorsementCount": 78
                    }
                ],
                "otherSkills": [
                    {
                        "skillName": "Hardware",
                        "endorsementCount": 2
                    },
                    {
                        "skillName": "USB",
                        "endorsementCount": 6
                    }
                ],
                "industrySkills": [
                    {
                        "skillName": "Mobile Devices",
                        "endorsementCount": 50
                    },
                    {
                        "skillName": "Wireless",
                        "endorsementCount": 78
                    }
                ],
                "toolsAndTechnologies": [
                    {
                        "skillName": "C++",
                        "endorsementCount": 15
                    },
                    {
                        "skillName": "Andriod",
                        "endorsementCount": 9
                    }
                ]
            },
            "licensesAndCertifications": [
                {
                    "issuedDate": "Nov 2019",
                    "Issuing authority": "Udemy",
                    "certificationName": "Certified Computer Professional (CCP)",
                    "certificationLinkURL": "https://www.udemy.com/certificate/UC-8U59W6QZ",
                    "credentialIdentifier": "Credential ID UC-8U59W6QZ"
                }
            ]
        },
        "data_state_c": 1
    },
    "in_JSON_value": "data_value",
    "in_sql_insert_1": "data_state_c,",
    "in_sql_insert_2": "1,"
}
';


set injason = cast(@hugeJason as json);
set json_text= JSON_EXTRACT((injason), CONCAT('$.', 'in_JSON'));
select json_text;


END master_block//
DELIMITER ;


call sp_jason(1);

有人知道是什么原因造成的吗？ @缺口！你在外面吗？ :)

【问题讨论】：

标签： mysql

【解决方案1】：

也许这个答案会帮助你： MySQL warning: Incorrect string value: '\x96

16 进制的 96 大概是 latin1 编码的破折号 (–)。但是你已经指定 CSV 文件是 utf8 编码（或 utf8mb4），这个字符是 utf8 无法理解的。

A 计划：更改文件。（这可能不实用。）

B 计划：告诉 MySQL 该文件是 latin1（而不是 utf8）。然后 MySQL 会将其正确转换为 utf8 编码的 E28093。

“排序”与排序和比较有关； “字符集”与“编码”有关。

可能包含查询的文件编码不正确。

【讨论】：

有趣；根据您的建议，我将如何更改“文本”？如您所见，我们尝试提取的 TEXT 变量已定义为 Latin1；如果那是问题，那不会解决问题吗？还是您的意思是将核心 JSON 变量设置为默认值 utf8mb4 以外的其他值？我该怎么做？此外，如您所见，这不是文件，而是在存储过程本身中定义的输入“字符串”（作为 JSON 变量）。
更多信息，如果我将大 JSON 字符串设置为 Latin1 的 TEXT 值（例如，将 @hugeJason 更改为 var_jason（作为文本 CHARACTER SET latin1）它会给出完全相同的错误，所以这绝对是一个问题使用字符集 - 但最终这对我来说毫无意义。整个字符集/UTF8 垃圾从来都不是 MSSQL 的问题。sigh
我猜想您必须先更改 @hugeJason 变量的编码，然后才能将其转换为 JSON 以删除无效字符。文档还说：JSON 文本，应该使用 ascii、utf8 或 utf8mb4 字符集进行编码。其他字符编码被隐式强制转换为 utf8mb4。 (dev.mysql.com/doc/refman/5.7/en/json.html)。不过我很高兴能帮到你！
问题是即使我将 TEXT 变量声明为 CHARACTER SET utf8mb4，将该值设置为“大 JSON 字符串”（如您在上面的示例中看到的）也会导致错误。 MySQL 根本不想“修复”字符（或编码）——据我所知。我不知道如何解决这个问题。
不幸的是我无法重现该问题，但也许这会有所帮助： SET @hugeJason = SELECT CONVERT(BINARY CONVERT(@hugeJason USING latin1) USING utf8); (stackoverflow.com/questions/1476356/…)

【解决方案2】：

因此，我通过将文本粘贴到 https://jsonformatter.curiousconcept.com/ 中解决了这个问题，这表明 JSON 结构正常且呈绿色，但它表示“已转义的未转义字符”。可悲的是，它没有告诉我在哪里或什么字符，而疯狂的是 '（例如 \'）的 3 个字符转义以前已经完成。所以这一切都没有任何意义，但很明显 MySQL 在管理某些类型的文本时存在严重问题，甚至试图将文本定义为 utf8 或 utf8mb4 甚至 latin1 对这样的问题没有影响，这是荒谬的。可悲的是，我没有从这个“修复”中学到任何东西，仍然不知道为什么会发生错误以及为什么我的好友（MySQL DBA）能够在完全相同的代码导致我的错误时毫无问题地运行代码;但是现在当我通过 json 处理站点运行它时，它神奇地得到了修复（请记住，问题是将其设置为 TEXT 值，最终与 JSON 格式无关）。我遇到的每一个 MySQL 问题中，95% 都与字符集或排序规则问题有关。我从来没有遇到过这样的 MSSQL 问题。我后悔当初选择 MySQL。

【讨论】：