如何使用 MariaDB 在 Django 中正确过滤 SIP 字符？

Question

我试图从我的数据库中过滤掉一些数据，但过滤失败并最终在过滤 SIP（​​补充表意平面）字符（例如“????”时从表中收集了所有行在分机 B 中，“？？？？？？”在 ext-F 中。

这仅适用于使用 MariaDB (10.1.41-MariaDB-0+deb9u1 Debian 9.9) 的生产环境，但在开发环境 (SQLite 3) 中完全正常。

mysite/settings/production.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        ...
        'OPTIONS': {
            'init_command': "SET sql_mode='STRICT_TRANS_TABLES'; SET NAMES utf8mb4;",
            'charset': 'utf8mb4',
            'use_unicode': True,
        }
    }
}

mysite/myapp/views.py

def query_dzih(dzih):
    result = list(Dzih.objects.filter(dzih=dzih))

查询解释 (MariaDB)

>>> Dzih.objects.filter(dzih="\U000200CB")
<QuerySet [<Dzih: Dzih object (0)>, <Dzih: Dzih object (10)>, <Dzih: Dzih
object (11)>, <Dzih: Dzih object (13)>, <Dzih: Dzih object (399)>, <Dzih: Dzih object (456)>, <Dzih: Dzih object (740)>, <Dzih: Dzih object (782)>, <Dzih: Dzih object (922)>, <Dzih: Dzih object (1006)>, <Dzih: Dzih object (1010)>, <Dzih: Dzih object (1020)>, <Dzih: Dzih object (1021)>, <Dzih: Dzih object (1212)>, <Dzih: Dzih object (1316)>, <Dzih: Dzih object (1528)>, <Dzih: Dzih object (1684)>, <Dzih: Dzih object (1720)>, <Dzih: Dzih object (1731)>, <Dzih: Dzih object (1734)>, '...(remaining elements truncated)...']>
>>> Dzih.objects.filter(dzih="\U000200CB").explain()
'1 SIMPLE myapp_dzih ref myapp_dzih_dzih_9a6b0a3f myapp_dzih_dzih_9a6b0a3f 766 const 59 Using where'

查询解释 (SQLite)

>>> Dzih.objects.filter(dzih="\U000200CB")
<QuerySet [<Dzih: Dzih object (3707)>]>

>>> Dzih.objects.filter(dzih="\U000200CB").explain()
'3 0 0 SEARCH TABLE myapp_dzih USING INDEX myapp_dzih_dzih_9a6b0a3f (dzih=?)'

玛丽亚数据库

MariaDB [django]> show variables like "chara%";
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8mb4                    |
| character_set_connection | utf8mb4                    |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8mb4                    |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+

MariaDB [django]> SHOW FULL COLUMNS FROM myapp_dzih;
+----------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+
| Field    | Type         | Collation          | Null | Key | Default | Extra | Privileges                      | Comment |
+----------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+
| dzihn    | int(11)      | NULL               | NO   | PRI | NULL    |       | select,insert,update,references |         |
| dzih     | varchar(255) | utf8mb4_general_ci | NO   | MUL | NULL    |       | select,insert,update,references |         |
| dzihm    | varchar(255) | utf8mb4_general_ci | NO   | MUL | NULL    |       | select,insert,update,references |         |
| dziho    | varchar(100) | utf8mb4_general_ci | NO   | MUL | NULL    |       | select,insert,update,references |         |
| dzihe    | varchar(255) | utf8mb4_general_ci | NO   | MUL | NULL    |       | select,insert,update,references |         |
+----------+--------------+--------------------+------+-----+---------+-------+---------------------------------+---------+

显示创建表（MariaDB）

CREATE TABLE `myapp_dzih` (
  `dzihn` int(11) NOT NULL,
  `dzih` varchar(255) NOT NULL,
  `dzihm` varchar(255) NOT NULL,
  `dziho` varchar(100) NOT NULL,
  `dzihe` varchar(255) NOT NULL,
  PRIMARY KEY (`dzihn`),
  KEY `myapp_dzih_dzih_9a6b0a3f` (`dzih`(191)),
  KEY `myapp_dzih_dzihm_e1ed3e47` (`dzihm`(191)),
  KEY `myapp_dzih_dziho_1e5984df` (`dziho`),
  KEY `myapp_dzih_dzihe_0e01a5cd` (`dzihe`(191))
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

MariaDB [django]> SELECT dzih, HEX(dzih) FROM myapp_dzih WHERE dzih=UNHEX('f0a0838b');
+------+-----------+
| dzih | HEX(dzih) |
+------+-----------+
| ????   | F0A093A5  |
| ????   | F0A6B38A  |
| ????   | F0A59DA9  |
| ????   | F0A995BE  |
| ????   | F0A786A8  |
| ????   | F0A190A8  |
| ????   | F0A9B1A1  |
| ????   | F0A4BABA  |
| ????   | F0A49494  |
| ????   | F0A2B588  |
| ????   | F0A0A8A7  |
| ????   | F0A986B5  |
| ????   | F0A986B5  |
| ????   | F0A68D92  |
| ????   | F0A0AF91  |
| ????   | F0A4A394  |
| ????   | F0A58191  |
| ????   | F0A58195  |
| ????   | F0A28489  |
| ????   | F0A1B182  |
| ????   | F0A69088  |
| ????   | F0A18FB3  |
| ????   | F0A58ABD  |
| ????   | F0A0AC9B  |
| ????   | F0A98E9F  |
| ????   | F0A082A4  |
| ????   | F0A59D8C  |
| ????   | F0A488A6  |
| ????   | F0A68AAE  |
| ????   | F0A0AA9A  |
| ????   | F0A3A5BA  |
| ????   | F0A2939C  |
| ????   | F0A7AA9C  |
| ????   | F0A7AEAB  |
| ????   | F0A1ADB4  |
| ????   | F0A5BCB6  |
| ????   | F0A5A5BB  |
| ????   | F0A188BC  |
| ????   | F0A9B0B2  |
| ????   | F0A892AA  |
| ????   | F0A0838B  |
| ????   | F0A0A8AE  |
| ????   | F0A38594  |
| ????   | F0A3859B  |
| ????   | F0A0A0A6  |
| ????   | F0A0AC9D  |
| ????   | F0A2A692  |
| ????   | F0A2A68F  |
| ????   | F0A7B39F  |
| ????   | F0A6A39E  |
| ????   | F0A8BDBF  |
| ????   | F0A7B38F  |
| ????   | F0A18DAC  |
| ????   | F0A7B6A0  |
| ????   | F0A083AC  |
| ????   | F0A28F9A  |
| ????   | F0A4BBB2  |
| ????   | F0A2809C  |
| ????   | F0A0AC9E  |
+------+-----------+

MariaDB [django]> SELECT * FROM myapp_dzih WHERE HEX(dzih)='f0a0838b';
+------+
| dzih |
+------+
| ????   |
+------+

Answer 1

首先要解决“奇怪的结果”。

select '?' = '?' collate utf8mb4_general_ci;      -- yields 1
select '?' = '?' collate utf8mb4_unicode_ci;      -- yields 1
select '?' = '?' collate utf8mb4_unicode_520_ci;  -- yields 0

也就是说，排序规则很重要。根据查询的具体情况，COLLATION 可能来自许多地方。 （请注意我是如何绕过设置来进行此测试的。）

utf8mb4_general_ci -- simple-minded
utf8mb4_unicode_ci -- from the old Unicode 4.0 standard
utf8mb4_unicode_520_ci -- Unicode 5.20, apparently handling Chinese differently
future:  Unicode 9.0 is available in MySQL 8.0

你的测试：

dzih=UNHEX('f0a0838b')  -- test as strings, based, I think, on the collation of dzih
HEX(dzih)='f0a0838b'  -- test the 8-char hex strings

以下是 191 kludge 的一些解决方法：http://mysql.rjweb.org/doc.php/limits#767_limit_in_innodb_indexes