当文本包含非日语字符时，日语中的 SQL 排序顺序会中断

Question

当文本包含非日语文本时，日语排序似乎“中断”，即使在查询的 sort 部分之后强制进行任何可能的排序时也是如此。

我想知道这是否是一种已知现象，以及可能的解决方案。

最后我在寻找不区分假名类型、区分大小写的排序，而搜索应该是不区分假名类型和大小写的不区分大小写

这是测试用例：

我会从下面的脚本中假设，我在两个查询中得到相同的结果（预期的排序顺序在第三列中）。基本上一次我按完整单词排序，一次我手动按第一个字母排序，然后是第二个字母，然后是第三个字母。

给定数据库排序规则SQL_Latin1_General_CP1_CI_AS

declare  @temp as table  (title nvarchar(5),  expected int,  script varchar(40) )

set nocount on
INSERT INTO @temp values(N'かか7', 4,'hiragana no accent')
INSERT INTO @temp values(N'がが6',7,'hiragana with accent') 
INSERT INTO @temp values(N'いい5',1,'earlier letter hiragana no accent') 
INSERT INTO @temp values(N'カカ4',3, 'katakana no accent') 
INSERT INTO @temp values(N'ガガ3',6, 'katakana with accent') 
INSERT INTO @temp values(N'かか2',2, 'hiragana no accent') 
INSERT INTO @temp values(N'がが1', 5, 'hiragana with accent')

--BAD
select unicode(left(title,1)) 'bin', * from @temp order by title  
--GOOD
select unicode(left(title,1)) 'bin', * from @temp order by left(title,1),substring(title,2,1), substring(title,3,1)

但是只有第二个版本有效，第一个版本排序不正确：

这似乎与title 字段中的数字有关，因为当我删除它们时，我确实得到了相同的顺序。

declare  @temp as table  (title nvarchar(5),  expected int,  script varchar(40) )

set nocount on
INSERT INTO @temp values(N'かか', 2,'hiragana no accent')
INSERT INTO @temp values(N'がが',3,'hiragana with accent') 
INSERT INTO @temp values(N'いい',1,'earlier letter hiragana no accent') 
INSERT INTO @temp values(N'カカ',2, 'katakana no accent') 
INSERT INTO @temp values(N'ガガ',3, 'katakana with accent') 
INSERT INTO @temp values(N'かか',2, 'hiragana no accent') 
INSERT INTO @temp values(N'がが', 3, 'hiragana with accent')

--GOOD
select unicode(left(title,1)) 'bin', * from @temp order by title  
--GOOD
select unicode(left(title,1)) 'bin', * from @temp order by left(title,1),substring(title,2,1)

查看结果：

有没有人知道原因，可能有解决方案？

Answer 1

蛮力方法：检查 SQL Server 中所有支持的排序规则：

create table ##temp(title nvarchar(5),  expected int,  script varchar(40) );

INSERT INTO ##temp values(N'かか7', 4,'hiragana no accent');
INSERT INTO ##temp values(N'がが6',7,'hiragana with accent');
INSERT INTO ##temp values(N'いい5',1,'earlier letter hiragana no accent'); 
INSERT INTO ##temp values(N'カカ4',3, 'katakana no accent');
INSERT INTO ##temp values(N'ガガ3',6, 'katakana with accent'); 
INSERT INTO ##temp values(N'かか2',2, 'hiragana no accent');
INSERT INTO ##temp values(N'がが1', 5, 'hiragana with accent');

还有脚本：

CREATE TABLE result(collation_name NVARCHAR(1000));
DECLARE @collate_name NVARCHAR(1000);
DECLARE @sql NVARCHAR(MAX);

DECLARE c CURSOR FOR
SELECT name FROM sys.fn_helpcollations() /* where name LIKE '%japan%'*/;

OPEN c;
FETCH NEXT FROM c INTO @collate_name;

WHILE @@FETCH_STATUS = 0  
BEGIN  
     SET @sql = REPLACE(
 N'with cte as (
  select bin = unicode(left(title,1)),expected
         ,rn= row_number() over(order by title collate <collate>)
         ,collation = ''<collate>''
   from ##temp 
)
select collation
from cte
where expected = rn GROUP BY collation HAVING COUNT(*) = 7'
     , '<collate>', @collate_name);
     -- debug
     --PRINT @sql;

     INSERT INTO result(collation_name) EXEC (@sql);
     FETCH NEXT FROM c INTO @collate_name;
END 

SELECT * FROM result;

CLOSE c; 
DEALLOCATE c;

db<>fiddle demo

结果：SQL Server 2017 中没有与“预期顺序”匹配的排序规则。

Answer 2

看起来您的预期订单对日文排序无效。

select case when N'か' COLLATE Japanese_BIN2 < N'カ' COLLATE Japanese_BIN2 then 'True' else 'False' end

否则排序似乎很好。

--CORRECT
select unicode(left(title,1)) 'bin', * from @temp order by title COLLATE Japanese_BIN2

--CORRECT
select unicode(left(title,1)) 'bin', substring(title,2,1), * from @temp order by left(title,2) COLLATE Japanese_BIN2, substring(title,3,1)

两者都生成：

bin    title  expected  script
12356  いい5  1          earlier letter hiragana no accent
12363  かか2  2          hiragana no accent
12363  かか7  4          hiragana no accent
12364  がが1  5          hiragana with accent
12364  がが6  7          hiragana with accent
12459  カカ4  3          katakana no accent
12460  ガガ3  6          katakana with accent

编辑（更多）：

declare  @temp as table  (title nvarchar(5) COLLATE Japanese_BIN2,  script varchar(40) )

set nocount on
INSERT INTO @temp values(N'が', 'hiragana with accent') 
INSERT INTO @temp values(N'い', 'earlier letter hiragana no accent') 
INSERT INTO @temp values(N'カ', 'katakana no accent') 
INSERT INTO @temp values(N'ガ', 'katakana with accent') 
INSERT INTO @temp values(N'か', 'hiragana no accent') 

select * from @temp order by title

title   script
い       earlier letter hiragana no accent
か       hiragana no accent
が       hiragana with accent
カ       katakana no accent
ガ       katakana with accent

使用排序规则集，添加尾随数字变得可预测。

declare  @temp as table  (title nvarchar(5) COLLATE Japanese_BIN2,  script varchar(40) )

set nocount on
INSERT INTO @temp values(N'かか7','hiragana no accent')
INSERT INTO @temp values(N'がが6','hiragana with accent') 
INSERT INTO @temp values(N'いい5','earlier letter hiragana no accent') 
INSERT INTO @temp values(N'カカ4','katakana no accent') 
INSERT INTO @temp values(N'ガガ3','katakana with accent') 
INSERT INTO @temp values(N'かか2','hiragana no accent') 
INSERT INTO @temp values(N'がが1','hiragana with accent')

select * from @temp order by title

title   script
いい5    earlier letter hiragana no accent
かか2    hiragana no accent
かか7    hiragana no accent
がが1    hiragana with accent
がが6    hiragana with accent
カカ4    katakana no accent
ガガ3    katakana with accent

编辑 (CI_AI)

不区分大小写/不区分重音的结果，末尾附加数字。

declare  @temp as table  (title nvarchar(5) COLLATE Japanese_90_CI_AI,  script varchar(40) )

set nocount on
INSERT INTO @temp values(N'かか7','hiragana no accent')
INSERT INTO @temp values(N'がが6','hiragana with accent') 
INSERT INTO @temp values(N'いい5','earlier letter hiragana no accent') 
INSERT INTO @temp values(N'カカ4','katakana no accent') 
INSERT INTO @temp values(N'ガガ3','katakana with accent') 
INSERT INTO @temp values(N'かか2','hiragana no accent') 
INSERT INTO @temp values(N'がが1','hiragana with accent')

select * from @temp order by title

平假名和片假名是不同的脚本，可以互换，只有尾随数字排序。

title   script
いい5    earlier letter hiragana no accent
がが1    hiragana with accent
かか2    hiragana no accent
ガガ3    katakana with accent
カカ4    katakana no accent
がが6    hiragana with accent
かか7    hiragana no accent

编辑 (3)

表定义中的排序规则会影响搜索结果。 您甚至可以在排序时动态更改排序规则。

• Japanese_90_CI_AI：行为（忽略假名，忽略重音）
• Japanese_90_CI_AS：行为（忽略假名，区分重音符号）
• Japanese_90_CS_AI：不影响假名类型
• Japanese_90_CI_AI_KS：行为（忽略重音，排序假名）
• Japanese_BIN2：行为（排序假名，排序重音）

更换下面的 COLLATE 以更好地了解预期内容

declare  @temp as table  (title nvarchar(5) COLLATE Japanese_90_CI_AS,  script varchar(40) )

set nocount on
INSERT INTO @temp values(N'い','earlier letter hiragana no accent') 
INSERT INTO @temp values(N'か','hiragana no accent')
INSERT INTO @temp values(N'カ','katakana no accent') 
INSERT INTO @temp values(N'ガ','katakana with accent') 
INSERT INTO @temp values(N'が','hiragana with accent');

select a.title, b.title, case when a.title = b.title then 'True' else 'False' end search_equivalent
 from @temp a join @temp b on 1=1

最后我正在寻找一个不区分假名类型、区分大小写的 排序，而搜索应该是假名类型不敏感和大小写 不敏感

For : 搜索应该不区分假名类型和大小写：

declare  @temp as table  (title nvarchar(5) COLLATE Japanese_90_CI_AI,  script varchar(40) )

对于：排序

-- Sorting is predictable using the collation above, predictable, but not what you originally expected.
order by title COLLATE Japanese_90_CS_AI_KS

更多（最终版 - v1）

查看您想要的输出：

• （把kata当作hira）
• （区分口音）
• （unicode 顺序）

因为没有 COLLATION ——来自@Lukasz Szozda 的回答

这是一种可能的解决方法：

declare  @temp as table  (title nvarchar(5),  expected int,  script varchar(40))

set nocount on
INSERT INTO @temp values(N'かか7', 4,'hiragana no accent');
INSERT INTO @temp values(N'がが6',7,'hiragana with accent');
INSERT INTO @temp values(N'いい5',1,'earlier letter hiragana no accent');
INSERT INTO @temp values(N'カカ4',3, 'katakana no accent');
INSERT INTO @temp values(N'ガガ3',6, 'katakana with accent'); 
INSERT INTO @temp values(N'かか2',2, 'hiragana no accent');
INSERT INTO @temp values(N'がが1', 5, 'hiragana with accent');

select * from @temp order by dbo.kata2hira(title) COLLATE Latin1_General_BIN2;

title    expected    script
いい5    1           earlier letter hiragana no accent
かか2    2           hiragana no accent
カカ4    3           katakana no accent
かか7    4           hiragana no accent
がが1    5           hiragana with accent
ガガ3    6           katakana with accent
がが6    7           hiragana with accent

CREATE OR ALTER FUNCTION dbo.Kata2Hira (@kana NVARCHAR(MAX))
RETURNS NVARCHAR(MAX)
WITH SCHEMABINDING
AS
BEGIN

  WITH filler AS (
     select value FROM STRING_SPLIT('1,1,1,1,1,1,1,1,1,1',',')
  ), Hira AS (
    -- 3040 - 309F (Hiragana) 
    SELECT TOP (96) 0x3040 + ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS [N]
    FROM filler x JOIN filler y on 1=1
  ), Kata AS (
    -- 30A0 - 30FF (Katakana)
    SELECT TOP (96) 0x30A0 + ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS [N]
    FROM filler x JOIN filler y on 1=1
  ) 
  SELECT @kana = REPLACE(@kana, NCHAR(Kata.[N]), NCHAR(Hira.[N]))
     FROM Hira JOIN Kata ON NCHAR(Hira.[N]) COLLATE JAPANESE_CS_AS = NCHAR(Kata.[N]) COLLATE JAPANESE_CS_AS

  RETURN @kana;

END;

GO

发布解决方法的想法：

我在检查时只是一个杂项仅供参考，它似乎不是非日语字符所独有的。

将 2 和 1 替换为 い 和 あ，默认排序规则的行为一致。

-- Implied Kana Insensitive vs JAPANESE_CS_AS_KS
select case when N'カ' COLLATE JAPANESE_CS_AS < N'ガ' COLLATE JAPANESE_CS_AS then 'True' else 'False' end as want_true union all
select case when N'カ2' COLLATE JAPANESE_CS_AS < N'ガ1' COLLATE JAPANESE_CS_AS then 'True' else 'False' end union all
select case when N'カb' COLLATE JAPANESE_CS_AS < N'ガa' COLLATE JAPANESE_CS_AS then 'True' else 'False' end union all 
select case when N'カい' COLLATE JAPANESE_CS_AS < N'ガあ' COLLATE JAPANESE_CS_AS then 'True' else 'False' end union all
select case when N'1カい' COLLATE JAPANESE_CS_AS < N'2ガあ' COLLATE JAPANESE_CS_AS then 'True' else 'False' end -- sanity check to make sure we're not reading right to left...haha

want_true
True
False
False
False
True

知道为什么会很有趣……

似是而非的理论：重音敏感排序作为第二遍应用在假名不敏感之后的字段（而不是每个字符）。