最常见的单词/字符组合 SQL Server

Question

我正在分析“RawDataDescriptions”表中的数据，该表中的“描述”字段是开放式供用户输入的。

我正在寻找方法，通过短语或经常出现的字符串（包括它们出现的次数）对描述进行广泛分类。

我没有特定的单词或短语要寻找我可以使用“like”语句的地方，而是寻找字段之间的共性。

在通过其他问题寻找这一点的同时，我设法找到了一个查询，该查询针对我自己的表进行了调整，该表得到了最常见的单词（粘贴在下面），但当然，仅一个单词就几乎没有 - 如果有的话 - 洞察力描述。

是否可以进行查询以提供短语计数而不仅仅是单个单词？如果是这样，它的主要组成部分是什么？

WITH E1(N) AS 
(
    SELECT 1 
    FROM (VALUES
        (1),(1),(1),(1),(1),(1),(1),(1),(1),(1)
    ) t(N)
),
E2(N) AS (SELECT 1 FROM E1 a CROSS JOIN E1 b),
E4(N) AS (SELECT 1 FROM E2 a CROSS JOIN E2 b)
SELECT
    x.Item,
    COUNT(*)
FROM RawDataDescriptions p
CROSS APPLY (
SELECT 
        ItemNumber = ROW_NUMBER() OVER(ORDER BY l.N1),
        Item       = LTRIM(RTRIM(SUBSTRING(p.[Description], l.N1, l.L1)))
        FROM (
            SELECT s.N1,
                L1 = ISNULL(NULLIF(CHARINDEX(' ',p.[Description],s.N1),0)-
s.N1,4000)
            FROM(
                SELECT 1 UNION ALL
                SELECT t.N+1 
                FROM(
                    SELECT TOP (ISNULL(DATALENGTH(p.[Description])/2,0))
                        ROW_NUMBER() OVER (ORDER BY (SELECT NULL))
                    FROM E4
                ) t(N)
                WHERE SUBSTRING(p.[Description] ,t.N,1) = ' '
            ) s(N1)
        ) l(N1, L1)
) x
WHERE x.item <> ''
GROUP BY x.Item
ORDER BY COUNT(*) DESC

*编辑 - 不可行。替代的期望结果：

示例表

Id | Description  
---+--------------------------
01 | Customer didn't like it  
02 | Person liked it  
03 | Person didn't like it  
04 | Client didn't like it  
05 | person liked it   

@Parameter = 3

想要的结果：

string           | count  
-----------------+-------
didn't like it   | 3  
Person liked it  | 2  

编辑 2** 原来的问题是可行的 - 查看答案

Answer 1

这是一种选择。我有几个问题，例如标点符号、控制字符，尤其是大型表格的性能

示例

Declare @RawDataDescriptions Table ([Id] varchar(50),[Description] varchar(50))
Insert Into @RawDataDescriptions Values 
 ('01','Customer didn''t like it')
,('02','Person liked it')
,('03','Person didn''t like it')
,('04','Client didn''t like it')
,('05','person liked it')

;with cte as (
    Select Id
          ,B.* 
      From  @RawDataDescriptions A
      Cross Apply (
                    Select RetSeq = Row_Number() over (Order By (Select null))
                          ,RetVal = LTrim(RTrim(B.i.value('(./text())[1]', 'varchar(max)')))
                    From  (Select x = Cast('<x>' + replace((Select replace(A.[Description],' ','§§Split§§') as [*] For XML Path('')),'§§Split§§','</x><x>')+'</x>' as xml).query('.')) as A 
                    Cross Apply x.nodes('x') AS B(i)
                  ) B 
)
Select Phrase
      ,Cnt  = count(*)
 From  cte A
 Cross Apply (
     Select Phrase = stuff((Select ' '+RetVal
                            From  cte 
                            Where ID = A.ID
                              and RetSeq between A.RetSeq and A.RetSeq+2
                            Order By RetSeq
                            For XML Path('')),1,1,'')

             ) B
  Where Phrase like '% % %'
  Group By Phrase
  Having count(*)>1
  Order By 2 Desc

退货

Phrase           Cnt
didn't like it   3
Person liked it  2

---

更新 - TVF - 更好的性能

我决定我可能想把它变成一个表值函数，并且对性能提升感到震惊。例如，我有来自 FRED（美联储经济数据）的 130,000 条描述，并且能够在 9 秒内生成常用短语（n 个单词）的列表。

用法

Select Phrase = B.RetVal
      ,Cnt    = count(*)
 From YourTable A
 Cross Apply [dbo].[tvf-Str-Parse-Phrase](A.YourColumn,' ',4) B
 Group By B.RetVal
 Having count(*)>1
 Order By 2 Desc

有兴趣的 TVF

CREATE FUNCTION [dbo].[tvf-Str-Parse-Phrase] (@String varchar(max),@Delimeter varchar(25),@WordCnt int)
Returns Table 
As
Return (  
 with cte as (
      Select RetSeq = Row_Number() over (Order By (Select null))
            ,RetVal = LTrim(RTrim(B.i.value('(./text())[1]', 'varchar(max)')))
      From  (Select x = Cast('<x>' + replace((Select replace(@String,@Delimeter,'§§Split§§') as [*] For XML Path('')),'§§Split§§','</x><x>')+'</x>' as xml).query('.')) as A 
      Cross Apply x.nodes('x') AS B(i)
)
Select RetSeq = Row_Number() over (Order By (Select Null))
      ,B.RetVal
 From  cte A
 Cross Apply (Select RetVal = stuff((Select ' '+RetVal From cte Where RetSeq between A.RetSeq and A.RetSeq+@WordCnt-1 For XML Path('')),1,1,'') ) B
 Where B.RetVal like Replicate('% ',@WordCnt-1)+'%'
);
--Select * from [dbo].[tvf-Str-Parse-Phrase]('This is some text that I want parsed',' ',4)

Answer 2

您可以在桌子上启用Microsoft Full Text Index 并在桌子上执行此类查找常用词和字符分析。

• sys.dm_fts_index_keywords_by_document

• sys.dm_fts_index_keywords_by_property

• sys.dm_fts_index_keywords_by_document