问题是 SQL 并没有像你想象的那样处理错误。
- SQL Server 保持 ACID 状态。
原子:全有或全无。所有工作都分解成事务性语句,这些语句必须是成功的,否则整个修改/创建被还原,或回滚,到之前的工作状态。您可以通过使用BEGIN TRANSACTION/COMMIT TRANSACTION 或使用ROLLBACK TRANSACTION 来明确一个作品在哪个事务中。 Read More: Transaction Statements 和 ROLLBACK TRANSACTION
一致:每个事务都必须使 SQL Server 处于有效状态。例如,虽然DROP TABLE MyTable; 可能是有效事务,但如果 MyTable 具有依赖项(即Foreign Key Constraints),则 SQL Server 将处于不一致状态并将事务回滚到最后一致状态。
孤立:每笔交易都发生在自己的时间和空间中。我们说 Serialized 是具体的。隔离允许在服务器上一次发出多个甚至相似的语句,并彼此排他地处理每个语句。术语阻塞是指语句正在等待事务提交,而死块是指两个事务无限期地相互等待。
持久性:与驻留在内存中并可能因突然断电而丢失的软件程序不同,SQL Server 的事务一旦提交就永久保存在磁盘上。这为声明提供了确定性。这也是 LOG 文件出现的地方,因为它记录了在数据库上执行的事务。
READ MORE: ACID PROPERTIES
我之所以提到所有这些,是因为您的 BEGIN TRY/CATCH 块会查找这些问题。
- 将表视为数据集
Recall SQL 基于关系集理论。当 SQL 可以对数据/对象的组/集执行操作时,它是最好的。它与继承完全不兼容,并且可以使用草书逻辑,但充其量是低效的。
因此,在您的 ETL 过程中,将数据视为一个整体。
相反,通过将数据视为一个整体,您可以转换用户/Web 界面的轻微拼写错误/错误,并使用谓词子句 (WHERE/ HAVING/ON)
您可以执行的操作类似于以下示例:
CREATE TABLE #ETLTable(ID INT NOT NULL
, Name VARCHAR(50) NOT NULL
, Comment VARCHAR(50) NULL
, Country VARCHAR(50) NOT NULL
, Dated VARCHAR(50)); --implicitly declared as allowing NULL values unless the type denies it
CREATE TABLE #MyTable (ID INT NOT NULL
, Name VARCHAR(50) NOT NULL
, Comment VARCHAR(50) NULL
, Country VARCHAR(50) NOT NULL
, Dated DATE);
CREATE TABLE #ERROR_TABLE (ObjectID INT NULL
, encrypted INT
, text VARCHAR(MAX)
, start_time DATETIME2
, Table_ID INT
, Table_Name VARCHAR(50)
, Table_Country VARCHAR(50)
, Table_Dated VARCHAR(20));
CREATE TABLE #ACTIONS ( [Action] VARCHAR(50)
, [inserted_ID] int
, inserted_Name VARCHAR(50)
, inserted_Country VARCHAR(50)
, inserted_Date Date
, deleted_ID int
, deleted_Name VARCHAR(50)
, deleted_Country VARCHAR(50)
, deleted_Date Date)
INSERT INTO #MyTable (ID, Name, Country, Dated)
VALUES (1, 'Mary', 'USA', '12/23/12')
, (2, 'Julio', 'Mexico', '12/25/12')
, (3, 'Marx', 'USA', '11/11/12')
, (4, 'Ann', 'USA', '11/27/12');
INSERT INTO #ETLTable(ID, Name, Country, Comment, Dated)
VALUES (1,'Mary', 'USA', 'Valid Date', '12/23/12')
, (2,'Julio', 'Mexico', 'Invalid Date', '12-25,12')
, (3,'Marx', 'USA', 'Valid but incorrect Date', '12-11/25') --this actually means YY-MM-DD
, (4,'Ann','USA', 'Not Matching Date', '12-23-12')
, (5, 'Hillary', 'USA', 'New Entry', '11/24/12');
/*SQL Server is fairly flexible to datatypes entries, so be explicit.
Note the date highlighted. This will fail your code since it is of datatype DATETIME. CAST is implicit anyways, and should not be depended on in important queries. Theoretically, you could catch this with your TRY/CATCH block, but the entire Merge statement would be rolled back...an expensive and probably unacceptable cost.
You should proof your INSERTIONS of errors before you start expensive transactions like the MERGE statement. In this example, I knew what dates were being entered and that only the Japanese version might be entered. Dates are very finicky (DATE has no format) and best handled outside the merge statement altogether. */
;WITH CTE AS (
SELECT ID, Name, Country, Comment, ISNULL(TRY_CAST(Dated AS Date), CAST(CONVERT(datetime, Dated, 11) AS DATE) ) AS Dated --TRY_CAST returns a NULL if it cannot succeed and will not fail your query.
FROM #ETLTable
WHERE ISDATE(Dated) = 1
)
MERGE INTO #MyTable TGT
USING CTE SRC ON TGT.ID = SRC.ID
AND TGT.Name = SRC.Name
WHEN MATCHED AND SRC.Dated > TGT.Dated
THEN UPDATE SET TGT.Dated = SRC.Dated
, TGT.Comment = SRC.Comment
WHEN NOT MATCHED BY TARGET
THEN INSERT(ID, Name, Country, Comment, Dated) VALUES (SRC.ID, SRC.Name, SRC.Country, SRC.Comment, SRC.DATED)
OUTPUT $action AS [Action]
, inserted.ID
, inserted.Name
, inserted.Country
, inserted.Dated
, deleted.ID
, deleted.Name
, deleted.Country
, deleted.Dated
INTO #Actions;
/* Note, you would have to run this query separately, as it only records active transactions. */
--CREATE PROC MyProject
--AS BEGIN
--WAITFOR DELAY '00:00:10'
--Print 'ME'
--END
;WITH CTE AS (
SELECT t.objectid, encrypted, text, start_time
FROM sys.dm_exec_requests AS r
CROSS APPLY sys.dm_exec_sql_text(r.sql_handle) AS t
INNER JOIN (SELECT object_id FROM sys.procedures
WHERE object_id = OBJECT_ID('MyProject') ) B ON t.objectid = B.object_id )
INSERT INTO #ERROR_TABLE (ObjectID, encrypted, text, start_time, Table_ID, Table_Name, Table_Country, Table_Dated)
SELECT CTE.objectid, CTE.encrypted, CTE.text, CTE.start_time, B.Table_ID, B.Table_Name, B.Table_Country, B.Table_Dated
FROM CTE
RIGHT OUTER JOIN ( SELECT ID AS Table_ID
, Name AS Table_Name
, Country AS Table_Country
, Dated AS Table_Dated
FROM #ETLTable
WHERE ISDATE(Dated) = 0) B ON 1 = 1
SELECT * FROM #ERROR_TABLE
SELECT * FROM #Actions
SELECT * FROM #MyTable
请注意,该脚本不会破坏SQL Server 中的任何内容,并且实际上在Merge 语句之前将错误输入分开。
- 注意
#ERROR_TABLE 中的有用信息。您实际上可以实时利用这些信息并提出建议。 这是错误捕获的目标。
- 提醒一下,请了解您的
RAISERROR 或TRY/CATCH 仅适用于整个事务...一个工作单元。如果您需要优雅地允许 Merge 语句失败,那很好。但请理解,正确的 ETL 将保证此类昂贵的语句永远不会失败。
因此,无论如何,请在写入决赛桌之前执行所有 ETL 流程。这就是暂存表的意义……因此您可以过滤掉批量表中的非法或拼写错误。
少做事就是懒惰和不专业。学习正确的习惯,它们将帮助您免于未来的灾难。