并发 upsert 上的 Postgres 死锁

Question

我们有一个应用程序，它从数据流中读取数据并将该信息插入到数据库中。数据是发生在 Google Drive 上的变化，这意味着影响相同对象的许多事件可能发生得非常接近。

在将此信息更新到数据库时，我们遇到了死锁，这是日志中的内容。我已经重构并清理了查询的可读性：

ERROR:  deadlock detected
DETAIL:  Process 10586 waits for ShareLock on transaction 166892743; blocked by process 10597.
  Process 10597 waits for ShareLock on transaction 166892741; blocked by process 10586.
  Process 10586: 
          INSERT INTO documents
              (version, source, source_id, ingestion_date)
          VALUES
              (0, 'googledrive', 'alpha', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'beta', '2017-09-21T07:03:51.074Z')
              (0, 'googledrive', 'gamma', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'delta', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'epsilon', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'zeta', '2017-09-21T07:03:51.074Z')

          ON CONFLICT (source, source_id)
          DO UPDATE
          SET
              ingestion_date = EXCLUDED.ingestion_date,
              version = documents.version + 1

          RETURNING source_id, source, uid

  Process 10597: 
          INSERT INTO documents
              (version, source, source_id, ingestion_date)
          VALUES
              (0, 'googledrive', 'delta', '2017-09-21T07:03:51.167Z'),
              (0, 'googledrive', 'gamma', '2017-09-21T07:03:51.167Z')

          ON CONFLICT (source, source_id)
          DO UPDATE
          SET
              ingestion_date = EXCLUDED.ingestion_date,
              version = documents.version + 1

          RETURNING source_id, source, uid

HINT:  See server log for query details.
CONTEXT:  while locking tuple (3908269,11) in relation "documents"
STATEMENT:  
          INSERT INTO documents
              (version, source, source_id, ingestion_date)
          VALUES
              (0, 'googledrive', 'alpha', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'beta', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'gamma', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'delta', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'epsilon', '2017-09-21T07:03:51.074Z'),
              (0, 'googledrive', 'zeta', '2017-09-21T07:03:51.074Z')

          ON CONFLICT (source, source_id)
          DO UPDATE
          SET
              ingestion_date = EXCLUDED.ingestion_date,
              version = documents.version + 1

          RETURNING source_id, source, uid

架构：

    Column      |            Type             |                             Modifiers
----------------+-----------------------------+-------------------------------------------------------------------
uid             | uuid                        | not null default gen_random_uuid()
date_created    | timestamp without time zone | not null default now()
sequence_id     | bigint                      | not null default nextval('documents__sequence_id__seq'::regclass)
version         | integer                     | not null default 0
source          | text                        | not null
source_id       | text                        | not null
ingestion_date  | timestamp without time zone | not null

Indexes:
    "documents__pkey" PRIMARY KEY, btree (uid)
    "documents__sequence_id__unique" UNIQUE CONSTRAINT, btree (sequence_id)
    "documents__source__source_id__deleted" UNIQUE, btree (source, source_id)
    "documents__ingestion_date__idx" btree (ingestion_date)
    "documents__source_id__source__idx" btree (source_id, source)

我怀疑问题类似于“第一个事务是依次锁定具有 source_id alpha、beta、gamma 的行，同时第二个事务是锁定具有 source_id delta、gamma 的相反顺序的行，并且死锁发生在指出他们都锁定了 gamma 和 delta”，但是这里的时间非常紧迫！

解决方案是什么？按 source_id 对我们的值列表进行排序？

Answer 1

您的查询语法允许轻松地对值进行排序：

INSERT INTO documents
          (version, source, source_id, ingestion_date)
   SELECT * FROM (
      VALUES
          (0, 'googledrive', 'alpha', '2017-09-21T07:03:51.074Z'),
          (0, 'googledrive', 'beta', '2017-09-21T07:03:51.074Z')
          (0, 'googledrive', 'gamma', '2017-09-21T07:03:51.074Z'),
          (0, 'googledrive', 'delta', '2017-09-21T07:03:51.074Z'),
          (0, 'googledrive', 'epsilon', '2017-09-21T07:03:51.074Z'),
          (0, 'googledrive', 'zeta', '2017-09-21T07:03:51.074Z')
      ) AS v ORDER BY source, source_id

      ON CONFLICT (source, source_id)

这应该可以解决您的问题。性能应该不错，因为排序会很小。

Answer 2

我能想到三个解决方案：

1. 每个语句只插入一行，但效率很低。

2. 在插入行之前对其进行排序。

3. 如果事务发生死锁或序列化错误，则重试事务。

除非错误经常发生，否则我更喜欢第三种解决方案。