Follow up - 5ca41c8389 - ActiveJobLock lock update fails due to PG::TRSerializationFailure, 'Could not serialize access due to concurrent update' exception

It indicates that the UPDATE or DELETE statement was queued behind another UPDATE/DELETE statement on the same row. That other statement finished, and due to the guarantees of the Serializable isolation level, the statement in this session was canceled.

The PostgreSQL states that the transaction should automatically retry the UPDATE or DELETE when seeing this error message, whilst taking the updated state of the row into account.

See 'Serializable Isolation Level' section of https://www.postgresql.org/docs/10/transaction-iso.html

app/jobs/concerns/has_active_job_lock.rb

@@ -89,6 +89,12 @@ module HasActiveJobLock
     ActiveJobLock.transaction(isolation: :serializable) do
       yield
     end
+  # PostgeSQL prevents locking on records that are already locked
+  # for UPDATE in Serializable Isolation Level transactions,
+  # but it's safe to retry as described in the docs:
+  # https://www.postgresql.org/docs/10/transaction-iso.html
+  rescue PG::TRSerializationFailure
+    retry
   rescue ActiveRecord::RecordNotUnique
     existing_active_job_lock!
   end

spec/jobs/concerns/has_active_job_lock_spec.rb

@@ -100,6 +100,29 @@ RSpec.describe HasActiveJobLock, type: :job do
         end.to have_enqueued_job(job_class).exactly(:twice)
       end
     end
+
+    if ActiveRecord::Base.connection.adapter_name == 'PostgreSQL'
+      context "when PG::TRSerializationFailure 'Could not serialize access due to concurrent update' is raised" do
+
+        it 'retries execution until succeed' do
+          allow(ActiveRecord::Base.connection).to receive(:open_transactions).and_return(0)
+          allow(ActiveJobLock).to receive(:transaction).and_call_original
+          exception_raised = false
+          allow(ActiveJobLock).to receive(:transaction).with(isolation: :serializable) do |&block|
+
+            if !exception_raised
+              exception_raised = true
+              raise PG::TRSerializationFailure, 'Could not serialize access due to concurrent update'
+            end
+
+            block.call
+          end
+
+          expect { job_class.perform_later }.to have_enqueued_job(job_class).exactly(:once)
+          expect(exception_raised).to be true
+        end
+      end
+    end
   end
 
   include_examples 'handle locking of jobs'