Global Enqueue Services Deadlock detected – TX deadlock in Exclusive(x) mode

In single instance environment, when a deadlock happens, it often reports ORA-60: Detecting and Resolving Locking Conflicts and Ora-00060 errors. In a Real Application Cluster (RAC) environment, instead of ORA-60, one would see the following messages in database alert log:

Global Enqueue Services Deadlock detected. More info in file
/u01/oracle/diag/rdbms/node/node1/trace/node1_ora_763770.trc.

OR

Global Enqueue Services Deadlock detected. More info in file
/u01/oracle/diag/rdbms/node/node1/trace/node1_lmd0_87933.trc.

You can see similar below lines in the trace file.

**************
---------resource 0x2efbe8510----------------------
resname       : [0xae0003][0x5149ef],[TX]
Local node    : 3
dir_node      : 2
master_node   : 2
hv idx        : 70
hv last r.inc : 138
current inc   : 140
hv status     : 0
hv master     : 3
.
.
.
.
.
.
.
----------enqueue 0x30d34bd30------------------------
lock version     : 232019
Owner node       : 3
grant_level      : KJUSEREX
req_level        : KJUSEREX
bast_level       : KJUSERNL
notify_func      : (nil)
resp             : 0x3170afe58
procp            : 0x30c334570
pid              : 13495
proc version     : 0

.

.
.
.
.
 

Global Wait-For-Graph(WFG) at ddTS[0.260b8] :

BLOCKED 0x30dcd0708 5 wq 2 cvtops x1001 TX 0xae0003.0x5149ef [8B000-0004-0036637E] 3

BLOCKER 0x30d29ce90 5 wq 1 cvtops x28   TX 0xae0003.0x5149ef [5D000-0003-001E8444] 2
BLOCKED 0x30e7999b0 5 wq 2 cvtops x1001 TX 0x270010.0x4c2def [5D000-0003-001E8444] 2
BLOCKER 0x30d34bd30 5 wq 1 cvtops x28   TX 0x270010.0x4c2def [8B000-0004-0036637E] 3
DUMP LOCAL BLOCKER/HOLDER: block level 5 res [0xae0003][0x5149ef],[TX]

  

**********

Here  mode 5 is exclusive lock and  instance# starts from 0

 

Above deadlock means two sessions involved in TX-0xae0003-0x5149ef and TX-0x270010-0x4c2def forms a deadlock, both sessions are from instance 4.

This is a typical application transaction TX enqueue lock, usually caused by SQL commit sequence and high concurrency. To avoid such deadlock, application code and logic need to be modified.
The application and SQL involved in the deadlock can be found in lmd0 or foreground trace (check all instances).