On the IBM AIX platform, all the servers hang. The hang is not
related to load, and usually occurs within 10 minutes or so of server
startup. A thread dump reveals two deadlocked threads, one doing a
socket close() and one doing a socket poll(). For example:
'[ACTIVE]
ExecuteThread:
'5' for queue: 'weblogic.kernel.Default (self-tuning)''
(TID:0x438F2000, sys_thread_t:0x438F9EF8, state:R, native
ID:0x0037F035) prio=5
at java/net/PlainSocketImpl.socketClose0(Native Method)
at
java/net/PlainSocketImpl.socketPreClose(PlainSocketImpl.java:744(Compiled
Code))
at java/net/PlainSocketImpl.close(PlainSocketImpl.java:568(Compiled
Code))
at java/net/SocksSocketImpl.close(SocksSocketImpl.java:1049(Compiled
Code))
at java/net/Socket.close(Socket.java:1341(Compiled Code))
at weblogic/socket/WeblogicSocket.close(WeblogicSocket.java:80(Compiled
Code))
at
weblogic/socket/SocketMuxer.closeSocket(SocketMuxer.java:449(Compiled
Code))
at weblogic/socket/SocketMuxer.cancelIo(SocketMuxer.java:773)
at
weblogic/socket/SocketMuxer$TimerListenerImpl.timerExpired(SocketMuxer.java:976(Compiled
Code))
at weblogic/timers/internal/TimerImpl.run(TimerImpl.java:273(Compiled
Code))
at
weblogic/work/SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:464(Compiled
Code))
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:197(Compiled
Code))
at weblogic/work/ExecuteThread.run(ExecuteThread.java:172(Compiled
Code))
'ExecuteThread: '7' for queue: 'weblogic.socket.Muxer''
(TID:0x3511B500, sys_thread_t:0x34FAD6C8, state:R, native
ID:0x003E80CB) prio=5
at weblogic/socket/PosixSocketMuxer.poll(Native Method)
at
weblogic/socket/PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102)
at weblogic/socket/SocketReaderRequest.run(SocketReaderRequest.java:29)
at
weblogic/socket/SocketReaderRequest.execute(SocketReaderRequest.java:42)
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:145)
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:117)
Internally, WebLogic Server polls the sockets to see if there is
any
new activity. The trouble is we are also calling socket.close() to try
and clear the sockets of existing clients at the same time.
According to IBM engineering:
- The underlying TCP implementation of socket close was changed
between IBM JDK SR5 and SR7. Part of that change is that dup2 is now
used.
- Because of the dup2 change, the poll() and close() methods
cannot
be called at the same time on the same socket.
This causes the deadlock.
Patches for WLS 8.x can be found in My Oracle Support. Open the
Patches & Updates tab. Search for patch ID 8173442 for the patches
for WLS 8.1mp3, 8.1mp4, and 8.1mp5. Search for patch ID 8179792 for the
patch for WLS 8.1mp6.
Patches for WLS 9.x and higher can be downloaded from Smart Update
using these patch IDs and passcodes:
------------------------------------------
PATCH REPOSITORY INFORMATION
------------------------------------------
WLS Version | Patch ID | Passcode
------------+----------+----------------
9.2 | T4DV | 7C7PYV9B
9.2mp1 | HZHQ | PTUYCCSI
9.2mp2 | WJD2 | GU1CW2AB
9.2mp3 | GNLT | 8J9L6Q4Y
10.0 | PMAJ | 9UQ69LLT
10.0mp1 | ITVL | K8RBHQQ2
10.3 | 9YT5 | I1DB5QSV
Fixed in: 10.0.2, 10.3.1