pstack: a method of checking program dead cycle and deadlock

The pstack command can display the stack trace of each process. pstack $pid is OK. The pstack command must be run by the owner or root of the $pid process.

This time, the cpu accounts for 100%, but there is no exception in memory. It is doubted that there is a dead cycle somewhere. After being reminded by colleagues, use pstack command to check related processes. By running pstack multiple times, it is found that the code stack always stops at the same location. Looking at the code, I found that it was the wrong place.

This is really a very powerful order! As powerful as the strace command

The following is the code stack printing of the process with seven threads

pstack 4551
Thread 7 (Thread 1084229984 (LWP 4552)):
#0  0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000000006f0730 in ub::EPollEx::poll ()
#2  0x00000000006f172a in ub::NetReactor::callback ()
#3  0x00000000006fbbbb in ub::UBTask::CALLBACK ()
#4  0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#5  0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#6  0x0000000000000000 in ?? ()
Thread 6 (Thread 1094719840 (LWP 4553)):
#0  0x000000302afc63dc in epoll_wait () from /lib64/tls/libc.so.6
#1  0x00000000006f0730 in ub::EPollEx::poll ()
#2  0x00000000006f172a in ub::NetReactor::callback ()
#3  0x00000000006fbbbb in ub::UBTask::CALLBACK ()
#4  0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#5  0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#6  0x0000000000000000 in ?? ()
Thread 5 (Thread 1105209696 (LWP 4554)):
#0  0x000000302b80baa5 in __nanosleep_nocancel ()
#1  0x000000000079e758 in comcm::ms_sleep ()
#2  0x00000000006c8581 in ub::UbClientManager::healthyCheck ()
#3  0x00000000006c8471 in ub::UbClientManager::start_healthy_check ()
#4  0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#5  0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#6  0x0000000000000000 in ?? ()
Thread 4 (Thread 1115699552 (LWP 4555)):
#0  0x000000302b80baa5 in __nanosleep_nocancel ()
#1  0x0000000000482b0e in armor::armor_check_thread ()
#2  0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#3  0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#4  0x0000000000000000 in ?? ()
Thread 3 (Thread 1126189408 (LWP 4556)):
#0  0x000000302af8f1a5 in __nanosleep_nocancel () from /lib64/tls/libc.so.6
#1  0x000000302af8f010 in sleep () from /lib64/tls/libc.so.6
#2  0x000000000044c972 in Business_config_manager::run ()
#3  0x0000000000457b83 in Thread::run_thread ()
#4  0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#5  0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#6  0x0000000000000000 in ?? ()
Thread 2 (Thread 1136679264 (LWP 4557)):
#0  0x000000302af8f1a5 in __nanosleep_nocancel () from /lib64/tls/libc.so.6
#1  0x000000302af8f010 in sleep () from /lib64/tls/libc.so.6
#2  0x00000000004524bb in Process_thread::sleep_period ()
#3  0x0000000000452641 in Process_thread::run ()
#4  0x0000000000457b83 in Thread::run_thread ()
#5  0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#6  0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#7  0x0000000000000000 in ?? ()
Thread 1 (Thread 182894129792 (LWP 4551)):
#0  0x000000302af8f1a5 in __nanosleep_nocancel () from /lib64/tls/libc.so.6
#1  0x000000302af8f010 in sleep () from /lib64/tls/libc.so.6
#2  0x0000000000420d79 in Ad_preprocess::run ()
#3  0x0000000000450ad0 in main ()

Tags: Linux

Posted on Sun, 03 May 2020 16:18:29 -0700 by haydndup