How to Check Why a Process Has Stopped Unexpectedly in Linux

How to Check Why a Process Has Stopped Unexpectedly in Linux

What Do I Need?

  • A Dedicated or VPS Linux Server
  • CentOS
  • ]Putty

Why do Processes Die?

Some unexpected behavior on the web server can at times be caused by system resource limitations. Processes often do this and sometimes quite excessively. Linux by its design aims to use all of the available physical memory as efficiently as possible. However, in practice, the Linux kernel follows a basic rule that a page of free ram is wasted RAM. The system holds a lot more in RAM than just application data; most importantly mirrored data from storage drives for faster access.

  1. Unexpectedly Stopped Process
  1. Suddenly killed tasks are often the result of the system running out of memory, which is when the so-called out-of-memory, or ‘OOM’, makes its presence known. Sort of like the eponymous blue screen of death, or ‘BSoD’, that plagues early versions of Windows.
  2. You can search the logs for messages of out of memory alerts:
sudo grep -i -r ‘out of memory’ /var/log/
  1. Grep goes through all the logs in the directory and will show all the commands just run from /var/log/auth.log. Actual indicators of an OOM killed process would look something like this:

How to Check Why a Process Has Stopped Unexpected in Linux

  1. The log noted here shows the process killed was mysqld with PID 9163 and OOM score of 511 at the time it was killed. Messages and their content will vary depending on the distribution of Linux that you’re using. If, for example, a process crucial to your web application was killed as a result of an OOM situation, you have a few options: reduce the amount of memory asked by the process, disallow processes to overcommit memory, or simply add more memory to your server configuration.
  1. Current Resource Usage
  1. Linux comes with a number of awesome tools for tracking processes that can help with identifying potential resource shortfalls:
free -h

How to Check Why a Process Has Stopped Unexpected in Linux

  1. Here it’s important to make the distinction between application used memory, buffers, and caches. On the ‘Mem’ line of the output it’d appear nearly 75% of our RAM is in use, but then again over half of the used memory is occupied by cached data.
  2. The difference is that while applications reserve memory for their own use, the cache is commonly used hard drive data that the kernel stores temporarily in memory space for faster access, which on the application level is considered free memory.
  3. Keeping that in mind, it’s easier to understand why used and free memory are listed twice; on the second line is the actual memory usage when taking into account the amount of memory occupied by buffers and cache.
  4. In this example, the system is using merely 234MB of the total available 993MB, and no process is in immediate danger of being killed to save resources.
  5. Another useful tool for memory monitoring is ‘top’, which displays useful continuously updated information about processes’ memory and CPU usage, runtime, and other statistics. This is particularly useful for identifying resource exhaustive tasks:
top

How to Check Why a Process Has Stopped Unexpected in Linux

  1. Demonstrated above is a system showing nominal to profile.
  1. Check if Your Processes are at Risk
  1. If your server’s memory gets used up to the extent to which it can threaten system stability, the out of memory killer will choose which process to eliminate based on many variables, such as the amount of work done that would be lost and total memory freed. Linux keeps a score for each running process, which represents the likelihood at which the process would be killed in an OOM situation.
  2. This score is stored on file in /proc/<pid>/oom_score. PID is the identification number for the process you’re looking into. The PID can be easily found using the following command:
ps aux ¦ grep <process name>
  1. The command when searching for MySQL would be similar to the following:

How to Check Why a Process Has Stopped Unexpected in Linux

Next Steps

After completing the above steps it’s always a good idea to disable the ‘over commit’ function that’s inherent in most Linux distributions. The kernel allows by default for processes to request more memory than is currently free in the system to improve memory utilization. This is based on heuristics that the processes never truly use all the memory they request. However, if your system is at risk of running out of memory, and you wish to prevent losing tasks to the OOM killer, it’s possible to disable memory overcommit. And let’s not forget, if you’re running low on usable memory you can always consider actually upgrading your web server hardware.

Conclusion

Overconsumption of system allocated resources can be an indicator of a number of things, including an actual attack by bad actors. Be aware, take precautions, and stay secure.

How To Set up a VSFTPD Server on a CentOS 7 VPS or Dedicated Server

Brief Description FTP is usually insecure exposing clear-text passwords, userna
2 min read
Avi Ilinsky
Avi Ilinsky
Hosting Expert

How To Set up a VSFTPD Server on an Ubuntu 16.04 VPS or Dedicated Server

Brief Description FTP data is usually insecure since information (usernames, pa
2 min read
Eliran Ouzan
Eliran Ouzan
Web Designer & Hosting Expert

How to use phpMyAdmin to develop a website (without MySQL experience)

Brief description A web developer who is not well versed into coding websites f
2 min read
Idan Cohen
Idan Cohen
Marketing Expert

How to Install MySQL on a Windows Web Server Running Apache

This tutorial will show you how to install the MySQL database on a Windows serve
3 min read
Michael Levanduski
Michael Levanduski
Expert Hosting Writer & Tester
HostAdvice.com provides professional web hosting reviews fully independent of any other entity. Our reviews are unbiased, honest, and apply the same evaluation standards to all those reviewed. While monetary compensation is received from a few of the companies listed on this site, compensation of services and products have no influence on the direction or conclusions of our reviews. Nor does the compensation influence our rankings for certain host companies. This compensation covers account purchasing costs, testing costs and royalties paid to reviewers.
Click to go to the top of the page
Go To Top