Comments on: How the Linux OOM killer works http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/ Blog O' Matty Tue, 07 Feb 2012 02:31:44 +0000 hourly 1 By: cedricp http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-830524 cedricp Mon, 20 Jun 2011 01:32:59 +0000 http://prefetch.net/blog/?p=2967#comment-830524 Those of you running billing apps or DB2 there might want to look into the fact that you can "immunize" process from oom-killer. This fellow has a nice writeup: http://linux-mm.org/OOM_Killer The key line being: "Any particular process leader may be immunized against the oom killer if the value of its /proc//oomadj is set to the constant OOM_DISABLE (currently defined as -17)" I suppose it won't really help with the memory leaking billing processes -- without oom-killer, they would crash on their own after leaking a bit more memory, so it's not really the operating system's fault that they're crashing; it's just killing them slightly before they would have completely f'd themselves, and maybe preventing the whole OS from going down with them. DB2, on the other hand, should know oom-killer is out there, and guard against it... :) Those of you running billing apps or DB2 there might want to look into the fact that you can “immunize” process from oom-killer.

This fellow has a nice writeup: http://linux-mm.org/OOM_Killer

The key line being: “Any particular process leader may be immunized against the oom killer if the value of its /proc//oomadj is set to the constant OOM_DISABLE (currently defined as -17)”

I suppose it won’t really help with the memory leaking billing processes — without oom-killer, they would crash on their own after leaking a bit more memory, so it’s not really the operating system’s fault that they’re crashing; it’s just killing them slightly before they would have completely f’d themselves, and maybe preventing the whole OS from going down with them.

DB2, on the other hand, should know oom-killer is out there, and guard against it… :)

]]> By: Mapage http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-829667 Mapage Fri, 05 Nov 2010 17:24:53 +0000 http://prefetch.net/blog/?p=2967#comment-829667 OOM killed our (production) DB2 databases yesterday. You should have seen my boss face when I tried to explain why we needed to restore a DB from backup during the midle of the day, FUN. OOM killed our (production) DB2 databases yesterday. You should have seen my boss face when I tried to explain why we needed to restore a DB from backup during the midle of the day, FUN.

]]> By: chm0dvii http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-829233 chm0dvii Tue, 13 Apr 2010 21:02:42 +0000 http://prefetch.net/blog/?p=2967#comment-829233 Aix does the same thing, just uses a different program to do this. Aix does the same thing, just uses a different program to do this.

]]> By: jowblow http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-829229 jowblow Thu, 08 Apr 2010 14:51:28 +0000 http://prefetch.net/blog/?p=2967#comment-829229 Aix does the same thing. Aix does the same thing.

]]> By: Julien Gabel http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-828866 Julien Gabel Mon, 19 Oct 2009 15:29:23 +0000 http://prefetch.net/blog/?p=2967#comment-828866 > How long until this feature decides to kill an important daemon like nfsd or other critical infrastructure processes which sends the whole box to go boom? Not too long in my case: http://blog.thilelli.net/post/2006/12/09/Memory-Behaviour%3A-Tuning-Linuxs-Kernel-Overcommit > How long until this feature decides to kill an important daemon like nfsd or other critical infrastructure processes which sends the whole box to go boom?

Not too long in my case:
http://blog.thilelli.net/post/2006/12/09/Memory-Behaviour%3A-Tuning-Linuxs-Kernel-Overcommit

]]> By: Robert Milkowski http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-827553 Robert Milkowski Thu, 08 Oct 2009 22:21:58 +0000 http://prefetch.net/blog/?p=2967#comment-827553 Woo - the problem with Linux is that by default it overcommits memory - basically when you do a malloc on linux it doesn't reserve ane swap areay (memory + swap disk) and alway returns as successful. Then if you have couple of programs and all of them actually do want to use the memory the problem starts as system is running out of memory but there is basically no interface to tell it to applications as it already told all of them that there is enough of it... so it starts killing application in order to avoid complete lock. In Solaris everytime malloc() some memory system will reserve required space by default so you won't end-up in such a situation. Woo – the problem with Linux is that by default it overcommits memory – basically when you do a malloc on linux it doesn’t reserve ane swap areay (memory + swap disk) and alway returns as successful. Then if you have couple of programs and all of them actually do want to use the memory the problem starts as system is running out of memory but there is basically no interface to tell it to applications as it already told all of them that there is enough of it… so it starts killing application in order to avoid complete lock. In Solaris everytime malloc() some memory system will reserve required space by default so you won’t end-up in such a situation.

]]> By: Woo http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-827439 Woo Wed, 07 Oct 2009 15:18:19 +0000 http://prefetch.net/blog/?p=2967#comment-827439 I'm really surprised that Linux contains such a weird feature. I wonder what state of mind a coder must be to think of a feature that more or less randomly kills processes when the only sane reaction would be an ENOMEM to the offending malloc and letting the offending application handle the error itself. This really sounds like a basis for entertaining debug sessions... especially as root-owned processes seem to only be avoided instead of fully exempt. How long until this feature decides to kill an important daemon like nfsd or other critical infrastructure processes which sends the whole box to go boom? I’m really surprised that Linux contains such a weird feature. I wonder what state of mind a coder must be to think of a feature that more or less randomly kills processes when the only sane reaction would be an ENOMEM to the offending malloc and letting the offending application handle the error itself.
This really sounds like a basis for entertaining debug sessions… especially as root-owned processes seem to only be avoided instead of fully exempt. How long until this feature decides to kill an important daemon like nfsd or other critical infrastructure processes which sends the whole box to go boom?

]]> By: implicate_order http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-827344 implicate_order Mon, 05 Oct 2009 16:04:48 +0000 http://prefetch.net/blog/?p=2967#comment-827344 We've run into the OOM Killer running some critical billing apps in our environment (the Vendor won't support anything besides RHEL4 on HP DL580s). Don't ask me why...we've beaten ourselves silly over this. In any case, their application running single-threaded processes bound to specific CPUs and has extensive memory leaks. As a result, OOM kills processes semi-randomly and has caused significant damage by panicking the system(s). We’ve run into the OOM Killer running some critical billing apps in our environment (the Vendor won’t support anything besides RHEL4 on HP DL580s). Don’t ask me why…we’ve beaten ourselves silly over this.

In any case, their application running single-threaded processes bound to specific CPUs and has extensive memory leaks. As a result, OOM kills processes semi-randomly and has caused significant damage by panicking the system(s).

]]> By: Robert Milkowski http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/comment-page-1/#comment-827150 Robert Milkowski Thu, 01 Oct 2009 10:07:25 +0000 http://prefetch.net/blog/?p=2967#comment-827150 OOM killer is in Linux mostly do to workaround problems with memory overcommiting in Linux. Linux is slowly moving into direction of getting rid of memory overcommiting approach (by tweaking /proc you can disable it to some extend for some time now). The truth is that memory overcommiting + OOM killer is a bad thing - killing semi-randomly applications because system allowed them to allocate more virtual memory than it has is just plain stupid in most environments. But as I said - Linux is slowly catching up and getting rid of that unpleasant feature. btw: in most other Unixes like Solaris OOM killer is not needed as system generally won't allow for memory overcommitment. See also - http://developers.sun.com/solaris/articles/subprocess/subprocess.html#overcom OOM killer is in Linux mostly do to workaround problems with memory overcommiting in Linux. Linux is slowly moving into direction of getting rid of memory overcommiting approach (by tweaking /proc you can disable it to some extend for some time now). The truth is that memory overcommiting + OOM killer is a bad thing – killing semi-randomly applications because system allowed them to allocate more virtual memory than it has is just plain stupid in most environments. But as I said – Linux is slowly catching up and getting rid of that unpleasant feature.

btw: in most other Unixes like Solaris OOM killer is not needed as system generally won’t allow for memory overcommitment.

See also – http://developers.sun.com/solaris/articles/subprocess/subprocess.html#overcom

]]>