Transcript
Meme format: Undertaker standing behind AJ Styles
AJ styles caption: Stubborn process consuming too much memory.
Undertaker caption: OOM Killer
heh one time I was compiling Linux kernel, I used full LTO with all 24 threads used for compilation, like within a minute of starting, it ate up all 48gb of ram before the oom killer could do anything about it and proceeded to lock up my system, since then I left 1 or 2 threads alone.
just go into system monitor, select processes, ctrl-a, right click end ___ processes. checkmate system.
True story:
- doing research. About 100 ff tabs open.
- about 50 emacs buffers open.
- run script that goes roge.
- oom killer kills emacs.
me: YOU, SON OF A BIT**H!
Well, earlyoom maybe. Linux’ own oom-killer is famously too slow to act, and can’t work a long time then, because the PC is already in memory pressure limbo. And also often kills the wrong thing, prelockd and nohang also help against this.
Thoughts and prayers go out to the innocent shell processes killed in the crossfire
Random but in Afrikaans, oom means uncle. Thought that was mildly humorous.
Ah yes, the uncle killer that kills uncle processes.
So an uncle process is the parent process’ sibling process. Makes sense.
Uncle killer
the only times I had OOM killer kick in, the machine was already frozen for a long time trying to swap things in and out and whatnot. So I started using earlyoom - it’ll just nuke that memory hogging process out of existence before it takes the whole machine down.
That sounds to me like it’s basically the same as turning off swap. I don’t think oom killer activates until you fill up your swap as well.
I guess it could still be useful since it’s more tunable than the kernel oom killer.
it’s not, because it takes swap usage into account, so you can benefit from swap without risking a frozen system
I guess swap can still be useful for background browser tabs and apps. I haven’t run swap on my machines for years now, so I can’t really remember how and when it decides to start moving stuff there.
The last few times I ran into OOM killer it’s been from compiling/linking some huge project that used up 90% of my ram on its own… so I’ve probably been having a different experience than most people.
yeah, swap is definitely still useful and it gets used by the kernel well before the main memory is full
For what applications? Is it for people who don’t have a lot if RAM? I have not had a swap partition in ages.
For most things. The only exception I see is if you have really over dimensioned memory, or have a very predictable load so that having the extra reclaimable space won’t make a difference.
I would rather change the meme to:
“Yeah, with modern RAM settings, you don’t need Swap.”
OOM killer lurking behind.
You do need swap with Linux, no matter the amount of RAM. It works without any swap, but not well.
Except that most likely, it kills something completely unrelated instead.
It is now my head canon that it randomly chooses a memory location and kills whatever process owns it, repeating as necessary.
The perverse logic of there being a greater chance of hitting a memory hog makes far too much sense.
It can’t actually be this though, can it?
Eeny, meeny, miny- YOU
I believe, and I could be very wrong, that it kills the largest memory process. The problem is that it may not be the problematic process. You could have a well behaved DB that is using a stable 80%, and a runaway process that rapidly fills the remaining 20%, and OOM will kill the DB, and let the out of control process keep consuming memory until it becomes the largest. Then your DB is down and you don’t known why.
I like the idea of just killing processes at random though. Determinism is overrated.
Go full Russian roulette and include the OOM killer itself in the list of targetable processes.
Oom killer is in the kernel, so that definitely would be exciting.
It isn’t.
On Linux, the OOM generally weighs 2 things, and based on those, starts killing processes until the system dies or goes back to life.The 2 things it weighs are memory footprint and how recent the process is.The reasoning is that, by killing a recently spawned process that is hogging a lot of memory, you will probably kill an unruly app that a user started and not a critical system component. Even in the case of servers with databases that use hundreds of GB of memory, it should in theory not kill them if there’s a recent process that has quickly hogged a lot of ram.Of course, it can still be a bit of a hit or miss (after all, it’s a last resource before crashing, and you can’t afford much complexity). But for the most part, it works well enough.Edit: I actually looked at the code. It doesn’t check if the process is recent. I don’t know where I read that it did. Maybe it was the OOM code for something else.
There are still ways to mark an app as non oom killable, like a critical DB process. But it needs to be manually done.
I’ll need to look into that again. At some point, OOM mostly annoyed me by killing Firefox while I my RAM wasn’t quite full and everything was running smoothly, so I disabled it and never bothered again since then. But occasionally I do run into issues.
Are you sure it was OOM? Just in case, run memtest.
Also make sure to setup enough swap. Swap is literally almost free RAM. Sure it won’t replace it, but on 8gb shitbox, having 32GB on swap really helps. Especially if swapping starts at 70% of ram usage.
And zswap to make less writes to nvme and make it last longer.
It will have been systemd-oomd
