Why does Linux still swap when I have 768 GB of RAM, and at least 50% of it is still free?

Operating Systems & Open Source Linux
21

I explained it twice above. The machine doesn’t NEED to swap right now, but it is doing it because it’s a task that can be done in advance of HAVING to do it, to improve system responsiveness under load. it’s pre-work the machine can do when it has nothing better to do with its time, that has basically zero penalty.

If you don’t want the machine to touch swap, turn off swap.

edit:
from the linked article, this:

  1. Swap is not generally about getting emergency memory, it’s about making memory reclamation egalitarian and efficient. In fact, using it as “emergency memory” is generally actively harmful.

i.e., don’t wait until you’re fucked to swap. as per above (twice, at least), swap early

6 Likes
22

It’s all there in the article.

People with experience in system programming perhaps find it easier to grasp.

I’m out of this thread. Cheers.

2 Likes
23

I use all HDDs in my “do-it-all” Proxmox server, after killing my 7th SSDs as a result of using said SSD for swap (as I mentioned previously).

…or…they should update the swap code base without the ca. 2018 set of assumptions, or update the swap code base such that if you have systems with a large amount of RAM, either it will turn itself off by default (and you have to “opt-in” to enable swap and/or they actually make swap “smarter” for large memory systems).

It’s absolutely bonkers to me that with 169 GB of RAM free, my system is swapping.

That makes no sense to me, at least when you compare and contrast how swap is actually operating vs. its theory of operation.

24

I read the article in its entirety.

It’s not in there.

Again, Chris makes certain assumptions about the swap use cases (as to when and why you’d want to use swap).

My system, in its present, operating state, doesn’t meet neither of his assumed criteria.

25

sounds like a you problem.

as above, if you don’t want the machine to swap, turn off swap.

you may have read the article, but you either didn’t comprehend it, or didn’t think about how a memory manager needs to respond to inbound requests.

3 Likes
26

HDD swap in 2025. /thread

Sane people would change things after the first SSD dies. Or buy more memory to begin with. But I’m talking about someone using HDDs for swap and arguing with people and criticizing the “swap code base” for his 300MB “swap problem” on a 768GB RAM machine.

It’s fine and you are just doing it wrong.

That’s a you-problem. Not a problem of either the code base or any other reasonable person.

3 Likes
27

Again, swapping to HDD is slow and unintelligent (on the part of the operation of swap).

Actually, it was having a performance penalty because it was causing the I/O wait to spike because it was swapping to HDDs.

As a result of that, it pushed the load averages up from ~38.42 to about 150, BECAUSE it was swapping to HDD (when it doesn’t need to).

So, it’s NOT “zero penalty”. The penalty is literally, an I/O wait of something like 60% and pushing the load average up by about 110.

THAT is why I finally decided to post about it.

Again, I understand that.

But you’ve got 169 GB of RAM free before you’re even REMOTELY CLOSE to being fucked.

And that’s if you COMPLETELY ignored the 271 GB that’s buffered/cached by the system itself, where you assume that you’re not going to start clearing that out so that it can be reclaimed for use by something else.

Again, I PERFECTLY understand this logic if you were working with 16, 32, 64, and maybe even 128 GB of RAM, as to why you would want it to do this.

But when you’re at 768 GB of RAM, and you have more free RAM available to the system than what most people have installed (typically), in their home system, you WELL under any sort of “getting fucked” threshold, if one exists.

That’d be like saying that the engine is going to start cutting off your fuel even when your foot is on wide open throttle, when you’re in a Formula 1 or Indy Car race, because you’re trying to save gas, when you still have over 1/2 a tank left.

That’s what swap is doing here.

28

Swapping to HDD well in advance of needing to page out is much faster than waiting until you’re out of RAM and THEN doing it.

Your memory is idle. The system is idle. It has no way of knowing if you’re going to have an inbound memory request for 300 GB of memory imminently. It is doing nothing useful.

SWAP OUT IDLE STUFF

is the strategy. the fact you’re using HDD for swap is not the decision of the system, that’s your decision (dumb or whatever is up for debate).

I’ve been all SSD for a decade or more at this point and have yet to kill an SSD from swapping to it.

5 Likes
29

Can’t. Platform limit is 64 GB.

Sure, if you ignore the fact that I just proved how it was and has used 8 GB before, and in the time since I started this thread, has already grown to 4 GB.

No. It’s crap and you know that it’s crap because you still can’t explain why it’s swapping at all when I have 169 GB of RAM free (excluding the 271 GB that’s held by buff/cache, which could be freed up, if needed).

No.

As I have shown, Chris Down makes certain assumptions about how he wrote the code for swap/kernel, NEITHER of which are the case here.

And yet, it still swaps when my system doesn’t meet neither of his criteria that necessitates swapping.

That’s a problem with the code.

This is super simple:

A system with 169 GB of RAM free and 440 GB of RAM that CAN be free, should NOT be swapping. Period. There is absolutely NO reason why, when you have 169 GB of RAM free, why it needs to swap.

Is the RAM about to be hit hard? No.

Is the system about to run out of RAM? No.

So then why the fuck, is it swapping when neither of the criteria that Chris Down wrote about, is true and present?

(And oh by the way, you can’t load a 440 GB direct sparse matrix in RAM, all that quickly neither. I’ve tried. That’s how I knew that using the distributed stripped gvol wasn’t really working all that particularly well.)

30

Which is very efficient to do. A lot of storage with different tiers work that way…if you have to shuffle around stuff if it is too late (memory pressure, need to compress stuff), will only make it worse. That’s my main complaint on e.g. zram or most storage tiering approaches.

Acting ahead of time, like swap or how ZFS does with filesystem read cache and special low-intensity feed threads for L2ARC in the background, avoids and often prevents these nasty congestion situations.

2 Likes
31

But again, you’re assuming that I am going to run out of RAM though.

That’s clearly NOT the case here.

Again, I have 169 GB of RAM free and an additional 271 GB that’s held by buff/cache that can be freed up for a grand total of 440 GB of RAM that can be freed.

I’m nowhere remotely close to running out of RAM.

Dunno about the memory being idle when about 324.4 GiB is currently in use. The rest of it, sure.

System LA is about 38 right now.

“It has no way of knowing if you’re going to have an inbound memory request for 300 GB of memory imminently.”
By that logic, it would be swapping out CONSTANTLY, since it will never know what it’s about to hit it.

(The last time I tried, it took somewhere between like 28-33 seconds to load a 400 GB sparse matrix into RAM. Despite the theoretical bandwidths, sparse memory allocation doesn’t really happen all that quickly. Dense matrix memory allocation is even slower.)

It really depends on how much RAM your system has installed and what you’re using/doing.

When you’re working with direct, sparse matrix solvers (for finite element analysis) all day, the scratch disk file alone was like 100 GB for a model that only had something like 500k elements. So, relatively speaking, it’s a pretty small model, but it scratches out a TON of data, during the solve process.

If you’re just browsing Facebook, you can probably have it last well over 100 years.

I killed 7 drives in 7 years which averaged out to killing a drive annually.

It’s really NOT that difficult to do, depending on what you’re doing.

(A single Monte Carlo simulation can consume 2% write endurance on an E1.S drive. It’s really not that hard, but again, it really depends on what you’re doing.)

CFD writes a lot of data at the end, but not as much during the iterative solution process.

FEA, when you’re using the direct sparse solver, craps out a TON of data, during the solution process, such that it wasn’t uncommon for me to be scratching about 100 GB per run, and swap behaving as it does, burned through a SSD annually, on average, for me.

32

Read the assumptions that Chris makes, in his blog post.

…or you can make swap work like how it’s supposed to work, according to its theory of operation: don’t swap when you have 169 GB of RAM free.

To (use) swap when you have 169 GB of RAM free is asinine and you know it.

33

LOVE it when people ignore data.

If that’s your key takeaway, then you’ve completely missed the title of my OP/thread.

34

Swapping out to ANYTHING when you have 169 GB of RAM free is asinine and you know it.

The “distance to empty” is between 440 GB (if you want to clear out what’s been buffered/cached by the system) to 169 GB (if you don’t want to clear out what’s been buffered/cached by the system).

In other words, you’re FARRRR from running out of memory.

Again, that would be like you’re either in a F1 or IndyCar race, and your engine is starting to cut the fuel when you still have more than 1/2 a tank of gas left, to save fuel (pre-emptively, because it thinks that you can run out of fuel at any moment here, or have a huge demand for fuel).

That’s the analogy of what you guys (and Chris) are saying here.

If you had only 1 gallon fuel tank, then it would makes sense to do that.

But if your tank is 100 gallons, then why would you operate the 100 gallon fuel tank using the same assumptions, criteria, and theory of operation, as you would, with the 1 gallon fuel tank where you would have the engine, pre-emptively start cutting the fuel super early on, so that you don’t run out of gas?

That’s what this is doing here.

Y’all realise that the way swap works here, in Linux – it doesn’t know nor take into consideration whether you’re working with the 1 gallon fuel tank or the 100 gallon fuel tank.

(Cuz what it is doing with the system with 100 gallon fuel tank is exactly the same thing that it is doing for the 1 gallon fuel tank. And what it is doing makes SENSE for the car with the 1 gallon fuel tank, but makes less sense for the car with the 100 gallon fuel tank.)

Y’all realise this, yes?

35

I can’t answer that question. Swapping is illogical, given your free resources.

This is why I don’t swap at all. It’s a layer of code doing stuff that doesn’t need to be done (on my systems, at least). I can see it making sense for other folks with fewer resources and less control, but it doesn’t do me any favours at all. Back in much earlier kernels, the swap code itself introduced a notable negative performance penalty. Merely running it — even if it didn’t do anything — harmed performance. That penalty has pretty-much evaporated now, but the fact remains that in certain use cases it is utterly pointless.

I don’t like having to recompile kernels to make low level changes to behaviour. I’m not in a position to fully appreciate how far-reaching the changes may be. So when I hit problems like this (which has happened on every OS I’ve ever used, not just Linux) I take a step back and solve the problem at a different level.

In the 90s when web browsers became a thing and Internet connections were very slow, caching web content was important and greatly sped up the browsing experience. Due to OS/browser/connection flakiness, however, crashes were common and often corrupted the cache files. Sometimes this corrupted the filesystem itself… which was bad.

To prevent damage to the filesystem, and greatly speed up the caching process itself, I created RAM Disks — took a chunk of memory I wasn’t using and made it act like a physical drive — and configured the browser to cache web content to the RAM Disk instead of the HDD that had the operating system on it. Caching was faster, so browsing was faster. It didn’t solve the issue of flakiness, but when crashes did occur the damage was confined to the RAM Disk — not the hard disk — and could thus be fixed by simply formatting the RAM Disk at any time, or rebooting the machine.

If the ultimate purpose of this thread is to try and find a way to avoid losing a drive every year, and a ‘proper’ solution is elusive, then consider this workaround/hack:

  1. Have your system create and mount a temporary file system (tmpfs) when it starts up.
  2. Use 8GB, 16GB or whatever you like from your vast pool of unused memory.
  3. Direct swap to a file on that filesystem.

Swapping will still happen… but it will swap to RAM instead of flash or spinning rust.

  • You won’t incur remotely the performance penalty that swapping to HDD does.
  • You will get performance on par with bursting to NVMe.
  • You won’t drill holes in any more NVMe drives.
  • It will buy you time to find a ‘proper’ solution.

Just an idea.

2 Likes
36

Look up how memory management works

2 Likes
37

I have.

Thus the question, I have 169 GB of RAM free, why it is swapping out AT ALL?

(Reading through some of the replies, there seems to be some kind of misconception about how fast you can actually load 300-400 GB of data into RAM. Hint: It’s NOT at the bandwidth of what the RAM nor its interface can do. It’s about like 10% of that. If that. Most of the time, in my experience, it’s closer to being like 5-8% of the theoretical bandwidth. Again, you can use something like GNU Octave to load 300-400 GB of data in a sparse matrix to RAM, VERY easily, so anybody here can run this test for themselves. Loading 300-400 GB of data to RAM isn’t fast, and therefore; the hit to RAM isn’t as instantaneous as people might perceive it to be, relative to how swap works in Linux.)

People here keep parroting what Chris wrote in his blog, and again, it would make sense IF you are VASTLY more limited with the amount of RAM that’s available to you.

But when you have a system where the amount of RAM that’s free, AFTER what’s been buffered/cached by Linux, is MORE than what people typically have in their system, the assumption that Chris has behind his blog post, that you should swap in a RAM-constrained system, doesn’t apply as much nor as well, to a system where it isn’t as limited/constrained by the amount of RAM that’s installed, in said system.

People write about swapping to a SSD like it’s a good thing.

And if you’re RAM limited, it is.

But when you’re not limited by the amount of RAM that’s installed, swapping to a SSD would STILL be slower than if you just left it in RAM. Even a U.2 NVMe SSD will STILL be slower than RAM.

The assumption about a RAM constrained environment (which drives how swap works in Linux) isn’t something that people are thinking about when you’re not in a RAM-constrained environment anymore.

I would imagine that you know that even a U.2 NVMe SSD is slower than RAM, in terms of random read/write performance.

Again, if you’re in a RAM constrained environment, how swap works makes sense.

But that’s not the case here, and therefore; writing the code for how swap is to operate, assuming a RAM constrained environment, doesn’t make sense when you’ve alleviated that constraint.

I would imagine that people here generally know and understand and recognise this even if the replies don’t always necessarily indicate such.

38

Yeah, I’ve tried that.

I had mixed results with that, because I interpreted some of the issues that I was running into where some of the read/write/sync operations that would normally be running on disk-based swap (whether it’s HDD or SSD), had like a wait timer to it.

But when I did it with a tmpfs drive/partition, RAM, of course, was able to commit the writes to RAM faster than swap might have expected, so it was creating some really weird “funkiness” to system operations.

It will work like for a little while, and then, it would like get “confused” (a la it was expecting the writes to take at minimum certain amount of time, but then writing to a RAM drive was obviously much faster, so it then thinks that there is an error with the write because it finished too quickly, and then I don’t really know what happens other than the system basically crashes because RAM drive is a little “too” fast for it to be swap.

It’s weird.

I’ve tried it.

Didn’t have stable results with it.

1 Like
39

That’s a pity.

The only other thing I can think of is to approach this from the process side — poison the apple.

  1. Make an educated guess about which process is responsible for generating the pages or files that are being swapped.
  2. Add that process to a cgroup.
  3. Set the swappiness of that cgroup to 0.
  4. Leave system-wide swap enabled, and swappiness to something ≥ 1.
  5. Test.
  6. If symptoms persist, goto 1 and pick a different process.

Basically, if you can’t stop the OS from swapping “something”, try work out what the “something” is, then tag it so that it gets skipped during the phase where the kernel scans the candidate list for things that could be swapped — make it ineligible for swapping.

PS: The last time I played with this sort of stuff was back in the v1 days. The current implementation — cgroups v2 — does things a bit differently. Control Group v2 — The Linux Kernel documentation

PPS: Since you are using a dual-CPU system, half of your memory is ‘owned’ by one CPU, and half by the other CPU. Since you are trying to allocate more than half of your memory to a process that is homed on a single CPU, could it be that your swap symptoms are a side-effect of the underlying memory architecture? The MySQL “swap insanity” problem and the effects of the NUMA architecture – Jeremy Cole (Dated, but possibly still relevant? I don’t have a dual-socket system that I can test.)

2 Likes
40

Unless I’m missing something that was already discussed, doesn’t zram address this very issue? Fedora uses zram by default, and then if you do custom partitioning in the installer (or edit your fstab post-install) you can just avoid creating a swap altogether.

Also adjusting vm.swappiness value can reduce the tendency to use swap. I think a value of 0 would disable swap too

(I’m guilty of doing the exact opposite here for my potato laptop :sweat_smile:) zswap to the moon with rolling TRIM operations on a compressed lz4 m.2 swap partition. I can actually run a lubuntu instance inside QEMU if there’s no animations