I’m gonna give it a try with Gentoo since it’s more flexible and i prefer it to Arch.
Side effects of being in a dsitro hopping mood…
I think you have to attach the snapshot to something. It wouldn’t be browsable like a dataset.
yeah someone on discord said theres a setting to show it… they are turned on by default like datasets
Is it just a list of snapshots? How would you browse it? I’m assuming the Zvol is from a vm or something that’s formatted it to lvm, ext4 or whatever.
dont want to browse it… but I could copy it somewhere else.
1 Like
I’ve never been able to see when just lsing the dataset, but if were to trust the system and ls the .zfs folder itself, I can see the stuff
Like I have /home on a separate dataset so ls -a /home gives me
.
..
<username>
but ls -a /home/.zfs gives me the actual snapshots
For a few weeks now I have had a little issue on Fedora KDE, which I suspect is a memory leak in a daemon. After a few days of runtime, I end up with ridiculous memory usage on packagekitd, see here:
This is the first time seeing this issue with plasma-discover as well, but 230 MB seems excessive considering I don’t use it and have never started it.
From what I can tell packagekitd is a background service to manage various package manager backends as an API for package manager frontends (like discover). Since this is the first time I’m seeing Discover up there too, I suspect it is something else though.
Did anyone ever have issues with memory usage on that?
I have a suspicion it is because the KDE “Software Updates” widget is using packagekitd in the background as well, and I have an unapplied update there for a few weeks now (Krita, which crashes immediately on the latest version, I haven’t added it to the dnf exclude list yet), and I have never noticed it before. Can someone confirm this is happening for them as well, or if it is my system?
Yes, I think that’s right that it doesn’t show up in getdents() when you check the parent directory but zfs will let you act on it directly. I actually don’t use zfs; however, this came up in a conversation with a friend randomly a few months ago. I don’t actually know all the stuff in there and limitations etc just that vfs implementation side sounded like it won’t list it if you just list the parent directory.
@mihawk90: Actually, what you’re seeing might be a newly opened open bug.
-
(FC32) 1857819 – PackageKit leaks large amounts of memory
Version-Release … 1.1.13-3
…
Steps to Reproduce:- Boot a Fedora 32 system with packagekitd enabled.
- Wait a week
Actual results: packagekitd consumes over 1 GB of RAM.
-
(RHEL8) 1854875 – Possible memory leak in packagekit
$ rpm -qa | grep .ackage.it
PackageKit-1.1.12-6.el8.x86_64
cockpit-packagekit-221-1.el8.noarch
PackageKit-glib-1.1.12-6.el8.x86_64
Both are assigned to the upstream maintainer; Hughes responded on the RHEL8 one though just to ask for specific reproducer steps (anything distinguishing it).
-
It is good that you noted you did have updates pending at least,
plasma-discovergoing up there might be relater or may just be sticking around longer at higher state now that packagekit may be having issues. With RHEL8 notexactlyshipping KDE I’m not sure if it’s KDE specific or not. Gnome certainly uses it though,(Unrelated realization - Oh crap did cockpit, yeah I bet it did get integration too; defiantly makes sense to use it.)
Yes. It does enable things like the gnome-software gui, and also things like the “Shutdown and install updates” option in GNOME3 menus.
I’m not sure I’ve ever had memory usage issues with it but am usually on a different/less-updated system. Though later I can check the fully updated F32 systems that may have it running. Also, I am as you might say “much less than a fan of packagekit”…
I believe it is dbus activated as well, but there may be a sytemd unit for it. It can also download it’s own set of yum/dnf repo metadata (many gigs) using up bandwidth/space; including pre-downloading updates depending on the default configuration, which makes me generally not a fan when it is active by default. I apologize for the snark, as I should not be so negative about it.
Per the bug comment above (which is sparse currently) there is a chance that it is within liblua, though I would lean towards blaming packagekit (usage over a library, or expect we might see it in a few other places); but I’m a little bias and the lean is mostly unfounded. I was about to suggest that if plasma-discover was using lua, but actually it may indeed be entangled with packagekit having an issue.
Yes it might be checking in with it regularly over the dbus interface.
At the moment I can’t confirm this myself, but maybe the existence of those BZ (different reporters) could be an instance of what you see. They did not mention desktop environment. I used an alternate one on my main system. I will try taking a peek at the other half updated system and the extra media laptop as that usually sits there doing mostly nothing but is regularly updated.
I assume you can restart the service as a workaround, possibly disable it though some things will wonder where it went , so I won’t go over those details unless unsure; however, it might be worth collecting the following details to keep on your system for later if you need to restart it (replace $pid with respective pid for packagekitd/discover).
-
cat /proc/$pid/maps > /tmp/packagekitd-maps.txt -
cat /proc/$pid/maps > /tmp/plasma-discover-maps.txt lsof > /tmp/packagekitd.lsof-
gcore -o /tmp/packagekitd-$(date +%s).gcore$pid rpm -ql &> /tmp/packagekitd-rpm_listing.txt- Maybe also save your recent system logs in case there is something in there from kde etc, a leak may not show there but if there is a chance of details about it’s activity or other applications using. At the least if it’s running then
systemctl status packagekitd > /tmp/packagekitd-status.txt
The gcore will take up around as much space as the memory usage of the process. Can be kept for later if it happens to be useful in investigations though it may not be.
EDIT2: OH HAI THARE!
systemctl status packagekit
● packagekit.service - PackageKit Daemon
Loaded: loaded (/usr/lib/systemd/system/packagekit.service; static; vendor preset: disabled)
Active: active (running) since Tue 2020-09-15 07:04:37 GMT; 1 weeks 2 days ago
Main PID: 3099998 (packagekitd)
Tasks: 14 (limit: 7067)
--> **Memory: 1.1G** <--
CPU: 10min 40.029s
CGroup: /system.slice/packagekit.service
├─3099998 /usr/libexec/packagekitd
├─4099647 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099660 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099673 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099683 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099699 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099712 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099724 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099735 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099748 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
├─4099762 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
└─4099773 gpg-agent --homedir /var/cache/PackageKit/32/metadata/....
Sep 24 06:17:09 localhost.localdomain PackageKit[3099998]: uid 1001 obtained auth for org.freedesktop.packagekit.system-sources-refresh
Sep 24 06:17:49 localhost.localdomain PackageKit[3099998]: refresh-cache transaction /4804_dcabaece from uid 1001 finished with success after 39649ms
Sep 24 06:17:59 localhost.localdomain PackageKit[3099998]: get-updates transaction /4805_ddbacbbc from uid 1001 finished with success after 2728ms
Sep 24 06:18:01 localhost.localdomain PackageKit[3099998]: get-updates transaction /4806_daecdaab from uid 1001 finished with success after 1293ms
....
We meet again~ First you eat my bandwidth, then my disk, then my ram, what more does this system have to give?? No, not my cycles! 
(By itself it’s around 668MB) which does seem a bit high, I’ll keep an eye on it. (PackageKit-1.1.13-3.fc32.x86_64)
Of course this is the system that is not fully updated, which is always a problem when it comes to this type of issue. The system with only systemd and a few packages not fully updated ha the same version, at 858mb, 2.2G group:
Though maybe that is the problem “please update and see if the issue occurs” <issue can’t occur… because it’s updated?> (it could be directly related but I’m unsure)
I may poke around a bit more you you might consider checking in on the bz if it seems appropriate (or reporting your observations and just that you’re not sure about other details but wanted to report it).
- How long has your
packagekitdprocess been running this time, did you say? (do you get any status out of it atsystemctl status packagekit(no “d” at the end)
Huh interesting… I searched for that but didn’t find anything. Gonna follow that, thanks!
Odd that it is a report from july, but I haven’t seen it until a few weeks ago… although I rarely check the memory usage unless I’m actually running out of memory, so this might have just flown under the radar for me.
sidenote: when you’re quoting, you don’t need @ 
people get notifications from being quoted by default 
edit:
Additional info: Restarting the packagekit service frees up the leaked memory.
For anyone coming across this who doesn’t know how to do this: open terminal and execute systemctl restart packagekit
1 Like
Oh sorry, I decided to edit my comment a bit.
There is a report for RHEL8 as well. And it looks like mine is up to 1GB on my less used and half updated F32. Will check the recently hardware refreshed (drive swap to newer-old laptop) and usually very up to date makeshift htpc laptop.
I suppose the steps to collect those details are less important at the moment with it restarted so don’t worry about those now.
Do you know how long it had been running in your case?
Well that’s unfortunate because I read that edit after I restarted the service already 
But anyway, at the time of writing the post the uptime of the system was around 7.5 days. What I noticed though it that the memory usage seems to peak at ~660MB and stay there for me. I get up to that in a one or two days, but then it just stays there, which doesn’t really make a whole lot of sense for a memory leak… but maybe that’s just Linux memory management doing its thing.
No worries. I accidentally slipped and hit tab when reaching for tilde and managed to hit enter and post it a bit earlier that I had planned; but then I got distracted checking systems as well.
On mine the process itself is at around 668MB (the systemctl output includes the whole cgroup/slice.)
If I can continue to see it grow (that system is not too important and can be left as is), then I may try poking around and reproducing it there or on another one to see if I can get anything useful out of it.
Yeah, now that I’ve checked it directly, it may be “expected” to be around 700mb (maybe it’s holding onto something associated with repo-medata that makes it stay around the same and not actually ‘leak’ continuously. I hope it’s not expected to always be at 700mb. Gnome shell has some javascript engines in it (as an example) and so you can choose if garbage collection cycles are worth it to do more often or less often; that’s not to say that that is the type of thing here, but to say that maybe it does eventually free things the next time it takes a specific action. Depends a lot on if it’s expected. More poking around will be needed.
EDIT: and on the newer and most recent updated system
cgroup at “2.2G”, packagekitd at 858MB; PackageKit-1.1.13-3.fc32.x86_64 (GNOME3) (uptime: 5 days; and process running about that long)
EDIT2: oh no no. I poked it and the existing one went up to 923M 
(on the system that was at 668M an hour ago). The poking only made it stronger. haha. It must have heard me talking about it.
(It or something that uses it also tends to happen check in once a day in the morning recently from the gnome session, likely just gnome-software or something (I think gpk had been phased out); or it checks and reports out to others; I’m not sure if it has a feature like that.
It’s possible that you caught it during/after a check-in interval of some sort (journalctl -u packagekit.service) while it was busy; though I’m not sure if it should be that high regardless; and if it stays that way and does not exit then that would still be a concern. I won’t be able to look more now but will update in the future if I get anything else more worthy of noting.
You probably found it by now, but wasn’t snapdev=visible the other one, in addition to snapdir=visible for svols?
IIRC they default to hidden?
Still need to set a mountpoint to view them from a commandline though?
i had already starting copying with zfs send/recv… so didnt play with trying to make them visible… yet
2 Likes
It’s been up for almost a day again and in the System Monitor it’s back up to ~170MB again. I don’t think it’s got to do with an interval since it doesn’t matter when I look at it.
systemctl status packagekit:
● packagekit.service - PackageKit Daemon
Loaded: loaded (/usr/lib/systemd/system/packagekit.service; static; vendor preset: disabled)
Active: active (running) since Thu 2020-09-24 13:10:19 CEST; 21h ago
Main PID: 1310007 (packagekitd)
Tasks: 14 (limit: 19083)
Memory: 324.0M
CPU: 32.293s
CGroup: /system.slice/packagekit.service
├─1310007 /usr/libexec/packagekitd
├─1342766 gpg-agent --homedir /var/cache/PackageKit/32/metadata/rpmfusion-free-updates-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342779 gpg-agent --homedir /var/cache/PackageKit/32/metadata/fedora-modular-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342803 gpg-agent --homedir /var/cache/PackageKit/32/metadata/updates-modular-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342817 gpg-agent --homedir /var/cache/PackageKit/32/metadata/rpmfusion-nonfree-updates-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342833 gpg-agent --homedir /var/cache/PackageKit/32/metadata/rpmfusion-nonfree-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342847 gpg-agent --homedir /var/cache/PackageKit/32/metadata/updates-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342861 gpg-agent --homedir /var/cache/PackageKit/32/metadata/rpmfusion-free-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342896 gpg-agent --homedir /var/cache/PackageKit/32/metadata/luminoso-Signal-Desktop-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342910 gpg-agent --homedir /var/cache/PackageKit/32/metadata/fedora-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
├─1342926 gpg-agent --homedir /var/cache/PackageKit/32/metadata/WineHQ-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
└─1342938 gpg-agent --homedir /var/cache/PackageKit/32/metadata/fedora-cisco-openh264-32-x86_64.tmp/gpgdir --use-standard-socket --daemon
Sep 24 20:11:14 localhost.localdomain PackageKit[1310007]: in /5637_cacabaac for update-packages package mediawriter;4.1.6-1.fc32;x86_64;updates was updating for uid 1000
Sep 24 20:11:14 localhost.localdomain PackageKit[1310007]: in /5637_cacabaac for update-packages package libproxy;0.4.15-18.fc32;x86_64;updates was updating for uid 1000
Sep 24 20:11:14 localhost.localdomain PackageKit[1310007]: in /5637_cacabaac for update-packages package systemd-libs;245.8-2.fc32;i686;updates was updating for uid 1000
Sep 24 20:11:14 localhost.localdomain PackageKit[1310007]: in /5637_cacabaac for update-packages package libproxy;0.4.15-18.fc32;i686;updates was updating for uid 1000
Sep 24 20:11:14 localhost.localdomain PackageKit[1310007]: update-packages transaction /5637_cacabaac from uid 1000 finished with success after 15017ms
Sep 24 20:11:18 localhost.localdomain PackageKit[1310007]: get-updates transaction /5638_baabdbdb from uid 1000 finished with success after 3431ms
Sep 25 08:09:09 localhost.localdomain PackageKit[1310007]: uid 1000 is trying to obtain org.freedesktop.packagekit.system-sources-refresh auth (only_trusted:0)
Sep 25 08:09:09 localhost.localdomain PackageKit[1310007]: uid 1000 obtained auth for org.freedesktop.packagekit.system-sources-refresh
Sep 25 08:09:10 localhost.localdomain PackageKit[1310007]: refresh-cache transaction /5639_caaaceed from uid 1000 finished with success after 543ms
Sep 25 08:09:10 localhost.localdomain PackageKit[1310007]: get-updates transaction /5640_daacbebb from uid 1000 finished with success after 54ms
Ok thanks. I happened to catch one of them while it was stracing (by chance) at the activity interval that I see on my system. An strace isn’t really want I wanted here but I had collected some details previously and I may be able to compare it afterwards. I had not restarted it to try running valgrind or other debugging just yet unfortunately.
Mostly by intervals I mean/meant to say both that the time it leaks (or leaks the most) might be at those intervals of activity, but also at the same time to suggest that looking at it during activity might mean it is higher but will drop (which is what you mention and I believe I understand what you mean; it’s observed to be up some and likely increasing even before that point when we see log activity has happened. ie. steadily increasing even between those interval endspoints when we see bursts of log activity).
As one random possibility, I wonder if when the other user talks about a small liblua leak if something like the following could be related. 1763008 – net-snmp leaks memory from not shutting down librpm (not this one but that scenario)
Unfortunately I was stracing at the time and not ltracing. I wasn’t planning to catch it at that time but suddenly I did. heh. (during the quiet time it was just glib polling /proc/self/mounts and my test of rapidly mount/unmount a test image caused activity but nothing to cause an obvious/major increase in memory; the system has some samba mounts etc so I wondered a bit.)
One of the maps that was large mostly had a list of rpm details within it. (looking mostly like changelog keys/rows from db2), but I only looked at one so far and not really others or any specifically picked due to -changes- between two points. Though once it settles down again maybe I can core it and compare. I may need to actually restart it and use valgrind or another LD_PRELOAD trick but figured I would look at it while it was in the bad state before trying and maybe not reproducing the issue or having to wait.
I’ll keep you updated if anything else interesting comes up.
I have this problem where Pop OS will occasionally freeze and requires a hard reset. Eventually, it won’t boot anymore and I need to reinstall it.
Specs are r9 3900x, vega 56, and msi mag b550.
It really only freezes if I step away from the computer for a while (half-hour or longer). After this happens a few times it is a toss-up whether or not it will keep working and I usually run into this error.
I am trying to setup KVM with Windows 10 and gpu passthrough but I don’t think it is related to this, since it keeps happening on installs before I set up KVM.
From what I’ve read about similar problems this is a power supply issue. Or related to power savings, depending on how you look at it.
Ryzen CPUs can power-down to just a watt or two of power usage. Then under sudden demand, spike to 95W almost instantly. This requires properly working PSU and motherboard VRMs.
I’ve heard that some people “solve” the problem by disabling deep C states and just basically keeping the CPU at 25W all the time.
And it doesn’t help that the GPU is basically powered off then the login screen comes up, and its power spikes too, at the same time.
1 Like
Made those power savings settings changes in the BIOS and that seemed to do the trick, thanks.
1 Like