Soooo…
I just god a popup that my Disk Space is running low, and checking FIlelight to see where the hell it went, it turns out my /var/log is getting close to 10gig now…
Well, that’s one thing, but then I also checked journal -b --reverse and I am getting literall hundreds of thousands of log messages like this:
Nov 02 23:00:19 localhost.localdomain audit[579316]: AVC avc: denied { unlink } for pid=579316 comm="sssd_ssh" name=>
Nov 02 23:00:19 localhost.localdomain audit[579317]: AVC avc: denied { unlink } for pid=579317 comm="sssd_nss" name=>
Nov 02 23:00:19 localhost.localdomain sssd_nss[579317]: Starting up
Nov 02 23:00:19 localhost.localdomain sssd_ssh[579316]: Starting up
Nov 02 23:00:19 localhost.localdomain rsyslogd[1128]: imjournal: rename() failed for new path: '/var/lib/rsyslog/imjour>
Nov 02 23:00:19 localhost.localdomain audit[1128]: AVC avc: denied { unlink } for pid=1128 comm="in:imjournal" name=>
Nov 02 23:00:19 localhost.localdomain rsyslogd[1128]: imjournal: rename() failed for new path: '/var/lib/rsyslog/imjour>
Nov 02 23:00:19 localhost.localdomain audit[1128]: AVC avc: denied { unlink } for pid=1128 comm="in:imjournal" name=>
Nov 02 23:00:19 localhost.localdomain rsyslogd[1128]: imjournal: rename() failed for new path: '/var/lib/rsyslog/imjour>
Nov 02 23:00:19 localhost.localdomain audit[1128]: AVC avc: denied { unlink } for pid=1128 comm="in:imjournal" name=>
Nov 02 23:00:19 localhost.localdomain rsyslogd[1128]: imjournal: rename() failed for new path: '/var/lib/rsyslog/imjour>
Nov 02 23:00:19 localhost.localdomain audit[1128]: AVC avc: denied { unlink } for pid=1128 comm="in:imjournal" name=>
Nov 02 23:00:19 localhost.localdomain audit[579310]: AVC avc: denied { unlink } for pid=579310 comm="sssd_nss" name=>
Nov 02 23:00:19 localhost.localdomain audit[579312]: AVC avc: denied { unlink } for pid=579312 comm="sssd_ssh" name=>
Nov 02 23:00:18 localhost.localdomain audit[579315]: AVC avc: denied { unlink } for pid=579315 comm="sssd_pam" name=>
Nov 02 23:00:18 localhost.localdomain audit[579314]: AVC avc: denied { unlink } for pid=579314 comm="sssd_sudo" name>
Nov 02 23:00:18 localhost.localdomain sssd_nss[579310]: Starting up
Nov 02 23:00:18 localhost.localdomain sssd_ssh[579312]: Starting up
Nov 02 23:00:18 localhost.localdomain audit[579311]: AVC avc: denied { unlink } for pid=579311 comm="sssd_pam" name=>
Nov 02 23:00:18 localhost.localdomain audit[579313]: AVC avc: denied { unlink } for pid=579313 comm="sssd_sudo" name>
Nov 02 23:00:18 localhost.localdomain audit[579309]: AVC avc: denied { read } for pid=579309 comm="sssd_be" name="in>
Nov 02 23:00:18 localhost.localdomain audit[579309]: AVC avc: denied { read } for pid=579309 comm="sssd_be" name="in>
Nov 02 23:00:18 localhost.localdomain audit[579309]: AVC avc: denied { read } for pid=579309 comm="sssd_be" name="pa>
Nov 02 23:00:18 localhost.localdomain audit[579309]: AVC avc: denied { read } for pid=579309 comm="sssd_be" name="pa>
Nov 02 23:00:18 localhost.localdomain sssd_be[579309]: Starting up
And no, I’m not joking about hundreds of thousands. I stopped scrolling at line 70.000 and that was an hour ago…
Any idea what the hell this even is or why it’s failing? Is it failing to start whatever it is because the disk is getting full and filling the log because of it?
Just while typing this, it filled another 5 MB…
edit:
seems this has been going on a couple days…
[tarulia@localhost]~% ls /var/log -lh | grep "G "
-rw-------. 1 root root 1.5G Nov 2 23:34 messages
-rw-------. 1 root root 2.1G Oct 30 16:11 messages-20211024
-rw-------. 1 root root 1.9G Nov 1 01:41 messages-20211031
And for the journal:
[tarulia@localhost]~% ls /var/log/journal/45a1b5f2623c470f8974756d48b6e068 -lh
total 4.1G
-rw-r-----+ 1 root systemd-journal 80M Oct 30 00:02 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000060943d-0005cf83130d0471.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 30 02:18 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000061c7dc-0005cf84fb16f452.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 30 04:33 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000062ff48-0005cf86e2c7c10c.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 30 06:49 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000064377e-0005cf88c6189c2b.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 30 09:05 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000006565a5-0005cf8aacb8587f.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 30 11:21 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000066943b-0005cf8c90da2728.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 30 13:37 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000067c203-0005cf8e775643ec.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 30 15:52 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000068f04a-0005cf905cffdd91.journal
-rw-r-----+ 1 root systemd-journal 112M Oct 30 23:07 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000006a1e8c-0005cf9242104120.journal
-rw-r-----+ 1 root systemd-journal 112M Oct 30 23:08 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000006bdce0-0005cf9855bb7be4.journal
-rw-r-----+ 1 root systemd-journal 112M Oct 30 23:09 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000006db5a3-0005cf985963fe3d.journal
-rw-r-----+ 1 root systemd-journal 112M Oct 30 23:45 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000006f96b0-0005cf985c7f19a2.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 31 02:01 system@6e6d1adefe5848eda346d9f0ebb3c6f1-0000000000714c81-0005cf98dd094172.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 31 03:17 system@6e6d1adefe5848eda346d9f0ebb3c6f1-0000000000727e39-0005cf9ac3f40a06.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 31 05:33 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000073b158-0005cf9ca9269c69.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 31 17:56 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000074e2f2-0005cf9e9128f0bc.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 31 20:13 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000007615d1-0005cfa8f14e5b5a.journal
-rw-r-----+ 1 root systemd-journal 80M Oct 31 22:29 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000007747ba-0005cfaadb2487b8.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 00:46 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000007879b4-0005cfacc317a796.journal
-rw-r-----+ 1 root systemd-journal 88M Nov 1 01:43 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000079ab9a-0005cfaeac5c7860.journal
-rw-r-----+ 1 root systemd-journal 104M Nov 1 01:44 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000007b3de1-0005cfaf7734b6e0.journal
-rw-r-----+ 1 root systemd-journal 112M Nov 1 01:45 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000007d13c9-0005cfaf7b1f35f0.journal
-rw-r-----+ 1 root systemd-journal 112M Nov 1 01:46 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000007ef6af-0005cfaf7e32e40f.journal
-rw-r-----+ 1 root systemd-journal 88M Nov 1 03:59 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000080db71-0005cfaf813fa9b3.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 06:15 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000082118d-0005cfb15d97fdc5.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 08:31 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008343e3-0005cfb343668f8a.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 10:47 system@6e6d1adefe5848eda346d9f0ebb3c6f1-0000000000847324-0005cfb52abd7508.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 13:00 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000085a0cf-0005cfb710f0a9d7.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 15:16 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000086d560-0005cfb8ecbec432.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 17:31 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008803f6-0005cfbad1689ada.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 19:46 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008934ed-0005cfbcb44421f4.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 1 22:02 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008a638a-0005cfbe9a12d9fa.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 00:19 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008b958e-0005cfc07efde7ab.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 02:35 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008cc750-0005cfc268789ac7.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 04:51 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008df8b9-0005cfc4508b764d.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 07:07 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000008f268d-0005cfc6372af6de.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 09:23 system@6e6d1adefe5848eda346d9f0ebb3c6f1-0000000000905474-0005cfc81d0a4d16.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 11:36 system@6e6d1adefe5848eda346d9f0ebb3c6f1-0000000000918231-0005cfca02f08157.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 13:50 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000092b6c9-0005cfcbddfbfcaa.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 16:06 system@6e6d1adefe5848eda346d9f0ebb3c6f1-000000000093e87a-0005cfcdbd75b59e.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 18:22 system@6e6d1adefe5848eda346d9f0ebb3c6f1-0000000000951831-0005cfcfa2a221a0.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 20:37 system@6e6d1adefe5848eda346d9f0ebb3c6f1-00000000009646ff-0005cfd18a02987b.journal
-rw-r-----+ 1 root systemd-journal 80M Nov 2 22:52 system@6e6d1adefe5848eda346d9f0ebb3c6f1-0000000000977e96-0005cfd36d3856a5.journal
-rw-r-----+ 1 root systemd-journal 72M Nov 3 00:35 system.journal
-rw-r-----+ 1 root systemd-journal 24M Oct 29 21:45 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000005f1f78-0005cf813fa46684.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 00:02 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000006096bc-0005cf8316da6168.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 02:18 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000061e751-0005cf852da1b0ed.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 04:33 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000063004b-0005cf86e489996d.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 06:49 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000644824-0005cf88e0e7a7ab.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 09:05 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000065805f-0005cf8ad7b017d2.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 11:21 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000066ae1f-0005cf8cba7c4b08.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 13:37 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000067d97f-0005cf8e9d488ae5.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 15:52 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000068f445-0005cf90636ffb50.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 23:07 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000006a32f8-0005cf9262e0e54d.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 23:08 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000006c1abe-0005cf9856353815.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 23:09 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000006dbfcc-0005cf985977f1d5.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 30 23:45 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000006ffc66-0005cf985d27d4a8.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 31 02:01 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000071516e-0005cf98e50e83ec.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 31 03:17 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000728858-0005cf9ad43af061.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 31 05:33 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000073b911-0005cf9cb5ab1bc8.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 31 17:56 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000007509b0-0005cf9ecf24e832.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 31 20:13 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000076274d-0005cfa90d57ee62.journal
-rw-r-----+ 1 root systemd-journal 8.0M Oct 31 22:29 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000774ad1-0005cfaae026b0ec.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 00:46 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000787b8a-0005cfacc5ddb1cd.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 01:43 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000079aeac-0005cfaeb16ebd15.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 01:44 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000007b92d8-0005cfaf77cccc56.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 01:45 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000007d3660-0005cfaf7b5c7cad.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 01:46 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000801dcd-0005cfaf80114816.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 03:59 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000080f0ba-0005cfaf96f289b4.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 06:15 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000008272d1-0005cfb1f81220a4.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 08:31 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000835a85-0005cfb367ce02dd.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 10:47 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000855fc9-0005cfb6a8001db3.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 13:00 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000085bcfc-0005cfb73e59a8bc.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 15:16 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000086e466-0005cfb904a80369.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 17:31 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000880a04-0005cfbadb03caa0.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 19:46 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000894f48-0005cfbcdeba6ae8.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 1 22:02 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000008a7a66-0005cfbebec15196.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 00:19 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000008b9e33-0005cfc08cdd86c8.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 02:35 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000008d76d1-0005cfc381c8599d.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 07:07 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000008f3bd5-0005cfc6591c40be.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 09:23 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000090783d-0005cfc856c11916.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 11:36 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000091de4e-0005cfca97208942.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 13:50 user-1000@e9fe74733a294831b0ff8e44725576c2-000000000092bbd9-0005cfcbe61d1fac.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 16:06 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000009405f1-0005cfcdeccd6d04.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 18:22 user-1000@e9fe74733a294831b0ff8e44725576c2-00000000009519a2-0005cfcfa4ea4b35.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 20:37 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000965883-0005cfd1a6308611.journal
-rw-r-----+ 1 root systemd-journal 8.0M Nov 2 22:52 user-1000@e9fe74733a294831b0ff8e44725576c2-0000000000977eac-0005cfd36d5ea031.journal
-rw-r-----+ 1 root systemd-journal 16M Nov 3 00:35 user-1000.journal
you’ll notice that the system logs are huge and I have no idea what happened there since the only config I changed recently was something related to pulseaudio. I mean 400MB of logs in 4 minutes is seriously impressive. What the actual hell?