Sysadmin Mega Thread

mutation666 · November 18, 2020, 6:02pm

thats most of my writing (large video files)

oO.o · November 18, 2020, 6:02pm

Are you thinking of continuous? Sync is for databases, vms, etc

Dynamic_Gravity · November 18, 2020, 6:03pm

I FTFY

oO.o · November 18, 2020, 6:03pm

ftfy

mutation666 · November 18, 2020, 6:08pm

“Unless explicitly declared as synchronous (by opening with O_SYNC set, or manually calling sync() ), all writes are asynchronous.” (Source:https://jrs-s.net/2019/05/02/zfs-sync-async-zil-slog/)

yeah guess wouldn’t be sync, damn it I want to use these SSDs oh well they will do something one day.

oO.o · November 18, 2020, 6:17pm

There’s that new tiered storage thing that Wendell talked about. I don’t think it’s in mainstream zfs yet though.

Dynamic_Gravity · November 20, 2020, 11:08pm

I like how when ever anyone asks a ZFS question 9/10 it is solved by just linking to the JRS System’s website.

Easy helpdesk tickets.

Dynamic_Gravity · November 25, 2020, 3:31pm

Just had a drive failure. Time to inspect.

[    2.940606] sd 0:0:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    2.940608] sd 0:0:0:0: [sda] 4096-byte physical blocks
[    2.940725] sd 0:0:0:0: [sda] Write Protect is off
[    2.940727] sd 0:0:0:0: [sda] Mode Sense: 46 00 10 08
[    2.940917] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    2.943496]  sda: sda1 sda9
[    2.945615] sd 0:0:0:0: [sda] Attached SCSI disk
[472445.318394] sd 0:0:0:0: [sda] tag#177 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.318397] sd 0:0:0:0: [sda] tag#177 CDB: Read(10) 28 00 00 00 00 00 00 01 00 00
[472445.318400] blk_update_request: I/O error, dev sda, sector 0 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
[472445.318475] sd 0:0:0:0: [sda] tag#178 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.318495] blk_update_request: I/O error, dev sda, sector 3697119936 op 0x1:(WRITE) flags 0x700 phys_seg 2 prio class 0
[472445.318508] blk_update_request: I/O error, dev sda, sector 1608444312 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[472445.318515] blk_update_request: I/O error, dev sda, sector 1608444776 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[472445.318526] blk_update_request: I/O error, dev sda, sector 3725858312 op 0x1:(WRITE) flags 0x700 phys_seg 5 prio class 0
[472445.318528] blk_update_request: I/O error, dev sda, sector 3703242488 op 0x1:(WRITE) flags 0x700 phys_seg 3 prio class 0
[472445.318540] blk_update_request: I/O error, dev sda, sector 1608444320 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[472445.318542] sd 0:0:0:0: [sda] tag#178 CDB: Read(10) 28 00 00 00 08 00 00 01 00 00
[472445.318547] blk_update_request: I/O error, dev sda, sector 2048 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
[472445.318549] blk_update_request: I/O error, dev sda, sector 3725858568 op 0x1:(WRITE) flags 0x700 phys_seg 2 prio class 0
[472445.318564] sd 0:0:0:0: [sda] tag#179 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.318580] blk_update_request: I/O error, dev sda, sector 1608444784 op 0x0:(READ) flags 0x700 phys_seg 1 prio class 0
[472445.318634] sd 0:0:0:0: [sda] tag#179 CDB: Read(10) 28 00 e8 e0 48 00 00 01 00 00
[472445.319098] sd 0:0:0:0: [sda] tag#345 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.319100] sd 0:0:0:0: [sda] tag#345 CDB: Write(10) 2a 00 d8 19 32 d8 00 00 20 00
[472445.319115] sd 0:0:0:0: [sda] tag#343 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.319117] sd 0:0:0:0: [sda] tag#343 CDB: Write(10) 2a 00 d6 0a 5f 28 00 00 20 00
[472445.319126] sd 0:0:0:0: [sda] tag#341 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.319128] sd 0:0:0:0: [sda] tag#341 CDB: Write(10) 2a 00 d8 19 36 70 00 00 08 00
[472445.319134] sd 0:0:0:0: [sda] tag#339 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.319135] sd 0:0:0:0: [sda] tag#339 CDB: Write(10) 2a 00 d8 19 3b a8 00 00 60 00
[472445.319143] sd 0:0:0:0: [sda] tag#337 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.319144] sd 0:0:0:0: [sda] tag#337 CDB: Write(10) 2a 00 d8 19 35 d0 00 00 48 00
[472445.319150] sd 0:0:0:0: [sda] tag#673 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.319151] sd 0:0:0:0: [sda] tag#673 CDB: Write(10) 2a 00 de 2b c0 a8 00 00 08 00
[472445.319160] sd 0:0:0:0: [sda] tag#675 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[472445.319161] sd 0:0:0:0: [sda] tag#675 CDB: Write(10) 2a 00 d4 3d be 30 00 00 08 00
[472456.329248] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[472456.329279] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

Atomic_Charge · November 26, 2020, 2:50am

Wow. I feel like i need to give you a hug with every line you wrote.

And yes, SolarWinds seems to be the norm.

kungr · November 29, 2020, 1:42am

I have a ghetto ubuntu-server and the / file system is 100% full. I can’t update, or install new programs, It’s starting to shutdown and services are crashing.

i ran a sudo apt-get autoremove --purge and got:

Reading package lists… Error!
E: Write error - write (28: No space left on device)
E: IO Error saving source cache
E: The package lists or status file could not be parsed or opened.

ya - soooooooo

zlynx · November 29, 2020, 9:27pm

cd /
du -x | sort -nr

But probably somewhere in /var/log is something you can delete to make space.

Some older Ubuntu/Debian installs had strange problems deleting old kernels so you might have gigabytes in /lib/modules too.

kungr · November 29, 2020, 9:33pm

This is a 32bit OS. So i can delete anything with an appended number?

Do i just delete the middle then update grub?
I tried to purge old kernels but the “no space” killed it.

I keep trying to remove kernels and it keeps telling me “no space left on the device”

SgtAwesomesauce · November 29, 2020, 11:46pm

Yep, those are old logs that have been “rotated”, feel free to delete anything in /var/log with a .gz extension.

I’d remove all of the 4.15.0-X with the exception of the -117 and -118 (as they’re the newest) and keep the 4.4.0-169 just in case there’s a problem with the 4.15 branch.

kungr · November 30, 2020, 1:45am

Did all that and it still says 100% full?

oO.o · November 30, 2020, 2:17am

Please post df -h and lsblk -f

kungr · November 30, 2020, 2:27am

I set this up a few years ago so. The arrangement might be questional.

user@computer:~$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
udev                            3.8G     0  3.8G   0% /dev
tmpfs                           784M   30M  754M   4% /run
/dev/mapper/computer--vg-root   54G   54G     0 100% /
tmpfs                           3.9G  4.0K  3.9G   1% /dev/shm
tmpfs                           5.0M     0  5.0M   0% /run/lock
tmpfs                           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                       720M  133M  550M  20% /boot
tmpfs                           3.9G  4.0K  3.9G   1% /tmp
/dev/sdb                        1.4T  260G  1.2T  19% /mnt/scratch
/dev/sde                        1.9T  1.5T  328G  83% /mnt/data
tmpfs                           784M     0  784M   0% /run/user/1000
/home/*user*/.Private             54G   54G     0 100% /home/*user*
user@computer:~$ lsblk -f
NAME                     FSTYPE      LABEL MOUNTPOINT
fd0                                                                               
sda                                                                               
├─sda1                   ext2             /boot
├─sda2                                                                            
└─sda5                   LVM2_member       
  ├─computer--vg-root    ext4             /
  └─computer--vg-swap_1  swap                
    └─cryptswap1         swap             [SWAP]
sdb                      btrfs            /mnt/scratch
sdc                      btrfs               
sdd                      btrfs                
sde                      btrfs            /mnt/data
sdf                      btrfs

All the parts are out of warranty, so technically it doubles as a waste bin.

/dev/mapper/computer–vg-root ---- is a 60gb vertex2 SSD

VERY NICE!!!

oO.o · November 30, 2020, 2:44am

Anything in /var/tmp? I’d troll through /var for things you can get rid of or boot into a live usb and copy the whole var to another partition temporarily mount that at /var while you clean it up and the copy back over.

zlynx · November 30, 2020, 2:48am

Did you do the

cd /
du -x | sort -n

Here’s a sample of the tail end of what I get on my Fedora NAS:

1724972 ./var/cache/dnf
1726236 ./var/cache
2803984 ./usr/lib
4218884 ./var/log/journal
4218884 ./var/log/journal/69d27b356a94476da859461d3a3bc6fd
4229048 ./var/local
4229048 ./var/local/sfdb
4483164 ./var/log
4624404 ./usr/lib64
4716308 ./usr/share
12129440        ./var
14070212        ./usr
26247852        .

You can see the big targets are /usr and /var, and /usr is usually important stuff you can’t delete. And by “delete” I mean rm -f or rm -rf not apt remove

You can see the logs and cache take up a lot of room on mine. Anything in cache is likely to be fair game to delete.

You don’t need a ton of space. Just free up a few hundred megabytes and then you can use apt again.

kungr · November 30, 2020, 3:10am

It did run it the first time, it errored. But i cleared the old kernels and it managed to run

18624	./sbin
19564	./usr/src/linux-headers-4.15.0-117/include/linux
19564	./usr/src/linux-headers-4.15.0-118/include/linux
19952	./usr/lib/i386-linux-gnu/samba
21032	./usr/share/perl/5.26.1
21036	./usr/share/perl
21332	./usr/lib/expressvpn
21920	./usr/lib/python2.7/dist-packages
22004	./usr/lib/python3.6
22076	./lib/firmware/intel
23924	./lib/firmware/netronome
24312	./lib/firmware/liquidio
24424	./usr/lib/python3.5
26216	./usr/lib/git-core
27572	./usr/lib/python3/dist-packages/twisted
27756	./lib/modules/4.15.0-117-generic/kernel/drivers/net
27756	./lib/modules/4.15.0-118-generic/kernel/drivers/net
28568	./lib/i386-linux-gnu
29004	./usr/share/man
29992	./usr/share/locale
31516	./lib/firmware/amdgpu
31880	./usr/share/vim/vim80
31900	./usr/share/vim
35184	./usr/lib/i386-linux-gnu/dri
39816	./usr/src/linux-headers-4.15.0-117/include
39816	./usr/src/linux-headers-4.15.0-118/include
41552	./usr/sbin
41896	./usr/share/icons
42716	./usr/lib/python2.7
48392	./usr/src/linux-headers-4.15.0-117/arch
48392	./usr/src/linux-headers-4.15.0-118/arch
48924	./var/lib/dpkg/info
49800	./usr/lib/python3/dist-packages
49804	./usr/lib/python3
50540	./usr/share/doc
51496	./var/lib/dpkg
54768	./usr/lib/snapd
84640	./usr/lib/gcc/i686-linux-gnu/7
84656	./usr/lib/gcc/i686-linux-gnu
84660	./usr/lib/gcc
115644	./usr/src/linux-headers-4.15.0-117
115644	./usr/src/linux-headers-4.15.0-118
121452	./lib/modules/4.15.0-117-generic/kernel/drivers
121456	./lib/modules/4.15.0-118-generic/kernel/drivers
129148	./var/lib/apt/lists
129232	./var/lib/apt
141904	./usr/include/boost
163608	./lib/modules/4.15.0-117-generic/kernel
163616	./lib/modules/4.15.0-118-generic/kernel
169000	./lib/modules/4.15.0-117-generic
169008	./lib/modules/4.15.0-118-generic
186456	./usr/include
204740	./var/lib
264428	./usr/src
335540	./lib/firmware
338848	./lib/modules
374692	./usr/bin
393832	./usr/share
507300	./var/log/journal/aa42336a97f7d0d122d9442a5dd1eb2d
507304	./var/log/journal
526764	./var/log
531104	./usr/lib/i386-linux-gnu
729312	./lib
740484	./var
917300	./usr/lib

zlynx · November 30, 2020, 3:15am

And if it is systemd journald files the right way to clean them is like:
journalctl --vacuum-size=2G
On my system that cut the logs down from 4G (which is my setting in /etc/systemd/journald.conf) to 2G.

To control journald sizes have a line in journald.conf like:
SystemMaxUse=4G

If you don’t use journald at all then you might change its Storage option from “persistent” to “volatile” which keeps it in RAM only. It will still copy it to rsyslogd or whatever logger you actually use.