Alright, so I’m way out of my depth here, but here’s what I want to acomplish:
- I want to sum everything in the ASIZE column, except for arbitrary rows like “L0 ZFS plain file”
The complete console output to operate on: (Note that this is a “pretty” format that keeps row layout, what I’d actually use is zdb -Pbbb which would output complete numbers rather than “4K” or whatever, so conversion won’t be needed)
root@pve:~# zdb -bbb kpool [180/1097]
Traversing all blocks to verify nothing leaked ...
loading concrete vdev 3, metaslab 222 of 223 .....
122T completed (148435MB/s) estimated time remaining: 0hr 00min 02sec
No leaks (block sum matches space maps exactly)
bp count: 444472779
ganged count: 0
bp logical: 90846428559360 avg: 204391
bp physical: 89297217174016 avg: 200905 compression: 1.02
bp allocated: 134069023522816 avg: 301636 compression: 0.68
bp deduped: 0 ref>1: 0 deduplication: 1.00
Normal class: 133986129666048 used: 74.47%
Special class 57124073472 used: 11.93%
Embedded log class 25769865216 used: 50.00%
additional, non-pointer bps of type 0: 471957
number of (compressed) bytes: number of bps
17: 588 *
18: 505 *
19: 253 *
20: 1066 *
21: 350 *
22: 423 *
23: 641 *
24: 587 *
25: 654 *
26: 414 *
27: 1080 *
28: 5951 ****
29: 66297 ****************************************
30: 481 *
31: 3835 ***
32: 675 *
33: 1022 *
34: 521 *
35: 889 *
36: 574 *
37: 892 *
38: 846 *
39: 2763 **
40: 1735 **
41: 845 *
42: 1171 * [135/1097]
43: 909 *
44: 893 *
45: 884 *
46: 1020 *
47: 1296 *
48: 899 *
49: 3367 ***
50: 2666 **
51: 3678 ***
52: 4111 ***
53: 6104 ****
54: 32086 ********************
55: 8111 *****
56: 14490 *********
57: 7151 *****
58: 8656 ******
59: 10764 *******
60: 8717 ******
61: 3116 **
62: 19021 ************
63: 6591 ****
64: 7238 *****
65: 8369 ******
66: 12347 ********
67: 20599 *************
68: 11178 *******
69: 7957 *****
70: 6108 ****
71: 4867 ***
72: 3660 ***
73: 4288 ***
74: 4353 ***
75: 6836 *****
76: 3916 ***
77: 3295 **
78: 12559 ********
79: 3035 **
80: 3336 ***
81: 4762 ***
82: 3337 ***
83: 3727 ***
84: 5289 ****
85: 4005 ***
86: 3454 ***
87: 6516 ****
88: 3559 ***
89: 2945 **
90: 3179 **
91: 2879 **
92: 2640 **
93: 5243 ****
94: 3093 **
95: 2906 **
96: 3570 ***
97: 4549 ***
98: 3248 **
99: 2688 **
100: 2742 **
101: 2986 **
102: 2534 **
103: 2565 **
104: 2853 **
105: 3181 **
106: 2626 **
107: 2946 **
108: 2572 **
109: 2960 **
110: 3733 ***
111: 2641 **
112: 2500 **
Dittoed blocks on same vdev: 2686694
Blocks LSIZE PSIZE ASIZE avg comp %Total Type
- - - - - - - unallocated
2 32K 8K 24K 12K 4.00 0.00 object directory
4 512K 16K 48K 12K 32.00 0.00 L1 object array
169 84.5K 84.5K 1.98M 12K 1.00 0.00 L0 object array
173 596K 100K 2.03M 12K 5.94 0.00 object array
2 32K 8K 24K 12K 4.00 0.00 packed nvlist
- - - - - - - packed nvlist size
2 256K 12K 36K 18K 21.33 0.00 L1 bpobj
267 33.4M 2.41M 7.23M 27.7K 13.85 0.00 L0 bpobj
269 33.6M 2.42M 7.27M 27.7K 13.88 0.00 bpobj
- - - - - - - bpobj header
- - - - - - - SPA space map header
2 32K 8K 24K 12K 4.00 0.00 L2 SPA space map
3.78K 60.5M 15.1M 45.4M 12.0K 3.99 0.00 L1 SPA space map
20.3K 180M 87.6M 263M 12.9K 2.06 0.00 L0 SPA space map
24.1K 241M 103M 308M 12.8K 2.34 0.00 SPA space map
5 20K 20K 60K 12K 1.00 0.00 ZIL intent log
44 5.50M 176K 352K 8K 32.00 0.00 L5 DMU dnode
44 5.50M 176K 352K 8K 32.00 0.00 L4 DMU dnode
44 5.50M 176K 352K 8K 32.00 0.00 L3 DMU dnode
45 5.62M 408K 820K 18.2K 14.12 0.00 L2 DMU dnode
1000 125M 44.5M 89.1M 91.2K 2.81 0.00 L1 DMU dnode
882K 13.8G 3.44G 6.89G 8.00K 4.00 0.01 L0 DMU dnode
883K 13.9G 3.49G 6.98G 8.10K 3.99 0.01 DMU dnode
77 308K 308K 620K 8.05K 1.00 0.00 DMU objset
- - - - - - - DSL directory
14 7.50K 1K 12K 877 7.50 0.00 DSL directory child map
12 56K 40K 120K 10K 1.40 0.00 DSL dataset snap map
16 74.5K 25K 156K 9.75K 2.98 0.00 DSL props
- - - - - - - DSL dataset
- - - - - - - ZFS znode
- - - - - - - ZFS V0 ACL
90.0K 11.3G 361M 722M 8.02K 31.92 0.00 L2 ZFS plain file
1.01M 130G 19.6G 39.2G 38.7K 6.61 0.03 L1 ZFS plain file
421M 82.5T 81.2T 122T 296K 1.02 99.96 L0 ZFS plain file
422M 82.6T 81.2T 122T 296K 1.02 99.99 ZFS plain file
4 512K 16K 32K 8K 32.00 0.00 L2 ZFS directory
68.5K 8.56G 275M 550M 8.04K 31.85 0.00 L1 ZFS directory
604K 3.86G 1.68G 3.87G 6.56K 2.30 0.00 L0 ZFS directory
672K 12.4G 1.95G 4.40G 6.71K 6.37 0.00 ZFS directory
11 5.50K 5.50K 104K 9.45K 1.00 0.00 ZFS master node
- - - - - - - ZFS delete queue
- - - - - - - zvol object
- - - - - - - zvol prop
- - - - - - - other uint8[]
- - - - - - - other uint64[]
- - - - - - - other ZAP
- - - - - - - persistent error log
2 256K 72K 216K 108K 3.56 0.00 L1 SPA history
1.14K 146M 13.4M 40.1M 35.1K 10.93 0.00 L0 SPA history
1.14K 146M 13.4M 40.3M 35.3K 10.89 0.00 SPA history
- - - - - - - SPA history offsets
- - - - - - - Pool properties
- - - - - - - DSL permissions
- - - - - - - ZFS ACL
- - - - - - - ZFS SYSACL
- - - - - - - FUID table
- - - - - - - FUID table size
1 1K 1K 12K 12K 1.00 0.00 DSL dataset next clones
- - - - - - - scan work queue
- - - - - - - ZFS user/group/project used
- - - - - - - ZFS user/group/project quota
- - - - - - - snapshot refcount tags
- - - - - - - DDT ZAP algorithm
- - - - - - - DDT statistics
194K 194M 194M 1.52G 8K 1.00 0.00 System attributes
- - - - - - - SA master node
11 16.5K 16.5K 104K 9.45K 1.00 0.00 SA attr registration
30 480K 120K 240K 8K 4.00 0.00 SA attr layouts
- - - - - - - scan translations
- - - - - - - deduplicated block
784 3.81M 2.97M 8.91M 11.6K 1.29 0.00 DSL deadlist map
- - - - - - - DSL deadlist map hdr
1 1K 1K 12K 12K 1.00 0.00 DSL dir clones
3 384K 12K 36K 12K 32.00 0.00 bpobj subobj
- - - - - - - deferred free
- - - - - - - dedup ditto
820 1.01M 325K 6.59M 8.22K 3.17 0.00 other
44 5.50M 176K 352K 8K 32.00 0.00 L5 Total
44 5.50M 176K 352K 8K 32.00 0.00 L4 Total
44 5.50M 176K 352K 8K 32.00 0.00 L3 Total
90.1K 11.3G 361M 723M 8.02K 31.90 0.00 L2 Total
1.08M 138G 19.9G 39.9G 36.8K 6.94 0.03 L1 Total
423M 82.5T 81.2T 122T 295K 1.02 99.97 L0 Total
424M 82.6T 81.2T 122T 295K 1.02 100.00 Total
Block Size Histogram
block psize lsize asize
size Count Size Cum. Count Size Cum. Count Size Cum.
512: 833K 416M 416M 833K 416M 416M 0 0 0
1K: 1.04M 1.22G 1.62G 1.04M 1.22G 1.62G 0 0 0
2K: 889K 2.31G 3.94G 889K 2.31G 3.94G 0 0 0
4K: 2.63M 11.6G 15.6G 963K 5.31G 9.25G 0 0 0
8K: 2.01M 21.8G 37.4G 1.61M 18.6G 27.8G 4.83M 50.6G 50.6G
16K: 1.44M 30.5G 67.9G 2.15M 40.6G 68.4G 2.77M 65.0G 116G
32K: 1.46M 65.3G 133G 802K 36.3G 105G 1.49M 67.1G 183G
64K: 3.08M 310G 443G 1.01M 93.3G 198G 1.64M 151G 334G
128K: 401M 50.2T 50.6T 405M 50.7T 50.9T 403M 75.6T 75.9T
256K: 602K 214G 50.8T 798K 287G 51.1T 734K 259G 76.1T
512K: 451K 320G 51.2T 585K 411G 51.5T 542K 385G 76.5T
1M: 331K 485G 51.6T 264K 371G 51.9T 385K 540G 77.1T
2M: 373K 1.03T 52.7T 153K 432G 52.3T 360K 1.03T 78.1T
4M: 7.14M 28.6T 81.2T 7.58M 30.3T 82.6T 7.35M 43.9T 122T
8M: 0 0 81.2T 0 0 82.6T 0 0 122T
16M: 0 0 81.2T 0 0 82.6T 0 0 122T
What I think I need to do:
- Write output to a file with tee. Easy enough.
- Read the file and ignore the first sections until the
Blocks LSIZE PSIZE ASIZE avg comp %Total Typeheader shows up. No idea where to start with this. - Once we hit the table header, start saving things to an array.
- The number of rows that will be outputted can be variable, such as if you have ZVOLs or not.
- I’m not sure how to distinguish the whitespace between the columns, and the spaces between the words at the end of the rows. I suppose checking if the entry begins with a number would suffice, or assuming a set number of columns which shouldn’t change between major versions.
- Stop filling the array when a blank line or
Block Size Histogramshows up. - Go down the ASIZE column, adding the numbers unless the row identifier is equal to a static value like “
L0 ZFS plain file” - Print out the final value. This will be the real space consumed by metadata on disk for a given pool.
Does anyone have a cool “wow that’s so much simpler than I thought it could be” linux way of accomplishing this? I am absolutely not a programmer or scripter, and pretty much just modify the work of others. While a bash oneliner would be simplest for others to copy+paste, I’m open to any way to solve this. My real sticking point is #2 where I’m not sure how to conceptually solve this with the text manipulation tools common to linux, without resorting to overkill cobbled together python.
I suppose I could try stuffing everything into an array, and then just searching for the rows with my keywords, but that seems like there’s gotta be a less bruteforce way to do it.
