Howdy,
I’ve been experimenting with CEPH for the past few years with mostly failures. But I would like to give it a true shot to see what it can do, both for my homelab and to introduce it at work as a storage option (ie as a component of OpenStack). I’m working on my homelab CEPH setup. I’m wondering if I could get some help working out some issues; I have many…
Any and all help would be muchly appreciated. Below is alot of details that I am thinking would be useful.
Thanks!
-Jake
Symptoms:
CephFS works for about 30 seconds, then stops, breaking my plex server/containers.
Known Issues:
- OSD.36 was deleted, but some services keep referencing it.
- Slow Heartbeats…
- Failed daemons
- unformatted discs not being added using “osd.all-available-devices” daemon/service
- 33 PGs stuck in “active+clean+remapped”
- 1PG incomplete (i’m willing to loose some data to clear up the PGs)
- OSD daemons are faling to start/restart (before I set them to ‘no-in’) I was thinking that they might be the cause of the PG issues…
- OSDs 17 & 26 are stuck in a deleting status (expected to delete)
- smartctl stats/info not showing up in the dashboard for disks, enabled & working on each host.
Notes:
- I had a seperate vm (jht-cephvm) that I setup with a 100G seperate drive to put the cluster into 3 node status, but that node failed and lost that vm…)
- I set no-scrub & no-deep-scrub on all OSDs, then manually kicked it off on some of the troubled OSDs
- I had some drives failed SMART, so I removed them after they drained, and it has helped some
- Ran a pg repair on the 1 inconsistant PG “2.1cc” to bring it from “down” status.
- Marked OSD 17,26 as lost in the dashboard.
Details:
2 Nodes, both running Debian 12, each with an HBA to a disk shelf and mixed size ssds and hdds. Ceph version is 19.2. Each server has a 10G link to a Netapp 1610 10G switch with MTU set at 9400. I have a few pools configured in both Replicated and with EC 4k/2m. I also have cephfs configured. All hardware except disks are work trashbin rescues.
Node 1: “media” UCSC-C240-M3S, CPU E5-2650 v2 @ 2.60GHz, 94G Ram, NetApp branded HBA (rebranded LSI 9200-IT…I think) 41 drives between 24 on-board 2.5 slots and a NetApp DS42xx Shelf for 3.5 drives. Roles configured are: Dashboard, _admin, mgr, grafana, mon, osd, mds
Node 2: “jht-cephdell” Dell 720, CPU E5-2620 0 @ 2.00GHz 190G Ram, 29 Drives, 15 on a EMC Branded Disk shelf with an 8088 HBA in IT mode. remaining on 2.5 front ports. Roles configured are mgr, mon, osd
ceph -s
cluster:
id: ed18013c-9996-11ef-9ede-90e2ba62e3b8
health: HEALTH_WARN
1 OSD(s) have spurious read errors
9 failed cephadm daemon(s)
1 MDSs report slow metadata IOs
nodeep-scrub flag(s) set
9 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
Slow OSD heartbeats on back (longest 4313.409ms)
Slow OSD heartbeats on front (longest 4313.639ms)
Reduced data availability: 1 pg inactive, 1 pg incomplete
555 pgs not deep-scrubbed in time
436 pgs not scrubbed in time
50 slow ops, oldest one blocked for 73431 sec, osd.38 has slow opsservices:
mon: 2 daemons, quorum media,jht-cephdell (age 6s)
mgr: media.ehsygr(active, since 23h), standbys: jht-cephdell.kmzpjq
mds: 1/1 daemons up, 1 standby
osd: 73 osds: 63 up (since 26h), 63 in (since 20h); 33 remapped pgs
flags nodeep-scrubdata:
volumes: 1/1 healthy
pools: 11 pools, 1121 pgs
objects: 12.18M objects, 46 TiB
usage: 70 TiB used, 113 TiB / 183 TiB avail
pgs: 0.089% pgs not active
21/73076359 objects misplaced (0.000%)
1000 active+clean
87 active+clean+scrubbing
33 active+clean+remapped
1 incomplete
ceph health
HEALTH_WARN 1 OSD(s) have spurious read errors; Failed to apply 1 service(s): osd.all-available-devices; 9 failed cephadm daemon(s); 1 MDSs report slow metadata IOs; nodeep-scrub flag(s) set; 9 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; Slow OSD heartbeats on back (longest 3764.874ms); Slow OSD heartbeats on front (longest 3764.785ms); Reduced data availability: 1 pg inactive, 1 pg incomplete; 555 pgs not deep-scrubbed in time; 436 pgs not scrubbed in time; 50 slow ops, oldest one blocked for 73511 sec, osd.38 has slow ops
ceph health detail
HEALTH_WARN 1 OSD(s) have spurious read errors; 9 failed cephadm daemon(s); 1 MDSs report slow metadata IOs; nodeep-scrub flag(s) set; 9 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; Slow OSD heartbeats on back (longest 3793.376ms); Slow OSD heartbeats on front (longest 3793.425ms); Reduced data availability: 1 pg inactive, 1 pg incomplete; 555 pgs not deep-scrubbed in time; 436 pgs not scrubbed in time; 50 slow ops, oldest one blocked for 73576 sec, osd.38 has slow ops
[WRN] BLUESTORE_SPURIOUS_READ_ERRORS: 1 OSD(s) have spurious read errors
osd.48 reads with retries: 1
[WRN] CEPHADM_FAILED_DAEMON: 9 failed cephadm daemon(s)
daemon osd.47 on jht-cephdell is in error state
daemon osd.57 on jht-cephdell is in error state
daemon osd.58 on jht-cephdell is in error state
daemon osd.50 on jht-cephdell is in error state
daemon osd.45 on jht-cephdell is in error state
daemon osd.42 on media is in error state
daemon osd.26 on media is in error state
daemon osd.41 on media is in error state
daemon osd.17 on media is in error state
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.media.media.ylxstp(mds.0): 5 slow metadata IOs are blocked > 30 secs, oldest blocked for 93932 secs
[WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set
[WRN] OSD_FLAGS: 9 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
osd.17 has flags noin
osd.26 has flags noin
osd.41 has flags noin
osd.42 has flags noin
osd.45 has flags noin
osd.47 has flags noin
osd.50 has flags noin
osd.57 has flags noin
osd.58 has flags noin
[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 3793.376ms)
Slow OSD heartbeats on back from osd.68 to osd.18 3793.376 msec
Slow OSD heartbeats on back from osd.68 to osd.27 3792.848 msec
Slow OSD heartbeats on back from osd.68 to osd.16 3792.458 msec
Slow OSD heartbeats on back from osd.68 to osd.28 3792.123 msec
Slow OSD heartbeats on back from osd.68 to osd.13 3792.055 msec
Slow OSD heartbeats on back from osd.68 to osd.31 3791.863 msec
Slow OSD heartbeats on back from osd.68 to osd.29 3791.856 msec
Slow OSD heartbeats on back from osd.68 to osd.9 3791.812 msec
Slow OSD heartbeats on back from osd.68 to osd.25 3791.662 msec
Slow OSD heartbeats on back from osd.68 to osd.19 3791.626 msec
Truncated long network list. Use ceph daemon mgr.# dump_osd_network for more information
[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 3793.425ms)
Slow OSD heartbeats on front from osd.68 to osd.16 3793.425 msec
Slow OSD heartbeats on front from osd.68 to osd.23 3793.135 msec
Slow OSD heartbeats on front from osd.68 to osd.19 3793.059 msec
Slow OSD heartbeats on front from osd.68 to osd.27 3792.873 msec
Slow OSD heartbeats on front from osd.68 to osd.24 3792.858 msec
Slow OSD heartbeats on front from osd.68 to osd.9 3792.710 msec
Slow OSD heartbeats on front from osd.68 to osd.18 3792.440 msec
Slow OSD heartbeats on front from osd.68 to osd.21 3792.420 msec
Slow OSD heartbeats on front from osd.68 to osd.8 3791.944 msec
Slow OSD heartbeats on front from osd.68 to osd.25 3791.871 msec
Truncated long network list. Use ceph daemon mgr.# dump_osd_network for more information
[WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg incomplete
pg 2.1cc is incomplete, acting [38,29,40,37,63,16] (reducing pool media-data min_size from 5 may help; search ceph.com/docs for ‘incomplete’)
[WRN] PG_NOT_DEEP_SCRUBBED: 555 pgs not deep-scrubbed in time
pg 2.1ff not deep-scrubbed since 2024-11-10T07:15:14.934557+0000
pg 2.1fe not deep-scrubbed since 2024-12-08T20:35:12.724011+0000
pg 2.1fd not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1fb not deep-scrubbed since 2024-11-30T03:32:14.517753+0000
pg 2.1fa not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1f9 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1f8 not deep-scrubbed since 2024-11-10T15:19:24.170880+0000
pg 2.1f7 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1f6 not deep-scrubbed since 2024-12-08T09:27:01.569691+0000
pg 2.1f5 not deep-scrubbed since 2024-11-29T11:35:02.227626+0000
pg 2.1f4 not deep-scrubbed since 2024-12-09T03:35:50.174305+0000
pg 2.1f0 not deep-scrubbed since 2024-12-12T09:13:46.120178+0000
pg 2.1ef not deep-scrubbed since 2024-12-06T12:03:01.446200+0000
pg 2.1ee not deep-scrubbed since 2024-11-08T15:27:55.482302+0000
pg 2.1ec not deep-scrubbed since 2024-12-14T23:54:07.887213+0000
pg 2.1eb not deep-scrubbed since 2024-11-07T04:22:57.819296+0000
pg 2.1e9 not deep-scrubbed since 2024-12-05T03:38:40.948009+0000
pg 2.1e8 not deep-scrubbed since 2024-11-10T10:29:35.070459+0000
pg 2.1e5 not deep-scrubbed since 2024-12-12T23:01:47.689635+0000
pg 2.1e4 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1e3 not deep-scrubbed since 2024-11-23T11:15:27.008175+0000
pg 2.1e1 not deep-scrubbed since 2024-12-07T23:12:47.478812+0000
pg 2.1df not deep-scrubbed since 2024-11-29T20:46:34.242203+0000
pg 2.1da not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1d9 not deep-scrubbed since 2024-11-18T20:01:39.432379+0000
pg 2.1d8 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1d3 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1d1 not deep-scrubbed since 2024-12-05T12:34:02.311247+0000
pg 2.1d0 not deep-scrubbed since 2024-11-21T08:51:30.887620+0000
pg 2.1cd not deep-scrubbed since 2024-12-17T01:33:56.292956+0000
pg 2.1cc not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1ca not deep-scrubbed since 2024-12-02T11:03:15.380890+0000
pg 2.1c9 not deep-scrubbed since 2024-12-07T14:54:09.734351+0000
pg 2.1c6 not deep-scrubbed since 2024-12-05T13:24:10.590103+0000
pg 2.1c4 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1c2 not deep-scrubbed since 2024-12-16T17:32:14.591840+0000
pg 2.1c1 not deep-scrubbed since 2024-12-17T18:11:45.277188+0000
pg 2.1bf not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1be not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1bb not deep-scrubbed since 2024-12-14T23:21:35.917463+0000
pg 2.1b9 not deep-scrubbed since 2024-12-17T10:10:13.948590+0000
pg 2.1b7 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1b2 not deep-scrubbed since 2024-11-29T17:33:57.035093+0000
pg 2.1b0 not deep-scrubbed since 2024-11-03T05:39:10.523001+0000
pg 2.1ae not deep-scrubbed since 2024-11-10T16:30:36.309732+0000
pg 2.1ad not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
pg 2.1ab not deep-scrubbed since 2024-12-02T02:24:38.071992+0000
pg 2.1aa not deep-scrubbed since 2024-12-05T05:03:44.976744+0000
pg 2.1a6 not deep-scrubbed since 2024-12-01T22:05:01.613146+0000
pg 2.1a5 not deep-scrubbed since 2024-11-03T05:31:47.948994+0000
505 more pgs…
[WRN] PG_NOT_SCRUBBED: 436 pgs not scrubbed in time
pg 2.1ff not scrubbed since 2024-11-29T07:44:30.303371+0000
pg 2.1fb not scrubbed since 2024-11-30T03:32:14.517753+0000
pg 2.1fa not scrubbed since 2024-11-03T08:31:15.155094+0000
pg 2.1f9 not scrubbed since 2024-11-03T05:48:18.259665+0000
pg 2.1f8 not scrubbed since 2024-11-29T05:48:06.048767+0000
pg 2.1f7 not scrubbed since 2024-11-29T03:56:56.395087+0000
pg 2.1f6 not scrubbed since 2024-12-08T09:27:01.569691+0000
pg 2.1f5 not scrubbed since 2024-11-29T11:35:02.227626+0000
pg 2.1f4 not scrubbed since 2024-12-09T03:35:50.174305+0000
pg 2.1ef not scrubbed since 2024-12-06T12:03:01.446200+0000
pg 2.1ee not scrubbed since 2024-11-10T01:42:36.655590+0000
pg 2.1eb not scrubbed since 2024-11-08T15:39:13.847403+0000
pg 2.1e9 not scrubbed since 2024-12-05T03:38:40.948009+0000
pg 2.1e8 not scrubbed since 2024-11-10T10:29:35.070459+0000
pg 2.1e3 not scrubbed since 2024-11-29T20:58:50.198720+0000
pg 2.1e1 not scrubbed since 2024-12-07T23:12:47.478812+0000
pg 2.1df not scrubbed since 2024-11-29T20:46:34.242203+0000
pg 2.1da not scrubbed since 2024-11-29T06:46:14.280675+0000
pg 2.1d9 not scrubbed since 2024-11-28T19:50:48.998475+0000
pg 2.1d8 not scrubbed since 2024-11-03T05:45:44.570836+0000
pg 2.1d3 not scrubbed since 2024-11-28T19:45:45.832197+0000
pg 2.1d1 not scrubbed since 2024-12-05T12:34:02.311247+0000
pg 2.1d0 not scrubbed since 2024-11-28T22:10:34.527035+0000
pg 2.1cc not scrubbed since 2024-11-03T05:46:11.074973+0000
pg 2.1ca not scrubbed since 2024-12-06T15:27:53.231222+0000
pg 2.1c9 not scrubbed since 2024-12-07T14:54:09.734351+0000
pg 2.1c8 not scrubbed since 2024-12-18T19:41:01.691948+0000
pg 2.1c6 not scrubbed since 2024-12-05T13:24:10.590103+0000
pg 2.1c4 not scrubbed since 2024-11-03T05:46:42.031076+0000
pg 2.1c2 not scrubbed since 2024-12-16T17:32:14.591840+0000
pg 2.1c1 not scrubbed since 2024-12-17T18:11:45.277188+0000
pg 2.1bf not scrubbed since 2024-11-03T10:13:04.030353+0000
pg 2.1b7 not scrubbed since 2024-11-03T06:50:37.588436+0000
pg 2.1b6 not scrubbed since 2024-12-19T06:26:21.330149+0000
pg 2.1b4 not scrubbed since 2024-12-19T01:30:44.659791+0000
pg 2.1b2 not scrubbed since 2024-12-04T09:16:15.366882+0000
pg 2.1ae not scrubbed since 2024-11-10T16:30:36.309732+0000
pg 2.1ad not scrubbed since 2024-11-03T05:47:29.154766+0000
pg 2.1ab not scrubbed since 2024-12-02T02:24:38.071992+0000
pg 2.1aa not scrubbed since 2024-12-05T05:03:44.976744+0000
pg 2.1a6 not scrubbed since 2024-12-01T22:05:01.613146+0000
pg 2.1a4 not scrubbed since 2024-12-13T03:16:42.061633+0000
pg 2.1a3 not scrubbed since 2024-12-17T07:22:46.327358+0000
pg 2.1a0 not scrubbed since 2024-11-27T21:37:00.161755+0000
pg 2.19f not scrubbed since 2024-12-04T10:14:49.701659+0000
pg 2.19e not scrubbed since 2024-12-14T10:18:14.359903+0000
pg 2.19d not scrubbed since 2024-12-13T15:42:07.500026+0000
pg 2.19a not scrubbed since 2024-11-09T22:46:18.050535+0000
pg 2.199 not scrubbed since 2024-11-03T05:45:40.173818+0000
pg 2.194 not scrubbed since 2024-12-18T08:22:55.910777+0000
386 more pgs…
[WRN] SLOW_OPS: 50 slow ops, oldest one blocked for 73576 sec, osd.38 has slow ops
error log:
30/12/24 02:13 PM
[WRN]
Health check update: 50 slow ops, oldest one blocked for 73846 sec, osd.38 has slow ops (SLOW_OPS)
30/12/24 02:11 PM
[WRN]
Health check update: Failed to apply 2 service(s): osd.all-available-devices,osd.cost_capacity (CEPHADM_APPLY_SPEC_FAIL)
30/12/24 02:11 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:11 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:11 PM
[INF]
fs media: max_mds=1 standbys_required=1, count=230/12/24 02:11 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:11 PM
[WRN]
Health check update: 50 slow ops, oldest one blocked for 73711 sec, osd.38 has slow ops (SLOW_OPS)30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[INF]
fs media: max_mds=1 standbys_required=1, count=230/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
Health check update: 50 slow ops, oldest one blocked for 73706 sec, osd.38 has slow ops (SLOW_OPS)30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[INF]
fs media: max_mds=1 standbys_required=1, count=230/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
Health check update: 50 slow ops, oldest one blocked for 73701 sec, osd.38 has slow ops (SLOW_OPS)30/12/24 02:10 PM
[WRN]
Health check update: Slow OSD heartbeats on front (longest 3734.970ms) (OSD_SLOW_PING_TIME_FRONT)30/12/24 02:10 PM
[WRN]
Health check update: Slow OSD heartbeats on back (longest 3734.807ms) (OSD_SLOW_PING_TIME_BACK)30/12/24 02:10 PM
[INF]
fs media: max_mds=1 standbys_required=1, count=230/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[WRN]
Health check update: Failed to apply 2 service(s): osd.all-available-devices,osd.cost_capacity (CEPHADM_APPLY_SPEC_FAIL)30/12/24 02:10 PM
[WRN]
50 slow requests (by type [ ‘delayed’ : 50 ] most affected pool [ ‘media-data’ : 50 ])30/12/24 02:10 PM
[ERR]
Failed to apply osd.cost_capacity spec DriveGroupSpec.from_json(yaml.safe_load(‘’‘service_type: osd service_id: cost_capacity service_name: osd.cost_capacity placement: host_pattern: ‘*’ spec: data_devices: rotational: 1 filter_logic: AND objectstore: bluestore ‘’’)): cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/mon.jht-cephdell/config Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a -e NODE_NAME=jht-cephdell -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/run/ceph:z -v /var/log/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/log/ceph:z -v /var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsmmwsyzn:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_e3r3lai:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a lvm batch --no-auto /dev/sdo --yes --no-systemd /usr/bin/podman: stderr → passed data devices: 1 physical, 0 LVM /usr/bin/podman: stderr → relative data size: 1.0 /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /dev/sdo /usr/bin/podman: stderr stdout: Physical volume “/dev/sdo” successfully created. /usr/bin/podman: stderr stdout: Volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” successfully created /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 35003 -n osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /usr/bin/podman: stderr stdout: Logical volume “osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8” created. /usr/bin/podman: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 /usr/bin/podman: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 /usr/bin/podman: stderr Running command: /usr/bin/ln -s /dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /var/lib/ceph/osd/ceph-36/block /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-36/activate.monmap /usr/bin/podman: stderr stderr: got monmap epoch 8 /usr/bin/podman: stderr → Creating keyring file for osd.36 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/keyring /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/ /usr/bin/podman: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 36 --monmap /var/lib/ceph/osd/ceph-36/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-36/ --osd-uuid 5523a8d2-5dff-425b-b818-6cfb6bfb54a8 --setuser ceph --setgroup ceph /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.769+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.773+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.773+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.773+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36/) _read_fsid unparsable uuid /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.113+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.113+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.113+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 bdev(0x563bc8b74a80 /var/lib/ceph/osd/ceph-36//block) open open got: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 e[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-36/: (13) Permission deniede[0m /usr/bin/podman: stderr → Was unable to complete a new OSD, will rollback changes /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.36 --yes-i-really-mean-it /usr/bin/podman: stderr stderr: purged osd.36 /usr/bin/podman: stderr → Zapping: /dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /usr/bin/podman: stderr → Unmounting /var/lib/ceph/osd/ceph-36 /usr/bin/podman: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-36 /usr/bin/podman: stderr stderr: umount: /var/lib/ceph/osd/ceph-36 unmounted /usr/bin/podman: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 bs=1M count=10 conv=fsync /usr/bin/podman: stderr stderr: 10+0 records in /usr/bin/podman: stderr 10+0 records out /usr/bin/podman: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0654842 s, 160 MB/s /usr/bin/podman: stderr → Only 1 LV left in VG, will proceed to destroy volume group ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /usr/bin/podman: stderr stderr: Removing ceph–707c0142–6f6e–473d–8436–b6593d0f4f04-osd–block–5523a8d2–5dff–425b–b818–6cfb6bfb54a8 (254:3) /usr/bin/podman: stderr stderr: Releasing logical volume “osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8” /usr/bin/podman: stderr Archiving volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” metadata (seqno 5). /usr/bin/podman: stderr stdout: Logical volume “osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8” successfully removed. /usr/bin/podman: stderr stderr: Removing physical volume “/dev/sdo” from volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” /usr/bin/podman: stderr stdout: Volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” successfully removed /usr/bin/podman: stderr stderr: Creating volume group backup “/etc/lvm/backup/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” (seqno 6). /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdo /usr/bin/podman: stderr stdout: Labels on physical volume “/dev/sdo” successfully wiped. /usr/bin/podman: stderr → Zapping successful for OSD: 36 /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr File “/usr/sbin/ceph-volume”, line 33, in /usr/bin/podman: stderr sys.exit(load_entry_point(‘ceph-volume==1.0.0’, ‘console_scripts’, ‘ceph-volume’)()) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/main.py”, line 54, in init /usr/bin/podman: stderr self.main(self.argv) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 59, in newfunc /usr/bin/podman: stderr return f(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/main.py”, line 166, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/terminal.py”, line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/main.py”, line 46, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/terminal.py”, line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py”, line 419, in main /usr/bin/podman: stderr self._execute(plan) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py”, line 437, in _execute /usr/bin/podman: stderr c.create() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py”, line 26, in create /usr/bin/podman: stderr self.objectstore.safe_prepare() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py”, line 128, in safe_prepare /usr/bin/podman: stderr self.prepare() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py”, line 169, in prepare /usr/bin/podman: stderr self.osd_mkfs() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/objectstore/baseobjectstore.py”, line 150, in osd_mkfs /usr/bin/podman: stderr raise RuntimeError(‘Command failed with exit code %s: %s’ % /usr/bin/podman: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 36 --monmap /var/lib/ceph/osd/ceph-36/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-36/ --osd-uuid 5523a8d2-5dff-425b-b818-6cfb6bfb54a8 --setuser ceph --setgroup ceph Traceback (most recent call last): File “”, line 198, in _run_module_as_main File “”, line 88, in _run_code File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 5579, in File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 5567, in main File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 409, in _infer_config File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 324, in _infer_fsid File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 437, in _infer_image File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 311, in _validate_fsid File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 3312, in command_ceph_volume File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/call_wrappers.py”, line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a -e NODE_NAME=jht-cephdell -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/run/ceph:z -v /var/log/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/log/ceph:z -v /var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsmmwsyzn:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_e3r3lai:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a lvm batch --no-auto /dev/sdo --yes --no-systemd Traceback (most recent call last): File “/usr/share/ceph/mgr/cephadm/serve.py”, line 599, in _apply_all_services if self._apply_service(spec): File “/usr/share/ceph/mgr/cephadm/serve.py”, line 714, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File “/usr/share/ceph/mgr/cephadm/services/osd.py”, line 79, in create_from_spec ret = self.mgr.wait_async(all_hosts()) File “/usr/share/ceph/mgr/cephadm/module.py”, line 796, in wait_async return self.event_loop.get_result(coro, timeout) File “/usr/share/ceph/mgr/cephadm/ssh.py”, line 136, in get_result return future.result(timeout) File “/lib64/python3.9/concurrent/futures/_base.py”, line 446, in result return self.__get_result() File “/lib64/python3.9/concurrent/futures/_base.py”, line 391, in __get_result raise self._exception File “/usr/share/ceph/mgr/cephadm/services/osd.py”, line 76, in all_hosts return await gather(*futures) File “/usr/share/ceph/mgr/cephadm/services/osd.py”, line 63, in create_from_spec_one ret_msg = await self.create_single_host( File “/usr/share/ceph/mgr/cephadm/services/osd.py”, line 95, in create_single_host raise RuntimeError( RuntimeError: cephadm exited with an error code: 1, stderr:Inferring config /var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/mon.jht-cephdell/config Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a -e NODE_NAME=jht-cephdell -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/run/ceph:z -v /var/log/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/log/ceph:z -v /var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsmmwsyzn:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_e3r3lai:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a lvm batch --no-auto /dev/sdo --yes --no-systemd /usr/bin/podman: stderr → passed data devices: 1 physical, 0 LVM /usr/bin/podman: stderr → relative data size: 1.0 /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph-authtool --gen-print-key /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /dev/sdo /usr/bin/podman: stderr stdout: Physical volume “/dev/sdo” successfully created. /usr/bin/podman: stderr stdout: Volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” successfully created /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 35003 -n osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /usr/bin/podman: stderr stdout: Logical volume “osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8” created. /usr/bin/podman: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 /usr/bin/podman: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 /usr/bin/podman: stderr Running command: /usr/bin/ln -s /dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /var/lib/ceph/osd/ceph-36/block /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-36/activate.monmap /usr/bin/podman: stderr stderr: got monmap epoch 8 /usr/bin/podman: stderr → Creating keyring file for osd.36 /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/keyring /usr/bin/podman: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36/ /usr/bin/podman: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 36 --monmap /var/lib/ceph/osd/ceph-36/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-36/ --osd-uuid 5523a8d2-5dff-425b-b818-6cfb6bfb54a8 --setuser ceph --setgroup ceph /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.769+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.773+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.773+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label unable to decode label /var/lib/ceph/osd/ceph-36//block at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3] /usr/bin/podman: stderr stderr: 2024-12-30T22:10:30.773+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36/) _read_fsid unparsable uuid /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.109+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.113+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.113+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.113+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 bluestore(/var/lib/ceph/osd/ceph-36//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-36//block: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 bdev(0x563bc8b74a80 /var/lib/ceph/osd/ceph-36//block) open open got: (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied /usr/bin/podman: stderr stderr: 2024-12-30T22:10:31.593+0000 7feae5a58640 -1 e[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-36/: (13) Permission deniede[0m /usr/bin/podman: stderr → Was unable to complete a new OSD, will rollback changes /usr/bin/podman: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.36 --yes-i-really-mean-it /usr/bin/podman: stderr stderr: purged osd.36 /usr/bin/podman: stderr → Zapping: /dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 /usr/bin/podman: stderr → Unmounting /var/lib/ceph/osd/ceph-36 /usr/bin/podman: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-36 /usr/bin/podman: stderr stderr: umount: /var/lib/ceph/osd/ceph-36 unmounted /usr/bin/podman: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04/osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8 bs=1M count=10 conv=fsync /usr/bin/podman: stderr stderr: 10+0 records in /usr/bin/podman: stderr 10+0 records out /usr/bin/podman: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0654842 s, 160 MB/s /usr/bin/podman: stderr → Only 1 LV left in VG, will proceed to destroy volume group ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-707c0142-6f6e-473d-8436-b6593d0f4f04 /usr/bin/podman: stderr stderr: Removing ceph–707c0142–6f6e–473d–8436–b6593d0f4f04-osd–block–5523a8d2–5dff–425b–b818–6cfb6bfb54a8 (254:3) /usr/bin/podman: stderr stderr: Releasing logical volume “osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8” /usr/bin/podman: stderr Archiving volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” metadata (seqno 5). /usr/bin/podman: stderr stdout: Logical volume “osd-block-5523a8d2-5dff-425b-b818-6cfb6bfb54a8” successfully removed. /usr/bin/podman: stderr stderr: Removing physical volume “/dev/sdo” from volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” /usr/bin/podman: stderr stdout: Volume group “ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” successfully removed /usr/bin/podman: stderr stderr: Creating volume group backup “/etc/lvm/backup/ceph-707c0142-6f6e-473d-8436-b6593d0f4f04” (seqno 6). /usr/bin/podman: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/sdo /usr/bin/podman: stderr stdout: Labels on physical volume “/dev/sdo” successfully wiped. /usr/bin/podman: stderr → Zapping successful for OSD: 36 /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr File “/usr/sbin/ceph-volume”, line 33, in /usr/bin/podman: stderr sys.exit(load_entry_point(‘ceph-volume==1.0.0’, ‘console_scripts’, ‘ceph-volume’)()) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/main.py”, line 54, in init /usr/bin/podman: stderr self.main(self.argv) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 59, in newfunc /usr/bin/podman: stderr return f(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/main.py”, line 166, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/terminal.py”, line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/main.py”, line 46, in main /usr/bin/podman: stderr terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/terminal.py”, line 194, in dispatch /usr/bin/podman: stderr instance.main() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py”, line 419, in main /usr/bin/podman: stderr self._execute(plan) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py”, line 437, in _execute /usr/bin/podman: stderr c.create() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/create.py”, line 26, in create /usr/bin/podman: stderr self.objectstore.safe_prepare() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py”, line 128, in safe_prepare /usr/bin/podman: stderr self.prepare() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/decorators.py”, line 16, in is_root /usr/bin/podman: stderr return func(*a, **kw) /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/objectstore/lvmbluestore.py”, line 169, in prepare /usr/bin/podman: stderr self.osd_mkfs() /usr/bin/podman: stderr File “/usr/lib/python3.9/site-packages/ceph_volume/objectstore/baseobjectstore.py”, line 150, in osd_mkfs /usr/bin/podman: stderr raise RuntimeError(‘Command failed with exit code %s: %s’ % /usr/bin/podman: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 36 --monmap /var/lib/ceph/osd/ceph-36/activate.monmap --keyfile - --osdspec-affinity cost_capacity --osd-data /var/lib/ceph/osd/ceph-36/ --osd-uuid 5523a8d2-5dff-425b-b818-6cfb6bfb54a8 --setuser ceph --setgroup ceph Traceback (most recent call last): File “”, line 198, in _run_module_as_main File “”, line 88, in _run_code File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 5579, in File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 5567, in main File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 409, in _infer_config File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 324, in _infer_fsid File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 437, in _infer_image File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 311, in _validate_fsid File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/main.py”, line 3312, in command_ceph_volume File “/var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/cephadm.a58127a8eed242cae13849ddbebcb9931d7a5410f406f2d264e3b1ed31d9605e/cephadmlib/call_wrappers.py”, line 310, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a -e NODE_NAME=jht-cephdell -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/run/ceph:z -v /var/log/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8:/var/log/ceph:z -v /var/lib/ceph/ed18013c-9996-11ef-9ede-90e2ba62e3b8/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpsmmwsyzn:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_e3r3lai:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:200087c35811bf28e8a8073b15fa86c07cce85c575f1ccd62d1d6ddbfdc6770a lvm batch --no-auto /dev/sdo --yes --no-systemd