Fiberchannel target dies after large file transfer over samba

Software & Operating Systems Linux
,
1

Intro

So, i’m a masochistic bastard and decided to set up a fiberchannel target using a bunch of old 4gbps qlogic fiberchannel cards some years ago. It has gotten to the point where all my stuff boots off this one machine. There’s this RAID 0 array of 2x seagate 4TB 2.5" drives that all my stuff boots from. This is cached in NVMe drives too to make it spoody, and for the most part, it is. Before i get people harping on about how RAID 0 is madness, i back stuff up ok? Nothing on it is irreplaceable. Just would take forever to download again.

How stuff’s set up

Everything is forwarded to the clients using fileio, so i’ve got a bunch of raw disk images just sat on this storage that i boot from. The remainder of the space i access via samba and this is where i keep the steam library and a ton of other stuff. Works out kinda nicely, though this has some weird issues after i upgraded the machine.

The problem

Every time i write a massive file (over 20GB) via samba, the target disconnects, then doesn’t reconnect until i power cycle the server everything’s booting from or switch the port and reconfigure the client. Irritating as one may imagine, as it bluescreens the affected machine. Doesn’t do it when i copy a 1TB file from the target to another one though. Just writing to it.

Some diags already done

Initially i thought this was due to write back caching or something, so i set it to write through instead to no success. I also thought this would be a disk utilization problem, though as far as i can see, the disks barely ever hit 100% due to most of it being reads, and the cache in my case is pretty huge (128GB of RAM, 2TB of 500GB NVMe drives, long story on how i got that). LIO doesn’t really spit out any errors into the log from what i can see either.

The weird thing is, it’s just that port that dies on the qlogic card. The moment i move the machine to another port, everything springs back into life after a bit of reconfiguration in the bios on the client machine. What’s more bizzarre is how it knows which machine is doing the writing. I can have anything else running without an issue. I might be copying stuff from a USB drive to the samba share, not even touching the SAN, still only affects the one machine that’s doing the copying.

I know what you’re thinking. “not possible” right? Exactly what i thought, but here we are!

To do

I have some more diags to come. Trying a linux client and seeing if the same thing happens is on my to-do list. That’d rule out a windows driver issue which at this point and due to all the weirdness, is the only thing i think it could be. I’m tempted to see if it’s heat related too, this server is kind of a fire breather (has 4 CPUs, silly TDP too). Doesn’t explain the rather specific disconnection but i’ve ordered myself some K type probes to have a look anyway.

Target specs

-- General stuff --
Chassis - IBM X3850 X5
CPU - 4x Intel(R) Xeon(R) CPU E7- 4830  @ 2.13GHz (32 cores, 64 threads)
RAM - 128GB of 1333MHz ECC RAM. IBM branded.
OS -  Fedora 29
Graphics - An actual potato (Matrox something or other. Even with drivers, text mode environments perform poorly let alone video & graphics)

-- Storage --
 - Pair of 72GB 15K drives in md RAID 1 (boot storage)
 - Pair of ST4000LM024-2AN17V (4TB drives) in md RAID 0 (LVM Storage)
 - 4x Samsung 970 EVO 500GB drives in md RAID 0 (LVM Cache)

-- Expansion cards --
 - Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (x4) - What i'm using as the target
 - InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter - (Intend to set up SRP for this, kinda got it working, but loads more needs doing.)

Client Specs

-- General Stuff --
Mobo - ASRock Fatal1ty X79 Champion (aptly named)
CPU - Intel I7 3820
RAM - 16GB Corsair Vengeance @ 1600MHz
OS - Windoge 10 Pro 64 bit (Version 1903)
Graphics - 2x AMD Radeon R9 290 (not in crossfire due to another issue i've been having)

-- Storage --
The target above
A metric assload of 1TB drives that probably have 30 seconds before failure - don't actually use them other than for shits and giggles though.

Any further suggestions are welcome.

2 Likes
2

I haven’t played with fibre chanel myself, but if the port dies…?

mybe with activity / a bump, some actual technical people might step in

3

The Server has four 2 port FC HBAs ?

The client is using what HBA?

Are they connected via FC switch or directly?

Are you using multipathing software?

grep your logs for messages related to the driver module.

4

So it’s 2x 4 port HBAs in the target
Not running through a switch at the moment, but used to have them running through a brocade DS-4900B. This didn’t really change anything though. It’s all just running point to point. The cards in the desktop machines are the 2 port variants of the above.

As for the logs, the below around the time the target shuts its connection off, i see this 458 times in the log. This happened about 2 minutes into the file transfer.

Aug 05 20:54:07 kernel: TARGET_CORE[qla2xxx]: Expected Transfer Length: 0 does not match SCSI CDB Length: 1048576 for SAM Opcode: 0x2f
1 Like
5

Are they using the same brand/family of qlogic , and all are 4Gb?

6

Yuppers. The target uses QLE2464 (4 port) cards, the client machines use QLE2462 (2 port) cards.

7

My gut says drivers. What does dmesg say?

1 Like
8

Grepped dmesg

[   13.588276] qla2xxx [0000:00:00.0]-0005: : QLogic Fibre Channel HBA Driver: 10.00.00.14-k.
[   13.588661] qla2xxx [0000:8d:00.0]-001d: : Found an ISP2432 irq 46 iobase 0x(____ptrval____).
[   13.911344] scsi host3: qla2xxx
[   13.917282] qla2xxx [0000:8d:00.0]-00fb:3: QLogic QLE2464 - PCI-Express Quad Channel 4Gb Fibre Channel HBA.
[   13.917289] qla2xxx [0000:8d:00.0]-00fc:3: ISP2432: PCIe (2.5GT/s x4) @ 0000:8d:00.0 hdma+ host#=3 fw=8.07.00 (9496).
[   13.917786] qla2xxx [0000:8d:00.1]-001d: : Found an ISP2432 irq 48 iobase 0x(____ptrval____).
[   14.239292] scsi host4: qla2xxx
[   14.245093] qla2xxx [0000:8d:00.1]-00fb:4: QLogic QLE2464 - PCI-Express Quad Channel 4Gb Fibre Channel HBA.
[   14.245102] qla2xxx [0000:8d:00.1]-00fc:4: ISP2432: PCIe (2.5GT/s x4) @ 0000:8d:00.1 hdma+ host#=4 fw=8.07.00 (9496).
[   14.245589] qla2xxx [0000:8e:00.0]-001d: : Found an ISP2432 irq 48 iobase 0x(____ptrval____).
[   14.576037] scsi host5: qla2xxx
[   14.582004] qla2xxx [0000:8e:00.0]-00fb:5: QLogic QLE2464 - PCI-Express Quad Channel 4Gb Fibre Channel HBA.
[   14.582011] qla2xxx [0000:8e:00.0]-00fc:5: ISP2432: PCIe (2.5GT/s x4) @ 0000:8e:00.0 hdma+ host#=5 fw=8.07.00 (9496).
[   16.290962] qla2xxx [0000:8e:00.1]-001d: : Found an ISP2432 irq 247 iobase 0x(____ptrval____).
[   16.617360] scsi host6: qla2xxx
[   16.623170] qla2xxx [0000:8e:00.1]-00fb:6: QLogic QLE2464 - PCI-Express Quad Channel 4Gb Fibre Channel HBA.
[   16.623178] qla2xxx [0000:8e:00.1]-00fc:6: ISP2432: PCIe (2.5GT/s x4) @ 0000:8e:00.1 hdma+ host#=6 fw=8.07.00 (9496).
[   54.047725] qla2xxx [0000:8e:00.1]-00af:6: Performing ISP error recovery - ha=00000000ab658227.
[   74.622501] qla2xxx [0000:8e:00.1]-8038:6: Cable is unplugged...
[   75.552735] qla2xxx [0000:8d:00.0]-00af:3: Performing ISP error recovery - ha=00000000d4253857.
[   76.354488] qla2xxx [0000:8d:00.0]-500a:3: LOOP UP detected (4 Gbps).
[   76.366527] Modules linked in: tcm_qla2xxx nf_conntrack_netbios_ns nf_conntrack_broadcast xt_CT ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat team_mode_loadbalance ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat team iptable_mangle iptable_raw iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 target_core_user uio target_core_pscsi target_core_file target_core_iblock ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables btrfs ib_isert iscsi_target_mod xor zstd_compress raid6_pq zstd_decompress ib_srpt rpcrdma target_core_mod ib_srp sunrpc scsi_transport_srp rdma_ucm ib_iser ib_umad rdma_cm libiscsi ib_ipoib scsi_transport_iscsi ib_cm iw_cm vfat fat mlx4_ib ib_uverbs ib_core intel_powerclamp coretemp kvm_intel dm_cache_smq kvm dm_cache dm_persistent_data irqbypass dm_bio_prison crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_cstate intel_uncore raid0 iTCO_wdt
[   76.366574]  gpio_ich iTCO_vendor_support ipmi_ssif i2c_i801 lpc_ich ioatdma i7core_edac ipmi_si dca ipmi_devintf pcc_cpufreq ipmi_msghandler acpi_cpufreq xfs libcrc32c mlx4_en mgag200 i2c_algo_bit drm_kms_helper qla2xxx ttm wmi nvme_fc ata_generic nvme_fabrics cdc_ether usbnet nvme drm crc32c_intel mlx4_core scsi_transport_fc pata_acpi megaraid_sas mii nvme_core bnx2
[   76.366638] Workqueue: qla2xxx_wq qla2x00_iocb_work_fn [qla2xxx]
[   76.366723]  qla24xx_do_nack_work+0x5e/0xd0 [qla2xxx]
[   76.366744]  qla2x00_do_work+0x348/0x9e0 [qla2xxx]
[   76.366775]  qla2x00_iocb_work_fn+0x42/0x80 [qla2xxx]
[   76.367587] qla2xxx [0000:8d:00.0]-d034:3: qla24xx_do_nack_work create sess success 00000000e94279cc
[   76.571837] qla2xxx [0000:8d:00.1]-00af:4: Performing ISP error recovery - ha=00000000b47c8e52.
[   97.150483] qla2xxx [0000:8d:00.1]-8038:4: Cable is unplugged...
[   98.075840] qla2xxx [0000:8e:00.0]-00af:5: Performing ISP error recovery - ha=00000000ecd25ccc.
[  101.536225] qla2xxx [0000:8d:00.0]-500b:3: LOOP DOWN detected (2 5 0 0).
[  101.952314] qla2xxx [0000:8d:00.0]-500a:3: LOOP UP detected (4 Gbps).
[  101.953299] qla2xxx [0000:8d:00.0]-d034:3: qla24xx_do_nack_work create sess success 00000000e94279cc
[  118.662313] qla2xxx [0000:8e:00.0]-8038:5: Cable is unplugged...

The error recovery message looks a tad worrying, though i see reports of this being normal when enabling/disabling target mode.

9

Don’t know if you get notified about this but i edited my posts above with the deets.

Imma run a test where i record the log outputs while the file transfer is running so i can see exactly what stages everything occurs.

10

Change the cable?

2 Likes
11

Id try cables, or blowing out dust… or even a bad card/port?

This is all that shows up in my client’s dmesg

[ 1.265646] qla2xxx [0000:00:00.0]-0005: : QLogic Fibre Channel HBA Driver: 10.00.00.12-k.
[ 1.265708] qla2xxx [0000:01:00.0]-001d: : Found an ISP2432 irq 16 iobase 0x00000000e1411bee.
[ 1.671970] scsi host0: qla2xxx
[ 1.672883] qla2xxx [0000:01:00.0]-00fb:0: QLogic QLE2462 - PCI-Express Dual Channel 4Gb Fibre Channel HBA.
[ 1.672887] qla2xxx [0000:01:00.0]-00fc:0: ISP2432: PCIe (2.5GT/s x4) @ 0000:01:00.0 hdma+ host#=0 fw=8.07.00 (9496).
[ 1.672984] qla2xxx [0000:01:00.1]-001d: : Found an ISP2432 irq 17 iobase 0x000000003354f893.
[ 1.933311] qla2xxx [0000:01:00.0]-500a:0: LOOP UP detected (4 Gbps).
[ 2.080074] scsi host7: qla2xxx
[ 2.081126] qla2xxx [0000:01:00.1]-00fb:7: QLogic QLE2462 - PCI-Express Dual Channel 4Gb Fibre Channel HBA.
[ 2.081145] qla2xxx [0000:01:00.1]-00fc:7: ISP2432: PCIe (2.5GT/s x4) @ 0000:01:00.1 hdma+ host#=7 fw=8.07.00 (9496).
[ 2.341406] qla2xxx [0000:01:00.1]-500a:7: LOOP UP detected (4 Gbps).

12

My feeling is that if those errors aren’t being specifically found by googling, I’d start playing with cables and ports.

13

Ok, so i measured the card temps with a thermocouple. Die temps on these cards is like 40 C idle. Airflow in the I/O cage is warm at 33 C ambient temp. The temp in the room this thing occupies is 18 C

I should have also mentioned the above is the target dmesg. Haven’t actually gotten around to installing fedora onto a test LUN to play with. Will probs do tomorrow night.

Will conduct a load test tomorrow afternoon along with recording exactly when the errors come up during a file copy. If it isn’t immediate, i’m thinking heat is my issue.

I forgot to mention earlier but the cards had been swapped out before, located in a different PCIe slot to no effect. I’ll change the cables cause it’s easiest to do, though this happens on multiple clients. A while back one of the ports shut off, i located the cable in the next available port (target side), just to see it happen again in that too. I did unplug everything and replug it all back in a week or so ago. Cleaned everything in the process. Seemed to work okay until i downloaded a large game directly to the NAS.

14

Are these logs taken with extended logging? If not, google ql2xextended_error_logging and reproduce the error (enable it on both target and initiator).
Edit: it’s not “boot from SAN” configuration, I hope?

15

It is boot from SAN config, just enabled extended error logging as a kernel module option now.

16

Turns out, with extended error logging enabled (i assume set to 1 means enabled?)
qla2xxx does not produce a single error during the transfer. It’s only when the machine shuts down or starts up that the errors occur. Every time the system boots, 4x SCSI commands go through that the adapter doesn’t recognise and this causes those errors. This is only from a cold boot too and it occurs as the adapter is initialising while in the OPROM menu. No issue is actually observed on the client other than the errors in the log. Everything just functions as intended.

Load tests done, die temps on the cards were monitored and logged, 42C was the highest things got so i doubt it’s that.

Didn’t think disk performance would be a thing and i did test this before, though perhaps my methods were bad. Could write to the disks at well over 200MBps over the network (from the clients perspective).
iostat -Nh 1 on the target showed the disks writing at a slower speed than things going to it over samba.
Ran dd if=/dev/zero | pv | dd of=/path/to/share/filename
pv showed a transfer rate way higher than what the disks were actually doing.

Changed how i mounted the disks in /etc/fstab to disable buffering/caching.
Turns out the disks are just slow as balls for writing. 30MBps kind of slow as balls.
Reads are great, does that no problem. I just had so much buffering/cache due to having so much RAM in this thing i didn’t notice any problems until about 20 - 30GB into a transfer.

This still doesn’t explain though why the ports basically shut down and don’t come back up until the target is rebooted for windows machines. The target doesn’t even acknowledge this is happening. Without grepping the log at all, the only things shown in there are messages about authentication when i sudo a command. It’s understandable it might shut off the port if the machine isn’t responding to disk reads/writes though, but it would be nice to know exactly what mechanism does this.

A further test i did earlier today showed that it’s only machines that are actively being used that end up getting disconnected. Though, with a Fedora 30 client, it recovers and the port doesn’t get disabled once the transfer finished. It just sits there not letting you do anything that requires an ounce of disk activity.

I will start my windows client back up and see if i can reproduce and watch the log as the machine reboots. I haven’t the slightest clue how fiberchannel actually works, though i have a feeling there’s some session that is created that isn’t getting reset due to the bluescreen not shutting the adapter down properly.

Will have a further poke and prod with a sharp stick on this one.

17

OK, let’s talk topology.
What acts as initiators in your SAN? Is it this Win10 machine, or do you have a bunch of VMs and you use passthrough to provide those with HBAs? If it’s a Win10, then yikes, BfSAN without MPIO (unless you got WS2016 or 2019 libraries somewhere)?

So what I think happens is client exhausts server’s memory and hits a raw write speed (and since drives are busy writing whatever comes in over usual network, IO latency becomes unmanageable). Client OS shits itself (if it’s Windows) or becomes temporary unresponsive (if it’s Linux). Then on target’s side it becomes an issue of either hanging scsi reservations (no idea if you’re using it, knowingly or otherwise, it’s basically a mechanism of concurrent access which keeps track of LUN usage and tells initiator to sod off if the old lock is still in place) or QLogic’s shenanigans (they tend to completely freeze on activity timeout, seen this crap on IBM servers way too many times).
Have you tried LIP reset and/or bus reset instead of reboot? Enabling stuff like full login and target reset in HBA BIOS might help as well.

1 Like
18

The Client machines are Windows 10
The disks they boot from are targetcli’s fileio backstore running on Fedora 29. One is 1TB in size, the other 250GB.
Memory doesn’t necessarily get “exhausted” as there’s 128GB and every time i’m logged into the thing, there’s only like 1GB of memory being used unless i’m running something from ramdisk or have VMs running. As far as i know, linux OSes tend to use unused memory for disk buffering.

So as far as i can tell, nothing really happens that would write that much to memory as the problems occur abut 25GB into the transfer. This may just be some buffer flushing setting somewhere though.

19

whats the output of : targetcli ls

20

This config was cleaner a week ago, but i truncated that GuestPC disk from 250GB to 1TB and added it back in and forgot to not get it to auto map them back in, so everything kinda just has permissions for everything at the moment. The issue occurred before this though. Also, ignore the srpt stuff, never got that working.

o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 4]
  | | o- DesktopPC ............................................. [/mnt/nas/imagefiles/LUN/DesktopPC (250.0GiB) write-back activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- Fedora ................................................ [/mnt/nas/imagefiles/LUN/Fedora.img (40.0GiB) write-back activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- FedoraInstall .......... [/mnt/nas/imagefiles/Installers/Fedora-Server-dvd-x86_64-30-1.2.iso (3.0GiB) write-back activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- GuestPC ................................................... [/mnt/nas/imagefiles/LUN/GuestPC (1.0TiB) write-back activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 2]
  |   o- DesktopRD ........................................................................................... [(20.0GiB) activated]
  |   | o- alua ................................................................................................... [ALUA Groups: 1]
  |   |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  |   o- GuestRD ............................................................................................. [(20.0GiB) activated]
  |     o- alua ................................................................................................... [ALUA Groups: 1]
  |       o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]
  o- qla2xxx .......................................................................................................... [Targets: 4]
  | o- naa.21000024ff6029bc ............................................................................................. [gen-acls]
  | | o- acls ............................................................................................................ [ACLs: 3]
  | | | o- naa.500143800636c784 ................................................................................... [Mapped LUNs: 4]
  | | | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  | | | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | | o- naa.500143800636c60c ................................................................................... [Mapped LUNs: 4]
  | | | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  | | | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | | o- naa.500143800636c78c ................................................................................... [Mapped LUNs: 4]
  | | |   o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  | | |   o- mapped_lun1 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  | | |   o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | |   o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | o- luns ............................................................................................................ [LUNs: 6]
  | |   o- lun0 .......................................... [fileio/DesktopPC (/mnt/nas/imagefiles/LUN/DesktopPC) (default_tg_pt_gp)]
  | |   o- lun1 .............................................. [fileio/GuestPC (/mnt/nas/imagefiles/LUN/GuestPC) (default_tg_pt_gp)]
  | |   o- lun2 ............................................................................. [ramdisk/DesktopRD (default_tg_pt_gp)]
  | |   o- lun3 ............................................................................... [ramdisk/GuestRD (default_tg_pt_gp)]
  | |   o- lun4 ............................................ [fileio/Fedora (/mnt/nas/imagefiles/LUN/Fedora.img) (default_tg_pt_gp)]
  | |   o- lun5 ..... [fileio/FedoraInstall (/mnt/nas/imagefiles/Installers/Fedora-Server-dvd-x86_64-30-1.2.iso) (default_tg_pt_gp)]
  | o- naa.21000024ff6029bd ............................................................................................. [gen-acls]
  | | o- acls ............................................................................................................ [ACLs: 3]
  | | | o- naa.500143800636c784 ................................................................................... [Mapped LUNs: 4]
  | | | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  | | | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | | o- naa.500143800636c60c ................................................................................... [Mapped LUNs: 4]
  | | | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  | | | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | | o- naa.500143800636c78c ................................................................................... [Mapped LUNs: 4]
  | | |   o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  | | |   o- mapped_lun1 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  | | |   o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | |   o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | o- luns ............................................................................................................ [LUNs: 6]
  | |   o- lun0 .......................................... [fileio/DesktopPC (/mnt/nas/imagefiles/LUN/DesktopPC) (default_tg_pt_gp)]
  | |   o- lun1 .............................................. [fileio/GuestPC (/mnt/nas/imagefiles/LUN/GuestPC) (default_tg_pt_gp)]
  | |   o- lun2 ............................................................................. [ramdisk/DesktopRD (default_tg_pt_gp)]
  | |   o- lun3 ............................................................................... [ramdisk/GuestRD (default_tg_pt_gp)]
  | |   o- lun4 ............................................ [fileio/Fedora (/mnt/nas/imagefiles/LUN/Fedora.img) (default_tg_pt_gp)]
  | |   o- lun5 ..... [fileio/FedoraInstall (/mnt/nas/imagefiles/Installers/Fedora-Server-dvd-x86_64-30-1.2.iso) (default_tg_pt_gp)]
  | o- naa.21000024ff6029be ............................................................................................. [gen-acls]
  | | o- acls ............................................................................................................ [ACLs: 3]
  | | | o- naa.500143800636c784 ................................................................................... [Mapped LUNs: 4]
  | | | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  | | | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | | o- naa.500143800636c60c ................................................................................... [Mapped LUNs: 4]
  | | | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  | | | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | | o- naa.500143800636c78c ................................................................................... [Mapped LUNs: 4]
  | | |   o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  | | |   o- mapped_lun1 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  | | |   o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  | | |   o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  | | o- luns ............................................................................................................ [LUNs: 6]
  | |   o- lun0 .......................................... [fileio/DesktopPC (/mnt/nas/imagefiles/LUN/DesktopPC) (default_tg_pt_gp)]
  | |   o- lun1 .............................................. [fileio/GuestPC (/mnt/nas/imagefiles/LUN/GuestPC) (default_tg_pt_gp)]
  | |   o- lun2 ............................................................................. [ramdisk/DesktopRD (default_tg_pt_gp)]
  | |   o- lun3 ............................................................................... [ramdisk/GuestRD (default_tg_pt_gp)]
  | |   o- lun4 ............................................ [fileio/Fedora (/mnt/nas/imagefiles/LUN/Fedora.img) (default_tg_pt_gp)]
  | |   o- lun5 ..... [fileio/FedoraInstall (/mnt/nas/imagefiles/Installers/Fedora-Server-dvd-x86_64-30-1.2.iso) (default_tg_pt_gp)]
  | o- naa.21000024ff6029bf ............................................................................................. [gen-acls]
  |   o- acls ............................................................................................................ [ACLs: 3]
  |   | o- naa.500143800636c784 ................................................................................... [Mapped LUNs: 4]
  |   | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  |   | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  |   | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  |   | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  |   | o- naa.500143800636c60c ................................................................................... [Mapped LUNs: 4]
  |   | | o- mapped_lun0 ................................................................................ [lun1 fileio/GuestPC (rw)]
  |   | | o- mapped_lun1 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  |   | | o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  |   | | o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  |   | o- naa.500143800636c78c ................................................................................... [Mapped LUNs: 4]
  |   |   o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  |   |   o- mapped_lun1 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  |   |   o- mapped_lun2 ................................................................................. [lun4 fileio/Fedora (rw)]
  |   |   o- mapped_lun3 .......................................................................... [lun5 fileio/FedoraInstall (rw)]
  |   o- luns ............................................................................................................ [LUNs: 6]
  |     o- lun0 .......................................... [fileio/DesktopPC (/mnt/nas/imagefiles/LUN/DesktopPC) (default_tg_pt_gp)]
  |     o- lun1 .............................................. [fileio/GuestPC (/mnt/nas/imagefiles/LUN/GuestPC) (default_tg_pt_gp)]
  |     o- lun2 ............................................................................. [ramdisk/DesktopRD (default_tg_pt_gp)]
  |     o- lun3 ............................................................................... [ramdisk/GuestRD (default_tg_pt_gp)]
  |     o- lun4 ............................................ [fileio/Fedora (/mnt/nas/imagefiles/LUN/Fedora.img) (default_tg_pt_gp)]
  |     o- lun5 ..... [fileio/FedoraInstall (/mnt/nas/imagefiles/Installers/Fedora-Server-dvd-x86_64-30-1.2.iso) (default_tg_pt_gp)]
  o- srpt ............................................................................................................. [Targets: 2]
  | o- ib.fe800000000000000010e0000132c349 ........................................................................... [no-gen-acls]
  | | o- acls ............................................................................................................ [ACLs: 2]
  | | | o- ib.fe800000000000000010e0000133bbbd .................................................................... [Mapped LUNs: 3]
  | | | | o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  | | | | o- mapped_lun2 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  | | | | o- mapped_lun3 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | | o- ib.fe800000000000000010e0000133bbbe .................................................................... [Mapped LUNs: 3]
  | | |   o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  | | |   o- mapped_lun2 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  | | |   o- mapped_lun3 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  | | o- luns ............................................................................................................ [LUNs: 3]
  | |   o- lun0 .......................................... [fileio/DesktopPC (/mnt/nas/imagefiles/LUN/DesktopPC) (default_tg_pt_gp)]
  | |   o- lun2 ............................................................................. [ramdisk/DesktopRD (default_tg_pt_gp)]
  | |   o- lun3 ............................................................................... [ramdisk/GuestRD (default_tg_pt_gp)]
  | o- ib.fe800000000000000010e0000132c34a ........................................................................... [no-gen-acls]
  |   o- acls ............................................................................................................ [ACLs: 2]
  |   | o- ib.fe800000000000000010e0000133bbbd .................................................................... [Mapped LUNs: 3]
  |   | | o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  |   | | o- mapped_lun2 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  |   | | o- mapped_lun3 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  |   | o- ib.fe800000000000000010e0000133bbbe .................................................................... [Mapped LUNs: 3]
  |   |   o- mapped_lun0 .............................................................................. [lun0 fileio/DesktopPC (rw)]
  |   |   o- mapped_lun2 ............................................................................. [lun2 ramdisk/DesktopRD (rw)]
  |   |   o- mapped_lun3 ............................................................................... [lun3 ramdisk/GuestRD (rw)]
  |   o- luns ............................................................................................................ [LUNs: 3]
  |     o- lun0 .......................................... [fileio/DesktopPC (/mnt/nas/imagefiles/LUN/DesktopPC) (default_tg_pt_gp)]
  |     o- lun2 ............................................................................. [ramdisk/DesktopRD (default_tg_pt_gp)]
  |     o- lun3 ............................................................................... [ramdisk/GuestRD (default_tg_pt_gp)]
  o- vhost ............................................................................................................ [Targets: 0]