Fedora DIY FC SAN

Community Blog
,
1

TLDR
Was hoping to setup this on CentOS 7.4 since I already have it setup … but only seems to work on Fedora currently.

Will try to get pictures added later.

#############

Links

#############


http://linux-iscsi.org/wiki/Fibre_Channel


https://bugs.launchpad.net/ubuntu/+source/targetcli/+bug/1390158
https://www.spinics.net/lists/target-devel/msg07910.html

#############

My thoughts

#############

Running Fedora 26… working great!!!
Using 4Gb FC HBA qla2462 model

If I cannot get a decent price on the 2562 model… I can just use multipathd to increase the throughput between the server and client.

The down side is all the work I have done with zfs on CentOS7 is for naught. But I look forward to seeing if I can import my zfs pool easily and just re-export the zvols using the FC HBA.

Excited!!

5 Likes
2

Spoiler alert…

You dont need the FC switch… you can go point to point.

# echo 'options qla2xxx qlini_mode="disabled"' > /usr/lib/modprobe.d/qla2xxx.conf
# rmmod qla2xxx
# modprobe qla2xxx



# yum install -y targetcli
# targetcli
/> qla2xxx/ info
Fabric module name: qla2xxx
ConfigFS path: /sys/kernel/config/target/qla2xxx
Allowed WWN types: naa
Allowed WWNs list: naa.210000e08b943494
Fabric module features: acls
Corresponding kernel module: tcm_qla2xxx
3

Found some cheap dual port 4Gb cards on newegg… got a couple.

The cards I used at work were oem dell ; the ones from newegg seem to be oem cisco. I dont expect any issues.

4

Still testing it out at work some while waiting for the stuff for my home lab to come in…but it definitely looks like SCSI over FC uses less CPU than iSCSI.

5

Cards here…

Putting in some work…

Got the card into target mode…
image

image

Got the two target wwns created in targetcli

image

Working on setting up multipathd on my proxmox server. Then I’ll configure the luns from my ZFS pool.

image

tbc…

6

Working so far…

image

7

Definitely working

image

image

8

Built a VM on proxmox by passing the lun thru.
Gave it two cpu threads…

Did some quick DD testing…

read
image
image

write
image

not to bad for zfs on linux using raidz on 2.5" drives.

3 Likes
9

To be able to use multipath and balance IO over both FC ports…(if I had more PCIe slots… I could install another FC card and raise the bandwidth potential to 16Gb/s, not that I have enough disk to reach those speeds) … I just setup each lun under each target.

image
image.png1063x411 11.9 KB

10

Figured out that putting both client wwns under both target wwns really helps if you never unplug and dont have the cables labeled.

image
image.png1069x490 18 KB

11

Any chance you might have run into the error “Could not create Target in configFS”? No matter what I do, I can’t get it to create. I have a lot of time in this but can’t get past it. It won’t let me post the Github link so I’ll try to copy and paste from there.

I installed targetcli via yum on CentOS 7 and this is the version in the repo:
# targetcli
targetcli shell version 2.1.fb49
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

I downloaded the .zip of the targetcli-fb version and tried it too from the scripts folder and same results.

Python:
# python --version
Python 2.7.5

Libs:
Installed Packages
python-configshell.noarch 1:1.1.fb25-1.el7 @base
python-rtslib.noarch 2.1.fb69-3.el7 @base
python-rtslib-doc.noarch 2.1.fb69-3.el7 @base

The error comes up when using create.
/> /qla2xxx create naa.xxxxxxxxxxxxxxx
Could not create Target in configFS

Below is the info on my FC
/> qla2xxx/ info
Fabric module name: qla2xxx
ConfigFS path: /sys/kernel/config/target/qla2xxx
Allowed WWN types: naa
Allowed WWNs list: naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx, naa.xxxxxxxxxxxxxxx
Fabric module features: acls
Corresponding kernel module: tcm_qla2xxx

The log file doesn’t show anything that explains what’s happening.

I checked to ensure that I have initiator mode disabled

# cat /sys/module/qla2xxx/parameters/qlini_mode
disabled

The kernel modules are loaded.
# lsmod | grep qla2xxx
tcm_qla2xxx 32768 1
target_core_mod 335872 4 tcm_qla2xxx,iscsi_target_mod
qla2xxx 634880 1 tcm_qla2xxx
scsi_transport_fc 65536 3 bfa,qla2xxx,tcm_qla2xxx

configfs is mounted:
# cat /proc/mounts | grep configfs
configfs /sys/kernel/config configfs rw,relatime 0 0

In the kernel config:
CONFIG_CONFIGFS_FS=y
CONFIG_TCM_QLA2XXX=y

Kernel:
4.4.207-1.el7.elrepo.x86_64

Should I be able to create a directory in /sys/kernel/config/target/qla2xxxx?
# mkdir naa.10008c7cffc7ef01
mkdir: cannot create directory ‘naa.xxxxxxxxxxxxxxx’: Invalid argument

I did notice that when I rebooted, there was no qla2xxx folder in /sys/kernel/config/target/ but after trying to use targetcli, it did make one so I’m not sure what else would cause it to fail.

Suggestions?

12

why did you get it from a zip?

why not install targetcli from the repo?

also,

There something the kernel is missing in Centos, so doing it on fedora is the way to go (IMO)

edit: seems like you may have taken care of the missing kernel parameters…

13

is the targetd service running?

14

Yes, I first tried the version in the repo. After trying for so long with that version and getting nowhere, Googling suggested trying a different version called targetcli-fb. It’s on Github but I can’t post the URL here. You can find it by searching for it by that name. I had no luck with that either.

I tried to make it work the latest kernel-3.10.0-1062.9.1.el7.x86_64 but ran into issues loading qla2xxxx. Modprobe returned an invalid parameter error but when using -f it say it’s an invalid key. Googling for that suggested it wasn’t signed. I didn’t want to deal with that so I found the elrepo.repo and it had the kernel-lt-4.4.207-1.elrepo.x86_64. That was the one that got me the closest. Kernel-ml-5.4.7-1.el7.elrepo.x86_64 had a bunch of weird errors so I reverted back to 4.4 That has me the closest to getting it to work. I know Fedora is an upstream version of RHEL/CentOS but I haven’t done much at all with it. I can wipe this and install CentOS 8 which may have better results. I just hate to abandon the progress so far. I can’t find any logs anywhere to give any indication as to why it fails. I have the newer 16gb cards which are supposed to be supported in the kernel natively now.

15

I thought I had. Target is there but not targetd.
# systemctl status target
● target.service - Restore LIO kernel target configuration
Loaded: loaded (/usr/lib/systemd/system/target.service; enabled; vendor preset: disabled)
Active: active (exited) since Thu 2020-01-02 23:04:36 MST; 12h ago
Process: 4370 ExecStart=/usr/bin/targetctl restore (code=exited, status=0/SUCCESS)
Main PID: 4370 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/target.service

Jan 02 23:04:36 xxx.tld systemd[1]: Starting Restore LIO kernel target configuration...
Jan 02 23:04:36 xxx.tld systemd[1]: Started Restore LIO kernel target configuration.

I just installed targetd and still, same error Could not create Target in configFS
Installed: libstoragemgmt-targetd-plugin.noarch 0:1.7.3-3.el7 targetd.noarch 0:0.8.6-1.el7

16

Ive setup several systems using Fedora since version 26 (the latest one I setup has been on Fedora30), I havent had any issues, I havent tried, nor do i have any 16Gb HBAs, but i do have and used 4Gb and 8Gb HBAs.

If nothing else, if you can do it on Fedora and get it working, you’ll have something to compare it against.

17

I think thats what the older daemon version was called before LIO

18

Well, target and targetd are in and running. No change. Anything you came across that contains any valid explanation of the errors or a means of debugging the issue?

19

if lsmod and modinfo show the correct modules are loaded , then maybe you could try a 8Gb HBA to see if it will work?

20

wait a min…

are you not using the targetcli command to setup your exports?