I have an existing ZFS mirrored zvol (4 x 3TB) running on Ubuntu 16-ish which is served to a win-10 desktop via samba. It is used for backups and long-term storage and anything else that I can’t have disappear on me. 10gbe LAN network if it matters.
I was RTFMing and I noticed that PrimoCache (I use it on that win-10 box) can cache an ISCSI target locally to my NVME which now has me intrigued.
What would be the simplest/quickest way to use the ZFS storage space to create an ISCSI target for playing with the PrimoCache?
Assuming that the local caching of the target works well, is there an alternate “fail-resistant” way to store the data and provide an ISCSI target with better performance?
You’ll have to create a disk image or a zvol and export that via iscsi. I’m not sure if ZFS on linux has built in iscsi target functionality but you can use the iscsitarget package on ubuntu. It’s fairly straight forward and you should be able to find a guide for how to set it up. On windows you use the iscsi initiator to connect to it, which is pretty intuitive to use.
That was kinda what I found, but would raid 5 or something be a better source for the ISCSI? What about cluster/sector/packet sizes or alignment?
Creating a file on a zvol (of a fixed size?) and then making it a target for ISCSI seems horrifically wasteful. Ok for a test, I guess, but sorta kills the whole point of the zfs flexibility. The pre-allocation of a multi-terrabyte empty file makes my skin crawl.
Also, I currently point steam at the network share by default, and then I use freefilesync to clone the game of the month to my SSD which gets cached to my NVME m.2 drive. This prevents any game updates from blind-siding me on the actual game that I’m currently playing, but it’s easy to sync any updates that I do want.
@Dexter_Kane, I just read the whole thread you mentioned, and clearly we were on the same wavelength. Thanks for the thread, and thanks for taking the time to point it out. I missed it in my search prior to posting.
On my 10gbe samba share, ATTO says I get between 600 MB/s and 5.6 GB/s depending on file size, direct I/O, 2GB or 32GB file, 256K or 4MB I/O blocks etc. In real life, games or installers usually get 80-190 MB/s with games on the low end. The bigger the files, the faster it copies when tossing files around. Even when the SSD is the destination, windows writes are criminally slow.
When I first read your 80MB story, my heart sank, but knowing it was much faster on 10gbe, I think I’ll have a go at it. It’s nearly the same as what you did already, so I have a fair chance to get it working.
You still get all the advantages of ZFS just not at a file system level. I prefer using a large image file rather than a ZVOL as a ZVOL becomes dedicated so you can’t use that space for anything else, an image file will be all zeros so it will compress to an empty file and fill as you use space.
RAID would probably be faster but with a cache on the other end the speed doesn’t really matter. That’s the main advantage I would say of using iscsi rather than samba, although iscsi by itself will be more efficient than samba anyway.
I upgraded to ubuntu 18.04 LTS and found that version has an unreliable driver for my 10gbe card. I got 19.04 (not LTS) working, so that will do for now. I also reconfigured the mirror zvols to a raidz instead. The performance of the mirror setup was disappointing, the raidz is working well.
I installed the ISCSI target via the file method, but it took many hours to DD a 2Tb file of zeros. It may be that zfs did NOT have compression enabled by default. It’s on now, but that might be closing the gate after the horse took a walk. I have plenty of space right now, so it doesn’t matter. And to answer your question, zfs on Linux does not natively support ISCSI targets, but tgt worked fine.
I’m getting a solid 290 MB/s on 200+ GB transfers, and non-cached bursts of 450+ MB/s on files of only a few hundred MB. When the cache kicks in, it’s insanely fast. An interesting and unexpected side-effect of using ISCSI instead of a share is that the “recycle bin” functionality is now working on that drive. Not as huge as having the cache, but handy when inadvertently deleting something. Way more convenient than doing a restore from a backup. Of course, I should use zfs to just snapshot, but I haven’t set that up yet, but the recycle bin is pretty painless, so it was a nice side-effect.
Thanks again for your help and the original thread.
Yeah without compression it takes a long time, with compression it’s pretty fast but not instant. Creating a zvol instead of a disk image file is instant but like i said before it always takes a fixed amount of space from the pool.
I might have to try switching to raidz from my mirror too.