I am looking for a fast but inexpensive way to add some cache to my freenas system. the main purpose for this cache is I want to use the freenas for my VM storage. and from what I have learned so far is if I want faster than gigabit I need some cache.
so far everything I found is way to much for me in my home lab, North of 400$. I like optane but it get it on a pcie card is not cheap and abit large for a cache.
any ideas that might be worth looking I to?
You can always pick up a cheap 128gb SSD. It will help and for 30-40 bucks you can play around and get a feel if your use case needs something like optane or Samsung nvme. Also remember you don’t need a large drive for this 120gb range will be plenty. Search for freenas cache on thier official documentation.
I thought about getting a pcie card to add some nvme or optane drives too but all the optane drives seem to be 2x? seem crazy. then i was think something like a small Intel 750 but thy large.
am I over thinking this?
Yeah I seriously would getting something along the lines of an adata su800 128gb.
Are you already on a 10gb link between freenas and the host?
What is your VM host?
I am using XCP-NG and I have a 40gb link, just because I got the cards dirt cheap.
https://www.amazon.com/XPG-SX6000Pro-Gen3x4-1500MB-ASX6000PNP-512GT-C/dp/B07H5355JF/ref=mp_s_a_1_5?ie=UTF8&qid=1547555102&sr=8-5&pi=AC_SX236_SY340_FMwebp_QL65&keywords=adata%2Bnvme&dpPl=1&dpID=41WwzClBYZL&ref=plSrch&th=1&psc=1
There is a 256gb NVME for 53 dollars. Give that a go and report back.
1 Like
The 280GB 900p should be under $300
Normal SSD’s will wear out very quickly if used as ZFS cache
1 Like
How much ram is in the system now?
2 Likes
I have a similar nvme that I linked as a cache for a 2tb mirror in my system. 24gb of ram but I just haven’t taken the time to do any testing g with VM over my iscsi 10gb link to my VMware host
1 Like
I have 32g right now, when I fine deal on some 16g ddr3 I want to upgrade my VM host and then move what I can to my freenas server
Ok, cool. 32 is probably fine. Just wanted to check cause the l2arc can cause more harm than good if the system is short on memory.
1 Like
thanks for help, I am going to have to wait on this cache drives till I figure out what’s going on.
last night moving a lot of files from the NAS, to a hard drive was all over the board again so I need to figure this out. it was even slow over gigabit network.
averaging 60MB’s on both networks.
This is quite aggravating. more testing
What is the array setup you are using?
I have 6 drives per Vdev, 3 Vdevs of raidz1.
i know I am not pushing the array, dd is around 300MB’s ish
before I added the connectx3 cards I was getting a good steady 100MB’s. on the 1gigbit network.
i have a feeling I need to pull things apart and one by one trace it down to the problem.
Samba or NFS?
MTU on cards set to 9000?
Any other info?
What’s the connection between the cards?
Windows 10 is on Samba, xcp-ng using a NFS. using ip numbers to connect to the shares.
Freenas is 11.2 if that matters.
the connectx3 cards are dual port qsfp+ connecting them DAC cable made by amphenol with a mellanox part number on it.
1gig connection I left at 1500 MTU. the mellanox I did set to 9000 MTU, for IP’s I am using on freenas is 192.169.10.250 for the 1gig, the mellanox I am using 10.1.10.250 and 10.1.20.250
there was a blurb on 45drives with freenas 10g turntables so I plugged thoses in.
http://45drives.blogspot.com/2016/05/how-to-tune-nas-for-direct-from-server.html?m=1.
i am not expecting full 10g saturation, but going backwards is frustrating
last night I had some time to do some troubleshooting.
so here is what seems to be the problem, I need a new windows box.
Looking at the block diagram for my motherboard it seems the on board SATA3 connections are on the chipset, my PCIE lanes are on the other chipset. oddly there are 2 eSATA ports off the back that are on the PCIE lanes.
so I moved the hard drive to the eSATA port my transfer rates when way up and so far it’s not be over the board like the chipset SATA.
so I my guess the chipset bus just can’t handle doing a lot of high speeds transfers at once.
the problems you find when you want to start pushing speeds limits
I picked up a pcie flash card, sadly I found out it does not work in freenas 11.2.
anyway it’s now in my window box and I am getting consistent 400 to 500 Mb’ s on large files like iso’s, with that flash card to and from it and freenas.
Small files however are really slow 50Mb’s is there any optimizations that I can make to bring that speed up.
as soon as I can I am going to try Nvme drive, but I figured might as well work getting everything optimized.
What PCIe flash card is that?
it’s a 640gb Fusion IO, I should have done more digging, it was Truenas compatable but thy removed the drivers. other said it did work in Freenas but thy did not say which version.
I plugged it in and I get a red warning in the fui that’s it’s not a supported card. I could get the drivers but ever time it updates it might brake. I don’t want that kind of trouble