So during the past week I have been trying to make my first FreeNAS setup work. I have overcome most of the problems regarding the basic functionality of the machine, however for some reason whenever I am making a file transfer from my Windows desktop over wired connection that is over 3-4 GBs the machine will most likely crash and reboot. Bellow I have pasted the error report from the system log.
I am thinking it could be a RAM problem even though I have 16gb, because the size of the array is 17.3 TB, I just would think that FreeNAS would prevent memory overflow somehow.
Thanks in advance.
Nov 2 02:17:34 freenas generate_smb4_conf.py: [generate_smb4_conf:1498] Unable to open /var/db/samba4/private/secrets.tdb: [Errno 2] No such file or directory Nov 2 02:17:34 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: /sbin/sysctl -n 'kern.maxfilesperproc' Nov 2 02:17:34 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: mount Nov 2 02:17:34 freenas generate_smb4_conf.py: [common.pipesubr:66] Popen()ing: /usr/local/bin/net -d 0 getlocalsid Nov 2 02:17:35 freenas smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1 Nov 2 02:17:35 freenas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd Nov 2 02:17:36 freenas smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2 Nov 2 02:17:37 freenas smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3 Nov 2 02:17:38 freenas smbd: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:35 Err:-1 Errno:2 No such file or directory Nov 2 02:17:46 freenas mDNSResponder: mDNSResponder (Engineering Build) (Oct 3 2016 06:58:50) starting Nov 2 02:17:46 freenas mDNSResponder: 8: Listening for incoming Unix Domain Socket client requests Nov 2 02:17:46 freenas mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0) Nov 2 02:17:46 freenas mDNSResponder: CheckNATMappings: Failed to allocate port 5350 UDP multicast socket for PCP & NAT-PMP announcements Nov 2 02:17:52 freenas mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000801406D60 freenas.local. (Addr) that's already in the list Nov 2 02:17:52 freenas mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000801407180 135.1.168.192.in-addr.arpa. (PTR) that's already in the list
Doesn't look like there's a whole lot of useful messages in the log. Have you tried bringing up the log level a notch or two in the SMB service configuration? Once that's done it may be worth doing a
tail -f /var/log/messages
and then doing the file transfer again. Just to see what comes up.
Also are there any other machines on the network? It would be nice if we could pin this on FreeNAS or the client machine.
Edit: Also for a memtest, grab a Linux LiveCD from the interwebs somewhere (probably the Ubuntu Desktop download if you're unsure), and booting to that you will see an option for Memtest86+.
If it can boot from a usb stick put a live linux into on a usb stick do the test ...won't change your install just run the test from the usb stick. Most current linux installs have a live version and I know Ubuntu does and even has a mem checker on boot as as option
On the live install under ubuntu you can sudo apt install stress
And that can load up your CPU to 100% but a home server barely hits 10% load. it more the memory and storage that fails.
And if your freeNAS box is solid and ok :)... Well I hate windows and cant help you there on your client PC :)
Hello again so I will be running the tests later tonight or tomorrow morning. However after a few crashes, I tried copying a folder but instead of letting the copy run throughout, I paused it a few times. I found out the problem is that the files are stored in the wired memory and then never deleted unless there is a reboot. Since it retained everything on memory for more than 30 minutes after the copying was finished.
I have attached a picture from the system monitoring for ram.
As it can be seen the system never frees the memory, but also whenever I ran out of ram the system will not use the swap memory for some reason. I have no idea what could be causing such problem.
You always want the system to use all the ram at least as cache which can be dumped when needed. At 16G i would not even have a swap disk space because if used will tank your system. I put enough ram in my machine so I never need swap storage. 16G is pretty safe.
Just a question, how many passes should I let memtest go through? I have already done 5 without any errors. I should be fine, but I do not want to stop it if I have to let go for longer.
As far as the memory usage goes, ZFS will use as much RAM as you let it to cache disk IO. "Free RAM is wasted RAM." The OS can ask for that memory back if it needs it, it's just a cache. I don't see any evidence that you're actually running out of memory, either.
Have you experienced any data errors with your pool? Have you run a scrub recently?
I do not have any problems with errors, at least not ones that are affecting my data. My actual problem is that, as I am using my machine at random intervals it tends to crash and reboot. I have posted both messages from the log and the messages posted on the console when it crashes but according to everyone, the messages do not show any kind of problem. So I honestly have no idea what it could be.
I have left memtest run for over 42 hours doing over 9 passes without any error so I think its not a memory problem.
Is there any way to check if the disks are fine apart from the SMART report? Some kind of test?
I was doing some digging around and I came across some potential problems related to my case. I am not sure if this could be an explanation, but could it be a problem with my SATA controller? If yes how can I test it to actually check for it?
Are you using dedupe for the dataset in which you experience crashes? Hoping you have ECC memory? Is this a bare metal install or is it virtualized?
You can run (in CLI format) short/long SMART tests (Google). I really doubt your drives are the problem however your controller definitely could be. It wouldn't hurt to just throw a little bit of money at it and buy a LSI HBA on eBay and test with that.
I am not using dedupe, I have 16 gb of ECC. Had 24gb but my mobo required different ranking to use all of it :(. I have already done short and long SMART tests and I have event created schedules to have them run on a monthly basis. I will look into the LSI card, even though I already shed so much money on this machine, I think its worth it to have some peace of mind and solve all of my problems...
Pretty much any HBA will work, you actually have a lot of options.....
Get an HBA like you linked, that will work fine. Don't be afraid of second hand on eBay though. I have seen these cards for as cheap as $25 with shipping in the USA. These cards are built like tanks, they never fail (except the batteries on RAID adapters).
Since your drives are SATA you can get pretty much use any SATA adapter, like the cheapo ones on NewEgg for $10. Perhaps the best option would be to buy a cheapo one on Amazon/NewEgg/eBay and test to see if your problem is fixed. If it is then invest into something better, aka enterprise class. Just make sure the card supports your drives (e.g. 5TB). I believe some chipsets max out at 4TB, so something to look out for.
Instead of an HBA you can get a RAID card instead. This would be a better investment over the HBA as they cost around the same used as that HBA you linked. You of course won't use the RAID functions with FreeNAS but if you ever switch to something else you will have the option to use it as its intended purpose, a RAID card. The LSI 9260-8i in Dell OEM format is really the best bang for the buck and Wendell himself has mentioned this card in many of his videos. These can be had for $50-120. The Dell version is the best as the firmware is the same as LSI and the eBay market is flooded with them so they are extremely cheap. The IBM versions can be had a bit less, I have scored them on eBay for as cheap as $50 with a good battery, however they utilized a different firmware than LSI/Dell OEM.
If you go the RAID card route you would setup the drives as Single-Mode (pretty sure you would have data loss though) or you can flash IT firmware to the card which would turn it into an HBA (no data loss). If you go the IT firmware route make sure to get Dell OEM as support is significantly better than IBM or another OEM. There may be some negative consequences running drives in FreeNAS on a RAID controller in Single-Mode use, so to be 100% safe you would want to go the IT firmware route.
Note that if you buy an HBA or RAID card you also need two SFF-8087 to 4x SATA breakout, plus possibly some Molex to SATA power adapters. I recommend getting the 1x SATA to 4x SATA power adapter vs 4x individual power cables. HBA + cables will bring your cost closer to $100 so as I mentioned in option two perhaps the best bet is go cheapo SATA card and test to make sure your problem is resolved.
EDIT:
Each port on the HBA supports 4 individual drives, 8 drives total. If you wanted to run more than that you would have to get another HBA. The only other alternative to this would be a SAS exapander. It works similar to a Network Switch, giving you more ports. SAS expanders are costly so I would recommend another HBA for additional drives over that. Plus having a dedicated HBA to each pool wouldn't be the worst thing either.
Again I am going to mention that you have SATA drives and all you need is a SATA controller. So investing in SAS equipment to run SATA drives is spending money you don't need to, right now at least. I really feel like your best option is to buy the cheap SATA card and test the system. Make sure everything is stable and then decide what you want to do.
Thanks for the lengthy and detailed reply! All is good apart from the fact that I can't seem to be able to find the kind of cheap controllers that you are referring to. The cheapest controller I could find is this
which is SAS so I would also have to buy the SAT-Sata cables. Can you please help me. Keep in mind I need something that will work with 6 drives. My mobo has one PCI-Ex X16, PCI-Ex X4 and PCI-32
This is what I was referring to - http://www.ebay.com/itm/172398533672 however they are $50 more than they have been in the past. I suspect it is because the market is flooded with the 4 port version http://www.ebay.com/itm/401220112503 - $65 for this is really a great deal, however it unfortunately doesn't work for your application. Now considering the 9260-8i is still relevant, it sells new at NewEgg for $450 + $150 for the battery. So $150 used price for both is still an amazing deal.
I also looked at cheapo SATA cards and prices are higher than I thought so an HBA is your best bet.
HBA - http://www.ebay.com/itm/381799790224 (make an offer of $50 or less) I am sure they will take it. Only thing is, your drives are not on the compatibility matrix. I am sure they will work fine but this is something to make note of. LSI is geared towards enterprise and your drives are consumer grade so this is not surprising.
For the mini-SAS cables, get 2 of either of these depending on length and connector preference.
You can also get these. I would recommend these because they don't lock you in with SATA only, they also work with SAS whereas the cables above are for SATA only (or a backplane).
This cleans things up and depending on your power supply may be necessity.
You can of course get a different brand or order from somewhere else. At the minimum you need 2x SFF-8087 mini-SAS to 4x SATA breakout. They make all different kinds of cables for all kinds of applications.
Also, you didn't answer one of my questions. Is this a bare metal install of FreeNAS or is it Virtualized (ESXi, etc)?
If you're going to use an lsi raid controller you'll want to flash it in to IT mode, this turns it in to a HBA. Even in single disk mode on a raid card zfs isn't interacting with the disk but the raid controller which makes it difficult for zfs to detect and fix some errors.
