I’m starting to experiment with ESXi 6.7 U2 and am running in a little hurdle:
I got 2 SATA SSDs with PLP and a Startech USB 3.0 to 2 x SATA RAID1 enclosure. The SATA drives are configured as a RAID1 volume.
ESXi can be installed just fine on that RAID1 volume however the remaining drive space is unusable.
I just plan to use this drive space as a Datastore for a very low IO storage managing VM that itself gets storage controllers and network adapters via PCI Passthrough and serves the created volumes as Datastore space to ESXi via NFS.
Can anyone assist with this VMware-unsupported way of getting space for a Datastore?
You could potentially use a second USB stick, but i’m not sure ESXI will allow you to run VMs from its own installation.
I would also caution that it is a very bad idea, as you’re putting read/write IO on the media that boots ESXI and USB boot drives for ESXi are unreliable enough as it is. They do die after a few years even with ESXi alone installed on them (I have a heap of them in the field in this configuration).
The only way I know of to accomplish this on 6.7 is to tell the enclosure to produce 2 arrays. At that point it will appear as a valid disk that is available to create a datastore on. A better way would be to boot ESXi off something less useful in the first place. It doesnt really run from the media it boots from – It boots, loads in to ram, and only updates the boot media for patches etc if so desired by the operator. Logs are default sent to a temporary location that is not persistent at least when using vcenter (i havent used stand alone ESXi in a very long time). This would let you use your external USB array as a fault tolerant storage device for said VM that will manage the underlying disks passed through to it. I speak from experience when I tell you that a failed ESXi boot device will not result in termination of VM’s running, although dealing with it after the failure is another story if vmotion isnt an option.
The electrical signal integrity is fine: The SSDs get SATA 6 Gbps links and C7 SMART values do not change after daily use
Connect 2 SATA SSDs to the 2 new SATA ports, the SSDs are powered by the system power supply
The RAID1 volume created by this cheap USB SATA controller is basic meaning you can take one of the used SSDs, connect it to a standard AHCI SATA controller and boot from it etc.
The gunk on the connectors is a combination of a thin layer hot glue protecting nearby components and epoxy glue making them (almost) indestructible (the PCB is going to break before any connectors get loose through wear).
Since the SSDs are powered externally the electrical load is minimal, the USB SATA adapter chip is only getting warm to the touch (10 K above ambient without any heat sink even after hours of sustained heavy use)
Could I create an additional partition on the ESXi boot volume in its unallocated free space and “format” it with the VMFS5 file system?
There seems to be a point where we just have to agree to disagree, then.
I agree that USB thumbdrives et al. are shitty (same experience) regarding their longtime reliabilty.
The whole purpose of this experiment is to never have a singular harddrive failure lead to data loss OR interruption of operation (like unplanned reboots).
So on purpose I do not intend to use “traditional” ESXi boot media but am willing to check out everyday behavior of such an atypical solution.
The less than optimal USB experience of the past is exactly why I’m using such a concortion with an USB 3.0-to-dual-SATA-for-RAID1 with enterprise-grade SSDs with Powerloss Protection after having evaluated it in longtime use and found it completely reliable as storage even for an operating system (Windows) with the “Windows-To-Go” feature.
Running in USB 3.0 mode, the RAID 1 volume delivers around 450 MB/s sequential read and 360 MB/s write performance so a single VM running on a Datastore that is located on the same device as ESXi itself should not be an IO strain problem at all.
The USB-SATA adapter’s board is designed to deliver power to 2 M.2 SATA SSDs only via USB 3.x, since I’m using the system’s msin power supply the power delivery section on this board is pretty much bored which in return should reduce the risk of hardware failure here.
If the “traditional” USB ESXi boot device dies during operation in a basic at-home environment the system should crash anyway at some point and you’re f’ed.
If this isn’t enough to clear things up then I guess I’m shit out of luck.
ESXi sees the SSDs as USB drives and in an effort to preserve itself it doesn’t make any unnecessary writes to USB flash media (because its usually garbage). If you installed ESXi on the SSDs directly, then ESXi will only take up 8gb of space and the rest will be made a Local Datastore
Sidenote. If you really want to try it, you can follow this guide. I’ve set it up on a 128gb USB drive before on ESXi 6.5. I will say that you lose the ability to passthrough any USB devices though as the usbarbitrator service needs to be disabled (don’t enable it or your datastore won’t be reusable on that device unless you clean up any trace of the hardware ever being on ESXi)
If I understand it correctly you don’t have to mess with USB services when only wanting to use the single USB storage device ESXi itself is installed on.
The summary seems to be:
Create an additional partition on unallocated disk space on ESXi’s USB boot device
Format it with VMFS6
Change that partition’s “number” (?) or ESXi may lament over something when installing updates
I’m currently trying to understand everything, part 3) seems to be the most difficult (for me, since I’m a Terminal layman).
Took a look at that guide, seems both interesting and is more useful than the guide I linked if it works on 6.5 (it was written with 6.0 or earlier)
Looks like the guide has all the commands you need, just mostly need something like a live boot USB of Linux Mint or Manjaro so you can use GParted and edit the ESXi USB stick
Got a subscription for “Parted Magic”, I find it very useful since it bundles recent releases of Linux tools, including GParted and is pretty much usable with a GUI which is nice for a normie-pleb like me.
I’ve reread the last guide and think I understand everything now. Hope I’m not too slow for users with expertise in this field.
So according to that guide am I right that the only “hack” is the partition number change from sdb2 to sdb10 meaning it is crucial to pick a number higher than ESXi’s standard partitions and ESXi is then able to properly format it with its VMFS file system?
Have there been instances when ESXi was altering the previously created partitions on its boot device after being installed, for example when doing a version update/upgrade?
I’d guess this would be the only issue/risk involving this guide.
I believe you need to increase the partition number for the spare space at the end so that the flash drive logically thinks the spare space is at the end of the drive, rather than somewhere inbetween the ESXi partitions. Likely something to do with how ESXi reads parition table data
After the process the Datastore just appears in the GUI and can be used as expected, tested it by using it to update ESXi to the latest release from June 20th, 2019 (not possible with an USB-only installation without an additional Datastore that is used for swapping).
Some pictures with a SanDisk Extreme USB thumbdrive, after the POC I’ll setup the proper SSD installation tomorrow.
