SOLVED! - ESXi 6.7 U2: Create Datastore on same USB drive as ESXI itself?

Operating Systems & Open Source
1

Hi,

I’m starting to experiment with ESXi 6.7 U2 and am running in a little hurdle:

I got 2 SATA SSDs with PLP and a Startech USB 3.0 to 2 x SATA RAID1 enclosure. The SATA drives are configured as a RAID1 volume.

ESXi can be installed just fine on that RAID1 volume however the remaining drive space is unusable.

I just plan to use this drive space as a Datastore for a very low IO storage managing VM that itself gets storage controllers and network adapters via PCI Passthrough and serves the created volumes as Datastore space to ESXi via NFS.

Can anyone assist with this VMware-unsupported way of getting space for a Datastore?

Thanks for the help!

Regards,
aBavarian Normie-Pleb

2

You could potentially use a second USB stick, but i’m not sure ESXI will allow you to run VMs from its own installation.

I would also caution that it is a very bad idea, as you’re putting read/write IO on the media that boots ESXI and USB boot drives for ESXi are unreliable enough as it is. They do die after a few years even with ESXi alone installed on them (I have a heap of them in the field in this configuration).

TLDR: Just don’t.

2 Likes
3

The only way I know of to accomplish this on 6.7 is to tell the enclosure to produce 2 arrays. At that point it will appear as a valid disk that is available to create a datastore on. A better way would be to boot ESXi off something less useful in the first place. It doesnt really run from the media it boots from – It boots, loads in to ram, and only updates the boot media for patches etc if so desired by the operator. Logs are default sent to a temporary location that is not persistent at least when using vcenter (i havent used stand alone ESXi in a very long time). This would let you use your external USB array as a fault tolerant storage device for said VM that will manage the underlying disks passed through to it. I speak from experience when I tell you that a failed ESXi boot device will not result in termination of VM’s running, although dealing with it after the failure is another story if vmotion isnt an option.

4

I understand your point of view that my plan isn’t official best-practice.

In my experience traditional ESXi USB boot media like thumb drives and SD cards “suck” when used 24/7.

Additionally I just despise it when a single drive failure interrupts operation this is why I even want to use SSDs with PLP for ESXi itself.

When needing USB storage I’ve been using multiple of these concoctions for some time without any failure:

  • Take the board out of a Startech SM22BU31C3R ( https://www.startech.com/HDD/Enclosures/dual-m2-raid-enclosure~SM22BU31C3R )
  • Add 2 M.2 SATA to 7-Pin SATA adapters
  • The electrical signal integrity is fine: The SSDs get SATA 6 Gbps links and C7 SMART values do not change after daily use
  • Connect 2 SATA SSDs to the 2 new SATA ports, the SSDs are powered by the system power supply
  • The RAID1 volume created by this cheap USB SATA controller is basic meaning you can take one of the used SSDs, connect it to a standard AHCI SATA controller and boot from it etc.

usb3-sata-raid1
usb3-sata-raid1.jpg1455×2027 964 KB

  • The gunk on the connectors is a combination of a thin layer hot glue protecting nearby components and epoxy glue making them (almost) indestructible (the PCB is going to break before any connectors get loose through wear).
  • Since the SSDs are powered externally the electrical load is minimal, the USB SATA adapter chip is only getting warm to the touch (10 K above ambient without any heat sink even after hours of sustained heavy use)

Could I create an additional partition on the ESXi boot volume in its unallocated free space and “format” it with the VMFS5 file system?

5

Sorry mate but i have to ask… but why are you trying to do this?

You can get a second USB flash drive for like… under $10. So do that, if you have to?

IMHO your plan isn’t just “not best practice” ; it is setting yourself up for failure…

  • if it is a lab, just make your life way, way easier and use seperate media?
  • if it is for actual production use (and there is some single USB port limitation)… just… no…

is there some special circumstance i’m not getting?

6

This is a personal homelab setup.

There seems to be a point where we just have to agree to disagree, then.

  • I agree that USB thumbdrives et al. are shitty (same experience) regarding their longtime reliabilty.

  • The whole purpose of this experiment is to never have a singular harddrive failure lead to data loss OR interruption of operation (like unplanned reboots).

  • So on purpose I do not intend to use “traditional” ESXi boot media but am willing to check out everyday behavior of such an atypical solution.

  • The less than optimal USB experience of the past is exactly why I’m using such a concortion with an USB 3.0-to-dual-SATA-for-RAID1 with enterprise-grade SSDs with Powerloss Protection after having evaluated it in longtime use and found it completely reliable as storage even for an operating system (Windows) with the “Windows-To-Go” feature.

  • Running in USB 3.0 mode, the RAID 1 volume delivers around 450 MB/s sequential read and 360 MB/s write performance so a single VM running on a Datastore that is located on the same device as ESXi itself should not be an IO strain problem at all.

  • The USB-SATA adapter’s board is designed to deliver power to 2 M.2 SATA SSDs only via USB 3.x, since I’m using the system’s msin power supply the power delivery section on this board is pretty much bored which in return should reduce the risk of hardware failure here.

  • If the “traditional” USB ESXi boot device dies during operation in a basic at-home environment the system should crash anyway at some point and you’re f’ed.

If this isn’t enough to clear things up then I guess I’m shit out of luck.

1 Like
7

ESXi sees the SSDs as USB drives and in an effort to preserve itself it doesn’t make any unnecessary writes to USB flash media (because its usually garbage). If you installed ESXi on the SSDs directly, then ESXi will only take up 8gb of space and the rest will be made a Local Datastore

Sidenote. If you really want to try it, you can follow this guide. I’ve set it up on a 128gb USB drive before on ESXi 6.5. I will say that you lose the ability to passthrough any USB devices though as the usbarbitrator service needs to be disabled (don’t enable it or your datastore won’t be reusable on that device unless you clean up any trace of the hardware ever being on ESXi)

2 Likes
8

Thanks for that link!

I think the key to the “best solution” was hinted to in the comments section linking to another website:

http://www.horizonbits.com/2017/02/19/squeezing-esxi-on-usb/

If I understand it correctly you don’t have to mess with USB services when only wanting to use the single USB storage device ESXi itself is installed on.

The summary seems to be:

  1. Create an additional partition on unallocated disk space on ESXi’s USB boot device

  2. Format it with VMFS6

  3. Change that partition’s “number” (?) or ESXi may lament over something when installing updates

I’m currently trying to understand everything, part 3) seems to be the most difficult (for me, since I’m a Terminal layman).

9

Took a look at that guide, seems both interesting and is more useful than the guide I linked if it works on 6.5 (it was written with 6.0 or earlier)

Looks like the guide has all the commands you need, just mostly need something like a live boot USB of Linux Mint or Manjaro so you can use GParted and edit the ESXi USB stick

1 Like
10

Got a subscription for “Parted Magic”, I find it very useful since it bundles recent releases of Linux tools, including GParted and is pretty much usable with a GUI which is nice for a normie-pleb like me.

I’ve reread the last guide and think I understand everything now. Hope I’m not too slow for users with expertise in this field.

So according to that guide am I right that the only “hack” is the partition number change from sdb2 to sdb10 meaning it is crucial to pick a number higher than ESXi’s standard partitions and ESXi is then able to properly format it with its VMFS file system?

Have there been instances when ESXi was altering the previously created partitions on its boot device after being installed, for example when doing a version update/upgrade?

I’d guess this would be the only issue/risk involving this guide.

11

I believe you need to increase the partition number for the spare space at the end so that the flash drive logically thinks the spare space is at the end of the drive, rather than somewhere inbetween the ESXi partitions. Likely something to do with how ESXi reads parition table data

1 Like
12

I can confirm that the method described in that guide also works with ESXi 6.7 U2 and VMFS6 instead of VMFS5.

http://www.horizonbits.com/2017/02/19/squeezing-esxi-on-usb/

After the process the Datastore just appears in the GUI and can be used as expected, tested it by using it to update ESXi to the latest release from June 20th, 2019 (not possible with an USB-only installation without an additional Datastore that is used for swapping).

Some pictures with a SanDisk Extreme USB thumbdrive, after the POC I’ll setup the proper SSD installation tomorrow.

esxi-usb-datastore-0
esxi-usb-datastore-0.png982×412 46 KB

esxi-usb-device
esxi-usb-device.png1089×600 72.8 KB

Thanks to 2bitmarksman for linking to that first guide that had the reference to the eventual “good” guide in its comments!

No BS with doctoring with the USB services etc.

2 Likes
13

Since the source to the article that truely answers this question has been relegated to ​wayback machine on internet archive at this point,

I found a way to do this from inside of vmware using partedUtil to partition and vmkfstools to format the partition to make it a datastore without using any other tools at all.

[Begin Disclaimer]
FYI: for all the datajockey’s out there saying DON’T do this… In my case I’m using an NVMe M.2 drive in a NVMe to USB-C cage for a “real” hard drive but plugging it into the USB header inside a server… that back in the day (pre vmware 6.7) was fully acceptable and supported way of booting ESXi.

I understand the normal concern with performance and danger when doing this with a normal USB stick, but in my case vmware recognizing this as USB storage it was refusing to create the datastore, which if this was just in an M.2 port on the server mainboard it would not have even been an issue.

My usage case was to separate the ESXi boot drive from the RAID in the server so the whole RAID in the server is available an segregated from the ESXi boot disk — but I still wanted to have a Datastore on the NVMe/USB to put ISOs I would use to install os that would be on the RAID Storage.
[disclaimer end]

ADDING NEW PARTITION:

partedUtil allows creation of table, deletion of partition, resizing of partition, but no “add partiton” other than printing the existing partition table, and then resetting it using the existing info, and appending the new info.

Since the ESXi boot disk numbers partitions 1, 5-9 any partition you make with a third party tool is going to try to be partition 2.

vmware documentation for parted util can be found at:

https://knowledge.broadcom.com/external/article/323144/using-partedutil-command-line-disk-parti.html

Turn on ESXi Shell and SSH either from the Console of the server, or start the services from the ESXi web ui.

connect to ESXi using putty

Issue following commands, and use the info they provide to help you craft the solution to match your drive:

ls -ltrh /vmfs/devices disks

get existing partition table using name from above command, using the tab key to complete the name of the device, so we dont have to type the whole thing:

partedUtil getptbl /vmfs/devices/disks/vml.01[tab]

craft the command to “replace” the existing partition table by over writing it with a new table that contains all of the existing table, and appending the new partiton we are wanting to create:

The command above shows the a disk info line for example in my case “31130 255 63 500118192”

Use this last number minus 2096 as the max number of sectors to determine the end of the new partition, and the end of partition 9…

“9 1843200 7086079”

add 1 to start it right after that partition, and specify the type of partition it will be using table found by running command:

partedUtil showGuids

so for example:

if end of partition 9 is 7086079

and end of the disk is 500118192 (minus 2096 = 500118192)

and we want to create a vmfs partition type “AA31E02A400F11DB9590000C2911D1B8”

we come up with:

“10 7086080 500118192 AA31E02A400F11DB9590000C2911D1B8 0”

partedUtil allows creation of table, deletion of partition, resizing of partition, but no “add partiton” other than printing the existing partition table, and then resetting it using the existing info, and appending the new info.

so in my example, I used the results from “partedUtil getptbl” above, and pasted them into a “partedUtil setptbl” command, putting quotes around each line, removing the text to put it in the right format, and then just added my new partition at the end of the command:

example result:

partedUtil setptbl /vmfs/devices/disks/vml.010000000030313233343536373839323353534b205374 gpt "1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B 128" "5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 0" "6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 0" "7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 0" "8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 0" "9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 0" "10 7086080 500116096 AA31E02A400F11DB9590000C2911D1B8 0"

next verify the table changed to match your input, again using the tab key to complete the name of the device, so we dont have to type the whole thing:

partedUtil getptbl /vmfs/devices/disks/vml.01[tab]

FORMATTING THE PARTITION AS VMFS:

vmkfstools -C vmfs6 -S USB-Stick /dev/disks/mpx.vmhba32:C0:T0:L0:10

In vCentre or ESXi web interface refresh and then check to see if the new Datastore is showing.

In my case it just showed up I didn’t have to do anything else.

14

Thanks for your detailed reply!

Maybe that helps users Broadcom hasn’t scared away from VMware, yet.

I’m currently trying to get into Proxmox in my spare time.