Looking for PMC/Microsemi NV-1616 NVRAM Drive Documentation/Tools

Hello LevelOneTechs Users

I apologize if I’m in the wrong category, this is my first time posting here. I recently ordered a used PMC NV-1616 (part of the Flashtec NV1600 family, sometimes listed also as NVRAM 1616 EMC 16GB DDR MN TCA-00364-08-D) PCIe card that acts as a standard NVMe storage device, but uses DRAM modules as storage to give great performance (perhaps rivaling or exceeding Intel’s Optane), then backing up to onboard flash memory on shutdown.

Mine sadly doesn’t have the original supercapacitor backup power source, so there’s a high likelihood mine will be unformatted each time the system is restarted.

I was wondering if anyone knows where I could find documentation and CLI utilities for this device?

When PMC was acquired by Microsemi/Microchip, it seems downloads migrated to their website, but the pages and downloads mostly appear to have been taken down (I’m guessing since the product has been marked end of life). All I’ve found so far are basic datasheets.

Thank you so much for any guidance or help you can provide.

1 Like

Interesting tech. PCIe gen3 x8, no apparent way to extend storage capacity.

You can easily find cards on eBay (~$95 w/capacitator, ~$50 w/o).

The spec sheet talks about the card offering an NVMe interface to the OS. So, the nvme cli utility (on Linux) should allow querying the card, formatting it, etc.

As you said, in 2024 it seems to compete with the last available 58GB Optane 1600X, which I would consider more practical (arguably more durable).
The card offers potentially better latency (tbd) and bandwidth (8x PCIe lanes vs. 2x) but ultimately has limited use cases due to the hard storage capacity limit.

What makes this card (for me) hard to recommend is the relatively strong resource requirements (8x PCIe slot) with seemingly niche benefits. I’d have the nagging question if I could use these (rare and valuable) PCIe lanes (slot) for something more useful?

Please keep us updated with your experiences.

1 Like

That is a really good point, it definitely isn’t the most efficient use of PCIe resources to take up 8 lanes for only 16GB of fast storage. I hadn’t thought about that part.

Thank you! I’ll definitely give nvme-cli a try on it once it arrives. Hopefully I can find a way to access what it chooses to do with the 16GB of DRAM and 16GB of flash, or perhaps find a manual to see what modes are available. I can see on Archive.org that a manual and CLI tool definitely existed for these cards, but unfortunately none were saved.

I’ll definitely share the results once the card arrives. Thank you again!

So far I have found a few more resources on the NV1600 cards.

While searching the other models of this card (the 4GB NV1604 and 8GB NV1608), I came across a guide for updating the firmware for the NV1604 from a company called Avere (now owned by Microsoft) who used them in their FXT 5000 servers. In the manual it contained a working link to download the NV1604 firmware!

Avere Documentation: Avere Legacy Documentation Home
NV1604 Firmware: https://download.averesystems.com/software/pmcnv_firmware_2.6.06.1

Sadly the tool used to install such firmware appears to have come already installed on the Avere servers, so not sure how useful the firmware is on its own. Also included in the documentation is the Avere FXT 5000 Series Field Service Guide, which shows the first record I’ve seen of any details on the operation of the NVRAM card. Below is a snapshot of the NVRAM page documenting the LED indicator behaviour for these NVRAM cards.

Lastly, I believe I’ve discovered what server the majority of these used cards were removed from. Most on eBay come with an extra adapter attached on the opposite side of the PCI slot cover labeled “HD Mini SAS To Internal SAS Adapter,” and the same sellers usually list at the same time an EMULEX LPE16304-M-E Fibre Channel card with an identical Mini SAS adapter on the PCI slot cover. That Fibre Channel card eventually lead me to the Dell/EMC XtremIO X2 server, which has both the NVRAM card and fibre channel card with the Mini SAS adapters on both brackets, as can be seen in pictures on this article: DellEMC XtremIO X2 – Part 2, The Hardware & Improved Data Reduction - Itzikr's Blog

Sadly all the XtremIO X2 downloads from Dell are locked behind having an account with access to those specific downloads, but perhaps a step in the right direction towards finding resources/tools for these NVRAM drives.

If anyone might have access to the tools for managing the NVRAM drives on either the Avere FXT 5000 or XtremIO X2 servers, I would greatly appreciate your help. Thank you again.

1 Like

Well, I got the card and… it hasn’t gone quite according to plan. Sadly the card does not show up as a storage device when connected to a PC.

In Linux the card is detected by lspci as “Non-Volatile memory controller: PMC-Sierra Inc. Device f117 (rev 06)” but does not show up as a storage device, and does not show up in nvme-cli either.
Strangely though, during bootup there is a long delay before an error shows up stating “nvme nvme0: Device not ready; aborting reset, CSTS=0x130df1”.

In Windows the drive is detected in Device Manager as “Standard NVM Express Controller” but with the error “This device cannot find enough free resources that it can use. (Code 12)”. It was the only device, aside from onboard devices, connected to the motherboard in this test. Testing was conducted on a 3rd gen Intel system with a B75 chipset.

I tried a separate machine in case of some strange motherboard conflict, but sadly my Dell T110 II’s motherboard is very picky and won’t detect the card on first boot, and will not POST with the card present on any successive boot attempts. I will have to take down another system to try it in one day.

As per the Avere guide, the card on boot will get to the flashing green state, indicating “Driver loaded successfully”. If one reboots the system without power cycling though, the light will go to solid green indicating “Firmware loaded successfully, ready for operation”. In either LED indicator case though, all results in both Linux and Windows are identical.

Has anyone encountered nvme not ready errors like this before and might have a solution? My guess at this point is that, since the card supports “Block Access Mode” and “Direct Memory Access Mode” (diagram from the datasheet below), perhaps the card was previously configured for the direct access to be read by a specifically built application, hence why it’s not showing up properly as an nvme device. Sadly without configuration tools, I can’t change that option and so far Microchip/Microsemi’s support team doesn’t seem able/willing to provide any documentation or CLI tools.

If anyone has any ideas, might have access to a tool that can configure the card, or know someone who might, please do let me know as I would greatly appreciate your help.

Thank you again. I will try to keep investigating the card and post if I make any progress.

1 Like

I tried on microchip site to request access for the device, I did not get a response. I opened a support case on the matter and the response I got is:
"Unfortunately, it seems that due to the nature of the product and the development required, we do not provide documentation on Flashtec devices.

Thank you for your understanding."

"
I discussed with the internal team. This product is very old and has been EOL’ed. The suggestion is to go back to the company, from where you bought the card from, and request for help. There is a good chance the drivers you needs are not available. The card should have come with software so something is wrong here and we can not support this request.

Thank you for your understanding."

And I found a person on reddit that claims that has the software and documentation but he does not want to share it because he says the pages are fingerprinted to him.

2 Likes

Thank you so much for all the work you did trying to get the documentation and software! You definitely got further with support than they let me go through the channels I made contact through. Do you also have one of these NV-RAM cards?

That’s interesting that you were able to find a person on Reddit who has access to the documentation and tools, though that’s too bad he seems to be under an agreement that doesn’t seem to allow him to share them. How did you find this user, is there a community of people with these cards for a specific use case?

Thank you again so much for your help.

any News here ? I get an Update Package from an avere System and it was possible to restore the Partition but not tested with card until now only in vm

Blockquote
root@nvramtest:~ # ./pmcnvutil
./pmcnvutil: no operation specified
./pmcnvutil [–unit=X] [–debug]
Operations:
–autorestore=BANK0/BANK1/BACKUP/DISABLED
–autobackup=BANK0/BANK1/DISABLED
–backup=BANK0/BANK1
–bad_block_scan=BANK0/BANK1
–banks
–erase=BANK0/BANK1
–overlap
–restore=BANK0/BANK1
–stats
–temp
–update=FILENAME
–info

There are Jumpers on the Card it Looks Like one is a reset jumper if unplug the sas Connection try to Remove Jumpers if there are any on it and diconnect the two wire cable that comes from the sas Port and try again
image

Thank you so much for this information! I will have to check the jumpers on my card. Where were you able to find this picture and information? Is there a table on what the two jumper configurations do?

If you have access to the manual, I’d greatly appreciate your help. Is it something you’re able to share with us?

Also, where were you able to acquire the Avere update package?

Thank you again for any help or information you can share.

1 Like

Check out the link in the image; there’s an update package behind it. They appear to be images. I tested a few other versions to download this are the working once.

https://azure.github.io/Avere/legacy/ops_guide/4_7/html/gui_software_update.html

image

I was able to extract the file using binwalk extract. After that, you can open the files with 7zip. It’s based on Free BSD 9.2. Unfortunately, I don’t know much about BSD and don’t have any hardware that boots BSD 9.2 so I could load the modules. There are kernel modules on the boot partition that are used for the NV1604, if I understood correctly.

The v5.3.4.2 based on freebsd 11.3

The newer packages i dont check untill now but it is possible to use this git project to extract the files if no linux system is ready with binwalk.

The image is from an Xtremio X2 file that I unfortunately can’t find anymore.

I think I also understand why the cards aren’t working as they should.

Since both the Xtremio and the averesystem have A/B partitions, we enabled the auto-backup mode of the active bank (2x16GB on the NV1616) during boot.
This way, in the event of a power failure, the data is written to the flash drive. Automatic recovery will always be disabled, so that upon booting, the recovery of the currently booted bank will be invoked.

During shutdown, the data will be cleanly written to the array, and the NV1616 backup will be disabled to save write operations. Unfortunately, I think our cards come from cleanly shut down systems.

Furthermore, there is an option in the servicemenu.py to disable the battery backup, which I think refers to the supercapacitor

class ConfigDisableNvramBatteries(object):
    def __init__(self):
        pass

    def __str__(self):
        return "Disable NVRAM batteries"

    def invoke(self):
        print
        print "THIS OPERATION WILL DISABLE NVRAM BATTERIES, WHICH CAN RESULT IN DATA LOSS"
        print
        while True:
            confirm=raw_input("Are you sure? [yn]:  ")
            if confirm.lower() in ("y","yes"):
                print "Disabling NVRAM batteries"
                sys.stdout.flush()
                cmd = '/home/armada/bin/maintenance/nvramutil --disable'
                if util.popen_cmd(cmd) != 0:
                    printerr('Could not disable NVRAM batteries')
                    break
                else:
                    logging.info("NVRAM batteries disabled")
                    print 'NVRAM batteries disabled'
                    break
            elif confirm.lower() in ("n","no"):
                print "Cancelling"
                break```

If you have problemms with boot you can try to cover the pins like on the dell sas cards, there is a managment part on it that will work with idrac but only in dell server systems, i think it is possible that this is a problem. Updated: SAS HBA crossflashing or flashing to IT mode, Dell Perc H200 and H310 – techmattr

you can boot with fr jumper wait until all leds back to green than remove jumper and disconnet power and super cap bbu the card is now reset to default firmware settings

I can now backup and restore data on nv1616 but the auto option is only working one backup than the function need a reenable via nvme admincomand i try to get the comands from the system but my bsd knollage is very small i can only google and test

1 Like

I have tested several NV1616 PCIe cards and noticed the following:
On newer systems, the card is detected by Windows without any issues. However, on older systems (e.g., Intel 6th generation), I receive an error stating that there are not enough resources available. This seems to be related to Above 4G Decoding, which is enabled by default on modern motherboards.

Another issue is that the card loses all data after every shutdown. My first suspicion was a faulty capacitor, but the electrolytic capacitors still have about 70–85% of their capacity, which should be sufficient.

After contacting the seller, I was told that this is an EMC OEM version of the card that has a firmware bug. Supposedly, this can be fixed with a firmware update. I did find a firmware file, but no tool to flash it. The card is said to be running firmware 2.7.18.0.

While researching the IC PMC 89HF08P08CG3, I also came across the Radian Memory Systems RMS‑200, which appears to be related.

If anyone has successfully restored full functionality of the card? – including proper backup and restore behavior – and is willing to share how they achieved it, I would greatly appreciate it.

So far, I have tried to replicate the behavior using Proxmox: when the system shuts down, the card is saved as an IMG file, and on startup, the image is written back. But of course, this is only a workaround.

1 Like

I don’t try another firmware

These cards originally sold for around 2,000–4,000 USD, so you’d expect the built‑in power‑loss protection to actually work. After all, that’s the whole point of NVRAM. But on all my units, the power‑loss backup function seems to be disabled or non‑functional.

Here’s what I’m observing across multiple NV1616 cards:

  • After every power loss, the entire content is wiped.

  • The SMART status looks “empty” after each outage, as if the card was brand‑new.

  • Swapping the flash chip with one from another NV1616 changes nothing.

  • Without the flash chip installed, the card is not detected at all.

There’s also a small add‑on board converting internal SAS to external SAS. It taps a voltage and routes it through two 470‑ohm resistors to a 2‑pin header. The product PDF calls this an “Auxiliary Power Optional 5V Input”. Adding a jumper didn’t change anything.

By default, the card identifies as:

“Non‑Volatile memory controller: Microchip Technology Device f117 (rev 06) (prog-if 02 [NVM Express])”

But if I set the middle jumper, it switches to:

“Microsemi / PMC / IDT F32P08xG3 [PCIe boot mode]”

In that mode, no memory is detected at all.

The other jumper positions had no noticeable effect.

At this point, I’m wondering whether the NV1616 expects a specific driver or firmware mode that sends a command during shutdown to trigger the backup routine. Without that command, the card would be unprotected during a power loss, which would explain why it behaves like a volatile RAM drive.

I’m considering buying an NV1604 to compare the models, but it’s quite expensive.

Unfortunately, the way the cards work makes sense. They are completely controlled by the operating system. Auto-backup is activated when the card becomes the cache. When the system shuts down, the cache is written to the hard drive and the backup is deactivated. This way, the few write operations the card performs can last indefinitely and are only used when there is data on the card and the power fails.

When I think about the Radian Memory Systems RMS‑200 NVRAM, which is quite similar, I haven’t seen anything indicating that backup/restore is disabled or only works once.

I’ve put together something in Proxmox that works independently of the PMC NV1616’s backup/restore function. The data is read from the NV1616 during shutdown and saved into an image, and when the system starts, the data is written back from the image onto the NV1616. Unfortunately, this does not solve crashes or power-loss issues in any way. In such cases, you would have to fall back to one of the older image states.

Because of this setup, it’s no longer possible to pass the card directly through to the guest (PCIe passthrough). Instead, you have to use the method of passing it through as a drive via IDE/SATA/SCSI or VirtIO. This would allow you to create additional backups using the backup function, but restoring those backups is not directly possible.

Unfortunately, some things shifted during the insertion process, and there are also display problems.
NV1616 Backup in Proxmox.txt (2.2 KB)

Anybody manage to get access to pmcnvutil or another utility for writing firmware to this thing? I have a Dell EMC NV-1616 (TCA-00364-63-D) with no supercap and I just want to use it as volatile swap space or a scratch disk. I don’t need data persistence.

I’ve been able to get it to light up all green in [PCIe boot mode] [111d:80cf] so I should be able to write to it. I haven’t been able to make it writeable no matter what jumpers I bridge.

Alright I got it figured out. For anybody who buys one of these and stumbles on this thread, here is how I got it working without the supercap.

  1. Disconnect the little wire that runs from the SAS2 piggyback module to the NVRAM card. Optionally, just unbolt the piggyback module and remove it completely.
  2. Put a jumper bridge where the SAS2 piggyback module wire connected. This puts the card in a reset mode of some kind.
  3. Put the card in a machine and boot it up. The NVRAM status LED will start blinking in 2hz indicating “Reset operation in progress”.
  4. Wait several minutes until the reset operation completes. On my card, all lights were green when it was finished.
  5. Turn off the machine.
  6. Remove the jumper bridge completely.
  7. Put the card back in your machine and start it up.

If you’re successful, the supercap light will still be solid red, but the NVRAM status LED will eventually blink green at 1hz and nvme –list or lsblk will show your NVMe device. From there, just use normal formatting steps.

Strange that the SAS2 piggyback module gets plugged into the firmware reset jumper. There must be some reason, but it just seems odd.

2 Likes