Situation
NFS is a mature and essential technology. Professionals will deploy it. Including in circumstances that I would advise against.
Background
I’ve never had a good experience with NFS. It’s too tightly coupled requiring coordination between server and client system’s Administrators, it’s kernel-land execution means when anything goes wrong I end up needing to reboot both the storage node and the clients to unstuck things, filesystems are usually written with interfaces that don’t tolerate the kind of inconsistency a network can offer. And filesystems tend to be designed in a way that expects low access times to random blocks, in a way that is incongruent with incurring 20ms+ between every primitive part of the filesystem driver.
That said, I think NFS has some serious real world applications, I would suggest that it fills a nitche similar to s3, where the backstore itself is practically unusuable directly (for example trying to index an NFS by running find ./ or tar -c ./ is going to go poorly), but if we offload the metadata storage to an application for example an SQL table of objects with pointers to the correct NFS pool and object path, suddenly we have very good latency for individual documents and very good operational index/backup/restore options available to us as long as we DIY.
Assessment
It occurred to me, if I could traverse the filesystem in parallel, I could probably parallelize a tremendous amount here and minimize the total wall time overhead introduced by NFS alone.
When I went searching for parallel-filesystem-access tools or benchmarks, I ended up finding basically nothing, and the sentiment that I could find was “you probably don’t actually want that [assuming you are using a normal one disk computer]”
I’m here to say, I think I do indeed want it, and I suspect that a high cpu core count, a 10gbe or better link to an NFS server of similar core count, and some clever cooperaton with file system atomics like inode traversal might be able to produce a tar or find a like tool that actually can run at some good fraction of the link speed, much closer to saturation than what I observe in practice.
PS, in “practice” this week, I was getting 1.1 MB/s when taring a real world application’s backstore over NFS to fulfill the expectation of a system backup. The total working set was only 20gb ( generated over 60 days) – these numbers give me a bleak outlook on the capability of a serious backup and restore option. In effect I expect to rapidly cross a threshold where generating backups become infeasible. And in this situation, the NFS server administrator will not grant remote shell sufficient to get conventional hardware speed filesystem access, so backup over NFS is the only option.
Recommendation
I am looking for proposals as to the theoretical correctness of the idea, that parallelized filesystem access would be possibly optimal in this NFS case/situation, whereas normally it would be always pessimistic to try in the non NFS case.
What would a serious benchmark tool look like.
(Is Bonnie++ a good base for generating a real world alike working set)
How would walking a filesystem in parallel might look.
( I imagine, a recursive parallel walk is the kind of computing atom/unit that we don’t have power tools and solved algorithms for by default today? Then the common problem of synchronizing and producing a log of outstanding object paths to be fed back into a parallel work queue of object to be read- and Then synchronizing and producing a tar document of fetched documents, seems like a lot of complication and opportunity for subtle mistakes)
How could we trivially show concurrent NFS access is useful, and further gauge the average object size above which the NFS access time overhead becomes comperable to local storage of various performance classes such as hard drives (equivalent to what an NFS server would likely use).
I think the optimistic caching at every level is going to complicate any such benchmark tool.
Cheers
This Felt like a fun and relevant problem to propose to a heavily parallel arm system like just feature on the YouTube channel, so I’m sharing it here in hopes of encouraging some fun community interest and catching Wendell’s attention