To whom it may benefit.
The why:
I started messing with drones a bit before Hurricane Helene irrevocably changed a lot of my favorite places in the Western part of North Carolina, and went up to volunteer in the aftermath by taking videos of infrastructure that was either severely damaged or just…gone. Didn’t have a clue how to do anything other than use it as a flying camera. Found some AI website that would eat your videos and spit out crude approximations of a 3D model but the quality was rather poor and I’m not sure the output was all that useful to the people that we shared links with. I figured I would teach myself how to build a computer last January so I could figure out how to process imagery myself and that turned into designing and building a full on processing stack and doing over 5000 hours of testing. Since I absorbed a lot of information from Mr. Wendell’s videos it felt right to give hard won knowledge back to the community.
I am going to attempt to distill a year’s worth of lessons learned by smashing my face through things into a brief summary, omitting the almost endless caveats to keep this part simple. All of my research was done on systems paid for out of my own pocket, I tried asking the big companies for help, expecting to be politely laughed at, and I got the expected outcome. The goal was on figuring out how to build systems for different use cases, from an air cooled PowerEdge you can run off a generator onsite to a dedicated stack with a 100G backbone and distributed processing all meant to go as fast as possible so you can feed data in and get useful imagery out so informed decisions can be made that might help people anywhere in the world.
The guts:
- WebODM
- NVIDIA CTK
- Ubuntu
- Docker
The most optimal hardware:
- the fastest cores you can get your hands on
- the largest amount of the fastest RAM you can get your hands on
- high speed high endurance low latency storage
- NVIDIA GPU with the highest count of relatively new cores possible
The most optimal configuration:
- One master node with tiered storage
- As many worker nodes as you want to chew up the data either together or working one dataset per node
- High speed networking
The bare minimum configuration:
- something made in the last 10 or 12 years
- 256GB of storage
- 32GB of RAM and some sort of 128GB SSD setup as sacrificial swap
- patience and good cooling
I’m rather new to contributing to tech forums so I am not sure how large of a post is taboo, hopefully I get some feedback on that.
Now to ramble a bit.
I have tested on everything I could get my hands on built out of scrap or scrounged off Facebook and eBay, from a Dell Latitude 5400, to R630s, R730s, old Precision desktops, then a lot of custom stuff that made my work area look about like a mix between a flea market and a Microcenter exploded over every flat surface.
My drone was a custom build that used a SIYI ZT6 camera, running ardupilot and QGround Control on my Herelink and my ground station laptop. I figured out how to program fully hands off survey missions that embarrass what DJI allows you to do, at the cost of having what felt like a near vertical learning curve.
My testing methodology sort of evolved to compare against the build I was using when I first figured out how to render my first dataset (Mandarin in the image file properties took a long time to stumble upon as the cause for endless failures.) This was a Dell R630 with dual E5-2699 v4s and 1,536GB of RAM that is my baseline, with two Tesla P4s, and 8 10K SAS HDDs in a hardware RAID10 (sorry!) running one “node” per CPU. I do two runs of 15 jobs per host, average together the runtime, standard deviation, and calculate the spread for each run. The finals are recorded as the results for that specific design. I have a 1,936 4K image test set that amounts to about 60 acres that I use as my yardstick, and WebODM has test sets here –> Test Data With that initial setup a la February of last year the average runtime was 300.6 minutes. I think my runtime for the Brighton Beach dataset that they use for testing was in the 7-9 minute mark. My current design can do my test set in 93.85 minutes and the Brighton Beach dataset in 1.55 minutes.
The good news is, those old PowerEdges run almost anywhere, seem to be essentially indestructible, don’t care about dust, or being knocked around a fair bit. The bad news, after testing every type of storage known to man that will work in one, in every configuration, with every trick I could discover, I only cut time by 6.67%. Core pinning has a pretty marked impact on performance and consistency. So does only running 1DPC. Hardware RAID, software RAID, SATA HDDs, SAS HDDs, SATA SSDs, SAS SSDs, HHHL AICs, m.2 to PCIe adapters, meh, +/- 5% of the baseline time. Consistency varies wildly across the board, when juggling the different types of storage and RAID configurations, the best standard deviation I got was .002113, the worst was .015074, shockingly, consumer drives suck at consistency in a workload that is deleterious to storage because of the sheer volume of writes.
Remembering that I am self funding all of this out of pocket so I can’t go stack up creator levels of benchmarks, I then tried the following CPUs:
- i9-10980XE
- i9-13900K
- i9-14700K
- i9-14900K
- 7950X
- 5955WX
- 5965WX
For storage I have tried almost everything from SAS HDDs made in 2012 to the 9100 Pro, with the exception of newer Gen4 Enterprise flash and Gen5 Enterprise flash.
For GPUs I have tried:
- Nothing
- Tesla P4
- Titan Xp
- 2070 Super (EVGA because it actually matters)
- 3060 12GB
- 3080 Ti (EVGA again)
- 3090 (EVGA again)
For networking I have tried the steps between 1GbE and 25GbE.
Some other data points for perusal:
- i9-10980XE, 990 Pro 4TB/P1600x swap, 3060 12GB, 128GB 3200MT/s (8 DIMMs), AIO
- 41.87% faster than the baseline
- 34.61% faster than throwing an entire R630 with all 88 threads at the problem with every trick and tweak enabled
- i9-13900K, MP600 2TB/P4610 swap, 2070 Super, 128GB 4800MT/s (4 DIMMs), AIO
- 65.85% faster than the baseline
- i9-13900K, MP600 2TB/P4610 swap, 3090, 64GB 6000MT/s (2 DIMMs), AIO
- 67.49% faster than the baseline
- i7-14700K, 870 Evo 2TB, 4090, 64GB 6000MT/s (4 DIMMs), air cooled
- 62.54% faster than the baseline
- i9-14900K, 990 Pro 4TB/905P swap, 3090, 64GB 6000MT/s (2 DIMMs), AIO
- 68.78% faster than the baseline
- i9-14900K PM1735 1.6TB/990 Pro swap, 3080Ti, 64GB 6000MT/s (2 DIMMs), AIO
- 68.72% faster than the baseline
- i9-14900K P Core Only, PM1735 1.6TB/990 Pro swap, 3080Ti, 64GB 6000MT/s (2 DIMMs), AIO
- 65.5% faster than the baseline but with fantastically low variance
- i9-14900K, 870 Evo 2TB/SN750 swap, 2070 Super, 64GB 6000MT/s (2 DIMMs), AIO
- 64.97% faster than the baseline with terrible variance
- 5955WX, KC3000 512GB, 3060 12GB, 128GB 2400MT/s ECC (8 DIMMs), air cooled
- 51.11% faster than the baseline
- 5965WX, 980 Pro 2TB, 3090, 512GB 3200MT/s ECC (8 DIMMs), air cooled
- 55.80% faster than the baseline
- 7950X, 9100 Pro 2TB/9100 Pro swap, 2070 Super, 64GB 6000MT/s (2 DIMMs), AIO
- 61.84% faster than the baseline
I’m omitting thousands of hours of intermediate tweaks, I know some of these aren’t apple to apple to apples, but I wanted to use the data to show some of the lessons learned.
The workload has 8 primary steps but I have broken out specific stages for benchmarking so I could isolate the impact single changes made:
- Matching - CPU
- Reconstruction - CPU
- Undistortion - CPU
- Estimated Depth Maps - GPU
- Geometric consistent estimated depth maps 1 - GPU
- Geometric consistent estimated depth maps 2 - GPU
- Filtered depth maps - GPU
- Fused depth maps - GPU
- Point visibility checks - CPU
- Decimated faces - CPU
- texturing procedure 1 - CPU
- texturing procedure 2 - CPU
For the 8 primary steps, if you use a GPU**, 2 of them are pure all core workloads that will cook load your CPU to 100% and fry it if you don’t have good cooling, if you omit the use of a GPU the job will still run on the CPU, will yield better results interestingly (I think this is because of error correction and accuracy,) but it will consume a bonkers amount of power and keep your CPU at 100% load for potentially hours. 3 of the steps are single core workloads, and the other 3 are a bursty workload for lack of a better term, where utilization flip flops between single core and periods of all core. If you want fast results you can use the GPU for feature extraction and for the point cloud steps, but the output will be…poor. If you want a meaningful reduction in render time without trashing your quality completely, do the point cloud densification and DEM steps on the GPU, VRAM size doesn’t matter, literally as long as you have 2GB free it is fine, bandwidth and core count/core generation/core speed seems to matter more. Something to keep in mind is that your returns diminish rapidly, the 3060 12GB handles the point cloud steps in about 11 minutes, a 3080Ti in about 7, and 5070 Ti in about 6. For runs that take multiple hours, saving time on this step is overshadowed by time savings gained with higher clock speeds and IPC on the CPU, and with better storage.
So to recap that part, CPU matters more than everything else. Single core clock speed matters more than core count, all core load clock speed matters as well but not as much. IPC matters as well. Homogenous cores matter for variance, I wish I could play with Bartlett Lake but Intel said “HA…no".”
On to storage, oh boy storage. Photogrammetry is write intensive and thus destructive to storage. I’ve done a lot of math here and actually created an entire category for ranking my builds focused just on storage endurance because for example, if you ran my test dataset with that i7-14700k on that Evo 2TB nonstop for 5 years, you’d reach the TBW of 8.7 drives. If for some strange reason you used a 1TB 860 QVO you’d hit the TBW in about 4 months. Endurance is king, followed by how the controller handles massive bursts of writes, followed by latency. I tried to see if there was a meaningful difference in sustained write speeds because during the densification step the GPU is offloading gigabytes of data to the storage per second, and the system would write at whatever speed the drives I tested would suffer, but I wasn’t able to clearly say that using a PM1735 was any better than a consumer SATA SSD beyond a few minutes saved over the course of the render. Now on to swap, you gotta have swap, otherwise you are going to be 9 hours into a render watching the RAM consumed creep to 100% and then your job will crash or start chunking if it recovers gracefully. Once the depth map step starts paging, my lord is it just abusing the snot out of your storage media. Go look in iostat for the stuff that consumer flash has nightmares about. I recommend using some flavor of Optane or old MLC enterprise flash for your swap, NVMe is a must, and if you can stripe your swap drives by setting the swappiness the same, even better. It can make a huge difference on how long the job takes to run, once you run out of RAM, which leads into the next bit.
For RAM, from what I can tell, faster is better than slower (water remains wet) but…more slow RAM beats less fast RAM because you can either avoid paging entirely or stave off paging for longer and get bottlenecked on swap for less time. I don’t know if latency makes any serious difference because I lack the variety of sticks of gold to test such a thing. I’m also not sure if bandwidth makes a meaningful difference but I could test that by yoinking sticks out of the 5965WX build. I haven’t figured out a rule of thumb yet along the lines of…”you need X RAM for Y images at Z quality” but I can tell you that at default settings, 1,936 4K images across ~60 acres will eat between 76 and 85GB of RAM during the densification stage depending on the GPU oddly enough. If you get into some of the larger ODM test datasets or some real world flights over a larger area or you start to combine the data from multiple flights together you are going to be getting into the 400GB and up range for RAM consumption. There is a reason I was rocking that dino R630 with 1.5TB of RAM when I was first starting out.
There are other things I can get into, but I figure I will stop here with saying that this workload is actually a good benchmark of a system because the workload stresses all of the components, and incorporates HPC/ML aspects.
I plan to test the 270K Plus or whatever Intel releases later this year with some high speed RAM if I can line some up because I have a hunch that specifically for small to medium scale jobs, it is the only thing that can beat the 14900K, I guess outside of an overclocked 14900K. I prefer my CPUs not fried however so I run everything at the default PLs. From the other data I have drawn on to compare against my real world findings, AMD doesn’t have the clock speed to win, and it doesn’t matter how many cores of Threadripper you throw at it, the single core steps are going to have them lagging behind.
Hopefully all of this helps someone start off with some informed decisions if you happen to work in the same niches that I do so you don’t have to spend as much as I did to learn things the hard way.
-M