I have been a long-time youtube viewer and lurker on the forums.
I am a neuroscientist studying neurodegenerative disease. A core tenant of neuroscience is that structure underlies function (or dysfunction). By studying those minute structural differences between healthy and diseased states, we might understand how diseases like ALS or Alzheimer’s manifest and thus build better treatments and interventions.
Recent advances in microscopy have made whole-brain imaging (while retaining single-cell resolution) much more technically feasible, however these datasets are often massive (100s of GB to TB sizes). Below is a sample of some of the whole-brain data acquired from light-sheet microscopy from the macro to micro-scale.
Full zoomed-out view of a fully intact mouse central nervous system with excitatory neurons labeled:
The goal is to systematically extract the structural features from these datasets and compare diseased neuronal structure and location to healthy states. I am using machine learning to confront this problem by training and fine-tuning models to segment those signals from background. Fine tuning and then utilizing models for segmentation requires a large amount of VRAM. A graphics card such as the R9700 with a large VRAM pool would speed up my efforts by allowing me to train with larger samples sizes held in VRAM.
Thanks for your attention, and I’d appreciate any and all perspectives!
Cool use case! My understanding is an early seminal paper in this area is on the “U-Net” from ~2015
The U-Net is still used in various modifications for image/data classification applications. The newest models seem to be a combination of unet style and transformer style (more like LLMs).
I fine-tuned a “SCUnet” for denoising astrophotography images a couple years ago on an 1080 TI GPU with 8GB VRAM. One hard part is coming up with a good clean labeled data set of clean vs diseased images for training, though there may be existing data sets available or likely you have access to that and possibly grad students to curate / label the data haha…
Often you can still train with smaller VRAM by using less batches per epoch or what not. It will just take longer. Once you get it kinda working you can scale up by renting then letting it run overnight etc.
Fine tuning is great for this application and works well as you said with smaller batch sizes for smaller VRAM pools. However, the currently available state of the art models (in my field) are usually trained on a broad subset of data which often times looks much different from whole-brain images. Think less z-depth and way smaller scale, maybe they’re a combination of confocal/light-microscopy/in vitro cellular imaging. This often leads to poor generalization across the many varied domains and scales of interesting features within just a single brain.
I think because healthy brain function is so reliant on both macro and micro structure that this problem necessitates larger context for training data. My hypothesis is that in order to represent both macro and micro scales within the brain you will need to train with annotated datasets of much larger scales to encompass all of the 3D context at both the macro and micro scale. Would love to see it run faster too so we can iterate and improve faster!
Wow I see, that will likely be a lot of data and probably need a little more VRAM just to fit batch size of 1 haha… It’s wild how much data it takes to specify 3d volumes despite the underlying structures being re presentable in small fractal equations…
Interestingly one of the guys who wrote most of the LLM quantization algorithms ik originally worked in medical volumentric imaging for radiation treatment planning. The same kind of algorithms he developed for quantizing those data are very similar for computing inference for LLMs with large tensor weight “volumes” hah…
The folks on this forum could definitely help if you’re setting up some kinda multi-GPU server for your lab or what not. I’m not sure what kind of funding you have or if you’re sharing space with a university or institution already.
I believe there are some kind of “kaggle competitions” as well for medical imaging, perhaps if you had a dataset you could make available other folks would try to build a model for it?
Right now I am running my own consulting gig and working with larger companies to build them custom models, so unfortunately I can’t make their data publicly available. I would love to get the equipment in my hands, build a proof of concept model which performs well on smaller datasets (TB) and then scale up to their datasets which are in the Petabyte range.
