Noob setting up a HomeLab in a culvert, underground, off grid in Alaska

Communities DIY Corner
1

bear
bear1200×712 139 KB

Figured I’d post about my jello “plans” before I implement so that people who actually know what they’re doing might save me some pain. The wife and I live off grid and we had some 12 foot diameter steel culverts delivered by helicopter to our property for use as root cellars. It’s below freezing here so getting stuff underground to avoid freezing is super nice. We have no neighbors around and there is no road access. We’re in the sticks.

One of our culvert cellars has our lithium batteries for our solar and is the power and network hub for the property. It has the inverter, MPPTs, etc and Starlink and we have trenched power and fiber from it to our cabin about 600 feet away and will trench to other structures on the property eventually. A hub and spoke design. I’d like to add an AI server into this culvert and make it into a home lab. The batteries will appreciate the generated heat to help them stay warm.

Edit to add: This short draft unpublished video covers our place and how we developed our homestead and what our goals are and it’s unlisted and we’re not farming for clicks or wanting it to be shared.

But we would appreciate any feedback on it before we start coding. We won’t launch a public channel for at least a year (if ever) and only if our AI (object detection, voice cloning, LLM drafted narration) video processing coding goes well so that we can automate the process to a large extent and focus on just manually filming us building robotic things and coding. If you want to see me spraying a bear point blank in the face (just one of three I sprayed that spring) or the helicopter sling loading an ATV and culverts to our property, or want to see real-life monsters destroying our yurt or to peer down the throat of the bear pictured above, or our solar setup, bacon cooked on a rocket stove, or the cutest baby squirrel ever doing the baby squirrel dance, or the raven who stayed in our cabin, or DJ E the ermine running around our cabin like she owns the place, or cabin insulation or just beautiful mountains and multiple glaciers, you might enjoy this video. Or if you want to see disgusting dirty fingernails, this video is definitely for you (off grid no running water life, represent). All video was taken from our homestead. We’d appreciate any feedback on if the narrative format is interesting enough or distracting or unappealing. Our desired brand for a potential channel is, as my wife says, “nerds in the wild.” Fusing tech and off grid living; automated nature video combined with us building tech for off grid life. We’re not good nerds, and we’re not good homesteaders but we dabble in both. Many thanks in advance to anybody who takes the time to watch and provide feedback!

For our homelab, what I’m envisioning so far:

An AI server:

Will accept a bunch of live feeds from a coupe dozen cameras, run custom YOLO models on each, and run some python scripts to process the videos (currently I have a script running a standard YOLO on videos, producing a JSON for detected object start stop times, then another script using FFMPEG to cut identified objects into separate videos and pass it down the line for future video compilation processing (to make nature videos). Primary purpose of the YOLO is to keep us appraised of the bear and moose, secondary purpose is me toying with automated nature video production. The brown bears destroyed our yurt a few years ago and really set us back but now we’ve got a cabin built but we don’t want to run into them when walking around the property and we want to know where the moose are because they’re even more dangerous than the bears.

I want to also use this server to train my own YOLO models from the footage collected to make it more purpose built. Just starting to work on this now, grabbing frames from collected footage and getting a dataset ready to train on brown bears, black bears, lynx, coyotes, moose, bird types, etc. I’d also like to eventually “train” a model for DIY drones to navigate our property although I’m fuzzy on how that might work. Basically want to feed a point cloud of the property into the model so that drones can use cameras to know where obstacles are.

Also want to run inference LLM models as a backend for a variety of applications (my own python scripts more than likely, maybe using Home Assistant), on some Pis on the network.

I’m currently thinking:

I’d also like to run FreeCAD and maybe Davinci Resolve on this server, and install a KVM switch if possible so that I could put a keyboard, mouse, and monitor in several structures on the property that could use the server to run FreeCAD for design and such. One structure might be a future woodworking shop, another might be a 3d printer at the cabin, etc.

I don’t have much experience with linux or AI. I just got a System76 notebook running Pop and it has a 4060 in it. I’m tinkering with Open WebUI and various LLMs.

I have some Pis. I have our solar power production system running Node Red and sending MQTT messages to a Mosquito server on a Pi at the cabin, and I have a Python script that plays voice recordings on a speaker to keep us appraised of battery status and solar coming in and generator status and such. Would like to figure out a good TTS but for now it’s just playing pre-recorded messages.

I want to replicate this on the other network that has the cameras, and where the AI server would be, to give us voice updates and text messages on moose and bear on the property. And I’m hoping to crack the ONVIF code for these cameras and have them working together to get good footage of eagles and other birds flying around with a 360 spotter camera directing PTZ cameras.

I’ve got an Asustor NAS that I will likely replace with a homebuilt in the future and place with the other server in the culvert but I’m not sure yet.

I don’t know much and have even less experience so any suggestions on how better to achieve my vision are appreciated.

7 Likes
2

Another project for AI (and I’m collecting images from our cameras to get the dataset ready) is to have a solid model for YOLO detection to allow our forest neighbors to visit and keep them de-conflicted. In the summer we normally keep our front doors open and our cabin is filled with squirrels (they have a tree in the center of the cabin although they usually sit on the couch with us or on our shoulders or laps) and all kinds of birds come in, it’s like an assembly line. But the squirrels we have are super territorial so they fight inside the cabin (although during the Spring they’ll do other things inside the cabin so I might start an OnlyFans for rodents) and we have other neighbors who visit like our ermine (we call her DJ E). She visits daily and gets fed very well, but she almost got nailed by a goshawk taking her to go order across the deck and we had a pair of Bald Eagles park on the deck (she chose not to visit that day). We installed a large tube across the deck for her so she can get to the woodpile in front of our cabin and another tube for getting into the cabin but we have to leave the door open. Soon we’re going to be building under our cabin and it will have squirrel and ermine doors with lots of entrances, t-intersections, and cameras so that only one visitor is allowed at a time. So the system will have to use AI to detect and then open and close doors automatically.

This was me feeding DJ E a few minutes ago:

6 Likes
3

Compelling thread title. I thought surely that’s hyperbole, but nope. As a city-slicker the idea of running a homelab in a place that doesn’t even have road access is wild. Good luck with it!

I can’t really offer any advice or suggestions other than consider how you might handle your data backups as I don’t see anything talking about that in your post, but I’ll watch the thread because that’s cool as hell.

2 Likes
4

Bookmarked! I have some experience with off grid networking and “non-standard” deployments. I have done gold mines in the Rockies and landfills. If you have any questions or concerns feel free to ask!

4 Likes
5

That’s one cute neighbour! :face_holding_back_tears:

I’m thinking that maybe you should check that what you want to do is even possible before investing in such an expensive system. There’s a lot of hype surrounding “AI”, but when it comes down to it, LLMs don’t seem to be too useful (yet?).

4 Likes
6

Thanks, I was hoping the title would be. It still feels surreal to be in an underground culvert in the middle of a huge forest messing around with computer tech.

I have an Austor NAS for my data backups right now but will probably replace with a homebuilt NAS in the future. Although thinking I’ll probably just get a healthy SSD on the AI Server when I finally get one, and have the camera footage monitored on that machine and YOLO on all the feeds and produce the JSON for each saved video chunk (with start/stop times for animals detected) and then have the subclips processed and longer videos made on that machine, then dump final products to the NAS. Just for performance.

Great advice, we’re trying to figure that out with our System76 first. Based on our early experience, it doesn’t seem like a monster machine is necessary for our LLM usage, but I imagine it might be for the real time YOLO of a dozen cameras or so along with processing all the videos from the various feeds but I don’t really know. I recently updated one of my video processing scripts to use the NVIDIA but I need to still set up multi-threading in my script as I try to get a handle on speeds and such.

And I still need to train my first YOLO model to see how long that takes. My models I’ll try to train will get bigger over time as we get more cameras and more footage of bears and moose and such so we can keep refining the models.

Eventually I want to have one of the PTZ cameras scan our peat moss bog and give us indications of when the cloud berries and blueberries are ripe too, so that we can beat the bears to them. And I’m dreaming up a tracked robot that will be sent out to collect berries. Got all kinds of cool plans because plans are easy and I don’t have to have any real skill to make a plan!

And I’d like to train our voices locally so we can have our automated video production provide narration with our voices.

And she is a cute neighbor for sure, she’s here every single day. But she is pure predator, my Lord. She scared me a couple days ago. At first she took food from my hand like a snapping turtle, but over time she got to the point where she’d be careful in my cupped hand and just get the offerings. But a couple days ago I had three pieces of raw steak in my cupped handed and she let out a squeak and grabbed the meat like a lightning bolt, far faster than I’ve seen her move before. It was like a frenzy. Scared me. I’m just glad she didn’t miss the steak. It’s only a matter of time before I earn a Darwin Award out here…

I have a system of tubes for her to keep her safe getting across the large deck to the wood pile in front our glass window/door, and a tube from that into the cabin when I crack the door. She has a habit of attracting feathered neighbors. She had a couple of eagles hanging out on the deck a few days ago, she nearly got nailed (pre-tubes) by a goshawk, and of course we have lots of owls. In the forest, we’re the animals in the zoo display. This Great Horned Owl came down to look at the human exhibit and he was nice enough to talk to me, although he wasn’t interested in me escaping my enclosure.

3 Likes
7

Thanks! I’m sure I’ll have some questions for sure. We want to host some websites ourselves as well, but we want to get smart on firewalls first. My wife and I took a networking class together in community college, got to play with the Cisco simulator but that was years ago and neither one of us retained much. Plus apparently there are all kinds of security vulnerabilities in that hardware anyway and of course the powers-that-be make vulnerability a requirement in our “free market.” Anyway, we need to get as smart as we can on networking and have a long way to go. We’re also interested in data diodes for our non-Internet fiber network where we have all of our solar equipment. We do not want that network on the Internet because we can’t stomach our equipment getting hacked and resulting in a fire or something, but would love it if there was a “one way” network method so we could monitor the system performance when not on the property. Seems data diodes are prohibitively expensive for some reason though.

I’m actually thinking about rigging up a Pi on the solar network that operates LED lights, and another Pi on the Internet network with a camera looking at the LEDs, and then programming a script to pass information from the one network to the other through morse-code type blinking patterns.

2 Likes
8

Sure beats Homer Simpson’s “Trying is the first step towards failure”! :smiley:

That’s cool and all :wink: but perhaps simpler - and a lot cheaper! - would be to use a unidirectional serial port connection? You could add an opto coupler for galvanic (electrical) isolation if you like. (Edit: but galvanic isolation would probably be pointless if you’re running your network equipment on the solar power anyway :slight_smile: )

1 Like
9

I like the cut of your jib. I don’t know how that differs from a flux capacitor but you can bet I’m going to research it, many thanks!

10

No worries. It’s simply two computers, each with a standard RS-232 serial port, and a simple “null modem” serial cable between them with one of the leads disconnected or cut. That way only one-way communication is possible.

(And most/many computers actually still have a serial port connector on the motherboard to this day! You only need to get a 10-pin IDC to 9-pin RS-232 connector-on-a-pcie-slot-bracket cable thingy to use it.)

2 Likes
11

Here’s one random offer on Ebay: 2-Port Serial RS232 DB9 Male COM Port to IDC 10pin Cable w/ Bracket | eBay

Note that there are two “standards” on the pinout of those IDC connectors so some trial and error and measurement may be involved. But you need to find the right lead to cut so you would have to check the pinout anyway.

1 Like
12

And a “null modem cable” is simply a (cross-connected) DB9 female-to-female cable.

1 Like
13

Thanks for that clarification. I was already getting lost in my research. This site suggests the 232 is bidirectional, which is why I imagine you said to cut one of the leads (or it comes like that I suppose). It also brings up:

Use of RS422

With unidirectional/non-reversible, terminated, or non-terminated transmission lines, point to point, or multi-drop, RS-422 allows for data transmission via balanced or differential signaling.

Advantages and Disadvantages of RS 422

Advantages of RS 422

  • High data rates
  • High noise immunity
  • Faster communication
  • Can be connected up to 1200 meters

Disadvantages of RS 422

  • Unidirectional
  • Not widely used

I don’t run a traditional computer on my solar network. Just a couple of Pis and routers and fiber converters. I would imagine I could rig one of these connections up on a Pi easy enough if it doesn’t already support it.

14

And there it is. I love it when a plan comes together!

https://forums.raspberrypi.com/viewtopic.php?t=50567

1 Like
15

RS-422 is a more complex multi-drop, differential standard: not what you want. (Although if you want to send data a longer distance you might want to look into RS-485.)

An RS-232 port has one TX pin for outgoing data, connected to the RX pin on the other computer, and one RX pin for incoming data. Cut the RX pin on the solar network side and it’s simply impossible for it to receive any kind of communication. Outgoing communications should still work. It really is a “data diode”.

Raspberry Pis support serial communications, but you need either a level converter or you can use a USB to serial adapter.

Edit: Yep, that’s a level converter. Perfect! (Although I’m sure there are cheaper ones; it’s a very simple circuit.)

1 Like
16

Awesome, thanks. I don’t need to go long distances and I think you just saved me a lot of work and headache!

1 Like
17

Great!

I should add that the way you use the serial port on a linux system is through the /dev/ttyS0 device (or /dev/ttyUSB0 or similar if it’s an USB serial adapter). I’m guessing /dev/ttyS0 already exists on a typical Pi system.

“In principle” it’s as simple as doing e.g. ‘echo “Hello World!” > /dev/ttyS0’ to send the text string “Hello World!” out. In practise some configuration of the port is required, which can be something of a pain. I’m sure you can find some Python library or something that handles the tricky bits for you. Typically you configure the port for 115200 baud, 8N1, no flow control.

Then write some script to collect the data you want on the solar network side and send it out as plain text through the serial port. Listen in on the other side and you should see the text there!

Perhaps there are ready-made “serial port listeners” to grab the text and add the data to a database? Some web searching finds lots of people doing this, mostly using custom Python scripts it seems.

2 Likes
18

I can’t wait to implement this and perhaps share it with others in a future video if I get my AI server set up and put all the pieces together for mostly automated video production. It would be perfect for a possible YT channel we’re considering launching, would be totally on brand, given that solar is so important for most off grid homesteaders and your solution mitigates the risk of technology. Which is what our potential future channel would be all about, fusing tech and off grid homesteading and finding the balance between the two.

For anybody interested, this short draft unpublished video covers our place and how we developed our homestead and what our goals are and it’s unlisted and we’re not farming for clicks or desiring for it to be shared. But we’d love feedback.

Now is the time to get some feedback before we start coding. I’m envisioning multiple cameras around our property, mostly PTZ, with my code using YOLO and other factors (sunset, sunrise) sending ONVIF commands to get great nature and scenery footage and then offering it up ready to go for us with voice-cloned (our voices) narration, so that if we publish a video all we have to do is add substance (building a robot that picks berries, building an automated chicken coop, etc) and we don’t have to spend time filming the nature and beauty that we want in the videos as well. We won’t launch a public channel for at least a year (if ever) and only if our AI (object detection, voice cloning, LLM drafted narration, ONVIF control) video processing coding goes well so that we can automate the process to a large extent and focus on just manually filming us building robotic things and coding.

This video is for those who might be interested in seeing me spray a bear point blank in the face (just one of three I sprayed that spring) or the helicopter sling loading an ATV and culverts to our property, or want to see real-life monsters destroying our yurt or to peer down the throat of the bear pictured in the OP, or our solar setup, bacon cooked on a rocket stove, or the cutest baby squirrel ever doing the baby squirrel dance, or the raven who stayed in our cabin, or DJ E the ermine running around our cabin like she owns the place, or cabin insulation or just beautiful mountains and multiple glaciers, you might enjoy this video. Or if you want to see disgusting dirty fingernails, this video is definitely for you (off grid no running water life, represent). All video was taken from our homestead. We’d appreciate any feedback on if the narrative format is interesting enough or distracting or unappealing. Our desired brand for a potential channel is, as my wife says, “nerds in the wild.” Fusing tech and off grid living; automated nature video combined with us building tech for off grid life. We’re not good nerds, and we’re not good homesteaders but we dabble in both. Many thanks in advance to anybody who takes the time to watch and provide feedback!

2 Likes
19

to go order
to go order1920×1254 160 KB

So our neighbor DJ E had a big day today. Despite the long tube I put on the deck to make her travel safer from the eagles and hawks, she smartly likes to vary her travel patterns. She’s very observant when taking her to go orders back to wherever she lives. But this forest is the real deal and threats are everywhere.

I heard some eagles outside and I went out to see if they were perched by the deck, but I didn’t see them so I figured they must have just flown by. I went back inside and had our main camera zoomed in on DJE under the raised grow bed on her way off the deck. Then I saw an eagle zoom over the deck low toward the grow bed. I ran out there and the eagle ended up in a tree, but our weasel friend was still under the raised garden bed. She was poking her head out every which way trying to find the eagle. I moved a tube over to give her a safe passage off the deck and after five minutes she departed safely. With her steak.

I’m going to trench several tubes into the nearby forest so she can vary her patterns safely from the birds of prey.

3 Likes
20

I managed to finally control my PTZ cameras in a python script. Now I’ve got to build some classes and figure out the best way to code it. Multiple cameras all running YOLO on live feeds. Should give me a good idea of how much it will task the future Server. Ideally, I’d like to run an individualized YOLO model for each camera and have them switch to nighttime models when the sun goes down. Will be interesting to see what kind of resources that would take though.

Then I’m going to look into setting up an Acoustic Localization scheme with multiple microphones, make some kind of YOLO model but for sound (YOHO?) – moose, bears, wolves, coyotes, eagles, etc, and have the nearest camera search for the source of the sound. I also want to get a good 180 dome camera to run a bird YOLO model and slew cameras to track eagles and other birds flying. At the cabin that will feed into some outdoor speakers to play sounds (like an eagle screaming) to alert our visitors when we spot a bird of prey so they’ll know to keep an eye out.

Then have code directing various cameras at various times. One that daily points at a blueberry bush and zooms in to get an idea of how ripe the berries are, having certain cameras position to get timelapse footage of sunsets and sunrises, have the thermal camera check the panels to look for any panel issues, and move from time to time to scan for animals and such.

And I’ve got the script to produce JSON of objects detected and make mini-clips for later video production.