Im building a datacenter

Potentially janky, but: Look at the part number of the fan, then take that to Digikey or similar and see what the naked fan costs.
Can maybe stockpile a hand full of those and “DIY-swap” the proprietary plastic over to the off the shelf part.

1 Like

A few updates:

Power

The electricians came in and redid the power for the building. We now have much fancier panels. I left things offline and did maintenance for the first three days, then ran the generator on the last day to see how it would hold up to sustained use.

I did run into one exciting problem. The system would run “fine” for 30-40 minutes, then it would get itself stuck in a sort of feedback loop. Something would kick on, the voltage would drop, the UPS would switch to battery, the voltage goes back up, the UPS jump back online, the voltage drops… Ill have to tune the generator to run a little faster and see if that does the trick.

Here’s a pic of the new panels:

Switch Swap

The new switch came in. I had a few bumps along the way, but things are sailing smoother now. Mainly issues with vendor interoperability. I assumed that since both the new and old switches were Dell parts, they would play nice, but they run different OS entirely.

I currently have one server plugged into the new switch and the other two plugged into the old one for testing. So long as nothing catastrophic happens ill get the other two switched over, then I can pull the old one and see the damage.

3 Likes

Yeah, I saw some reboots on my VM. I didn’t really test the VPN very much, but it worked for all that matters. The SSH connection is kinda unstable though, it kinda works for a few seconds, then drops, then reconnects after a few tcp packets retransmissions. Didn’t feel much when using the VPN though. Watched some odysee videos, because those tend to be much larger than youtube, and they weren’t buffering and stuttering.

My plan is to use that as a gateway from the internet to my home network to access my files, not to use it as a self-hosted VSP (vpn service provider, i.e. things like Mullvad).

I suppose some maintenance is happening, as the VM is down. That, or the tunnel is down, or the ipv6 of my VM changed.

1 Like

The tunnel is down. I need to log into the router and see what is going on but I don’t have a way to do that remotely.

The funny part is I got the “tunnel ping failure” notification after I had gotten back home from the DC. If it had gone down 10 minutes sooner I could have done something about it.

On the brighter side, the fact that it went down only after I finished means it’s likely not a problem with the new hardware or configs.

I’ll be back in the DC likely Sunday. Will poke at it then.

2 Likes

I also left a ping script running 3 pings every 10 seconds on ipv6, seems like it didn’t help.

I woke up to an alert at 1AM saying the tunnel is back up. No idea what’s up but I’ll take it.

I also have a ping script running. I do think it’s helping, I don’t think this outage is related. Not sure though, will have to check logs.

1 Like

Anyone have any interest in GPU instances? If so, what would you use them for?

Software

I have been working on a user-facing portal for the system, something I can integrate a billing platform with. In order to do this I had to write a basic PHP library to interact with Proxmox via their API. Now that this is done though, I can create and delete VMs from the new portal and have cloud-init auto populate passwords and network configs.

The next step is to figure out SSH key management. Users can input their SSH keys into the application but I don’t yet have a way for the user to select which key they want injected.

So far I have tinkered with Ubuntu and Debian cloud images. I didn’t want to get too ahead of myself, especially since this isn’t ready for production yet.

Peopleware

With the struggle with sales I put out some job ads. LinkedIn and Handshake for starters. So far I have had 4 interviews scheduled, but nobody has bothered to show up yet.
If anyone knows any tech-interested people who are up for some contract customer interfacing, yo.

2 Likes

I have questions, … of a non judgemental kind.

  1. Why PHP?

Reason I’d avoid it would be because it makes it fundamentally hard (well, harder at least) to share resources across requests. For example, if you wanted a pool of pre-initialized connections from your frontend webserver to any backend service it might be harder to maintain. I haven’t used it in a while so I think common things like memcached or database handlers have custom code in PHP, but what about other stuff like if you wanted to stream logs from some script configuring VMs, or running some debugging tool… or maybe you don’t want to do this at all. Hmmm.

The next step is to figure out SSH key management. Users can input their SSH keys into the application but I don’t yet have a way for the user to select which key they want injected

I know it’s early days, but consider supporting custom images and console / virtual serial access from the beginning. For example, you could grab hterm and hook it up to your own simple JavaScript handlers that shuttle data over a websocket to your user console webserver url. Your webserver can check the previous user authentication and shuttle reads/writes from/to websocket over to whatever you have running on the hypervisors that exposes serial ports, possibly another websocket server/service to start with,… although you may want to switch to plain gRPC over mTLS.

Familiarity mostly. The ecosystem is mature and it does everything I need of it. It can’t do everything, but that is what APIs are for. I should post some screenshots of what I have so far…

I do plan on using an asynchronous queue worker though. That will make it easier to scale.

That would be cool. I’d really have to learn a lot more about them, but if people want to bring their own cloud images I don’t see why not.

Luckily this is a thing that Proxmox supports out of the box. I just need to set up a proper reverse proxy for the web socket connection. I’m crossing my fingers that the serial API is as robust as the QEMU and Cloud-Init APIs.


I don’t claim to be an expert, just someone interested in the stuff. I am sure there are better ways to do things.

Some screenshots of the frontend I am working on:

User management:

VM Index:

The email you get when a VM is created:

Don’t mind the password there, it’s a local VM that has already been deleted.

6 Likes

Hi there,

I beg you a pardon, I’m coming to this thread after a long time no longer posting, but, why didn’t you just go with WHMCS? I believe it already has integrations for most of the things you want regarding user/customer-facing interfaces, billing methods and automatic deployment of servers/vms/containers etc.

Best regards,

vhns

As far as my understanding goes, WHMCS is mainly for cPanel servers where I am trying to be a bit more broad. I do resell cPanel services though, so if I get more business going in that direction it could be a great asset!

How’s the datacenter going, I see it’s been a few months, so I hope it’s been humming along well.

1 Like

Not a whole lot to report. Its been smooth sailing since.

I got a job with a pretty great company and just have not had the time or energy to put into the project as of late.

3 Likes

I haven’t had energy to put into my projects either. The alpine VM just idled for a long time now.

Good to hear about the job, stable money sure does help for buying new toys :joy:

2 Likes