The title might be a little sensational. It’s not a datacenter. It’s literally just one room, but I am crossing fingers I can expand into multiple.
Why?
A long time ago (7-8 years) in a far away land (about 6 blocks from where I sit) I started a website hosting business to spite a former employer. Since then it has been passive income. Not enough to live off of, but a few hundred dollars a month for only a few hours of time.
Since the pandemic killed off both of my other businesses (RIP) which forced me to get a “real job,” I figure it’s time to kick things up a notch. Worst case scenario I end up bankrupt, best case someone hires me.
Anyways; I ran the numbers and hosting simply wouldn’t be sustainable if I had to keep paying the AWS tax, colocation was more expensive than cloud hosting, and options were limited. I realized that my dad’s business, a cabinet shop, couldn’t utilize the 2nd floor of the building just due to logistical reasons. It was impossible to move equipment and materials up and down stairs all day, so it was mostly used for storage. I asked him if I could move my business in and lease the floorspace. He was excited about the opportunity and that is where this story begins.
Also partially because I am so far beyond sick of family members telling me all about these “bootstraps” and just “working harder,” so damn it to heck and I’ll make a moon shot. Besides, I am young enough that a bankruptcy wont hurt too bad in the long term.
So… Thread?
I love seeing stories of folks doing neat projects here on the forum and I want to see more of them. I shall be the change I wish to see in the forum, so stand by while I make a few follow-up posts here documenting the progress so far.
This is the place I was moving into. A decade or two back there was an attempt to remodel the upstairs to allow for offices. I don’t remember why construction had stopped, I think there was an issue with bringing plumbing upstairs and the city had an issue with that. Not sure.
Anyways, one of my first tasks was to break down some framing and get a few decades worth of dust removed. There were also a few vacuum pumps and an air compressor that needed segregated, they put some serious vibrations into the floor and made enough noise that I was concerned for the health of any spinning rust. You can see some of the neat seismic dampers below.
Don’t mind that the framing isn’t square. With these lumber prices I couldn’t afford both lumber and a carpenter. I put a door up and if you slam it hard enough and squint your eyes, it looks fine.
I had picked out my room in the corner. It was placed above the main power distribution panel for the building, so it was the cheapest bet for an electrician to run power to. I am starting off with 100A, anything more and we will need to call the power company. This should be plenty to get me up and running though.
Once the room was finished it was time to install servers. Maybe should have measured the rack, but with a few strong helpers we managed to stick it in sideways. It has about 4in of clearance to the ceiling too, had to remove its casters to stand it upright, then put them back on.
A “cabinet shop” is likely to have high-current machinery (e.g. table saws). Depending on how the building is wired, you may have to contend with ‘dirty’ power when those machines are in use. If the power cables to those machines are routed along the ground floor ceiling, you may also have non-trivial interference to contend with. Intermittent data corruption due to induced currents is a PITA to troubleshoot and expensive to shield against.
This is a concern. I got some halfway decent (heh, not really) line-interactive UPS units that I am hoping will do the trick. When the large machines start up the lights dim and often trip the AVR correction on other UPS units in the building, but with enough redundancy I am hopeful.
Quick infrastructure update. The last server for the quorum has been delivered. I forgot to grab rails so I cant rack it up just yet, and I have no power cables so cant turn it on, but its here!
I got some unfortunate news about my network plans.
For some backstory, before I started building I saw there was a new ISP moving into town. They had purchased a building directly across the street from me, so I called them up and asked if they could service my building and explained what I was trying to accomplish. Their salesman’s words were, and I quote, “our network hub will be right across the street from you, we can totally hook you up!” A few months and lots of remodeling pass and I call them back to get hooked up, and they respond to explain that when the salesman said “network hub” then meant “this is where we park our trucks and we have no plans to lay fiber anywhere near you.”
I left to go on a quest for internet. As it turns out the city has a dark fiber junction not 400ft from the corner of the building. The solution is so simple, just run fiber to that junction, get a block of IPs from ARIN, peer with the local ISPs where my customers are, and find a transit provider.
My initial phone calls with utility contractors was promising. To go aerial it would cost about $6,000 (plus pole attachment fees) and to lay conduit about $8,000. Only one problem…
Yup, that’s a state highway. I got a quote to bore under the highway, $15-20,000. I got a quote to use existing poles to go over the highway, $15-20,000. Not necessarily a problem, this is what loans are for, I just need to justify the cost by getting a good transit agreement. I contact the city and ask who provides this service and it turns out the only provider available is the parent company of the ISP who initially screwed me over, and they want over $1,000/mo for transit. I suppose its fair, but not in my budget since ill also be spending $250/mo on the dark fiber lease.
My other option is to avoid needing an IP transit service by connecting to a larger public IX. There happens to be one at a university in a neighboring town. This means I would have to lease dark fiber from my place to city hall ($250/mo) then from city hall to the University of Idaho ($700/mo) then from there across the state border into Washington ($1,000/mo) to connect into Washington State Universities public IX, but that’s even more expensive.
Thank god I didn’t pull the trigger on getting a few blocks of IPs, that would have burnt $1,300 and I wouldn’t even be able to use them.
The silver lining is that with the new ISP laying fiber all over the place, Comcast has stepped up their game. They are offering gigabit service over coax. Not ideal, leaves a nasty taste in my mouth, I wont have my own block of IPs, but it may be my only option. At least until I scale up a bit to justify the IP transit costs.
I’m not sure what your skyline looks like, but if ‘grid power’ is, erm, sub-optimal, and you have lots of empty roof space, you may want to consider covering it with a PV array. Feed that into a large bank of batteries and you can power some (or perhaps even all) of your systems using solar through an inverter. Datacentres have very stable power requirements that can be neatly matched to solar setups.
That would give you a clean power source, and redundancy, both at the same time. Generating your own power might make more financial sense than a boatload of UPSs that need replacing every few years.
Use solar during the day when the sun is up, the machines downstairs are running and the power is dirty. If your battery bank is small, perhaps switch over to (or boost from) mains at night when the sun is down, the machines are off, and the power is clean.
Depending on how large your roof space is, you might be able to generate surplus power during the day. That surplus could be used to decrease the old man’s power bill by powering some of his circuits, or sold back to the grid for a bit of income.
I know solar is not really ‘core business’, but power is needed by every enterprise, you seem to have some nice roof space available, and solar is (virtually) always profitable in the long run. Maybe something to consider?
Hm, interesting. I’m assuming your website hosting business is currently running on AWS.
What kind of traffic is is getting, how much CPU do you need? And can you reduce that? (E.g. replace Wordpress with static site gens? Requires cooperation with your customer though…)
Scaling up doesn’t necessarily mean building your own infrastructure completely, and my gut reaction is I’d say hosting yourself is more expensive for the same performance, at least initially/“on the small scale”.
Instead of using AWS for hosting, have you considered renting colo space, or a (root/VPS/dedicated) server in a “proper” datacenter?
You can also do both, a small Server in a Datacenter(say with 2.5G/10G symetrical) as a front-end caching server, with some more servers “on-site” for the CPU/Storage-intensive stuff.
If you cleanly separate static content, you can think about building your own “CDN” for static assets “in the cloud”(other peoples computers around the world for low latency, you don’t need much per location)
The thought is, If you go that route, you have less strict requirements for you network in your new location - You can easily access multiple IPs from your “front-end”, allowing you e.g. use a fast wireless link as primary uplink, with a secondary regular, more consumer-grade “cable internet connection” as a backup.
Just my uninformed thoughts, feel free to ignore.
This seems like an awesome project and I wish you the best of luck!
I wouldn’t use standard NEMA 5-15R electrical outlets for a data center as in your pictures. Instead L6-30R outlets (240V) will provide 4X as much power… one is often enough for a full rack of servers. L6-30P PDUs and UPSes (6000VA) are common.
If I zoom in on that electrical box shot and squint, it looks like you have a bridged breaker that says 100A on both sides, which would mean you actually have 200A@120V, or if you use similar bridged breakers for the output, you can use it as 100A@240V. Or are my squinting skills failing me? A better photo would help, preferably after taking the cover off.
Didn’t see anything in your plans for aircon, which is #3 right behind data and power. The power all your equipment draws will be the amount of heat you have to remove, constantly. You should reserve 1/3rd of your power to cooling. Assuming you only have one 100A circuit is 120V, that’s 41,000 BTUs/hour, while a window air conditioner is typically 8,000BTU. At a bare minimum, you also want to be able to handle one aircon unit failing.
I like the idea. Budget is tight, but I will definitely keep it in mind. Especially since, as you said, peak power draw will be in the day.
A good chunk of it is, but after having some issues with cloud providers (cough DigitalOcean SFO2 cough OVH nonsense cough GCP issues) I no longer trust anything to just one provider. I have redundant DNS with Digital Ocean, lots of storage and some compute with AWS, my big compute servers are with Linode, and so on.
I did some rather extensive research into colocation before the pandemic. I don’t have my old numbers in front of me but it looked like the low-end was about $100 per month per unit of rack space, plus networking. If I were to have three servers, which is the bare minimum required for true redundancy, that would run $600/mo and not even include any networking. Not to mention that any physical maintenance would require me to purchase a plane ticket, possibly on short notice. These numbers didn’t add up.
I have also rented physical servers from a few providers, mainly DotBlock and OVH. The rates seem to be about 30% higher than simple colocation. The service is great, but no physical access makes troubleshooting real difficult.
Besides, by having my own place and infrastructure I can have “more server to sell” at the same price. Part of my business plan is offering colocation services anyways, since I’ll have the rack space and infrastructure.
This was a hot debate early on.
The building does not have 240V run to it, though we do have a transformer in the building that steps up the power for the water jet. Doing some napkin math, that transformer wouldn’t be able to push enough amps to cover both the water jet and my equipment with a reasonable safety margin. Meanwhile the building is currently set up for about 300A of “normal” 110V power, plus another 200A of 3-phase power (also 110V). More than enough for both companies.
Our decision was to stick with a 100A circuit of 110V for the initial build. Once I outgrow these power demands we will call the power company in to get us a 240V circuit, this way we can decommission the transformer on the water jet and have cleaner power to my systems.
This was just forward thinking. The upstream breaker box (the big one downstairs) is only 100A.
We know that we want to shoot for 240V longer term, but we were not too sure on the direction we wanted to go for cooling. I figured that if we over-budgeted on power then when it came time to get a better cooling system and had the power to do it, we would already have the wiring in place. Who knows though, we might opt for a fancy 3-phase HVAC system and end up needing yet another breaker box.
Look away if you are easily offended.
Short term I am absolutely sticking with a window unit. I did some napkin math on expected peak heat output and I have some wiggle room for ambient heat from outside.
Then I realized that my measurements were bad. You would think that in a cabinet shop we would have someone with a tape measure, but oh well. So, I cut another hole!
Long term we are looking to invest in a heat pump. There is a small patch of land that is on the property where the building does not sit on, but its big enough to stick a pump down in it. Its basically a big heat-sink underground, that uses the thermal mass of the earth to maintain temperature for a phase-change medium.
The idea is to have this shared with the rest of the building. This way its a closed loop and excess heat from my system can help heat the rest of the warehouse in the winter, while efficiently pumping heat underground in the summer.
No idea if it will work, have not planned that far ahead.
Across any two of those three hot wires, you should measure 208V. That’s what “phase” means… the three 120V lines are each as out-of-phase with each other as possible. One is at +104V while the next is at -104V. So you get a higher voltage across two hot wires than you do from any one hot wire just to neutral/ground.
Solar’s good for your use case. The main issue with renewables on the grid is storing the energy generated from the sun during the daytime, as peak usage occurs during the night. Also the main losses in efficiency is in the inverters as switching from DC to AC.
EEVBlog covers a lot on solar if you want to do more research.
Geothermal I heard is good for the summer. Technology connections would argue its good for all year
You know what, you are absolutely right. I had a few assumptions on how things worked.
I asked my dad why we had a transformer for the water jet instead of using two legs of our three-phase feed, and he said its because the water jet needs 480v instead of 240v. I was mistaken there.
Ill definitely shoot to have the 2nd room use 220v. When the time comes I will ask my electrician how feasible it will be to convert the current room to 220v. I appreciate you saying something, I would have been missing out otherwise.
