kinda reminds me of the Brett Kelly / 45 drives example of CEPH working despite drives being pulled
3 Likes
haha, true XD That video was great!
2 Likes
I managed to land an ok programming gig at a web development shop. Lots of nice people but it’s glorified data entry.
I’m crossing fingers I can set aside about half my income to try to re-start this project. I am cautiously optimistic that I can get over the initial hurdle of the uplink costs if I sign a 5-year contract.
The only thing I am concerned about is who my customers are going to be. At one point I figured I could get by with website or game hosting, but after doing a bunch of market research that seems like a dead end. Anyone who knows that they need server space already knows a DigitalOcean droplet is $5/mo and I am running 10 year old hardware.
My expectations for humanity are just low enough that I feel like I might have better luck trying to sell server space to an amoeba.
3 Likes
Ouch. I was split between going with a VPS from you, or getting an openbsd.amsterdam VPS. I was hoping your data center business would be successful.
What I see to be a potential growth for VPS enterprise in general is DIY VPN services. Going with Wickr or SquareSpace is better than anything else for web hosting right now. But some people are skeptical of VPN Service Providers and may want to do their own. Or VSPs may get their IPs banned on certain streaming services, but a DIY VPN could get around that, because the traffic is small enough to possibly not trigger IP filters and is unlikely to be added to a monitor.
I personally would not underestimate the power of this forum. If you want to find customers that don’t mind old hardware, I think some of us are willing to get a VPS from you, me included (probably starting from December or January I think).
3 Likes
I’d second that, if you could have competitive (or at least ballpark with linode) I think that there are plenty of people that are looking for a small vps provider
3 Likes
Business offsite backup maybe. Servers can be old so long as the drives aren’t and you have sufficient redundancy and failover built in. Performance doesn’t need to be great. Competitors (backblaze for instance) are still pretty pricey when you get into dozens or hundreds of terabytes. Also, cost is relatively easy to calculate based on MTBF and warranty.
2 Likes
The trendy thing is to get in on mining crypto coins. Perhaps you can find someone with gold-rush fever and a bankroll.
Data centers typically just offer space in a rack (with power and internet), and let customers bring in their equipment. I would think medium sized local businesses would be interested in that… Better than servers in the office, and less travel time to a distant data center.
But for that to work you do need staff on-site (or very close-by) 24/7 to answer the inevitable remote-hands requests, UPSes and generators backup for all the servers. It’s certainly not an easy business to get into.
1 Like
One of the other fine people on this forum mentioned Scaleway Stardust instances - for 0.5/month.
… 1 vCPU, 1 GB of RAM, 1 IPv4 address, 10GB of local storage, up-to 100Mbps Bandwidth …
Actually they’re 2.5/mo but if you drop the IPv4 they go to 0.5/mo
Then there’s these free tiers:
e.g.
Oracle Cloud Free Tier
* 2 Autonomous Databases, 20 GB each
* Up to 4 instances of Arm Compute
* 200 GB block volume
* 10 GB object storage
Google Compute Engine
1 e2-micro instance per month
...
Google Cloud Run
2 million requests per month
360,000 GB-seconds of memory, 180,000 vCPU-seconds of compute time
1 GB network egress from North America per month
So… you get 5 hosts to ssh into for free + a few containers that can host a blog or redirect stuff
For most small workloads, slow CPUs or the fact some got preempted for CPU for 50-100ms basically doesn’t matter.
…
So my point is, knowing you can’t do free, can you do cheaper than digital ocean for example or cheaper than scaleway …
… or can you do specialized workloads for super cheap? e.g. hosting unifi controllers (they’re basically java web app + mongodb + whatever for graphs) . You could take $5 for 5y of hosting for 128MB of ram and still make a profit probably
2 Likes
I’m not sure how profitable that would be, unless he’s getting a lot of customers. Doing it cheaper might be an option, but I would suspect giving perks like free subdomains he owns for the same price as a VPS might be a better deal (because you only have to pay, say $2.5/month for both a VPS and a domain name, if that fits your needs).
I’m rather interested in an “Internet gateway” and an internet facing reverse proxy in a VPS. Since I can’t easily do stuff at home anymore, it seems really worth to spend some money on a VPS. I am willing to scrap ipv4 if I at least get a NAT for that, just in case some websites are still ipv4 only.
1 Like
I’m too stupid to give up and find myself with lots of free time. Medical issues prevent me from being terribly productive, but any progress is still progress.
Some updates:
This morning I racked a new server and unboxed a fancy new switch to play with. I didn’t have the brainpower needed to configure anything, and I need another UPS to power anything safely anyways. Still, it’s like Christmas opening up those boxes.
Those refurbished drives are not doing so hot, have had several failures so far. I’m going to need to order more soon.
Since my ISP does not offer IPv6 (shame on you Ziply), I grabbed a tunnel from Hurricane Electric. The real amazing part about it is I can failover the tunnel endpoint in the event that one ISP falls over. This is really fantastic, because I’m still connected to my dads business internet in case mine fails.
In short, redundant uplinks with the same IP space, without needing an ASN/IX-Peers. That’s the plan at least, work in progress.
I’m also thinking about doing a free tier. Most of the other big companies offer a free tier, or at least credit when signing up. Now realistically speaking, looking at my own servers utilization, the cost of a “personal” server is negligible. Plex, Nextcloud, ZNC, VPN, heck even Minecraft…
So, I’m thinking of offering a small server for personal or open source usage for free. 1CPU, 2GB RAM, 16GB Disk, an IPv6 /64 and 10Mbps uplink.
Notice no IPv4. If you want that then you gotta pay. Same for the uplink speeds. Shits expensive yo.
For monetization my plan is to focus on commercial and professional. I am hoping that I can attract the kinds of customers I want with the free tier, and if I do good they might remember me when it comes to more beefy requirements.
I have been working on some marketing materials. Nothing to show off just yet, but I have a tentative deal with a live-streamer buddy of mine. It’s nothing special, I’m basically “sponsoring” their Minecraft server in exchange for a shoutout. It’s not great advertising, but I’m not paying for it either.
One of my ideas for marketing/support/whatever is to maybe set up a Discord server. I don’t know if it’s just me, but I absolutely love seeing the “behind the scenes” stuff that other hosts put out. There isn’t a whole lot of “opsec” stuff, why not have a medium where I can interact with folks more freely? Besides, when it comes to downtime (when not if) it would be pretty neat to be able to post pictures and describe the issues. Plus, I’m sure I’ll get some great pointers from the community.
I have found both as the host and the customer, that when there is downtime it’s absolutely infuriating when things break with no explanation. On the flip side, if the host goes “yo, we see connectivity issues, hang tight” then suddenly the attitude is completely different.
A buddy of mine and I talked a bit and came up with the idea; Why not give customers the option to help choose what new equipment and upgrades to buy next? If I was going to throw money at infrastructure and some folks are saying things like “disk IO is suffering” - then I have a pretty clear direction I need to go.
8 Likes
Anyone have any ideas on last minute things to check before opening things up for testing?
I have been running some production stuff off the stack for a few weeks now. It’s blazing fast, but a fiber connection to a box down the street is kinda cheating.
I still have some things to hammer out (like the IPv6 tunnel) but I think it’s in a pretty-OK condition for a first launch. More upgrades are coming soon, but if I don’t start now I might never start.
4 Likes
Have you done any testing for your redundancy? Power, cooling, storage, networking, backups, etc? While you don’t have much production traffic these kinds of tests are much easier to do.
4 Likes
Go for it…the MuleShed is real…https://muleshed.com/ and we have our first paying customer actually, $10.00 a month for 1 Cpanel account. I believe you CAN do it…very inspirational! People thought I was playing when I started www.liftedmoving.com but I make $80.00 to $130.00 an hour just moving junk and furniture around, plus I love lifting heavy things and get a paid workout. Life is good!
2 Likes
Not sure how to emulate stability testing. Servers aren’t always going full tilt, so try to run a benchmark on a server intermittently, unplug the power and plug it back in. Then switch PSUs. Unplug a UPS and plug it back in. Then switch UPSes.
Then unplug a switch and see if things are still working. Then plug it back and unplug the other one. Not sure how you are connected to ISPs. but if you don’t have a redundant connection yet, at least make sure that your connection doesn’t die if your main fiber gets disconnected or the switch is powered off.
You should test the power generator weekly for about 1 hour.
Finally, do a DR test. Make some test VMs, pretend they are your customers’ services. Unplug your main hypervisor host’s Ethernet cable, see if HA works. If you aren’t doing HA, which would be understandable, not a lot of VPSes do that, preferring to actually have a sane recovery tactic via snapshots (duplicating VMs is expensive), then just test and see how fast can you get a VM back up and running.
If you are doing snapshots every 4 to 6 hours and transferring the snapshot to your backup servers or replicating it to another NAS or hypervisor (depending on how your network was setup), you should be able to recover in less than 30 minutes. If you’re using something like ceph, it should be even easier.
2 Likes
More Updates
Power Delivery
Just got word that the office needs to plan a 3-day shutdown. They are redoing the wiring in the building and need to kill the power. On the plus side, now is the chance to up the voltage and pull more power into the room.
The downside is… well… 3 days of being offline. I have a generator, except for the price of gas. Bad timing. At the end of the day $200 in gas isn’t the end of the world. Ill post a separate update with generator testing, it needs some love before prime time.
With generator testing in mind, I flipped the breaker just out of curiosity. Servers and gear worked just fine, but I realized my fiber modem thingy isn’t on a battery. This is what testing is for!
Networking
I am struggling getting the IPv6 tunnel working. I don’t think its anything wrong with the tunnel, its routing is just complicated.
I did install my new (previously enjoyed) 10G switch. Its not configured just yet but its in the pipeline.
Discord
I set something basic up. I would love to chat with you guys there, if you are interested in what I am doing here. I love the forum format but for quick chit-chat a dedicated chat server works. Once you land say hi and I’ll give you a role so you can see the “infrastructure” channel.
I’ll still be posting fun updates here, no need to worry.
Public Testing
Speaking of Discord… If anyone wants to help me try test things out feel free to hop over there. If you have a project that needs hosting, I would love to work with you.
At this point things are pretty “set” hardware wise. There is a lot of room for improvement, but I have upgrades in the pipeline. A 10-gig network backbone, for starters.
There are some limitations of course. No uptime promises, might need some flexibility to shuffle IP addresses around, the rack could just burst into flames, or I might just need to shut down because the power to the building is cut. You know, normal things.
6 Likes
I’d like to help you test, but I am boycotting discord, so oh well.
Still, good luck on your endeavor.
GG Testing. This is what it’s about, plan for failure and don’t let failure plan for you.
MTU issue or asymmetric routing? (Since it sounds like you have two tunnels up?) The former may cause issues if you’re blocking ICMP globally (most firewalls can do stateful ICMP which helps here), and the latter will kill most firewall state tables if its not designed right.
1 Like
Best tip I can offer on generators is to avoid the GFCI outlets they put on them for 120V output these days. UPSes (or is it the servers?) do a good job of tripping them even at pretty low loads. Won’t work worth a damn… Meanwhile, you can get L14-30p adapters that break-out to pairs of 120V 5-20r outlets that’ll work nicely:
1 Like
I think I have tracked it down to a L2 issue.
I have a /64 and a /48 routed via a HE tunnel. Using the /64 I can ping the remote endpoint from my router, and get out on the IPv6 “open internet” just fine. I have not tested connectivity to the /48 just yet, but I assume that since the /64 works that the /48 will as well.
I set up a new VLAN for this network, trunked that over to my main switch, added the new VLAN tag to the switch port, then added that VLAN to the servers “client” NIC. I created two test VMs and assigned each a /64 from the assigned /48, neither can ping each other or the router, even from the same host.
For example:
- 2001:abc:def::/48 routed
- 2001:abc:def:0::1/64 assigned to router
- 2001:abc:def:1::1/64 assigned test1 vm
- 2001:abc:def:2::1/64 assigned test2 vm
This narrows the problem down to a few options that I see.
- The problem is on the hypervisor, I can test by bringing in a laptop and plugging it into an “access” switch port.
- It could be the switch, but IPv6 is L3 and the switch is L2, little uncertain here
- The problem could exist between keyboard and chair, which is most likely
Just to be clear here, your firewall/router needs an IP in the same /64 as your router. This part is identical to IPv4. It sounds like you’re just giving a new /64 to each VM, without adding an IP to your firewall for each?