Hello Levelone techs community! I have a piece of software i have “Vibe Coded” that is made to allow users to cluser a bunch of physical machines, and it is designed to try and make the barrier to entry into technology like this as easy as possible, while also allowing users to cluster a bunch of machines they have handy and getting more use out of their hardware! It is far from complete, and I have only just started on it. But would love to hear some input on how to proceed with this. The project is completely open source and will be available for anyone to look at. I am not a developer at all and cant code at all. But I love open source technology and would love to make something that anyone can benefit from! Or if other solutions exist that are newbie friendly i would love to try them!
Whilst the enthusiasm is to be commended, i have to ask - what are you trying to do that isn’t already covered by an existing project?
What are your design goals and why would i consider your project? What’s the current status? Which hypervisor are you using?
Be aware you’ve embarked on a very complex project, what you’re attempting to do here is very much not simple, but if you’re trying it as a learning exercise, go nuts!
What do you mean by cluster? Theres different kinds.
There is no hyper visor for this software: it is a general purpose harness to allow multiple physical computers to cluster for different workloads. You install it to a linux system of your choice (Fedora is what i use but added support for ubuntu based distros as well) its supposed to be a lower point of entry. You just install the software, set it up and run programs in a cluster. No vm’s , no difficult network setup (static Ip’s or the like. You just plug in systems together by ethernet, thunderbolt, usb4 and just let them go at it! I tried to make it to be the easiest setup as possible to just fire up a cluster of machines and go. Code removed from Github.
It is a piece of software made for physical machines to cluster them up. I couldn’t find alot of options that made this easy to use and just fire up and go. Originally designed with LLM’s in mind, but made more as a general compute cluster, without the complicated network setup you have to do with server os’s like PROXMOX and VMWARE. It could not replace these options. But it is made to be just an easy point of entry for people who want to just play around with it, or maybe just set up a simple cluster of phsyical machines to work on things with! Code removed from Github
I’m glad you posted this because last night I was conceptualizing exactly this configuration of a LLM work queue.
I have a small handful of systems that are capable of running different size models and thought it could be useful to have a team of LLMs which could collectively work on the same projects.
Edit:
Run one LLM cumulatively across every machine
Not exactly, then. How about many LLMs collaborating on a larger task?
Are you actually a software developer, or did you just “vibe code” something without any experience?
If its the latter, I would recommend people stay away from it, whatever it is.
Using AI can make you more efficient at things you are already an expert at, but it should never be used to substitute expertise, because then you don’t have the necessary expertise to catch things when (not if) they go off the rails.
There is too much vibe-coded AI slop code out there already, and almost without exception it is a security disaster, as inexperienced “vibe coders” can run code and feed back error messages into the model and keep doing these iterations until it works, but even when using models to “code review” the output, they rarely have a good security focused architecture, and almost always have some pretty bad gaping security holes.
This seems particularly bad when clustering up multiple machines together, presumably over a network.
This article from a few months back seems relevant here.
Your condecending attitude aside: Like previously stated in in the title. I did vibe code it and said from the start i am not a developer, and it is completely open source. I couldn’t find anything that suited my needs without having to take a huge leap into the realm of enterprise/ server software. So i messed around with a LLM and tried to make something i couldn;t find a replacement for. The enitre project is made to be open source if you wanted to take a look at it. It isn’t made to be deployed in a corperate fleet, nor is it designed to be a professional pice of software. Maybe i should have put this in a homelab topic rather than a Dev corner….
My bad, I didn’t mean to be condescending. I am not an expert at coding either.
I just think it is important to point out at any available opportunity, that hvaing access to AI assistance does not replace the need for expertise.
Any use of AI is best used in topics that the user already fundamentally understands well.
I’m not sure I understand your usecase very well. For generic distributed inference it seems to use llama.cpp RPC thing but you still need to optimize the networking portion of your stack to get the optimal performance out of it.
For generic distributed computing, there are better options out there depending upon your usecase.
As long as you know what your code is doing then great!
Historically this sort of thing has been called a beowulf cluster. I wouldn’t say it’s been done as a way to make it easier to get started, but otherwise yeah it’s exactly the kind of thing you’re describing where someone puts together a hodge podge of spare and under-utilized hardware in order to take on bigger tasks than they could otherwise. I guess I have a soft spot for using this kind of project as a stepping stone for deeper investigation into distributed computing and computer science in general because it’s exactly what I did years ago when first getting started.
If you want to do the same starting with an LLM, and then pick through the parts it hasn’t got quite right, then cool.
It is like the @redocbew said, i am trying to do something like a beowulf cluster. Just have a piece of software that you install on any machine you have, and can run distributed workloads (whether they be llms or video renders) on old hardware you can get cheaply or already have lying around due to this ongoing RAM crisis and whatnot. What are some options that you know of? I couldn’t find any open source options that has a more “Plug and play” approach. I have seen a few tools that are all command line driven or Proxmox. But when you are first starting out. It is a huge undertaking for someone to learn an entire operating systems ontop of the command line structure. I want either to try and make an open source project that is newbie friendly, or if one already exists, i would love to play around with it. I would love to see options similar to what i have cobbled together here if they exist! I only relied on an LLM because i could not find anything like I want to try.
For generic inference, it’s llama.cpp’s RPC. The same thing your project is using.
For other usecases, I’m not sure either. But I can assure you options exist although they aren’t newbie friendly so to speak.
MIT OCW might have lectures on clustering and associated things if you want to learn about it.
Yeah, it gets complicated quickly, and that’s probably part of the reason why you didn’t find much.
Also, I think maybe you’re mixing High Availability where a virtual machine can transparently switch to being run on a different host if the other one dies with clustering where the execution of software really is being distributed across multiple hosts at the same time, but that’s all stuff that’ll come out in the wash once you start digging in.
Your original response wasn’t the point of my post or related what i was asking, but understand your views and concerns. But i really appreciate you backstepping and taking another look at it. If you know of an active open source project being developed that is intended to make clustering for newbies easy and to get their feet wet I am all ears.
I really appreciate this info, it is really helpful! But i am kind mixed minded currently in my journey. I want a bunch of my old computers sitting in a closet to work together to solve an array of different workloads (thinking just one massive program crunching numbers on a fleet of computers, no migration or any of that, maybe fault tolerant if they die like availability but without all the networking stuff you have to know ontop of it. Whether it be llms, or something that requires alot of compute power like rendering or something like that, and I also want it to work entirely offline if need be and not need a router or a managed switch to get in the way, and with a gui. Command line is a great fallback. But most user facing programs have a GUI these days, and having one is just easier to work with when you first start out. I know it is a big ask. So that why tried my best with the tools i had to get this lol but i would love to hear of other software that might fill this need!
OK, I mean cool, but you should try to understand what the LLM is doing otherwise you end up with this:
def _write_icon(path):
"""Icon generator: orange rounded square with bold CR letters — stdlib only."""
import struct
import zlib
W = H = 128
BG = (232, 163, 61) # orange #E8A33D
INK = (40, 26, 10) # dark brown letters for contrast
# letter placement: CR centered, each glyph cell scaled to 12px, 7 rows tall
scale = 12
glyph_w = 5 * scale # 60 per letter
total_w = glyph_w * 2 + 2 * scale # letters + 1-cell gap
x0 = (W - total_w) // 2
y0 = (H - 7 * scale) // 2
letters = [(_GLYPH_C, x0), (_GLYPH_R, x0 + glyph_w + 2 * scale - scale)]
def letter_pixel(x, y):
for glyph, gx in letters:
lx, ly = (x - gx) // scale, (y - y0) // scale
if 0 <= lx < 5 and 0 <= ly < 7:
in_col = (x - gx) % scale
if in_col < scale and glyph[ly][lx] == "1":
return True
return False
rows = []
for y in range(H):
row = b"\x00"
for x in range(W):
r = 20
inside = (r < x < W - r and r < y < H - r) or \
((r <= x < 2 * r or W - 2 * r < x < W - r) and
(2 * r <= y <= H - 2 * r)) or \
((r <= y < 2 * r or H - 2 * r < y < H - r) and
(2 * r <= x <= W - 2 * r))
if not inside:
row += bytes((0, 0, 0, 0))
elif letter_pixel(x, y):
row += bytes((*INK, 255))
else:
row += bytes((*BG, 255))
rows.append(row)
raw = b"".join(rows)
def chunk(tag, data):
c = struct.pack(">I", len(data)) + tag + data
return c + struct.pack(">I", zlib.crc32(tag + data) & 0xffffffff)
ihdr = struct.pack(">IIBBBBB", W, H, 8, 6, 0, 0, 0)
png = (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", ihdr)
+ chunk(b"IDAT", zlib.compress(raw, 6)) + chunk(b"IEND", b""))
with open(path, "wb") as f:
f.write(png)
The whole build.py is pretty wild, but why the heck would you start handcrafting a png in python?
And did you forget to check in the function/class definition of ranch? I see a lot whole lot of from ranch import but no def ranch.
Sorry to say, but this code is pretty unmaintainable in it’s current state and also from first glance not functional.
What are your needs exactly? What application you need to run as a cluster? Clustering a hypervisor or a database or a render farm or whatever else is not the same thing
I did not code it myself, as i stated in the original message (except fo the local llm bit). I am not a coder/ developer and tried to feed information into a smaller local LLM to make what i couldn’t find. But i appreciate you explaining this to me. I like to learn about this stuff,. But also if you are aware of something that can acheive what the LLM failed to do i am all ears! I Am trying to find an open source program that can distribute cluster workloads to a stock of old machines or on any physical machines i choose that doesn’t require the whole complicated network static IP stuff or with installing a whole other operating system to learn on top of a program. I have change the title of this post to specify what exactly i am looking for. If you know of open source alternatives that exist i will gladly try them!
I am trying to find a piece of software that is in a better state than the one my LLM coded to do to do the following: Cluster a bunch of old machines together to accomplish workloads such as LLM hosting (Through lamma.cpp with a easy to use GUI) or make it to where it can handle workloads for other pieces of software as well wiithout learning a whole new operating system or dealing with all the complicated networking once it gets to a certain size, kind of like a plug and play with a gui instead of command line. If you know of any open source software like that I will gladly try them!