Building a supercomputer/cluster computer?

Communities Blog
1

I currently have 2 computers at my disposal: my main PC and my server. For the most part, my server is just idling, hosting a local file server. What I'm planning on doing is to utilise the unused power of my server for things like faster code compilation and backup compression for archiving as well as bragging rights :)

There are loads of youtube tutorials on how to do this with raspberry Pi's and, since I'll be using Debian for both my server and main PC I shouldn't have to change too many steps.

I'll update this thread once I've made some progress.

UPDATES:
Day 1: backed up all the data on my server (only about 600GB or so), reinstalled/upgraded to a 64 bit version of Debian (was originally in an old P4 system, now and FX-4300), restored the backup and optimised storage configuration due to a switch from ext4 to xfs and configured some basic functionality.

TODO: actually set up the supercomputing software, finish setting up core server functionality (home file server and PXE/tftp server)

3 Likes
2

Nice, do keep us updated :) Pinging @FaunCB for server idea.

3

Before doing anything, just gotta backup the server so I don't nuke everything accidentally. This may take a while...

4

Nice - I currently have my servers 24 threads thrown at Seti@home currently as its a bit OP for what I currently actually need it for.

5

Alright heres what you do. Go buy a well sealed shed. Not some bullshit I'm talking like its basically a house. Cut some holes, 6 or 8, and put window air conditioners in them. Then you get a dual fan unit like this for the ceiling:

http://www.target.com/p/holmes-slim-window-fan-hwf0522m/-/A-14384113?ref=tgt_adv_XS000000&AFID=google_pla_df&CPNG=PLA_Home%2BImprovement%2BShopping&adgroup=SC_Home%2BImprovement&LID=700000001170770pgs&network=g&device=c&location=9051962&gclid=Cj0KEQjw1cS6BRDvhtKL89em1oIBEiQAtZO5x3pDf_nrdpvsYNrAPJcNZAGNtTlXCWBioo0HlWN73Y4aAiUV8P8HAQ&gclsrc=aw.ds

Fuck that I found a triple fan unit. Hang that shit from the cieling, right? AIR IS KEY ON THIS. Now your server housing is separated. No leaks, no dirt, and if you can you have it on a separate breaker, k? THEN you go out and get a fuck load of powermac G5 A1117's (M9592LL/A or M9749XX/A), maybe like 20? (I lied do like 60), all with PCIe (THIS IS FUCKING KEY YOU NEED PCIE ON THIS SHIT) and a BUNCH of 9800 GX2's (the best GPU that could be handled. They're about 30 bucks I think?) and EITHER do gentoo OR Ubuntu Mate. They are the only things I would bother with.... Well maybe debian.... Idunno do debin its not an asshole... THEN you install Java, C# libs, C++ libs, and the latest python... throw flash in there too..... and samba that shit together. Essentially all this should work in a giant RAID now. THEN (IT GETS FUCKING BETTER) you get a desktop server... 2 desktop servers, and fill one with SSD's and just give the other like, I dunno, a boot SSD and a 1 TB drive? Gotta be SAS or the newest possible SCSI you can do. So the SSD loaded one you have as your content hub and the second moot one you have as you buffer K??

DON'T STOP READING AAA. Get fibre channel, YES FIBRE CHANNEL, and link all this shit together. THEN once that is done put handbrake on all of your PPC systems. Once that is done you need to make sure you can SSH into all of them at once... I have no clue personally how to do this though I imagine some sort of hub could be used? Get like a pentium 4 or 3 from a garage sale and throw arch on it or something.

AAAAAAA SHIT I FORGOT TO SAY GET ECC MEMORY IN THE G5'S OR THEY WILL FUCK UP CONSTANTLY. SHIT. Max out the ram too plz.

THEN get a bunch of core 2 machines. Have that shit lined up for core related tasks and web setup. All still connected via samba, mind you. Have some machines handle apache2 or nginx and have some machines act as high bandwith routers and buffers. These might not need to be fibre channel? But if you can do it TOTALLY DO IT.

THEN get a couple new servers (they don't have to be a billion dollarz but enough power to do like 800 VM's all in one group) and set up a XEN group (XEN is a corporate VM utility / server environment) and link that shit up to the G5 group. Install linux OR OSX to all 800 of the VM's and link that shit up.

THEN make a website service and offer streaming and video production VIA VM's through KDenLive in linux or Adobe in OSX (or kdenlive I guess but only I am that stupid) and have all of your maxed out 2.7 GHZ ALTIVEC LINUX POWERED G5's render videos via handbrake and shoot that through the fibre channel samba connection to host the videos on your SSD based server and have the other server handle firewall stuff.

Hell throw another 4 air conditoners up by the cieling of your shed and put in 6 more servers to handle DDOS reflection.

You now have youtube with VM's set up like rabb.it for video editing, storage, and hosting, with 800 rental slots.

THERE YOU WANTED AN IDEA THAT SHIT IS WORTH GOLD.

EDITITITTITITTITIT: DAMMIT GET UPS'S TOO

SHIT I AM FORGETFUL ON THAT STUFF

6

Sloooooow down there mate! This isn't some production grade supercomputer implementation, it's literally just my my main PC and my file server/NAS sharing computational power. Maybe when I have a few hundred grand to throw around I'll think about it :)

2 Likes
7

I'M SO BORED HELP ME

I might die

8

maybe @Logan has an idea

9

The "supercomputers" you hear about aren't used like a normal PC, and it normally involves writing custom software built using libraries like MPI or OpenMP and the work is run in batches as resources become available. It's not something you're going to find useful for general purpose computing, as there aren't any mainstream operating systems designed around distributed compute architecture. Plan 9 is probably one of the more well known attempts at developing a distributed compute operating system, and people are still hacking away at it's forks but who knows if it will ever get mainstream use.

For distributed code compilation though you could look at distcc or icecream which is a fork of the former. I haven't done much research into distributed backup solutions. Pretty much any kind of distributed workload you want to perform is going to take application specific support to implement.

10

MP as in CHALLENGERMP?

11

You can missuse continous integration tools to offload work.
Never worked with jenkins.
I like team city.

12

Sounds good to me. As for the backup solution, I already have rsync set up for that, all I'm looking for is some kind of distributed compression program for an extra boost.