To preface, I was using my 5 year old dual core pentium Asus laptop with a whopping 4gb of ram sporting windows 10 patched up to date.
I have been working on a pet programming project for the past week, my language of choice is python 3.6
the project involved a lot of compression using LZ4 which I enabled with multi-threading, finally today I got to the point in the project where I could point the program at a file and let it loose.
to start off with I tried compressing a few random files of varying formats which resulted in slightly larger files, this was expected to a certain degree so I thought maybe I should try some larger files.
The compression ratio of LZ4 is supposed to be approximately 2.7 at best so I decided to throw a 750mb avi file at it and then on the prompt I see ‘… file found, file encoded in base64’, then …nothing, laptop was completely unresponsive.
from what I can tell throwing a 750mb file into memory must have decimated whatever ram I had left beyond the capacity that the swap/page file could handle due to the garbage cpu.
I had to do a hard reboot to get the pc back operating … aaaand half of the project is gone.
learn from my mistakes especially before trying something potentially risky always save your progress or even create a backup if possible even more so if you are about to do something potentially risky on a 5 year old laptop.
TLDR: tried to run alpha scalable compression software on a garbage laptop which resulted in the loss of about 9 hours of coding.
Snapshots are bae. BtrFS is the way.
That’s a very strange way of spelling ZFS
I mention it because it is built into some distros of Linux. Fedora 25 and 26 have installation options that are drop downs that allow you to have the root drive be on a BtrFS partition.
And Linux Containers (OS based Linux virtualization) have a flag for putting the whole container in a BtrFS subvolume.
I haven’t found any other built-in functionality like that for ZFS in any of the Linux Distros I’ve tried. The only thing I’ve found that has the option to use ZFS built into it to begin with is FreeNAS or NAS4Free.
If all you want is snapshots that work and are relatively simple to manage, BtrFS can do that.
So… github?
that they are however ,Windows 10 is strictly NTFS as far as I am aware, I think microsoft are slowly rolling out an alternative in the enterprise which is more reliable eg. more redundancy etc.
I dont think there is any need for a project of this size to have its own github repo, i should have just created a local and a NAS copy.
Are you telling me that you didn’t save once in 9 hours? Or that it flipped out and erased it? Git it, even locally.
I didnt save the project in its entirety in 9 hours, I saved individual modules but evidently missed some or they were somehow deleted
Easy way to keep a backup is Briefcases on Windows.
They’re disabled in Windows 10 for some reason, but you can re-enable them with a registry edit.
A briefcase is a fancy folder that syncs files between two locations on a system when it sees there is a change. It’s meant for people who use a laptop and regularly connect/disconnect from storage sources, or work on the same project on two different computers without a common storage location like a server, but it can be used to back things up in situations like this where files may get overwritten on one storage location but not another.
From that web page I linked:
You can use Briefcase to keep files in sync between two different computers, even if the computers aren’t on the same network. If the computers aren’t on the same network, you can copy files from one computer to another by using removable media, work on those files using the other computer, and then use Briefcase to sync the changes back to the original computer.
So you’d create a briefcase with a corresponding one on something like a flash drive, external HDD, or network server via a share. Then put all your project files in it. Any changes on either side (server or client) will update to the other side.
This can cause issues if somehow the wrong side has the newer date information causing the old to overwrite the new accidentally. But that’s hard to make happen unless you intentionally change a file’s metadata to be the wrong date.
Just a thought.
1 backup is better than none, 2 is good, but 3 is ideal. If I were you, I’d use a briefcase for an automated instant backup, have Windows backup your project folder to an external HDD (separate from the briefcase target location) weekly, and manually copy your project to a flash drive every month or so.
But that’s probably too much to bother with for a personal project. Up to you. Just my 2 cents.
ERROR! Wrong way of thinking.
Everything should always be backup up. Github is not an exclusive club, anyone is free to use it for public repos. Everything from serious huge projects to dumb shit or just a directory full of README’s.
By integrating Github, or any other SVC, into your project workflow you drastically improve its reliability and version control.
My first error was not backing up everything prior to each test however I have learned that lesson the hard way, since its just a pet project im not sure if I really need version control
I may just write a script that does that and just adds a 1 to the file names to avoid overwriting the same file eg. ( dir / all files), (dir1 /allfiles1)
At that point Github is less effort and better features.
Just make your repo private if it bothers you. https://help.github.com/articles/making-a-public-repository-private/
I may as well give it a shot, I’m not really worried about people ‘stealing’ my work as I’m self taught and my code would probably drive any professional dev crazy, that said if the project ever goes anywhere I will throw a link on the forum.
In principle the program is for people who have rigs with multicore processors eg i5,i7,i9 & Ryzen, but at the same time suffer from shitty network architecture eg being stuck with a sub gigabit lan due to having to use a shitty isp provided router/switch/ access point all in one solution , the idea being instead of taxing the network by trying to transfer hundreds of gigs over a network that cant handle the bandwidth, the program uses lossless compression to automatically compress the target file (multi-threaded) and automatically send it to another pc on the lan which decompresses the file, using sha512 hashes to verify file integrity.
In short it is lessening the strain on low bandwidth Lan by pushing the work to the processors lessening the bottleneck.
docker has zfs driver, as does openVZ
you’re right though, zfs is much better integrated with BSD jails/containers than linux at the moment.
I mean zfs snapshot pool/volume@date isn’t exactly complicated.
You also get boot environments, which can be a huge help in more severe development screwups.
also, @redking please just use a VCS, svn or git both work fine
Neat. Didn’t know that.
That’s cool. Is this for Linux, Windows, Mac, or a combo of those 3?
I’m just saying, rsync does this and it’s niiice. If you could get something equivalent on Windows, that’d be good. I’ve never found an equivalent to rsync on Windows.
They exist, they’re just behind massive licensing paywalls. People know you can make way more money on it on windows.
currently im developing it on windows however im not using any libs that should prevent it from being ported to linux easily( maybe one or two changes to directory and file creation given that the directories in linux are structured differently eg move the base directory to /home, file creation as well could be slightly modified to deal with not requiring file extensions in linux eg using bash: touch to create new files instead of the hamfisted win method.