Creating a reading AI with the flaws of the human vision system

For Devember this year, I want to get around to an idea I’ve had for a while but never got around to.

I want to try and build an AI that can ‘read’ (transcribe text from an image), but with all the flaws of the human vision system. I have almost no experience with AI, and I figure Devember is the perfect time to jump into it.

Github link:

Since I don’t have experience with AI (and a decade old computer), I literally have no idea how far I can get with the idea, which is part of the fun. Radical questions require radical tools (which I have a habit of half building) =P

Here is a reference video that can give an idea of how we, human beings, read:
The Unbelievable Science of How We Read - YouTube

On the side, I plan to keep working on my Devember project from last year. 700 pages down, a couple thousand more pages to go ='P

TLDR:

4 Likes

Random Update [2022-12-22] - Drive Crash

My desktop drive crashed, restoring from backups, the last few months of stuff is ‘gone’ (best guess, the partition table got scrambled or something something 0xC00000E9 Boot Error)

Up side, I now have time to blog the hell out of this Devember (while my life is being slowly copied over… yay), though it’ll be more the ideas then actual code until I get things up and running and together and fixed and etc…

TLDR

1 Like

Random Update [2022-12-25] - Backup

Since I’ll be spending a solid week getting my desktop back up to ship-shape (minus a few months of data) after a hard drive crash, I figure now is the perfect time to talk about how I do backups, lessons learned, etc.

The first rule of backups is ACTUALLY HAVING A BACKUP

The 3-2-1 rule of backups is to have 3 copies of your data on 2 different media with at least 1 copy off site. That’s a good rule to follow, but it means nothing if you don’t put in the effort to RUN THE BACKUP in the first place. In my case, I keep two backups, one I run about once every few months (although I’ve been lazy about it and let it slip by a few more months, and am now paying the price), and one once a year so that if there’s a file that I accidentally deleted, it’s likely in the longer backup (In my case, that drive failed sometime in the last year, which I only discovered when I tried to access it [THIS IS WHY YOU NEED TWO BACKUPS, which I had, bacon partially saved]).

Lesson Learned, Backup more often, and check the drives are working more often

The second rule of backups is don’t name it ‘Backup’

If a virus gets on your computer, having files and folders named ‘Backup’ or ‘Important Stuff’ or something as obvious is going to be a target. Don’t do it.

Repair Box

I found it useful to have a folder ‘Repair Box’ full of utilities, installers, ISOs, etc. I keep multiple copies of it wherever I can put it. Every computer I have has a copy on the desktop, every spare flash drive has a copy, every external hard drive has a copy. When stuff breaks it’s very useful to have the tools you need ready at hand. Below are some of the stuff I like to have.

Lesson learned, make sure all the copies of the Repair Box are up to date.

Note: Notice how the folder name has a time stamp? It’s actually really useful to timestamp when the last change was made, to help keep track of how old a particular copy is, or whether it’s “been a while and should probably update that installer”.

Windows Recovery

I use the Windows Installation Media Creation Tool to create a flash drive that I can install Windows from, then I chuck the flash drive with my PC tools and stuff. The tool is easy to use and available to use for anyone with Windows. There is no real reason not to use it.

It does come with a catch though, it can only install versions of windows that are equal or less than the version of windows the Installation Flash Drive was created on. IE: my desktop is Windows 10 Pro, so the Installation Flash Drive I create can install Windows 10 Pro, Windows 10 Home, and lower. BUT if I created the flash drive from a Windows 10 Home computer, I would not be able to install Windows 10 Pro using it. This seems like a small inconvenience, but keep in mind, you’re going to be needing this when your desktop is dead. The last thing you want to deal with when stressed out trying to put your desktop back together is hoping your spare laptop/friend’s desktop/family computer has the same (or better) version of Windows as you to create the Installation Flash Drive that can install the same level of Windows as you had before. Be Prepared.

Windows 7 Backup and Restore

I use the Windows 7 Backup and Restore utility for backing up my main hard drive. It’s the ‘backup everything and the kitchen sink’ approach. It’s simple enough to use (what partitions to copy [with boot partitions being mandatory], where to put the copy, etc). The main benefit is that it can copy files in use (though those files may not be usable) and it’ll copy everything (program folders, windows folders, user setting folders, etc).

Plus, you can also mount the backup file as a removable drive, if you want to retrieve a single file, or like having options for data retrieval. Extremely useful to have options when restoring a desktop.

The Catch is there are a number of limitations. Apparently there is an upper size limit of 2TB for backing up partitions. There are also requirements for smaller sized partitions. Below is an example of the Microsoft Recovery Partition not being big enough to copy. It’s a mandatory partition to backup, you can’t unselect it. You can’t resize the partition easily, if at all, on a running system. Your best bet is to get the partition size right when initially installing windows, which is a pain in the ass as you ‘jiggle’ the windows install disk format wizard to get the right partition layout and sizes (it took a lot of trial and error on a Virtual Machine to figure it out)

Also (If I remember correctly) the system repair disk it prompts you to create doesn’t let you do a fresh install when using said disk. It’s annoying to not have the option. Further, it would have been helpful if the prompt to create the disk outlined better what said system repair disk could and could not do.

The Archive

I maintain a folder of installers for the various programs I use, a readme.txt with some installation instructions, plus any save data (like for games [I still got my SimCIty 4 Rush Hour save!). It’s a bit annoying at times to keep it maintained with modern installers, but it’s proven useful. Especially with older and harder to find programs.

InstallWindows.txt

I keep a text file that lists all the settings I change and programs I install. (I regularly reinstall Virtual Machines, so it comes in handy to have a checklist, no matter the operating system). It’s not all encompassing, but it helps get things ‘feeling like normal’ quickly and efficiently, especially when you are trying to hunt down a particular setting.

Below is a small snippet of said text file:


#user settings=================================================================

#config taskbar
    settings -> personalization -> taskbar
        set use small taskbar buttons to "on"
        Multiple Displays -> Show Taskbar buttons on
            set to "Main taskbar and taskbar where window is open"
#set wallpaper
    settings -> personalization -> background
        set background to "slideshow"
        set folder
        set change picture every "10 minutes"
        set shuffle "on"
        choose a fit "fill"
#set system colour
    settings -> personalization -> colors
        choose accent colour
    settings -> personalization -> colors -> Transparency effects
        set to "off"
#set high contrast
    settings -> ease of access -> high contrast
        set turn on high contrast to "on"
        set theme to "high contrast #2"
#set lock screen displayed apps
    settings -> personalization -> lock screen
        Set detailed to "calander"
        # set quick status to other needed apps
#set nightlight
    settings -> system -> display -> colour
        set night light "on"
    settings -> system -> display -> colour -> night light settings
        set strength to "32" #adjust so that it is barly noticable
        night light settings -> schedule
            set schedule night light "on"
            set to "sunset to sunrise" #may require location detection enabled
#show my computer on desktop
    settings -> personalization -> themes -> related settings -> desktop icon settings
#set size allocated to system restore points
    System Properties -> system protection -> protection settings
        Select system drive
            Configure
                # set to about ~50GB? for drive C:\
#pin folders to quick access
    folder -> right click -> pin to quick access
#change time/date format
    settings -> time and language -> region -> regional format data -> change data formats
        adjust as neccissary
#set multitasking so Halo doesn't glitch out and perminently get stuck 'in the background' when alt-tabbing
    # last checked [2022-04-18]
    settings -> search 'multitasking' -> Multitasking -> Virtual Desktops -> Pressing Alt+Tab shows winsows that are open on -> set to 'All Desktops'

Lesson Learned, keep a copy of this text file in the repair box. Because having to restore a file to set up a computer to be able to restore files to is backwards, and needlessly stressful.

TarBall

Another strategy I use for backing up other drives (IE: not the C:\ drive) is to use 7-Zip and throw the entire drive contents in a tar file on my backup drive. It’s aggressively dumb and aggressively simple. The down side being it takes a long time when backing up a large amount of data (plus it’s unwieldy to manage those large tar files).

The Catch is that you can’t use it on your ‘user folder’. It can’t copy files that are in use, which is understandable. But more frustratingly with Windows, there are a number of ‘Not Files’ in your user folder that are like Windows Hardcoded Legacy Shortcuts that exist enough for programs to see and try to touch them, but don’t exist enough to be able to access them, causing errors.

The Cloud

I don’t like using ‘The Cloud’ to backup data at all, I made a mistake. For example, I don’t want my documents in the cloud, nor could my documents fit in (the free tier) of the cloud; but at the same time, Steam stores the save data of my games (most of the time).

Lesson Learned: since I haven’t been keeping up with my backups, having some of the smaller more dynamic stuff automatically sync to the cloud would have been super helpful. While not being able to backup a large amount of data, being able to have a couple small things automatically synced/backed upped/up to date is an additional layer of protection that I can’t accidentally forget or delay or etc… would have eased my stress significantly. I plan on investigating syncing browser bookmarks, moving some files to Microsoft OneDrive, and using Google Docs more heavily (Like I am now to write this post =P)

Windows File History

I don’t use this at all. I should investigate it further. The Windows 7 Backup and Restore tool is getting kind of old, but I didn’t invest time into investigating Windows File History because the old Windows 7 Backup and Restore tool was working well enough.

Lesson Learned I should have investigated Windows File History more. Potentially it could have completely mitigated all the data loss from a failed drive. Or it might not have. It could completely replace the Windows 7 Backup and Restore tool, or it could have been an additional layer of protection. By not investigating Windows File History in detail, I don’t know its strengths and weaknesses, thus I can’t properly compare it to the strategies I’m currently using.

Having a plan

Having a general idea of what to do is important. Following through on said idea/plan is harder. The point is, not having a plan is a recipe for disaster. Running a backup regularly or choosing which files are important enough to keep in the cloud is adding a layer of protection that significantly reduces the chances of data loss and needless stress.


TLDR: When life gives you a broken hard drive, make a blog about backups

1 Like

Try using git diff to make a backups, but remember to have a starting point for git to compare diff-erences with, both on Win & Lin is easy to set-up;
For the HXD part i will say that Notepad++ has a module, that replaces almost all of the HXD functionality;
By the way, you can automate setting up windows by making a .bat or .vbs file to auto-configure it after or even at the installation, the only thing you have to do when installing Windows (at least <= 8.1) is to unmount installation media drive on program side and reassign its letter to drive C beforehand, then remount and configure system - the scripts will affect the installed system instead of media, but will work with install media environment configuration), other way around is to make a small USB stick, put scripts on it and configure autorun.ini

1 Like

Neat, I’m going to have to try some of those ideas

1 Like

Random Update [2023-01-06] - The Shape of Data: A Primer on some Astrophysics

To understand some of the ideas I want to experiment with will require some knowledge of astrophysics. I’ll keep it simple, mostly because I’m not an expert and am punching above my weight class here.

Energy Scale Large and Small

(Note: numbers are rounded to within a couple orders of magnitude, which is common in astrophysics)

The sun produces ~10^26 Watts of energy
(3.83x10^26 Watts → 3.83x10^26 Joules/Second)
[ energy output of sun - Wolfram|Alpha ]

A Modern CPU uses ~10^-11 Joules to flip a bit
(#Using an 64 Core AMD Epic as a rough estimate anchor; (64_BitInt * 64_Cores * 4_ExecutionSocketsPerCore * 3GHz = A = Bits Per Second;
A Bits/Second / (360 Watts) → A Bits/Second / (360 Joules/Second) = B Bits per Joule;
1 / ( B Bits/Joule ) = (1/B) Joules/Bit)
[ https://www.wolframalpha.com/input?i2d=true&i=Divide[1%2C\(40)Divide[\(40)64+*+64+*+4+*+\(40)3*Power[10%2C9]\(41)\(41)%2C360]\(41)] ]
(Note: A CPU also moves data around, so that one Bit Flip also includes potentially storing that bit, moving that bit around, loading the program to say to flip a bit, Operating System overhead, etc, etc, etc)

A Modern Transistor uses about ~10^-19 Joules to activate
[
https://www.researchgate.net/publication/256546316_Activation_energy_values_from_the_temperature_dependence_of_silicon_PN_junction_reverse_current_and_its_origin ]

Storing a Bit to DRAM takes about ~10^-12 Joules per Bit
[ https://www.seas.upenn.edu/~leebcc/teachdir/ece299_fall10/Vogelsang10_dram.pdf ]
[ computation-v-memory.jpg (602×398) (eetimes.com) ]
(Note: These numbers are from a decade ago, so there is some guesstimation to get a modern number)

The Minimum Energy required to Flip a Bit in the Universe is ~10^-23 Joules (per bit erased)
[ (1.38Power[10,-23] Divide[Joules,Kelvin])(2.73 Kelvin)*ln2 - Wolfram|Alpha (wolframalpha.com) ]
[ Landauer's principle - Wikipedia ]
(Note: there are ways around this limit, like using ‘reversible computation’, or waiting for the Universe to ‘cool down’ in a few billion years. In short, to understand the limits of what this universal limit says, you must understand how the theory was constructed and from what fields of physics is draws from in order to figure out how it can be subverted, or where it applies)

The Minimum Energy required to Flip a Quantum Bit is ~10^-34 Joules (per quantum bit orthogonalization)
[ Margolus–Levitin theorem - Wikipedia ]
[ The Mathematics of Quantum Computers | Infinite Series - YouTube ]
(Note: again, there is some nuance to this number, as a ‘bit’ and ‘quantum bit’ are fundamentally different, and can’t be mapped directly to each other, nor do they behave similarly. I recommend watching the PBS Infinite Series youtube video about it, I find the explanation given is much better and fuller then the usual simple explanation usually given [It really helped me better understand the concept years ago when I first watched it]. Also, the theorem is an upper limit of computation speed per energy, rather than a lower limit of energy per computation, so subtle but important difference when trying to rearrange the numbers and inadvertently changing it’s meaning, which I have probably done here)

The Minimum Mass required to Compute is ~10^50 Bits per Second per Kilogram
[ Bremermann's limit - Wikipedia ]
(Note: I just learned about this just now, it’s neat)

These are some interesting numbers, but the main thing to remember is that computation takes energy, and moving information takes energy. Plus it helps answer some questions.

IE: If you could brute force the decryption of AES-256 encrypted data, how possible is it? Welp, let’s assume decryption is a single operation like a single bit flip, therefor it would take ~10^77 bit flips, resulting in needing ~10^54 Joules used, or about the entire energy output of the sun for One Billion Billion times the age of the universe. (or a mere Billion times the age of the universe if using quantum bit flipping orthogonalization =P)

IE: If a modern CPU can do ~10^14 Bit Flips per Second, That would require a minimum of ~10^36 Kilograms of ‘Matter’ to be the ‘CPU’, which is less then the mass of a proton. It’s not that it’s possible to build a computer the size of a proton and as powerful as an AMD ThreadRipper, it’s that our current understanding of physics don’t prevent us from the possibility of building that Proton Sized ThreadRipper =P

(…I spent waaaaaay to much time writing this section)

Random Walk of Stellar Photons

A fun fact about the sun, it takes a photon 100000 years to move from the core of the sun to the surface of the sun to escape the sun. In short, because the sun is a GIANT BALL OF PLASMA, photons are constantly bouncing around until they escape, this movement is called a Random Walk. There is a specific property of random walks I want to talk about.

The one dimensional case. Let’s say you have a number X, and you flip a coin, heads is +1, tails is -1. As you flip said coin, you are constantly adding and subtracting from X, you might go from 0 to 1000 to -100000 and back again. The important point is you’ll eventually end up back at zero. Specifically, the more you flip the coin, the more likely you are to end up back at zero at some point. IE: the random walk in one dimension approaches the origin the more random coin flips you take.

The two dimensional case. A similar setup, except you have an X and Y axis, and you are flipping two coins for the X and Y axis respectively. The same thing happens in two dimensions for a random walk, you always approach the origin the more random coin flips you take.

In the three dimensional case, things are different. You have three dimensions to move around in randomly. The random walk this time approaches an infinite distance from the origin the more random coin flips you take.

This video covers the idea much better than I can (Thank you PBS)
[ What is a Random Walk? | Infinite Series - YouTube ]

In short, the path of a photon from the core of the sun is called a Random Walk, and because statistics, it will eventually move away from the core of the sun and leave the sun, due to it being a 3 Dimensional Random Walk. A Random Walk in one and two dimensions will always return to the origin (eventually).

String Theory Dimensionality

The general idea of string theory is that everything in the universe is made out of tiny vibrating strings (incredibly tiny, like trillions of times smaller than the smallest particles in physics small), and how these strings wiggle and vibrate determines what particle they are. The part I want to talk about is that these strings vibrate in 10 Dimensions, 9 spatial dimensions (give or take a dimension depending on the specific flavour of string theory) plus 1 dimension for time. We live in a 4 dimensional universe, 3 spatial dimensions (X Y Z axis) plus 1 dimension for time. The way strings can vibrate and move in 9 spatial dimensions is that 6 of those spatial dimensions are ‘compacted’.

Think of looking at a power line running through a neighbourhood. To you, looking at it from a distance, it’s a one dimension line. But if you were looking at the power line from the perspective of an ant walking on said power line, the ant would see two dimensions, 1 dimension along the line that they could walk forwards and backwards on, and 1 dimension that wraps around the power line that they could move clockwise and counterclockwise around. That second tiny ‘pac-man’ dimension is a compacted dimension. Even though it’s extremely tiny, and wraps around, it’s still an extra direction of movement and freedom for the ant to move through.

Similar idea with string theory, there are 3 large spatial dimensions a string can move through, plus 6 extra extremely tiny pac-man compacted dimensions that the string can wiggle in.

This video covers string theory better than I can (Thank you again PBS)
[ What are the Strings in String Theory? - YouTube ]


TLDR


Random Update [2023-01-12] - My Hard Drive was Framed

Welp, my desktop is being unstable again, in the exact same way it was before the hard crash a few weeks ago. Considering I swapped the hard drive with a new one, I suspect either my motherboard is failing (note: I did some tests blah blah blah and discovered a dead-ish SATA port [Alive enough for the BIOS to detect a connected CD-Drive, Dead enough for Windows Disk Management to not see it]) or there is some weird critical bug with how Windows interacts with my DECADE OLD motherboard, or something I never thought of. Either way, the best (and least stressful) course of action is to replace my motherboard/CPU/RAM. (Though my wallet will not be happy about a surprise necessary computer component shopping spree)

To be clear, it’s a minor bug that could possibly fix itself with a reboot, …but I won’t actually know for sure until I reboot my desktop and it doesn’t boot along with a (probably) missing partition table. I also won’t be able to try some recovery options on my new and old hard drive until I shut down my desktop and swap the drives and use the repair tools from the Windows boot disk; resulting in a chicken and egg problem on multiple levels.

I could be wrong, with everything being fine and the random minor glitch i’m seeing is just a minor glitch that solves itself via reboot; but just in case I’ve spent a good week figuring out how to shuffle my data around then actually shuffling my data around with more data shuffling expected in the future (along with spending the previous week attempting redoing/queuing up as much things as I could remember I did from the period between my last backup and the crash [MY SPRING CLEANING!? all my spring cleaning got un-spring-cleaned =S).


(Note: for those curious, as far as I can tell, except for the glitch where non-boot drives are showing zero activity in task manager (after restarting task manager because of a glitch where it was showing nonexistent ghost activity on non-boot drives), everything is showing up fine. I can access and read ALL my data on all connected drives without issue or corruption (as far as I can test and tell). Resource Monitor is showing all activity properly (including the activity on non-boot drives having the issue in task manager). Nothing unusual in the Windows Event Logs. Windows Update has been paused (after installing ALL updates) for a solid week or two before the issue showed up. All drives are showing up properly in Disk Management. My desktop is behaving exactly as it was a few months ago [my desktop isn’t exactly rock-solid to begin with] before any hint of the issue showed up. I ran SFC and CheckDisk when I reinstalled and updated Windows on the new drive without much issue. Frankly, I’m running out of ideas for tests, and I’m hesitant to run (and rerun) more intensive and deeper tests for fear of pushing my desktop ‘over the edge’ of whatever is keeping it afloat. This is all assuming this is an actual issue that will hard crash my desktop again)


TLDR: The hardest part is not knowing till it’s too late, Schrödinger’s Partition Table

Random Update [2023-02-05] - Derailed Devember

Welp, Devember got derailed, life happens. I fully intend to continue this project, once I ‘get my shit together’. I still managed to learn a fair bit, just not the AI stuff I intended.

TLDR

I leaned about: MS Paint 3D, Microsoft File History, the limits of bookmark sync on Microsoft Edge, Microsoft installation media command line recovery tools, network file sharing/transfers, a bit of PyTorch AI stuff, a bunch of other random stuff, AND I’m swapping out my damanged motherboard; Yay

Hal fixing a light bulb (from Malcolm in the Middle S03E06 - Health Scare) - YouTube

1 Like