Disabled Gamer needs help w/ accessibility in Linux

Software & Operating Systems Linux
,
1

Wendel helped me build a PC a few years back.

I have a disease like ALS and it’s getting more difficult for me to use the computer. I need some help with accessibility in Linux. I need voice recognition which is really lacking in Linux. Everything I’ve seen is pretty bad. Nothing comparing to Dragon Naturally Speaking on Windows. Best solution I’ve found is my Android phone over scrcpy. I don’t want to have to go back to windows… Any ideas welcome. I type painfully slow so I’ll be adding more to this post.

I’ve updated my post. Hopefully it makes more sense. I’m needing to start from scratch. I’m biased towards Linux, but I’m pragmatic and at the end of the day I need what works. Please offer suggestions!

What host OS should I use?

My biggest complaint about different distros and desktops

  • Logging in to the Display Manager with the on-screen keyboard.
  • Logging back in when the screen saver is activated with the on-screen keyboard.
  • Authenticating system dialogs not in the terminal.

This is why I keep coming back to xfce. Gnome in Ubuntu was awful last time I tried when they switched.

  • Assuming Linux, what distro, desktop environment, window manager…
  • What applications?
  • Voice recognition
  • On Screen Keyboard
  • Browser
  • Text editor
  • Terminal

What software I use:

  • Voice recognition
  • On Screen Keyboard
    • Onboard - This is what I’m using. Onboard is pretty great except the text suggestions don’t work gmail in Firefox, telegram or Atom. This just could be my particular Xubuntu.
    • Florence
    • xvkbd
  • Screen Magnifier
    • KMag
    • xzoom
  • Browser
    • Firefox
    • w/ ScrollAnywhere
  • Text editor - These are the only ones I’ve found that support text suggestions with onboard.
    • mousepad
    • gedit
  • Terminal
    • Terminator

My unique needs:

  • I use my pc at my desk in my office, from my bed in the bedroom and from my power wheelchair on the go.
  • I’m a CS student getting into machine learning and more advanced coursework.
  • I switched to Linux 2 years ago.

How I use the computer:

  • I use the joystick on my wheelchair as a Bluetooth mouse to control the computer and my phone.

  • Bluetooth mouse so I can switch between my pc and my phone (Pixel xl). Again suggestions needed. Maybe a kvm or something could be better.

  • On screen keyboard

  • In my wheelchair in front of my desk.

    Office:

    • I have a 2019 65’’ 4k Samsung TV for a monitor.

    • I have a dual monitor arm w/ a 27’’ 1080p monitor and a 21.5’’ 1080p monitor.

    Bedroom:

    • I’d like to use the computer from my adjustable bed. Currently, I just have an old tv, so I need everything from scratch. Displays, mounts, arms, tables…

    On the go:

    • I can’t really type on a laptop keyboard or trackpad, so I don’t really use the computer on the go. I need ideas!

    • The chair has usb power.

What hardware I have:

  • Modest Upgraded Coffee Lake Gaming PC that Wendell helped me build.

OS: Xubuntu 19.10 x86_64
Host: Z370 AORUS ULTRA GAMING WIFI
Kernel: 5.3.0-42-generic
Shell: bash 5.0.3
Resolution: 3840x2160
DE: Xfce
WM: Xfwm4
WM Theme: Breeze
Theme: Greybird [GTK2/3]
Icons: Numix-Circle [GTK2], elementary-xfce-darker [GTK3]
Terminal: terminator
CPU: Intel i7-8700 (12) @ 4.600GHz
GPU: NVIDIA GeForce GTX 970
Memory: 15954MiB

  • 2019 65’’ 4k Samsung TV for a monitor.

  • Dual monitor arm w/ a 27’’ 1080p monitor and a 21.5’’ 1080p monitor.

  • Electric sit/stand desk

  • Thecus NAS

  • Pixel xl

1 Like
2

Hmmmmm…

Seems like there HAS to be something

1 Like
3

Everything I’ve seen is pretty bad. Nothing comparing to Dragon Naturally Speaking on Windows. Best solution I’ve found is my Android phone over scrcpy

1 Like
4

I’m pretty sure @FaunCB was refering to a thread he started

Here

For using Mycroft on his hone, but it was also planning to be a voice assistant.

Example

5

I did find this

I was considering mycroft but it sounds buggy and @FaunCB makes it sound difficult to setup

1 Like
6

I’ve used mycroft. The initial config isn’t terribly difficult, it’s basically git clone, run the executable. And I’m sure you can set up a unit file to have it run when you log in.

Obviously, there are preferences and whatnot to configure, and addons/packages for extra functionality, but that’s a bit of a rabbit hole that can be done once the computer is up and running.

I wonder if Googles stt engine could be leveraged somehow…

7

If you already paid for the software, could you use it in WINE or a VM?

That sounds really awesome! I help take care of a disabled elderly couple that had polio and the times they have had issues with their wheelchair controls (usually wiring/control box issues) have been an ordeal. The only saving grace being that they have very similar wheelchairs so if one is acting up they can share the good one until the other is fixed.

If at all feasible, keeping a spare control on hand might be a good investment. Using it as a mouse might wear it out sooner, and problems rarely ever occur at a convenient time. Just some food for thought.

8

Very good point about wearing out the joystick. I can tell only using a few weeks.

I am getting a copy of Dragon Naturally Speaking 15 on Windows.

Talks about using in a vm. I’ve been trying to pass my mic through to vbox but no luck

9

I’ve updated my post. Hopefully it makes more sense. I’m needing to start from scratch.

What host OS should I use?
Assuming Linux, what distro, desktop environment, window manager…
What applications?
Voice recognition
On Screen Keyboard
Browser
Text editor
Terminal

10

I keep Florence installed. It’s handy and somewhat customizable. I haven’t used others much, just remember it being installed by default in LXLE and it was useful. This along with the other things you are seeking out might take a bit of search engine diving to find what programs others use regularly for accessibility features and find useful.

Most of the other items on your list are hotly contested debates in Linux communities. Since you are already using a distro, I would just stick with that as the bare metal OS and do some distro hopping in VMs. One option is DistroTest:

https://distrotest.net/

I just tried out Zorin OS. They used an on screen keyboard called Onboard which appears to be available in the Ubuntu repositories. It looked nice and a bit more customizable, so there’s another option.

I would start off with Ubuntu based distros since that is what you are currently using and get an idea of what Desktop Environment you like. Try out all of the official flavors like Kubuntu, Lubuntu, Ubuntu Mate, Ubuntu Budgie. Look through what they do and play with their settings to see how customizable they are. If you break a VM, just nuke it and try again or move on. I would use whatever Window Manager they use by default. You can change it, but my experience was a little less than perfect as It causes unintended consequences. From there you can move on to different respins of various distros and different base OSes using the DE’s that you like.

I’m not going to beat on the distro hopping dead horse too much, but the really cool thing is finding new software. Searching through the default programs different distros ship with and various software stores/centers/package managers always turns up very interesting results. I was just looking in Muon (Lubuntu default graphical package manager, like Synaptic) and saw it has an XFCE category. I browsed it out of curiosity and found xfce4-equake-plugin:

Finding awesome stuff like this hiding in plain sight in the repos is one of the reasons I love Linux. The grass is always greener on the other side, but there is vibrantly verdant grass beneath us right now.

For a browser, I’d imagine all of the extensions available for Firefox, and the fact it tends to be default in many distros, would make it a good starting point. I basically feel the same about a lot of programs. Sure, there are some defaults that I really dislike, but there are also plenty of good ones that really don’t differ much from one to the next.

As far as passing through voice to a VM, I haven’t played around with it. It is pretty straightforward to pass USB devices through to Virtualbox guests, so maybe those cheap USB sound dongles would be a good option. I know people using Raspberry Pi’s use this option for various things.

11

Onboard is pretty great except the text suggestions don’t work gmail in Firefox, telegram or Atom

My biggest complaint about different distros and desktops

  • Logging in to the Display Manager with the on-screen keyboard.
  • Authenticating system dialogs not in the terminal.

This is why I keep coming back to xfce. Gnome in Ubuntu was awful last time I tried when they switched.

12

This has been a painful point for a while. They aren’t unified, so even if they come up with a fix for some, others will still have issues. Most people complain about the aesthetics and window size/placement, but I can easily see how it could affect accessibility. Maybe something systemd related will unify it and people can complain about systemd even more, but at least it could give a single focus point for making it work uniformly with various input methods.

I don’t know how well the different display managers work in that regard, but I bet it would be easier to change Display Managers with XFCE than some other DE’s so that might be something to try if you find a DM that works better.

I just tried Onboard, and it wasn’t as nice as on Zorin OS out of the box, but a few minutes of fighting got it to work. I didn’t have the same issue with Gmail in Firefox, so I don’t know why it would act that way.

Vanilla Ubuntu does things to test out new features that other distros don’t or won’t do. I never recommend it. I hear good things about Pop OS for an Ubuntu Gnome alternative. Fedora or Manjaro are probably better for using Gnome than Ubuntu.

13

Yeah I’m trying all 3 versions of Mint in vms. It’s so slow to type it takes ages to reply.

How do you like Zorin? Did you pay?

14

No, I only tried the VM from the link above for about 15 mins. The VM was obviously a bit laggy/choppy, but if you are already familiar with a DE then it can be interesting to see the changes other distros make. I haven’t done it in a while, but sometimes forcing yourself to run things in a different layout than what you are used to can open your mind to certain positive aspects you would never otherwise find. It’s pretty easy to have a knee-jerk reaction to things that are different.

15

I’m really needing voice recognition on Linux. Any ideas?

  • I have a copy of Dragon Naturally Speaking 13. I’d love to run it in a windows vm.
  • I’m using my android phone with scrcpy and using gboard to type by voice on the phone with its mic. Then copying and pasting. There has to be a better way.
16

It’s a niche situation for a niche platform. I’m not aware of anything that hasn’t been suggested already.

I’ll keep an eye out though.

17

Hmmm, I’d set up a FULL pass-through GPU VM (not sure if AMD or NVIDIA is best) and just get more memory to compensate, 4 cores 8 threads should be enough, leaving 2 for Linux. (not sure if it passes them back once VM is down).

Just seems like your trying to run a software VM which is going to be very very slow. A full pass-through VM should allow natural speaking to work flawlessly.

18

Do I need gpu pass-through for Dragon I don’t know if it would use it?

How easy is it to setup?

19

I’m not sure but it would allow the Windows VM to run at near full speed which could help I dunno.

You could also try using a USB sound card and pass it through the VM?

20

That’s probably the best solution. $20 on Amazon/Newegg.

I got this one, but it blocks the surrounding ports, so I got a little USB extension.

It’s pretty good for $8. It’s not hi-fi, but you shouldn’t expect that at $8.