PC Controls via mqtt (mqPC)

[JDev]

Hi all! First post is for Devember. I’ve been working on this for a few weeks now and am just getting around to posting on here. Below are the specs I decided for this project which is already underway. Proto-typing, early build screens, etc. coming in follow up post.

This project is a combination of a need to control specific functions on various windows computers via Home Assistant and a desire to learn win32/winforms.

The goal is to create an mqtt client which will allow users to define computer controls and provide a topic/path which triggers the function. For example, for a computer named “ArcadePC01”, one should be able to publish to an mqtt topic such as “ArcadePC01/save-state” and Ctrl+Alt+F1 is pressed on the keyboard, publishing to “ArcadePC01/mute” would mute windows audio, etc. MQTT Payloads will be utilized only when a command has a value, such as payload “5” for “vol-up” would obviously raise payload volume by 5.

The coding is taking place in c# with visual studio.

Feature needs:

• Must maintain a connection to mqtt broker and should be able to connect to server on startup
• Must accept password authentication credentials (no SSL - local server)
• UI must close to tray / Right click tray to close out application
• Must allow exposure of media controls (play/pause, track skip, volume)
• Must allow exposure of mouse controls (move by x, move to x,y, click, double click, etc.)
• Must allow exposure of arbitrary keyboard commands (including modifiers such as ctrl, ctrl+alt, etc.)
• All exposed commands must contain granular control (nothing should be exposed without user control)
• All commands should be nameable / contain custom naming scheme for topic paths
• All control sets should be able to be enabled/disabled
• All control sets should be user savable and should save whether control is enabled/disabled
• All mqtt server settings, etc. should also be saved to the same save file

Nice to have (planned to build now):

• Override default windows styles to contain dark mode
• Volume up/down and mute via keyboard commands all natively cause Windows UI volume UI to appear; desirable to hook directly into windows volume and allow user to decide between keyboard press(ui shows) and ui invisible(for something like an arcade or virtual pinball machine with rotated monitor, etc.)
• Collapsible control detail interface (if implemented, should contain collapse all/expand all)
• Collapse/expand state should be user saved
• Ability to re-order individual control items

Future upgrades desired (plan architechture in advance for expansion):

• Run batch scripts (pre-defined/already loaded on computer - no arbitrary code execution here!)
• Custom macros (with separate UI window for creation of macro)
• Import/Export macros
• Import/Export “control sets” for applications

[JDev]

I probably should have mentioned in the intro post that this is my first windows forms application (actually also my first time coding anything in c#). So this is really just a pragmatic way to quickly learn something new.

While I’m decently far along in this project (2 weeks in), I did save my work throughout the process - so I’m documenting much of this in hindsight but will try my best to keep things in the perspective as if I was posting as I go:

Attached is a mock-up / quick and dirty wireframe for the UI. I mostly sketched things out on paper but I’m going to spare you my illegible child-like handwriting and instead attach a horrible msPaint mock-up I did somewhere along the way fairly early on.

Architechture decisions include using c# PAHO library which is really just M2MQTT. This will serve as the foundation for connecting to my mqtt broker (mosquito) installed through the HASSIO add on which I have installed on a vm on my unraid server.

No other libraries will be used. I will hook straight into dll for mouse control/keyboard controls for fast access and to ensure I’m only “using what I need”.

---

Fun side note, about a month ago, I recently installed the mosquito mqtt integration for a separate project to control an RF fan in our living room. It’s one of those 2 story situations which would require scaffolding to install a different controller on the fan, so the hope was to be able to have basic automation (turning the fan on when temps rise above a certain temp ahead of using AC, turning it off at certain hours or during certain conditions, etc.). Also the fan remote was from previous owner and seems to be in percarious shape and since there’s one switch for both fan and lights, I’m nervous if the remote goes, I’d be SOL.

For this project I used the python paho integration for mqtt (which it turns out has some considerable architechtural differences for how it is programmed vs. the c# paho library). The hardware control is taken care of via a raspberry pi I had laying around. Assuming like most rf projects, it was a 433 mhz remote, I grabbed the transmitter/receiver and was able to get a signal but nothing that made any sense.

And now for the fun part - doing a search on fcc’s website gave me the docs I needed to discover it was a 315mhz remote. I figured the folks here would enjoy a story about how the FCC was actually useful for something haha. (Although I mean… I’d already wasted the time on the 433 controller so process of elimination I was going to grab a 315mhz transmitter/receiever pair either way). That project was quite a bit of fun and involved piscope, building a custom rf script since none of the scripts /tutorials on rf control for RPI were applicable to my particular fan controller, etc. It served as the inspiration for me to build a PC mqtt client to control various machines such as a console emulator, arcade machine, virtual pinball machine, etc.

[JDev]

Day 1

I thought I had saved an earlier build than this mid-day but I guess I saved over it because I was working quickly to make the most of the weekend. This is the end of the first day. There’s nothing functional, but I was starting from not knowing anything at all whatsoever about windows forms so was quite happy with the amount I was able to learn in a single day. Windows Forms/C# development is VERY accessible and easy to learn which was useful for a simple utility project like this.

Tray Icon

I started out with learning how to do the simple stuff in windows forms. This was mostly just laying out the table controls, setting up a tray icon, and hooking into the windows close functionality to override the basic close button.

I set-up a tray icon which you can right click->settings or close via right click->close. Double clicking the tray icon also opens the window.

The icon is a simple gear vector with the play symbol cut out done in inkscape (open source) and exported to ico via gimp (also open source) since I have better luck using gimp for icos. This was created because I HAD to have an ico to use the tray icon functionality of windows forms.

(see status graphic below for tray icon image - merged together since new users can only upload two media items per post!)

Tabs / Status layout

The build directly before this I had used the native win32 tab pane. This would have been a lot more convenient. However, as it turns out, without overwriting the paint, there’s no way to control the style of colors on the win32 tabs (most likely due to the legacy nature where all that styling could be controlled via windows themes).

Since I really wanted the app to be dark mode colors, I decided to use buttons for tabs and just show/hide control panes.

I went with the orange color scheme quite honestly just because it matches the orange and dark green colors in my office (no team affiliation or any reason in particular other than the mid-century meets atomic age vibe I’ve got for decor happened to contain orange accent items).

(ph = placeholder - like I said this is a very early wip)

(There’s nothing in the MQTT “tab” yet)

Control Groups

Below is a screenshot of the first version of the control interface (translating the mock-up and tweaking things for space reasons).

The checkboxes are just for show; I don’t know that I’ll need this many checkboxes but did want to make sure there was a bit more room in case I had overlooked a needed feature.

I’m aware I don’t have an enable button here - I was debating shrinking the path and just having a “check” square button in top left corner, but it was at this point that I realized it’s likely I’ll want to be able to collapse these things and sort them so I went back to the drawing board.

[risk]

Sounds like mqtt-exec (which doesn’t have a ui), but can be used as a building block to do something similar

[JDev]

@risk

mqtt-exec

Yeah similar - but with a lot of built-in functionality to allow for a quick stand-alone set-up which will quickly allow all those arcade/visual pinball machines out their to quickly include functionality without requiring multiple additional tools.

Also without requiring one to go into the source code in order to figure out how authentication works since that’s entirely undocumented on mqtt-exec for some odd reason haha!

[JDev]

1 Week Later

This should be the last retro-active post. The next post will be my “current” status post! Most of the work is the invisible back-end stuff. The UI has made some progress toward where I need it to be but there’s a whole lot of bugs and quirks that need to be sorted out.

MQTT Tab

I finally have the PAHO MQTT integration working how I want. I’ve tested out sending commands from a separate machine and receiving the subscribed topic/payload on the c# client.

Nothing revolutionary with the UI here. Device name is inserted into the topic at first level. This allows one to take the same config from one machine, change the device name, connect, and have a whole new set of topics so that machine can be controlled separately.

Beyond that you’ve just got the basic broker IP/port and authentication fields. Currently it’s set up to require authentication via username but I do hope to eventually incorporate the ability to use SSL cert instead (I’ll consider a no-authentication setting if someone really needs it but honestly think that’s a terrible idea).

Connect locks the fields so they cannot be edited (assuming it’s able to make a connection). I still don’t have UI set-up to indicate when connection fails - currently it just flickers for a second and then the connect button re-appears. (Obviously fake IP displaying but that is connected to the actual IP, etc.)

Non-UI MQTT Functionality

I’ve built in all the desired media control functionality (vol up, vol dwn, mute, play/pause, stop, track next, track prev). None of this is currently exposed via UI. I’ve just got some test topics that are auto subscribed on connect. Publishing to the topic triggers the event. Currently volume isn’t accepting payload (easy to code but just wanted to isolate and flesh out the mqtt stability).

Everything is surprisingly fast! It makes me question why there is so much latency in off the shelf IOT given that I’m getting so little latency it’s unnoticable from pressing pause on phone to having audio pause on the PC. Started off with simple test publishes and then built a simple media control interface in Home Assistant. Notice the “Show Volume OSD” - currently I can get this to be toggleable for vol up, dwn, and mute but not for play/pause, stop, or track next/prev. Fortunately not showing OSD for volume is really all I wanted to target anyway.

All of these buttons are simply publishing empty payloads to topics such as “COMPUTER-NAME/play” or “COMPUTER-NAME/vol-up” (which will eventually accept a numeric payload from 0-100 as well to allow for something like MAX-VOL buttons using the same topic publish).

Here are some screenshots below to help further clarify what exactly what I mean by not showing the OSD on volume controls. These are actually separate topics (“COMPUTER-NAME/vol-up-osd” vs “COMPUTER-NAME/vol-up” for example). Home Assistant toggle is just changing which topic is being published to.

Whenever you use the keyboard controls to change your volume on Windows 10 you most likely have noticed this little volume control display. When the vol-up-osd topic is used, the c# app sends keyboard commands same as your keyboard volume control which displays this OSD. While this was a controversial UI decision from M$ that has left tons of people trying to figure out how to make it go away forever, there are honestly situations where I don’t mind it (it actually will also show the “now playing” card info for media that sends the metadata to windows).

While this honestly can be useful in certain situations, there’s also situations where it is really really undesirable. Consider a virtual pinball cabinet or a dedicated arcade machine where you’re going out of your way to create the illusion that this fun stuff is all magic free of the typical computer UI. You may even have a rotated display configured such that the volume UI will display rotated.

So by having a separate topic without the OSD, I was able to hook directly into Windows Core Audio API and implement the bare minimum set of COM interfaces necessary to directly control volume level and mute status for the active audio device. Manipulating this directly vs. sending the key-press for volume controls will not display the windows metro style audio UI. This allows a best of both worlds situation!

Updated Controls Tab

MQTT media functionality aside, I did re-design the control layout for the UI. I made the interface collapsible, added some basic sort controls, added a status indicator, and ensured there were Enable/Disable and Delete buttons available. Also I styled the form controls as much as possible out-the-box to be darker. Those stupid dropdown buttons are not stylable without user custom paint and I don’t feel like mucking around with that at this point.

The control Enable button serves for validation. For example, duplicate topics/paths are forbidden. Also obviously you can’t have a control with no key actually selected so it will pop up a window alert saying “please select a key” (or “please enter a topic” if one is not present, etc.).

The validation layer tries it’s best to “make it work” when it comes to key mods. The way the UI works, you can select up to 3 modifiers + the actual control key. Modifiers can be pressed by themselves if there’s a need such as just pressing ctrl vs. doing ctrl+c. Since you select the number of mods and they appear blank by default if you were to select 3 mods but only populate 2, it will scrunch it together for you, change the # of mods, and enable. This seems better than harassing users needlessly.

Despite the progress there’s still a tone of work to go. Adding controls works correctly and there’s intelligent numbering, dictionaries being used to ensure resources are released properly / no memory leaks or resources held over. BUT tab indexing isn’t being dynamically assigned correctly yet so I do need to work on implementing that. Likewise, obviously controls should all lock down once enabled to simplify that whole workflow (vs. subscribing and un-subscribing from topics every time a user types a key into path or trying to use focus which involves other potential issues).

Saving the control sets is still a work in progress as well. They’re currently saving to XML fine but there’s some sort issues on saving that need to be addressed. I think I’m actually going to have to change from System.XML to the System.XML.LINQ as it will make the sorting procedures a lot easier to ensure sort order is saved corretly when items are deleted, etc. without needing to over-complicate the storage procedures.

That being said, it’s starting to become “almost functional” at this point. There’s only a few steps left until I can get a usable test build running for personal daily use while I work on developing out more of the desired functionality!

[JDev]

Current Release

This release is my current progress. At this point, I’ve cleaned up a lot of the various threading bugs (paho server running on separate thread from UI, updating status, etc. so need to use safe invokes) as well as some other potential spots for things to go wrong.

The UI for the control sets now lock whenever they’re enabled. I chose to allow the name to be editable even when the control is enabled as that is just for display. (You do have to click the save button to save those changes - the save button graphic adds a little asterisk whenever any edits are made).

The above is definitely not an exhaustive list of all the bug-fixes that went into this release. It really was the bulk of the work. There was a LOT of clean-up. Everything that was sort of working (such as saving control sets, etc.) is now fully functional and working as expected. System.XML functionality ported to the LINQ implementation for easy sort saves. This really helped cut down on the code complexity quite a bit.

There was actually a memory leak in the PAHO code. Someone beat me to calling it out on GIT (almost 2 years ago at this point) but apparently Eclipse isn’t doing a good job maintaing the library as it still hasn’t been pulled into the release. I made some variant code changes to fix the memory leak to the library for my build which have worked perfectly thus far.

I’m going on about a week now of using it straight. It’s configured to start on boot and the program by default opens to the tray with very low impact to start-up and only around 20 megs of ram utilization, even with hundreds of control sets enabled (for testing purposes - in reality I only need like 20). While there’s still tons of work to go to get it “fully functional” I’m happy I have a tool that ALREADY accomplishes what I originally set out to accomplish.

There are two main features implemented that are new: Media Controls (in ui) and Mouse Control. These involved a full re-write of how the subscribe engine worked to allow for a single command to represent multiple commands (with wild-card * to differentiate the topic subscribes).

Media Controls

Media controls are now fully in the UI. Previously I had them thrown in without UI for testing purposes but that was removed. Users can choose to either pick and choose the media functions they want to enable OR they can choose “Full Stack” which will implement the entire set of controls. With full stack, users can still customize the topics by creating whatever topic path they want as a pre-fix/suffix to the core command. I choose to just use the wild card by itself, but wanted to include the option in case there was a potential use case.

For example: media-*-control as the MQTT path/topic will subscribe the following topics:
DEVICE-NAME/media-vol-up-control - Volume Up
DEVICE-NAME/media-play-control - Play/Pause
…

I really tackled this re-write because I want to eventually create extensible control sets where the wild-card may be more useful and wanted to just have it built into the subscribe engine up front. (One of many use cases: retro cheat code control sets in the format of -* where is the game name and * is the cheat code based on a library of code look-ups that trigger the appropriate macros, etc.)

For now though, I’m able to have my 3rd monitor playing the Level 1 news while I work on my work computer hooked up to monitors 1 and 2 without having to toggle the KVM to pause the youtube video whenever I need to take a work call ;).

Mouse Controls

Mouse controls are also now fully functional in the UI. In reality this isn’t terribly useful at the current time unless I build a control client to simplify this. However the nifty thing about that is once it’s working it’s a lightweight method of remote mouse control that’s surprisingly fast.

Really though the end goal here is just to allow the macro feature to have mouse functionality as an option in case there’s something that can only be accessed via mouse. While I don’t have the macro functionality built in yet, I hard coded a few for testing purposes. In testing running auto-hotkey scripts vs. using this, I can cut down on end-to-end latency from push button to desired effect by close to a second. That is very much so the difference between something feeling magical and not being sure if it’s working! So this is a feature that is more so about the “potential” than anything BUT I can control the mouse from home assistant which is nifty.

The implementation here involves 2 “stacks”, 1 coordinate move function, and 4 click commands.
The 4 click commands are: Click, Double Click, Right Click, Middle Click.
(There may be a need for double middle, double right, and separating click-up and click down which I plan to implement as well)

The coordinate move function (Move X/Y) takes a comma separated payload and will move the cursor to the appropriate point on the screen. For example on a dual 4k montior setup, 4040,200 would move the cursor to monitor 2 top left corner 200 pixels from left, 200 pixels from right. Once again a feature that in and of itself isn’t terribly useful BUT can obviously be useful for some macros to for example start up an old school RTS straight into a certain mode, etc.

The first stack is Move (ALL) which includes 5 commands:
Same as media the path requires a single * which will be replaced by the text below
Move X/Y (topic replace: move-to)
Move X- (topic replace: move-left)
Move X+ (topic replace: move-right)
Move Y- (topic replace: move-down)
Move Y+ (topic replace: move-up)

I do plan to eventually add a gear edit to the UI to allow these defaults to be edited on any stack. This would probably be very useful for the media controls and I suspect not so useful here on the mouse which I doubt many will implement unless I build a hassio mouse card (thinking about it).

The other stack is “Full Stack” which implements Move (ALL) plus all 4 click commands.

Room to Grow

At this point I’ve got a tool that I use all of the time that is always present on my taskbar tray. I’m definitely looking to put the pin in this for “now” (I don’t plan to get batch/macro done by the end of devember because of how ambitious my plan is for macro).

Short term:
Custom title bar
Win32 doesn’t let you over-write the app titlebar colors by default. So I’m just going to make the app frameless, put a 40 pixel border on the top and implement my own title bar as it’s just bugging me as-is.

Long term:

• User customizable color schemes
  I realize the color scheme is very very personal. Heck I would only want it on this one pc that happens to be stylized with the orange/green but would not want it like that on all machines either! (I plan to use this on like 6 pcs).

• Batch Functionality (will be implemented such that it’s able to be used in macros)

• Macro functionality (this really is the BIG missing feature!)

• Additional keysets (I don’t have F13 - F2x implemented. Those will probably be useful to have)

• UI updates for Key controls. Dropdown will become clunkier and clunkier as more controls are added.

• Web UI - optional mini-web-server that would allow

• Python Port with web-ui (headless support) - this will require a LOT more work as it needs to be cross-platform compatible

Additional Exploration:
Will UWP allow xbox controls via background? The xbox remote control is very un-reliably slow. Faster remote media controls would be too great an opportunity to pass up. It’s already C# right now, so it would just be a matter of cutting out the features that don’t apply to an xbox.

[JDev]

This is a simple update. I’ve tackled 2 key things here:
1.) Replaced native win32 titlebar with a custom titlebar
2.) Began the work on fully user-customizable UI colors

Custom Titlebar

I just made the main form window frameless and threw a picturebox in a margin up top along with an icon and 2 labels.

This was actually a lot easier to implement than I expected and I fully replace all of the native functions. For the icon I chose not to implement the part where you can click the icon to open the context menu for move/size/close. I may consider it in the future but it seems unnessary.

User Customizable UI Colors

This was a much larger lift. I did not complete this, but I did all of the back-end work and did the first 3 UI colors so I just need to extend it to the 29 other possible colors that could be customizable (button background colors, etc.).

I also intend to allow for user customizable save icons and tray icons.

I realize this is an entirely unnecessary and weird functionality to implement. It’s a tool that should exist in the toolbar for 99.9% of it’s life anyway. So why? Well this actually involved a heavy lift on the back-end for saving and loading the color schemes so it was more about practicing programming. Remember - this is my first C# application at all, so it’s an opportunity to learn new skills while building a nifty, if unnecessary, extra feature for a tool I use.

[JDev]

While this is very much so still early development, I will be trying to get the source posted for this before the end of devember in case anyone has a use for this for their own projects. It will not be terribly documented to begin with due to recent time constraints, but I will check Level1 periodically if anyone has any questions, etc.

[JDev]

jdev-obs/mqPC

I realized that I’ve got a modified version of PAHO source integrated in and eclipse license is not a standard open source license so I do need to read it more. For now, GitHub has the binary (based on the nov 21 post - the custom ui version from later on is still in the works).

The source code is based on most recent version with work in progress UI edits. I’ll try to finish these closer to the new year and update the repository (and will post here as soon as it’s finished).

PM me if you want to know more about the PAHO edits so you can just pull down PAHO C# and make those adjustments yourself (I do ask that you piggy-back on the other GitHub requests for the maintainer to get it together and make the adjustment to the loop - it’s a massive memory leak fix that’s made to the version in the built binary).

[wendell]

@JDev

There was actually a memory leak in the PAHO code. Someone beat me to calling it out on GIT (almost 2 years ago at this point) but apparently Eclipse isn’t doing a good job maintaing the library as it still hasn’t

Omg this is something that was bugging me on another project for ages. It never occurred to me they never updated git in eclipse…omg…

[JDev]

Let me know if you need me to shoot you the lines I edited. I’m pretty sure it’s in an open pull request on the Git but I may have had to tweak it a bit.

It’s a really simple fix and allows it to release memory on reconnects so you don’t wind up with a program that’s guaranteed to inevitably crash over time.

[JDev]

Just saw I got #8 There were some awesome projects in the top 8. Glad to be among them.

To make your verification easier: https://github.com/jdev-obs/mqPC/blob/master/devember2020

Also didn’t clone any pre-existing project. Just used the C# PAHO library for the mqtt communications like one would any library… and you know had to fix their code and try to encourage them to commit the code - I don’t know if they ever got around to it.

It’s actually QUITE a bit of coding that all started once you announced Devember. (Which was in November so I had almost 2 months haha). All of the persistence, etc. is build from scratch and the accessing Windows audio, kbd controls through user dlls, etc. are all without a pre-existing source.

I do need to document the use cases better and create a batch script run exposure (should be fairly easy) and also create an option to break screen saver (because Windows is fussy when screen saver kicks in) and then this should actually become a fairly widely used tool in the emulation community given that as far as I can tell there’s no “easy” alternative for lamen to quickly integrate windows computers into Home Assistant or other mqtt compatible smart home systems.

But the people here at the Level 1 forums have first access to the source to build themselves if they happen to have such a use case

[JDev]

FYI, for anyone curious but concerned that as a single-man program this might not have good reliability, the persistence has been incredibly stable - I’ve had some wifi outages, etc. and the mqPC client has re-established communication with the mosquitto server I’m running on its own consistently without me having to tamper with it at all since some time in November on one of the computers on my network running on the build pre-UI customization. (There haven’t been any changes to the persistence functionality since then since I haven’t managed to find any bugs with persistence which was the most important piece for me personally)

[JDev]

Update: For those interested in this, I have started (slowly) looking at replacing the UI functionality as an Angular build so that the UI components would not be Windows specific.

This was always how I intended on accomplishing the functionality for this (but I wanted to learn C# forms for funsies). With my currrent workload outside of this project, I do anticipate this moving slowly, but I’ll post an update on here once I have a build I can commit to a repository and share.

For now it will still be Windows specific, but I anticipate extending my Python build for Linux after I finish the implementation. I just expect that this will have far more use-cases with Windows machines since so many emulator/front-end combos are Windows specific, etc.

I’m also going to work through getting this to work elegantly with Launchbox. (This should allow for RFID box scan launching of games on Launchbox).

[Joshndroid]

Literally just googled this sort of thing and first couple of results was an L1… therefore here i am as my first result and looking through the thread this is a cool little project. Are you still working on it to much extent (sans me checking Git for recent pushes)

[JDev]

Life has been crazy the last 6 months. I’ve been pulled into about 5 different projects at work so pulling 70 - 90 hours a week right now :(. I’m jealous of the hourly contractor works right now

But let me know if you are able to use this for your purposes (right now it’s windows only, yuck I know) and if so if you run into any issues.

[RiFaT1]

ive been having ome issues with this popup lately suddenly im seeing this popup and my backspace button has become volume controller hortcut and slash or ? key gone locked in just the ? mark whenever im pressing backspace volume going down