[Devember 2021] [Complete!] MFD-Thingy: Send keystroke to game from HTML page

[Nic_s]

Can we have more than one Devember project? Let me know if not and I’ll update the thread.

My idea is to have a little HTML page running on your PC which can send keystrokes to your game… or any application that is currently focused for that matter. Some sort of MFD Thingy… but with HTML. In fact, I’m calling it MFD-Thingy or Multi-Function Display Thingy.

The idea here is to access the HTML page over your local network from any device that can run a browser and have controls like buttons you can press which will translate into a key press on the main system.

This is not a new idea, stuff like this has been around for a long time. Most recent example I’ve tried is Game Glass and it just about works on my old little Samsung galaxy A3 (Don’t want to ruin the battery on my new phone). It has a free tier and you can create your own little MFD with the free tier. You can also share your designs. It’s pretty cool and worth a look IMO.

However, after all the google searches I have yet to come across something that runs as a single app/service utilizing hosting a HTML page. If you know of such and example, please share.

How does Game Glass work and why do I want to make my own?
Game Glass, and most such applications, require you to install a app/service on your PC and another app/client on your phone or tablet. The client sends your input from the phone or tablet to the service on your desktop which then simulates a key press.

I want to have just the service and have that service host an HTML page which I can then browse to from any device that has a browser on my local network.

The plan:
Since there is only a few days left for Devember, I’m going to focus on a working proof of concept for Star Citizen and not a complete, user customizable solution. My Gui-Fu isn’t great either and it’s often what takes so much time for me. My day job is shoveling data from point A to point B, not making pretty guis

I will create a little HTML page with a couple of buttons and when you press one of those buttons it will send a Ajax request to the back end which will then simulate a key press on the machine it’s running on. Typically this will be your gaming rig.

I’ll be using ASPNET Core as I can compile the HTML page, javascript, CSS, and C# logic into a single exe. There’s almost certainly a better way, but for now I just want to make it happen. Please share if anyone has ideas.

I’ve already done a very dirty test. Here’s a sneak peak sending a button click from Chrome on my main rig to the service running in the VM. I have tested this from my phone as well.

ToDo:
Create proper project
Create HTML front end with some buttons and basic theming.
Create script to handle buttons press events and ajax call to API.
Create C# logic to translate buttons into key press.

Notes:
-In the before times you could use SendKeys function in Win Forms. At least I think you could use that to send keys to any application. Anyway, this does not exist in ASPNET core. I found a library called H.InputSimulator that works which will do just fine for a proof of concept.
-I don’t want to enable someone to inject script and just send what ever keys they want. For now I’m going to have the buttons represent an Action and then translate that Action into a key server side.
-I also want to limit when a key will be simulated. Basically only simulate a key press IF Star Citizen is running and IF it’s currently focused. Will include notepad in this limitation for testing purposes.

EDIT: GitHub - Nicks182/MFD_Thingy: Web Based Multi-Function Display for Sim games... or any games

[Nic_s]

Proof of concept complete.

Basically worked on this all day. Most of my time was actually spent on simplifying the ASP NET Core project so it only has what is needed and then figuring out how to embed the static files like the HTML page, CSS, and js. It seems embedding the files like this is not something many do, but I think this is so cool and I love the flexibility of HTML.

As of now, it’s a ASP NET Core project which has a API controller and then an embedded HTML page with a javascript ajax call which is the first thing you see if you browse to it. I used JQuery thinking that I will have use for it, but not so sure at the moment. Left it for now.

Currently I publish it as a self contained exe which means I don’t need to install any .NET frameworks on my main rig to run it. I should be able to publish this to run on Linux as well, but I’ve not tried that yet… don’t really have a need to at the moment

Actually learned a lot today with this and it’s resulted in some ideas for stuff and things I can do at work.

Right now I have a couple of hard coded buttons on the HTML page and you can see what it looks like below. In terms of delay/lag, I can’t tell a difference between pressing the key on the keyboard vs pressing the button on the html page on my old crappy phone.

Sorry for crappy video quality, was the only way I could think to show it working as intended.

This is the proof of concept I set out to build and as such the main goal is done.

That’s not to say that I’m done working on it as I do actually want to expand on this and make it more configurable. I want to be able to combine actions so I can do more than one thing in the game with one button. For Example, turn off ship power and then exit the ship. Currently this requires pressing and holding 2 different keys, but with this Thingy I can have it be just one button.

I’m not that big into SIM games and I may only ever use this in Star Citizen, but I think this is pretty cool either way and I want to build on it some more.

[paulwratt]

This one of those projects that begs the (philisophical) question “It can be done, but should it be done”, so i am glad of your Notes: and the fact its a proof of concept with some “clamps”. I’d hate to see 20,000 of these popup over the next 12 months because someone figure out how to force it onto a platform.

FWIW: there was someone on the BSD kernel mailing list that was bent out of shape for 20+ years, because you could do this by default on ANY command line. He finally got it patched out a couple of years ago. THANK GOD he was unable to convince the Linux Kernel developers to do the same. It has never been exploited ever, and is in fact a base functionality, and I was able to use it recently from a version of BASIC which does not have access to system output (or input in this case).

Anyway, I am glad you got it finish, its projects like this that originally made me realise just how powerful a web page can be, and how much more creative ideas can be pushed with such a simple “interface”.

BTW by keeping the JQuery in, you will find lots nice “interface design” examples to help spruce up the GUI (amoungst other things).

Cheers

Paul

[Nic_s]

I have been working on expanding this a little. I’ve changed gears and I’m now working on a WPF app that hosts the HTML page and API controller. The WPF app double’s as a control and config interface. So the HTML will only do the most basic part which I’ve already shown.

It’s not open to the internet, but I am still somewhat concerned that basically anyone on your local network can browse to this and start pressing buttons. They can’t press what ever they want, only what’s been defined and the HTML has no info what the actual key is that will be “pressed”.

Adding the logic to limit it to certain processes adds about 300ms delay which is not terrible, but not great. The other problem with this approach is the fact that I need to know what the process name is you want to limit the app to. Not difficult, but requires that the user knows enough to be able choose the right thing from a list or something.

I’ll probably want to add some sort of permission system. When you browse to the service the first time, you have to approve the IP/device on the WPF app.

Will see how far I get, but I would like to have a semi working version of what I’m busy with now by end of the month. I’m busy doing the Config part which allows the user to setup Applications, themes, Actions (keys to sim), different screens per app, Buttons for the screen, and which Actions the user want to trigger per Button.

I also don’t want people to look at this as merely a key sim app. Being able to embed html, js, css, and host it from within a WPF exe I think is pretty cool even if use cases for it might be limited.

[paulwratt]

using non-standard ports might be enough to hide it better on a local network, especially if you dont have any login for at least the approval process on ther WPF app. But yeah, nice, stuff like this is cool, and can have some valid use cases “in the real world”.

[Nic_s]

As stated above, I completed my proof of concept and have since then switched gears a little by going with a WPF app with a built in web host.

Git updated.

The WPF app acts as a control and setup interface. I’ve not tried to make the DataGrids be all pretty… they are just auto generating their columns, but it’s functional.

Something I discovered is that the h.inputsimulator library I’m using has some… err… different names for certain keys. Left and Right Bracket [ ] for example is OEM4 and OEM6 in the enum of keys. It means that some keys are just impossible to find when I just list them as is in a Combobox. I really did not want to make my own list of keys and then translate my list to the Library’s list, but I may have to…

What I’ve learned this month is that I really hate making GUIs. Yes, the fact that I’m not well versed in front end stuff and things is in large part why I find it tedious and frustrating. For the most part, I just need some simple Display, Add, Edit, and Deletes and I can’t help but feel there should be something that can generate such basics for you as a place to start.

I did try and search for something that can generate some basics GUIs based on data, but I’ve not really found anything. If anyone knows of something, please share.

Anyway, It’s now possible to setup different applications with multiple screens, multiple Controls/Buttons, and you can also have multiple Actions/Keys linked to a single Button. You can also do some basic theming which currently only applies to the Screens and Buttons.

Below is quick test using Notepad. The app and Notepad is running on my VM and then I browse to it from my main machine. I will be doing some testing and will probably add some more vids and picks later.

[Nic_s]

Another vid, although this time on the youtubes as it was way to big to upload here. The volume is a little low, sorry.

Note: Getting out of the seat or changing the power distribution requires the keys to be held down. Right now I have it set to 1000ms, but it’s configurable in the WPF app. Can also do mouse inputs like click or scrolling up or down.

MFD Thingy Test 3

Screenshot of what was being shown on the phone:

I think this turned out pretty decent and the input lag is almost nothing… once it’s loaded and has talked to the SQLite DB for the first time.

I’m considering this complete

[Nic_s]

Git now has a zip in the Build folder.

I realized I didn’t really talk much about the HTML side of things. I’m using a CSS Grid to make the layout a bit more dynamic to try and better fill different screen sizes.

You can specify what the minimum width of the buttons should be and as the screen size changes, that will in part determine when they form more or less columns.

You can also specify the spacing between the buttons which will also to some extend have an impact on when more or less columns are formed as screen size changes.

Some Demos:

The HTML and css from the themes is actually being generated on the fly in C# code as you navigate the screens. I can define the screen in code and then pass that to a function that will generate the HTML and css. It’s very limited and is written to do just this. I thought it would be interesting to try it this way.

I was also thinking I can make a Kodi or youtube “remote” with this. I already have an actual remote, but I could make a “remote” for any app that is usable with keyboard shortcuts. Maybe with a bit more work it could even be used as a wireless “controller” for games as the input latency is actually decent.

Something to think about and maybe work on in the future