Third party Steam Controller software. Part 2 - My take on it.

In part one of this two-part article, I went over what we have seen in the first year and a bit since the Steam Controller has launched, in terms of hardware mods and (mostly third party) software for it.

One of the programs discussed was my project “GloSC“, that I want to go more in depth on in this part two.

You’re going to see code and assembly here.
If that’s up your alley you might wanna grab a cup of coffee and make yourself comfortable as it’s gonna be a bit of a read.
Otherwise, you’re welcome to stay, too!

Obvious disclaimer before we start:

• Lots of the code has changed from the beginning and will continue to change and improve.
• Discussing the most up to date version of the code at the time of writing, with the latest Steam client beta
• I am not an expert when it comes to reverse engineering of binaries. Nonetheless, I’ve found all the pieces relevant for GloSC and will share that knowledge with you.

My basic idea before creating GloSC was simple: An application that makes all of the Steam Controller functionality available, without beeing locked to focus a single specific window.

List of things needed for that:

• A Gamepad emulation driver for a system-wide gamepad
• A transparent, clickthrough, always on top window for Steam to draw its overlay into.
• A way to enforce a specific controller binding

With that outta the way, grab some coffee and let’s work our way down the list.

The easy stuff:

1. Controller emulation

Probably the most important part of the whole project is the ability to use the Controller as, well, a controller and not only keyboard/mouse on a system-wide level.
Since I have no idea on how to write Windows drivers, even less idea of ones that create virtual devices, I started out contacting the author of the SCPToolkit, that previously has been used to get system-wide XInput bindings for the Steam Controller.
The previous solution was clunky and used another third party framework, in addition to relying on a driver that originally only was intended for DualShock 3 controllers.
So, naturally, I asked for a better way and he told me about this really cool project he’s been working on:
ViGEm, the Virtual Gamepad Emulation Framework.

A sample program of the project is a bridge between ViGEm and x360ce, another XInput controller emulator.

Instead of x360ce faking XInput devices and piping that to ViGEm, we just let Steam fake the XInput device, like it already does for applications launched through Steam and then
Copy pasta code and… profit?

BINGO!
It really was that easy…

Later on, when Steam added support for any gamepad other than the Steam Controller, Steam also detected the virtual controllers from ViGEm and faked an additional Controller.
That one got piped to ViGEm, creating yet another virtual controller and so Steam faked another one that got piped to ViGEm, creating…
Look, You know where this is going.

Luckily, though, ViGEm was also updated in the meantime and implemented a feature suggestion of mine: Changing the USB Vendor- and Product-ID of the created virtual device.
Now, we just set the IDs of Valve’s Steam Controller and Steam doesn’t know what to do with the (virtual) device since the HID reports are wrong and it doesn’t detect it anymore.

As long as we only want to support a single controller, everything is fine and dandy.
For multiple controllers, the story looks a little bit different, though.
We have Steam not detecting our virtual controllers, but we ourselves have no way of knowing which is which!
The XInput-API has no way of reading the USB-IDs from a device and Steam doesn’t hide the virtual controllers from us. That creates a bit of a problem…
Or does it?

The solution is simple:
Steam hooks the “XInputGetState()” function in our executable, and we use the result of said function for our controller emulation.
We un-patch and re-patch said hook every time we want to read the state of a controller.
Conveniently, as API hooks usually work, the first five bytes of the “XInputGetState()” function get replaced with a JMP (==jump) instruction and the address to jump to.
All we need to do is to wait until Steam has hooked said function, store the patched bytes in a buffer, replace them with the real five bytes and we can call the now un-hooked function.
Afterward, we re-patch Steam’s hook for the next cycle.
Using the results of the patched and real function, we can decide if we need to “plug in” a new virtual controller or not.

2. Transparent Window

I found a little Windows specific snippet somewhere on how to create a transparent OpenGL window with SFML and tried that.
It’s nothing really more than creating an SFML window and these four lines of code

MARGINS margins;
margins.cxLeftWidth = -1;
SetWindowLong(window.getSystemHandle(), GWL_STYLE, WS_POPUP | WS_VISIBLE); //Remove window frame
DwmExtendFrameIntoClientArea(window.getSystemHandle(), &margins); //Actually make the window transparent
SetWindowPos(sfWindow.getSystemHandle(), HWND_TOPMOST, 0, 0, 0, 0, SWP_NOMOVE | SWP_NOSIZE); //Additionaly set the window to be always on top

You can find a minimal example here.

Compile that -> Launch via Steam -> Yep, the overlay is showing up!
Combine with copy pasta controller emulation code, and we are starting to make some real progress.

In fact, great progress so far. We’ve got a virtual system-wide XInput device and the Steam overlay can be drawn on top of every other window of the system.
We could just stop here and re-use our desktop-config setup with XInput bindings and call it a day.
But I wanted more.

For the Steam overlay to work properly, the window of the application launched by Steam has to be currently focused. So, we need a way to detect that.
Quickly checking the exported functions of “GameOverlayRenderer.dll” we get this:

“BOverlayNeedsPresent” looks promising, but didn’t turn out to be all too useful when playing with it. So we need another way.

At first, I’ve tried using CheatEngine to just get a pointer to a variable that tells me if the overlay is currently opened or not.
And while that works great, Steam updates most of the time change the address of this pointer and I’d have to update GloSC. Bad! We need a better way.

It’s getting more difficult.

So I went to work with x64dbg, poking in my running executable with the Steam overlay attached to it.
When searching for intermodular calls in “GameOverlayRenderer.dll”, we see, aside from tons of OpenGL calls, a lot of calls to “PostMessage()” and a call to “SetWindowsHookEx()”
Jumping to the address of said call, this little string constant greets us

“WH_GETMESSAGE hook failed”

That seems interesting… The controller hook, as well as the overlay rendering, shouldn’t need that. Worth a shot looking at I guess…
A few instructions before the call, we see the address of the hook callback function and jump to it, set a breakpoint and try to open up the overlay.
Breakpoint hit!

Inspecting the arguments of the call, with a bit of reading up on MSDN, we know that the third argument (in the image, passed in r8, x64 Microsoft fastcall) is a pointer to an MSG structure.
A bit more reading and we know that values in the range of 0x0400 to 0x7FFF of the message member are available for private message identifiers.

We check against that, and it seems like we got a winner here…
But, it’s not over yet.
We still don’t know the message code and if we hit continue a few times in our debugger, we quickly notice that a metric butt-ton of messages gets sent. (More specifically: We are stuck in a loop.)
We need to dig deeper.

Let’s set a breakpoint on those “PostMessage()” calls.
Open up the overlay and…
Breakpoint hit! Continue a few times and see that we’re stuck in a loop… Again!
We disable this particular breakpoint and leave the others checked. The next one breaks…
Again, hitting continue, but this time the program doesn’t hit the breakpoint again and the overlay opens.
Let’s close the overlay and hit yet another one of our breakpoints, continue again and the program continues, the overlay closes.
After a bit more reading on MSDN, we inspect the Msg argument of the function and find that the values are 0x14FA / 0x14F7 when the overlay opens / closes. (At least on 64bit, Unicode)

JACKPOT!

A bit of coding later, setting the same hook on WH_GETMESSAGE as Steam does (For those asking: No, an own WindowProc function or GetMessage, PeekMessage, etc. didn’t work!) and inspecting the message and a bit of window-focus-switching, we have our overlay detection implemented and can continue on with the next task:

Per application bindings

Now, this one took me a while to do correctly, and I still am not really happy with how this works. But it’s the best we get for now.

As you may or may not know, the Steam Controller switches to desktop-config when the “Game” or in my case the GloSC window looses focus.
There currently is no way to disable this behavior officially, except when you have your very own Steam AppID.

Getting an arbitrary amount of AppIDs with arbitrary names attached to them obviously wouldn’t work for me.
So, we need another way to disable this behavior and force a specific set of bindings.

While debugging I found out that, if you have a console window you can somewhat enforce this by a bit of focus switch trickery that I don’t want to go on any deeper here. It’s not very reliable, uses a bug in Steam and overall is rather cringeworthy code.
You can still find it in the source, though.

Instead of this, I figured, the most reliable way to do this is to set another hook.
Unfortunately, though, the code that’s responsible for changing configs, couldn’t be found in the overlay-dll, so we have to look at the Steam executable and hook into that. (The part I’m not very happy about…)

When we go through the list of command-line arguments for Steam, we notice that there is a “-console” switch.
When we try that, Steam gets a lovely extra tab with a console.
A few “OnFocusWindowChanged” messages and the Steam AppID, as well as the full path to the config files, greet us here.

At first, I tried hooking “GetFocusedWindow()“, “GetActiveWindow()” as well as a few other WinAPI calls but didn’t really have any luck.
I didn’t really want to change the config files on the hard drive, which would have also been a valid option. Instead, I focused on the AppIDs.

If we hold down the Steam button, the controller is switching to the so-called “Steam Chord”-configuration. The AppID of which is 443510 or 0x6C47C in hex.
AppIDs of desktop and BigPicture config are 413080 and 413090 respectively.
I chose the AppID of the chords because it’s easier to just hold down the Steam button on the controller, opposed to switching windows all the time. I will also call this button “Breakpoint button” for this section from now on.

If we use x64dbg to search for constants in “Steam.exe” and according dll-files, we find a few comparisons with said value in “steamclient.dll”.
Jumping to the first one, it looks promising right away.

Some value gets moved into the EAX register and then compared to the Steam chord AppID.

We set a breakpoint, hit the “Breakpoint button” on our controller and see that it actually hit it. After hitting continue one or two times and watching the EAX register, we see that it now contains the AppID of the desktop config.

If we now use x64dbg to change that value to a different AppID every time it is the desktop config’s one to one of our choice, we might have enforced a specific set of controller bindings.
We do exactly that and find out that it actually works!

Now we need to write a bit of code and that does that without x64dbg.
I used a simple mid-function hook for that, as it is probably the easiest solution to do this.
To write such a hook, we write a handy little .dll-file that places a JMP  instruction at the desired address, jumps to our own bit of assembly, in which we do our stuff and execute the overwritten instructions and then jump back.
We can grab the AppID of our launched program while we’re at it at in our assembly too!

Having done exactly that, about 100 lines of code later, we’ve got our dll which we can inject into Steam to enforce our controller bindings.
I’ve also included a function to reverse this hook and ship a custom dll-injector / -ejector with GloSC, too!

Combining all of the things we achieved so far, it seems like we got our minimum viable product.
Adding a bit more fluff on top, like a GUI for creating and configuring multiple different shortcuts, the result is neat little software package that allows one to use the Steam Controller with any game that might has trouble with all the hooks Steam sets and on top adds lacking functionality, like touch-menus, to the desktop, making the Steam Controller an even more versatile device that in already is.

What’s left, is to keep an eye out for what Valve does with their controller going forward and to support and maintain GloSC in the future.