Readme Markdown from a not-yet-public project I am working on.
The Repeatable Benchmarking Platform Project
This project is designed to automate the creation of easily repeatable and reproducible benchmark environments on Windows through the use of powershell and other automation technologies.
The idea is to:
1) programmatically bring a machine fully up to date
2) configure it to minimize interference from background processes
3) install drivers appropriate for the graphics hardware
4) catalog & inventory the machine hardware/software settings
5) provide a platform for standardized repeatable software testing including
a. games
b. professional software
We can extend this platform for games testing, such as GTA V, by providing batch files or power shell scripts to automate benchmarking and dropping associated configuration profiles in game settings. In general a benchmark script should:
1) (optionally) download the game from somewhere
2) configure the game by copying a settings profile
3) run the game to benchmark it with the documented settings
4) (optionally) run a program like HWMonitor, but capture the data to a csv file
a. Ideally we have temps,
b. Fan speeds
c. Cpu frequency
d. Cpu utilization
e. Gpu utilization(?)
5) capture the results from the benchmark, store it with the machine catalog from Part 1, #4 above, along with the name of the settings file
6) (optionally) programmatically generate game-specific graphs of the data
The game_benchmarks directory contains one directory for each game – a powershell script that is game-specific that does the above steps. Any dependencies, such as game configurations, should be stored in separate sub-directories. We suggest directory names such as 3840x2160_high_fsaa_nohairworks, 0800x0600_low_noaa_nohairworks but if there is ever any confusion, the raw game settings file can be examined inside this folder for confirmation/clarification.
The desired result of this project is to improve the transparency and repeatability of software benchmarks. Further, it should be user-friendly enough that a user of hardware could download this project, run the benchmarks, and compare the results from similar systems.
For anyone out there that may be writing benchmarks and doing pull requests, please keep in mind that the users of the scripts may not be power users; your benchmarking script will have to have error-checking and helpful hints when things go wrong (“Did you install GTA V? Did you configure your steam user/pass?” etc). It may be that a gamer would want to run these scripts to confirm their system is performing appropriately.
The main script in this project can be edited in a text editor. Inside the main script you will find calls to all the benchmark scripts one might want to include to benchmark and to generate data.
The program_benchmarks directory contains one directory for each app – inside is a powershell script meant to benchmark a particular program. There may be additional powershell scripts to download dependent projects e.g. an Adobe Premiere benchmark may require the download and unpacking of a Premiere Project and sample footage.
USAGE:
UpdateEverything.ps
This script updates your machine with the latest windows updates. It was designed for us on Windows 10. It may require multiple reboots to complete. This script does not run automatically as part of the benchmark system.
Inventory.ps
This script inventories the system. It will automatically run as part of the benchmark process, unless it is commented out from the RunBenchmark script.
RunBenchmarks.ps
This script runs the benchmark scripts that are not commented out. Note that for steam games, you may need to specify your steam username and password so that a game may be downloaded. You may also have to specify what drive letter and path a particular game is installed on.
This section TBD.
This project does not exist yet it does kinda in that I have bits and pieces of this. For example I can run GTA V benchmarks in a canned fashion. I suspect some third party programs like AutoIT would be needed to automated "every" game testing but for getting started toward a 0.01 release on github of all this, that may not be necessary.
The goal here is open source level the playing field to do everything.
We could even include software like Asus' realbench as long as the powershell script for that benchmark downloads + configures it in a mostly automatic kind of way -- why not?
All of this on github, accepting pull requests, etc.
I thought I would post this here, now, to get feedback and to see if there are similar projects out there that I have missed. Perhaps there are open and permissive scripts that already exist for things like updates, configuring steam (for example on gta v I have to set the command line manually, via steam client, still, for benchmarking.. which is annoying). Perhaps the community would get involved and develop some of the game benchmarking scripts, scour forums for them, and we can organize them into a central repository.