[WIP] Making a Video Editing algorithm that cuts out respawn portions

Development
1

Intro

My youtube feed had some youtube programmers that have been working on video editing algorithms



Seeing these videos gave me a crazy idea.

What if I took the code these guys wrote for cutting videos and adapted it to gameplay
Make a algorithm go through a video frame by frame with a OCR library to look for text that has image
If “respawn” is found, save the starting postion and the ending position when it disappears
After finding all the start and ends, cut out the portions that are were found and sync it with the audio

Examples of how current editors cut out respawns

This is an example of how some channels manually edit out respawns.


This example cuts just the respawn section

In this case the editor may cut the respawn time and skip to the next part where the player returns to the same spot

Which isn’t bad, but it can get tedious if you have watch through multiple videos spanning from 15-20 minutes and cut out every death

Questions/concerns

  • The biggest concern is actually implementing and coding this into existence (if I write it from scratch)
  • Is there an already existing program that does this?
  • Is it easier to make my own or to modify someone else’s?
  • Is it within reason to create create with basic knowledge from introductory coding class or is more research/experience needed?
    • what are some topics I need to look up and read to be able to implement this?

Goals/todolist

  • Read the code and write up the idea/goals
    • Decide by ask community
  • If writing something from scratch,
    • write pseudo code for what the program will do
    • implement pseudo code into actual functions
  • If modifying the carykh jumpcutter,
    • Change the variable setup in the intro
      • have ffmpeg cut up the videos into individual image files
      • change the audio count to image count
    • Change the algorithm itself
    • Change the threshold (lines 129-135) to compare images instead of audio
      • have OCR library look at the image for image
    • Change the append loop

Code that can potentially be used to make this tool

  • This project has tools that will make cuts either randomly or soon with motion.
  • Looks like something that would take a good amount of time to go through and understand what’s happening.
  • It does have some good potential as more features get added in the future.
    • Looks like it is written in C++, so performance might be better
  • This project makes cuts by speeding up silent portions to a point where it is a cut
  • looks easier to understand the code and use
    • has issues with the amount of space it uses for video files (17 minutes seems to be fine, but be warned for anything longer)
  • Really leaning on trying to modify this section
  • Things noticed from the code
    • lines 129-135 define the threshold
      • line 133 and 134 tells the algorithm compares the current section to the threshold portion and assigns 1 if the audio at that portion is above the threshold
    • lines 138-144 looks to have a loop that looks for portions where audio is above the max and will append the parts that have sound
  • Portions that would probably be modified.
    • Lines 129-135 and 138-134. Instead of having algorithm look at the audio, I can look up a ocr library to look at the video as individual images.
      • If it finds the word “Respawn” the algorithm will assign a number probably (to make it still work with the original audio portion)
      • I would probably have to use ffmpeg to split the video into individual images in order for the algorithm to use ocr.
    • I would also need to make some adjustments to the audio variables
1 Like
2

Look into the Crash Bandicoot Auto-splitter. They have experience on making accurate matching on shapes and patterns for loading screens:

3 Likes