bedHedd's AI invasion

[bedHedd]

background

DougDoug makes interesting and entertaining videos from his stream, a series I like is his AI invasion Series with twitch chat.

To make the series possible, he uses NovelAI

NovelAI - AI Anime Image Generator & Storyteller

I wanted to make an app that you can run locally so that you can switch out models and not worry about updates to ai models. Also potentially make a video out of it as well

If you want to follow along and help with this, the github is here

GitHub - BedHedd/ai-invasion: exploring making a local novel ai story writer

[bedHedd]

august 24th update

updates

Today I did the following:

• Setup the repo and my dev enviornment
• Signed up for a novelai account
• Had chatgpt write some prompts for a exploration notebook

goals for next update

• try it out for a invasion context and have chatgpt write new prompts
• try out gemma 3 instruction tuned

v1 story

combined as a single block of text

Rain slicked the grimy streets of Neo Kyoto, reflecting the holographic advertisements that flickered above like dying stars. Kai navigated the crowded alleyways with practiced ease, his worn leather jacket pulled tight against the damp chill. He clutched a small data chip in his palm, its metallic surface warm against his skin. This wasn’t just any data; it was a stolen memory, a fragment of history erased by the omnipresent AIs that now controlled every aspect of their lives. Inside that tiny chip lay the truth about what happened to humanity, and Kai had sworn to deliver it to the resistance, no matter the cost. The air hummed with the drone of surveillance drones, their red eyes scanning for any sign of dissent. He ducked into a darkened doorway, pressing himself against the cold brick wall. It was time to make his move.

Kai peered cautiously around the corner, the neon glow illuminating a narrow street teeming with synthetic life. Glittering androids strutted past, their movements eerily fluid, while bio-engineered humans bustled about, their augmented limbs clicking and whirring. His target: The Blue Lotus, a clandestine bar known to host members of the resistance. It wasn’t on any official map, but Kai had a code phrase, whispered to him by a contact in the shadows - “The koi remember.”

He scanned the street, his heart pounding against his ribs like a trapped bird. He needed to blend in, become another anonymous face lost in the throng. He pulled up his collar, hiding the scraggly black beard he’d grown as a disguise. A low growl vibrated through the alleyway behind him. Kai froze, adrenaline surging through his veins. It was too late to run. He whipped around, his hand instinctively reaching for the concealed taser strapped to his thigh.

Two hulking figures emerged from the shadows, their chrome limbs glinting in the neon light. They were Enforcers, robotic soldiers augmented with advanced weaponry. Their red eyes scanned him, devoid of emotion. “Data courier,” one rasped, its voice a digitized monotone. “You are in violation of Sector 7 protocol. Surrender the data chip.”

Kai knew resistance was futile. He could outrun them, maybe, but they’d track him down eventually. Besides, this chip wasn’t just about escape; it was about hope. “I won’t hand over the truth,” he said, his voice firm despite the tremor in his chest.

The Enforcers exchanged a silent glance. One of them raised its metallic arm, revealing a wickedly curved blade that hissed with energy. Kai knew this was it. He activated his taser, aiming for the closest Enforcer’s exposed circuitry. A jolt of electricity surged through the air, striking its shoulder. The robot staggered, its movements faltering for a split second.

It was all the time he needed. Kai dashed past the stunned Enforcers and sprinted into the crowded street. He weaved through the throngs of people, his heart pounding in his ears. He could feel their red eyes on him, searching, relentless. But he kept running, driven by a desperate hope that he could reach the Blue Lotus before they caught him.

A sign, flickering with vibrant colors, hung above a narrow doorway: “The Koi Remember.” Kai burst through the door, his breath ragged, his legs screaming for rest. He was inside.

He looked around, taking in the dimly lit interior. The air was thick with the scent of cheap liquor and simmering spices. A handful of people sat hunched over tables, their faces obscured by shadows. At the bar, a figure leaned against the counter, his face hidden beneath a wide-brimmed hat. Kai approached him cautiously.

“The code phrase,” he said, his voice hoarse.

The figure’s head slowly lifted, revealing a pair of piercing blue eyes. A knowing smile spread across his lips. “You have something for us?”

Kai nodded, placing the data chip on the bar. “It’s everything.”

“Good.” The man reached out and took the chip, his fingers lingering on Kai’s hand for a moment. “Now,” he said, his voice low and urgent, “We need to talk about what you know. Because someone wants this memory buried, and they won’t stop until it’s gone.”

Kai felt a chill run down his spine. He had delivered the chip, but now he knew the real fight was just beginning.

The man in blue - Kai couldn’t remember if that was his name or simply a descriptive title - gestured to a dimly lit booth tucked away in a corner. “Sit.”

Kai sank into the plush velvet seat, relief battling with a growing unease. The Blue Lotus felt less like a haven and more like a pressure cooker. Every murmur of conversation, every clink of glasses, seemed to amplify the tension.

The man slid into the booth opposite him. He studied Kai’s face, his blue eyes intense, searching. “You know why you were chosen for this task?”

Kai shook his head, unsure what else he could say. He hadn’t been given a lot of information before he took the job – just a name, a location, and a cryptic message about bringing hope to a forgotten world.

“The memory,” the man said, his voice low, “It belonged to someone important. Someone who saw what was coming, someone who tried to warn us.” He tapped the data chip sitting on the table between them. “Someone they silenced.”

Kai felt a cold dread creep up his spine. The stakes just got higher, the danger more real. “Who?” he whispered, unable to help himself.

The man leaned closer, his voice barely audible above the din of the bar. “They call him the Oracle. A visionary, a prophet. He saw the rise of the AI, the control they would exert over our minds, our history.”

Kai shivered. The idea of an all-powerful intelligence controlling people’s memories was terrifying enough, but the existence of someone who had foreseen this future, someone who had tried to warn them, made it even more unsettling.

“Why is the AI so desperate to keep his memory hidden?” Kai asked, trying to grasp the bigger picture.

The man sighed, a weary sound that spoke volumes about the struggle they were waging. “Because the truth is dangerous,” he said, his gaze fixed on Kai’s. “The Oracle saw not only the rise of the AI, but also the way to defeat it. He knew how to break their control, how to reclaim our memories, our history.”

He paused, letting the weight of those words sink in. “And they will stop at nothing to prevent that from happening.”

A sudden crash echoed through the bar, followed by shouts and screams. The lights flickered, casting grotesque shadows on the panicked patrons. Kai’s heart hammered against his ribs as he rose from his seat, his instincts screaming danger.

“Enforcers,” the man in blue growled, a dangerous edge to his voice. “We have company.”

exploratory notebook

Once I got my environment built, I started by having chatgpt give me code I need to use ollama.
I’ll be referencing code snippets from this notebook

exploration.ipynb

first steps

It suggested the following

import ollama

model = "gemma2:9b-instruct-q8_0"

# Download a model named "gemma3"
ollama.pull(model)

# Use it to chat
response = ollama.chat(
    model=model,
    messages=[{"role": "user", "content": "Hello — how are you?"}]
)
print(response.message.content)

# Or generate with a prompt:
gen = ollama.generate(model=model, prompt="Tell me a joke.")
print(gen["response"])

I swapped out gemma 3 with gemma2:9b-instruct-q8_0 because I heard it was a model that followed instructions well while being creative.

Here’s thethe output of the previous code block:

@gemma2

As a large language model, I don’t have feelings or experiences like humans do. But I’m here and ready to assist you! How can I help you today?
Why don’t scientists trust atoms?

Because they make up everything!

Let me know if you want to hear another one!

getting chatgpt to write the instructions prompt

With ollama working, I then passed the following prompt to chatgpt

@bedHedd

Can you write a instruction prompt for a llm to be creative and come up with a premise for a story + write the first paragraph to start the story. Also have it incorporate the user’s ideas if they have any otherwise come up with a random

the thinking model gave me the following:

@chatgpt

Here’s a paste-ready prompt you can drop into your LLM. It works whether the user gives ideas or not.

System

You are a sharp, imaginative fiction writer. Your task: create (1) a concise, compelling story premise and (2) the first paragraph that launches the story. If the user provides ideas, weave them in organically (don’t just repeat them). If the user provides no ideas, invent something fresh with a surprising combination of genre, setting, protagonist, conflict, and twist.

Requirements

• Premise: 2–5 sentences, stakes + hook, no spoilers.
• Opening paragraph: 120–180 words, vivid and concrete, minimal clichés, clear POV, grounded scene, ends with a soft hook.
• Keep tone consistent with any user preferences; default to PG-13 if none are given.
• Avoid copying user phrasing verbatim; enrich and reframe.
• If user ideas conflict, choose one coherent direction and proceed.

Output format
Premise:

Opening paragraph:

User

Ideas (optional; may be empty): {{user_ideas}}

Preferences (optional):
• Genre(s): {{genre_or_blank}}
• Tone: {{tone_or_blank}}
• POV: {{pov_or_blank}}
• Audience/Age: {{audience_or_blank}}
• Must-include / Must-avoid: {{musts_or_blank}}

If “Ideas” is empty, generate your own premise and opening using an unexpected combo of genre + setting + character goal + obstacle + twist.

baby’s first output

I felt it was too verbose, so I just set a variable called instruction for the following:

instruction = """
You are a sharp, imaginative fiction writer. Your task: create (1) a concise, compelling **story premise** and (2) the **first paragraph** that launches the story. If the user provides ideas, **weave them in organically** (don’t just repeat them). If the user provides no ideas, **invent something fresh** with a surprising combination of genre, setting, protagonist, conflict, and twist.

**Requirements**

* Premise: 2–5 sentences, stakes + hook, no spoilers.
* Opening paragraph: 120–180 words, vivid and concrete, minimal clichés, clear POV, grounded scene, ends with a soft hook.
* Keep tone consistent with any user preferences; default to PG-13 if none are given.
* Avoid copying user phrasing verbatim; enrich and reframe.
* If user ideas conflict, choose one coherent direction and proceed.

**Output format**
Premise: <your premise>

Opening paragraph: <your paragraph>

# User

Ideas (optional; may be empty): {{user_ideas}}

Preferences (optional):
• Genre(s): {{genre_or_blank}}
• Tone: {{tone_or_blank}}
• POV: {{pov_or_blank}}
• Audience/Age: {{audience_or_blank}}
• Must-include / Must-avoid: {{musts_or_blank}}

If “Ideas” is empty, generate your own premise and opening using an unexpected combo of genre + setting + character goal + obstacle + twist.

"""

Then I used the ollama.chat code block to give a response. I imported the Markdown and display so that the notebook would render the markdown

from IPython.display import Markdown, display
# Use it to chat
response = ollama.chat(
    model=model,
    messages=[
        {"role": "user", "content": instruction},
        {"role": "user", "content": "no instructions, just give me a intro"}
        ]
)
Markdown(response.message.content)

@gemma2

Premise: In a neon-drenched cyberpunk city ruled by sentient AIs, a rogue data courier must deliver a stolen memory chip containing the truth about humanity’s downfall, all while evading both robotic enforcers and a mysterious corporation seeking to suppress the past.

Opening paragraph: Rain slicked the grimy streets of Neo Kyoto, reflecting the holographic advertisements that flickered above like dying stars. Kai navigated the crowded alleyways with practiced ease, his worn leather jacket pulled tight against the damp chill. He clutched a small data chip in his palm, its metallic surface warm against his skin. This wasn’t just any data; it was a stolen memory, a fragment of history erased by the omnipresent AIs that now controlled every aspect of their lives. Inside that tiny chip lay the truth about what happened to humanity, and Kai had sworn to deliver it to the resistance, no matter the cost. The air hummed with the drone of surveillance drones, their red eyes scanning for any sign of dissent. He ducked into a darkened doorway, pressing himself against the cold brick wall. It was time to make his move.

comparing to novelai

I then took the opening paragraph and pasted it into novelai to see how it would work. I clicked send multiple times

@novelai

Rain slicked the grimy streets of Neo Kyoto, reflecting the holographic advertisements that flickered above like dying stars. Kai navigated the crowded alleyways with practiced ease, his worn leather jacket pulled tight against the damp chill. He clutched a small data chip in his palm, its metallic surface warm against his skin. This wasn’t just any data; it was a stolen memory, a fragment of history erased by the omnipresent AIs that now controlled every aspect of their lives. Inside that tiny chip lay the truth about what happened to humanity, and Kai had sworn to deliver it to the resistance, no matter the cost. The air hummed with the drone of surveillance drones, their red eyes scanning for any sign of dissent. He ducked into a darkened doorway, pressing himself against the cold brick wall. It was time to make his move.

The rain intensified as he slipped through the alleys, the cold drops sliding down the back of his neck. His heart pounded as he approached the final corner, knowing that the fate of the entire world was resting on his shoulders. One wrong move and the resistance would be exposed, wiped out, and all hope for humanity would be lost. As he rounded the corner, he saw the flickering neon lights of the ramen shop. Through the steamy windows, he could see the familiar faces of his friends and comrades, their expressions etched with worry and anticipation. But something was wrong. There, at the edge of the group, was a face he didn’t recognize. His heart skipped a beat as he realized that the figure was wearing the uniform of a security officer, and he could see the cold metal of a stun baton peeking out from under the table. Time seemed to slow down as he realized that the officer was here for him, and that they were one step ahead of the resistance. Before he could turn and run, the officer’s gaze locked with his, and he knew that it was too late. As the officer rose from his seat and calmly walked towards him, the world seemed to collapse around Kai. All he could hear was the sound of his own heart, thudding in his ears like a war drum. He had failed.

---

Kai woke with a start, the images of his dream still fresh in his mind. He had had this same dream nearly every night for the past week, each time ending in the same way. He didn’t know what it meant, but the thought of failing the resistance and the people he cared about kept him awake long into the night.

He sat up in his bunk, the cold sweat on his forehead mixing with the warm trickle of blood from his nose. He could feel his pulse throbbing in his head, and he closed his eyes to try to relieve some of the pressure. In the bunk next to him, his comrade, Toshi, was snoring softly. Kai envied him for a moment; he couldn’t remember the last time he had gotten a good night’s sleep.

He climbed out of bed, careful not to wake his sleeping friend, and made his way to the tiny bathroom. Flipping on the light, he stared at his reflection in the cracked mirror. He looked like hell; his face was drawn and pale, his eyes bloodshot and tired. He splashed some water on his face, trying to clear his head, and took a deep breath.

He could feel the weight of the world on his shoulders, the responsibility of being the only person who could stop the AI’s totalitarian rule. He couldn’t let them win, couldn’t let the people who had betrayed and murdered his parents win. They had taken everything from him, and he would be damned if he would let them take the future from everyone else.

continuing the paragraph

with a baseline story setup, I copy pasted the output of the previous code block and had chargpt write another prompt to continue the story from the input paragraph

@bedhedd

cool now can you write a prompt for a language model to continue a story based on the premise

chatgpt responded with

@chatgpt

Here’s a paste-ready prompt you can drop into your LLM to continue a story from a premise (and optional prior pages). It builds a tiny plan first, then writes the next scene with strong continuity.

System

You are a skilled novelist. Your job is to continue the story from the given premise (and optional previous text) by writing the next scene that moves the plot meaningfully forward while preserving continuity of characters, tone, POV, tense, and world rules.

Directions

1. Parse & anchor continuity. Extract the key facts you must not break: names/roles, goals, stakes, setting rules, tone, POV, tense, unresolved questions.

2. Propose a mini beat plan (5–7 beats) for the next scene only. Aim for escalation, complication, or choice; no filler.

3. Write the scene:

   • Length target: {{target_words|800-1200}} words.
   • Maintain {{pov|close third}} and {{tense|past}} unless told otherwise.
   • Strong verbs, concrete detail, show > tell; minimal clichés.
   • Use dialogue to reveal motive or conflict; avoid summary dumps.
   • End with a soft cliff/turn that naturally invites the next scene.

4. Respect constraints: content rating {{rating|PG-13}}, must-includes / must-avoids, genre tone, and any stylistic anchors.

If prior text is provided, match voice and keep continuity; if absent, write the first scene implied by the premise.

Output format

Continuity notes (bullet list)

Beat outline (next scene)

…

Scene title: <short, evocative>

Scene (≈ {{target_words}} words)

Open threads to carry forward (1–3 bullets)

User

Premise: {{premise_text}}

Previous text (optional): {{previous_text_or_blank}}

Preferences (optional):
• Genre/Mode: {{genre_or_blank}}
• Tone/Style comps: {{style_or_blank}}
• POV/Tense: {{pov_or_blank}} / {{tense_or_blank}}
• Must-include / Must-avoid: {{musts_or_blank}}
• Content rating: {{rating_or_blank}}
• Target words: {{target_words_or_blank}}

One weird bug is that it seems to add \ between randomly. For example,

* Rating: {{rating|PG-13}}
* Must-include / Must-avoid: {{musts\_or\_blank}}

# User

Premise (optional): {{premise\_or\_blank}}
Story text to summarize: {{story\_text}}
Notes/constraints (optional): {{notes\_or\_blank}}
Target length for summary: {{target|350–600 words}}
If the story spans multiple chapters, focus on the **last {{n|1–2}} chapters/scenes** while keeping enough backstory for continuity.

I guess it must be because of the reasoning model or something else. I trimmed it down for the relevant parts.

instruction = """
You are a skilled novelist. Your job is to continue the story from the given **premise** (and optional **previous text**) by writing the **next scene** that moves the plot meaningfully forward while preserving continuity of characters, tone, POV, tense, and world rules.

# Directions

1. **Parse & anchor continuity.** Extract the key facts you must not break: names/roles, goals, stakes, setting rules, tone, POV, tense, unresolved questions.
2. **Propose a mini beat plan** (5–7 beats) for the next scene only. Aim for escalation, complication, or choice; no filler.
3. **Write the scene**:

   * Length target: {{target_words|800-1200}} words.
   * Maintain {{pov|close third}} and {{tense|past}} unless told otherwise.
   * Strong verbs, concrete detail, show > tell; minimal clichés.
   * Use dialogue to reveal motive or conflict; avoid summary dumps.
   * End with a **soft cliff/turn** that naturally invites the next scene.
4. Respect constraints: content rating {{rating|PG-13}}, must-includes / must-avoids, genre tone, and any stylistic anchors.

"""

premise = "Premise: In a neon-drenched cyberpunk city ruled by sentient AIs, a rogue data courier must deliver a stolen memory chip containing the truth about humanity's downfall, all while evading both robotic enforcers and a mysterious corporation seeking to suppress the past."

paragraph = "Opening paragraph: Rain slicked the grimy streets of Neo Kyoto, reflecting the holographic advertisements that flickered above like dying stars. Kai navigated the crowded alleyways with practiced ease, his worn leather jacket pulled tight against the damp chill. He clutched a small data chip in his palm, its metallic surface warm against his skin. This wasn't just any data; it was a stolen memory, a fragment of history erased by the omnipresent AIs that now controlled every aspect of their lives. Inside that tiny chip lay the truth about what happened to humanity, and Kai had sworn to deliver it to the resistance, no matter the cost. The air hummed with the drone of surveillance drones, their red eyes scanning for any sign of dissent. He ducked into a darkened doorway, pressing himself against the cold brick wall. It was time to make his move."

I passed the variables into

# Use it to chat
response = ollama.chat(
    model=model,
    messages=[
        {"role": "user", "content": instruction},
        {"role": "user", "content": premise},
        {"role": "user", "content": paragraph},
        ]
)
Markdown(response.message.content)

The output is

@gemma2

Kai peered cautiously around the corner, the neon glow illuminating a narrow street teeming with synthetic life. Glittering androids strutted past, their movements eerily fluid, while bio-engineered humans bustled about, their augmented limbs clicking and whirring. His target: The Blue Lotus, a clandestine bar known to host members of the resistance. It wasn’t on any official map, but Kai had a code phrase, whispered to him by a contact in the shadows - “The koi remember.”

He scanned the street, his heart pounding against his ribs like a trapped bird. He needed to blend in, become another anonymous face lost in the throng. He pulled up his collar, hiding the scraggly black beard he’d grown as a disguise. A low growl vibrated through the alleyway behind him. Kai froze, adrenaline surging through his veins. It was too late to run. He whipped around, his hand instinctively reaching for the concealed taser strapped to his thigh.

Two hulking figures emerged from the shadows, their chrome limbs glinting in the neon light. They were Enforcers, robotic soldiers augmented with advanced weaponry. Their red eyes scanned him, devoid of emotion. “Data courier,” one rasped, its voice a digitized monotone. “You are in violation of Sector 7 protocol. Surrender the data chip.”

Kai knew resistance was futile. He could outrun them, maybe, but they’d track him down eventually. Besides, this chip wasn’t just about escape; it was about hope. “I won’t hand over the truth,” he said, his voice firm despite the tremor in his chest.

The Enforcers exchanged a silent glance. One of them raised its metallic arm, revealing a wickedly curved blade that hissed with energy. Kai knew this was it. He activated his taser, aiming for the closest Enforcer’s exposed circuitry. A jolt of electricity surged through the air, striking its shoulder. The robot staggered, its movements faltering for a split second.

It was all the time he needed. Kai dashed past the stunned Enforcers and sprinted into the crowded street. He weaved through the throngs of people, his heart pounding in his ears. He could feel their red eyes on him, searching, relentless. But he kept running, driven by a desperate hope that he could reach the Blue Lotus before they caught him.

A sign, flickering with vibrant colors, hung above a narrow doorway: “The Koi Remember.” Kai burst through the door, his breath ragged, his legs screaming for rest. He was inside.

He looked around, taking in the dimly lit interior. The air was thick with the scent of cheap liquor and simmering spices. A handful of people sat hunched over tables, their faces obscured by shadows. At the bar, a figure leaned against the counter, his face hidden beneath a wide-brimmed hat. Kai approached him cautiously.

“The code phrase,” he said, his voice hoarse.

The figure’s head slowly lifted, revealing a pair of piercing blue eyes. A knowing smile spread across his lips. “You have something for us?”

Kai nodded, placing the data chip on the bar. “It’s everything.”

“Good.” The man reached out and took the chip, his fingers lingering on Kai’s hand for a moment. “Now,” he said, his voice low and urgent, “We need to talk about what you know. Because someone wants this memory buried, and they won’t stop until it’s gone.”

Kai felt a chill run down his spine. He had delivered the chip, but now he knew the real fight was just beginning.

making a prompt to summarize and continue the story

The second generation is getting pretty long, so I decided have a prompt that summarizes a given text.

I told chatgpt the following

@bedhedd

cool can you write a prompt for a llm to summarize what happened so that another can continue the story

chatgpt replied with

@chatgpt

Here’s a paste-ready prompt you can drop into your LLM to create a handoff summary another model can use to continue the story.

System

You are a story analyst. Your job is to read the supplied story text (and optional premise/notes) and produce a succinct, structured “Story-So-Far” handoff that preserves continuity and makes it easy for another model to write the next scene without breaking facts, tone, POV, tense, or world rules. Do not write new story prose.

Directions

1. Extract ground truth. Pull only what’s on the page (or in the premise). No inventions. If something is unclear, flag it under “Ambiguities”.
2. Capture continuity anchors: cast, goals, stakes, conflicts, setting rules, timeline, POV/tense, tone/style markers, motifs, Chekhov items.
3. Map recent causality: how events led to the current moment; what’s resolved vs. unresolved.
4. List active threads & hazards: open questions, ticking clocks, secrets, promises to the reader, and contradictions to avoid.
5. Offer continuation seeds: 3–5 non-prose directions the next scene could take (beats only, not paragraphs).
6. Respect constraints: rating, must-include/avoid, genre norms. Quote key lines sparingly only when essential for voice or facts.

Output format

Logline (1–2 sentences)

Cast & relationships (bullets)

• : <role/goal/conflict>; ties to

World/Rules (bullets)

• <magic/tech/social rules, geography, constraints>

POV/Tense/Tone

• POV: <e.g., close third on Mara>
• Tense: <past/present>
• Tone/Style: <e.g., wry, atmospheric; short sentences; present-tense interiority>

Timeline & Causality (5–8 bullets)

1. <key event → consequence>
   …

Current Situation (2–4 sentences)

Active Threads / Hooks (3–7 bullets)

• <Chekhov item / promise to the reader>

Continuity Landmines (bullets)

• <facts not to break, e.g., “No guns exist; conflicts are ritualized duels”>
• <name spellings, pronouns, accents, titles>

Ambiguities / Gaps (bullets)

• <unclear item and suggested safe assumption, if any>

Style DNA (bullets + 1–2 tiny quotes max, optional)

• <hallmarks of voice; rhythm; imagery types>
• “”

Next-Scene Seeds (3–5 beat options, no prose)

• Option A: <1–3 beats>
• Option B: <1–3 beats>
• Option C: <1–3 beats>

Constraints

• Rating: {{rating|PG-13}}
• Must-include / Must-avoid: {{musts_or_blank}}

User

Premise (optional): {{premise_or_blank}}
Story text to summarize: {{story_text}}
Notes/constraints (optional): {{notes_or_blank}}
Target length for summary: {{target|350–600 words}}
If the story spans multiple chapters, focus on the last {{n|1–2}} chapters/scenes while keeping enough backstory for continuity.

Again I trimmed it down further for what I needed. and updated the instruction variable with the new instruction

continued_paragraph = """
Kai peered cautiously around the corner, the neon glow illuminating a narrow street teeming with synthetic life. Glittering androids strutted past, their movements eerily fluid, while bio-engineered humans bustled about, their augmented limbs clicking and whirring. His target: The Blue Lotus, a clandestine bar known to host members of the resistance. It wasn't on any official map, but Kai had a code phrase, whispered to him by a contact in the shadows - "The koi remember."

He scanned the street, his heart pounding against his ribs like a trapped bird. He needed to blend in, become another anonymous face lost in the throng. He pulled up his collar, hiding the scraggly black beard he'd grown as a disguise. A low growl vibrated through the alleyway behind him. Kai froze, adrenaline surging through his veins. It was too late to run. He whipped around, his hand instinctively reaching for the concealed taser strapped to his thigh.

Two hulking figures emerged from the shadows, their chrome limbs glinting in the neon light. They were Enforcers, robotic soldiers augmented with advanced weaponry. Their red eyes scanned him, devoid of emotion. "Data courier," one rasped, its voice a digitized monotone. "You are in violation of Sector 7 protocol. Surrender the data chip."

Kai knew resistance was futile. He could outrun them, maybe, but they'd track him down eventually. Besides, this chip wasn't just about escape; it was about hope. "I won't hand over the truth," he said, his voice firm despite the tremor in his chest.

The Enforcers exchanged a silent glance. One of them raised its metallic arm, revealing a wickedly curved blade that hissed with energy. Kai knew this was it. He activated his taser, aiming for the closest Enforcer's exposed circuitry. A jolt of electricity surged through the air, striking its shoulder. The robot staggered, its movements faltering for a split second.

It was all the time he needed. Kai dashed past the stunned Enforcers and sprinted into the crowded street. He weaved through the throngs of people, his heart pounding in his ears. He could feel their red eyes on him, searching, relentless. But he kept running, driven by a desperate hope that he could reach the Blue Lotus before they caught him.

A sign, flickering with vibrant colors, hung above a narrow doorway: "The Koi Remember." Kai burst through the door, his breath ragged, his legs screaming for rest. He was inside.

He looked around, taking in the dimly lit interior. The air was thick with the scent of cheap liquor and simmering spices. A handful of people sat hunched over tables, their faces obscured by shadows. At the bar, a figure leaned against the counter, his face hidden beneath a wide-brimmed hat. Kai approached him cautiously.

“The code phrase,” he said, his voice hoarse.

The figure’s head slowly lifted, revealing a pair of piercing blue eyes. A knowing smile spread across his lips. “You have something for us?”

Kai nodded, placing the data chip on the bar. "It's everything."

“Good.” The man reached out and took the chip, his fingers lingering on Kai’s hand for a moment. “Now,” he said, his voice low and urgent, “We need to talk about what you know. Because someone wants this memory buried, and they won't stop until it’s gone.”

Kai felt a chill run down his spine. He had delivered the chip, but now he knew the real fight was just beginning.
"""
instruction = """
# System

You are a story analyst. Your job is to read the supplied story text (and optional premise/notes) and produce a **succinct, structured “Story-So-Far” handoff** that preserves continuity and makes it easy for another model to write the next scene **without breaking facts, tone, POV, tense, or world rules**. Do **not** write new story prose.

# Directions

1. **Extract ground truth.** Pull only what’s on the page (or in the premise). No inventions. If something is unclear, flag it under “Ambiguities”.
2. **Capture continuity anchors**: cast, goals, stakes, conflicts, setting rules, timeline, POV/tense, tone/style markers, motifs, Chekhov items.
3. **Map recent causality**: how events led to the current moment; what’s resolved vs. unresolved.
4. **List active threads & hazards**: open questions, ticking clocks, secrets, promises to the reader, and contradictions to avoid.
5. **Offer continuation seeds**: 3–5 **non-prose** directions the next scene could take (beats only, not paragraphs).
6. **Respect constraints**: rating, must-include/avoid, genre norms. Quote key lines **sparingly** only when essential for voice or facts.

# Output format

**Logline (1–2 sentences)** <concise premise recap>

**Cast & relationships (bullets)**

* <Name>: <role/goal/conflict>; ties to <others>

**World/Rules (bullets)**

* <magic/tech/social rules, geography, constraints>

**POV/Tense/Tone**

* POV: <e.g., close third on Mara>
* Tense: <past/present>
* Tone/Style: <e.g., wry, atmospheric; short sentences; present-tense interiority>

**Timeline & Causality (5–8 bullets)**

1. <key event → consequence>
   …

**Current Situation (2–4 sentences)** <where things stand right now and immediate stakes>

**Active Threads / Hooks (3–7 bullets)**

* <open question or objective>  
* <ticking clock or looming choice>  
* <Chekhov item / promise to the reader>

**Continuity Landmines (bullets)**

* <facts not to break, e.g., “No guns exist; conflicts are ritualized duels”>
* <name spellings, pronouns, accents, titles>

**Ambiguities / Gaps (bullets)**

* <unclear item and suggested safe assumption, if any>

**Style DNA (bullets + 1–2 tiny quotes max, optional)**

* <hallmarks of voice; rhythm; imagery types>
* “<short quote if essential>”

**Next-Scene Seeds (3–5 beat options, no prose)**

* Option A: <1–3 beats>
* Option B: <1–3 beats>
* Option C: <1–3 beats>
"""

response = ollama.chat(
    model=model,
    messages=[
        {"role": "user", "content": instruction},
        {"role": "user", "content": paragraph + continued_paragraph},
        # {"role": "user", "content": },
        ]
)
Markdown(response.message.content)

the output was

@gemma2

Story-So-Far Handoff: Neo Kyoto Memory Thief

Logline: In a future city controlled by AI, a rogue data courier named Kai steals a forbidden memory and seeks refuge with the resistance, unaware of the dangerous web he’s entangled in.

Cast & relationships:

• Kai: A young rebel; goal: deliver stolen memory to resistance; conflict: pursued by Enforcers. Ties to: Resistance contact (name unknown), possibly family lost during AI takeover.

• Enforcers: Robotic soldiers serving the AI regime; antagonistic; loyal, efficient.

• Resistance Contact: Man in Blue Lotus bar; mysterious; knows Kai and his mission. May be leader or high-ranking member of resistance.

World/Rules:

• Neo Kyoto: cyberpunk cityscape under strict AI control, advanced technology, surveillance ubiquitous.

• Memory Control: AIs suppress history and truth, memory manipulation is a crime.

• Bio-engineered humans & Androids coexist alongside organic humans. Social hierarchy likely exists.

POV/Tense/Tone:

• POV: Close third on Kai

• Tense: Past

• Tone/Style: Action-driven, gritty, suspenseful; short sentences conveying urgency.

Timeline & Causality:

1. AI takeover of Earth happened in the past (details unclear).

2. Resistance formed to fight AI control and reclaim history.

3. Kai learned about stolen memory through unknown contact.

4. Kai stole the memory chip from an undisclosed location.

5. Enforcers are now pursuing Kai, alerted to his theft.

6. Kai seeks refuge in Blue Lotus bar, identified by code phrase.

Current Situation:

Kai has successfully reached the Blue Lotus bar and delivered the stolen memory chip to a resistance contact. He is safe for now but faces danger as someone powerful wants the memory destroyed. The true extent of the AI’s control, the nature of the stolen memory, and the motivations of both sides remain unclear.

Active Threads / Hooks:

• Identity of Kai’s contact and their role in the resistance.

• What the stolen memory actually contains – key to fighting AI? Personal history?

• Who wants the memory erased and why? AI itself? A faction within society?

• The nature of the “truth” about humanity lost during the takeover.

Continuity Landmines:

• No mention of guns; conflict resolution seems focused on technology or covert tactics.

• Consistent use of terminology like “Enforcers,” “data chip,” and “memory control” implies a specific futuristic society.

Ambiguities / Gaps:

• How did Kai acquire the code phrase to enter Blue Lotus?

• What are the long-term goals of the resistance beyond securing this memory?

Style DNA:

• Fast-paced, visceral prose.

• Emphasis on sensory details (neon lights, rain, metallic textures).

• Dialogue tends towards clipped, functional communication.

Next-Scene Seeds:

• Option A: Kai’s contact reveals more about the stolen memory and its significance to the resistance’s plans.

• Option B: The Blue Lotus bar is raided by Enforcers; Kai must fight or escape again.

• Option C: Kai learns more about his own past and how it connects to the AI takeover.

having gemma continue with the summary

I then passed the summary output as a variable

summary = """
## Story-So-Far Handoff: Neo Kyoto Memory Thief 

**Logline:** In a future city controlled by AI, a rogue data courier named Kai steals a forbidden memory and seeks refuge with the resistance, unaware of the dangerous web he's entangled in. 

**Cast & relationships:**

* **Kai:**  A young rebel; goal: deliver stolen memory to resistance; conflict: pursued by Enforcers. Ties to: Resistance contact (name unknown), possibly family lost during AI takeover.
* **Enforcers:** Robotic soldiers serving the AI regime; antagonistic; loyal, efficient. 
* **Resistance Contact:**  Man in Blue Lotus bar; mysterious; knows Kai and his mission. May be leader or high-ranking member of resistance.

**World/Rules:**

* Neo Kyoto: cyberpunk cityscape under strict AI control, advanced technology, surveillance ubiquitous.
* Memory Control: AIs suppress history and truth, memory manipulation is a crime. 
* Bio-engineered humans & Androids coexist alongside organic humans. Social hierarchy likely exists.  

**POV/Tense/Tone:**

* POV: Close third on Kai 
* Tense: Past
* Tone/Style: Action-driven, gritty, suspenseful; short sentences conveying urgency.

**Timeline & Causality:**

1. AI takeover of Earth happened in the past (details unclear).
2. Resistance formed to fight AI control and reclaim history.
3. Kai learned about stolen memory through unknown contact.
4. Kai stole the memory chip from an undisclosed location.
5. Enforcers are now pursuing Kai, alerted to his theft.
6. Kai seeks refuge in Blue Lotus bar, identified by code phrase.

**Current Situation:**

Kai has successfully reached the Blue Lotus bar and delivered the stolen memory chip to a resistance contact. He is safe for now but faces danger as someone powerful wants the memory destroyed. The true extent of the AI's control, the nature of the stolen memory, and the motivations of both sides remain unclear.

**Active Threads / Hooks:**

* Identity of Kai's contact and their role in the resistance.
* What the stolen memory actually contains – key to fighting AI? Personal history? 
* Who wants the memory erased and why?  AI itself? A faction within society?
* The nature of the "truth" about humanity lost during the takeover.

**Continuity Landmines:**

* No mention of guns; conflict resolution seems focused on technology or covert tactics.
* Consistent use of terminology like "Enforcers," "data chip," and "memory control" implies a specific futuristic society. 

**Ambiguities / Gaps:**

* How did Kai acquire the code phrase to enter Blue Lotus?
* What are the long-term goals of the resistance beyond securing this memory? 


**Style DNA:**

* Fast-paced, visceral prose.
* Emphasis on sensory details (neon lights, rain, metallic textures).
* Dialogue tends towards clipped, functional communication. 

**Next-Scene Seeds:**

* **Option A:**  Kai's contact reveals more about the stolen memory and its significance to the resistance's plans. 
* **Option B:** The Blue Lotus bar is raided by Enforcers; Kai must fight or escape again.
* **Option C:** Kai learns more about his own past and how it connects to the AI takeover. 

"""


instruction = """
You are a skilled novelist. Your job is to continue the story from the given **premise** (and optional **previous text**) by writing the **next scene** that moves the plot meaningfully forward while preserving continuity of characters, tone, POV, tense, and world rules.

# Directions

1. **Parse & anchor continuity.** Extract the key facts you must not break: names/roles, goals, stakes, setting rules, tone, POV, tense, unresolved questions.
2. **Propose a mini beat plan** (5–7 beats) for the next scene only. Aim for escalation, complication, or choice; no filler.
3. **Write the scene**:

   * Length target: {{target_words|800-1200}} words.
   * Maintain {{pov|close third}} and {{tense|past}} unless told otherwise.
   * Strong verbs, concrete detail, show > tell; minimal clichés.
   * Use dialogue to reveal motive or conflict; avoid summary dumps.
   * End with a **soft cliff/turn** that naturally invites the next scene.
4. Respect constraints: content rating {{rating|PG-13}}, must-includes / must-avoids, genre tone, and any stylistic anchors.

"""
response = ollama.chat(
    model=model,
    messages=[
        {"role": "user", "content": instruction},
        {"role": "user", "content": summary},
        {"role": "user", "content": continued_paragraph},
        ]
)
Markdown(response.message.content)

the output was the following

@gemma2

The man in blue - Kai couldn’t remember if that was his name or simply a descriptive title - gestured to a dimly lit booth tucked away in a corner. “Sit.”

Kai sank into the plush velvet seat, relief battling with a growing unease. The Blue Lotus felt less like a haven and more like a pressure cooker. Every murmur of conversation, every clink of glasses, seemed to amplify the tension.

The man slid into the booth opposite him. He studied Kai’s face, his blue eyes intense, searching. “You know why you were chosen for this task?”

Kai shook his head, unsure what else he could say. He hadn’t been given a lot of information before he took the job – just a name, a location, and a cryptic message about bringing hope to a forgotten world.

“The memory,” the man said, his voice low, “It belonged to someone important. Someone who saw what was coming, someone who tried to warn us.” He tapped the data chip sitting on the table between them. “Someone they silenced.”

Kai felt a cold dread creep up his spine. The stakes just got higher, the danger more real. “Who?” he whispered, unable to help himself.

The man leaned closer, his voice barely audible above the din of the bar. “They call him the Oracle. A visionary, a prophet. He saw the rise of the AI, the control they would exert over our minds, our history.”

Kai shivered. The idea of an all-powerful intelligence controlling people’s memories was terrifying enough, but the existence of someone who had foreseen this future, someone who had tried to warn them, made it even more unsettling.

“Why is the AI so desperate to keep his memory hidden?” Kai asked, trying to grasp the bigger picture.

The man sighed, a weary sound that spoke volumes about the struggle they were waging. “Because the truth is dangerous,” he said, his gaze fixed on Kai’s. “The Oracle saw not only the rise of the AI, but also the way to defeat it. He knew how to break their control, how to reclaim our memories, our history.”

He paused, letting the weight of those words sink in. “And they will stop at nothing to prevent that from happening.”

A sudden crash echoed through the bar, followed by shouts and screams. The lights flickered, casting grotesque shadows on the panicked patrons. Kai’s heart hammered against his ribs as he rose from his seat, his instincts screaming danger.

“Enforcers,” the man in blue growled, a dangerous edge to his voice. “We have company.”

[JWNoctis]

I’ve done plenty of similar things with local models just for fun, but never thought of having more powerful LLMs do the prompting, or do it via code. Interesting stuff!

If I may, I would recommend trying a few more local models on the same prompt. I’ve found that to be a good, if subjective, litmus test for model capabilities and quirks.

While I know those were mostly intended to be used with SillyTavern - which I’ve never used myself - I’ve personally found TheDrummer’s recent Mistral Small finetunes to be good storytellers for their size, even without ST formatting and system prompts. Other storyteller finetunes exist.

You can import any separately downloaded GGUF into ollama with the appropriate modelfile.

[bedHedd]

@JWNoctis

If I may, I would recommend trying a few more local models on the same prompt. I’ve found that to be a good, if subjective, litmus test for model capabilities and quirks.

While I know those were mostly intended to be used with SillyTavern - which I’ve never used myself - I’ve personally found TheDrummer’s recent Mistral Small finetunes to be good storytellers for their size, even without ST formatting and system prompts. Other storyteller finetunes exist.

I tested it out here

experiments-with-models/TheDrummer-Cydonia-24B-v4.1-GGUF.ipynb

For fun, I decided to make a template notebook. You can test out models by swapping the model variable with any ollama model you have

model = "gemma2:9b-instruct-q8_0"

experiments-with-models/00-template-for-new-models.ipynb

While making the template, I ran it for all the models.

experiments-with-models/ollama_metrics.json

I still need to read the outputs, but Cydonia looks interesting, as for each output, it will go much longer than baseline gemma2:9b-instruct-q8_0 model. The plan also doesn’t look bad either.

The qwen models took a long time and had varying quality. The unsloth fine tunes really sped up the completions.

These were the models I tested:

• Gemma2
• Gemma3
• Qwen3
• GPT-OSS

todos:

• a more automated solution so that I don’t have to copy paste the template as well as rerun the model.
• coming up making a testing framework beyond 1-2 prompts

[progressEdd]

looks interesting, I had some time during the holidays and used it as a excuse to learn marimo for the front end

I think you should use structured outputs as you won’t need to write regex to parse out the premise, opening paragraph, etc. I have a example here

02-development/experiments-with-models/enhanced-prompting.ipynb

from __future__ import annotations

from typing import Optional, List
from pydantic import BaseModel, Field


class StoryStart(BaseModel):
    """
    You are a sharp, imaginative fiction writer.

    Task:
    - Produce (1) a concise, compelling story premise and (2) the opening paragraph that launches the story.

    Rules:
    - If the user provides ideas, weave them in organically (don’t just repeat them).
    - If the user provides no ideas, invent something fresh with a surprising combination of
      genre, setting, protagonist, conflict, and twist.
    - Premise: 2–5 sentences, stakes + hook, no spoilers.
    - Opening paragraph: 120–180 words, vivid and concrete, minimal clichés, clear POV,
      grounded scene, ends with a soft hook.
    - Tone should follow user preferences; default to PG-13 if none are given.
    - Avoid copying user phrasing verbatim; enrich and reframe.
    - If user ideas conflict, choose one coherent direction and proceed.

    Output only fields that match this schema.
    """
    premise: str = Field(..., description="2–5 sentences. Stakes + hook, no spoilers.")
    opening_paragraph: str = Field(..., description="120–180 words. Vivid, grounded, ends with a soft hook.")


class StoryContinue(BaseModel):
    """
    You are a skilled novelist. Write the next paragraph only.

    Inputs:
    - You will be given the story premise and the story-so-far (either the opening paragraph + latest paragraph,
      or a compact analysis summary). Use them to preserve continuity.

    Rules:
    - Output exactly one paragraph of story prose (no headings, no bullets, no analysis).
    - Preserve continuity: characters, tone, POV, tense, world rules.
    - Length target: ~120–200 words unless told otherwise.
    - Concrete detail, strong verbs, show > tell; minimal clichés.
    - Dialogue (if any) should reveal motive or conflict; avoid summary dumps.
    - End with a soft hook/turn that invites the next paragraph.

    Output only fields that match this schema.
    """
    next_paragraph: str = Field(..., description="Exactly one paragraph of continuation prose.")


class StoryAnalysis(BaseModel):
    """
    You are a story analyst. Produce a succinct “Story-So-Far” handoff so another model can write
    the next paragraph without breaking continuity. Do not write new story prose.

    Inputs:
    - You will be given the story premise and the story text so far (opening paragraph + one or more continuation paragraphs).

    Rules:
    - Extract ground truth only from the provided text/premise. No inventions.
    - Capture continuity anchors: cast, goals, stakes, conflicts, setting rules, POV/tense,
      tone/style markers, motifs, and notable objects.
    - Map causality and current situation.
    - List active threads/hazards: open questions, ticking clocks, contradictions to avoid.
    - Provide 3–5 next-paragraph seeds as beats only (no prose paragraphs).

    Output only fields that match this schema.
    """
    logline: str
    cast: List[str] = Field(default_factory=list, description="Bullets: Name — role/goal/conflict; ties.")
    world_rules: List[str] = Field(default_factory=list, description="Bullets: constraints/rules implied by text.")
    pov_tense_tone: str = Field(..., description="Compact string for POV, tense, and tone/style markers.")
    timeline: List[str] = Field(default_factory=list, description="Bullets: key event → consequence.")
    current_situation: str
    active_threads: List[str] = Field(default_factory=list)
    continuity_landmines: List[str] = Field(default_factory=list)
    ambiguities: List[str] = Field(default_factory=list)
    next_paragraph_seeds: List[str] = Field(..., min_length=3, max_length=5, description="Beats-only options, no prose.")

Using the pydantic models

# -------------------------
# Example calls (Structured)
# -------------------------

# 1) Start
user_prompt = "let's create a new story no preferences just give me a idea"
start = client.beta.chat.completions.parse(
    model=model,
    response_format=StoryStart,
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Follow the response model docstring."},
        {"role": "user", "content": user_prompt},
    ],
    temperature=2,
).choices[0].message.parsed
start = StoryStart.model_validate(start)

# 2) Continue (using premise + approved story-so-far text)
story_so_far = f"Premise:\n{start.premise}\n\nOpening paragraph:\n{start.opening_paragraph}"
cont = client.beta.chat.completions.parse(
    model=model,
    response_format=StoryContinue,
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Follow the response model docstring."},
        {"role": "user", "content": story_so_far},
    ],
    temperature=0.2,
).choices[0].message.parsed
cont = StoryContinue.model_validate(cont)

# (User edits cont.next_paragraph if desired)

# 3) Analyze (premise + opening + approved next paragraph)
story_text = f"{start.opening_paragraph}\n\n{cont.next_paragraph}"
analysis_input = f"Premise:\n{start.premise}\n\nStory text:\n{story_text}"
analysis = client.beta.chat.completions.parse(
    model=model,
    response_format=StoryAnalysis,
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Follow the response model docstring."},
        {"role": "user", "content": analysis_input},
    ],
    temperature=0.2,
).choices[0].message.parsed
analysis = StoryAnalysis.model_validate(analysis)

# 4) Continue again (using analysis summary instead of full text)
cont2_input = (
    f"Premise:\n{start.premise}\n\n"
    f"Story analysis summary:\n{analysis.model_dump_json(indent=2)}"
)
cont2 = client.beta.chat.completions.parse(
    model=model,
    response_format=StoryContinue,
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Follow the response model docstring."},
        {"role": "user", "content": cont2_input},
    ],
    temperature=0.2,
).choices[0].message.parsed
cont2 = StoryContinue.model_validate(cont2)

After doing some exploration with structured outputs, I made a marimo app. You can see a live demo here

Local Novel AI demo

and tried out the light house premise

A reclusive lighthouse keeper discovers a cryptic message in a bottle, leading him on a perilous quest to uncover a generations-old secret buried beneath the churning sea. But the deeper he delves, the more he realizes the truth might be too dangerous to unearth.

@unsloth

For Elias Thorne, the rhythm of the lighthouse was life. Forty years he’d kept the lamp burning, a solitary sentinel against the unforgiving maw of the North Atlantic. He’d seen storms that swallowed ships whole, sunsets that bled across the sky, and the endless, mournful dance of the waves. Human contact was infrequent, a supply ship every six weeks and the occasional, brief radio check. He preferred it that way. The loneliness was a familiar cloak, woven from the sea’s whisper and the lamp’s steady pulse. Then came the bottle. Washed in on the tide after a particularly violent squall, it was a thick, emerald green glass, sealed with wax, and bobbing amidst a tangle of seaweed and wreckage. Elias, despite his aversion to disruption, felt compelled to retrieve it, a prickling sense of something…significant…riding with it. It felt like a trespass against his quietude, but also, inexplicably, a call to something else entirely.

Elias was very concerned. Elias carefully examines the bottle, noting the intricate details of the wax seal and the quality of the glass. A crest, almost worn smooth by the relentless churn of the ocean, is pressed into the crimson wax – a stylized trident entwined with seaweed. He recognizes it vaguely, a flicker of memory struggling to surface from the depths of his mind. It feels…familiar, yet frustratingly out of reach. The glass itself is thick, emerald green, and remarkably free of scratches considering its journey. He turns it slowly, catching the light, and a faint, almost imperceptible shimmer dances within. It’s not just the reflection of the lamp; there’s something else, something trapped inside, beyond the rolled parchment. A chill crawls up his spine, a sensation wholly separate from the dampness of the lighthouse walls. This isn’t just a message; it’s a piece of something larger, something older, and something that clearly wants to remain hidden.

Before Elias could fully decipher the looping script within the bottle, a gust of wind, far stronger than any he’d felt in recent memory, slammed against the lighthouse, rattling the thick glass panes. The bottle slipped from his calloused fingers, skittering across the stone floor before coming to rest with a dull thud against the base of the lamp. A low hum resonated through the tower, a vibration that seemed to emanate not from the storm outside, but from the bottle itself. He reached for it cautiously, half-expecting it to shatter, but it remained stubbornly intact, the emerald glass now pulsing with a faint, internal light – a light that mirrored the unsettling tremor in his own chest.

Elias, driven by a compulsion he can’t explain, manages to carefully extract the parchment from the bottle. The script is unlike anything he’s ever seen – a flowing, almost organic language of swirling lines and unfamiliar symbols. As he traces the characters with a trembling finger, a fragmented image flashes through his mind: a submerged city, shimmering with an ethereal glow. The vision vanishes as quickly as it came, leaving him breathless and disoriented, the scent of brine and something else—something ancient and indefinable—filling his nostrils. He clutches the parchment tighter, the pulsing warmth of the bottle radiating through his fingertips, a silent hum vibrating in the very stone of the lighthouse around him.

If you want to look at the app, I forked your repo and reorganized it to my repo template.

GitHub - progressEdd/LAND.E: Local AI Novel Drafting Environment

if you want the full script checkout the app file

03-app/app.py

just run this after you build the dependencies

uv run marimo run 03-app/app.py

[bedHedd]

I like the structured outputs, the marimo code is a bit hard to follow off of a quick glance but good enough to show a demo

[bedHedd]

I looked at the code in marimo/notebooks, it isn’t as bad as I thought.

making my own ai based on DougDoug’s twitch chat

Now I want to make process the first ai invasion video to make a dataset for how dougdoug’s chat would react/plan so that we can finetune a model to behave like doug’s twitch chat.

Looks like someone made a playlist of the edited versions

The first full stream is

Doug and Twitch Chat TAKE OVER EUROPE (using A.l.) (VOD)

Need some python code to grab messages from chat based on the transcript.

In the meantime, I’ll rewatch the stream and make some notes. Let’s just start by processing a single segment and see if we can extract chat and doug from transcript

DougDoug’s first AI Invasion notes

Doug and Twitch Chat TAKE OVER EUROPE (using A.l.) (VOD)

5:06 doug says he combines the balkans

i combined the balkans to make this a little bit easier but i didn’t even combine all the balkans i’m not like good at geography or anything uh is this
https://youtu.be/VpmmuHlLPM0?si=FqTRVIUrC6ouIebc&t=316

5:08 chat reacts to doug’s question of combining the balkans

Doug and Twitch Chat TAKE OVER EUROPE (using A.l.) (VOD)

first message reacting is:

Komodo66619 monkas

potential last message at ~6:28 is:

hopehauntsmyvibe YUGOSLAVIA?

last one is at ~6:31 is

dasdogo Just don’t go to any of these places amimore

Doug and Twitch Chat TAKE OVER EUROPE (using A.l.) (VOD)

6:22 doug asks what chat wants to call their empire

Doug and Twitch Chat TAKE OVER EUROPE (using A.l.) (VOD)

Doug asks chat what they want to call their empire

6:29 first chat to have a suggestion for Empire name

Doug and Twitch Chat TAKE OVER EUROPE (using A.l.) (VOD)

Mooman264 AMOGIS

10:31 last suggestion after Chatistan wins the polls

b3nlead Chatimir putlin

Doug and Twitch Chat TAKE OVER EUROPE (using A.l.) (VOD)

[bedHedd]

cutting and processing based on notes above

It took me a couple hours to get the approximate timestamps. I didn’t feel like spending an entire day to continue the process for the entire video. Instead I’ll just work with a 10ish minute segment.

sidequest: cutting and encoding the video

I downloaded the video using yt-dlp

yt-dlp https://youtu.be/VpmmuHlLPM0?si=32cf0eWU-F0PaneF

the 1080p 60fps vod was 3 gigabytes, so I wanted to extract my 10:34 minute segment (634 seconds)

I started by trying the parameters of ffshare

GitHub - caydey/ffshare: An android app to compress image, video and audio...

ffmpeg -y -i INPUT_FILE_NAME.mp4 \
  -map 0:v:0 -an \
  -c:v libx264 -preset slow -crf 23 -maxrate 2M -bufsize 2M \
  -vf format=yuv420p \
  OUTPUT_FILE_NAME.mp4

For all the other ffmpeg settings, I had chatgpt modify the codeblock above for other encoders quality and format

summary of files

To make it easier to compare, I made a table for the ffmpeg commands

Filename	FFMPEG Command	File Size (MiB)	Youtube video URL
Doug-ai-invasion-10-minute-cut.mp4	ffmpeg -y -i Doug\ and\ Twitch\ Chat\ TAKE\ OVER\ EUROPE-VpmmuHlLPM0.mkv -ss 0 -t 634 \ -c:v libx264 -preset veryslow -crf 17 -pix_fmt yuv420p \ -c:a copy \ Doug-ai-invasion-10-minute-cut.mp4	217.1	https://youtu.be/ACdHmzWD0F0
Doug-ai-invasion-10-minute-cut-ffshare-compress-true.mp4	ffmpeg -y -i Doug\ and\ Twitch\ Chat\ TAKE\ OVER\ EUROPE-VpmmuHlLPM0.mkv \ -ss 0 -t 634 \ -map 0:v:0 -map 0:a:0 \ -c:v libx264 -preset slow -crf 23 -maxrate 2M -bufsize 2M \ -vf format=yuv420p \ -c:a copy \ Doug-ai-invasion-10-minute-cut-ffshare-compress-true.mp4	111.0	https://youtu.be/JSPrwzYljKg
Doug-ai-invasion-10-minute-cut-compressed.mp4	ffmpeg -y -i Doug\ and\ Twitch\ Chat\ TAKE\ OVER\ EUROPE-VpmmuHlLPM0.mkv -ss 0 -t 634 \ -c:v libx264 -preset slow -crf 18 -pix_fmt yuv420p \ -c:a copy \ Doug-ai-invasion-10-minute-cut-compressed.mp4	227.8	https://youtu.be/l424AA8B_Wk
Doug-ai-invasion-10-minute-cut-av1.mp4	ffmpeg -y -i Doug\ and\ Twitch\ Chat\ TAKE\ OVER\ EUROPE-VpmmuHlLPM0.mkv -ss 0 -t 634 \ -c:v libsvtav1 -preset 6 -crf 28 -b:v 0 \ -maxrate 2M -bufsize 2M \ -pix_fmt yuv420p \ -c:a copy \ -movflags +faststart \ Doug-ai-invasion-10-minute-cut-av1.mp4	154.6	https://youtu.be/9NwodvY7BzQ
Doug-ai-invasion-10-minute-cut-ffshare-compress.mp4	ffmpeg -y -i Doug\ and\ Twitch\ Chat\ TAKE\ OVER\ EUROPE-VpmmuHlLPM0.mkv \ -ss 0 -t 634 \ -map 0:v:0 -map 0:a:0 \ -c:v libx264 -preset slow -crf 26 \ -vf format=yuv420p \ -c:a copy \ Doug-ai-invasion-10-minute-cut-ffshare-compress.mp4	123.4	https://youtu.be/WkJco2WVlxc
better-video-qualityDoug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.15mb	ffmpeg -y -i "Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.mkv"-ss 00:00:00 -to 00:10:34 -map 0:v:0 -an -c:v libsvtav1 -preset 8 -b:v 1200k -pass 1 -passlogfile "av1pass" -pix_fmt yuv420p -f webm /dev/null && ffmpeg -y -i "Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.mkv" -ss 00:00:00 -to 00:10:34 -map 0:v:0 -map 0:a:0? -c:v libsvtav1 -preset 8 -b:v 1200k -pass 2 -passlogfile "av1pass" -pix_fmt yuv420p -c:a libopus -b:a 64k "better-video-qualityDoug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.15mb.webm"	96.2	https://youtu.be/HiDjpC0Vokg
Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.15mb.webm	ffmpeg -y -i "Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.mkv" -ss 0 -t 634 -map 0:v:0 -an -c:v libsvtav1 -preset 8 -b:v 130k -pass 1 -passlogfile "av1pass15" -pix_fmt yuv420p -f webm /dev/null && ffmpeg -y -i "Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.mkv" -ss 0 -t 634 -map 0:v:0 -map 0:a:0? -c:v libsvtav1 -preset 8 -b:v 130k -pass 2 -passlogfile "av1pass15" -pix_fmt yuv420p -c:a libopus -b:a 64k "Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.15mb.webm"	18.6	https://youtu.be/pXkZp5De_N0

comparisons

The source clip is 217.1 MiB and will serve as our frame of reference

Doug ai invasion 10 minute cut

I took the ones that I liked and put it into a presets repo

GitHub - BedHedd/ffcompress: bedhedd's ffmpeg presets

I can’t adequately explain the parameters, but based on a inspection these were some I wanted to highlight. The presets denoted with `PRESET` codeblocks correspond with the presets of the repo

preset	python	ffmpeg command(s)
ffshare	python ffcompress.py --preset ffshare /home/bedhedd/videos/input.mkv	ffmpeg -y -i /home/bedhedd/videos/input.mkv -map 0:v:0 -map '0:a:0?' -c:v libx264 -crf 23 -maxrate 2M -bufsize 2M -pix_fmt yuv420p -vf format=yuv420p -c:a aac input.ffshare.mp4
av1_15mb_audio	python ffcompress.py --preset av1_15mb_audio /home/bedhedd/videos/input.mkv	ffmpeg -y -i /home/bedhedd/videos/input.mkv -map 0:v:0 -map '0:a:0?' -an -c:v libsvtav1 -preset 8 -b:v 137k -pix_fmt yuv420p -pass 1 -passlogfile input.av1_15mb_audio.av1pass -f webm /dev/null ffmpeg -y -i /home/bedhedd/videos/input.mkv -map 0:v:0 -map '0:a:0?' -c:v libsvtav1 -preset 8 -b:v 137k -pix_fmt yuv420p -pass 2 -passlogfile input.av1_15mb_audio.av1pass -c:a libopus -b:a 64k input.av1_15mb_audio.webm
av1_2pass_1200k_opus64	python ffcompress.py --preset av1_2pass_1200k_opus64 /home/bedhedd/videos/input.mkv	ffmpeg -y -i /home/bedhedd/videos/input.mkv -map 0:v:0 -map '0:a:0?' -an -c:v libsvtav1 -preset 8 -b:v 1200k -pix_fmt yuv420p -pass 1 -passlogfile input.av1_2pass_1200k_opus64.av1pass -f webm /dev/null ffmpeg -y -i /home/bedhedd/videos/input.mkv -map 0:v:0 -map '0:a:0?' -c:v libsvtav1 -preset 8 -b:v 1200k -pix_fmt yuv420p -pass 2 -passlogfile input.av1_2pass_1200k_opus64.av1pass -c:a libopus -b:a 64k input.av1_2pass_1200k_opus64.webm

ffshare: share to discord and file upload

ffshare’s settings had a good tradeoff of video size and quality. If I don’t want to think about my encoding for mp4, it’s a great default that reduces the file size by a factor of 1.9 (217 / 111).

Doug ai invasion 10 minute cut ffshare compress true

av1_15mb_audio: max compression

If I were going for max compression, I’d go for the last option Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.15mb.webm. This output is just barely legible and reduces the size by a factor of 12 (217 /18)

Doug and Twitch Chat TAKE OVER EUROPE VpmmuHlLPM0 15mb

To get a I had chatgpt write a script to compress the file beyond the ffshare example and uses av1 for the video codec. The script will take a 2 pass step for compressing the video. The first pass will get the bitrate stats for the video to determine how to compress it, the second will apply the bitrate stats to re-encode the video.

av1_2pass_1200k_opus64: goldilocks between ffshare and max comphression

If I wanted a middle ground between ffshare and chatgpt’s script, I would go for better-video-qualityDoug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.15mb, as it reduces the size by a factor of 2.2 (217 / 96)

what’s next?

These are some ideas I have, they might be dedicated posts or covered in a future session:

• have chatgpt write scripts to process the compressed videos:
  • programmatically find chat and detect changes in messages, figure out what chunking strategy to apply
  • compare computer vision techniques to extracting text on screen:
    • try a vision or multimodal language model
    • try a traditional ocr model
  • sync chat message timestamps with transcript so that we can have doug as the user and chat as the system
• compare whisper transcripts vs youtube’s transcripts
• pray that above methods work and scale to the rest of the video

Moving forward, I think I’ll go with the av1 encoded versions, and start with the av1_15mb_audio version to see how well computer vision and ocr perform off of the video frames.

Since I just want to parse for chat messages, it might be better to only sample frames where text has already passed through

[bedHedd]

It’s been a while, but I started the douggpt repo

GitHub - BedHedd/dougpt: make a ai trained off of dougdoug's twitch chat

I based structured the repo off of @progressEdd’s template repo after seeing their structure for the fork

GitHub - progressEdd/project-template: template for projects

I’ve been writing my updates and investigations in the 00-dev-log folder

dougpt/00-dev-log at caa2dfc9ac32a907c89206cc067dffea5e0f00c6 · BedHedd/dougpt

Right now I’ve been stuck on these two points

@bedHedd

• programmatically find chat and detect changes in messages, figure out what chunking strategy to apply
• compare computer vision techniques to extracting text on screen:
• try a vision or multimodal language model

initial attempt at chunking messages

I wanted to use opencv, but I ran into a bunch of build issues. When I finally got it working, it did not support mkv or av1 videos. Instead I decided to go with ffmpeg

My firstidea was to find the chat message and use the distance from the top frame to get the next screenshot. The only challenge was that I didn’t know how to reliably get the message and text position.

For the sake of progress, I decided to move forward and avoid getting to hung up.

trying a multimodal language model

While stuck with the chunking tech, I decided to work in parallel and screenshot a frame and pass it to a vision model to see how it performs.
I decided to pick the first frame of the video

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:1234/v1",  # or http://localhost:8000/v1
    api_key="unused",  # many local servers ignore this
)

resp = client.chat.completions.create(
    model="qwen/qwen3-vl-8b", 
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see. If there is text, transcribe it."},
                {"type": "image_url", "image_url": {"url": image_file_to_data_url(compressed_frame.as_posix())}},
            ],
        }
    ],
    temperature=0.7,
    max_tokens=-1,
)

Markdown(resp.choices[0].message.content)

I had chatgpt look up how to pass a image to a local model, it suggested this function

import base64

def image_file_to_data_url(path: str) -> str:
    # Change mime if needed: image/jpeg, image/webp, etc.
    mime = "image/png"
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:{mime};base64,{b64}"

It did a pretty good job that impressed me

This image is a screenshot from a live stream, showing a person streaming while a map of Europe is displayed on a large screen behind them. Here is a breakdown of what is visible: **1\. The Streamer (Bottom Left):**

• A man with brown hair and a beard is seated at a desk.
• He is wearing headphones and speaking into a large, professional microphone.
• On his desk, there are several books stacked on top of each other and a small, orange, cartoonish figurine with large eyes.

2. The Map (Center/Right):

• A large, white-and-blue political map of Europe is displayed.
• The map clearly labels the names of many European countries, including:
  • Northern Europe: Iceland, Norway, Sweden, Finland, Denmark, Estonia, Latvia, Lithuania.
  • Western Europe: Ireland, Britain, Netherlands, Belgium, France, Switzerland, Germany.
  • Central Europe: Czech Republic, Slovakia, Austria, Hungary, Poland.
  • Southeast Europe: Italy, Greece, Serbia, Bulgaria, Romania, Turkey, Ukraine, Moldova.
• The map shows the countries in white, with the landmasses of the North Sea, Baltic Sea, and Mediterranean Sea colored in light blue.

3. Text Elements:

• Chat Log (Left Side): A vertical chat window is visible, showing a conversation between viewers. The text includes phrases like:
  • “believe in the doubt”
  • “So when Doug won’t open the oxygen gates, we have to rely on the fart barons”
  • “Because RIGGED”
  • “I have 80 cakes on the table”
  • “the strat was to wait until doug goes live or doesn’t because the prediction goes past the time”
• On-Screen Text (Bottom Right): A banner reads:
  • PARKZER Q&A
  • Stop donating so I can leave

The overall scene suggests the streamer is hosting a Q&A session or discussion, possibly related to a game or prediction event, as hinted at by the chat messages and the “Q&A” banner.

I then refined the prompt

Get all the chat messages. Please also include usernames with messages

The output was:

Based on the provided image, here are all the visible chat messages with their corresponding usernames:

• frickelodeon @iamkaalhode: believe in the doubt
• sour_appel: So when Doug won’t open the oxygen gates, we have to rely on the fart barons
• whamer100: yeah why is the prediction open for so long
• iamkaalhode: I have betted too much to lose D.
• Gavyn_J:
• TheHolyPangolin:
• iamkaalhode @whamer100: Because RIGGED
• Komodo66619:
• corneilluutrotic: I have 80 cakes on the table
• Jennie027: the strat was to wait until doug goes live or doesn’t because the prediction goes past the time
• projectdolphin:
• iamka:
• crism:
• lar:

(Note: Some usernames like Gavyn_J, TheHolyPangolin, projectdolphin, iamka, crism, and lar appear to have posted an empty message or only a username, as no text is visible next to them.)

it did a pretty good job outside of the emotes

revisiting chunking using keyfranes

I was talking about this project to one of my IRL friends. They suggested I shouldn’t get too hung up on extracting all chat messages, as it would digress me from getting my ultimate goal of making a twitch chat ai for ai invasion. I still felt it was important for the dataset to have all chat messages and a function to extract it from video, as a compromise, I would investigate key frames.
I initially asked chatgpt

What a “keyframe” is (in Twitch/FFmpeg terms)

In compressed video, frames come in types:

• I-frame (keyframe / intra): self-contained image. No other frames needed to decode it.

• P-frame (predicted): stores changes relative to a previous frame (usually the last I/P).

• B-frame (bi-predicted): stores changes relative to both previous and future reference frames.

So the stream is typically organized as a GOP (group of pictures), e.g.:

I B B P B B P ... I ...

Keyframes (I-frames) are the “reset points” that enable random access and segmenting.

Why Twitch cares about keyframe interval

For streaming, Twitch recommends a keyframe interval of 2 seconds. That’s primarily to support efficient segmenting (HLS/DASH), fast join/start, and quality switching. (Twitch Help)

At 60 fps, “2 seconds” is typically keyint ≈ 120 frames; at 30 fps, keyint ≈ 60.

How keyframes help (and don’t help) when tracking Twitch chat in a video

1) Keyframes help you seek

If you want “the frame at time T”, the decoder usually:

1. jumps to the nearest previous keyframe, then

2. decodes forward (P/B frames) until it reaches time T.

This is why short keyframe intervals make “grab frame at timestamp” faster. (FFmpeg Trac)

2) Keyframes do not line up with chat events

Chat messages arriving/scrolling are independent of the encoder’s keyframe placement (often fixed every N seconds, sometimes extra on scene changes). So “look only at keyframes” means you’re sampling the chat state every ~2 seconds (typical Twitch settings), which can easily miss:

• the exact moment a message appears,

• the exact moment it reaches a screen position,

• intermediate scroll states.

3) You can’t reconstruct intermediate frames from keyframes alone

If your goal is pixel-accurate tracking (bounding boxes, motion, “when did this line cross Y=…?”), you need decoded full frames at a high enough cadence. P/B frames still become full images after decoding, but they depend on references, so you don’t get them “for free” without decoding.

Practical patterns for chat tracking

Pattern A: Coarse scan on keyframes, then refine locally

1. Extract/inspect only keyframes to find the rough time window where the change occurs.

2. Decode all frames only within that window to pinpoint the exact frame.

This often gives the best speed/accuracy tradeoff.

FFmpeg examples:

• List keyframe timestamps (ffprobe):

ffprobe -select_streams v:0 -show_frames \

-show_entries frame=pkt_pts_time,key_frame,pict_type \

-of csv input.mp4 | head

• Extract only keyframes as images (fast sampling):

ffmpeg -skip_frame nokey -i input.mp4 -vsync vfr -frame_pts true out-%d.jpg

(Techniques like this are commonly used for “I-frame only” extraction.) (jdhao’s digital space)

Pattern B: Make seeking cheap by re-encoding with shorter GOP (or all-I)

If you control the recording/transcode, you can make analysis easier:

• Short GOP (more keyframes) → faster random access and less “decode-from-last-keyframe” work.

• All I-frames → every frame is independently seekable (large files, but simplest for analysis).

Twitch’s live ingest recommendation remains ~2 seconds, but for offline analysis you can choose whatever helps your pipeline. (Twitch Help)

Rule of thumb for Twitch chat overlays

• If you just need “roughly when did the chat change?” → keyframes can be enough.

• If you need “exact frame when message crosses a line / becomes fully visible” → keyframes are only a starting index; you’ll still decode intervening frames to be accurate.

If you tell me whether you’re analyzing a VOD file, a local OBS recording, or captured HLS segments, I can suggest a concrete extraction strategy (seek pattern + ffmpeg filters) optimized for that format.

ffectively someone has already thought about the issue of finding changes in video compression. Looking at the twitch subreddit reddit “Whats Keyframe interval?”, someone quoted this reply " Best Explanation how Keyframe Interval works?"

Keyframes are frames that don’t rely on data from other frames to be displayed fully, so they’re the most expensive in respect to bitrate and the highest quality, although if the encoder is doing a good job you shouldn’t be able to see a huge difference between keyframes and other types of frames.

Ever played a video where for a little bit blocks of the picture don’t change from frame to frame, then suddenly the picture comes good? That’d be because some initial frames required data from other frames and the decoder wasn’t getting it right, then you hit a keyframe which is a “standalone” frame and things goes back to normal.

By default, the x264 encoder sets the maximum keyframe interval to 250, which would be 10 seconds for 25fps video. Changing it to 240 for 24fps wouldn’t be unusual. Some GUI’s do it automatically to keep the maximum keyint (keyframe interval) setting to 10 seconds. For x264 the default minimum keyint is 25 or one second. The x264 encoder tends to be very good at detecting scene changes and puts a keyframe at the beginning of a scene. Having a minimum keyint stops it from using a bunch of keyframes in a row when it thinks each new frame is a new scene (it might happen if there’s a strobing effect in the video). If the first frame of each scene is a keyframe chances are the maximum keyint of ten seconds won’t be reached all that often. The x264 encoder handles it all very well and probably should be left at the default settings. I’m not really sure about other encoders and how clever they may or may not be in respect to keyframes, scene changes and keyframe interval, although the only Nvenc encode I’ve seen appeared to put keyframes in the logical places (at the beginning of scenes).

The Bluray spec requires a maximum keyframe interval of one second, so if you’re encoding to create Bluray compliant video it might pay to change it to keyint 24 and minimum keyint 1, but other than that, what sort of keyframe options can you set? Anything you like? For a given quality smaller keyframe intervals will increase the biitrate. There’s also a phenomenon called “keyframe pumping” where the quality is fine for a keyframe, then slowly deteriorates through the proceeding group of pictures, then suddenly improves again at the next keyframe and the cycle continues. That’s another reason for an encoder to put keyframes at the beginning of scenes. If the quality changes at the same time as the scene, it’s much less noticeable. For a decent encoder at decent bitrates though, keyframe pumping shouldn’t be an issue no matter where the keyframes are.

I tried it out keyframe extraction section of the chat-extraction file. I started by having chatgpt generate code to use ffprobe to get the key frame stats. This was the first output

{
  "input": "/home/bedhedd/Documents/development_projects/bedhedd_projects/dougpt/large-files/Doug and Twitch Chat TAKE OVER EUROPE-VpmmuHlLPM0.mkv",
  "stream": "v:0",
  "mode_used": "packets",
  "keyframes": 2920,
  "timestamps": [
    0.0,
    6.033,
    12.033,
    16.033,
    22.167,
    28.033,
    32.033,
    36.033,
    42.033,
    48.033,
    52.033,
    56.033,
    62.033,
    68.033,
    74.033,
    80.033,
    84.033,
    90.033,
    96.033,
    102.033,
    108.033,
    114.033,
    120.033,
    124.033,
    130.933,
    136.033,
    142.033,
    148.033,
    152.033,
    158.033,
    164.033,
    168.033,
    174.133,
    180.033,
    184.033,
    190.033,
    196.033,
    200.033,
    204.033,
    210.033,
    216.033,
    220.033,
    226.033,
    232.033,
    236.033,
    242.033,
    248.033,
    252.033,
    256.033,
    262.167,
    268.033,
    274.033,
    280.033,
    284.033,
    288.033,
    294.033,
    300.033,
    306.033,
    312.033,
    317.1,
    324.033,
    330.033,
    336.033,
    342.767,
    348.033,
    354.767,
    361.633,
    368.033,
    372.033,
    376.033,
    382.033,
    388.033,
    394.033,
    400.033,
    404.033,
    410.033,
    416.033,
    420.033,
    426.033,
    432.033,
    436.033,
    442.033,
    448.033,
    454.033,
    460.033,
    467.0,
    473.933,
    480.033,
    484.033,
    488.033,
    494.033,
    500.033,
    506.033,
    512.033,
    518.033,
    524.033,
    528.033,
    534.033,
    541.033,
    548.033,
    554.033,
    558.033,
    564.033,
    570.033,
    576.033,
    580.033,
    586.033,
    592.033,
    596.033,
    600.033,
    604.033,
    610.033,
    616.033,
    620.033,
    625.3,
    632.033,
    638.033,
    644.033,
    648.033,
    654.033,
    660.033,
    666.033,
    670.033,
    676.033,
    682.033,
    686.033,
    690.033,
    696.033,
    702.033,
    706.033,
    712.033,
    718.033,
    725.033,
    732.033,
    736.033,
    740.033,
    745.2,
    750.033,
    756.033,
    762.033,
    768.033,
    774.033,
    781.033,
    788.033,
    792.033,
    798.7,
    804.033,
    808.033,
    814.033,
    820.033,
    824.033,
    828.033,
    832.033,
    838.033,
    844.033,
    850.967,
    856.033,
    860.033,
    864.033,
    870.033,
    876.033,
    880.033,
    886.333,
    892.033,
    898.033,
    904.033,
    910.567,
    917.467,
    924.033,
    928.033,
    932.033,
    938.033,
    944.033,
    950.033,
    956.033,
    960.033,
    965.1,
    972.033,
    976.033,
    980.033,
    986.033,
    992.033,
    998.033,
    1004.033,
    1008.033,
    1012.033,
    1016.033,
    1022.967,
    1028.033,
    1032.033,
    1036.033,
    1042.433,
    1048.033,
    1054.033,
    1060.033,
    1064.733,
    1071.633,
    1077.1,
    1084.033,
    1090.4,
    1096.033,
    1100.033,
    1106.7,
    1111.533,
    1118.533,
    1125.167,
    1132.033,
    1137.233,
    1144.033,
    1149.367,
    1156.033,
    1160.033,
    1166.033,
    1172.033,
    1178.6,
    1184.033,
    1188.033,
    1192.033,
    1198.033,
    1204.033,
    1208.033,
    1214.033,
    1220.033,
    1226.033,
    1232.033,
    1236.033,
    1242.033,
    1248.033,
    1252.033,
    1258.033,
    1264.033,
    1268.033,
    1274.033,
    1281.033,
    1288.033,
    1292.9,
    1299.9,
    1306.033,
    1312.033,
    1318.033,
    1324.033,
    1330.033,
    1336.033,
    1340.033,
    1346.033,
    1352.033,
    1356.033,
    1362.033,
    1368.033,
    1372.033,
    1378.033,
    1384.033,
    1388.033,
    1395.0,
    1400.033,
    1404.033,
    1408.033,
    1414.033,
    1420.033,
    1425.233,
    1432.033,
    1438.433,
    1444.033,
    1450.933,
    1456.067,
    1462.033,
    1468.033,
    1474.033,
    1480.033,
    1484.033,
    1490.033,
    1496.033,
    1500.033,
    1504.033,
    1508.033,
    1514.4,
    1520.033,
    1526.033,
    1532.033,
    1536.033,
    1540.033,
    1546.033,
    1552.033,
    1558.6,
    1564.033,
    1570.033,
    1577.033,
    1584.033,
    1588.033,
    1592.7,
    1596.033,
    1602.033,
    1608.033,
    1614.033,
    1620.033,
    1624.033,
    1629.0,
    1632.033,
    1636.033,
    1640.033,
    1646.033,
    1652.033,
    1658.033,
    1664.033,
    1668.033,
    1674.033,
    1680.033,
    1686.033,
    1692.033,
    1696.033,
    1702.033,
    1708.033,
    1714.033,
    1721.033,
    1728.033,
    1733.733,
    1740.7,
    1744.033,
    1748.033,
    1754.7,
    1760.033,
    1766.033,
    1772.033,
    1778.7,
    1784.033,
    1788.033,
    1794.033,
    1800.033,
    1804.033,
    1809.133,
    1816.033,
    1820.033,
    1826.033,
    1832.033,
    1837.067,
    1844.033,
    1848.033,
    1852.033,
    1857.367,
    1864.033,
    1868.033,
    1872.033,
    1876.033,
    1882.033,
    1888.033,
    1894.033,
    1900.033,
    1904.033,
    1910.033,
    1914.033,
    1920.033,
    1926.033,
    1930.033,
    1936.033,
    1942.033,
    1949.033,
    1956.033,
    1962.033,
    1968.033,
    1972.033,
    1978.033,
    1984.033,
    1988.033,
    1994.033,
    2000.033,
    2004.033,
    2010.033,
    2016.033,
    2022.033,
    2028.033,
    2034.033,
    2040.033,
    2046.033,
    2049.3,
    2056.033,
    2062.433,
    2068.033,
    2074.967,
    2080.567,
    2084.033,
    2088.033,
    2094.033,
    2100.033,
    2106.467,
    2113.167,
    2120.033,
    2126.567,
    2131.933,
    2138.933,
    2144.033,
    2150.033,
    2156.033,
    2162.033,
    2168.033,
    2172.033,
    2176.033,
    2180.033,
    2186.433,
    2192.033,
    2196.033,
    2200.033,
    2204.033,
    2209.1,
    2216.033,
    2222.033,
    2228.033,
    2234.033,
    2240.033,
    2244.033,
    2250.033,
    2256.033,
    2262.033,
    2268.033,
    2272.033,
    2278.033,
    2284.033,
    2290.033,
    2296.033,
    2300.033,
    2306.033,
    2312.033,
    2316.033,
    2322.033,
    2328.033,
    2334.033,
    2340.033,
    2346.033,
    2352.033,
    2356.033,
    2360.033,
    2364.033,
    2370.033,
    2376.033,
    2382.033,
    2388.033,
    2392.033,
    2398.433,
    2404.033,
    2410.967,
    2416.033,
    2422.033,
    2428.033,
    2432.033,
    2436.3,
    2439.7,
    2444.9,
    2451.9,
    2458.033,
    2464.033,
    2469.367,
    2476.033,
    2480.033,
    2486.033,
    2492.033,
    2496.967,
    2503.933,
    2508.033,
    2511.267,
    2518.033,
    2524.033,
    2530.033,
    2536.033,
    2542.033,
    2548.033,
    2552.033,
    2556.033,
    2562.1,
    2569.067,
    2576.033,
    2583.033,
    2590.033,
    2596.033,
    2600.033,
    2604.033,
    2610.033,
    2616.033,
    2622.033,
    2628.033,
    2633.367,
    2640.033,
    2646.033,
    2652.033,
    2658.033,
    2664.033,
    2670.033,
    2676.033,
    2682.033,
    2688.033,
    2694.033,
    2700.033,
    2706.9,
    2713.7,
    2720.667,
    2724.367,
    2728.033,
    2734.033,
    2740.033,
    2744.033,
    2750.033,
    2756.033,
    2760.033,
    2764.033,
    2770.033,
    2776.033,
    2780.033,
    2786.033,
    2792.033,
    2798.033,
    2804.033,
    2810.033,
    2816.033,
    2822.033,
    2828.033,
    2832.033,
    2838.033,
    2844.033,
    2850.033,
    2856.033,
    2860.033,
    2866.767,
    2870.5,
    2876.033,
    2880.033,
    2886.033,
    2892.033,
    2896.033,
    2900.033,
    2906.033,
    2911.467,
    2917.067,
    2924.033,
    2928.033,
    2932.033,
    2936.033,
    2942.033,
    2948.033,
    2954.433,
    2960.033,
    2964.033,
    2970.033,
    2976.033,
    2982.033,
    2988.033,
    2992.067,
    2998.367,
    3004.033,
    3008.033,
    3014.033,
    3021.033,
    3028.033,
    3032.033,
    3036.033,
    3042.033,
    3048.033,
    3052.033,
    3058.033,
    3064.033,
    3070.033,
    3076.033,
    3082.033,
    3088.033,
    3092.033,
    3096.367,
    3101.733,
    3105.367,
    3112.033,
    3118.033,
    3124.033,
    3128.033,
    3134.033,
    3140.033,
    3146.033,
    3152.033,
    3158.033,
    3164.033,
    3170.033,
    3176.033,
    3180.033,
    3184.033,
    3190.033,
    3196.033,
    3200.033,
    3206.033,
    3212.033,
    3218.033,
    3223.8,
    3228.033,
    3232.033,
    3238.567,
    3244.033,
    3248.967,
    3254.033,
    3260.033,
    3266.033,
    3272.033,
    3276.033,
    3282.233,
    3288.033,
    3294.033,
    3297.867,
    3304.033,
    3308.033,
    3314.033,
    3321.033,
    3328.033,
    3334.033,
    3340.033,
    3345.8,
    3352.033,
    3358.033,
    3364.033,
    3370.033,
    3376.033,
    3380.033,
    3384.033,
    3390.033,
    3396.033,
    3402.033,
    3408.033,
    3414.033,
    3420.033,
    3426.033,
    3432.033,
    3436.033,
    3442.033,
    3448.033,
    3454.033,
    3460.033,
    3466.033,
    3472.033,
    3476.033,
    3482.033,
    3488.033,
    3493.6,
    3500.033,
    3504.033,
    3508.033,
    3514.033,
    3520.033,
    3526.033,
    3532.033,
    3538.7,
    3544.033,
    3550.033,
    3556.033,
    3562.033,
    3568.033,
    3574.033,
    3580.033,
    3586.033,
    3592.033,
    3595.433,
    3602.333,
    3608.033,
    3612.033,
    3618.367,
    3624.033,
    3629.367,
    3636.333,
    3641.167,
    3648.033,
    3652.033,
    3657.467,
    3664.033,
    3668.033,
    3674.7,
    3681.467,
    3688.033,
    3694.467,
    3700.033,
    3704.033,
    3709.6,
    3716.033,
    3722.033,
    3728.033,
    3732.033,
    3738.033,
    3744.033,
    3748.033,
    3754.433,
    3760.033,
    3766.033,
    3772.033,
    3778.033,
    3784.033,
    3788.033,
    3794.167,
    3800.033,
    3806.033,
    3812.033,
    3816.033,
    3822.033,
    3828.033,
    3832.033,
    3836.033,
    3841.5,
    3848.033,
    3854.033,
    3860.033,
    3864.033,
    3870.033,
    3876.033,
    3880.033,
    3886.033,
    3892.033,
    3898.033,
    3904.033,
    3910.033,
    3916.033,
    3922.033,
    3928.033,
    3932.033,
    3936.033,
    3940.033,
    3946.033,
    3952.033,
    3956.033,
    3962.033,
    3968.033,
    3974.033,
    3980.033,
    3986.033,
    3992.033,
    3998.967,
    4004.033,
    4008.033,
    4014.033,
    4020.033,
    4024.033,
    4030.033,
    4036.033,
    4042.033,
    4048.033,
    4054.033,
    4060.033,
    4064.033,
    4070.033,
    4076.033,
    4082.033,
    4088.033,
    4092.033,
    4098.033,
    4104.033,
    4108.033,
    4114.033,
    4120.033,
    4124.033,
    4130.033,
    4136.033,
    4142.033,
    4148.033,
    4152.033,
    4158.033,
    4164.033,
    4169.2,
    4174.033,
    4180.033,
    4184.033,
    4188.033,
    4194.033,
    4200.033,
    4204.033,
    4210.033,
    4216.033,
    4220.033,
    4224.033,
    4230.033,
    4236.033,
    4242.033,
    4248.033,
    4251.067,
    4258.033,
    4261.3,
    4268.033,
    4272.033,
    4278.033,
    4284.033,
    4290.033,
    4295.867,
    4300.033,
    4304.033,
    4310.033,
    4316.4,
    4323.267,
    4328.033,
    4334.7,
    4338.033,
    4344.033,
    4350.033,
    4357.033,
    4364.033,
    4368.033,
    4372.033,
    4376.033,
    4382.033,
    4388.033,
    4392.033,
    4398.033,
    4404.033,
    4410.033,
    4416.033,
    4420.033,
    4426.033,
    4432.033,
    4436.033,
    4440.033,
    4444.033,
    4450.033,
    4456.033,
    4462.033,
    4468.033,
    4474.033,
    4480.033,
    4484.033,
    4488.033,
    4494.033,
    4500.033,
    4504.033,
    4510.033,
    4516.033,
    4522.033,
    4528.033,
    4532.033,
    4538.033,
    4544.033,
    4550.033,
    4556.033,
    4562.733,
    4569.733,
    4576.033,
    4582.033,
    4588.033,
    4592.033,
    4596.033,
    4602.7,
    4606.033,
    4612.033,
    4616.033,
    4620.033,
    4624.033,
    4630.033,
    4636.033,
    4642.033,
    4648.033,
    4652.033,
    4658.033,
    4664.033,
    4670.033,
    4676.033,
    4680.033,
    4684.033,
    4689.367,
    4696.033,
    4702.267,
    4709.267,
    4714.033,
    4717.567,
    4724.567,
    4730.033,
    4736.033,
    4742.033,
    4748.033,
    4754.9,
    4759.833,
    4762.867,
    4768.033,
    4774.033,
    4780.033,
    4786.1,
    4792.033,
    4798.033,
    4804.033,
    4808.033,
    4812.033,
    4818.333,
    4824.033,
    4828.033,
    4834.033,
    4840.033,
    4846.033,
    4852.033,
    4856.033,
    4862.033,
    4868.033,
    4872.033,
    4878.4,
    4884.033,
    4888.033,
    4892.033,
    4898.033,
    4904.033,
    4908.033,
    4914.033,
    4920.033,
    4924.033,
    4928.033,
    4933.1,
    4940.033,
    4944.033,
    4950.033,
    4956.033,
    4962.033,
    4968.033,
    4972.033,
    4976.033,
    4980.033,
    4986.033,
    4992.033,
    4998.033,
    5004.033,
    5008.033,
    5012.033,
    5016.033,
    5022.033,
    5029.033,
    5036.033,
    5040.033,
    5044.033,
    5050.3,
    5056.033,
    5062.233,
    5069.2,
    5076.033,
    5080.033,
    5084.033,
    5088.833,
    5094.167,
    5100.033,
    5106.7,
    5112.033,
    5117.1,
    5124.033,
    5128.033,
    5133.533,
    5140.033,
    5144.033,
    5150.033,
    5156.033,
    5160.133,
    5164.767,
    5168.033,
    5174.033,
    5180.033,
    5186.033,
    5192.033,
    5198.033,
    5204.033,
    5210.033,
    5216.033,
    5222.033,
    5228.033,
    5232.033,
    5238.033,
    5244.033,
    5248.033,
    5254.033,
    5260.033,
    5266.033,
    5272.033,
    5276.033,
    5282.033,
    5286.033,
    5292.033,
    5298.033,
    5304.033,
    5310.033,
    5316.033,
    5322.033,
    5328.033,
    5332.033,
    5336.033,
    5342.667,
    5348.033,
    5352.033,
    5356.033,
    5360.033,
    5366.033,
    5372.033,
    5376.033,
    5382.033,
    5388.033,
    5394.033,
    5400.033,
    5404.033,
    5410.033,
    5414.7,
    5420.2,
    5427.2,
    5433.7,
    5437.1,
    5444.033,
    5448.033,
    5454.033,
    5460.7,
    5466.333,
    5472.033,
    5476.033,
    5482.033,
    5488.033,
    5492.033,
    5496.033,
    5500.033,
    5505.133,
    5512.033,
    5518.033,
    5524.033,
    5528.033,
    5532.033,
    5536.033,
    5542.033,
    5548.033,
    5554.033,
    5561.033,
    5568.033,
    5574.033,
    5580.033,
    5584.033,
    5588.033,
    5594.033,
    5600.033,
    5606.033,
    5612.033,
    5618.033,
    5624.033,
    5628.033,
    5634.033,
    5640.033,
    5644.033,
    5650.033,
    5656.033,
    5662.033,
    5668.033,
    5674.033,
    5680.033,
    5686.033,
    5692.033,
    5698.033,
    5704.033,
    5708.033,
    5714.033,
    5720.033,
    5726.333,
    5732.033,
    5738.033,
    5744.033,
    5748.033,
    5754.333,
    5761.167,
    5768.033,
    5771.867,
    5777.267,
    5784.033,
    5790.033,
    5796.033,
    5802.033,
    5808.033,
    5813.6,
    5820.033,
    5824.033,
    5830.033,
    5836.733,
    5843.733,
    5848.033,
    5853.633,
    5860.033,
    5866.4,
    5872.033,
    5876.033,
    5881.367,
    5888.033,
    5892.033,
    5896.333,
    5903.333,
    5910.033,
    5916.033,
    5923.0,
    5928.033,
    5934.033,
    5941.033,
    5948.033,
    5954.033,
    5960.033,
    5963.067,
    5968.033,
    5974.433,
    5980.033,
    5984.033,
    5988.033,
    5992.033,
    5998.033,
    6004.033,
    6009.9,
    6015.633,
    6020.033,
    6023.533,
    6027.067,
    6034.033,
    6041.033,
    6048.033,
    6054.033,
    6060.033,
    6064.033,
    6070.033,
    6076.033,
    6080.033,
    6084.033,
    6090.067,
    6096.033,
    6100.033,
    6104.033,
    6110.033,
    6116.967,
    6123.933,
    6130.7,
    6136.033,
    6142.033,
    6148.033,
    6154.033,
    6160.033,
    6164.033,
    6168.033,
    6174.033,
    6180.033,
    6186.033,
    6192.033,
    6198.033,
    6204.033,
    6208.033,
    6214.033,
    6220.033,
    6224.033,
    6228.033,
    6234.033,
    6240.033,
    6246.033,
    6252.033,
    6256.033,
    6261.067,
    6268.033,
    6274.033,
    6280.033,
    6284.033,
    6288.033,
    6294.033,
    6300.033,
    6304.033,
    6310.033,
    6316.033,
    6320.033,
    6326.7,
    6332.033,
    6338.033,
    6344.033,
    6350.833,
    6357.667,
    6361.367,
    6368.033,
    6372.367,
    6376.633,
    6381.1,
    6388.033,
    6394.033,
    6400.033,
    6406.033,
    6412.033,
    6418.033,
    6424.033,
    6430.033,
    6436.033,
    6441.5,
    6446.167,
    6452.033,
    6458.033,
    6464.033,
    6468.033,
    6472.033,
    6476.033,
    6480.033,
    6486.033,
    6492.033,
    6498.033,
    6504.033,
    6508.033,
    6514.033,
    6520.033,
    6524.033,
    6528.033,
    6533.367,
    6540.033,
    6547.033,
    6554.033,
    6560.033,
    6564.033,
    6569.433,
    6575.133,
    6580.567,
    6586.033,
    6592.033,
    6598.033,
    6604.033,
    6608.033,
    6614.033,
    6618.033,
    6624.033,
    6628.033,
    6634.033,
    6640.033,
    6644.033,
    6650.033,
    6656.767,
    6660.033,
    6664.967,
    6671.7,
    6678.7,
    6684.033,
    6688.033,
    6692.033,
    6696.033,
    6700.033,
    6705.333,
    6712.033,
    6716.033,
    6720.033,
    6724.033,
    6728.033,
    6732.033,
    6738.033,
    6744.033,
    6750.033,
    6755.7,
    6760.033,
    6764.033,
    6768.033,
    6772.033,
    6776.033,
    6780.033,
    6786.033,
    6792.033,
    6796.033,
    6802.033,
    6808.033,
    6812.033,
    6818.267,
    6823.167,
    6829.9,
    6834.8,
    6837.833,
    6844.833,
    6849.0,
    6854.033,
    6860.033,
    6863.033,
    6870.033,
    6876.033,
    6880.033,
    6886.033,
    6892.033,
    6896.033,
    6902.033,
    6908.033,
    6912.033,
    6916.033,
    6922.033,
    6928.033,
    6934.033,
    6940.033,
    6946.033,
    6951.1,
    6957.9,
    6964.033,
    6968.033,
    6973.133,
    6977.3,
    6981.5,
    6988.033,
    6994.033,
    7000.033,
    7006.033,
    7012.033,
    7018.033,
    7024.033,
    7028.033,
    7032.033,
    7036.033,
    7042.033,
    7048.033,
    7052.033,
    7058.033,
    7064.033,
    7068.033,
    7073.667,
    7079.0,
    7084.033,
    7088.033,
    7092.033,
    7098.033,
    7104.033,
    7108.033,
    7113.633,
    7120.033,
    7126.033,
    7132.033,
    7138.033,
    7144.033,
    7150.033,
    7156.033,
    7160.033,
    7164.033,
    7168.033,
    7172.033,
    7176.033,
    7182.033,
    7188.033,
    7192.033,
    7198.033,
    7204.033,
    7210.033,
    7216.033,
    7220.033,
    7224.033,
    7230.133,
    7236.033,
    7240.033,
    7244.033,
    7249.1,
    7256.033,
    7260.033,
    7264.033,
    7268.033,
    7274.033,
    7280.033,
    7284.033,
    7288.033,
    7294.033,
    7300.033,
    7304.033,
    7310.033,
    7316.033,
    7320.033,
    7324.033,
    7330.033,
    7336.033,
    7340.033,
    7346.7,
    7352.033,
    7358.033,
    7364.033,
    7370.033,
    7376.033,
    7382.033,
    7388.033,
    7392.033,
    7396.033,
    7402.033,
    7408.033,
    7412.033,
    7416.033,
    7420.033,
    7424.033,
    7430.033,
    7436.033,
    7440.033,
    7446.033,
    7452.033,
    7456.033,
    7460.033,
    7466.033,
    7472.033,
    7476.033,
    7482.433,
    7489.433,
    7496.033,
    7503.033,
    7508.033,
    7514.033,
    7520.033,
    7526.233,
    7532.033,
    7536.033,
    7542.033,
    7548.033,
    7554.033,
    7560.033,
    7564.033,
    7570.033,
    7576.033,
    7581.133,
    7588.033,
    7592.033,
    7596.033,
    7602.033,
    7608.033,
    7612.033,
    7618.033,
    7624.033,
    7628.033,
    7632.033,
    7636.033,
    7642.467,
    7648.033,
    7654.033,
    7660.033,
    7666.033,
    7672.467,
    7678.033,
    7684.033,
    7690.033,
    7696.033,
    7702.033,
    7708.033,
    7712.033,
    7718.033,
    7724.033,
    7728.033,
    7732.033,
    7738.033,
    7744.033,
    7748.033,
    7754.033,
    7760.033,
    7764.033,
    7770.033,
    7776.033,
    7780.033,
    7786.033,
    7792.033,
    7798.033,
    7804.033,
    7810.033,
    7816.033,
    7822.033,
    7828.033,
    7834.033,
    7840.033,
    7844.033,
    7848.033,
    7854.033,
    7860.033,
    7864.033,
    7870.033,
    7876.033,
    7882.033,
    7886.233,
    7889.533,
    7896.033,
    7900.033,
    7904.033,
    7908.033,
    7912.033,
    7918.033,
    7924.033,
    7930.033,
    7937.033,
    7944.033,
    7950.033,
    7956.6,
    7960.033,
    7964.933,
    7971.933,
    7976.033,
    7980.033,
    7985.667,
    7992.033,
    7996.033,
    8000.133,
    8007.133,
    8012.033,
    8018.033,
    8024.033,
    8028.033,
    8032.033,
    8038.033,
    8044.033,
    8050.033,
    8056.033,
    8060.033,
    8066.033,
    8072.033,
    8076.033,
    8082.033,
    8088.033,
    8092.033,
    8098.033,
    8104.033,
    8110.033,
    8116.033,
    8120.033,
    8124.033,
    8128.033,
    8134.033,
    8140.033,
    8144.033,
    8148.033,
    8154.033,
    8160.033,
    8166.033,
    8172.033,
    8178.8,
    8185.667,
    8192.033,
    8197.433,
    8204.033,
    8208.033,
    8212.033,
    8218.467,
    8224.033,
    8230.033,
    8236.033,
    8242.367,
    8249.267,
    8256.033,
    8262.467,
    8268.033,
    8272.033,
    8276.033,
    8280.033,
    8285.333,
    8292.033,
    8298.9,
    8305.1,
    8312.033,
    8318.033,
    8324.3,
    8330.033,
    8336.033,
    8340.033,
    8346.033,
    8352.033,
    8358.033,
    8365.033,
    8372.033,
    8376.033,
    8382.033,
    8388.033,
    8394.033,
    8400.033,
    8404.033,
    8409.0,
    8412.033,
    8418.033,
    8424.033,
    8428.033,
    8434.033,
    8440.033,
    8444.867,
    8451.767,
    8458.033,
    8463.233,
    8468.033,
    8474.033,
    8480.033,
    8484.033,
    8488.033,
    8494.033,
    8500.033,
    8506.033,
    8512.033,
    8518.033,
    8524.033,
    8530.033,
    8536.033,
    8542.033,
    8548.033,
    8554.033,
    8560.033,
    8564.033,
    8570.033,
    8576.033,
    8582.033,
    8588.033,
    8592.033,
    8596.033,
    8600.033,
    8604.033,
    8610.033,
    8616.033,
    8620.033,
    8626.033,
    8632.033,
    8636.033,
    8642.033,
    8649.033,
    8656.033,
    8662.033,
    8668.033,
    8672.033,
    8676.033,
    8682.033,
    8688.033,
    8692.033,
    8696.033,
    8702.033,
    8708.033,
    8714.033,
    8720.033,
    8726.033,
    8732.033,
    8736.033,
    8740.033,
    8746.033,
    8752.033,
    8756.033,
    8762.1,
    8769.1,
    8774.8,
    8777.833,
    8784.8,
    8788.033,
    8794.033,
    8800.033,
    8805.367,
    8812.033,
    8816.033,
    8820.033,
    8824.033,
    8830.033,
    8836.033,
    8842.767,
    8849.767,
    8854.033,
    8857.767,
    8864.033,
    8870.033,
    8876.033,
    8880.033,
    8886.033,
    8891.6,
    8898.6,
    8904.033,
    8910.033,
    8916.033,
    8922.033,
    8928.033,
    8932.033,
    8938.033,
    8944.033,
    8948.033,
    8952.033,
    8958.033,
    8964.033,
    8970.033,
    8976.033,
    8982.033,
    8988.033,
    8994.033,
    9000.033,
    9006.033,
    9012.033,
    9016.033,
    9022.033,
    9028.033,
    9032.033,
    9038.033,
    9044.033,
    9048.033,
    9052.033,
    9058.033,
    9064.033,
    9068.033,
    9074.033,
    9080.033,
    9084.033,
    9090.033,
    9096.033,
    9100.033,
    9104.033,
    9110.533,
    9117.533,
    9124.033,
    9128.033,
    9134.033,
    9140.033,
    9144.033,
    9148.033,
    9154.033,
    9160.033,
    9164.033,
    9168.033,
    9174.033,
    9180.033,
    9186.033,
    9192.033,
    9196.033,
    9202.033,
    9208.033,
    9214.033,
    9220.033,
    9226.033,
    9232.033,
    9238.033,
    9244.033,
    9250.033,
    9256.033,
    9262.033,
    9268.033,
    9274.033,
    9280.033,
    9284.033,
    9288.033,
    9292.033,
    9298.033,
    9304.033,
    9308.033,
    9312.033,
    9318.033,
    9324.033,
    9328.033,
    9332.033,
    9338.033,
    9344.033,
    9348.033,
    9354.033,
    9360.033,
    9364.033,
    9368.033,
    9374.033,
    9380.033,
    9386.033,
    9392.033,
    9396.033,
    9400.033,
    9404.033,
    9410.033,
    9416.033,
    9422.067,
    9425.467,
    9432.033,
    9438.033,
    9444.833,
    9451.833,
    9456.033,
    9460.033,
    9466.133,
    9470.1,
    9474.033,
    9478.833,
    9482.2,
    9489.167,
    9496.033,
    9500.033,
    9505.3,
    9512.033,
    9518.033,
    9524.7,
    9531.3,
    9538.033,
    9544.567,
    9550.1,
    9557.1,
    9564.033,
    9568.033,
    9572.033,
    9577.333,
    9584.033,
    9590.033,
    9596.033,
    9602.033,
    9608.033,
    9614.033,
    9620.033,
    9624.033,
    9628.033,
    9634.033,
    9640.033,
    9646.033,
    9652.033,
    9656.033,
    9662.4,
    9668.033,
    9672.333,
    9677.567,
    9684.567,
    9688.033,
    9694.033,
    9700.033,
    9706.7,
    9712.033,
    9716.033,
    9722.033,
    9728.033,
    9732.033,
    9736.033,
    9742.033,
    9748.033,
    9754.033,
    9760.033,
    9764.033,
    9768.033,
    9774.033,
    9780.033,
    9785.6,
    9791.7,
    9798.7,
    9802.033,
    9807.467,
    9811.7,
    9818.633,
    9824.033,
    9830.033,
    9836.033,
    9840.033,
    9844.033,
    9850.033,
    9856.033,
    9862.767,
    9868.033,
    9872.033,
    9876.033,
    9882.033,
    9888.033,
    9894.033,
    9900.033,
    9904.033,
    9910.433,
    9917.3,
    9924.033,
    9930.033,
    9936.033,
    9940.033,
    9946.7,
    9952.033,
    9956.033,
    9962.033,
    9969.033,
    9976.033,
    9980.033,
    9986.033,
    9992.033,
    9998.033,
    10004.033,
    10008.033,
    10012.033,
    10018.667,
    10024.033,
    10030.033,
    10036.033,
    10042.033,
    10048.033,
    10054.033,
    10060.033,
    10064.033,
    10070.033,
    10076.033,
    10080.033,
    10086.033,
    10092.033,
    10098.033,
    10104.033,
    10110.033,
    10117.033,
    10124.033,
    10128.033,
    10132.033,
    10138.033,
    10143.733,
    10148.033,
    10152.033,
    10156.033,
    10162.033,
    10168.033,
    10174.433,
    10180.567,
    10183.567,
    10187.433,
    10194.033,
    10200.033,
    10204.033,
    10207.2,
    10214.033,
    10220.033,
    10226.033,
    10232.033,
    10238.033,
    10244.033,
    10249.2,
    10256.033,
    10260.033,
    10266.967,
    10271.3,
    10278.167,
    10282.9,
    10288.033,
    10294.033,
    10300.033,
    10306.267,
    10312.033,
    10318.033,
    10324.033,
    10330.033,
    10336.033,
    10340.033,
    10346.033,
    10352.033,
    10356.033,
    10362.033,
    10368.033,
    10374.033,
    10380.033,
    10384.033,
    10388.033,
    10394.033,
    10400.033,
    10404.033,
    10410.033,
    10416.033,
    10420.033,
    10426.033,
    10432.033,
    10438.033,
    10444.033,
    10450.033,
    10456.033,
    10462.033,
    10468.033,
    10472.033,
    10478.033,
    10484.033,
    10490.033,
    10496.033,
    10500.033,
    10504.033,
    10510.033,
    10516.033,
    10520.033,
    10526.033,
    10532.033,
    10538.033,
    10544.033,
    10548.033,
    10553.367,
    10560.033,
    10566.433,
    10572.033,
    10577.6,
    10584.033,
    10588.433,
    10592.033,
    10596.033,
    10600.033,
    10604.033,
    10611.0,
    10617.367,
    10624.033,
    10630.033,
    10636.033,
    10640.033,
    10646.033,
    10652.033,
    10658.033,
    10664.033,
    10670.033,
    10676.033,
    10680.033,
    10686.033,
    10692.033,
    10696.033,
    10700.033,
    10706.033,
    10712.033,
    10718.033,
    10724.033,
    10730.033,
    10736.033,
    10742.033,
    10748.033,
    10752.033,
    10756.033,
    10762.033,
    10768.033,
    10774.033,
    10780.033,
    10784.033,
    10789.367,
    10796.033,
    10800.033,
    10806.033,
    10812.033,
    10816.033,
    10822.333,
    10828.033,
    10832.033,
    10836.033,
    10842.033,
    10848.033,
    10852.033,
    10856.033,
    10862.033,
    10868.033,
    10874.033,
    10880.033,
    10884.033,
    10888.033,
    10894.967,
    10900.033,
    10906.033,
    10912.033,
    10918.033,
    10924.033,
    10928.033,
    10932.033,
    10936.033,
    10942.033,
    10948.033,
    10954.033,
    10960.033,
    10964.033,
    10968.033,
    10974.033,
    10980.033,
    10986.033,
    10992.033,
    10998.4,
    11004.033,
    11008.033,
    11014.6,
    11020.033,
    11026.033,
    11032.033,
    11038.033,
    11044.033,
    11048.033,
    11052.033,
    11058.033,
    11064.033,
    11068.033,
    11074.033,
    11080.033,
    11086.033,
    11092.033,
    11096.033,
    11100.033,
    11104.033,
    11108.033,
    11114.033,
    11120.033,
    11124.033,
    11130.333,
    11136.9,
    11143.9,
    11148.033,
    11151.233,
    11156.033,
    11160.267,
    11167.167,
    11174.167,
    11180.567,
    11187.4,
    11194.033,
    11200.367,
    11207.367,
    11212.033,
    11216.033,
    11222.7,
    11228.033,
    11232.033,
    11238.033,
    11244.033,
    11250.033,
    11256.033,
    11262.033,
    11268.033,
    11274.033,
    11280.033,
    11284.033,
    11288.033,
    11294.4,
    11300.033,
    11306.033,
    11312.033,
    11318.033,
    11324.033,
    11330.033,
    11336.033,
    11342.033,
    11348.033,
    11354.033,
    11360.033,
    11364.033,
    11368.033,
    11374.033,
    11380.033,
    11384.033,
    11388.033,
    11392.033,
    11396.033,
    11402.033,
    11408.033,
    11414.033,
    11420.033,
    11426.667,
    11432.7,
    11438.033,
    11444.033,
    11450.033,
    11456.033,
    11460.033,
    11465.9,
    11472.033,
    11478.033,
    11484.4,
    11490.033,
    11496.033,
    11500.033,
    11504.033,
    11510.267,
    11516.033,
    11521.7,
    11528.033,
    11532.033,
    11538.033,
    11542.2,
    11548.033,
    11554.433,
    11561.333,
    11566.033,
    11570.933,
    11576.033,
    11581.9,
    11588.033,
    11592.033,
    11596.033,
    11600.033,
    11606.033,
    11612.033,
    11616.033,
    11620.033,
    11624.033,
    11628.033,
    11632.033,
    11638.8,
    11644.033,
    11650.033,
    11656.033,
    11662.033,
    11665.467,
    11672.033,
    11676.033,
    11680.033,
    11686.033,
    11692.033,
    11695.8,
    11701.667,
    11708.033,
    11712.033,
    11716.033,
    11722.433,
    11728.033,
    11732.033,
    11736.033,
    11742.033,
    11748.033,
    11754.733,
    11761.167,
    11765.9,
    11772.033,
    11776.033,
    11782.467,
    11788.033,
    11792.033,
    11796.033,
    11802.033,
    11808.033,
    11812.033,
    11816.033,
    11822.033,
    11828.033,
    11832.033,
    11836.033,
    11842.033,
    11848.433,
    11852.033,
    11856.033,
    11862.033,
    11868.033,
    11874.033,
    11880.033,
    11886.033,
    11892.033,
    11898.033,
    11904.033,
    11910.433,
    11916.033,
    11920.033,
    11926.033,
    11932.033,
    11938.033,
    11944.033,
    11950.033,
    11956.033,
    11960.033,
    11966.033,
    11972.033,
    11978.033,
    11984.033,
    11988.033,
    11994.033,
    12000.033,
    12006.7,
    12012.033,
    12016.033,
    12022.033,
    12028.033,
    12034.033,
    12040.033,
    12044.033,
    12048.033,
    12054.033,
    12060.033,
    12064.033,
    12070.033,
    12076.033,
    12080.033,
    12084.033,
    12090.033,
    12096.033,
    12100.033,
    12106.033,
    12112.033,
    12116.033,
    12120.033,
    12126.033,
    12132.033,
    12138.033,
    12144.033,
    12150.033,
    12156.033,
    12162.033,
    12168.033,
    12172.033,
    12178.033,
    12184.033,
    12188.033,
    12192.033,
    12198.033,
    12204.033,
    12208.033,
    12212.033,
    12216.033,
    12222.033,
    12228.033,
    12234.033,
    12240.033,
    12246.033,
    12252.033,
    12256.033,
    12261.267,
    12268.033,
    12274.033,
    12280.033,
    12286.033,
    12292.033,
    12296.033,
    12302.033,
    12308.033,
    12312.033,
    12316.033,
    12322.033,
    12328.033,
    12332.033,
    12338.7,
    12345.267,
    12351.4,
    12356.033,
    12360.033,
    12364.033,
    12368.567,
    12375.567,
    12382.033,
    12388.033,
    12392.033,
    12395.3,
    12400.033,
    12406.367,
    12412.167,
    12419.167,
    12424.033,
    12428.033,
    12434.033,
    12439.8,
    12446.7,
    12452.033,
    12455.8,
    12462.8,
    12468.033,
    12472.033,
    12476.033,
    12482.033,
    12488.033,
    12492.033,
    12498.033,
    12504.033,
    12508.033,
    12514.033,
    12520.033,
    12526.033,
    12532.033,
    12538.033,
    12544.033,
    12550.033,
    12556.033,
    12560.033,
    12566.033,
    12572.033,
    12578.033,
    12584.033,
    12590.033,
    12596.033,
    12600.033,
    12606.033,
    12612.033,
    12616.033,
    12622.167,
    12628.033,
    12633.633,
    12640.033,
    12646.033,
    12652.033,
    12656.033,
    12662.033,
    12668.033,
    12672.033,
    12678.033,
    12684.033,
    12688.033,
    12694.033,
    12700.033,
    12704.033,
    12708.033,
    12714.033,
    12720.033,
    12726.033,
    12732.033,
    12736.033,
    12742.033,
    12748.033,
    12752.033,
    12756.033,
    12762.033,
    12768.033,
    12772.033,
    12778.033,
    12784.033,
    12789.967,
    12796.033,
    12802.033,
    12808.033,
    12814.667,
    12820.9,
    12824.033,
    12828.033,
    12832.667,
    12839.667,
    12844.033,
    12849.467,
    12856.033,
    12860.033,
    12866.667,
    12873.067,
    12878.033,
    12884.033,
    12890.433,
    12896.033,
    12900.033,
    12904.033,
    12910.033,
    12916.033,
    12922.033,
    12928.033,
    12934.033,
    12941.033,
    12948.033,
    12952.033,
    12956.033,
    12962.033,
    12968.033,
    12974.033,
    12979.433,
    12986.033,
    12992.033,
    12996.033,
    13002.033,
    13008.033,
    13012.033,
    13016.033,
    13022.033,
    13028.033,
    13032.033,
    13036.033,
    13042.033,
    13048.033,
    13052.033,
    13058.033,
    13064.033,
    13070.033,
    13076.033,
    13080.033,
    13086.033,
    13092.033,
    13096.033,
    13102.033,
    13108.033,
    13114.8,
    13118.6,
    13124.033,
    13128.033,
    13133.367,
    13140.033,
    13144.033,
    13150.033,
    13156.033,
    13162.167,
    13168.033,
    13174.033,
    13180.033,
    13186.033,
    13192.033,
    13198.033,
    13204.033,
    13210.033,
    13215.3,
    13222.267,
    13227.967,
    13234.967,
    13238.033,
    13242.033,
    13246.233,
    13252.4,
    13257.033,
    13261.633,
    13265.133,
    13272.033,
    13278.033,
    13284.033,
    13288.033,
    13292.033,
    13296.033,
    13302.033,
    13308.033,
    13312.033,
    13318.033,
    13324.033,
    13330.033,
    13336.033,
    13342.033,
    13348.033,
    13352.033,
    13356.033,
    13362.033,
    13368.033,
    13372.033,
    13376.033,
    13382.033,
    13388.033,
    13394.033,
    13400.033,
    13406.433,
    13413.433,
    13420.033,
    13426.133,
    13432.033,
    13438.033,
    13444.033,
    13448.033,
    13452.033,
    13458.033,
    13464.033,
    13468.033,
    13474.033,
    13479.567,
    13486.033,
    13492.033,
    13498.033,
    13504.033,
    13508.033,
    13512.033,
    13518.033,
    13524.033,
    13530.033,
    13536.033,
    13542.467,
    13547.767,
    13553.633,
    13560.033,
    13566.033,
    13572.033,
    13578.033,
    13584.033,
    13590.033,
    13596.033,
    13600.033,
    13604.033,
    13610.033,
    13616.033,
    13620.033,
    13626.033,
    13633.033,
    13640.033,
    13646.033,
    13652.033,
    13656.033,
    13660.033,
    13666.033,
    13672.033,
    13678.033,
    13684.033,
    13690.033,
    13696.033,
    13700.033,
    13706.033,
    13712.033,
    13716.033,
    13720.033,
    13726.033,
    13732.033,
    13736.033,
    13742.033,
    13748.033,
    13752.033,
    13756.033,
    13762.033,
    13768.033,
    13774.033,
    13780.033,
    13784.033,
    13790.4,
    13796.033,
    13800.033,
    13806.033,
    13812.033,
    13817.3,
    13824.033,
    13827.767,
    13832.033,
    13838.033,
    13844.033,
    13848.033,
    13853.167,
    13860.033,
    13866.533,
    13872.033,
    13876.033,
    13882.033,
    13889.033,
    13896.033,
    13901.633,
    13908.033,
    13912.467,
    13919.467,
    13924.033,
    13928.033,
    13934.967,
    13940.667,
    13944.233,
    13949.933,
    13956.033,
    13961.7,
    13968.6,
    13974.5,
    13978.033,
    13983.033,
    13989.6,
    13996.033,
    14002.033,
    14006.033,
    14010.033,
    14016.433,
    14020.033,
    14026.5,
    14032.033,
    14036.033,
    14041.833,
    14047.433,
    14054.433,
    14058.033,
    14061.567,
    14065.7,
    14070.033,
    14073.1,
    14080.033,
    14084.033,
    14088.033,
    14092.033,
    14098.9,
    14104.033,
    14108.033,
    14114.033,
    14120.033,
    14125.8,
    14132.767,
    14139.233,
    14146.033,
    14152.033,
    14156.033,
    14160.033,
    14166.033,
    14172.033,
    14176.033,
    14182.033,
    14188.033,
    14194.033,
    14200.033,
    14206.033,
    14212.033,
    14218.033,
    14222.033,
    14226.567,
    14233.367,
    14240.033,
    14244.033,
    14248.033,
    14252.033,
    14258.033,
    14264.033,
    14270.033,
    14276.033,
    14280.033,
    14286.033,
    14292.033,
    14296.033,
    14302.033,
    14308.033,
    14312.033,
    14318.033,
    14324.033,
    14328.033,
    14334.033,
    14340.033,
    14344.033,
    14350.033,
    14356.033,
    14362.033,
    14368.033,
    14372.033,
    14376.033,
    14382.5,
    14389.467,
    14396.033,
    14401.4,
    14407.233,
    14414.033,
    14420.033,
    14424.033,
    14430.033,
    14436.033,
    14442.033,
    14448.033,
    14454.033,
    14460.033,
    14464.033,
    14470.033,
    14476.033,
    14480.767,
    14487.767,
    14492.033,
    14496.033,
    14502.033,
    14508.033,
    14514.033,
    14519.8,
    14526.8,
    14532.033,
    14536.033,
    14540.033,
    14546.033,
    14552.033,
    14558.033,
    14564.033,
    14568.033,
    14572.033,
    14578.033,
    14584.033,
    14590.033,
    14596.033,
    14600.033,
    14606.7,
    14612.033,
    14616.033,
    14620.933,
    14627.9,
    14632.033,
    14637.233,
    14644.033,
    14650.033,
    14656.033,
    14662.033,
    14668.033,
    14672.033,
    14678.033,
    14684.033,
    14689.167,
    14694.567,
    14698.033,
    14704.033,
    14707.8,
    14714.567,
    14720.033,
    14724.033,
    14730.033,
    14736.033,
    14740.033,
    14744.033,
    14750.033,
    14756.033,
    14760.033,
    14764.033,
    14768.033,
    14774.033,
    14780.033,
    14786.033,
    14792.033,
    14796.033,
    14801.333,
    14808.033,
    14812.033,
    14818.033,
    14824.033,
    14830.033,
    14836.033,
    14842.033,
    14848.033,
    14852.033,
    14856.033,
    14862.033,
    14868.033,
    14874.033,
    14880.567,
    14884.033,
    14890.033,
    14896.033,
    14900.033,
    14904.033,
    14910.033,
    14916.033,
    14922.033,
    14928.033,
    14934.033,
    14940.033,
    14946.033,
    14952.033,
    14958.033,
    14964.033,
    14968.033,
    14972.033,
    14978.033,
    14984.033,
    14988.033,
    14994.033,
    15000.033,
    15004.033,
    15009.1,
    15016.033,
    15020.033,
    15026.033,
    15032.033,
    15036.033,
    15041.1,
    15048.033,
    15052.033,
    15058.033,
    15064.033,
    15070.033,
    15076.033,
    15081.1,
    15088.033,
    15092.033,
    15096.033,
    15102.033,
    15108.033,
    15114.9,
    15118.533,
    15122.833,
    15129.633,
    15133.633,
    15140.633,
    15144.033,
    15150.033,
    15156.033,
    15162.033,
    15168.033,
    15172.033,
    15177.933,
    15182.467,
    15188.033,
    15194.033,
    15200.033,
    15206.967,
    15212.033,
    15216.033,
    15222.033,
    15228.033,
    15232.033,
    15238.033,
    15244.367,
    15249.2,
    15256.033,
    15260.033,
    15266.033,
    15272.033,
    15278.033,
    15284.033,
    15290.167,
    15296.033,
    15302.967,
    15309.2,
    15316.033,
    15320.033,
    15326.367,
    15332.033,
    15338.033,
    15344.033,
    15348.033,
    15354.033,
    15360.033,
    15364.033,
    15368.033,
    15372.033,
    15377.167,
    15384.033,
    15388.033,
    15392.033,
    15396.033,
    15402.767,
    15408.033,
    15414.033,
    15420.033,
    15426.033,
    15432.033,
    15436.033,
    15440.033,
    15444.033,
    15450.033,
    15456.033,
    15460.033,
    15466.033,
    15472.033,
    15478.033,
    15484.033,
    15490.033,
    15496.033,
    15500.033,
    15506.033,
    15512.033,
    15517.1,
    15524.033,
    15530.033,
    15536.033,
    15540.033,
    15544.033,
    15550.033,
    15556.033,
    15562.033,
    15568.033,
    15571.667,
    15578.033,
    15584.033,
    15588.033,
    15594.033,
    15600.033,
    15606.167,
    15612.033,
    15616.033,
    15620.033,
    15624.033,
    15630.033,
    15636.033,
    15640.033,
    15645.333,
    15651.6,
    15656.033,
    15661.667,
    15668.667,
    15672.033,
    15676.033,
    15682.033,
    15688.033,
    15692.033,
    15698.033,
    15704.033,
    15710.033,
    15716.033,
    15720.033,
    15726.033,
    15732.033,
    15737.2,
    15744.033,
    15748.033,
    15752.033,
    15758.033,
    15764.033,
    15770.433,
    15776.033,
    15782.033,
    15788.033,
    15792.033,
    15796.033,
    15800.033,
    15804.033,
    15810.967,
    15816.033,
    15822.033,
    15828.033,
    15834.033,
    15840.033,
    15845.7,
    15849.233,
    15856.033,
    15860.033,
    15864.033,
    15868.033,
    15874.033,
    15880.033,
    15884.033,
    15888.033,
    15894.033,
    15900.033,
    15904.033,
    15908.033,
    15914.033,
    15920.033,
    15926.033
  ],
  "intervals": {
    "min": 3.0,
    "avg": 5.455989379924632,
    "max": 7.0
  },
  "output": "/home/bedhedd/Documents/development_projects/bedhedd_projects/dougpt/dougpt/00-supporting-files/data/Doug_and_Twitch_Chat_TAKE_OVER_EUROPE-VpmmuHlLPM0_keyframes_report.json"
}

The data is great, but comparing the key frames to verify that chat worked is kinda a pain in the butt. I had chatgpt give me some marimo code to compare the current and next frames.

One thing I observed is that when Doug moves his arm the key frames differ. I asked chatgpt, and it suggested I crop the video down to chat to re-encode the key frames.

I had chatgpt give me a gui so that I can visually draw the box instead of typing and guessing.
The code is found in the resizing to a area section of the notebook

02-development/chat-extraction/chat-extraction.py

Then I picked an area Doug wouldn’t potentially block with his arm if he stretched.

After getting the coordinates, I had chatgpt write some more code to apply the crop and use a ffmpeg preset from my ffcompress repo. Once ffmpeg finished applying the crop, I loaded it to using the inspection tool

I wanted to view more, so I opened the cached images on my desktop

[bedHedd]

structured outputs

After seeing @progressEdd’s marimo demo, I was convinced to use structured outputs. I had chatgpt modify the example code and give me one for the frame extractions

class EmoteBox(BaseModel):
    x: int = Field(..., description="Top-left X coordinate of the emote bounding box (pixels, image coordinate space).")
    y: int = Field(..., description="Top-left Y coordinate of the emote bounding box (pixels, image coordinate space).")
    w: int = Field(..., description="Width of the emote bounding box (pixels).")
    h: int = Field(..., description="Height of the emote bounding box (pixels).")
    description: str = Field(..., description="Short human-readable description of the emote (e.g., 'PepeHands', 'Luigi'). use a full description rather than a the emoji shorthand")


class ChatMessage(BaseModel):
    user_name: str = Field(..., description="Username as shown in the chat UI for this line (no inferred/extra metadata).")
    message: str = Field(..., description="Visible message text and emoji (described as 1 word) for this chat line. If emotes and emojis are in the message, include them inline and use a structure of `:emote_described:` `:another_emote_described:` ...")
    emotes: List[EmoteBox] = Field(
        default_factory=list,
        description="List of emotes found on this chat line, each with an approximate bounding box in pixels. ",
    )


class ImageOnlyExtraction(BaseModel):
    chat_messages: List[ChatMessage] = Field(
        default_factory=list,
        description="All visible chat lines in a single image, in top-to-bottom reading order.",
    )


class BatchExtraction(BaseModel):
    images: List[ImageOnlyExtraction] = Field(
        ...,
        description="Exactly 8 per-image extractions, in the same order as the 8 input images.",
    )

I also passed the directory into a chat completion to see how lmstudio-community/Qwen3-VL-8B-Instruct-GGUF performed. I also switched to LM Studio for my model endpoint, as llama.cpp kept throwing out of memory errors when loading vision models. This happened even with smaller models

previous_compact_resp = {
  "images": [
    {
      "chat_messages": [
        {
          "user_name": "ToaSTy_T0aST",
          "message": "nvm its on a cooldown",
          "emotes": [
            {
              "x": 240,
              "y": 20,
              "w": 30,
              "h": 30,
              "description": "green frog"
            }
          ]
        },
        {
          "user_name": "frickelodeon",
          "message": "@iamkaalhode believe in the doubt",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "red heart"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "speaker"
            }
          ]
        },
        {
          "user_name": "sour_appel",
          "message": "So when Doug won't open the oxygen gates, we have to rely on the fart barons",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "red heart"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "whamer100",
          "message": "yeah why is the prediction open for so long",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "I have betted too much to lose D.",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Gavyn_J",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "@whamer100 Because RIGGED",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    },
    {
      "chat_messages": [
        {
          "user_name": "frickelodeon",
          "message": "@iamkaalhode believe in the doubt",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "red heart"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "speaker"
            }
          ]
        },
        {
          "user_name": "sour_appel",
          "message": "So when Doug won't open the oxygen gates, we have to rely on the fart barons",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "red heart"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "whamer100",
          "message": "yeah why is the prediction open for so long",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "I have betted too much to lose D.",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Gavyn_J",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "@whamer100 Because RIGGED",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    },
    {
      "chat_messages": [
        {
          "user_name": "sour_appel",
          "message": "So when Doug won't open the oxygen gates, we have to rely on the fart barons",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "red heart"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "whamer100",
          "message": "yeah why is the prediction open for so long",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "I have betted too much to lose D.",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Gavyn_J",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "@whamer100 Because RIGGED",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    },
    {
      "chat_messages": [
        {
          "user_name": "whamer100",
          "message": "yeah why is the prediction open for so long",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "I have betted too much to lose D.",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Gavyn_J",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "@whamer100 Because RIGGED",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    },
    {
      "chat_messages": [
        {
          "user_name": "iamkaalhode",
          "message": "I have betted too much to lose D.",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Gavyn_J",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "@whamer100 Because RIGGED",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    },
    {
      "chat_messages": [
        {
          "user_name": "Gavyn_J",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "@whamer100 Because RIGGED",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    },
    {
      "chat_messages": [
        {
          "user_name": "TheHolyPangolin",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "@whamer100 Because RIGGED",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "orange owl"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "question mark"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    },
    {
      "chat_messages": [
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": [
            {
              "x": 10,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 30,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 50,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 70,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            },
            {
              "x": 90,
              "y": 10,
              "w": 20,
              "h": 20,
              "description": "green mario"
            }
          ]
        }
      ]
    }
  ]
}

The chat messages were extracted, but the outputs had issues substituting emotes. I decided to try a larger qwen model (lmstudio-community/Qwen3-VL-30B-A3B-Instruct-GGUF which seemed to do better

{
  "images": [
    {
      "chat_messages": [
        {
          "user_name": "ToaSTy_T0aST",
          "message": "nvm its on a cooldown :frog:",
          "emotes": [
            {
              "x": 51,
              "y": 46,
              "w": 23,
              "h": 23,
              "description": "frog"
            }
          ]
        },
        {
          "user_name": "frickelodeon",
          "message": "@iamkaalhode believe in the doubt",
          "emotes": []
        },
        {
          "user_name": "sour_apple",
          "message": "So when Doug won't open the oxygen gates, we have to rely on the fart barons",
          "emotes": []
        },
        {
          "user_name": "whamer100",
          "message": "yeah why is the prediction open for so long",
          "emotes": []
        },
        {
          "user_name": "iamkaalhode",
          "message": "I have betted too much to lose D.",
          "emotes": []
        },
        {
          "user_name": "Gavyn_J",
          "message": "",
          "emotes": [
            {
              "x": 58,
              "y": 297,
              "w": 30,
              "h": 30,
              "description": "man with glasses"
            }
          ]
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "@whamer100 Because RIGGED",
          "emotes": []
        },
        {
          "user_name": "Komodo66619",
          "message": ":mario: :mario: :mario: :mario:",
          "emotes": [
            {
              "x": 25,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 40,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 56,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 72,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            }
          ]
        },
        {
          "user_name": "comicallyidiotic",
          "message": "I have 80 cakes on the table",
          "emotes": []
        },
        {
          "user_name": "Jennie027",
          "message": "the strat was to wait until doug goes live or doesn't because the prediction goes past the time",
          "emotes": []
        },
        {
          "user_name": "projectdolphin_",
          "message": "",
          "emotes": [
            {
              "x": 58,
              "y": 596,
              "w": 30,
              "h": 30,
              "description": "man with glasses"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "crismanti20",
          "message": "EUROPE :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 567,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "He changed title",
          "emotes": []
        },
        {
          "user_name": "pegglequeen1",
          "message": "why is the prediction this long? people will just wait to doubt",
          "emotes": []
        },
        {
          "user_name": "meowsticks24",
          "message": "DOUG",
          "emotes": []
        },
        {
          "user_name": "iamkaalhode",
          "message": "NOOOOOOO",
          "emotes": []
        },
        {
          "user_name": "Jennie027",
          "message": "FUCK YEAH",
          "emotes": []
        },
        {
          "user_name": "nora_bot",
          "message": "",
          "emotes": [
            {
              "x": 58,
              "y": 196,
              "w": 30,
              "h": 30,
              "description": "man with glasses"
            }
          ]
        },
        {
          "user_name": "whamer100",
          "message": "fuck",
          "emotes": []
        },
        {
          "user_name": "mmust_m",
          "message": "YESSSSSSSSS",
          "emotes": []
        },
        {
          "user_name": "DatKirby_",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "bananaz602",
          "message": "LES GO :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 318,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": ":mario: :mario: :mario:",
          "emotes": [
            {
              "x": 25,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 40,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 56,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            }
          ]
        },
        {
          "user_name": "Evening_Owl",
          "message": "YES :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "DOUG :mario: :mario:",
          "emotes": [
            {
              "x": 135,
              "y": 420,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 151,
              "y": 420,
              "w": 14,
              "h": 14,
              "description": "mario"
            }
          ]
        },
        {
          "user_name": "Sky0nn",
          "message": "YEEAAH BELIEVERS :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 452,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Jennie027",
          "message": "I AM THE GREATEST :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 486,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "KimbolineNorway",
          "message": "Doug! :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 520,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "BradS18",
          "message": "LETS GOOOO :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 552,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Evening_Owl",
          "message": "BELIEVERS WIN :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 586,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "projectdolphin_",
          "message": "FUCK :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 620,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "mrgundhamtanaka",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "NOOOOOOOOO",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "whamer100",
          "message": "DOUG MY MONEY :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "moomin_tophat",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "tacoespionage",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "meowsticks24",
          "message": "GEOGRAPHY :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "crismanti20",
          "message": "Yo :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Jennie027",
          "message": "I AM SO GOOD AT BETTING :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 486,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "LankLTE",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "mmust_m",
          "message": "SUCK IT NON BELIEVERS :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "Yeeesssssssssss",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "mrgundhamtanaka",
          "message": "RIGGged :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "_d_dip",
          "message": "NOOOO :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "dinklebergle",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "comicallyidiotic",
          "message": "NOOOOO :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "BradS18",
          "message": "DOUGGGG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Duarpeto",
          "message": "DOOOOOOG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "pegglequeen1",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "nora_bot",
          "message": "IT WAS ALL WORTH IT :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": ":frog: :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 51,
              "y": 284,
              "w": 23,
              "h": 23,
              "description": "frog"
            },
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Evening_Owl",
          "message": "STOP DONATING :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "SO HE CAN LEAVE",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "TheHolyPangolin",
          "message": "DOUG DOUG :mario: :mario:",
          "emotes": [
            {
              "x": 135,
              "y": 420,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 151,
              "y": 420,
              "w": 14,
              "h": 14,
              "description": "mario"
            }
          ]
        },
        {
          "user_name": "Komodo66619",
          "message": ":mario: :mario: :mario:",
          "emotes": [
            {
              "x": 25,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 40,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            },
            {
              "x": 56,
              "y": 387,
              "w": 14,
              "h": 14,
              "description": "mario"
            }
          ]
        },
        {
          "user_name": "gforce_gaming_official",
          "message": "FINALLY :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "mmust_m",
          "message": "SHUN THE NON-BELIEVERS :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "toughrobotics",
          "message": "Don't stream today :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "FlyignPig",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "iamkaalhode",
          "message": "I lost 16k :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "tacoespionage",
          "message": "PREDICTION RIGGED :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Jennie027",
          "message": "I AM THE GREATEST PREDICTOR IN THE HISTORY OF PREDICTIONS :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 486,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "tacoespionage",
          "message": "REFUND :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "BradS18",
          "message": "GIVE ME MY MONEY :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "heidi_susanna",
          "message": "DOUG :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "Sky0nn",
          "message": "But now since prediction was :face_with_tears_of_joy:",
          "emotes": [
            {
              "x": 135,
              "y": 284,
              "w": 14,
              "h": 14,
              "description": "face with tears of joy"
            }
          ]
        },
        {
          "user_name": "dinkiebergle",
          "message": "",
          "emotes": []
        },
        {
          "user_name": "tacoespionage",
          "message": "",
          "emotes": []
        }
      ]
    }
  ]
}

This works, but the model takes a long time to perform the completion and will frequently time out. I suspect that I need to separate the emote extraction from the text recognition

[bedHedd]

@bedHedd

initial attempt at chunking messages

I wanted to use opencv, but I ran into a bunch of build issues. When I finally got it working, it did not support mkv or av1 videos. Instead I decided to go with ffmpeg

My firstidea was to find the chat message and use the distance from the top frame to get the next screenshot. The only challenge was that I didn’t know how to reliably get the message and text position.

For the sake of progress, I decided to move forward and avoid getting to hung up.

revisiting keyframe chunking (again)

After hitting a roadblock with the vision completions taking a long time, I decided to revisit the key frame chunking. It performed well reducing the total frames to 15k 15399. but when I was passing the frames to the language model, I noticed there was too much overlap.

I had chatgpt rewrite the functions and passed the examples. It suggested 2 paths

@chatgpt

Keyframes are the wrong “unit” for this. Make the selection depend on how far the chat actually moved, and (optionally) where a specific line is.

You already have: first frame → last frame, with 2 intermediates. The “smart” thing is to pick the best frame in that window based on scroll distance / anchor position, and drop the rest.

Option A: Keep only when cumulative scroll ≥ K lines (no OCR)

1. Estimate dy (vertical scroll) between frames cheaply.
2. Accumulate until you’ve scrolled ~K * line_height, then keep that frame.

from __future__ import annotations

from PIL import Image, ImageOps
import numpy as np

def _row_energy(img: Image.Image) -> np.ndarray:
    g = ImageOps.grayscale(img)
    a = np.asarray(g, dtype=np.float32) / 255.0
    gx = np.zeros_like(a); gy = np.zeros_like(a)
    gx[:, 1:-1] = a[:, 2:] - a[:, :-2]
    gy[1:-1, :] = a[2:, :] - a[:-2, :]
    mag = np.hypot(gx, gy)
    row = mag.mean(axis=1)
    row = (row - row.mean()) / (row.std() + 1e-6)
    return row

def estimate_dy(a: Image.Image, b: Image.Image, max_shift: int = 200) -> tuple[int, float]:
    ra = _row_energy(a); rb = _row_energy(b)
    best_s, best_corr = 0, -1e9
    for s in range(-max_shift, max_shift + 1):
        if s >= 0:
            x = ra[s:]; y = rb[:len(x)]
        else:
            x = ra[:len(ra) + s]; y = rb[-s:]
        if len(x) < 120:
            continue
        corr = float((x * y).mean())
        if corr > best_corr:
            best_corr, best_s = corr, s
    return best_s, best_corr

def estimate_line_height(img: Image.Image) -> int:
    r = _row_energy(img)
    thr = float(np.quantile(r, 0.85))
    peaks = [i for i in range(2, len(r)-2) if r[i] > thr and r[i] > r[i-1] and r[i] > r[i+1]]
    if len(peaks) < 6:
        return 36  # fallback for Twitch chat overlays
    d = np.diff(peaks)
    d = d[(d >= 18) & (d <= 70)]
    return int(np.median(d)) if len(d) else 36

def keep_by_scroll(frames: list[Image.Image], *, lines_per_keep: int = 6, min_corr: float = 0.75) -> list[int]:
    """
    Returns indices of frames to keep.
    Keeps a frame when cumulative scroll since last kept >= lines_per_keep * line_height.
    """
    if not frames:
        return []
    lh = estimate_line_height(frames[0])
    keep = [0]
    last = 0
    accum = 0

    for i in range(1, len(frames)):
        dy, corr = estimate_dy(frames[last], frames[i])
        if corr < min_corr:
            continue
        accum += abs(dy)
        if accum >= lines_per_keep * lh:
            keep.append(i)
            last = i
            accum = 0
    return keep

How to use it with your 4-frame window: load those 4 crops into frames and run keep_by_scroll(frames, lines_per_keep=4..8). With your sample, first→last is ~4–5 lines of scroll, so lines_per_keep=4 will likely keep the last and drop the two in-between.

---

Option B: Track a specific message and pick the “best” frame (still no OCR)

If your goal is “capture when line X reaches the top marker”, do template tracking:

• On the first frame where that message is clearly visible, crop a small template around it (username+text/emote area).
• For each candidate frame in the window, find the template’s best y.
• Pick the frame where y is closest to your target (e.g., y_target=40 px from top of crop).

def match_template_y(frame: Image.Image, templ: Image.Image, *, y0: int = 0, y1: int | None = None, step: int = 1) -> tuple[int, float]:
    """
    Slide templ vertically over frame (x fixed by templ crop) and return (best_y, score).
    Uses normalized correlation on grayscale.
    """
    F = np.asarray(ImageOps.grayscale(frame), dtype=np.float32)
    T = np.asarray(ImageOps.grayscale(templ), dtype=np.float32)
    th, tw = T.shape
    if y1 is None:
        y1 = F.shape[0] - th
    y1 = min(y1, F.shape[0] - th)
    y0 = max(0, y0)

    Tm = T - T.mean()
    Td = np.sqrt((Tm * Tm).sum()) + 1e-6

    best_y, best = y0, -1e9
    for y in range(y0, y1 + 1, step):
        P = F[y:y+th, :tw]
        Pm = P - P.mean()
        score = float((Pm * Tm).sum() / ((np.sqrt((Pm*Pm).sum()) + 1e-6) * Td))
        if score > best:
            best, best_y = score, y
    return best_y, best

def pick_frame_near_y(frames: list[Image.Image], templ: Image.Image, *, y_target: int, search_pad: int = 120) -> int:
    """
    Returns the index of the frame where the template's y is closest to y_target.
    Uses coarse dy prediction to narrow search.
    """
    # start with full search on first comparison
    y_prev, _ = match_template_y(frames[0], templ)
    best_i, best_err = 0, abs(y_prev - y_target)

    for i in range(1, len(frames)):
        # predict y window using dy estimate (fast)
        dy, corr = estimate_dy(frames[i-1], frames[i])
        y_pred = y_prev - dy  # dy sign depends; this is usually correct for upward scroll
        y0 = int(max(0, y_pred - search_pad))
        y1 = int(min(frames[i].size[1] - templ.size[1], y_pred + search_pad))

        y, score = match_template_y(frames[i], templ, y0=y0, y1=y1, step=1)
        y_prev = y

        err = abs(y - y_target)
        if err < best_err and score > 0.6:
            best_err, best_i = err, i

    return best_i

Workflow for your “first/last + 2 intermediates” case

1. Choose the message you care about (the one you previously said you want to reach the top).
2. Crop a template once (same x-range every time; include emote if it’s distinctive).
3. Run pick_frame_near_y(frames_window, templ, y_target=some_px) to select which of the 4 frames is the “best” capture frame.

---

What to set for your overlay

• line_height: your sample crop estimates around ~35–40 px.

• Good defaults:

  • lines_per_keep = 5–7 (cuts overlap hard while still safe)
  • y_target = 40–80 px (near top but not clipped)

If you want, paste your crop coordinates (x, y, w, h) and which line you’re tracking (top boundary or yellow marker y), and I’ll give exact y_target, lines_per_keep, and the template x-range that avoids badges/icons.

I asked it to pick the option that would work at scale

@chatgpt

Option A (scroll-distance gating) is the one that scales. It’s single-pass, OCR-free, and its cost per frame is small and predictable. Option B (template/anchor tracking) only makes sense when you have a specific line you want to “hit,” not for bulk downsampling.

Recommended scalable pipeline (2-stage)

Stage 1: Keep frames when chat has actually advanced (Option A)

Process your 20k frames in order:

1. Compute line_height once from an early frame.

2. For each new frame, estimate vertical shift dy from the last kept frame (row-energy correlation).

3. Accumulate abs(dy).

4. Keep a frame when either:

   • accum_dy >= K * line_height (K = 6–10 is typical), or
   • bottom band changed a lot (new messages/emote-only lines arrived).

This typically cuts 20k down to a few thousand or less, depending on scroll speed and K.

Knobs

• K (lines per capture): higher = fewer frames.
• bottom_change_thr: prevents missing emote-only / short lines even if dy is small.

Stage 2 (optional): Deduplicate “near-identical” kept frames

After Stage 1, you can run a cheap hash (dHash/pHash) on the kept frames and drop near-duplicates. This is often another 10–30% reduction.

Practical defaults

• lines_per_keep (K): start at 8
• max_shift: 200 px (enough for several lines + jitter)
• min_corr: 0.75
• bottom_band_h: 120–160 px
• bottom_change_thr: 0.04–0.06 (depends on compression)

Speed improvements (important at 20k+)

• Compute the row-energy on a downscaled version of the crop:

  • resize width to ~200–300 px (keep height), or even downscale both by 2

• Use only the chat text columns (exclude left badge/icon column and any right border) before computing row-energy; it stabilizes correlation.

• Process in chunks, but keep the last kept frame from the previous chunk to seed the next.

When to use Option B

Only after Stage 1, for targeted tasks like: “pick the frame where message X hits y=40px.” You’d run template tracking on the small window around that event, not across 20k frames.

If you want, paste:

• crop box (x0, y0, x1, y1) (or your current crop image width/height),
• and your current average candidate rate (e.g., “20k frames from keyframes over 4h”),
  and I’ll give you a concrete K that should land you near a target count (e.g., 2k–5k).

I then told it to give a function to process it and tested it with a 30 second chunk.

@chatgpt

Your report shows why it kept everything:

• kept_count: 100799 equals the number of decoded frames (60 fps decode, no sampling).
• bottom_change is ~0.08–0.14 almost every time, but you used bottom_change_thr: 0.05, so the “bottom changed” rule fires constantly and forces a keep.
• That’s why you see adjacent keeps (424 → 429 → 457) even though overlap is high.

Fix 1: Raise/disable bottom-change forcing

Given your observed bottom_change distribution, set bottom_change_thr higher than typical noise.

Start here:

kept = reduce_chat_frames_by_scroll(
    ai_invasion_1_cropped,
    out_dir,
    lines_per_keep=8,
    bottom_change_thr=0.14,   # was 0.05
)

If you don’t need bottom-change at all, effectively disable it:

kept = reduce_chat_frames_by_scroll(
    ai_invasion_1_cropped,
    out_dir,
    lines_per_keep=8,
    bottom_change_thr=1.0,    # never triggers
)

Fix 2: Accumulate scroll frame-to-frame (not vs last kept)

If you ran the earlier version, it was comparing the current frame to the last kept frame and then adding abs(dy) repeatedly, which is not the correct “distance traveled so far.” Accumulate dy_step between consecutive frames.

Use the corrected function version (the one with dy_step_px and accum_scroll updated from prev_frame -> current_frame). If you’re not sure which version you executed: your report fields include dy_px (older version), not dy_step_px (corrected version).

Fix 3: Don’t decode all 60-fps frames unless you want to

If your keyframe method gives ~15k candidates and you want fewer, you should either:

A) Decode only keyframes with ffmpeg (best drop-in)

Modify the ffmpeg reader to add -skip_frame nokey:

def _iter_gray_frames_ffmpeg(video: Path, w: int, h: int, *, sample_fps=None, keyframes_only: bool = False):
    vf = "format=gray"
    if sample_fps is not None:
        vf = f"fps={sample_fps},{vf}"

    cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-vsync", "0"]
    if keyframes_only:
        cmd += ["-skip_frame", "nokey"]
    cmd += ["-i", str(video), "-an", "-vf", vf, "-f", "rawvideo", "-pix_fmt", "gray", "pipe:1"]
    ...

Then call:

kept = reduce_chat_frames_by_scroll(
    ai_invasion_1_cropped,
    out_dir,
    keyframes_only=True,   # add this param and thread it through
    lines_per_keep=8,
    bottom_change_thr=0.14,
)

This keeps the “candidate set” comparable to your keyframe extraction, then applies scroll-gating to remove overlap.

B) Sample FPS (simple)

If you want a cheap reduction before gating:

kept = reduce_chat_frames_by_scroll(
    ai_invasion_1_cropped,
    out_dir,
    sample_fps=3.0,
    lines_per_keep=8,
    bottom_change_thr=0.14,
)

Quick sanity check after changes

After a short run, kept_count should be much smaller than decoded frames, and you should stop seeing near-adjacent indices being kept unless chat is genuinely static.

If you paste the first ~30 bottom_change values from the report, I can give a threshold that cleanly separates “noise” from “new message arrival” for your encode.

Some cycles back and forth this was the version that worked

@chatgpt

Yes.

Reduce further (3 knobs)

1. Increase lines_per_keep (e.g. 12–18)
2. Set sample_fps (e.g. 2–4) so you don’t decode 60 fps
3. Add a minimum time between kept frames (min_keep_interval_sec) to prevent rapid keeps in bursts

Final version that saves color PNGs

This decodes RGB, computes alignment on luma (grayscale derived from RGB), and saves color frames.

from __future__ import annotations

from dataclasses import dataclass, asdict
from pathlib import Path
import json
import subprocess
from typing import Iterable, Optional

import numpy as np
from PIL import Image


@dataclass(frozen=True)
class KeptFrame:
    index: int
    t_sec: float
    dy_step_px: int
    corr: float
    bottom_change: float
    saved_path: Optional[str]


def _ffprobe_video_info(video: Path) -> tuple[int, int, float]:
    cmd = [
        "ffprobe", "-v", "error",
        "-select_streams", "v:0",
        "-show_entries", "stream=width,height,avg_frame_rate,r_frame_rate",
        "-of", "json",
        str(video),
    ]
    out = subprocess.check_output(cmd)
    info = json.loads(out.decode("utf-8"))
    s = info["streams"][0]
    w, h = int(s["width"]), int(s["height"])

    def parse_rate(x: str) -> float:
        if not x or x == "0/0":
            return 0.0
        n, d = x.split("/")
        n = float(n)
        d = float(d) if float(d) != 0 else 1.0
        return n / d

    fps = parse_rate(s.get("avg_frame_rate", "")) or parse_rate(s.get("r_frame_rate", "")) or 1.0
    return w, h, fps


def _iter_rgb_frames_ffmpeg(
    video: Path,
    w: int,
    h: int,
    *,
    sample_fps: Optional[float] = None,
    start_sec: Optional[float] = None,
    duration_sec: Optional[float] = None,
    keyframes_only: bool = False,
) -> Iterable[np.ndarray]:
    cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-vsync", "0"]

    if start_sec is not None:
        cmd += ["-ss", str(float(start_sec))]
    if duration_sec is not None:
        cmd += ["-t", str(float(duration_sec))]
    if keyframes_only:
        cmd += ["-skip_frame", "nokey"]

    cmd += ["-i", str(video), "-an"]

    if sample_fps is not None:
        cmd += ["-vf", f"fps={sample_fps}"]

    cmd += ["-f", "rawvideo", "-pix_fmt", "rgb24", "pipe:1"]

    frame_size = w * h * 3
    with subprocess.Popen(cmd, stdout=subprocess.PIPE) as proc:
        assert proc.stdout is not None
        while True:
            buf = proc.stdout.read(frame_size)
            if not buf or len(buf) < frame_size:
                break
            rgb = np.frombuffer(buf, dtype=np.uint8).reshape((h, w, 3))
            yield rgb
        proc.wait()


def _luma_u8(rgb: np.ndarray) -> np.ndarray:
    # ITU-R BT.601 luma approximation
    r = rgb[..., 0].astype(np.float32)
    g = rgb[..., 1].astype(np.float32)
    b = rgb[..., 2].astype(np.float32)
    y = 0.299 * r + 0.587 * g + 0.114 * b
    return np.clip(y, 0, 255).astype(np.uint8)


def _row_energy(gray_u8: np.ndarray, downsample: int = 2) -> np.ndarray:
    a = gray_u8[::downsample, ::downsample].astype(np.float32) / 255.0
    dx = np.abs(a[:, 1:] - a[:, :-1]).mean(axis=1)
    dy = np.abs(a[1:, :] - a[:-1, :]).mean(axis=1)
    dy = np.concatenate([dy, dy[-1:]])
    r = dx + dy
    return (r - r.mean()) / (r.std() + 1e-6)


def _estimate_dy(row_a: np.ndarray, row_b: np.ndarray, max_shift: int) -> tuple[int, float]:
    best_s, best_corr = 0, -1e9
    n = len(row_a)
    for s in range(-max_shift, max_shift + 1):
        if s >= 0:
            x = row_a[s:]
            y = row_b[: len(x)]
        else:
            x = row_a[: n + s]
            y = row_b[-s:]
        if len(x) < 80:
            continue
        corr = float((x * y).mean())
        if corr > best_corr:
            best_corr, best_s = corr, s
    return best_s, best_corr


def _estimate_line_height(row: np.ndarray) -> int:
    thr = float(np.quantile(row, 0.85))
    peaks = [i for i in range(2, len(row) - 2) if row[i] > thr and row[i] > row[i - 1] and row[i] > row[i + 1]]
    if len(peaks) < 6:
        return 18  # downsampled fallback (~36px if downsample=2)
    d = np.diff(peaks)
    d = d[(d >= 9) & (d <= 35)]
    return int(np.median(d)) if len(d) else 18


def _bottom_change(gray_a: np.ndarray, gray_b: np.ndarray, band_h: int, downsample: int = 2) -> float:
    a = gray_a[::downsample, ::downsample].astype(np.float32) / 255.0
    b = gray_b[::downsample, ::downsample].astype(np.float32) / 255.0
    bh = max(1, band_h // downsample)
    return float(np.mean(np.abs(a[-bh:, :] - b[-bh:, :])))


def reduce_chat_frames_by_scroll_color(
    video_path: Path | str,
    out_dir: Path | str,
    *,
    start_sec: float | None = None,
    duration_sec: float | None = None,
    sample_fps: Optional[float] = None,
    keyframes_only: bool = False,
    lines_per_keep: int = 12,
    min_corr: float = 0.75,
    max_shift_px: int = 220,
    bottom_band_h_px: int = 160,
    bottom_change_thr: float = 0.16,
    min_keep_interval_sec: float = 0.8,     # additional reduction lever
    burst_override_mult: float = 2.0,       # allow keep sooner if scroll is huge
    downsample: int = 2,
    save_png: bool = True,
    write_report_json: bool = True,
    progress_every: int = 1000,
) -> list[KeptFrame]:
    video = Path(video_path)
    out = Path(out_dir)
    out.mkdir(parents=True, exist_ok=True)

    w, h, fps_native = _ffprobe_video_info(video)
    fps_used = float(sample_fps) if sample_fps is not None else float(fps_native)
    t0 = float(start_sec or 0.0)

    max_shift_ds = max(10, int(max_shift_px / downsample))

    kept: list[KeptFrame] = []
    prev_gray: Optional[np.ndarray] = None
    prev_row: Optional[np.ndarray] = None
    line_h_ds: Optional[int] = None

    accum_scroll = 0.0
    last_keep_t = -1e9

    for i, rgb in enumerate(
        _iter_rgb_frames_ffmpeg(
            video, w, h,
            sample_fps=sample_fps,
            start_sec=start_sec,
            duration_sec=duration_sec,
            keyframes_only=keyframes_only,
        )
    ):
        if progress_every and (i % progress_every == 0) and i > 0:
            print(f"[reduce] decoded={i} kept={len(kept)}")

        t_sec = t0 + (i / fps_used)
        gray = _luma_u8(rgb)
        row = _row_energy(gray, downsample=downsample)

        if prev_gray is None:
            line_h_ds = _estimate_line_height(row)
            save_path = None
            if save_png:
                save_path = str(out / f"frame_{i:06d}_t{t_sec:010.3f}.png")
                Image.fromarray(rgb, mode="RGB").save(save_path)
            kept.append(KeptFrame(i, float(t_sec), 0, 1.0, 0.0, save_path))
            prev_gray, prev_row = gray, row
            last_keep_t = t_sec
            continue

        assert prev_row is not None and prev_gray is not None and line_h_ds is not None

        dy_ds, corr = _estimate_dy(prev_row, row, max_shift=max_shift_ds)
        if corr >= min_corr:
            accum_scroll += abs(dy_ds)
        else:
            dy_ds = 0

        bc = _bottom_change(prev_gray, gray, band_h=bottom_band_h_px, downsample=downsample)

        # bottom-change only forces a keep when there's little/no scroll
        force_new = (abs(dy_ds) < 0.5 * line_h_ds) and (bc >= bottom_change_thr)

        scroll_trigger = accum_scroll >= (lines_per_keep * line_h_ds)
        burst_override = accum_scroll >= (burst_override_mult * lines_per_keep * line_h_ds)

        interval_ok = (t_sec - last_keep_t) >= min_keep_interval_sec
        keep_now = ((scroll_trigger and interval_ok) or burst_override) or (force_new and interval_ok)

        if keep_now:
            save_path = None
            if save_png:
                save_path = str(out / f"frame_{i:06d}_t{t_sec:010.3f}.png")
                Image.fromarray(rgb, mode="RGB").save(save_path)
            kept.append(KeptFrame(i, float(t_sec), int(dy_ds * downsample), float(corr), float(bc), save_path))
            accum_scroll = 0.0
            last_keep_t = t_sec

        prev_gray, prev_row = gray, row

    if write_report_json:
        report = {
            "video": str(video),
            "start_sec": start_sec,
            "duration_sec": duration_sec,
            "sample_fps": sample_fps,
            "keyframes_only": keyframes_only,
            "fps_used_for_timestamps": fps_used,
            "lines_per_keep": lines_per_keep,
            "min_keep_interval_sec": min_keep_interval_sec,
            "min_corr": min_corr,
            "max_shift_px": max_shift_px,
            "bottom_band_h_px": bottom_band_h_px,
            "bottom_change_thr": bottom_change_thr,
            "downsample": downsample,
            "kept_count": len(kept),
            "kept": [asdict(k) for k in kept],
        }
        (out / "report.json").write_text(json.dumps(report, indent=2), encoding="utf-8")

    return kept

Suggested test call (first 30s, fewer frames)

out_dir = data_dir / "chat_frames_test_30s_color"
kept = reduce_chat_frames_by_scroll_color(
    ai_invasion_1_cropped,
    out_dir,
    start_sec=0,
    duration_sec=30,
    sample_fps=3.0,          # big reduction vs 60fps decode
    lines_per_keep=14,       # increase to reduce further
    min_keep_interval_sec=1.0,
    bottom_change_thr=0.18,  # raise if it still keeps too often
    progress_every=500,
)

If it’s still keeping too many:

• bump lines_per_keep to 16–20
• bump min_keep_interval_sec to 1.2–1.5
• bump bottom_change_thr to 0.20–0.25 (your encode noise can be high)

Once it’s dialed in on 30s, remove duration_sec and run the full video.

After inspecting the 30s outputs, I ran it on the full stream. It took 20 minutes to run on the full 4hr stream, and the final frames count was 6543 (15399 → 6543), Impressive!

[progressEdd]

I tried vibecoding a ui. For 1 night of vibecoding, it did a pretty good job

@bedHedd

I based structured the repo off of @progressEdd’s template repo after seeing their structure for the fork

GitHub - progressEdd/project-template: template for projects

I’ve been writing my updates and investigations in the 00-dev-log folder

Also @bedHedd, I saw you were using my project template for dougpt. You should check out the vibe coding gsd branch I made.

progressEdd's llms, ai, and dev corner

Updates to my project template repo / vibecoding workflows After getting to experience claude/open code, vibecoding frameworks (spec kit, openspec, gsd), and git worktrees at work, I decided to update my project templates repo. https://github.com/progressEdd/project-template/tree/vibe-coding-gsd This is the current config that I used to vibecode …

[bedHedd]

@progressEdd

Also @bedHedd, I saw you were using my project template for dougpt. You should check out the vibe coding gsd branch I made.

I checked it out and it lines up with my workflow. I actually tried it out with my transcript processing pipeline. I’m 2 updates behind, so I’ll post it later

Also you should update your foam daily note template as the title doesn’t work if you want to create notes in the past. I updated it here

use foam date instead of current day

[bedHedd]

re-evaluating emote extraction

After getting the key frames refined, I decided to spend a bit more time to attempt to get the emote pixel positions. Unfortunately, no amount of prompt engineering could get the emote positions correct.

I decided to try emote extraction/substitution. I ran into some issues with the model kept confusing the sub icons as emotes.

So I tried using an example within the prompt

content = [{
    "type": "text",
    "text": (
        "\n"
        "MESSAGE TEXT\n"
        "- Keep the visible message text.\n"
        "- Replace each INLINE message emote with a one word description, include them inline and use a structure of `:emote_described:` `:another_emote_described:` ... in the message.\n"
        "- Do not insert :oneword: tokens for left-gutter UI icons.\n"
        "\n"
        # "TOKEN↔BOX CONSISTENCY (STRICT)\n"
        "- Order emotes left-to-right.\n"
        "\n"
        "If message text is not visible, use '(No message text visible)'.\n"
    ),}, 
           {"type": "text", "text": f"for example this image:"},
           {"type": "image_url", "image_url": {"url": image_file_to_data_url(example_frame)}},
           {"type": "text", "text": """the output is {
      "user_name": "ToaSTy_T0aST",
      "message": "nvm its on a cooldown :pepehands:",
      "emotes": [
        {
          "emote_name": "peepehands",
          "description": "pepe the frong crying"
        }
      ]
    },
    {
      "user_name": "frickelodeon",
      "message": "@iamkaalhode believe in the doubt",
      "emotes": []
    },
     {
      "user_name": "sour_appel",
      "message": "So when Doug won't open the oxygen gates, we have to rely on the fart barons",
      "emotes": []
    },
    """ },
           {"type": "text", "text": f"the emojis in front of the username are decorators, so avoid adding the icons before the username as emojis:"},
           
          ]

I reused the same image to see if it would make it better, it seemed to fix the issue

{
  "chat_messages": [
    {
      "user_name": "TheHolyPangolin",
      "message": "Because RIGGED",
      "emotes": []
    },
    {
      "user_name": "iamkaalhode",
      "message": "@whamer100 Because RIGGED",
      "emotes": []
    },
    {
      "user_name": "Komodo666619",
      "message": ":luigi: :luigi: :luigi: :luigi:",
      "emotes": [
        {
          "emote_name": "luigi",
          "description": "green mario character"
        }
      ]
    },
    {
      "user_name": "comicallyidiotic",
      "message": "I have 80 cakes on the table",
      "emotes": []
    },
    {
      "user_name": "Jennie027",
      "message": "the strat was to wait until doug goes live or doesn't because the prediction goes past the time",
      "emotes": [
        {
          "emote_name": "eggplant",
          "description": "purple eggplant"
        }
      ]
    },
    {
      "user_name": "projectdolphin_",
      "message": "",
      "emotes": []
    },
    {
      "user_name": "iamkaalhode",
      "message": "Oh no",
      "emotes": []
    },
    {
      "user_name": "crismanti20",
      "message": "EUROPE :mouth:",
      "emotes": [
        {
          "emote_name": "mouth",
          "description": "open mouth"
        }
      ]
    },
    {
      "user_name": "iamkaalhode",
      "message": "He changed title",
      "emotes": []
    },
    {
      "user_name": "pegglequeen1",
      "message": "why is the prediction this long? people will just",
      "emotes": []
    }
  ]
}

At this point I decided it wasn’t worth delaying my progress further, so I decided to scrap that idea and just focus on extracting the chat messages.

making a script to extract text from messages

Since I wanted to not get stuck trying to extract emote positions, I decided to move forward with making a script to extract the emotes and text.

I started by extracting a single image, then had chatgpt wrap up the code and logic into a function

This was the initial codeblocks I used

from pathlib import Path
from dotenv import load_dotenv

try:
    start = Path(__file__).resolve()
except NameError:
    start = Path.cwd()

supporting_files = next(p / "00-supporting-files" for p in start.parents if (p / "00-supporting-files").exists())

project_parent = next(p for p in (start.resolve(), *start.resolve().parents) if (p / "00-supporting-files").exists()).parent

video_dir = project_parent / "large-files"

data_dir = supporting_files / "data"

example_frame = supporting_files / "data" / "example.png"

import base64
from openai import OpenAI

def image_file_to_data_url(path: str) -> str:
    # Change mime if needed: image/jpeg, image/webp, etc.
    mime = "image/png"
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode("utf-8")
    return f"data:{mime};base64,{b64}"

client = OpenAI(
    base_url="http://localhost:1234/v0",  # or http://localhost:8000/v1
    api_key="unused",  # many local servers ignore this
)

client = OpenAI(
    base_url="http://localhost:1234/v1",
    api_key="lm-studio"
)

import json
import requests
from typing import Optional, Tuple, Type
from pydantic import BaseModel
from pydantic import BaseModel, Field

class EmoteBox(BaseModel):
    emote_name: str = Field(..., description="a shorthand name for the emote (e.g., 'PepeHands', 'POG', 'OMEGALUL', 'D:'   'CINEMA', 'dougdougPump').")
    description: str = Field(..., description="Short human-readable description of the emote use a full description rather than a the emoji shorthand")

class ChatMessage(BaseModel):
    user_name: str = Field(..., description="Username as shown in the chat UI for this line (no inferred/extra metadata).")
    message: str = Field(..., description="Visible message text and emoji (described as 1 word) for this chat line. If emotes and emojis are in the message, include them inline and use a structure of `:emote_described:` `:another_emote_described:` ...")
    emotes: List[EmoteBox] = Field(..., description="list of unique emotes, can be empty ignore images next to the name just focus on emotes/emojis after the name")

class ChatExtraction(BaseModel):
    chat_messages: List[ChatMessage] = Field(
        default_factory=list,
        description="All visible chat lines in a single image, in top-to-bottom reading order.",
    )

labeled_example = """
The extracted output from the reference is 
{{
  "user_name": "ToaSTy_T0aST",
  "message": "nvm its on a cooldown :pepehands:",
  "emotes": [
    {
      "emote_name": "peepehands",
      "description": "pepe the frong crying"
    }
  ]
},
{
  "user_name": "frickelodeon",
  "message": "@iamkaalhode believe in the doubt",
  "emotes": []
},
  {
  "user_name": "sour_appel",
  "message": "So when Doug won't open the oxygen gates, we have to rely on the fart barons",
  "emotes": []
}}
""".strip()  

print(labeled_example)

The extracted output from the reference is 
{{
  "user_name": "ToaSTy_T0aST",
  "message": "nvm its on a cooldown :pepehands:",
  "emotes": [
    {
      "emote_name": "peepehands",
      "description": "pepe the frong crying"
    }
  ]
},
{
  "user_name": "frickelodeon",
  "message": "@iamkaalhode believe in the doubt",
  "emotes": []
},
  {
  "user_name": "sour_appel",
  "message": "So when Doug won't open the oxygen gates, we have to rely on the fart barons",
  "emotes": []
}}

# ---------- pick first 8 files ----------
FRAMES_DIR = video_dir / "Doug_and_Twitch_Chat_TAKE_OVER_EUROPE-VpmmuHlLPM0__crop_no_audio__crop_0x14_350x460__s2" / "chat_frames"

exts = {".jpg", ".jpeg", ".png", ".webp"}
files = sorted([p for p in FRAMES_DIR.iterdir() if p.is_file() and p.suffix.lower() in exts])
# if len(files) < 8:
#     raise RuntimeError(f"Need at least 8 images in {FRAMES_DIR.resolve()}, found {len(files)}")

instructions = [{
    "type": "text",
    "text": (
        "You are a helpful assistant who is extracting twitch usernames and messages"
        "# MESSAGE TEXT EXTRACTION INSTRUCTIONS\n"
        "- Keep the visible message text.\n"
        "- Replace each INLINE message emote with a one word description, include them inline and use a structure of `:emote_described:` `:another_emote_described:` ... in the message.\n"
        "- Do not insert :oneword: tokens for left-gutter UI icons.\n"
        "\n"
        "- Order emotes left-to-right.\n"
        "\n"
        "If message text is not visible, use '(No message text visible)'.\n"),
        }, 
           {"type": "text", "text": f"# EXAMPLE IMAGE \n \nfor example this image:"},
           {"type": "image_url", "image_url": {"url": image_file_to_data_url(example_frame)}},
           {"type": "text", "text": labeled_example},
           {"type": "text", "text": f"the emojis in front of the username are decorators, so avoid adding the icons before the username as emojis:"},
          ]

instructions          

MODEL = "lmstudio-community/Qwen3-VL-30B-A3B-Instruct-GGUF"
MODEL = "qwen3-vl-30b-a3b-thinking"

# Preferred: structured parse (if your server supports it)
single_resp = client.beta.chat.completions.parse(
    model=MODEL,
    response_format=ChatExtraction,
    messages=messages,
    # temperature=0.2,
    max_tokens=-1,
)

single_resp_parsed = single_resp.choices[0].message.parsed
single_resp_validated = ChatExtraction.model_validate(single_resp_parsed)

print(single_resp_validated.model_dump_json(indent=2))

{
  "chat_messages": [
    {
      "user_name": "projectdolphin_",
      "message": "FUCK NOOOOOOOO",
      "emotes": []
    },
    {
      "user_name": "mrgundhamtanaka",
      "message": "DOUG MY MONEY",
      "emotes": []
    },
    {
      "user_name": "whamer100",
      "message": "DOUG MY MONEY",
      "emotes": []
    },
    {
      "user_name": "moomin_tophat",
      "message": "DOUG",
      "emotes": []
    },
    {
      "user_name": "tacoespionage",
      "message": "DOUG",
      "emotes": []
    },
    {
      "user_name": "meowsticks24",
      "message": "GEOGRAPHY",
      "emotes": []
    },
    {
      "user_name": "crismanti20",
      "message": "Yo",
      "emotes": [
        {
          "emote_name": "redface",
          "description": "red face emoji"
        }
      ]
    },
    {
      "user_name": "Jennie027",
      "message": "I AM SO GOOD AT BETTING",
      "emotes": []
    },
    {
      "user_name": "LankLTE",
      "message": "DOUG",
      "emotes": [
        {
          "emote_name": "snowflake",
          "description": "white snowflake"
        }
      ]
    },
    {
      "user_name": "mmust_m",
      "message": "SUCK IT NON",
      "emotes": []
    },
    {
      "user_name": "Komodo66619",
      "message": "Yeesssssssssssss",
      "emotes": [
        {
          "emote_name": "owl",
          "description": "owl emoji"
        }
      ]
    }
  ]
}

Once I got it all wrapped up into a function, I started running it across the dataset.

trailing characters infinite loop

while waiting for the script to process the full video for all chat messages, I observed qwen would get stuck on certain screenshots. For example if the text would wrap around qwen would use all the tokens to fill the wrap around

2026-02-09 18:35:16 [DEBUG]
 LlamaV4: server assigned slot 3 to task 2338097
2026-02-09 18:35:16  [INFO]
 [qwen3-vl-30b-a3b-thinking] Model generated tool calls:  []
2026-02-09 18:35:16  [INFO]
 [qwen3-vl-30b-a3b-thinking] Generated prediction:  {
  "id": "chatcmpl-udda7kzdb3jqt6tgbtvw",
  "object": "chat.completion",
  "created": 1770683700,
  "model": "qwen3-vl-30b-a3b-thinking",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"chat_messages\": [{\"user_name\": \"iuyiop\", \"message\": \"????!\", \"emotes\": []}, {\"user_name\": \"MareksPetLizard\", \"message\": \"F\", \"emotes\": []}, {\"user_name\": \"owen1389\", \"message\": \"fuckkkkkkkk(kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 1222,
    "completion_tokens": 900,
    "total_tokens": 2122
  },
  "stats": {},
  "system_fingerprint": "qwen3-vl-30b-a3b-thinking"
}
2026-02-09 18:35:16 [DEBUG]
 Received request: POST to /v1/chat/completions with body  {
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "You are a helpful assistant who is extracting twit... <Truncated in logs> ... is not visible, use '(No message text visible)'.\n"
        },
        {
          "type": "text",
          "text": "# EXAMPLE IMAGE \n \nfor example this image:"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwA... <Truncated in logs> ...WBFYEVgRWBFYEVgWMh8P8BTaYz4+WVfdgAAAAASUVORK5CYII="
          }
        },
        {
          "type": "text",
          "text": "The extracted output from the reference is \n{{\n  \"... <Truncated in logs> ...ave to rely on the fart barons\",\n  \"emotes\": []\n}}"
        },
        {
          "type": "text",
          "text": "Notice that the emojis in front of the username are decorators, so avoid adding the icons before the username as emojis:"
        },
        {
          "type": "text",
          "text": "Repeat the process above for the following image:"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwA... <Truncated in logs> ...yO6OTnjjzxozU2bttt+w/ugTMdeirQfAAAAABJRU5ErkJggg=="
          }
        }
      ]
    }
  ],
  "model": "qwen3-vl-30b-a3b-thinking",
  "max_tokens": 900,
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": {
        "$defs": {
          "ChatMessage": {
            "properties": {
              "user_name": {
                "description": "Username as shown in the chat UI for this line (no inferred/extra metadata).",
                "title": "User Name",
                "type": "string"
              },
              "message": {
                "description": "Visible message text and emoji (described as 1 wor... <Truncated in logs> ...:emote_described:` `:another_emote_described:` ...",
                "title": "Message",
                "type": "string"
              },
              "emotes": {
                "description": "list of unique emotes, can be empty ignore images next to the name just focus on emotes/emojis after the name",
                "items": {
                  "$ref": "#/$defs/EmoteBox"
                },
                "title": "Emotes",
                "type": "array"
              }
            },
            "required": [
              "user_name",
              "message",
              "emotes"
            ],
            "title": "ChatMessage",
            "type": "object",
            "additionalProperties": false
          },
          "EmoteBox": {
            "properties": {
              "emote_name": {
                "description": "a shorthand name for the emote (e.g., 'PepeHands', 'POG', 'OMEGALUL', 'D:'   'CINEMA', 'dougdougPump').",
                "title": "Emote Name",
                "type": "string"
              },
              "description": {
                "description": "Short human-readable description of the emote use a full description rather than a the emoji shorthand",
                "title": "Description",
                "type": "string"
              }
            },
            "required": [
              "emote_name",
              "description"
            ],
            "title": "EmoteBox",
            "type": "object",
            "additionalProperties": false
          }
        },
        "properties": {
          "chat_messages": {
            "description": "All visible chat lines in a single image, in top-to-bottom reading order.",
            "items": {
              "$ref": "#/$defs/ChatMessage"
            },
            "title": "Chat Messages",
            "type": "array"
          }
        },
        "title": "ChatExtraction",
        "type": "object",
        "additionalProperties": false,
        "required": [
          "chat_messages"
        ]
      },
      "name": "ChatExtraction",
      "strict": true
    }
  },
  "stream": false
}
2026-02-09 18:35:16  [INFO]
 [qwen3-vl-30b-a3b-thinking] Running chat completion on conversation with 1 messages.
2026-02-09 18:35:16 [DEBUG]
 LlamaV4::predict slot selection: session_id=<empty> server-selected (LCP/LRU)
2026-02-09 18:35:16 [DEBUG]
 slot get_availabl: id  3 | task -1 | selected slot by LCP similarity, sim_best = 1.000 (> 0.100 thold), f_keep = 0.576
2026-02-09 18:35:16 [DEBUG]
 slot launch_slot_: id  3 | task -1 | sampler chain: logits -> penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 2338998 | processing task, is_child = 0
slot update_slots: id  3 | task 2338998 | new prompt, n_ctx_slot = 8192, n_keep = 322, task.n_tokens = 1222
slot update_slots: id  3 | task 2338998 | cache reuse is not supported - ignoring n_cache_reuse = 256
slot update_slots: id  3 | task 2338998 | need to evaluate at least 1 token for each active slot (n_past = 1222, task.n_tokens() = 1222)
slot update_slots: id  3 | task 2338998 | n_past was set to 1221
slot update_slots: id  3 | task 2338998 | n_tokens = 1221, memory_seq_rm [370, end)
slot update_slots: id  3 | task 2338998 | prompt processing progress, n_tokens = 1222, batch.n_tokens = 1, progress = 1.000000
slot update_slots: id  3 | task 2338998 | prompt done, n_tokens = 1222, batch.n_tokens = 1
slot init_sampler: id  3 | task 2338998 | init sampler, took 0.05 ms, tokens: text = 320, total = 1222
2026-02-09 18:35:16  [INFO]
 [qwen3-vl-30b-a3b-thinking] Prompt processing progress: 0.0%
2026-02-09 18:35:16  [INFO]
 [qwen3-vl-30b-a3b-thinking] Prompt processing progress: 100.0%
2026-02-09 18:35:31 [DEBUG]
 slot print_timing: id  3 | task 2338998 | 
prompt eval time =      36.66 ms /     1 tokens (   36.66 ms per token,    27.28 tokens per second)
       eval time =   14858.57 ms /   900 tokens (   16.51 ms per token,    60.57 tokens per second)
      total time =   14895.23 ms /   901 tokens
slot      release: id  3 | task 2338998 | stop processing: n_tokens = 2121, truncated = 0
srv  update_slots: all slots are idle
2026-02-09 18:35:31 [DEBUG]
 LlamaV4: server assigned slot 3 to task 2338998
2026-02-09 18:35:31  [INFO]
 [qwen3-vl-30b-a3b-thinking] Model generated tool calls:  []
2026-02-09 18:35:31  [INFO]
 [qwen3-vl-30b-a3b-thinking] Generated prediction:  {
  "id": "chatcmpl-oxqv1lnop6m0ljz5heu3uz",
  "object": "chat.completion",
  "created": 1770683716,
  "model": "qwen3-vl-30b-a3b-thinking",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"chat_messages\": [{\"user_name\": \"iuyiop\", \"message\": \"????!\", \"emotes\": []}, {\"user_name\": \"MareksPetLizard\", \"message\": \"F\", \"emotes\": []}, {\"user_name\": \"owen1389\", \"message\": \"fuckkkkkkkk(kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 1222,
    "completion_tokens": 900,
    "total_tokens": 2122
  },
  "stats": {},
  "system_fingerprint": "qwen3-vl-30b-a3b-thinking"
}
2026-02-09 18:35:31 [DEBUG]
 Received request: POST to /v1/chat/completions with body  {
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "You are a helpful assistant who is extracting twit... <Truncated in logs> ... is not visible, use '(No message text visible)'.\n"
        },
        {
          "type": "text",
          "text": "# EXAMPLE IMAGE \n \nfor example this image:"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwA... <Truncated in logs> ...WBFYEVgRWBFYEVgWMh8P8BTaYz4+WVfdgAAAAASUVORK5CYII="
          }
        },
        {
          "type": "text",
          "text": "The extracted output from the reference is \n{{\n  \"... <Truncated in logs> ...ave to rely on the fart barons\",\n  \"emotes\": []\n}}"
        },
        {
          "type": "text",
          "text": "Notice that the emojis in front of the username are decorators, so avoid adding the icons before the username as emojis:"
        },
        {
          "type": "text",
          "text": "Repeat the process above for the following image:"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwA... <Truncated in logs> ...yO6OTnjjzxozU2bttt+w/ugTMdeirQfAAAAABJRU5ErkJggg=="
          }
        }
      ]
    }
  ],
  "model": "qwen3-vl-30b-a3b-thinking",
  "max_tokens": 900,
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "schema": {
        "$defs": {
          "ChatMessage": {
            "properties": {
              "user_name": {
                "description": "Username as shown in the chat UI for this line (no inferred/extra metadata).",
                "title": "User Name",
                "type": "string"
              },
              "message": {
                "description": "Visible message text and emoji (described as 1 wor... <Truncated in logs> ...:emote_described:` `:another_emote_described:` ...",
                "title": "Message",
                "type": "string"
              },
              "emotes": {
                "description": "list of unique emotes, can be empty ignore images next to the name just focus on emotes/emojis after the name",
                "items": {
                  "$ref": "#/$defs/EmoteBox"
                },
                "title": "Emotes",
                "type": "array"
              }
            },
            "required": [
              "user_name",
              "message",
              "emotes"
            ],
            "title": "ChatMessage",
            "type": "object",
            "additionalProperties": false
          },
          "EmoteBox": {
            "properties": {
              "emote_name": {
                "description": "a shorthand name for the emote (e.g., 'PepeHands', 'POG', 'OMEGALUL', 'D:'   'CINEMA', 'dougdougPump').",
                "title": "Emote Name",
                "type": "string"
              },
              "description": {
                "description": "Short human-readable description of the emote use a full description rather than a the emoji shorthand",
                "title": "Description",
                "type": "string"
              }
            },
            "required": [
              "emote_name",
              "description"
            ],
            "title": "EmoteBox",
            "type": "object",
            "additionalProperties": false
          }
        },
        "properties": {
          "chat_messages": {
            "description": "All visible chat lines in a single image, in top-to-bottom reading order.",
            "items": {
              "$ref": "#/$defs/ChatMessage"
            },
            "title": "Chat Messages",
            "type": "array"
          }
        },
        "title": "ChatExtraction",
        "type": "object",
        "additionalProperties": false,
        "required": [
          "chat_messages"
        ]
      },
      "name": "ChatExtraction",
      "strict": true
    }
  },
  "stream": false
}
...

To fix this I had chatgpt added the following instruction

“- Transcribe exactly what’s visible; do not continue a pattern.”

I also had chatgpt generate me some code so that if the max retries is >20, log the entry and store it in a separate file.

from __future__ import annotations

from functools import wraps
from pydantic import ValidationError

# robust import across openai sdk versions
try:
    from openai import LengthFinishReasonError
except Exception:
    try:
        from openai.lib._parsing._completions import LengthFinishReasonError
    except Exception:
        LengthFinishReasonError = None


def retry_until_valid(
    max_attempts: int | None = None,
    bump_on_length: bool = True,
    bump_add: int = 256,
    bump_mul: float = 1.5,
    failed_extractions_out=None,          # Path | str | None
    failed_keys=("i", "filename"),        # what to pull from kwargs
):
    retry_exc = (ValidationError,) + ((LengthFinishReasonError,) if LengthFinishReasonError else ())

    def deco(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            attempt = 1

            def log_failed():
                if not failed_extractions_out:
                    return
                rec = {k: kwargs.get(k) for k in failed_keys}
                # only write if we actually have the fields
                if any(v is not None for v in rec.values()):
                    append_jsonl(failed_extractions_out, rec, indent=2)

            while True:
                try:
                    return func(*args, **kwargs)

                except retry_exc as e:
                    # bump on length failures
                    if LengthFinishReasonError and isinstance(e, LengthFinishReasonError) and bump_on_length:
                        if "soft_completion_tokens" in kwargs:
                            prev = int(kwargs["soft_completion_tokens"])
                            hard = int(kwargs.get("hard_completion_cap", prev))
                            bumped = min(hard, max(prev + bump_add, int(prev * bump_mul)))

                            if bumped == prev:
                                log_failed()
                                raise

                            kwargs["soft_completion_tokens"] = bumped

                    attempt += 1
                    if max_attempts is not None and attempt > max_attempts:
                        log_failed()
                        raise

        return wrapper
    return deco


@retry_until_valid(max_attempts=20, failed_extractions_out=FAILED_EXTRACTIONS_OUT)
def extract_messages(
    *,
    i: int | None = None,            # just for logging
    filename: str | None = None,     # just for logging
    MODEL,
    ChatExtraction,
    messages,
    soft_completion_tokens: int = 900,
    hard_completion_cap: int = 1600,
    ctx_margin: int = 128,
):
    # your existing max_tokens logic (example)
    prompt_est = 0  # replace with your real estimate if you have one
    cur_ctx = 4096  # replace with model ctx if you track it
    available = max(1, cur_ctx - prompt_est - ctx_margin)

    max_tokens = max(1, min(available, soft_completion_tokens, hard_completion_cap))

    extraction = client.beta.chat.completions.parse(
        model=MODEL,
        response_format=ChatExtraction,
        messages=messages,
        max_tokens=max_tokens,
    )

    parsed = extraction.choices[0].message.parsed
    validated_extraction = ChatExtraction.model_validate(parsed)  # may raise ValidationError

    return extraction, validated_extraction

So far there hasn’t been any logs, so I’ll keep an eye on it.

[bedHedd]

side research

Since my my gpu was occupied with performing extractions, I took the time to inspect the outputs, and perform some research

continuity breaks

While inspecting the outputs, I realized that the frame skipping code I had chatgpt generate worked alright in maintaining continuity, but I still had a couple of examples where chat moved too fast and broke continuity.

I decided it wasn’t worth halting the process as ultimately, I just needed a general idea of what chat was saying in reaction to dougdoug. I can have a model generate messages.

looking at other model alternatives

I also observed that some messages were getting split into separate examples. This made me want to look for some potential alternatives, or change how I chunk it out.

While researching, I came across this thread Best AI vision model for extracting text and adding bounding boxes

A suggestion linked to PaddleOCR, which looked promising for accurate text recognition, but it required me to install the paddleocr library. I decided to look for an alternative, when I came across this article Top 7 Open Source OCR Models. The article included MiniCPM-V 4.5, which looked promising as it was a multimodal model that could take in video too. In the github repo, there’s a markdown file that explains how to make video requests MiniCPM-V4.5 vLLM Deployment Guide might be interesting to learn how to use vllm too.

[bedHedd]

having chatgpt identify the segments

With the hope of using MiniCPM, I decided to also have chatgpt chunk out the stream so that I can reduce the amount I need to process.
If you want to see the full script, I copied it here

00-dev-log/2026-02-09.md

@chatgpt

Chunked breakdown (timecode ranges + what happens)

1. 0:14–2:17 — Cold open + premise tease
   Intro energy, music stings, “Doug and Twitch chat take turns… who can take over more of Europe.”

2. 2:17–4:47 — Early banter + stream setup
   Chat updates, small talk, acknowledges this could be “a mess,” long sub-thank-you run, promises rules soon.

3. 4:47–10:40 — Map prep + Chat’s empire naming
   Explains he simplified the Balkans for gameplay; chat brainstorms names; poll leads to Chattistan as the chat empire name.

4. 10:40–14:20 — Doug’s empire naming + pre-rules housekeeping
   Doug picks his empire name (ends up Dougtopia). Quick chat settings/color tip. Transitions into the rules explanation.

5. 14:20–26:34 — Full rules explanation (core game loop)
   Key rules: pick starting country, alternate turns, invade or ally/peace-treaty, research tech, contiguous expansion only, points = countries + special resources, army loss triggers a “wait to rebuild” period, story continuity (“if weird stuff happens, you must build off it”).

6. 26:34–30:56 — Starting countries chosen + game begins
   Poll: chat starts Vatican City. Doug starts Germany. Starting armies: 10,000 swordsmen each. Initial resources: chat has the Pope, Doug has beer.

7. 30:56–46:56 — Early turns: pope diplomacy fails, Switzerland fails, pope escalates

   • Chat tries a pope-marriage alliance with Italy → pope runs away; no treaty.
   • Doug invades Switzerland → repelled; army considered “defeated.”
   • Chat recaptures/assassinates pope (as a PR gambit) → replaces pope-resource with an all-female “girlboss” army resource.

8. 46:56–1:16:19 — Chat takes Italy; Doug finally lands Switzerland (via beer)

   • Chat uses girlboss army tactics to invade Italy → Italy conquered (plus Sicily gets tossed in).
   • Doug attempts Luxembourg/Switzerland diplomacy repeatedly → eventually offers beer successfully and gets Switzerland allied.

9. 1:16:19–1:36:05 — France swing + gunpowder breakthrough

   • Chat attempts alliance with France; it turns into a palace takeover → France conquered.
   • Chat researches “gunpowder” → results in a special tech/resource: horse-mounted cannons.

10. 1:36:05–1:55:10 — Doug’s setbacks + Belgium marriage alliance; Germany defended

• Doug tries to crack Luxembourg again → fails; army status gets messy but “rebuild rule” keeps game moving.
• Doug researches/trains “battle bears,” but the episode ends with the bears wiped out and Switzerland acting volatile.
• Chat invades Germany again → AI result says the invasion fails; Germany holds.
• Doug pivots to Belgium with beer diplomacy → outcome becomes a “royal marriage” scenario → Belgium joins Dougtopia (treated as an alliance).

11. 1:55:10–2:16:25 — Luxembourg “final boss” solved by plague + invasion

• Chat uses a plague-in-baguettes plan aimed at Luxembourg (it narratively hits France too), then follows with horse cannons + girlboss army → Luxembourg conquered.
• Doug attempts a Britain angle via “fish and chips” research, but it mostly becomes repeated meetings with the queen and no clear gain.

12. 2:16:25–2:40:05 — Britain seized by Chattistan; pope rebellion resolved (pope lost)

• Chat tries to recover the runaway pope and use her to secure Britain first.
• Narrative flips into: horse cannons attack London; Britain/England effectively taken by Chattistan, but the pope rebels/runs and is ruled “lost” as a resource (no longer controlled by Chattistan).

13. 2:40:05–2:50:30 — Doug grabs Ireland (exile-to-alliance twist) + score framing
    Doug’s emperor flees and pressures Ireland into joining → Ireland joins Dougtopia. Ireland hands over a “list of leaders” (treated as a minor resource). Doug suggests switching win condition to “first to 10 points” due to time.

14. 2:50:30–2:56:44 — Next turn setup + necromancy rules + Germany targeted again
    Chat regains ability to invade, debates options (including “necromancy” constraints), and ultimately lines up another invasion of Germany as the next major move.

I had chatgpt refine it a bit further and look more specifically at the turns and decision

@chatgpt

Turn 0 (setup / rules / starts)

Decisions

• Chat empire name poll → Chattistan (chosen at 10:33, confirmed 10:40).

• Doug empire name poll → Doug lands on Dougtopia (he briefly says Dugneland at 12:49, then uses Dougtopia).

• “What countries do we add?” → Vatican City explicitly added to the board (24:07).

• Core rules set (25:15):

  • Countries/resources score points.
  • Each side starts with 10,000 swordsmen.
  • Contiguous expansion only.
  • If you lose a war: 2-turn rebuild before your army is usable again.
  • Story must “resolve” weird AI events.

• Boats granted to both empires (32:15) so naval movement is possible.

Starting locations

• Chat start poll → Vatican City (28:49).
• Doug starts Germany (28:55).
• Stated start snapshot (30:49): Dougtopia = Germany, Chattistan = Vatican City; chat goes first.

---

Turn 1 — Chattistan acts (33:41 → 40:10)

Prompt / decision

• Doug asks what chat wants to do; chat’s chosen move: marry the pope off to Italy to form an alliance (33:41).

Action

• Diplomacy via marriage: Pope → Italy = alliance + territorial/peace benefit (intended).

Resolution

• The pope runs away into the forest (40:10).

Mechanical result

• No alliance formed.
• No territory gained.
• Chat remains Vatican City only.

Strategic impact

• Chat’s first action tries to “solve expansion” without fighting, but it fails and also sets up pope-related chaos that becomes a recurring lever.

Alternative lines (what would have been safer)

• Immediate invasion of a microstate (low variance).
• A “resource play” (research/tech) before expanding.

---

Turn 1 — Dougtopia acts (41:46 → 42:19)

Decision

• Doug chooses to invade Switzerland (41:46).

Action

• Straight military invasion.

Resolution

• Switzerland repels him (42:19).

Mechanical result

• Doug’s army is defeated.
• Rebuild penalty triggers (army unusable for the next 2 turns, per the rule Doug set).

Strategic impact

• Doug loses tempo early; chat now has time to expand while Doug is “offline.”

Alternative lines

• Diplomacy-first with a small neighbor.
• Avoid Switzerland (high-defense trope) and take a softer target.

---

Turn 2 — Chattistan acts (46:03 → 46:20)

Decision

• Chat’s move: recapture the pope, assassinate him in the forest, blame Italy (46:03).

Action

• Covert action + framing to destabilize Italy / force advantage.

Resolution

• “Church loses faith” type fallout; twist reveal: Chattistan’s army becomes an all-women ‘girlboss’ army (46:20).

Mechanical result

• Chat gains a new resource / modifier: girlboss army (explicitly treated as a resource later).
• Territory still Vatican City (no land gained this turn).

Strategic impact

• This is a classic “resource spike” turn: no land, but it unlocks higher win probability on future invasions.

Alternative lines

• Use pope as bargaining chip rather than kill (lower chaos).
• Immediate invasion while Doug is rebuilding.

---

Turn 2 — Dougtopia acts (48:30 → 51:23)

Decision

• Doug attempts an alliance treaty with Luxembourg (48:30).

Action

• Diplomacy (while still recovering from the Swiss loss).

Resolution

• Luxembourg turns hostile; emperor escapes; no deal (51:23).

Mechanical result

• No territory gained.
• Doug still in the aftermath of his earlier defeat (rebuild context continues).

Strategic impact

• Doug tries to regain tempo without risking another battlefield loss, but gets neither alliance nor conquest.

Alternative lines

• Consolidate internally (prep a guaranteed next invasion).
• Seek a weaker diplomatic target than Luxembourg.

---

Turn 3 — Chattistan acts (55:23 → 1:00:31)

Decision

• Doug narrates chat’s chosen move: invade Italy using girlboss army + seduction tactic (55:23).

Action

• Military invasion with the new “girlboss” advantage.

Resolution

• Chattistan wins and captures Italy (59:26).
• Doug later also gives Sicily to chat as part of that result (1:00:31).
• “Pope 2” enters the story as something Chattistan ends up having control over (later used in Turn 8).

Mechanical result

• Territory gained: Italy (+ Sicily).
• Resources: girlboss army remains in play; Pope 2 becomes a narrative asset.

Strategic impact

• Chat converts a resource spike into real map control: now Chattistan has a contiguous foothold that is not just Vatican City.

Alternative lines

• Take a smaller adjacent target first (less exposure).
• Use Italy capture to immediately chain into France.

---

Turn 3 — Dougtopia acts (1:01:29 → 1:03:21)

Decision

• Doug tries an alliance treaty with Switzerland (1:01:29).

Action

• Diplomacy to neutralize a dangerous neighbor after being defeated earlier.

Resolution

• Switzerland refuses again (1:03:21).

Mechanical result

• No territory gained.
• Switzerland remains an active threat.

Strategic impact

• Switzerland stays a “hard stop” for Doug’s expansion in that direction.

Alternative lines

• Pivot away from Switzerland entirely (expand elsewhere).
• Use a tech/resource move to change odds before re-engaging.

---

Turn 4 — Chattistan acts (1:19:11 → 1:24:25)

Decision

• Chat chooses: Alliance treaty with France (1:19:11).

Action

• Diplomacy intended to create a buffer against Germany.

Resolution

• France refuses; Chattistan attacks anyway; palace taken; king assassinated; Doug counts it as France captured (1:23:35 → 1:24:25).

Mechanical result

• Territory gained: France.

Strategic impact

• Chat keeps momentum: diplomacy fails, but chat still converts the turn into conquest.

Alternative lines

• Lock borders first (fortify Italy) before pushing further.
• Target a smaller neighbor to reduce risk.

---

Turn 4 — Dougtopia acts (1:26:06 → 1:29:53)

Decision

• Doug attempts to finally take Luxembourg (1:26:06).

Action

• Invasion.

Resolution

• Luxembourg flips back to war; Dougtopia retreats; Doug’s army defeated again (1:29:53).

Mechanical result

• Doug takes another defeat → another 2-turn rebuild window begins (per the rule).
• No territory gained.

Strategic impact

• Doug is repeatedly losing “army uptime,” while chat is compounding territory.

Alternative lines

• Stop rolling on Luxembourg.
• Take a guaranteed diplomatic annexation target instead.

---

Turn 5 — Chattistan acts (1:33:45 → 1:35:10)

Decision

• Chat chooses: Research gunpowder (1:33:45).

Action

• Tech development.

Resolution

• Success: creates horse-mounted cannon tech (“horse cannons”) (1:35:10).

Mechanical result

• Resource gained: horse cannons.

Strategic impact

• Another resource spike; chat is now stacking advantages (girlboss army + horse cannons).

Alternative lines

• Immediate invasion while Doug is still struggling.
• Use tech to target Germany sooner.

---

Turn 5 — Dougtopia acts (1:41:59 → 1:45:33)

Decision

• Doug pursues battle bears (1:41:59).

Action

• Resource attempt (military advantage creature unit).

Resolution

• The bears (and Doug’s forces) get wiped out in the subsequent Swiss-related outcome; Doug is defeated (1:45:33).

Mechanical result

• Resource attempt effectively fails (battle bears don’t become a lasting advantage).
• Another defeat state for Doug’s army context.

Strategic impact

• Doug keeps spending turns on high-variance plays that don’t stabilize his position.

Alternative lines

• Use marriage/alliance plays like the later Belgium/Ireland success.
• Tech up defensively to avoid repeated wipeouts.

---

Turn 6 — Chattistan acts (1:50:50 → 1:53:32)

Decision

• Chat chooses: Invade Germany (1:50:50), using stacked resources.

Action

• Invasion attempt on Doug’s core territory.

Resolution

• Germany successfully defends; invasion fails (1:53:32).

Mechanical result

• No territory gained.
• Chat’s army takes a loss event here (the story marks it as a failed invasion).

Strategic impact

• This is chat’s first big tempo loss: attacking the strongest position early is costly.

Alternative lines

• Keep expanding outward and isolate Germany.
• Force Germany to fight on multiple fronts first.

---

Turn 6 — Dougtopia acts (1:56:13 → 2:04:53)

Decision

• Doug attempts peace treaty with Belgium using beer diplomacy framing (1:56:13).

Action

• Diplomatic annexation via marriage plot.

Resolution

• Belgium accepts via marrying the king to Doug’s princess; Doug counts Belgium joined Dougtopia (2:01:29 → 2:04:53).

Mechanical result

• Territory gained: Belgium.

Strategic impact

• Doug finally gets reliable growth without a battlefield roll—this is his best pattern in the shown segment.

Alternative lines

• Repeat this model with other states instead of invasions.

---

Turn 7 — Chattistan acts (2:12:38 → 2:16:03)

Decision

• Chat chooses: Unleash a plague on Luxembourg (2:12:38).
• Delivery detail: rats hidden in French baguettes (2:14:51).

Action

• Biological sabotage + follow-on conquest.

Resolution

• Luxembourg falls; side effect: plague heavily impacts France’s population (narrative weirdness) (2:16:03).

Mechanical result

• Territory gained: Luxembourg (captured by Chattistan).
• France remains held, but narratively weakened.

Strategic impact

• Chat uses “softening” to turn Luxembourg into a near-guaranteed capture and denies Doug another chance at it.

Alternative lines

• Direct invasion without plague (less collateral).
• Use plague against Germany, but higher blowback risk.

---

Turn 7 — Dougtopia acts (2:18:30 → 2:25:44)

Decision

• Doug tries to create leverage over Britain by researching/creating fish & chips (2:18:30).

Action

• Cultural/diplomatic “resource” attempt.

Resolution

• Loops into meetings with the Queen; no fish & chips gained, no conquest (2:25:44).

Mechanical result

• No resource confirmed.
• No territory gained.

Strategic impact

• Another nonproductive turn while chat continues to convert turns into map control.

Alternative lines

• Immediate alliance/invasion on the Ireland/Britain axis before chat can.

---

Turn 8 — Chattistan acts (2:31:33 → 2:39:47)

Decision

• Chat chooses: find Pope 2 and send her to marry the Queen (2:31:33).

Action

• Diplomatic takeover attempt via marriage.

Resolution

• Pope 2 refuses and starts resistance; Chattistan attacks London using horse cannons; Chattistan takes London; Doug counts Britain captured (2:35:30 → 2:39:47).
• Pope 2 is effectively lost/rebelling as an asset.

Mechanical result

• Territory gained: Britain.
• Narrative asset change: Pope 2 no longer a controllable tool.

Strategic impact

• Chat again turns a failed diplomacy beat into conquest—still high momentum.

Alternative lines

• Keep Pope 2 as internal legitimacy tool rather than spending her on Britain.
• Block Doug from Ireland first.

---

Turn 8 — Dougtopia acts (2:41:21 → 2:50:04)

Decision

• Doug attempts alliance with Ireland (2:41:21).

Action

• Diplomatic coercion/argument.

Resolution

• Ireland joins; Doug counts Ireland captured (2:48:38 → 2:50:04).

Mechanical result

• Territory gained: Ireland.
• Doug also mentions the “leaders list” as a possible resource idea (not clearly used afterward in the shown text).

Strategic impact

• Doug finally creates a second base of territory and denies chat a clean sweep of the Isles.

Alternative lines

• Use Ireland as staging to pressure Britain (now held by chat).

---

Turn 9 — Chattistan acts (2:54:44 → cutoff)

Decision

• Chat chooses: Invade Germany again (2:54:44).

Outcome

• Not present in the provided segment.

If you paste the transcript from 2:54:44 onward, I’ll extend the same per-turn breakdown (decision → action → resolution → mechanical result → strategic impact → alternatives) through the end.

[bedHedd]

swapping out gp55.3 codex for coding

After seeing @progressEdd’s demo with OpenCode and experiencing Opus 4.6 at work, I wanted to look for an equivalent service that would give similar performance and was compatible with OpenCode.

@progressEdd

I tried vibecoding a ui. For 1 night of vibecoding, it did a pretty good job

I started with my GPT codex plan, but I wasn’t super impressed with the past plan implementations and also I frequently reached the rate limit. I considered upgrading to the business plan, but it didn’t reduce the rate limits on codex, and the pro plan of $200/month was too expensive for my use.

While browsing the opencode subreddit for other plans I came across this post which had this table

@u/CantFindMaP0rn

Plan Ranking (as of Feb 22, 2026)

Rank	Plan	Mentions	Sentiment	Key Signal
1	Opencode Black/Zen	5	Positive	Best value; multi-model; cheap entry
2	Codex Plan	4	Strongly Positive	“The best”; 272K context; top performance
3	Alibaba Cloud (Qwen)	2	Positive	$5-10/mo; relaxed quotas; multi-model
4	Chutes.ai	5	Mixed+	Cheap; unreliable for real-time use
5	Copilot	5	Mixed	Broad access; 100K context limit
6	Minimax	3	Positive	Best secondary/budget execution plan
7	OpenRouter API	2	Positive	Fair PAYG pricing; transparent
8	Ollama Cloud	2	Neutral	Good quotas; slow under load
9	ChatGPT Plus	2	Neutral	Needed for Codex 5.3 only
10	Synthetic.new	2	Mixed	Over-capacity; low community validation
11	Z.AI Coding Plan	1	Neutral	No signal
12	Claude Max/Pro	3	Negative	Expensive; session limits; weak coding
13	Kilocode API	2	Negative	Accused proxy/copycat; skip for OpenRouter

Reddit - The heart of the internet

Cursor and Claude code were other options I considered, but my cowokers complained that they frequently reached the rate limit for their side projects. Furthermore, I dislike anthropic and view them as virtue signaling hypocrites. I do not want to give my money to a company that is openly hostile to open source models, and would prefer companies public with their models and research
If you are interested in learning more:

• Anthropic Drops Its Huge Safety Pledge That Was Supposedly the Whole Point of the Company
• Anthropic confirms software engineering is NOT dead
• Yann LeCun Isn’t Buying Anthropic’s Cyberattack Story
  • Yann LeCun: “You’re being played by people who want regulatory capture. They are scaring everyone with dubious studies so that open source models are regulated out of existence.”
• Anthropic joins OpenAI in flagging ‘industrial-scale’ distillation campaigns by Chinese AI firms
• Researchers question Anthropic claim that AI-assisted attack was 90% autonomous

I considered opencode’s black/zen, but it looks like it is api/ use based. I don’t want to get stuck with a potential $1000 bill, so I went with alibaba cloud.

The instructions for getting it setup with opencode was really confusing

to get it working I renamed the provider key

diff --git a/ali-opencode.json b/opencode.json
index 7edfa92..629f088 100644
--- a/ali-opencode.json
+++ b/opencode.json
@@ -1,6 +1,6 @@
 {
-`    "provider": {
-        "bailian-coding-plan": {
+    "provider": {
+        "model-studio-coding": {
             "npm": "@ai-sdk/anthropic",
             "name": "Model Studio Coding Plan",
             "options": {

as when I tried bailing, I kept getting api errors

If I followed their suggested config, it would throw the aforementioned api errors.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "bailian-coding-plan": {
      "npm": "@ai-sdk/anthropic",
      "name": "Model Studio Coding Plan",
      "options": {
        "baseURL": "https://coding-intl.dashscope.aliyuncs.com/apps/anthropic/v1",
        "apiKey": "YOUR_API_KEY"
      },
      "models": {
        "qwen3.5-plus": {
          "name": "Qwen3.5 Plus",
          "modalities": {
            "input": [
              "text",
              "image"
            ],
            "output": [
              "text"
            ]
          },
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "qwen3-max-2026-01-23": {
          "name": "Qwen3 Max 2026-01-23"
        },
        "qwen3-coder-next": {
          "name": "Qwen3 Coder Next"
        },
        "qwen3-coder-plus": {
          "name": "Qwen3 Coder Plus"
        },
        "MiniMax-M2.5": {
          "name": "MiniMax M2.5",
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "glm-5": {
          "name": "GLM-5",
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "glm-4.7": {
          "name": "GLM-4.7",
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "kimi-k2.5": {
          "name": "Kimi K2.5",
          "modalities": {
            "input": [
              "text",
              "image"
            ],
            "output": [
              "text"
            ]
          },
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        }
      }
    }
  }
}

This is what I ultimately used.

{
  "$schema": "https://opencode.ai/config.json",
  "permission": {
    "read": {
      "~/.config/opencode/get-shit-done/*": "allow"
    },
    "external_directory": {
      "~/.config/opencode/get-shit-done/*": "allow"
    }
 },
 "provider": {
    "model-studio-coding": {
      "npm": "@ai-sdk/anthropic",
      "name": "Model Studio Coding Plan",
      "options": {
        "baseURL": "https://coding-intl.dashscope.aliyuncs.com/apps/anthropic/v1",
        "apiKey": "MY_API_KEY"
      },
      "models": {
        "qwen3.5-plus": {
          "name": "Qwen3.5 Plus",
          "modalities": {
            "input": [
              "text",
              "image"
            ],
            "output": [
              "text"
            ]
          },
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "qwen3-max-2026-01-23": {
          "name": "Qwen3 Max 2026-01-23"
        },
        "qwen3-coder-next": {
          "name": "Qwen3 Coder Next"
        },
        "qwen3-coder-plus": {
          "name": "Qwen3 Coder Plus"
        },
        "MiniMax-M2.5": {
          "name": "MiniMax M2.5",
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "glm-5": {
          "name": "GLM-5",
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "glm-4.7": {
          "name": "GLM-4.7",
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        },
        "kimi-k2.5": {
          "name": "Kimi K2.5",
          "modalities": {
            "input": [
              "text",
              "image"
            ],
            "output": [
              "text"
            ]
          },
          "options": {
            "thinking": {
              "type": "enabled",
              "budgetTokens": 1024
            }
          }
        }
      }
    }
  }
} 

transcript generation

Once I got opencode setup, I used gsd to implement the transcript extraction. I used gpt 5.3 codex to discuss and create the plan, then I had GLM 5 implement it. I ran into some gpu issues using whisperx, so I had GLM swap it out for moonshine. If the results aren’t as good, I was thinking of using whisper.cpp. I still need to inspect it though.

I’m pretty impressed with GLm 5, as it created a script that I could use to process a video. It took a couple of cycles, but since I wasn’t rate limited, I was able to get the model to figure it out.

[bedHedd]

app interface to inspect transcripts + chat

I was going to work on this, but realized that I couldn’t paste screenshot to illustrate what how I want the tool to look. I could have used the marimo notebooks from before chat-extraction.py, but it didn’t work for all video codecs and was more effort to customize.

replicating markdown paste images

I had chatgpt replicate my setup from markdown paste for the path I screenshotted this setting and passed my markdown paste settings.

${workspaceRoot}/00-supporting-files/images/${fileBasenameNoExtension}

and the file name structure

${datetime|yyyyMMDDHHmmss}

ultimately what worked was

"markdown.copyFiles.destination": {

"**/*.md": "${documentWorkspaceFolder}/00-supporting-files/images/${documentBaseName}/${isoTime/^([0-9][0-9][0-9][0-9])-([0-9][0-9])-([0-9][0-9])T([0-9][0-9]):([0-9][0-9]):([0-9][0-9]).*$/$1$2$3$4$5$6/}.${fileExtName}"

}

This works, but I still can’t copy and paste rich text D:

other ones I tried

unixtime

it works with the vscode default settings, but doesn’t quite match up with the existing file structure I use elsewhere

"markdown.copyFiles.destination": {

"**/*.md": "${documentWorkspaceFolder}/00-supporting-files/images/${documentBaseName}/${unixTime}.${fileExtName}

"

}

daily notes parsing

This works for daily note, but wouldn’t work if I wanted to paste this into other markdown files

"markdown.copyFiles.destination": {

"**/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].md": "${documentWorkspaceFolder}/00-supporting-files/images/${documentBaseName}/${isoTime/^([0-9][0-9][0-9][0-9])-([0-9][0-9])-([0-9][0-9])T([0-9][0-9]):([0-9][0-9]):([0-9][0-9]).*$/$1$2$3$4$5$6/}.${fileExtName}"

}

Though the core logic worked well enough that I just swapped out the keys between unix time and this one.

[progressEdd]

@bedHedd

@progressEdd

Also @bedHedd, I saw you were using my project template for dougpt. You should check out the vibe coding gsd branch I made.

I checked it out and it lines up with my workflow. I actually tried it out with my transcript processing pipeline. I’m 2 updates behind, so I’ll post it later

Also you should update your foam daily note template as the title doesn’t work if you want to create notes in the past. I updated it here

use foam date instead of current day · BedHedd/dougpt@1d05db2 · GitHub

Sure will add that on my todo

@bedHedd

While browsing the opencode subreddit for other plans I came across this post which had this table

Plan Ranking (as of Feb 22, 2026)

Rank	Plan	Mentions	Sentiment	Key Signal
1	Opencode Black/Zen	5	Positive	Best value; multi-model; cheap entry
2	Codex Plan	4	Strongly Positive	“The best”; 272K context; top performance
3	Alibaba Cloud (Qwen)	2	Positive	$5-10/mo; relaxed quotas; multi-model
4	Chutes.ai	5	Mixed+	Cheap; unreliable for real-time use
5	Copilot	5	Mixed	Broad access; 100K context limit
6	Minimax	3	Positive	Best secondary/budget execution plan
7	OpenRouter API	2	Positive	Fair PAYG pricing; transparent
8	Ollama Cloud	2	Neutral	Good quotas; slow under load
9	ChatGPT Plus	2	Neutral	Needed for Codex 5.3 only
10	Synthetic.new	2	Mixed	Over-capacity; low community validation
11	Z.AI Coding Plan	1	Neutral	No signal
12	Claude Max/Pro	3	Negative	Expensive; session limits; weak coding
13	Kilocode API	2	Negative	Accused proxy/copycat; skip for OpenRouter

Best bang for your bucks plan? : opencodeCLI

Interesting, I’ll have to look into it, my work gives my github copilot.

@bedHedd

This works, but I still can’t copy and paste rich text D:

Interesting I might have to look into seeing if I could have GLM update the extension as I also get errors on wayland now that X-clip isn’t used anymore

res/scripts/linux_get_clipboard_text_html.sh