A lot of my research begins with video. When a talk is worth studying closely, I usually extract its transcript with yt-dlp, pass it to a text model, and turn the result into structured notes.
That workflow looked good enough until I began checking those notes against the original videos. The notes looked complete. They were not.
Speakers often finish spoken ideas with something visible on screen. They use phrases like “this,” “right here,” and “as you can see,” whose meaning depends on a visual target. Flatten the video into text and the target disappears.
The model could not inspect what the speaker was referring to. Worse, the missing evidence did not present itself as missing. The notes could supply a plausible interpretation without saying that the source representation was incomplete.
In a research workflow, that is more dangerous than an obvious error. An unsupported inference can quietly enter the record as though it came from the source.
I rebuilt the workflow around that failure.
I built video-context-mcp to prepare a video as an aligned working set: transcript segments, OCR-derived screen text, keyframes, metadata, and timestamps.
Through the Model Context Protocol, an agent can search that material and inspect a relevant moment. Speech and screen evidence remain distinct, but they can be retrieved together.
The text model did not learn to watch video. Its native capabilities stayed the same.
What changed was the system around it.
I call this evidence-path engineering: preparing a source so the model can retrieve relevant evidence and the resulting claims can retain resolvable links to the original artifact.
Video is a useful stress test because its meaning is spread across speech, visuals, and time. The larger question is how much practical capability can be built around a model—and where that composition breaks down.
“...right here...”
A transcript contains the phrase right here but not its visual referent. Source preparation aligns timestamped transcript segments, OCR and keyframes, and timestamps so an agent can search and inspect the relevant moment. The prepared evidence crosses the interaction boundary without collapsing the source into one session-bound answer.
A model boundary is not a source boundary
The failure became easier to understand once I separated two boundaries.
The interaction boundary determines what can reach the model during a run. A text model can receive language but cannot directly inspect a frame. An image-capable model can inspect selected frames. A model with native-video support can receive a larger temporal sequence.
The source boundary determines what remains available around and after that run. A video treated as a source should remain searchable across questions. Claims should resolve to specific moments. Another person or agent should be able to inspect the evidence from which an answer was assembled.
My original notes failed at both boundaries. The transcript crossed into context, but the slide did not. Once the transcript had been summarized, the resulting claims also lost their connection to the moments that supported them.
This changed how I thought about model capability. The model may lack the modality. The harness may fail to deliver a frame. The prepared representation may omit the evidence. Retrieval may select the wrong moment. The final answer may have no usable path back to the source.
Those are different failures, and they do not share one remedy.
Changing models may be the right choice. Native-video input is often simpler for a one-off question, especially when motion or timing carries the answer.
But a research workflow may already depend on a particular harness, tool set, privacy boundary, or local deployment. Switching models may solve the input mismatch while requiring the surrounding workflow to change.
Native input also does not guarantee that a source remains searchable or reusable across later questions. A prepared evidence path can provide those properties, although it may lose motion, gesture, spatial relationships, or other features preserved by direct video analysis.
Native input optimizes for direct analysis of an interaction. Prepared evidence optimizes for selective reuse, stable source handles, and repeated inspection. The right path depends on the evidence the task needs, where that evidence is blocked, and what an alternative representation would lose.
Building the evidence path
Sending more material to the model was not enough. An entire transcript plus hundreds of frames would replace missing evidence with noise.
I needed to prepare the source broadly and disclose it narrowly.
video-context-mcp indexes speech, OCR, keyframes, metadata, and timestamps as one aligned working set. A question begins with lightweight search results and identifiers. From there, the agent can retrieve a transcript range, inspect a frame, crop a region, or search again. The full artifact stays outside the immediate context.
Search results are labeled speech or screen and include a timestamp. A speech hit points the agent toward get_transcript; a screen hit points toward peek_frame and, for dense material, crop_region. The server does not infer every ambiguous visual reference or force an inspection. The agent must choose the follow-up action.
A research question enters five stages: represent source, select context, expose actions, preserve state, and verify claim. A return path from verification to context is labeled insufficient evidence, showing that the source can be inspected again.
Representation came first. I had to decide what could survive the conversion. Speech became timestamped text. Visible language became OCR. Selected screen states became keyframes. Each representation preserved part of the video and discarded something else.
Then came selection. Indexing the whole source made later discovery possible, but the agent still needed a small evidence bundle for the current question.
Retrieval also had to be actionable. Search results were useful only if the agent could follow them into transcript ranges, frames, crops, and nearby moments.
State mattered because I did not want every question to restart ingestion. A stable video identifier, prepared index, and resolvable evidence handles allowed the investigation to continue across turns and retries.
Finally, the answer needed a route back to the source. Timestamps and channel labels provide that route. They do not prove that an interpretation is correct, but they make the interpretation inspectable.
These are not stages in a one-way pipeline. Verification can reveal that the current evidence is insufficient and send the agent back to search or inspection.
The division of labor is fairly clear. Deterministic infrastructure extracts and locates evidence. Tool descriptions make the available inspection actions legible. The model chooses what to inspect and how the pieces relate. The user supplies the research question and reviews consequential or ambiguous conclusions.
MCP is how I implemented that loop. The architecture does not depend on MCP. Another protocol or application could expose the same evidence operations.
The result also degrades according to the connected model. A text-only model can use transcripts, OCR, search results, and timestamps but cannot directly inspect the geometry of a diagram. An image-capable model can inspect frames and crops. A native-video model may not need the same decomposition for a single question, but repeatable retrieval and stable source handles can still matter when the video will be used again.
A narrow mechanism trace
A deterministic replay of an investigation into Frank Coyle’s “Why Agentic Systems Need Ontologies” shows the retrieval mechanism at work. The task is narrow: retrieve the spoken distinction between structural and ontology-level validation together with the screen artifacts that accompany it, while preserving channel and timestamp.
- At 18:14, speech distinguishes structural input checks from ontology-level result checks.
- At 18:43, the screen supplies the compact formulation “Pydantic in, ontology out.”
- At 19:11, concurrent speech and screen present the one-refund constraint and the double-payout failure together.
→ double-payout bug
A schematic timestamped sequence shows speech evidence at 18 minutes 14 seconds, the Pydantic in and ontology out formulation on screen at 18 minutes 43 seconds, and the one-refund rule with its double-payout failure in concurrent speech and screen at 19 minutes 11 seconds. Connector lines converge on a source-linked cross-channel evidence bundle.
The replay shows search returning evidence from both indexed channels and follow-up inspection resolving the screen hits to source moments. A transcript-only workflow cannot retrieve a slide as a slide. The prepared system can.
That is the capability demonstrated here: source-linked access to a previously absent screen channel.
It is not evidence that visual retrieval improved the interpretation. Coyle speaks most of the semantic distinction aloud, so the transcript can support the broad Pydantic-versus-ontology explanation on its own.
A claim about answer quality would require a paired comparison that holds the model, harness, video, prompt, and answer requirements constant. One condition would receive only the relevant transcript. The other could retrieve the speech-and-screen bundle. Both answers would then be checked against the same claims and timestamps.
Until that comparison exists, the defensible result is narrower: the system made an absent evidence channel retrievable and inspectable while preserving its source handles.
Where the capability resides
Calling this a new model capability would be inaccurate.
The model is the underlying model and its supported interface. The harness is the runtime that manages tools, state, and result delivery. An agent is the model acting through that harness. The system is the complete assembly of model, harness, prepared evidence, environment, and supervision.
Capability belongs to that assembly for a defined task. In this case, the task-relative capability is retrieving and inspecting screen evidence that is absent from the transcript representation.
A source-model mismatch can be addressed in several places. I can choose a model that accepts the source natively. I can transform the source into representations the current model accepts. I can also give the model a way to request and inspect those representations as it works.
Each choice changes what the full system can do.
This fits the broader shift toward compound AI systems, where useful behavior emerges from models combined with retrieval, tools, and control logic. Recent harness-engineering practice asks whether an environment’s tools, knowledge, and feedback loops are legible to an agent. Context engineering controls what reaches the model at a given moment.
Evidence-path engineering applies the same concern to the source. It asks how an artifact becomes inspectable, how necessary evidence crosses the model’s interface, and how resulting claims can retain resolvable links to that artifact.
I encountered the same pattern while building semantic-code-mcp. Instead of pushing an entire private codebase into context, external infrastructure creates an addressable semantic index. The agent retrieves narrow code slices, and those results retain links to the source files.
In both cases, the useful unit of analysis is the task-system pair, not the model alone.
What survives when the harness changes?
Better notes were the first payoff. The second appeared when I tried to use the same source from different reasoning environments.
Preparing a video spends compute and storage before the first question. That cost can be justified when the working set supports repeated retrieval, reuse, or verification. If preparation lives only inside one conversation, changing harnesses means starting over.
If the working set exists outside the conversation, another compatible harness connected to the same prepared store—and operating with the necessary permissions—may begin from the same transcript, OCR, keyframes, metadata, and timestamps.
That does not make the full capability portable.
A shared protocol can preserve an interface. The result still depends on tool discovery, permissions, modality support, context management, reasoning, and supervision.
A source artifact feeds a relatively stable layer containing representations, addressable evidence, a retrieval contract, and verification handles. That layer branches to three harness and model pairs, which each produce a separate task-relative outcome. Dashed return paths show that each outcome can resolve back to the source evidence.
The most stable part is the prepared source and its evidence handles. Compatible harnesses may also be able to perform the same pattern of search, inspection, follow-up, and verification.
Interpretation travels less reliably.
Two agents can inspect the same frame and understand it differently. They can choose different actions, stop at different points, or produce answers of different quality. Shared evidence makes comparison and verification possible; it does not guarantee the same outcome.
This is why I think of evidence access as more portable than capability itself.
The current implementation provides a reusable working set, not durable institutional memory. Long-lived use would need persistence, export, access control, lifecycle policies, and evidence handles that continue to resolve over time.
Where composition stops
Composition works only when decomposition preserves the evidence required by the task.
A decision flowchart asks six questions in sequence. Each yes branch continues to the next decision. Each no branch ends at a named outcome. Six yes answers lead to task-relative evidence access.
Video provides obvious failure cases. A question about the exact sequence of keypresses in a coding demonstration may depend on a button that flashes red for one frame. A keyframe extractor based on scene changes or fixed intervals may omit it. The transcript may never mention it. OCR may never capture it.
The prepared evidence path would then be blind to the event the question is asking about.
The same problem appears when meaning depends on gesture, motion, spatial structure, or metadata the harness does not deliver. In those cases, the system should expose its insufficiency. The next step may be another inspection, human review, continuous-video analysis, or a different system.
Composition also has a cost. Every retriever, cache, permission, and control loop adds latency, complexity, debugging work, and attack surface. A workaround that makes sense today may become unnecessary when a later model supports the task directly.
That is fine. The infrastructure should remain only while it earns its complexity.
A model’s interface does not always define what the complete system can do. Some gaps can be crossed with tools. Others cannot.
My original notes looked complete because the missing evidence had disappeared before the model could object to its absence. The workflow improved when that loss became visible, inspectable, and sometimes recoverable.
Implementation
The implementation is available in the video-context-mcp repository and on npm.