Permalink to Timing & AssemblyTiming & Assembly

For spoken video, a SemanticTrack connects the authored Script to the actual performance. This is the natural time source for captions, word-triggered graphics and coverage. Build it in segment-sized pieces:

1. normalize each accepted A/V take into one exact frame domain;
2. align that normalized media with its authored Script Segment to create a self-contained SemanticTake;
3. assemble the Semantic Takes in program order with speech:Track.

Every Take is already semantic before it enters the Speech Track. Pictures can be presented by Media Track or a project component independently of this semantic and audio assembly.

A purely visual animation can instead declare its own duration and frame rate in a ProgramSpace. See an authored film clock. It can use seconds or frames for events; spoken work can use Script Selections and Moments for the same visual behavior.

<import as="program" from="@hypit/program-space@1"/>
<import as="pipeline" from="@hypit/media-pipeline@1"/>
<import as="whisperx" from="@hypit/whisperx@1"/>
<import as="speech" from="@hypit/speech-track@1"/>
<import as="media-track" from="@hypit/media-track@1"/>
<import as="space" from="@hypit/spatial@1"/>
<import as="recipes" source="./recipes.svs"/>

Permalink to normalize-each-takeNormalize each take

Normalization makes video, audio, duration and frame rate one explicit SynchronizedMedia fact. The Clock is authored once and shared by every Take that will enter the same SemanticTrack.

<program:Clock id="clock" frame-rate="30"/>

<pipeline:Normalize id="opening-media" source={opening-video.video}
  video="primary-moving" audio="default" span-authority="video" clock={clock}/>
<pipeline:Normalize id="answer-media" source={answer-video.video}
  video="primary-moving" audio="default" span-authority="video" clock={clock}/>

Normalization contains no Script meaning and performs no transcription. It only establishes the media facts that later semantic alignment can trust.

Permalink to create-one-semantictake-per-segmentCreate one SemanticTake per Segment

whisperx:SemanticTake measures one normalized Take and aligns the evidence with exactly one authored Segment:

<whisperx:SemanticTake id="opening-semantic" narrative={story}
  segment={story.segment.opening} media={opening-media.media} language="en"/>
<whisperx:SemanticTake id="answer-semantic" narrative={story}
  segment={story.segment.answer} media={answer-media.media} language="en"/>

language is required on every alignment call and currently accepts en or zh. It is passed unchanged to WhisperX; Hypit does not detect or route languages from Script text or audio.

Each output contains the normalized media, the Segment identity, every authored word's local frame window, and all of that Segment's structural anchors. There are two anchors for the Segment and two for each word. Acoustic evidence is an implementation input to this step; downstream components see the completed SemanticTake, not a second evidence-shaped timing structure.

Permalink to assemble-the-semantictrackAssemble the SemanticTrack

speech:Track concatenates already-semantic Takes in document order and provides the semantic timeline and its original sound. Media presents the performance separately:

<space:Canvas id="vertical" width="1080" height="1920"/>
<space:Frame id="speech-frame" within={vertical}
  left="0%" top="0%" right="100%" bottom="100%"/>

<speech:Track id="speech">
  <speech:Take source={opening-semantic.take}/>
  <speech:Take source={answer-semantic.take}/>
</speech:Track>
<media-track:Track id="performance" semantic={speech.semantic} canvas={vertical}>
  <media-track:Performance during="program" frame={speech-frame}
    appearance={recipes.media.performance}/>
</media-track:Track>

Picture consumers and the original audio use the same prepared Takes and source positions. SemanticTrack derives global frames by prefix-summing the local Take lengths, and also supplies the program duration and frame domain required by Film and Render.

Permalink to consume-semantic-timeConsume semantic time

Selections, Moments and whole Segments remain authored Script identities. A downstream component receives the SemanticTrack once and projects those identities into frames only when it builds its deterministic Track:

<media-track:Track id="cards" semantic={speech.semantic} canvas={vertical}>
  <media-track:Item image={card.image} extent={card-extent}
    during={story.selection.demo} frame={card-frame}
    appearance={recipes.media.card} motion={recipes.motion.card}/>
</media-track:Track>

<caption-fine:Track id="captions"
  document={story.caption}
  semantic={speech.semantic}
  program={caption-program}/>

<film:Film id="main" canvas={vertical}
  semantic={speech.semantic} appearance={recipes.film.vertical}>
  <film:Track source={performance.visual}/>
  <film:Track source={speech.audio}/>
  <film:Track source={cards.visual}/>
  <film:Track source={captions.track}/>
</film:Film>

<render:Video id="final"
  composition={main.composition} semantic={speech.semantic}/>

Use during={story.segment.answer} for a whole Segment, a Selection for an authored range, a Moment for a point event, and during="program" for the complete SemanticTrack domain. Components consume semantic={speech.semantic}.

prepared Takes → Speech Track ── .semantic → Media / project scene → .visual ─┐
                         │             └──→ Caption / semantic graphics ───┤
                         └───── .audio ────────────────────────────────────┤
                                                                         Film