JSON Driver Masterclass

What is a Driver?

A driver is a JSON file that tells Onset Engine what visual content to assign to each energy tier of your music. Without a driver, the engine uses raw CLIP similarity and motion scores. With a driver, you get precise creative control over what appears at each musical intensity level.

Think of it as a content brief for the AI: “During quiet sections, show landscapes. During medium energy, show dialogue. During the drop, show explosions.”

Tip

Don’t want to write JSON? Use Text Descriptions — two simple text fields that auto-generate a driver for you.

The v3 Schema

{
  "meta": {
    "name": "Action Anime Driver",
    "version": "3.0",
    "description": "Maps anime content to energy tiers with subject awareness"
  },
  "global": {
    "min_rating": 3,
    "exclude_tags": ["@Filler", "@Recap"]
  },
  "tiers": {
    "1_LOW": {
      "descriptions": [
        "character standing in peaceful landscape",
        "calm sky with clouds",
        "characters talking quietly"
      ],
      "subjects": ["@Goku"],
      "moods": ["serene", "melancholic"],
      "scene_types": ["wide", "medium"]
    },
    "2_MED": {
      "descriptions": [
        "character powering up with glowing aura",
        "intense stare between fighters",
        "character flying through sky"
      ],
      "subjects": ["@Goku", "@Vegeta"],
      "moods": ["tense"],
      "scene_types": ["medium", "close-up"]
    },
    "3_HIGH": {
      "descriptions": [
        "fast martial arts combat with punching",
        "energy beam attack",
        "character dodging rapid attacks"
      ],
      "subjects": ["@Goku"],
      "moods": ["aggressive", "epic"],
      "scene_types": ["close-up", "medium"]
    },
    "4_MAX": {
      "descriptions": [
        "massive energy explosion",
        "character transforming with blinding light",
        "devastating beam clash"
      ],
      "subjects": ["@Goku"],
      "moods": ["epic"],
      "min_rating": 4
    }
  }
}

Schema Reference

Top-Level Blocks

Block	Description
meta	Display name, version, and description
global	Settings applied across all tiers
tiers	Energy tier definitions keyed as 1_LOW, 2_MED, 3_HIGH, 4_MAX

Meta Fields

Field	Type	Description
name	string	Display name for the driver
version	string	Schema version (use "3.0" for current)
description	string	What this driver is designed for

Global Settings

Field	Type	Default	Description
min_rating	integer	0	Baseline rating preference for all tiers (0–5). A soft penalty, not a floor — see below
exclude_tags	string[]	[]	Drop clips carrying these tags from the pool entirely. The only hard filter in the driver system
tier_thresholds	number[]	[0.25, 0.50, 0.75]	The three energy cutoffs between the four tiers. Raise them to make peak tiers rarer
tag_weight	number	0.3	How much a tier’s @Tag references count toward the final score, vs. its descriptions

Tier Fields

Each tier is keyed as 1_LOW, 2_MED, 3_HIGH, or 4_MAX:

Field	Type	Description
descriptions	string[]	CLIP text descriptions — the engine computes cosine similarity against clip embeddings
tags (or subjects)	string[]	Subject tag references using @TagName syntax
moods	string[]	Preferred mood classification. Mismatches take a 0.50x score penalty
scene_types	string[]	Preferred scene type. Mismatches take a 0.60x penalty
min_rating	integer	Per-tier rating preference (overrides global). Lower-rated clips take a 0.70x penalty

moods, scene_types and min_rating are preferences, not filters

They multiply a clip’s score rather than removing it from the pool. A clip that matches none of them can still be selected if it is the best available fit for that moment — which is what stops your edit developing gaps when the library runs thin. Only exclude_tags removes clips outright.

Energy Tiers

The engine maps musical energy (0.0–1.0) to four tiers:

Tier	Energy Range	Musical Moment
1_LOW	0.0–0.25	Intros, breakdowns, quiet sections
2_MED	0.25–0.50	Verses, building tension
3_HIGH	0.50–0.75	Choruses, buildups
4_MAX	0.75–1.0	Drops, climaxes, peak energy

CLIP Descriptions — How Matching Works

Each description string is encoded into a 768-dim vector using the CLIP text encoder. The engine computes cosine similarity between the description vector and every clip’s embedding.

Description: "massive energy explosion"
   ↓ CLIP text encoder
   ↓ 768-dim vector
   ↓ cosine similarity vs. all clips
   ↓ ranked results

Clip #4821: cos_sim = 0.31  ← best match
Clip #1203: cos_sim = 0.28
Clip #0892: cos_sim = 0.24

Contrastive Scoring

v3 drivers use contrastive scoring — the engine doesn’t just pick clips with the highest absolute similarity to a tier’s descriptions. It measures tier specificity: how much more similar is this clip to the target tier than to all other tiers?

contrast = target_tier_sim - best_other_tier_sim
score    = raw_similarity * 0.7 + contrast * 0.3

This prevents the common failure mode where the CLIP model’s highest-confidence clips dominate every tier. Contrastive scoring ensures that calm tiers get genuinely calm content, not just the model’s most confident matches.

Subject Tags (@ Syntax)

Reference tagged clips using the @TagName syntax in the subjects array:

{
  "4_MAX": {
    "descriptions": ["massive energy explosion"],
    "subjects": ["@Goku", "@Vegeta"],
    "moods": ["epic"],
    "min_rating": 4
  }
}

Tags are created via the few-shot propagation system (tag 5 clips → engine finds 800 more).

Tags steer, they don’t lock

Subject tags are blended with the description score, not AND-ed with it. The engine takes the clip’s best similarity to any tag centroid in the tier and mixes it in at tag_weight (default 0.3 — so 70% descriptions, 30% tag).

A tier tagged @Goku will lean heavily toward Goku clips, but a clip that matches the descriptions superbly can still beat a weak Goku clip. Raise tag_weight in global if you want the subject to dominate; there is no setting that makes it absolute.

Mood and Scene Type Preferences

Available Moods

Mood	Description
epic	Grand, powerful, heroic content
melancholic	Sad, reflective, emotional
tense	Suspenseful, high-stakes
comedic	Light, funny, playful
romantic	Intimate, warm, affectionate
serene	Peaceful, calm, meditative
aggressive	Intense, violent, forceful
mysterious	Dark, eerie, shadowy

Available Scene Types

Scene Type	Description
close-up	Face or detail shot
medium	Waist-up or small group
wide	Full environment or establishing shot
aerial	Drone or overhead perspective
pov	First-person or subjective camera
slow_motion	Reduced playback speed content (note the underscore)
group	Several people together, crowd scene

Spell these exactly — a typo fails silently

These two lists are the complete vocabulary. Anything else (joyful, dramatic, slow-motion with a hyphen) matches no clip at all, so every candidate takes the penalty equally, it cancels out of the ranking, and the preference does nothing. The tier looks configured and behaves as though the field were absent.

Onset Engine now warns on load when a driver asks for a value that doesn’t exist — watch the console if a mood or scene preference seems to be ignored.

Building Your First Driver

Step 1: Identify Your Content

What footage is in your library? Anime fights? Drone landscapes? Wedding ceremonies? The driver should reflect your actual content, not aspirational queries.

Step 2: Write Descriptions

Be specific and visual — but write captions, not prose. CLIP was trained on image captions and alt-text, so it responds to how people label pictures: short, blunt, concrete. Full descriptive sentences consistently score worse than the tag-style phrasing of the same idea when measured against a real library.

// ❌ Too vague — matches everything, so it distinguishes nothing
"descriptions": ["action"]

// ❌ Too prose-y — the extra words dilute the match
"descriptions": ["a character performing a spinning kick in mid-air"]

// ✅ Concrete, caption-style
"descriptions": [
  "spinning kick, mid-air",
  "explosion, flying debris",
  "sword slash, motion blur"
]

Step 2b: Escalate by content, not by speed

This is the single biggest driver mistake. If your four tiers describe the same thing at different speeds — “slow movement” → “fast movement” → “very fast movement” — they will all pull the same clips. CLIP reads what is in the frame far better than it reads pace.

Build the ladder out of things you could point at in a still: subject, framing, clothing, location, act, weather, time of day. Rewriting a driver from a pace ladder to a content ladder has been measured to cut tier overlap from ~27% to under 13%.

// ❌ Pace ladder — tiers will overlap heavily
"1_LOW":  ["gentle unhurried movement"]
"4_MAX":  ["blurred fast motion"]

// ✅ Content ladder — genuinely different clips
"1_LOW":  ["empty street, dawn"]
"4_MAX":  ["car chase, night, sparks"]

Step 3: Use the Driver Wizard

In Studio Mode, click ✨ Create in the Clip Direction section to open the Driver Wizard — a visual tier builder with live JSON preview. If you’ve entered text descriptions, the wizard pre-populates from those.

Step 4: Test and Iterate

Use DJ Mode to preview how the driver selects clips in real time. Adjust descriptions based on what you see. The console shows per-tier diagnostic logging with similarity breakdowns.

The thing to watch for is tier separation — whether your tiers actually pull different clips. Step through the tiers in DJ Mode and look at what each one serves up. If two adjacent tiers keep offering you the same footage, their descriptions are saying the same thing to the AI even though they read differently to you, and the edit will feel flat no matter how the energy curve moves. Rewrite one of them to describe something the other genuinely doesn’t.

Tip

Wording that seems obviously different to a human is often near-identical to the model. “Intense close-up action” and “high-energy tight shots” will usually pull the same pool. Separate tiers by content — what is physically in the shot — rather than by intensity adjectives.

Example Drivers

Nature / Drone

{
  "meta": {
    "name": "Nature Reel",
    "version": "3.0"
  },
  "tiers": {
    "1_LOW": {
      "descriptions": ["calm ocean waves", "forest canopy from above", "sunrise over mountains"],
      "scene_types": ["wide", "aerial"]
    },
    "2_MED": {
      "descriptions": ["flowing river through valley", "birds in flight"],
      "scene_types": ["wide", "medium"]
    },
    "3_HIGH": {
      "descriptions": ["fast drone dive through canyon", "waterfall close-up"],
      "scene_types": ["aerial", "pov"],
      "min_rating": 2
    },
    "4_MAX": {
      "descriptions": ["storm clouds time-lapse", "lightning strike over ocean"],
      "scene_types": ["wide"],
      "min_rating": 3
    }
  }
}

Wedding Highlight

{
  "meta": {
    "name": "Wedding Highlights",
    "version": "3.0"
  },
  "global": {
    "min_rating": 2
  },
  "tiers": {
    "1_LOW": {
      "descriptions": ["wedding venue exterior", "floral decorations", "guests arriving"],
      "moods": ["serene"],
      "scene_types": ["wide"]
    },
    "2_MED": {
      "descriptions": ["bride walking down aisle", "exchanging rings", "emotional guests"],
      "moods": ["romantic", "melancholic"],
      "scene_types": ["medium"]
    },
    "3_HIGH": {
      "descriptions": ["first dance", "wedding party celebration", "champagne toast"],
      "moods": ["romantic", "epic"],
      "scene_types": ["medium", "close-up"]
    },
    "4_MAX": {
      "descriptions": ["crowd dancing at reception", "confetti throw", "sparkler exit"],
      "moods": ["epic"],
      "scene_types": ["wide", "close-up"],
      "min_rating": 3
    }
  }
}

Tips

• Write 3–6 descriptions per tier for best results — more variety = better coverage
• Use @Tag subjects only after running clip tagging in the GUI
• The penalty multipliers stack: a clip with wrong mood AND scene can get 0.50 × 0.60 = 0.30× score
• Per-tier min_rating overrides the global setting for that tier only

Related Articles

• Text Descriptions
• Configuration Reference