From c49d1ddb9e1048eb61c4b75827bf9925e8614eae Mon Sep 17 00:00:00 2001 From: Diwakar-odds Date: Tue, 23 Jun 2026 13:48:17 +0530 Subject: [PATCH 1/2] feat: add intra-stage generation checkpoint and resume support (closes #129) --- skills/meta/checkpoint-protocol.md | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/skills/meta/checkpoint-protocol.md b/skills/meta/checkpoint-protocol.md index 688a65c6..21e2bdb8 100644 --- a/skills/meta/checkpoint-protocol.md +++ b/skills/meta/checkpoint-protocol.md @@ -52,7 +52,22 @@ The checkpoint utility will: - Write the checkpoint JSON to disk - Include timestamp and stage metadata -### Step 4: Human Approval (If Required) +### Step 4: Intra-Stage Checkpointing (Resume Support) + +Long-running stages (like `assets` or `compose` loops) can fail midway due to API errors, rate limits, or session interruptions. To allow resuming from the exact point of failure (e.g., Scene 4): + +1. **Write partial progress**: Every time you successfully generate a significant item (e.g., one scene's assets, one clip), write an `in_progress` checkpoint. Because `status` is `"in_progress"`, validation will not fail even if the canonical artifact is incomplete. + ```python + write_checkpoint( + pipeline_dir, project_name, + stage="assets", + status="in_progress", + artifacts={"asset_manifest": partial_manifest_dict} + ) + ``` +2. **Resume from partial progress**: When starting a stage, ALWAYS check if an `in_progress` checkpoint exists for it. See Step 7 (Resume Protocol) for how to handle it. + +### Step 5: Human Approval (If Required) When `human_approval_default: true`: @@ -87,7 +102,7 @@ When `human_approval_default: true`: - `compose` — Rarely. But human may want to preview. - `publish` — Always. Human must approve before anything goes public. -### Step 5: Determine Next Stage +### Step 6: Determine Next Stage After checkpoint is written and approved (if needed): @@ -97,7 +112,7 @@ next_stage = get_next_stage(pipeline_dir, project_name) This reads all existing checkpoints and returns the next stage that needs to run, or `None` if the pipeline is complete. -### Step 6: Resume Protocol +### Step 7: Resume Protocol At the START of any pipeline run (not just after a stage), always check for existing progress: @@ -107,8 +122,13 @@ next_stage = get_next_stage(pipeline_dir, project_name) If `next_stage` is not the first stage: 1. Inform the human: "Found existing progress. Resuming from stage: [next_stage]" -2. Load prior artifacts from checkpoints for context -3. Continue from that stage +2. **Check for partial progress**: Read the checkpoint for `next_stage`: + ```python + current_cp = read_checkpoint(pipeline_dir, project_name, next_stage) + ``` + If `current_cp` exists and its status is `"in_progress"`, inform the human you are resuming from the middle of the stage. +3. **Load artifacts**: Load prior artifacts from checkpoints for context. If resuming from `"in_progress"`, load the partial artifact (e.g. `asset_manifest`) from `current_cp` and skip the sub-tasks (like scenes) that are already completed. +4. **Continue**: Continue generation from the next successful step, appending to the partial artifact. If a checkpoint exists with status `"awaiting_human"`: 1. Inform the human: "Stage [name] is awaiting your approval" From d9793ed0e4486c5ae50bb38b0eb5ad67103cdf4e Mon Sep 17 00:00:00 2001 From: calesthio Date: Tue, 23 Jun 2026 12:12:32 -0700 Subject: [PATCH 2/2] docs: clarify partial checkpoint validation --- skills/meta/checkpoint-protocol.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/skills/meta/checkpoint-protocol.md b/skills/meta/checkpoint-protocol.md index 21e2bdb8..700be44f 100644 --- a/skills/meta/checkpoint-protocol.md +++ b/skills/meta/checkpoint-protocol.md @@ -56,15 +56,24 @@ The checkpoint utility will: Long-running stages (like `assets` or `compose` loops) can fail midway due to API errors, rate limits, or session interruptions. To allow resuming from the exact point of failure (e.g., Scene 4): -1. **Write partial progress**: Every time you successfully generate a significant item (e.g., one scene's assets, one clip), write an `in_progress` checkpoint. Because `status` is `"in_progress"`, validation will not fail even if the canonical artifact is incomplete. +1. **Write partial progress**: Every time you successfully generate a significant item (e.g., one scene's assets, one clip), write an `in_progress` checkpoint. + + `in_progress` checkpoints may omit the stage's canonical artifact, but any artifact stored under a known artifact name is still schema-validated. If the partial data is not yet a valid canonical artifact, store it under `metadata.partial_progress` instead of `artifacts`. ```python write_checkpoint( pipeline_dir, project_name, stage="assets", status="in_progress", - artifacts={"asset_manifest": partial_manifest_dict} + artifacts={}, # no incomplete canonical artifact yet + metadata={ + "partial_progress": { + "asset_manifest_draft": partial_manifest_dict, + "completed_scene_ids": completed_scene_ids, + } + }, ) ``` + If the partial artifact already satisfies its schema (for example, an `asset_manifest` with `version: "1.0"` and valid `assets[]` entries), it may be stored in `artifacts` directly. 2. **Resume from partial progress**: When starting a stage, ALWAYS check if an `in_progress` checkpoint exists for it. See Step 7 (Resume Protocol) for how to handle it. ### Step 5: Human Approval (If Required) @@ -127,7 +136,7 @@ If `next_stage` is not the first stage: current_cp = read_checkpoint(pipeline_dir, project_name, next_stage) ``` If `current_cp` exists and its status is `"in_progress"`, inform the human you are resuming from the middle of the stage. -3. **Load artifacts**: Load prior artifacts from checkpoints for context. If resuming from `"in_progress"`, load the partial artifact (e.g. `asset_manifest`) from `current_cp` and skip the sub-tasks (like scenes) that are already completed. +3. **Load artifacts**: Load prior artifacts from checkpoints for context. If resuming from `"in_progress"`, first load any schema-valid partial artifact from `current_cp["artifacts"]`. If the partial data is stored in `current_cp["metadata"]["partial_progress"]`, use that draft data and its completion markers (such as `completed_scene_ids`) to skip sub-tasks that are already done. 4. **Continue**: Continue generation from the next successful step, appending to the partial artifact. If a checkpoint exists with status `"awaiting_human"`: