mirror of
https://github.com/calesthio/OpenMontage.git
synced 2026-08-05 15:20:40 +08:00
feat: add intra-stage generation checkpoint and resume support (closes #129)
This commit is contained in:
@@ -52,7 +52,22 @@ The checkpoint utility will:
|
||||
- Write the checkpoint JSON to disk
|
||||
- Include timestamp and stage metadata
|
||||
|
||||
### Step 4: Human Approval (If Required)
|
||||
### Step 4: Intra-Stage Checkpointing (Resume Support)
|
||||
|
||||
Long-running stages (like `assets` or `compose` loops) can fail midway due to API errors, rate limits, or session interruptions. To allow resuming from the exact point of failure (e.g., Scene 4):
|
||||
|
||||
1. **Write partial progress**: Every time you successfully generate a significant item (e.g., one scene's assets, one clip), write an `in_progress` checkpoint. Because `status` is `"in_progress"`, validation will not fail even if the canonical artifact is incomplete.
|
||||
```python
|
||||
write_checkpoint(
|
||||
pipeline_dir, project_name,
|
||||
stage="assets",
|
||||
status="in_progress",
|
||||
artifacts={"asset_manifest": partial_manifest_dict}
|
||||
)
|
||||
```
|
||||
2. **Resume from partial progress**: When starting a stage, ALWAYS check if an `in_progress` checkpoint exists for it. See Step 7 (Resume Protocol) for how to handle it.
|
||||
|
||||
### Step 5: Human Approval (If Required)
|
||||
|
||||
When `human_approval_default: true`:
|
||||
|
||||
@@ -87,7 +102,7 @@ When `human_approval_default: true`:
|
||||
- `compose` — Rarely. But human may want to preview.
|
||||
- `publish` — Always. Human must approve before anything goes public.
|
||||
|
||||
### Step 5: Determine Next Stage
|
||||
### Step 6: Determine Next Stage
|
||||
|
||||
After checkpoint is written and approved (if needed):
|
||||
|
||||
@@ -97,7 +112,7 @@ next_stage = get_next_stage(pipeline_dir, project_name)
|
||||
|
||||
This reads all existing checkpoints and returns the next stage that needs to run, or `None` if the pipeline is complete.
|
||||
|
||||
### Step 6: Resume Protocol
|
||||
### Step 7: Resume Protocol
|
||||
|
||||
At the START of any pipeline run (not just after a stage), always check for existing progress:
|
||||
|
||||
@@ -107,8 +122,13 @@ next_stage = get_next_stage(pipeline_dir, project_name)
|
||||
|
||||
If `next_stage` is not the first stage:
|
||||
1. Inform the human: "Found existing progress. Resuming from stage: [next_stage]"
|
||||
2. Load prior artifacts from checkpoints for context
|
||||
3. Continue from that stage
|
||||
2. **Check for partial progress**: Read the checkpoint for `next_stage`:
|
||||
```python
|
||||
current_cp = read_checkpoint(pipeline_dir, project_name, next_stage)
|
||||
```
|
||||
If `current_cp` exists and its status is `"in_progress"`, inform the human you are resuming from the middle of the stage.
|
||||
3. **Load artifacts**: Load prior artifacts from checkpoints for context. If resuming from `"in_progress"`, load the partial artifact (e.g. `asset_manifest`) from `current_cp` and skip the sub-tasks (like scenes) that are already completed.
|
||||
4. **Continue**: Continue generation from the next successful step, appending to the partial artifact.
|
||||
|
||||
If a checkpoint exists with status `"awaiting_human"`:
|
||||
1. Inform the human: "Stage [name] is awaiting your approval"
|
||||
|
||||
Reference in New Issue
Block a user