civitai__civitai/docs/workflow-metadata-refactor.md

Workflow Metadata Refactor

Goal

Establish clean, scalable metadata architecture for generation workflows that handles:

• Single-step workflows (standard generation)
• Multi-step workflows (batch upscale: one form → multiple steps)
• Chained workflows (txt2img → face-fix: multiple form inputs → one workflow)

Architecture

Two-Layer Model

workflow.metadata = the generation form input snapshot (workflow-level replay)

• What the user submitted: params + resources + flags
• Used for workflow-level replay ("redo this entire generation")
• Shared across all steps — same regardless of how many steps the workflow has

step.metadata.params/resources = the original generation (step-level remix)

• Always resolved: for enhancements, this is the source generation (what you'd remix to); for standard generations, this is the step's own params
• step.metadata.workflow = what action this step performed (e.g., img2img:upscale). Undefined for standard generations.
• Clients never need to branch or check for source — metadata.params is always the answer

Data model (what the orchestrator API stores)

Standard generation (new writes):

workflow.metadata = {
  params: { prompt: "a cat", steps: 30, workflow: "image:create", ... },
  resources: [{ id: 123, strength: 1 }],
  remixOfId?: 42,
  isPrivateGeneration?: true,
}
workflow.steps[0].metadata = {}
// Step metadata is empty — params/resources live only on workflow.metadata.
// Per-image feedback goes here: { images: { "img-id": { feedback: "liked" } } }

Standard generation (legacy — step has params):

workflow.metadata = undefined | { params, resources }  // may or may not exist
workflow.steps[0].metadata = {
  params: { prompt: "a cat", steps: 30, ... },
  resources: [{ id: 123, strength: 1 }],
  remixOfId?: 42,
  isPrivateGeneration?: true,
  images?: { "img-id": { feedback: "liked" } }
}

Single-image upscale:

workflow.metadata = {
  params: { upscaler: "4x-ultrasharp", creativity: 0.5, workflow: "img2img:upscale", ... },
  resources: [],
}
workflow.steps[0].metadata = {
  params: { upscaler: "4x-ultrasharp", ... },   // step's own action
  resources: [],
  source: {                                       // original generation (raw, stored)
    params: { prompt: "a cat", steps: 30, ... },
    resources: [{ id: 123, strength: 1 }],
    remixOfId?: 42,
  }
}

Batch upscale (4 images):

workflow.metadata = {
  params: { upscaler: "4x-ultrasharp", creativity: 0.5, workflow: "img2img:upscale", ... },
  resources: [],
}
workflow.steps[0].metadata = {
  ...stepOwnParams,
  source: { params: { prompt: "a cat" }, resources: [{ id: 1 }] }
}
workflow.steps[1].metadata = {
  ...stepOwnParams,
  source: { params: { prompt: "a dog" }, resources: [{ id: 2 }] }
}
// ...etc for each image

Chained workflow (txt2img → face-fix):

workflow.metadata = {
  params: { prompt: "a portrait", steps: 30, workflow: "image:create", ... },
  resources: [{ id: 123, strength: 1 }],
}
workflow.steps[0] = {
  $type: "textToImage",
  name: "$0",
  metadata: { params: {...}, resources: [...] }
}
workflow.steps[1] = {
  $type: "comfy",
  name: "$1",
  input: { image: { $ref: "$0", path: "output.images[0].url" } },
  metadata: {
    params: { faceFixStrength: 0.8, workflow: "img2img:face-fix", ... },
    resources: [],
    source: {
      params: { prompt: "a portrait", steps: 30, ... },
      resources: [{ id: 123, strength: 1 }],
    }
  }
}

Key principles

1. workflow.metadata = what the user submitted to the form. Single source of truth for standard generations. Used for workflow-level replay.
2. step.metadata = step-specific data only. Empty for new standard generation writes. Enhancement steps have their own params/resources + source.
3. step.metadata.source = the original generation this enhancement acts upon (raw stored format). Only present on enhancement steps.
4. step.metadata.params/resources = optional. Present on enhancement steps and legacy data. Absent on new standard gen writes.
5. metadata.workflow = what action this step performed (badge display). Undefined for standard generations.
6. Source always points to the original generation, not intermediate enhancements.
7. isPrivateGeneration and remixOfId live on workflow.metadata, not per-step.

Read path normalization

formatStep resolves stored metadata into a consistent normalized shape. It accepts workflowMetadata from the parent workflow for dimension/seed resolution when steps have no params.

New format — enhancement (source field on step):

// step.metadata.source has original, step.metadata root has step's own action
normalized.metadata.params     = resolve(source.params)       // original generation
normalized.metadata.resources  = enrich(source.resources)
normalized.metadata.workflow   = step.metadata.params.workflow // what this step did

Legacy (transformations[]):

// Root params/resources are the original generation. Last transformation = step's action.
normalized.metadata.params     = resolve(step.metadata.params)    // original generation
normalized.metadata.resources  = enrich(step.metadata.resources)
normalized.metadata.workflow   = lastTransformation.workflow       // what this step did

Legacy standard (step has params, no source/transformations):

normalized.metadata.params     = resolve(step.metadata.params)    // the generation itself
normalized.metadata.resources  = enrich(step.metadata.resources)
normalized.metadata.workflow   = undefined                         // standard generation

New standard (step has no params — data on workflow.metadata):

// Step metadata stays empty (no params/resources fabricated)
// Dimension/seed resolution uses workflow.metadata.params as fallback
normalized.metadata             = {}   // or just { images: { ... } }
normalized.metadata.workflow    = undefined

formatGenerationResponse2 also enriches workflow.metadata.resources alongside step resources, building a typed NormalizedWorkflowMetadata on the response.

Client consumption — data class model

Data flows through a parent chain: WorkflowData → StepData → BlobData. Each class resolves metadata with automatic fallback (step → workflow). Components receive BlobData and access everything via getters:

image.step.params       // step.metadata.params ?? workflow.metadata.params
image.step.resources    // step.metadata.resources ?? workflow.metadata.resources
image.workflow          // parent WorkflowData
image.step              // parent StepData

Key getters on StepData:

• succeededImages — images with status === 'succeeded' && !blockedReason && !hidden
• displayImages — images suitable for display (not hidden, not hard-blocked, upgradeable included)
• completedCount / processingCount / blockedCount / blockedReasons

Key getters on WorkflowData:

• Same aggregate getters across all steps (completedCount, succeededImages, etc.)
• params / resources / remixOfId — resolved from workflow.metadata

Usage patterns:

• Workflow replay (QueueItem "redo"): reads request.params (workflow-level)
• Per-image remix: reads image.step.params / image.step.resources (automatic fallback)
• Badge display: step.metadata.workflow (e.g., "img2img:upscale")
• EXIF/Posting: getStepMeta(image.step) — reads resolved params/resources directly
• Prompt display: image.step.params.prompt or request.params.prompt
• Image filtering: step.succeededImages, request.displayImages, matchesMarkerTags(image, markerTags)

Current Implementation Status

What's been built

Helper modules (tested, 73 tests passing):

• workflow-metadata.ts — buildStepSource() / resolveStepSource() for writing and reading source on step metadata (removed) — this "Model A" representation (source nested under step.metadata.source, enhancement action at the root) was never wired into production. The live path stores the source generation's params at the step-metadata root with a workflow lineage marker, resolved inline by formatStep() in orchestration-new.service.ts and surfaced to clients as NormalizedStepMetadata.sourceLineage.
• step-ref.ts — $ref helpers: isStepRef, buildStepRef, assignStepNames
• multi-graph.ts — processMultiGraphInputs() for chained workflows
• index.ts — getStepParams() / getStepResources() client helpers with fallback logic

Write path (production):

• buildResolvedSource() puts step's own params at root, original generation in source. Looks up per-image metadata from sourceMetadataMap (or falls back to single sourceMetadata).
• SourceCtx carries sourceMetadata, sourceMetadataMap, and workflow key for source resolution
• StepInput.resolvedSource — pre-computed field that step creators set when they've already computed their source metadata
• createWorkflowStepsFromGraph() returns { steps, workflowMetadata } — workflow metadata is the form input snapshot
• Standard gen steps write empty metadata (additionalMetadata ?? {}), not full params/resources
• isPrivateGeneration and remixOfId live on workflowMetadata, not per-step
• generateFromGraph() reads isPrivateGeneration from workflowMetadata
• sourceMetadataMap supports per-image source for batch upscale
• needsSourceMetadata flag (on workflow config) determines whether to build source context

Read path (production — formatStep):

• Handles 4 formats: transformations[], source field, step-has-params (legacy), no-step-params (new)
• Enhancement steps: resolves source server-side, metadata.params = original generation
• New standard gen: step metadata stays empty, dimension/seed resolution uses workflow params as fallback
• formatGenerationResponse2() enriches workflow.metadata.resources and builds typed NormalizedWorkflowMetadata
• metadata.workflow = what this step did (badge display)
• getResourceRefsFromStep() collects IDs from both step resources AND source resources for enrichment

Client-side (data class model):

• WorkflowData → StepData → BlobData parent chain in src/shared/orchestrator/workflow-data.ts
• WorkflowData constructor handles the full chain: wraps steps in StepData, wraps images in BlobData, wires parent refs
• StepData has image getters: succeededImages, displayImages, completedCount, processingCount, blockedCount, blockedReasons
• WorkflowData has aggregate getters across all steps
• BlobData has step, workflow, params, resources getters via parent chain
• BlobData handles NSFW blocking logic (private gen, site restricted, enable nsfw, can upgrade)
• Components receive image: BlobData as single prop — no separate step/request/workflowMetadata props
• matchesMarkerTags(image, markerTags) used in Feed, Lightbox, and Queue/QueueItem for marker filtering
• transformations and source references removed from all client code
• No standalone new StepData() construction in any component — all instances created by WorkflowData

Backward Compatibility

Historic workflows have everything on step.metadata. The read path handles all four formats:

Format	workflow.metadata	step.metadata	Era
New standard gen	Has params/resources/flags	Empty (or images feedback only)	New writes
New enhancement	Has form input snapshot	Has own params + source with original gen	New writes
Legacy + transformations	Empty	Root = original gen, transformations = enhancements	Existing enhancements
Legacy (no transformations)	Empty	Root = generation params/resources	Existing standard

Multi-Step Workflows

The current branch (feature/multi-step-workflows) supports multi-step workflows for batch upscale:

• createWorkflowStepsFromGraph() returns WorkflowStepTemplate[] (one step per image for batch upscale)
• sourceMetadataMap maps image URLs → source metadata for per-step source tracking
• buildResolvedSource() looks up per-image source from sourceMetadataMap (or falls back to single sourceMetadata)
• The form's appendUpscaleImage() accumulates images, FormFooter collects sourceMetadataMap for multi-image enhancements
• QueueItem supports stepDisplay: 'separate' for multi-step workflows — renders each step with its own label and image grid
• Queue/QueueItem filter displayed images via matchesMarkerTags when marker filters are active

Multi-Graph Submission (Future)

Concept

Submit multiple generation-graphs in a single workflow request. Each graph = one orchestrator step. Steps can reference earlier step outputs via $ref.

Orchestrator $ref support (already exists)

// Step 1: named step
{ $type: 'wdTagging', name: 'tags', input: { mediaUrl: { $ref: '$arguments', path: 'mediaUrl' } } }

// Step 2: references step 1's output
{ $type: 'repeat', input: { for: { $ref: 'videoFrames', path: 'output.frames', as: 'frame' } } }

Helpers (built in step-ref.ts)

type StepRef = { $ref: string; path: string };
function isStepRef(value: unknown): value is StepRef;
function buildStepRef(stepIndex: number, outputPath: string): StepRef;
function assignStepNames(steps: WorkflowStepTemplate[]): WorkflowStepTemplate[];

Key Files

File	Role
src/server/services/orchestrator/orchestration-new.service.ts	Main service — write + read paths
src/server/services/orchestrator/index.ts	Barrel exports, getStepParams / getStepResources helpers
src/server/services/orchestrator/step-ref.ts	$ref helpers (tested)
src/server/services/orchestrator/legacy-metadata-mapper.ts	mapDataToGraphInput() — maps raw metadata to graph format
src/shared/utils/resource.utils.ts	toStepMetadata() — converts graph output to params/resources
src/store/source-metadata.store.ts	Client SourceMetadata type
src/utils/metadata/extract-source-metadata.ts	EXIF extraction for re-uploaded images
src/components/generation_v2/hooks/useGeneratedItemWorkflows.ts	Client source metadata + workflow application
src/components/ImageGeneration/GenerationForm/generation.utils.ts	getStepMeta() — builds EXIF data
src/shared/orchestrator/workflow-data.ts	WorkflowData, StepData, BlobData data classes

Completed: Standalone StepData Construction Eliminated

All components now use the WorkflowData → StepData → BlobData parent chain. No standalone new StepData() construction exists in any component code.

File	Old pattern	New pattern
GeneratedImage.tsx	Separate step + request + image props	Single image: BlobData prop — accesses image.step, image.workflow
GeneratedItemWorkflowMenu.tsx	step + workflowMetadata → new StepData()	image: BlobData — accesses image.workflow.id, image.ecosystemKey
useGeneratedItemWorkflows.ts	ApplyWorkflowOptions with step + workflowMetadata, 4× new StepData()	ApplyWorkflowOptions with image: BlobData — image.params, image.resources
generation.utils.ts	getStepMeta(step?, wfMeta?) → new StepData() internally	getStepMeta(step?) — accepts step-like object with resolved params/resources
ChallengeSubmitModal.tsx	Fabricated { params, resources, metadata: {} } as any	image: BlobData — getStepMeta(image.step)
ImageSelectModal.tsx	Same as Challenge	Same
QueueItem.tsx	Cast (step as StepData)	step: StepData typed directly (from WorkflowData.steps)
Feed.tsx / Lightbox.tsx	{ ...image, step, request } spread	image: BlobData — no spread needed