skills/langfuse__skills
1
0
Fork 0
mirror of https://github.com/langfuse/skills.git synced 2026-09-14 20:00:05 +08:00
Code Issues Packages Projects Releases Wiki Activity

docs(langfuse): enhance migration guidance for trace and dataset-item evaluators

Browse Source
This commit is contained in:
Marlies Mayerhofer
2026-07-23 13:12:17 +02:00
parent e569195a41
commit d226810635
2 changed files with 21 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/langfuse/SKILL.md
+2 -1
View File
@@ -36,7 +36,8 @@ Follow these principles for ALL Langfuse work:
- creating a prompt or changing any part of an existing prompt, including small edits and debugging/tuning: references/prompt-engineering.md
- capturing user feedback (thumbs, ratings, implicit signals) as scores on traces: references/user-feedback.md
- further tips on using the Langfuse CLI: references/cli.md
- upgrading legacy trace-level evaluators and their application instrumentation to observation-level evaluators: references/trace-evaluator-upgrade.md
- upgrading legacy trace-level evaluators and their application instrumentation to observation-level evaluators: references/trace-evaluator-upgrade.md. Use the [trace-level evaluator upgrade guide](https://langfuse.com/faq/all/llm-as-a-judge-migration) as the primary reference when the migration path is unclear.
- upgrading legacy dataset-item evaluators to experiment evaluators: follow the dataset-item evaluator migration section in references/trace-evaluator-upgrade.md
- preparing an application and Langfuse project for the v4 platform migration: references/v4-project-migration.md
- judge calibration (LLM-as-a-Judge reliability, simple accuracy checks, advanced split-based validation, confusion matrices, and metric ingestion): references/judge-calibration.md
- systematic error analysis — reading traces, building failure taxonomy, deciding what to fix: references/error-analysis.md
skills/langfuse/references/trace-evaluator-upgrade.md
+19 -3
View File
@@ -14,9 +14,10 @@ In the v4 data model, a trace groups observations sharing a trace ID; it is not
## Sources of truth
Fetch the applicable pages in full before editing:
Fetch the [trace-level evaluator upgrade guide](https://langfuse.com/faq/all/llm-as-a-judge-migration) in full before planning or editing. Treat it as the primary reference and fetch it again whenever the API shape, instrumentation guidance, or migration semantics are unclear. Do not infer those details from memory.
Fetch the other applicable pages in full before editing:
- [Trace-level evaluator upgrade guide](https://langfuse.com/faq/all/llm-as-a-judge-migration)
- [Langfuse v4 overview](https://langfuse.com/docs/v4)
- [Observation evaluator context](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge#observation-evaluator-context)
- [Python v3 to v4](https://langfuse.com/docs/observability/sdk/upgrade-path/python-v3-to-v4)
@@ -31,10 +32,25 @@ Discover the current unstable API schema before use. Use unstable evaluation-rul
- Page through every evaluation rule, then fetch each trace-target rule and its referenced evaluator. If a bulk page fails to parse, retry with `limit=1` to isolate unreadable entries and report them as blockers.
- Record status, block reason, filters, variable mappings and JSONPaths, sampling, delay, time scope, evaluator definition, and score name.
- Migrate only trace-target rules. Do not include observation, experiment, event, or dataset rules in this workflow.
- Migrate only trace-target rules in this section. Dataset-target rules follow the short migration below; do not include observation, experiment, or event rules in this workflow.
- Classify trace rules as active, inactive, or blocked/paused before asking which ones to retain. Offer to delete blocked rules the user no longer needs. Require approval for deletion; if the unstable API cannot mutate legacy trace rules, give the exact UI action instead.
- Show the complete inventory and get one consolidated decision about retained and deleted rules. Do not let discarded rules drive code changes.
## Dataset-item evaluator migration
Use the unstable Evaluation Rules API to read the legacy `dataset` rule, then
create an `experiment` rule with the same evaluator, score name, sampling, and
mappings. Translate the dataset filter from `datasetId` to
`experimentDatasetId`; map `dataset_item.expected_output` and
`dataset_item.metadata` to `experimentItemExpectedOutput` and
`experimentItemMetadata`.
Keep the legacy rule active while the project still uses low-level dataset-run
APIs. Disable it once the project emits experiment context; leaving both copies
active can create duplicate scores for Experiment Runner runs. Review mappings
that use `dataset_item.input` or named observations, since those are not always
one-to-one.
## 2. Confirm v4 instrumentation
The SDK upgrade should already have happened. Inspect the repository and a representative newly ingested trace to confirm the resolved SDK or OpenTelemetry path follows the applicable guide under [SDK upgrade paths](https://langfuse.com/docs/observability/sdk/upgrade-path); do not infer readiness from a dependency declaration alone.