skills: document single-parse JSON extraction pattern for materialized views (#20)

* skills: document single-parse JSON extraction pattern for materialized views
This commit is contained in:
Paco González López
2026-07-29 12:59:32 +02:00
committed by GitHub
parent 960fb75167
commit 2c075dd2d3
2 changed files with 48 additions and 1 deletions
@@ -101,7 +101,7 @@ Apply only when runtime thresholds are exceeded, based on `pipe_stats_rt` and `E
### Expensive JSON extraction at query time
- **When**: p95 > 3s, or CPU > 70%.
- Fix: Extract JSON fields into typed columns at ingestion time.
- Fix: Extract JSON fields into typed columns at ingestion time. If many fields are extracted from the same JSON payload (per-field `visitParamExtract*`/`JSONExtract*` calls), see the single-parse JSON pattern in `materialized-files.md` — it applies here too, but matters most in materialized views since the parse cost compounds over every ingested row.
### Large IN lists
- **When**: p95 > 3s, or query planning time is high.
@@ -35,6 +35,53 @@ ENGINE_PARTITION_KEY "toYYYYMM(date)"
ENGINE_SORTING_KEY "date, dimension_1, dimension_2"
```
## JSON extraction: parse once, not once per field
- **When to apply**: the query calls `JSONExtractString`/`JSONExtractInt`/`JSONExtractBool`/`JSONExtractFloat`/`simpleJSONExtractString`/`visitParam*` multiple times against the same JSON/string column — one call per field. Each call re-parses the raw JSON from scratch, so N fields means N full parses per row. This is most costly in materialized views, since it runs on every inserted block for the pipe's lifetime.
- **How to apply**: parse the JSON once into a typed `Tuple` with `JSONExtract(...)`, then read each field from it with `getSubcolumn`.
Bad (one parse per field):
```
NODE typed_events
SQL >
SELECT
at AS timestamp,
visitParamExtractString(payload, 'field_a') AS field_a,
visitParamExtractInt(payload, 'field_b') AS field_b,
visitParamExtractBool(payload, 'field_c') AS field_c,
simpleJSONExtractString(payload, 'field_d') AS field_d
FROM raw_events
TYPE MATERIALIZED
DATASOURCE typed_events_ds
```
Good (one parse total):
```
NODE typed_events
SQL >
WITH
JSONExtract(payload, 'Tuple(
field_a String,
field_b Int64,
field_c Bool,
field_d String
)') AS payload_json
SELECT
at AS timestamp,
getSubcolumn(payload_json, 'field_a') AS field_a,
getSubcolumn(payload_json, 'field_b') AS field_b,
getSubcolumn(payload_json, 'field_c') AS field_c,
getSubcolumn(payload_json, 'field_d') AS field_d
FROM raw_events
TYPE MATERIALIZED
DATASOURCE typed_events_ds
```
- Missing fields default to their type's default value.
- Reuse a field via a `WITH` alias if multiple derived expressions depend on it.
## Usual gotchas
- Materialized Views work as insert triggers, which means a delete or truncate operation on your original Data Source doesn't affect the related Materialized Views.