Sanderhoff-alt a08ec8ddfe fix(embeddings): send each Gemini input as its own Content (#4001) ...

fix(embeddings): send each Gemini input as its own Content

Passing a plain `list[str]` to the Google GenAI SDK's `models.embed_content()`
reaches the API as several Parts of ONE Content, and the multimodal models
(`gemini-embedding-2`+) fuse those into a single vector — the whole batch
collapses to one embedding. Verified against the live Gemini API: three texts
sent as a `list[str]` come back as 1 vector on `gemini-embedding-2` and
`gemini-embedding-2-preview`, and as 3 when each text is its own `Content`.

We worked around the fusion by forcing `batch_size = 1` for those models, which
kept 1:1 alignment at the cost of one request per text. Since #4039 those
requests fan out concurrently through `_encode_batched`, so this was never the
~30s serial stall the original report described — but it still burns one
upstream request per fact, which is what runs into per-minute rate limits.

Wrapping each text in its own `Content` fixes the cause instead of the symptom:
every model returns one vector per input, so the whole batch travels in a single
request and the `batch_size = 1` special case (and the model-name sniffing that
drove it) goes away. Vectors are byte-identical on `gemini-embedding-001`, so
existing embeddings do not need regenerating.

- Wrap each input text in a distinct `Content` in `_embed_batch()`.
- Drop the forced `batch_size = 1` and `_gemini_model_aggregates_inputs()`.
- Add `HINDSIGHT_API_EMBEDDINGS_GEMINI_BATCH_SIZE` (default 100), matching the
  other providers' configurable batch size.

The live `test_gemini_embedding_2_vertexai_one_vector_per_input` test now covers
the batched case directly — all three texts go out in one request, which is
exactly the shape that used to aggregate. A mocked client cannot prove this;
the behaviour lives in the API.

2026-09-07 14:05:51 +02:00

39 KiB

Raw Blame History

View Raw