feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
package core
|
|
|
|
|
|
|
|
|
|
import "time"
|
|
|
|
|
|
|
|
|
|
// QueryEcho echoes the interpreted query parameters back to the client.
|
|
|
|
|
type QueryEcho struct {
|
|
|
|
|
Text string `json:"text"`
|
|
|
|
|
Lang string `json:"lang,omitempty"`
|
2026-05-21 20:45:10 +03:00
|
|
|
Region string `json:"region,omitempty"`
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
EnginesRequested []string `json:"engines_requested"`
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// ResponseMeta carries request-level metadata for observability and debugging.
|
|
|
|
|
type ResponseMeta struct {
|
2026-05-13 01:30:54 +03:00
|
|
|
RequestID string `json:"request_id"`
|
|
|
|
|
RequestedAt string `json:"requested_at"`
|
|
|
|
|
TookMs int64 `json:"took_ms"`
|
|
|
|
|
EnginesResponded []string `json:"engines_responded,omitempty"`
|
|
|
|
|
EnginesFailed []string `json:"engines_failed"`
|
|
|
|
|
EngineErrors []EngineErrorDetail `json:"engine_errors,omitempty"`
|
|
|
|
|
Version string `json:"version"`
|
2026-04-29 02:45:16 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// EngineErrorDetail is a client-facing, sanitized per-engine failure summary.
|
|
|
|
|
type EngineErrorDetail struct {
|
|
|
|
|
Engine string `json:"engine"`
|
|
|
|
|
Error string `json:"error"`
|
|
|
|
|
Message string `json:"message,omitempty"`
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Pagination carries cursor information for client-side loop termination.
|
|
|
|
|
type Pagination struct {
|
|
|
|
|
Page int `json:"page"`
|
|
|
|
|
HasMore bool `json:"has_more"`
|
|
|
|
|
NextStart int `json:"next_start"`
|
|
|
|
|
}
|
|
|
|
|
|
2026-05-13 01:30:54 +03:00
|
|
|
// Envelope is the top-level v2 response wrapper for all search endpoints.
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
type Envelope struct {
|
|
|
|
|
Query QueryEcho `json:"query"`
|
|
|
|
|
Meta ResponseMeta `json:"meta"`
|
|
|
|
|
Results []Result `json:"results"`
|
|
|
|
|
Pagination Pagination `json:"pagination"`
|
|
|
|
|
// Clusters is only populated by /mega/search (see clusters.go).
|
|
|
|
|
Clusters *[]Cluster `json:"clusters,omitempty"`
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Cluster groups results that refer to the same canonical URL across engines.
|
|
|
|
|
// Populated only by /mega/search. Full type defined in clusters.go.
|
|
|
|
|
type Cluster struct {
|
|
|
|
|
ID string `json:"id"`
|
|
|
|
|
CanonicalURL string `json:"canonical_url"`
|
|
|
|
|
Domain string `json:"domain"`
|
|
|
|
|
Title string `json:"title"`
|
|
|
|
|
Occurrences []ClusterOccurrence `json:"occurrences"`
|
|
|
|
|
EnginesCount int `json:"engines_count"`
|
|
|
|
|
BestRank int `json:"best_rank"`
|
|
|
|
|
Score float64 `json:"score"`
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// ClusterOccurrence links one engine result back into the flat results list.
|
|
|
|
|
type ClusterOccurrence struct {
|
|
|
|
|
Engine string `json:"engine"`
|
|
|
|
|
Rank int `json:"rank"`
|
|
|
|
|
ResultID string `json:"result_id"`
|
|
|
|
|
}
|
|
|
|
|
|
2026-05-13 01:30:54 +03:00
|
|
|
// ImageEnvelope is the top-level v2 response wrapper for image search endpoints.
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
type ImageEnvelope struct {
|
2026-04-26 02:40:08 +03:00
|
|
|
Query QueryEcho `json:"query"`
|
|
|
|
|
Meta ResponseMeta `json:"meta"`
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
Results []ImageResult `json:"results"`
|
2026-04-26 02:40:08 +03:00
|
|
|
Pagination Pagination `json:"pagination"`
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
}
|
|
|
|
|
|
2026-05-19 00:32:11 +03:00
|
|
|
const apiVersion = "2.1"
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
|
|
|
|
|
// NewEnvelope builds a fresh Envelope pre-filled with query echo and an open
|
|
|
|
|
// meta block. Call Finalize before serializing.
|
|
|
|
|
func NewEnvelope(q Query, requestID string, startedAt time.Time, engines []string) *Envelope {
|
|
|
|
|
return &Envelope{
|
|
|
|
|
Query: QueryEcho{
|
|
|
|
|
Text: q.Text,
|
|
|
|
|
Lang: q.LangCode,
|
2026-05-21 20:45:10 +03:00
|
|
|
Region: q.Region,
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
EnginesRequested: engines,
|
|
|
|
|
},
|
|
|
|
|
Meta: ResponseMeta{
|
2026-04-26 02:40:08 +03:00
|
|
|
RequestID: requestID,
|
|
|
|
|
RequestedAt: startedAt.UTC().Format(time.RFC3339),
|
|
|
|
|
EnginesFailed: []string{},
|
|
|
|
|
Version: apiVersion,
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
},
|
|
|
|
|
Results: []Result{},
|
|
|
|
|
Pagination: Pagination{},
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// NewImageEnvelope builds a fresh ImageEnvelope.
|
|
|
|
|
func NewImageEnvelope(q Query, requestID string, startedAt time.Time, engines []string) *ImageEnvelope {
|
|
|
|
|
return &ImageEnvelope{
|
|
|
|
|
Query: QueryEcho{
|
|
|
|
|
Text: q.Text,
|
|
|
|
|
Lang: q.LangCode,
|
2026-05-21 20:45:10 +03:00
|
|
|
Region: q.Region,
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
EnginesRequested: engines,
|
|
|
|
|
},
|
|
|
|
|
Meta: ResponseMeta{
|
2026-04-26 02:40:08 +03:00
|
|
|
RequestID: requestID,
|
|
|
|
|
RequestedAt: startedAt.UTC().Format(time.RFC3339),
|
|
|
|
|
EnginesFailed: []string{},
|
|
|
|
|
Version: apiVersion,
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
},
|
|
|
|
|
Results: []ImageResult{},
|
|
|
|
|
Pagination: Pagination{},
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Finalize stamps the elapsed time and computes pagination fields.
|
|
|
|
|
func (e *Envelope) Finalize(startedAt time.Time, q Query) {
|
|
|
|
|
e.Meta.TookMs = time.Since(startedAt).Milliseconds()
|
|
|
|
|
|
|
|
|
|
limit := q.Limit
|
|
|
|
|
if limit <= 0 {
|
|
|
|
|
limit = 25
|
|
|
|
|
}
|
|
|
|
|
page := q.Start/limit + 1
|
|
|
|
|
e.Pagination = Pagination{
|
|
|
|
|
Page: page,
|
2026-05-13 01:30:54 +03:00
|
|
|
HasMore: countNonAdResults(e.Results) >= limit,
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
NextStart: q.Start + limit,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2026-05-13 01:30:54 +03:00
|
|
|
func countNonAdResults(results []Result) int {
|
|
|
|
|
count := 0
|
|
|
|
|
for _, result := range results {
|
|
|
|
|
if result.Type != ResultTypeAd {
|
|
|
|
|
count++
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return count
|
|
|
|
|
}
|
|
|
|
|
|
feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
meta (request_id, took_ms, engines_responded, engines_failed, version),
results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
id, type, display_url, domain, favicon, is_ad, position, engine_meta,
domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-24 05:21:11 +03:00
|
|
|
// Finalize stamps the elapsed time and computes pagination fields.
|
|
|
|
|
func (e *ImageEnvelope) Finalize(startedAt time.Time, q Query) {
|
|
|
|
|
e.Meta.TookMs = time.Since(startedAt).Milliseconds()
|
|
|
|
|
|
|
|
|
|
limit := q.Limit
|
|
|
|
|
if limit <= 0 {
|
|
|
|
|
limit = 25
|
|
|
|
|
}
|
|
|
|
|
page := q.Start/limit + 1
|
|
|
|
|
e.Pagination = Pagination{
|
|
|
|
|
Page: page,
|
|
|
|
|
HasMore: len(e.Results) >= limit,
|
|
|
|
|
NextStart: q.Start + limit,
|
|
|
|
|
}
|
|
|
|
|
}
|