mirror of
https://github.com/vectorize-io/hindsight.git
synced 2026-09-14 19:31:49 +08:00
feat(llm): add GitHub Copilot subscription provider (#3597)
Adds a `github-copilot` LLM provider backed by the official GitHub Copilot SDK, using the signed-in Copilot entitlement with no `HINDSIGHT_API_LLM_API_KEY`. Verified end-to-end against a live Copilot subscription: retain, consolidation and reflect all run on the provider. Review fixes on top of the original submission: - default to `gpt-5.6-terra`; the submitted `gpt-5.6-sol` is not in the model list the runtime serves, so every call failed and the provider was unusable out of the box - make a rejected request configuration terminal instead of a runtime failure; an unavailable model was invalidating the shared runtime and respawning `copilot --headless` once per attempt (5 spawns at max_retries=3, ~12 at the default of 10) - let a token-authenticated host run without `~/.copilot`, so the documented COPILOT_GITHUB_TOKEN / GH_TOKEN / GITHUB_TOKEN path works in containers and CI - regenerate skills/hindsight-docs (the provider list also feeds the generated faq.md) Co-authored-by: Max Marino <dudujuju828@users.noreply.github.com>
This commit is contained in:
+7
-2
@@ -2,11 +2,11 @@
|
||||
# Copy this file to .env and fill in your values
|
||||
|
||||
# LLM Configuration (Required)
|
||||
# Supported providers: openai, openai-responses, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, zai, atlas, volcano
|
||||
# Supported providers: openai, openai-responses, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, zai, atlas, volcano, openai-codex, claude-code, github-copilot
|
||||
HINDSIGHT_API_LLM_PROVIDER=openai
|
||||
HINDSIGHT_API_LLM_API_KEY=your-api-key-here
|
||||
HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
|
||||
HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
|
||||
# HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
|
||||
# Reasoning effort for providers/models that support it. Examples: none, low, medium, high, xhigh.
|
||||
# Set it and the value is sent as given, whatever the model is called — use `none` to stop a
|
||||
# self-hosted reasoning model (vLLM, Ollama, llama.cpp, TGI) emitting thinking blocks. Unset,
|
||||
@@ -74,6 +74,11 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
|
||||
# HINDSIGHT_API_LLM_API_KEY=your-anthropic-api-key
|
||||
# HINDSIGHT_API_LLM_MODEL=claude-sonnet-4-20250514
|
||||
|
||||
# Example: GitHub Copilot subscription via the official Copilot SDK
|
||||
# Sign in with Copilot CLI first; no HINDSIGHT_API_LLM_API_KEY is needed.
|
||||
# HINDSIGHT_API_LLM_PROVIDER=github-copilot
|
||||
# HINDSIGHT_API_LLM_MODEL=gpt-5.6-terra
|
||||
|
||||
# Example: Google Vertex AI configuration
|
||||
# HINDSIGHT_API_LLM_PROVIDER=vertexai
|
||||
# HINDSIGHT_API_LLM_MODEL=google/gemini-2.0-flash-001
|
||||
|
||||
@@ -89,7 +89,7 @@ cd hindsight-control-plane && npm run dev
|
||||
|
||||
### Core Engine (hindsight-api-slim/hindsight_api/engine/)
|
||||
- `memory_engine.py`: Main orchestrator for retain/recall/reflect operations
|
||||
- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code
|
||||
- `llm_wrapper.py`: LLM abstraction supporting OpenAI, Anthropic, Gemini, VertexAI, Groq, MiniMax, Ollama, LM Studio, LiteLLM, Claude Code, GitHub Copilot
|
||||
- `embeddings.py`: Embedding generation (local sentence-transformers or TEI)
|
||||
- `cross_encoder.py`: Reranking (local or TEI)
|
||||
- `entity_resolver.py`: Entity extraction and normalization
|
||||
|
||||
@@ -77,7 +77,7 @@ docker run -it --pull always --name hindsight --restart unless-stopped -p 8888:8
|
||||
>API: http://localhost:8888
|
||||
>UI: http://localhost:9999
|
||||
|
||||
Hindsight works with **25+ LLM providers** via `HINDSIGHT_API_LLM_PROVIDER` — hosted (`openai`, `anthropic`, `gemini`, `groq`, `bedrock`, `vertexai`, `minimax`, `deepseek`, `atlas`, …), fully local (`ollama`, `lmstudio`, `llamacpp`), any OpenAI-compatible endpoint, and gateways (`litellm`, `litellmrouter`) that reach the rest. Existing subscriptions work too: `openai-codex` (ChatGPT Plus/Pro) and `claude-code` (Claude Pro/Max) need no API key. See [supported models](https://hindsight.vectorize.io/developer/models).
|
||||
Hindsight works with **25+ LLM providers** via `HINDSIGHT_API_LLM_PROVIDER` — hosted (`openai`, `anthropic`, `gemini`, `groq`, `bedrock`, `vertexai`, `minimax`, `deepseek`, `atlas`, …), fully local (`ollama`, `lmstudio`, `llamacpp`), any OpenAI-compatible endpoint, and gateways (`litellm`, `litellmrouter`) that reach the rest. Existing subscriptions work too: `openai-codex` (ChatGPT Plus/Pro), `claude-code` (Claude Pro/Max) and `github-copilot` (GitHub Copilot) need no API key. See [supported models](https://hindsight.vectorize.io/developer/models).
|
||||
|
||||
#### Docker (external PostgreSQL)
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@ class HindsightEmbedded:
|
||||
Args:
|
||||
profile: Profile name for data isolation (default: "default")
|
||||
llm_provider: LLM provider ("groq", "openai", "ollama", "gemini", "anthropic",
|
||||
"lmstudio"). Omit to inherit; the server default is "openai".
|
||||
"lmstudio", "github-copilot"). Omit to inherit; the server default is "openai".
|
||||
llm_api_key: API key for the LLM provider. Omit to inherit; pass "" to
|
||||
explicitly run without a key (local services that need no auth).
|
||||
llm_model: Model name to use. Omit to inherit; the server picks a default
|
||||
|
||||
@@ -80,7 +80,7 @@ Configure via environment variables:
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_API_DATABASE_URL` | PostgreSQL connection string | `pg0` (embedded) |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | LLM provider, including `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio`, and `github-copilot` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | API key for LLM provider | - |
|
||||
| `HINDSIGHT_API_LLM_MODEL` | Model name | `gpt-4o-mini` |
|
||||
| `HINDSIGHT_API_HOST` | Server bind address | `0.0.0.0` |
|
||||
|
||||
@@ -911,6 +911,7 @@ PROVIDER_DEFAULT_MODELS = {
|
||||
"vertexai": "google/gemini-3.1-flash-lite",
|
||||
"openai-codex": "gpt-5.4-mini",
|
||||
"claude-code": "claude-sonnet-4-5-20250929",
|
||||
"github-copilot": "gpt-5.6-terra",
|
||||
"mock": "mock-model",
|
||||
"none": "none",
|
||||
"litellm": "gpt-4o-mini",
|
||||
|
||||
@@ -264,6 +264,7 @@ _PROVIDERS_WITHOUT_API_KEY = frozenset(
|
||||
"llamacpp",
|
||||
"openai-codex",
|
||||
"claude-code",
|
||||
"github-copilot",
|
||||
"mock",
|
||||
"none",
|
||||
"vertexai",
|
||||
@@ -374,6 +375,7 @@ def create_llm_provider(
|
||||
CodexLLM,
|
||||
FireworksLLM,
|
||||
GeminiLLM,
|
||||
GitHubCopilotLLM,
|
||||
LiteLLMLLM,
|
||||
LiteLLMRouterLLM,
|
||||
LlamaCppLLM,
|
||||
@@ -410,6 +412,16 @@ def create_llm_provider(
|
||||
reasoning_effort=reasoning_effort,
|
||||
)
|
||||
|
||||
elif provider_lower == "github-copilot":
|
||||
return GitHubCopilotLLM(
|
||||
provider=provider,
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
model=model,
|
||||
reasoning_effort=reasoning_effort,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
elif provider_lower == "mock":
|
||||
return MockLLM(
|
||||
provider=provider,
|
||||
@@ -774,6 +786,7 @@ class LLMProvider:
|
||||
"vertexai",
|
||||
"openai-codex",
|
||||
"claude-code",
|
||||
"github-copilot",
|
||||
"mock",
|
||||
"none",
|
||||
"minimax",
|
||||
@@ -1476,9 +1489,7 @@ class LLMProvider:
|
||||
if not api_key and not requires_api_key(provider):
|
||||
pass # Provider handles its own auth
|
||||
elif not api_key:
|
||||
raise ValueError(
|
||||
f"{ENV_LLM_API_KEY} environment variable is required (unless using openai-codex, claude-code, or litellm)"
|
||||
)
|
||||
raise ValueError(f"{ENV_LLM_API_KEY} environment variable is required for provider '{provider}'")
|
||||
|
||||
base_url = os.getenv(ENV_LLM_BASE_URL, "")
|
||||
model = os.getenv(ENV_LLM_MODEL) or _get_default_model_for_provider(provider)
|
||||
|
||||
@@ -9,6 +9,7 @@ from .claude_code_llm import ClaudeCodeLLM
|
||||
from .codex_llm import CodexLLM
|
||||
from .fireworks_llm import FireworksLLM
|
||||
from .gemini_llm import GeminiLLM
|
||||
from .github_copilot_llm import GitHubCopilotLLM
|
||||
from .litellm_llm import LiteLLMLLM
|
||||
from .litellm_router_llm import LiteLLMRouterLLM
|
||||
from .llamacpp_llm import LlamaCppLLM
|
||||
@@ -23,6 +24,7 @@ __all__ = [
|
||||
"CodexLLM",
|
||||
"FireworksLLM",
|
||||
"GeminiLLM",
|
||||
"GitHubCopilotLLM",
|
||||
"LlamaCppLLM",
|
||||
"LiteLLMLLM",
|
||||
"LiteLLMRouterLLM",
|
||||
|
||||
@@ -0,0 +1,869 @@
|
||||
"""
|
||||
GitHub Copilot LLM provider using the official Copilot SDK.
|
||||
|
||||
This provider uses the GitHub identity already authenticated by Copilot CLI, or
|
||||
one of the SDK-supported GitHub token environment variables. It does not use
|
||||
``HINDSIGHT_API_LLM_API_KEY``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Awaitable, Mapping
|
||||
from contextlib import AbstractAsyncContextManager, nullcontext
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Callable
|
||||
|
||||
from pydantic import ValidationError
|
||||
|
||||
from hindsight_api.engine.llm_interface import LLM_TOOL_CHOICE_AUTO, LLMInterface, LLMToolChoice, LLMToolChoiceMode
|
||||
from hindsight_api.engine.llm_trace import LLMResponseUsage, stash_response_usage
|
||||
from hindsight_api.engine.response_models import LLMToolCall, LLMToolCallResult, TokenUsage
|
||||
from hindsight_api.metrics import get_metrics_collector
|
||||
from hindsight_api.worker.stage import set_stage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from copilot import CopilotClient
|
||||
from copilot.session_events import AssistantMessageData, SessionEvent
|
||||
from copilot.tools import Tool
|
||||
|
||||
_STRUCTURED_TOOL_NAME = "structured_response"
|
||||
_DEFAULT_TIMEOUT_SECONDS = 120.0
|
||||
_RUNTIME_CLEANUP_TIMEOUT_SECONDS = 5.0
|
||||
_SESSION_CLEANUP_TIMEOUT_SECONDS = 2.0
|
||||
_TEMPLATE_OPENAI_BASE_URL = "https://api.openai.com/v1"
|
||||
_SUPPORTED_REASONING_EFFORTS = frozenset({"low", "medium", "high", "xhigh", "max"})
|
||||
|
||||
|
||||
@dataclass
|
||||
class _PromptParts:
|
||||
system_prompt: str
|
||||
user_prompt: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class _TurnCapture:
|
||||
assistant_messages: list[AssistantMessageData] = field(default_factory=list)
|
||||
input_tokens: int = 0
|
||||
output_tokens: int = 0
|
||||
cached_tokens: int = 0
|
||||
thoughts_tokens: int = 0
|
||||
finish_reason: str | None = None
|
||||
|
||||
def handle_event(self, event: SessionEvent) -> None:
|
||||
from copilot.session_events import AssistantMessageData, AssistantUsageData
|
||||
|
||||
data = event.data
|
||||
if isinstance(data, AssistantMessageData):
|
||||
self.assistant_messages.append(data)
|
||||
elif isinstance(data, AssistantUsageData):
|
||||
self.input_tokens += data.input_tokens or 0
|
||||
self.output_tokens += data.output_tokens or 0
|
||||
self.cached_tokens += data.cache_read_tokens or 0
|
||||
self.thoughts_tokens += data.reasoning_tokens or 0
|
||||
self.finish_reason = data.finish_reason or self.finish_reason
|
||||
|
||||
def token_usage(self) -> TokenUsage:
|
||||
return TokenUsage(
|
||||
input_tokens=self.input_tokens,
|
||||
output_tokens=self.output_tokens,
|
||||
total_tokens=self.input_tokens + self.output_tokens,
|
||||
cached_tokens=self.cached_tokens,
|
||||
thoughts_tokens=self.thoughts_tokens,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _InvocationResult:
|
||||
content: str
|
||||
tool_calls: list[LLMToolCall]
|
||||
usage: TokenUsage
|
||||
finish_reason: str | None
|
||||
|
||||
|
||||
_TOKEN_ENV_VARS = ("COPILOT_GITHUB_TOKEN", "GH_TOKEN", "GITHUB_TOKEN")
|
||||
|
||||
|
||||
def _has_token_credentials() -> bool:
|
||||
"""Whether the SDK can authenticate without a signed-in Copilot CLI account."""
|
||||
return any(os.environ.get(name) for name in _TOKEN_ENV_VARS)
|
||||
|
||||
|
||||
def _sync_copilot_auth_metadata(target_home: Path, source_home: Path | None = None) -> None:
|
||||
"""Copy only the account selector into the isolated runtime home.
|
||||
|
||||
The normal Copilot home also contains user hooks, plugins, MCP servers,
|
||||
skills, and session state. Pointing Hindsight's internal runtime there made
|
||||
every internal LLM request run the Hindsight hooks again, creating an
|
||||
explosive recursion loop. ``config.json`` carries the signed-in account
|
||||
selection, so copying only that file preserves the account choice without
|
||||
importing executable config. The credential itself is never in this file:
|
||||
the runtime resolves it from the system keychain or from a `gh` CLI login.
|
||||
"""
|
||||
target_home.mkdir(parents=True, exist_ok=True)
|
||||
source = (source_home or Path.home() / ".copilot") / "config.json"
|
||||
if not source.is_file():
|
||||
if _has_token_credentials():
|
||||
# A headless deployment (container, CI) authenticates from the token
|
||||
# environment and never runs Copilot CLI, so there is no signed-in
|
||||
# account on disk to carry over. An empty home is correct there.
|
||||
logger.info(
|
||||
"No Copilot account metadata at %s; authenticating from the token environment instead",
|
||||
source,
|
||||
)
|
||||
return
|
||||
raise RuntimeError(
|
||||
f"GitHub Copilot account metadata was not found at {source}. "
|
||||
"Start Copilot CLI and sign in, or set COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN."
|
||||
)
|
||||
shutil.copy2(source, target_home / "config.json")
|
||||
|
||||
|
||||
class _SharedCopilotRuntime:
|
||||
"""One Copilot runtime shared by every Hindsight LLM lane in this process."""
|
||||
|
||||
def __init__(self, runtime_url: str) -> None:
|
||||
self.runtime_url = runtime_url
|
||||
self.ref_count = 0
|
||||
self._started = False
|
||||
self._lifecycle_lock = asyncio.Lock()
|
||||
self._copilot_home = None if runtime_url else Path(tempfile.mkdtemp(prefix="hindsight-github-copilot-"))
|
||||
self.client = self._new_client()
|
||||
|
||||
def _new_client(self) -> CopilotClient:
|
||||
from copilot import CopilotClient, RuntimeConnection
|
||||
|
||||
connection = RuntimeConnection.for_uri(self.runtime_url) if self.runtime_url else None
|
||||
base_directory = None
|
||||
if self._copilot_home is not None:
|
||||
_sync_copilot_auth_metadata(self._copilot_home)
|
||||
base_directory = str(self._copilot_home)
|
||||
return CopilotClient(
|
||||
connection=connection,
|
||||
working_directory=tempfile.gettempdir(),
|
||||
base_directory=base_directory,
|
||||
use_logged_in_user=True,
|
||||
session_idle_timeout_seconds=300,
|
||||
mode="copilot-cli",
|
||||
)
|
||||
|
||||
async def ensure_started(self) -> None:
|
||||
if self._started:
|
||||
return
|
||||
async with self._lifecycle_lock:
|
||||
if self._started:
|
||||
return
|
||||
client = self.client
|
||||
try:
|
||||
await client.start()
|
||||
auth = await client.get_auth_status()
|
||||
if not auth.isAuthenticated:
|
||||
raise RuntimeError(
|
||||
"GitHub Copilot is not authenticated. Start Copilot CLI and sign in, "
|
||||
"or set COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN."
|
||||
)
|
||||
self._started = True
|
||||
except BaseException:
|
||||
self.client = self._new_client()
|
||||
self._started = False
|
||||
await self._force_stop_client(client)
|
||||
raise
|
||||
|
||||
async def get_client(self) -> CopilotClient:
|
||||
await self.ensure_started()
|
||||
return self.client
|
||||
|
||||
async def invalidate(self, expected_client: CopilotClient, reason: str) -> None:
|
||||
"""Replace a dead or wedged runtime without disrupting a newer generation."""
|
||||
async with self._lifecycle_lock:
|
||||
if self.client is not expected_client:
|
||||
return
|
||||
logger.warning("Resetting shared GitHub Copilot runtime: %s", reason)
|
||||
self.client = self._new_client()
|
||||
self._started = False
|
||||
await self._force_stop_client(expected_client)
|
||||
|
||||
@staticmethod
|
||||
async def _force_stop_client(client: CopilotClient) -> None:
|
||||
try:
|
||||
await asyncio.wait_for(client.force_stop(), timeout=_RUNTIME_CLEANUP_TIMEOUT_SECONDS)
|
||||
except Exception:
|
||||
logger.warning("Failed to force-stop GitHub Copilot runtime", exc_info=True)
|
||||
|
||||
async def stop(self) -> None:
|
||||
async with self._lifecycle_lock:
|
||||
if self._started:
|
||||
client = self.client
|
||||
self._started = False
|
||||
try:
|
||||
await asyncio.wait_for(client.stop(), timeout=_RUNTIME_CLEANUP_TIMEOUT_SECONDS)
|
||||
except Exception:
|
||||
logger.warning("GitHub Copilot runtime cleanup failed; forcing shutdown", exc_info=True)
|
||||
await self._force_stop_client(client)
|
||||
if self._copilot_home is not None:
|
||||
try:
|
||||
shutil.rmtree(self._copilot_home)
|
||||
except OSError:
|
||||
logger.warning(
|
||||
"Failed to remove isolated Copilot home %s",
|
||||
self._copilot_home,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
|
||||
_runtime_registry_lock = threading.Lock()
|
||||
_runtime_registry: dict[str, _SharedCopilotRuntime] = {}
|
||||
|
||||
|
||||
def _acquire_runtime(runtime_url: str) -> _SharedCopilotRuntime:
|
||||
key = runtime_url.strip().rstrip("/")
|
||||
with _runtime_registry_lock:
|
||||
runtime = _runtime_registry.get(key)
|
||||
if runtime is None:
|
||||
runtime = _SharedCopilotRuntime(key)
|
||||
_runtime_registry[key] = runtime
|
||||
runtime.ref_count += 1
|
||||
return runtime
|
||||
|
||||
|
||||
def _normalize_runtime_url(base_url: str) -> str:
|
||||
value = base_url.strip().rstrip("/")
|
||||
# Older generated .env files actively set the OpenAI default URL. Treat
|
||||
# that one known template value as unset so switching only provider/model
|
||||
# works; every other non-empty value remains explicit and is validated by
|
||||
# the Copilot SDK as a headless runtime URI.
|
||||
if value == _TEMPLATE_OPENAI_BASE_URL.rstrip("/"):
|
||||
logger.info("Ignoring the OpenAI template base URL for the github-copilot provider")
|
||||
return ""
|
||||
return value
|
||||
|
||||
|
||||
async def _release_runtime(runtime: _SharedCopilotRuntime) -> None:
|
||||
should_stop = False
|
||||
with _runtime_registry_lock:
|
||||
if runtime.ref_count > 0:
|
||||
runtime.ref_count -= 1
|
||||
if runtime.ref_count == 0 and _runtime_registry.get(runtime.runtime_url) is runtime:
|
||||
del _runtime_registry[runtime.runtime_url]
|
||||
should_stop = True
|
||||
if should_stop:
|
||||
await runtime.stop()
|
||||
|
||||
|
||||
def _build_prompt(messages: list[dict[str, Any]]) -> _PromptParts:
|
||||
system_parts: list[str] = []
|
||||
conversation: list[dict[str, Any]] = []
|
||||
|
||||
for message in messages:
|
||||
role = str(message.get("role", "user"))
|
||||
if role == "system":
|
||||
content = message.get("content")
|
||||
if isinstance(content, str) and content:
|
||||
system_parts.append(content)
|
||||
continue
|
||||
conversation.append(message)
|
||||
|
||||
system_prompt = "\n\n".join(system_parts).strip()
|
||||
if not system_prompt:
|
||||
system_prompt = "Act as a stateless language-model backend and follow the supplied conversation."
|
||||
|
||||
if (
|
||||
len(conversation) == 1
|
||||
and conversation[0].get("role") == "user"
|
||||
and isinstance(conversation[0].get("content"), str)
|
||||
and set(conversation[0]).issubset({"role", "content"})
|
||||
):
|
||||
user_prompt = conversation[0]["content"]
|
||||
else:
|
||||
serialized = json.dumps(conversation, ensure_ascii=False, default=str)
|
||||
user_prompt = (
|
||||
"Continue the conversation represented by the JSON messages below. "
|
||||
"Produce only the next assistant turn.\n\n"
|
||||
f"<conversation_messages>{serialized}</conversation_messages>"
|
||||
)
|
||||
|
||||
return _PromptParts(system_prompt=system_prompt, user_prompt=user_prompt)
|
||||
|
||||
|
||||
def _normalize_tool_arguments(arguments: Any) -> dict[str, Any]:
|
||||
if arguments is None:
|
||||
return {}
|
||||
if isinstance(arguments, Mapping):
|
||||
return dict(arguments)
|
||||
if isinstance(arguments, str):
|
||||
parsed = json.loads(arguments)
|
||||
if isinstance(parsed, dict):
|
||||
return parsed
|
||||
raise ValueError(f"GitHub Copilot returned non-object tool arguments: {type(arguments).__name__}")
|
||||
|
||||
|
||||
def _is_authentication_error(error: Exception) -> bool:
|
||||
text = str(error).lower()
|
||||
return any(
|
||||
marker in text
|
||||
for marker in (
|
||||
"not authenticated",
|
||||
"authentication failed",
|
||||
"unauthorized",
|
||||
"sign in",
|
||||
"login required",
|
||||
"invalid token",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _is_quota_error(error: Exception) -> bool:
|
||||
text = str(error).lower()
|
||||
return any(marker in text for marker in ("quota", "usage limit", "weekly limit", "ai credits"))
|
||||
|
||||
|
||||
def _is_configuration_error(error: BaseException) -> bool:
|
||||
"""Whether the runtime will reject this request identically on every attempt.
|
||||
|
||||
These surface as JsonRpcError, which is otherwise indistinguishable from a
|
||||
transport failure. Treating one as a runtime failure costs a full Copilot
|
||||
CLI restart per attempt and can never succeed, so they are terminal.
|
||||
"""
|
||||
text = str(error).lower()
|
||||
# Gate on "model" so a transient outage phrased as "service is not
|
||||
# available" stays retryable; only the model selection is terminal.
|
||||
if "model" not in text:
|
||||
return False
|
||||
return any(
|
||||
marker in text
|
||||
for marker in (
|
||||
"is not available",
|
||||
"unknown model",
|
||||
"not found",
|
||||
"invalid model",
|
||||
"unsupported model",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _is_runtime_failure(error: BaseException) -> bool:
|
||||
if _is_configuration_error(error):
|
||||
return False
|
||||
if isinstance(error, (TimeoutError, ConnectionError, OSError)):
|
||||
return True
|
||||
if type(error).__name__ in {"ProcessExitedError", "JsonRpcError"}:
|
||||
return True
|
||||
text = str(error).lower()
|
||||
return any(
|
||||
marker in text
|
||||
for marker in (
|
||||
"client not connected",
|
||||
"process exited",
|
||||
"connection closed",
|
||||
"connection lost",
|
||||
"broken pipe",
|
||||
"transport closed",
|
||||
"unexpected eof",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class GitHubCopilotLLM(LLMInterface):
|
||||
"""LLM provider backed by the authenticated GitHub Copilot runtime."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
provider: str,
|
||||
api_key: str,
|
||||
base_url: str,
|
||||
model: str,
|
||||
reasoning_effort: str | None = None,
|
||||
timeout: float | None = None,
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
super().__init__(provider, api_key, base_url, model, reasoning_effort, **kwargs)
|
||||
self._released = False
|
||||
self._timeout = timeout
|
||||
|
||||
if self.reasoning_effort == "none":
|
||||
self.reasoning_effort = None
|
||||
elif self.reasoning_effort is not None and self.reasoning_effort not in _SUPPORTED_REASONING_EFFORTS:
|
||||
raise ValueError(
|
||||
f"Unsupported GitHub Copilot reasoning effort {self.reasoning_effort!r}. "
|
||||
f"Use one of: {', '.join(sorted(_SUPPORTED_REASONING_EFFORTS))}, or leave it unset."
|
||||
)
|
||||
self.base_url = _normalize_runtime_url(base_url)
|
||||
self._runtime = _acquire_runtime(self.base_url)
|
||||
|
||||
async def verify_connection(self) -> None:
|
||||
try:
|
||||
await self._runtime.ensure_started()
|
||||
await self.call(
|
||||
messages=[{"role": "user", "content": "Reply with exactly: ok"}],
|
||||
max_completion_tokens=10,
|
||||
scope="verification",
|
||||
max_retries=0,
|
||||
)
|
||||
logger.info("GitHub Copilot connection verified successfully")
|
||||
except Exception as error:
|
||||
raise RuntimeError(f"GitHub Copilot connection verification failed: {error}") from error
|
||||
|
||||
async def _invoke(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
sdk_tools: list[Any],
|
||||
available_tools: list[str],
|
||||
system_suffix: str = "",
|
||||
timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS,
|
||||
) -> _InvocationResult:
|
||||
prompt = _build_prompt(messages)
|
||||
system_prompt = prompt.system_prompt
|
||||
if system_suffix:
|
||||
system_prompt = f"{system_prompt}\n\n{system_suffix}"
|
||||
|
||||
capture = _TurnCapture()
|
||||
session = None
|
||||
session_id: str | None = None
|
||||
client = None
|
||||
runtime_invalidated = False
|
||||
loop = asyncio.get_running_loop()
|
||||
deadline = loop.time() + timeout_seconds
|
||||
|
||||
async def wait_with_deadline(factory: Callable[[], Awaitable[Any]]) -> Any:
|
||||
remaining = deadline - loop.time()
|
||||
if remaining <= 0:
|
||||
raise TimeoutError(f"GitHub Copilot attempt exceeded {timeout_seconds:g}s")
|
||||
return await asyncio.wait_for(factory(), timeout=remaining)
|
||||
|
||||
try:
|
||||
client = await wait_with_deadline(self._runtime.get_client)
|
||||
session = await wait_with_deadline(
|
||||
lambda: client.create_session(
|
||||
model=self.model,
|
||||
reasoning_effort=self.reasoning_effort,
|
||||
tools=sdk_tools,
|
||||
available_tools=available_tools,
|
||||
system_message={"mode": "replace", "content": system_prompt},
|
||||
enable_experimental_mode=False,
|
||||
enable_session_telemetry=False,
|
||||
enable_file_change_tracking=False,
|
||||
skip_custom_instructions=True,
|
||||
custom_agents_local_only=True,
|
||||
coauthor_enabled=False,
|
||||
manage_schedule_enabled=False,
|
||||
streaming=False,
|
||||
include_sub_agent_streaming_events=False,
|
||||
mcp_oauth_token_storage="in-memory",
|
||||
embedding_cache_storage="in-memory",
|
||||
enable_config_discovery=False,
|
||||
skip_embedding_retrieval=True,
|
||||
enable_on_demand_instruction_discovery=False,
|
||||
enable_file_hooks=False,
|
||||
enable_host_git_operations=False,
|
||||
enable_session_store=False,
|
||||
enable_skills=False,
|
||||
memory={"enabled": False},
|
||||
on_event=capture.handle_event,
|
||||
)
|
||||
)
|
||||
session_id = session.session_id
|
||||
response = await wait_with_deadline(
|
||||
lambda: session.send_and_wait(
|
||||
prompt.user_prompt,
|
||||
timeout=max(deadline - loop.time(), 0.001),
|
||||
)
|
||||
)
|
||||
|
||||
if response is not None:
|
||||
from copilot.session_events import AssistantMessageData
|
||||
|
||||
if isinstance(response.data, AssistantMessageData) and response.data not in capture.assistant_messages:
|
||||
capture.assistant_messages.append(response.data)
|
||||
|
||||
tool_calls: list[LLMToolCall] = []
|
||||
seen_call_ids: set[str] = set()
|
||||
content = ""
|
||||
for assistant_message in capture.assistant_messages:
|
||||
if assistant_message.content:
|
||||
content = assistant_message.content
|
||||
for request in assistant_message.tool_requests or []:
|
||||
if request.tool_call_id in seen_call_ids:
|
||||
continue
|
||||
seen_call_ids.add(request.tool_call_id)
|
||||
tool_calls.append(
|
||||
LLMToolCall(
|
||||
id=request.tool_call_id,
|
||||
name=request.name,
|
||||
arguments=_normalize_tool_arguments(request.arguments),
|
||||
)
|
||||
)
|
||||
|
||||
return _InvocationResult(
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
usage=capture.token_usage(),
|
||||
finish_reason=capture.finish_reason,
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
if client is not None:
|
||||
await self._runtime.invalidate(client, "attempt was cancelled")
|
||||
runtime_invalidated = True
|
||||
raise
|
||||
except Exception as error:
|
||||
if client is not None and _is_runtime_failure(error):
|
||||
await self._runtime.invalidate(client, str(error))
|
||||
runtime_invalidated = True
|
||||
raise
|
||||
finally:
|
||||
if session is not None and client is not None and not runtime_invalidated:
|
||||
cleanup_ok = await self._cleanup_session(
|
||||
client=client,
|
||||
session=session,
|
||||
session_id=session_id,
|
||||
deadline=deadline,
|
||||
)
|
||||
if not cleanup_ok:
|
||||
await self._runtime.invalidate(client, "transient session cleanup timed out or failed")
|
||||
|
||||
def _timeout_seconds(self) -> float:
|
||||
return self._timeout if self._timeout is not None else _DEFAULT_TIMEOUT_SECONDS
|
||||
|
||||
@staticmethod
|
||||
async def _cleanup_session(
|
||||
client: CopilotClient,
|
||||
session: Any,
|
||||
session_id: str | None,
|
||||
deadline: float,
|
||||
) -> bool:
|
||||
loop = asyncio.get_running_loop()
|
||||
|
||||
async def cleanup_step(factory: Callable[[], Awaitable[Any]]) -> None:
|
||||
remaining = deadline - loop.time()
|
||||
if remaining <= 0:
|
||||
raise TimeoutError("GitHub Copilot attempt deadline expired before session cleanup")
|
||||
await asyncio.wait_for(
|
||||
factory(),
|
||||
timeout=min(remaining, _SESSION_CLEANUP_TIMEOUT_SECONDS),
|
||||
)
|
||||
|
||||
try:
|
||||
await cleanup_step(session.disconnect)
|
||||
if session_id is not None:
|
||||
await cleanup_step(lambda: client.delete_session(session_id))
|
||||
return True
|
||||
except Exception:
|
||||
logger.warning("Failed to clean up transient Copilot session %s", session_id, exc_info=True)
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _terminal_tool(name: str, description: str, parameters: dict[str, Any]) -> Tool:
|
||||
from copilot.tools import Tool, ToolInvocation, ToolResult
|
||||
|
||||
async def capture_tool(_invocation: ToolInvocation) -> ToolResult:
|
||||
# The assistant.message event carries the full tool request. Marking
|
||||
# this tool terminal prevents Copilot's own agent loop from feeding
|
||||
# this placeholder result back to the model; Hindsight executes the
|
||||
# real tool in its outer reflect loop.
|
||||
return ToolResult(text_result_for_llm="Tool call captured.", result_type="success")
|
||||
|
||||
return Tool(
|
||||
name=name,
|
||||
description=description,
|
||||
parameters=parameters,
|
||||
handler=capture_tool,
|
||||
skip_permission=True,
|
||||
defer="never",
|
||||
is_terminal=True,
|
||||
)
|
||||
|
||||
async def call(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
response_format: Any | None = None,
|
||||
max_completion_tokens: int | None = None,
|
||||
temperature: float | None = None,
|
||||
scope: str = "memory",
|
||||
max_retries: int = 10,
|
||||
initial_backoff: float = 1.0,
|
||||
max_backoff: float = 60.0,
|
||||
skip_validation: bool = False,
|
||||
strict_schema: bool = False,
|
||||
return_usage: bool = False,
|
||||
cached_prefix: str | None = None,
|
||||
attempt_context: Callable[[], AbstractAsyncContextManager[None]] | None = None,
|
||||
) -> Any:
|
||||
start_time = time.time()
|
||||
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
sdk_tools: list[Any] = []
|
||||
available_tools: list[str] = []
|
||||
system_suffix = ""
|
||||
|
||||
if response_format is not None:
|
||||
schema = response_format.model_json_schema()
|
||||
sdk_tools = [
|
||||
self._terminal_tool(
|
||||
_STRUCTURED_TOOL_NAME,
|
||||
"Return the structured response matching the required schema.",
|
||||
schema,
|
||||
)
|
||||
]
|
||||
available_tools = [f"custom:{_STRUCTURED_TOOL_NAME}"]
|
||||
system_suffix = (
|
||||
f"You MUST call the {_STRUCTURED_TOOL_NAME!r} tool exactly once. Do not answer with prose."
|
||||
)
|
||||
|
||||
async with attempt_context() if attempt_context is not None else nullcontext():
|
||||
set_stage(f"llm.github_copilot.{scope}.attempt={attempt + 1}/{max_retries + 1}")
|
||||
invocation = await self._invoke(
|
||||
messages=list(messages),
|
||||
sdk_tools=sdk_tools,
|
||||
available_tools=available_tools,
|
||||
system_suffix=system_suffix,
|
||||
timeout_seconds=self._timeout_seconds(),
|
||||
)
|
||||
|
||||
stash_response_usage(
|
||||
LLMResponseUsage(
|
||||
input_tokens=invocation.usage.input_tokens,
|
||||
output_tokens=invocation.usage.output_tokens,
|
||||
cached_tokens=invocation.usage.cached_tokens,
|
||||
)
|
||||
)
|
||||
|
||||
if response_format is not None:
|
||||
structured_call = next(
|
||||
(call for call in invocation.tool_calls if call.name == _STRUCTURED_TOOL_NAME),
|
||||
None,
|
||||
)
|
||||
if structured_call is None:
|
||||
raise RuntimeError("GitHub Copilot did not return the required structured_response tool call")
|
||||
result = (
|
||||
structured_call.arguments
|
||||
if skip_validation
|
||||
else response_format.model_validate(structured_call.arguments)
|
||||
)
|
||||
else:
|
||||
if not invocation.content:
|
||||
raise RuntimeError("GitHub Copilot returned an empty response")
|
||||
result = invocation.content
|
||||
|
||||
duration = time.time() - start_time
|
||||
self._record_success(
|
||||
messages=messages,
|
||||
result=result,
|
||||
invocation=invocation,
|
||||
scope=scope,
|
||||
duration=duration,
|
||||
)
|
||||
if return_usage:
|
||||
return result, invocation.usage
|
||||
return result
|
||||
except ValidationError:
|
||||
raise
|
||||
except Exception as error:
|
||||
if _is_configuration_error(error):
|
||||
raise RuntimeError(f"GitHub Copilot rejected the request configuration: {error}") from error
|
||||
if _is_authentication_error(error):
|
||||
raise RuntimeError(
|
||||
"GitHub Copilot authentication failed. Start Copilot CLI and sign in, "
|
||||
"or set COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN. "
|
||||
"HINDSIGHT_API_LLM_API_KEY is not used by this provider."
|
||||
) from error
|
||||
if _is_quota_error(error):
|
||||
raise RuntimeError(f"GitHub Copilot usage limit reached: {error}") from error
|
||||
if attempt >= max_retries:
|
||||
raise
|
||||
backoff = min(initial_backoff * (2**attempt), max_backoff)
|
||||
logger.warning(
|
||||
"GitHub Copilot error (attempt %d/%d): %s",
|
||||
attempt + 1,
|
||||
max_retries + 1,
|
||||
error,
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
|
||||
raise RuntimeError("GitHub Copilot call failed after all retries")
|
||||
|
||||
async def call_with_tools(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
tools: list[dict[str, Any]],
|
||||
max_completion_tokens: int | None = None,
|
||||
temperature: float | None = None,
|
||||
scope: str = "tools",
|
||||
max_retries: int = 5,
|
||||
initial_backoff: float = 1.0,
|
||||
max_backoff: float = 30.0,
|
||||
tool_choice: LLMToolChoice = LLM_TOOL_CHOICE_AUTO,
|
||||
cached_prefix: str | None = None,
|
||||
cached_prefix_message_count: int = 0,
|
||||
attempt_context: Callable[[], AbstractAsyncContextManager[None]] | None = None,
|
||||
) -> LLMToolCallResult:
|
||||
start_time = time.time()
|
||||
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
selected_tools = tools
|
||||
system_suffix = ""
|
||||
if tool_choice.mode is LLMToolChoiceMode.NONE:
|
||||
selected_tools = []
|
||||
elif tool_choice.mode is LLMToolChoiceMode.NAMED:
|
||||
selected_name = tool_choice.selected_function_name
|
||||
selected_tools = [tool for tool in tools if tool.get("function", {}).get("name") == selected_name]
|
||||
if not selected_tools:
|
||||
raise ValueError(f"Requested GitHub Copilot tool {selected_name!r} is not available")
|
||||
system_suffix = f"You MUST call the {selected_name!r} tool. Do not answer with prose."
|
||||
elif tool_choice.mode is LLMToolChoiceMode.REQUIRED:
|
||||
system_suffix = "You MUST call at least one available tool. Do not answer with prose."
|
||||
|
||||
sdk_tools = []
|
||||
available_tools = []
|
||||
for tool in selected_tools:
|
||||
function = tool.get("function", {})
|
||||
name = function.get("name", "")
|
||||
if not name:
|
||||
continue
|
||||
sdk_tools.append(
|
||||
self._terminal_tool(
|
||||
name=name,
|
||||
description=function.get("description", ""),
|
||||
parameters=function.get("parameters", {}),
|
||||
)
|
||||
)
|
||||
available_tools.append(f"custom:{name}")
|
||||
|
||||
async with attempt_context() if attempt_context is not None else nullcontext():
|
||||
set_stage(f"llm.github_copilot.tools.attempt={attempt + 1}/{max_retries + 1}")
|
||||
invocation = await self._invoke(
|
||||
messages=messages,
|
||||
sdk_tools=sdk_tools,
|
||||
available_tools=available_tools,
|
||||
system_suffix=system_suffix,
|
||||
timeout_seconds=self._timeout_seconds(),
|
||||
)
|
||||
|
||||
stash_response_usage(
|
||||
LLMResponseUsage(
|
||||
input_tokens=invocation.usage.input_tokens,
|
||||
output_tokens=invocation.usage.output_tokens,
|
||||
cached_tokens=invocation.usage.cached_tokens,
|
||||
)
|
||||
)
|
||||
|
||||
if tool_choice.mode is LLMToolChoiceMode.NAMED:
|
||||
selected_name = tool_choice.selected_function_name
|
||||
if not any(call.name == selected_name for call in invocation.tool_calls):
|
||||
raise RuntimeError(f"GitHub Copilot did not call the required {selected_name!r} tool")
|
||||
elif tool_choice.mode is LLMToolChoiceMode.REQUIRED and not invocation.tool_calls:
|
||||
raise RuntimeError("GitHub Copilot did not call any tool when a tool call was required")
|
||||
|
||||
duration = time.time() - start_time
|
||||
result = LLMToolCallResult(
|
||||
content=invocation.content or None,
|
||||
tool_calls=invocation.tool_calls,
|
||||
finish_reason="tool_calls" if invocation.tool_calls else invocation.finish_reason or "stop",
|
||||
input_tokens=invocation.usage.input_tokens,
|
||||
output_tokens=invocation.usage.output_tokens,
|
||||
cached_tokens=invocation.usage.cached_tokens,
|
||||
thoughts_tokens=invocation.usage.thoughts_tokens,
|
||||
)
|
||||
self._record_success(
|
||||
messages=messages,
|
||||
result=result,
|
||||
invocation=invocation,
|
||||
scope=scope,
|
||||
duration=duration,
|
||||
)
|
||||
return result
|
||||
except Exception as error:
|
||||
if _is_configuration_error(error):
|
||||
raise RuntimeError(f"GitHub Copilot rejected the request configuration: {error}") from error
|
||||
if _is_authentication_error(error):
|
||||
raise RuntimeError(
|
||||
"GitHub Copilot authentication failed. Start Copilot CLI and sign in, "
|
||||
"or set COPILOT_GITHUB_TOKEN, GH_TOKEN, or GITHUB_TOKEN."
|
||||
) from error
|
||||
if _is_quota_error(error):
|
||||
raise RuntimeError(f"GitHub Copilot usage limit reached: {error}") from error
|
||||
if attempt >= max_retries:
|
||||
raise
|
||||
backoff = min(initial_backoff * (2**attempt), max_backoff)
|
||||
logger.warning(
|
||||
"GitHub Copilot tool-call error (attempt %d/%d): %s",
|
||||
attempt + 1,
|
||||
max_retries + 1,
|
||||
error,
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
|
||||
raise RuntimeError("GitHub Copilot tool call failed after all retries")
|
||||
|
||||
def _record_success(
|
||||
self,
|
||||
messages: list[dict[str, Any]],
|
||||
result: Any,
|
||||
invocation: _InvocationResult,
|
||||
scope: str,
|
||||
duration: float,
|
||||
) -> None:
|
||||
usage = invocation.usage
|
||||
metrics = get_metrics_collector()
|
||||
metrics.record_llm_call(
|
||||
provider=self.provider,
|
||||
model=self.model,
|
||||
scope=scope,
|
||||
duration=duration,
|
||||
input_tokens=usage.input_tokens,
|
||||
output_tokens=usage.output_tokens,
|
||||
cached_input_tokens=usage.cached_tokens,
|
||||
thoughts_tokens=usage.thoughts_tokens,
|
||||
success=True,
|
||||
)
|
||||
|
||||
try:
|
||||
from hindsight_api.tracing import _serialize_for_span, get_span_recorder
|
||||
|
||||
get_span_recorder().record_llm_call(
|
||||
provider=self.provider,
|
||||
model=self.model,
|
||||
scope=scope,
|
||||
messages=messages,
|
||||
response_content=_serialize_for_span(result),
|
||||
input_tokens=usage.input_tokens,
|
||||
output_tokens=usage.output_tokens,
|
||||
cached_tokens=usage.cached_tokens,
|
||||
duration=duration,
|
||||
finish_reason=invocation.finish_reason,
|
||||
error=None,
|
||||
tool_calls=[
|
||||
{"id": call.id, "name": call.name, "arguments": call.arguments} for call in invocation.tool_calls
|
||||
]
|
||||
or None,
|
||||
)
|
||||
except Exception:
|
||||
logger.debug("GitHub Copilot span recording failed", exc_info=True)
|
||||
|
||||
if duration > 10.0:
|
||||
logger.info(
|
||||
"slow llm call: scope=%s, model=%s/%s, time=%.3fs",
|
||||
scope,
|
||||
self.provider,
|
||||
self.model,
|
||||
duration,
|
||||
)
|
||||
|
||||
async def cleanup(self) -> None:
|
||||
if self._released:
|
||||
return
|
||||
self._released = True
|
||||
await _release_runtime(self._runtime)
|
||||
|
||||
def supports_attempt_scoped_concurrency(self) -> bool:
|
||||
return True
|
||||
@@ -141,6 +141,7 @@ PROVIDER_NAME_MAPPING = {
|
||||
"lmstudio": "lmstudio",
|
||||
"openai-codex": "openai",
|
||||
"claude-code": "anthropic",
|
||||
"github-copilot": "github",
|
||||
"mock": "mock",
|
||||
}
|
||||
|
||||
|
||||
@@ -95,6 +95,7 @@ dependencies = [
|
||||
"aiohttp>=3.14.3", # GHSA-cq5v-8q36-5273: OOB heap read in the C response parser, plus earlier DoS fixes
|
||||
"pygments>=2.20.0", # ReDoS via inefficient GUID regex fix
|
||||
"claude-agent-sdk>=0.2.82",
|
||||
"github-copilot-sdk>=1.0.11",
|
||||
"boto3>=1.42.74",
|
||||
"croniter>=2.0.0", # Cron parsing for scheduled mental model refresh
|
||||
"json-repair>=0.63.2", # Structural repair of malformed LLM JSON (last-resort parse fallback); >=0.60.1 also fixes the circular-$ref unbounded-CPU DoS
|
||||
|
||||
@@ -0,0 +1,601 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
from pydantic import BaseModel
|
||||
|
||||
from hindsight_api.config import PROVIDER_DEFAULT_MODELS
|
||||
from hindsight_api.engine.llm_interface import LLM_TOOL_CHOICE_REQUIRED, LLMToolChoice
|
||||
from hindsight_api.engine.llm_wrapper import create_llm_provider, requires_api_key
|
||||
from hindsight_api.engine.providers import github_copilot_llm as provider_module
|
||||
from hindsight_api.engine.providers.github_copilot_llm import GitHubCopilotLLM
|
||||
|
||||
|
||||
class _StructuredAnswer(BaseModel):
|
||||
answer: str
|
||||
|
||||
|
||||
class _FakeSession:
|
||||
def __init__(self, on_event, events, *, error: Exception | None = None) -> None:
|
||||
self.session_id = "copilot-session"
|
||||
self._on_event = on_event
|
||||
self._events = events
|
||||
self._error = error
|
||||
self.disconnected = False
|
||||
self.aborted = False
|
||||
|
||||
async def send_and_wait(self, _prompt: str, *, timeout: float):
|
||||
if self._error is not None:
|
||||
raise self._error
|
||||
for event in self._events:
|
||||
self._on_event(event)
|
||||
assistant_events = [event for event in self._events if event.data.__class__.__name__ == "AssistantMessageData"]
|
||||
return assistant_events[-1] if assistant_events else None
|
||||
|
||||
async def disconnect(self) -> None:
|
||||
self.disconnected = True
|
||||
|
||||
async def abort(self) -> None:
|
||||
self.aborted = True
|
||||
|
||||
|
||||
class _FakeClient:
|
||||
def __init__(self, events, *, error: Exception | None = None) -> None:
|
||||
self.events = events
|
||||
self.error = error
|
||||
self.create_kwargs = None
|
||||
self.create_count = 0
|
||||
self.session = None
|
||||
self.deleted_sessions: list[str] = []
|
||||
|
||||
async def create_session(self, **kwargs):
|
||||
self.create_count += 1
|
||||
self.create_kwargs = kwargs
|
||||
self.session = _FakeSession(kwargs["on_event"], self.events, error=self.error)
|
||||
return self.session
|
||||
|
||||
async def delete_session(self, session_id: str) -> None:
|
||||
self.deleted_sessions.append(session_id)
|
||||
|
||||
|
||||
class _FakeRuntime:
|
||||
def __init__(self, client: _FakeClient) -> None:
|
||||
self.client = client
|
||||
self.runtime_url = ""
|
||||
self.ref_count = 1
|
||||
self.started = False
|
||||
self.invalidations: list[str] = []
|
||||
|
||||
async def ensure_started(self) -> None:
|
||||
self.started = True
|
||||
|
||||
async def get_client(self):
|
||||
self.started = True
|
||||
return self.client
|
||||
|
||||
async def invalidate(self, _expected_client, reason: str) -> None:
|
||||
self.invalidations.append(reason)
|
||||
|
||||
async def stop(self) -> None:
|
||||
self.started = False
|
||||
|
||||
|
||||
def _assistant_event(content: str, tool_requests=None):
|
||||
from copilot.session_events import AssistantMessageData
|
||||
|
||||
return SimpleNamespace(
|
||||
data=AssistantMessageData(
|
||||
content=content,
|
||||
message_id="message-1",
|
||||
tool_requests=tool_requests,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _usage_event():
|
||||
from copilot.session_events import AssistantUsageData
|
||||
|
||||
return SimpleNamespace(
|
||||
data=AssistantUsageData(
|
||||
model="gpt-5.6-terra",
|
||||
input_tokens=120,
|
||||
output_tokens=30,
|
||||
cache_read_tokens=20,
|
||||
reasoning_tokens=10,
|
||||
finish_reason="stop",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _provider(monkeypatch, events, *, error: Exception | None = None) -> tuple[GitHubCopilotLLM, _FakeClient]:
|
||||
client = _FakeClient(events, error=error)
|
||||
runtime = _FakeRuntime(client)
|
||||
monkeypatch.setattr(provider_module, "_acquire_runtime", lambda _url: runtime)
|
||||
provider = GitHubCopilotLLM(
|
||||
provider="github-copilot",
|
||||
api_key="",
|
||||
base_url="",
|
||||
model="gpt-5.6-terra",
|
||||
timeout=15,
|
||||
)
|
||||
return provider, client
|
||||
|
||||
|
||||
def test_provider_registration_and_default_model(monkeypatch):
|
||||
assert requires_api_key("github-copilot") is False
|
||||
assert PROVIDER_DEFAULT_MODELS["github-copilot"] == "gpt-5.6-terra"
|
||||
|
||||
runtime = _FakeRuntime(_FakeClient([]))
|
||||
monkeypatch.setattr(provider_module, "_acquire_runtime", lambda _url: runtime)
|
||||
provider = create_llm_provider(
|
||||
provider="github-copilot",
|
||||
api_key="",
|
||||
base_url="",
|
||||
model="gpt-5.6-terra",
|
||||
reasoning_effort=None,
|
||||
)
|
||||
|
||||
assert isinstance(provider, GitHubCopilotLLM)
|
||||
|
||||
|
||||
def test_openai_template_base_url_is_ignored(monkeypatch):
|
||||
runtime = _FakeRuntime(_FakeClient([]))
|
||||
acquired_urls: list[str] = []
|
||||
|
||||
def acquire(url: str):
|
||||
acquired_urls.append(url)
|
||||
return runtime
|
||||
|
||||
monkeypatch.setattr(provider_module, "_acquire_runtime", acquire)
|
||||
provider = GitHubCopilotLLM(
|
||||
provider="github-copilot",
|
||||
api_key="",
|
||||
base_url="https://api.openai.com/v1",
|
||||
model="gpt-5.6-terra",
|
||||
)
|
||||
|
||||
assert provider.base_url == ""
|
||||
assert acquired_urls == [""]
|
||||
|
||||
|
||||
def test_isolated_copilot_home_copies_only_account_metadata(tmp_path):
|
||||
source_home = tmp_path / "source"
|
||||
target_home = tmp_path / "target"
|
||||
source_home.mkdir()
|
||||
(source_home / "config.json").write_text(
|
||||
'{"lastLoggedInUser":{"host":"https://github.com","login":"user"}}',
|
||||
encoding="utf-8",
|
||||
)
|
||||
(source_home / "hooks").mkdir()
|
||||
(source_home / "hooks" / "dangerous.json").write_text("{}", encoding="utf-8")
|
||||
|
||||
provider_module._sync_copilot_auth_metadata(target_home, source_home)
|
||||
|
||||
assert (target_home / "config.json").read_text(encoding="utf-8") == (source_home / "config.json").read_text(
|
||||
encoding="utf-8"
|
||||
)
|
||||
assert not (target_home / "hooks").exists()
|
||||
|
||||
|
||||
def test_invalid_reasoning_effort_does_not_acquire_runtime(monkeypatch):
|
||||
acquire = MagicMock()
|
||||
monkeypatch.setattr(provider_module, "_acquire_runtime", acquire)
|
||||
|
||||
with pytest.raises(ValueError, match="Unsupported GitHub Copilot reasoning effort"):
|
||||
GitHubCopilotLLM(
|
||||
provider="github-copilot",
|
||||
api_key="",
|
||||
base_url="",
|
||||
model="gpt-5.6-terra",
|
||||
reasoning_effort="extreme",
|
||||
)
|
||||
|
||||
acquire.assert_not_called()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_plain_call_isolates_session_and_reports_usage(monkeypatch):
|
||||
provider, client = _provider(monkeypatch, [_assistant_event("ok"), _usage_event()])
|
||||
|
||||
result, usage = await provider.call(
|
||||
messages=[
|
||||
{"role": "system", "content": "Reply concisely."},
|
||||
{"role": "user", "content": "Say ok."},
|
||||
],
|
||||
return_usage=True,
|
||||
max_retries=0,
|
||||
)
|
||||
|
||||
assert result == "ok"
|
||||
assert usage.input_tokens == 120
|
||||
assert usage.output_tokens == 30
|
||||
assert usage.cached_tokens == 20
|
||||
assert usage.thoughts_tokens == 10
|
||||
assert client.create_kwargs["model"] == "gpt-5.6-terra"
|
||||
assert client.create_kwargs["available_tools"] == []
|
||||
assert client.create_kwargs["system_message"] == {
|
||||
"mode": "replace",
|
||||
"content": "Reply concisely.",
|
||||
}
|
||||
assert client.create_kwargs["enable_file_hooks"] is False
|
||||
assert client.create_kwargs["enable_config_discovery"] is False
|
||||
assert client.create_kwargs["skip_custom_instructions"] is True
|
||||
assert client.create_kwargs["enable_session_store"] is False
|
||||
assert client.create_kwargs["enable_skills"] is False
|
||||
assert client.create_kwargs["memory"] == {"enabled": False}
|
||||
assert client.session.disconnected is True
|
||||
assert client.deleted_sessions == ["copilot-session"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_structured_call_uses_terminal_schema_tool(monkeypatch):
|
||||
from copilot.session_events import AssistantMessageToolRequest
|
||||
|
||||
request = AssistantMessageToolRequest(
|
||||
name="structured_response",
|
||||
tool_call_id="structured-1",
|
||||
arguments={"answer": "captured"},
|
||||
)
|
||||
provider, client = _provider(
|
||||
monkeypatch,
|
||||
[_assistant_event("", [request]), _usage_event()],
|
||||
)
|
||||
|
||||
result = await provider.call(
|
||||
messages=[{"role": "user", "content": "Return an answer."}],
|
||||
response_format=_StructuredAnswer,
|
||||
max_retries=0,
|
||||
)
|
||||
|
||||
assert result == _StructuredAnswer(answer="captured")
|
||||
tool = client.create_kwargs["tools"][0]
|
||||
assert tool.name == "structured_response"
|
||||
assert tool.parameters == _StructuredAnswer.model_json_schema()
|
||||
assert tool.is_terminal is True
|
||||
assert tool.skip_permission is True
|
||||
assert client.create_kwargs["available_tools"] == ["custom:structured_response"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_named_tool_choice_exposes_only_requested_tool(monkeypatch):
|
||||
from copilot.session_events import AssistantMessageToolRequest
|
||||
|
||||
request = AssistantMessageToolRequest(
|
||||
name="recall",
|
||||
tool_call_id="recall-1",
|
||||
arguments={"query": "project decisions"},
|
||||
)
|
||||
provider, client = _provider(
|
||||
monkeypatch,
|
||||
[_assistant_event("", [request]), _usage_event()],
|
||||
)
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "recall",
|
||||
"description": "Recall memories",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"query": {"type": "string"}},
|
||||
"required": ["query"],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "done",
|
||||
"description": "Finish",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
result = await provider.call_with_tools(
|
||||
messages=[{"role": "user", "content": "Find project decisions."}],
|
||||
tools=tools,
|
||||
tool_choice=LLMToolChoice.named("recall"),
|
||||
max_retries=0,
|
||||
)
|
||||
|
||||
assert result.finish_reason == "tool_calls"
|
||||
assert result.tool_calls[0].name == "recall"
|
||||
assert result.tool_calls[0].arguments == {"query": "project decisions"}
|
||||
assert [tool.name for tool in client.create_kwargs["tools"]] == ["recall"]
|
||||
assert client.create_kwargs["available_tools"] == ["custom:recall"]
|
||||
assert "MUST call the 'recall' tool" in client.create_kwargs["system_message"]["content"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_named_tool_choice_retries_then_rejects_prose(monkeypatch):
|
||||
provider, client = _provider(monkeypatch, [_assistant_event("I will answer directly."), _usage_event()])
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "recall",
|
||||
"description": "Recall memories",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
with pytest.raises(RuntimeError, match="did not call the required 'recall' tool"):
|
||||
await provider.call_with_tools(
|
||||
messages=[{"role": "user", "content": "Recall first."}],
|
||||
tools=tools,
|
||||
tool_choice=LLMToolChoice.named("recall"),
|
||||
max_retries=1,
|
||||
initial_backoff=0,
|
||||
)
|
||||
|
||||
assert client.create_count == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_required_tool_choice_rejects_prose(monkeypatch):
|
||||
provider, _client = _provider(monkeypatch, [_assistant_event("No tool needed."), _usage_event()])
|
||||
tools = [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "recall",
|
||||
"description": "Recall memories",
|
||||
"parameters": {"type": "object", "properties": {}},
|
||||
},
|
||||
}
|
||||
]
|
||||
|
||||
with pytest.raises(RuntimeError, match="did not call any tool"):
|
||||
await provider.call_with_tools(
|
||||
messages=[{"role": "user", "content": "Use a tool."}],
|
||||
tools=tools,
|
||||
tool_choice=LLM_TOOL_CHOICE_REQUIRED,
|
||||
max_retries=0,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_timeout_aborts_and_deletes_session(monkeypatch):
|
||||
provider, client = _provider(monkeypatch, [], error=TimeoutError("slow"))
|
||||
|
||||
with pytest.raises(TimeoutError, match="slow"):
|
||||
await provider.call(
|
||||
messages=[{"role": "user", "content": "wait"}],
|
||||
max_retries=0,
|
||||
)
|
||||
|
||||
assert provider._runtime.invalidations == ["slow"]
|
||||
assert client.session.disconnected is False
|
||||
assert client.deleted_sessions == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_attempt_timeout_includes_runtime_startup(monkeypatch):
|
||||
client = _FakeClient([])
|
||||
|
||||
class _SlowRuntime(_FakeRuntime):
|
||||
async def get_client(self):
|
||||
await asyncio.sleep(1)
|
||||
return self.client
|
||||
|
||||
runtime = _SlowRuntime(client)
|
||||
monkeypatch.setattr(provider_module, "_acquire_runtime", lambda _url: runtime)
|
||||
provider = GitHubCopilotLLM(
|
||||
provider="github-copilot",
|
||||
api_key="",
|
||||
base_url="",
|
||||
model="gpt-5.6-terra",
|
||||
timeout=0.01,
|
||||
)
|
||||
|
||||
started = time.monotonic()
|
||||
with pytest.raises(TimeoutError):
|
||||
await provider.call(
|
||||
messages=[{"role": "user", "content": "wait"}],
|
||||
max_retries=0,
|
||||
)
|
||||
|
||||
assert time.monotonic() - started < 0.2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_runtime_failure_invalidates_shared_client(monkeypatch):
|
||||
provider, _client = _provider(monkeypatch, [], error=RuntimeError("Client not connected"))
|
||||
|
||||
with pytest.raises(RuntimeError, match="Client not connected"):
|
||||
await provider.call(
|
||||
messages=[{"role": "user", "content": "hello"}],
|
||||
max_retries=0,
|
||||
)
|
||||
|
||||
assert provider._runtime.invalidations == ["Client not connected"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_session_cleanup_timeout_invalidates_runtime(monkeypatch):
|
||||
client = _FakeClient([_assistant_event("ok"), _usage_event()])
|
||||
|
||||
class _SlowCleanupSession(_FakeSession):
|
||||
async def disconnect(self) -> None:
|
||||
await asyncio.sleep(1)
|
||||
|
||||
async def create_session(**kwargs):
|
||||
client.create_count += 1
|
||||
client.create_kwargs = kwargs
|
||||
client.session = _SlowCleanupSession(kwargs["on_event"], client.events)
|
||||
return client.session
|
||||
|
||||
client.create_session = create_session
|
||||
runtime = _FakeRuntime(client)
|
||||
monkeypatch.setattr(provider_module, "_acquire_runtime", lambda _url: runtime)
|
||||
provider = GitHubCopilotLLM(
|
||||
provider="github-copilot",
|
||||
api_key="",
|
||||
base_url="",
|
||||
model="gpt-5.6-terra",
|
||||
timeout=0.05,
|
||||
)
|
||||
|
||||
started = time.monotonic()
|
||||
assert (
|
||||
await provider.call(
|
||||
messages=[{"role": "user", "content": "say ok"}],
|
||||
max_retries=0,
|
||||
)
|
||||
== "ok"
|
||||
)
|
||||
|
||||
assert time.monotonic() - started < 0.2
|
||||
assert runtime.invalidations == ["transient session cleanup timed out or failed"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_shared_runtime_invalidation_replaces_dead_client(monkeypatch, tmp_path):
|
||||
old_client = MagicMock()
|
||||
old_client.force_stop = AsyncMock()
|
||||
new_client = MagicMock()
|
||||
clients = iter([old_client, new_client])
|
||||
monkeypatch.setattr(
|
||||
provider_module.tempfile,
|
||||
"mkdtemp",
|
||||
lambda **_kwargs: str(tmp_path / "runtime-home"),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
provider_module._SharedCopilotRuntime,
|
||||
"_new_client",
|
||||
lambda _self: next(clients),
|
||||
)
|
||||
|
||||
runtime = provider_module._SharedCopilotRuntime("")
|
||||
runtime._started = True
|
||||
await runtime.invalidate(old_client, "connection closed")
|
||||
|
||||
assert runtime.client is new_client
|
||||
assert runtime._started is False
|
||||
old_client.force_stop.assert_awaited_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_shared_runtime_stops_only_after_last_release(monkeypatch, tmp_path):
|
||||
with provider_module._runtime_registry_lock:
|
||||
provider_module._runtime_registry.clear()
|
||||
|
||||
monkeypatch.setattr(
|
||||
provider_module.tempfile,
|
||||
"mkdtemp",
|
||||
lambda **_kwargs: str(tmp_path / "runtime-home"),
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
provider_module._SharedCopilotRuntime,
|
||||
"_new_client",
|
||||
lambda _self: MagicMock(),
|
||||
)
|
||||
runtime_a = provider_module._acquire_runtime("")
|
||||
runtime_b = provider_module._acquire_runtime("")
|
||||
runtime_a.stop = AsyncMock()
|
||||
|
||||
assert runtime_a is runtime_b
|
||||
assert runtime_a.ref_count == 2
|
||||
|
||||
await provider_module._release_runtime(runtime_a)
|
||||
runtime_a.stop.assert_not_awaited()
|
||||
|
||||
await provider_module._release_runtime(runtime_b)
|
||||
runtime_a.stop.assert_awaited_once()
|
||||
|
||||
|
||||
def test_prompt_serializes_tool_history():
|
||||
prompt = provider_module._build_prompt(
|
||||
[
|
||||
{"role": "system", "content": "Use memory tools."},
|
||||
{"role": "user", "content": "What changed?"},
|
||||
{
|
||||
"role": "assistant",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call-1",
|
||||
"type": "function",
|
||||
"function": {"name": "recall", "arguments": '{"query":"changes"}'},
|
||||
}
|
||||
],
|
||||
},
|
||||
{"role": "tool", "tool_call_id": "call-1", "content": '{"memories":["A"]}'},
|
||||
]
|
||||
)
|
||||
|
||||
assert prompt.system_prompt == "Use memory tools."
|
||||
assert '"tool_call_id": "call-1"' in prompt.user_prompt
|
||||
assert '"name": "recall"' in prompt.user_prompt
|
||||
|
||||
|
||||
class JsonRpcError(RuntimeError):
|
||||
"""Stand-in for the SDK error class, which is matched by name."""
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configuration_error_is_terminal_and_spares_the_runtime(monkeypatch):
|
||||
"""An unavailable model is rejected identically forever.
|
||||
|
||||
It arrives as JsonRpcError, which _is_runtime_failure would otherwise treat
|
||||
as a dead transport — restarting the Copilot CLI once per attempt for an
|
||||
error that can never succeed.
|
||||
"""
|
||||
error = JsonRpcError('Request session.create failed with message: Model "gpt-9" is not available.')
|
||||
provider, client = _provider(monkeypatch, [], error=error)
|
||||
|
||||
with pytest.raises(RuntimeError, match="rejected the request configuration"):
|
||||
await provider.call(messages=[{"role": "user", "content": "hi"}], max_retries=3, initial_backoff=0)
|
||||
|
||||
assert client.create_count == 1
|
||||
assert provider._runtime.invalidations == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_configuration_error_is_terminal_for_tool_calls(monkeypatch):
|
||||
error = JsonRpcError("Model is not available.")
|
||||
provider, client = _provider(monkeypatch, [], error=error)
|
||||
|
||||
with pytest.raises(RuntimeError, match="rejected the request configuration"):
|
||||
await provider.call_with_tools(
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
tools=[{"type": "function", "function": {"name": "noop", "parameters": {}}}],
|
||||
max_retries=3,
|
||||
initial_backoff=0,
|
||||
)
|
||||
|
||||
assert client.create_count == 1
|
||||
assert provider._runtime.invalidations == []
|
||||
|
||||
|
||||
def test_missing_account_metadata_falls_back_to_token_credentials(monkeypatch, tmp_path):
|
||||
"""A container that never ran Copilot CLI authenticates from the token env."""
|
||||
monkeypatch.delenv("COPILOT_GITHUB_TOKEN", raising=False)
|
||||
monkeypatch.delenv("GH_TOKEN", raising=False)
|
||||
monkeypatch.setenv("GITHUB_TOKEN", "ghp_token")
|
||||
target = tmp_path / "isolated-home"
|
||||
|
||||
provider_module._sync_copilot_auth_metadata(target, source_home=tmp_path / "absent")
|
||||
|
||||
assert target.is_dir()
|
||||
assert not (target / "config.json").exists()
|
||||
|
||||
|
||||
def test_missing_account_metadata_without_credentials_raises(monkeypatch, tmp_path):
|
||||
for name in provider_module._TOKEN_ENV_VARS:
|
||||
monkeypatch.delenv(name, raising=False)
|
||||
|
||||
with pytest.raises(RuntimeError, match="COPILOT_GITHUB_TOKEN"):
|
||||
provider_module._sync_copilot_auth_metadata(tmp_path / "isolated-home", source_home=tmp_path / "absent")
|
||||
|
||||
|
||||
def test_transient_service_outage_is_not_mistaken_for_a_config_error():
|
||||
""" "is not available" alone must stay retryable — only the model is terminal."""
|
||||
assert provider_module._is_configuration_error(JsonRpcError("Copilot service is not available.")) is False
|
||||
assert provider_module._is_runtime_failure(JsonRpcError("Copilot service is not available.")) is True
|
||||
assert provider_module._is_configuration_error(JsonRpcError('Model "gpt-9" is not available.')) is True
|
||||
@@ -0,0 +1,33 @@
|
||||
from pathlib import Path
|
||||
from runpy import run_path
|
||||
|
||||
|
||||
def test_all_copied_llm_registries_allow_github_copilot_without_an_api_key(monkeypatch):
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
registry_files = sorted((root / "hindsight-integrations").glob("**/llm.py"))
|
||||
|
||||
monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "github-copilot")
|
||||
monkeypatch.setenv("HINDSIGHT_API_LLM_MODEL", "gpt-5.6-terra")
|
||||
monkeypatch.delenv("HINDSIGHT_API_LLM_API_KEY", raising=False)
|
||||
|
||||
assert registry_files
|
||||
for path in registry_files:
|
||||
namespace = run_path(str(path))
|
||||
no_key_required = namespace.get("NO_KEY_REQUIRED")
|
||||
assert isinstance(no_key_required, set), f"{path} has no no-key provider registry"
|
||||
assert "github-copilot" in no_key_required, f"{path} does not register github-copilot"
|
||||
detected = namespace["detect_llm_config"]({})
|
||||
assert detected["provider"] == "github-copilot"
|
||||
assert detected["api_key"] == ""
|
||||
assert detected["model"] == "gpt-5.6-terra"
|
||||
|
||||
|
||||
def test_openclaw_no_key_registries_include_github_copilot():
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
paths = [
|
||||
root / "hindsight-integrations" / "openclaw" / "src" / "index.ts",
|
||||
root / "hindsight-integrations" / "openclaw" / "src" / "setup-lib.ts",
|
||||
]
|
||||
|
||||
for path in paths:
|
||||
assert '"github-copilot"' in path.read_text(encoding="utf-8")
|
||||
@@ -31,6 +31,7 @@ def test_provider_name_mapping():
|
||||
assert PROVIDER_NAME_MAPPING["ollama"] == "ollama"
|
||||
assert PROVIDER_NAME_MAPPING["openai-codex"] == "openai"
|
||||
assert PROVIDER_NAME_MAPPING["claude-code"] == "anthropic"
|
||||
assert PROVIDER_NAME_MAPPING["github-copilot"] == "github"
|
||||
|
||||
|
||||
def test_truncate_content_short():
|
||||
|
||||
@@ -80,8 +80,8 @@ Configure via environment variables:
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_API_DATABASE_URL` | PostgreSQL connection string | `pg0` (embedded) |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | API key for LLM provider | - |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | LLM provider, including `openai`, `anthropic`, `gemini`, `groq`, `ollama`, `lmstudio`, and `github-copilot` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | API key for providers that require one; unused by `github-copilot` | - |
|
||||
| `HINDSIGHT_API_LLM_MODEL` | Model name | `gpt-4o-mini` |
|
||||
| `HINDSIGHT_API_HOST` | Server bind address | `0.0.0.0` |
|
||||
| `HINDSIGHT_API_PORT` | Server port | `8888` |
|
||||
|
||||
@@ -213,8 +213,8 @@ For non-English banks (especially CJK) and the language/extraction-language trad
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | Provider: `openai`, `openai-responses`, `openai-codex`, `claude-code`, `anthropic`, `gemini`, `groq`, `minimax`, `deepseek`, `zai`, `opencode-go`, `nous`, `xai-oauth`, `fireworks`, `ollama`, `ollama-cloud`, `lmstudio`, `llamacpp`, `vertexai`, `bedrock`, `litellm`, `litellmrouter`, `volcano`, `openrouter`, `requesty`, `none` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | API key for LLM provider | - |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | Provider: `openai`, `openai-responses`, `openai-codex`, `claude-code`, `github-copilot`, `anthropic`, `gemini`, `groq`, `minimax`, `deepseek`, `zai`, `opencode-go`, `nous`, `xai-oauth`, `fireworks`, `ollama`, `ollama-cloud`, `lmstudio`, `llamacpp`, `vertexai`, `bedrock`, `litellm`, `litellmrouter`, `volcano`, `openrouter`, `requesty`, `none` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | API key for providers that require one; unused by `github-copilot` | - |
|
||||
| `HINDSIGHT_API_LLM_MODEL` | Model name | `gpt-5-mini` |
|
||||
| `HINDSIGHT_API_LLM_BASE_URL` | Custom LLM endpoint | Provider default |
|
||||
| `HINDSIGHT_API_LLM_MAX_CONCURRENT` | Max concurrent LLM requests | `32` |
|
||||
|
||||
@@ -11,7 +11,7 @@ Guide to setting up a local development environment for contributing to Hindsigh
|
||||
- Python 3.11+
|
||||
- [uv](https://docs.astral.sh/uv/) - Fast Python package manager
|
||||
- Docker and Docker Compose
|
||||
- An LLM API key (OpenAI, Groq, or Ollama)
|
||||
- An LLM provider credential, or a local/subscription-backed provider such as Ollama or GitHub Copilot
|
||||
|
||||
## Local Development Setup
|
||||
|
||||
|
||||
@@ -268,6 +268,34 @@ export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-gcp-project-id
|
||||
|
||||
---
|
||||
|
||||
### GitHub Copilot Setup
|
||||
|
||||
Use a GitHub Copilot subscription for Hindsight's extraction, consolidation, and reflection calls through the official GitHub Copilot SDK.
|
||||
|
||||
**Prerequisites:**
|
||||
- An active GitHub Copilot entitlement
|
||||
- Copilot CLI signed in under the same operating-system user that runs Hindsight
|
||||
|
||||
**Configure Hindsight:**
|
||||
|
||||
```bash
|
||||
export HINDSIGHT_API_LLM_PROVIDER=github-copilot
|
||||
export HINDSIGHT_API_LLM_MODEL=gpt-5.6-terra
|
||||
# No HINDSIGHT_API_LLM_API_KEY is needed.
|
||||
```
|
||||
|
||||
The provider starts one shared Copilot runtime for all Hindsight LLM lanes. That runtime uses a hook-free temporary `COPILOT_HOME` containing only the signed-in account selection; the credential itself is resolved by the runtime from the system keychain or an existing `gh` CLI login. Each call then uses an isolated transient session with repository instructions, skills, Copilot memory, built-in tools, and the cross-session store disabled. This prevents the memory integration from recursively retaining its own extraction calls.
|
||||
|
||||
For automation, the Copilot SDK also accepts `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`. With one of those set, Copilot CLI never has to have been run on the host, so containers and CI images work with no `~/.copilot` at all. GitHub Actions and server-to-server deployments must have the appropriate Copilot organization policy and token permissions.
|
||||
|
||||
**Important notes:**
|
||||
- Usage counts against the authenticated account or organization's Copilot allowance and AI Credits.
|
||||
- `HINDSIGHT_API_LLM_BASE_URL` optionally points to an existing headless Copilot runtime, such as `http://127.0.0.1:4321`; it is not an LLM-provider endpoint for this provider.
|
||||
- GitHub-hosted Copilot sessions do not expose temperature or maximum-output-token controls through the SDK, so those Hindsight settings are not applied.
|
||||
- Embeddings and reranking continue to use Hindsight's separately configured providers; their local defaults require no API key.
|
||||
|
||||
---
|
||||
|
||||
### OpenAI Codex Setup (ChatGPT Plus/Pro)
|
||||
|
||||
Use your ChatGPT Plus or Pro subscription for Hindsight without separate OpenAI Platform API costs.
|
||||
|
||||
@@ -19,6 +19,7 @@
|
||||
{"id": "requesty", "label": "Requesty", "iconKey": "openai-compatible", "defaultModel": "openai/gpt-4o-mini"},
|
||||
{"id": "openai-codex", "label": "OpenAI Codex", "iconKey": "openai", "defaultModel": "gpt-5.4-mini"},
|
||||
{"id": "claude-code", "label": "Claude Code", "iconKey": "anthropic", "defaultModel": "claude-sonnet-4-5-20250929"},
|
||||
{"id": "github-copilot","label": "GitHub Copilot", "iconKey": "terminal", "defaultModel": "gpt-5.6-terra"},
|
||||
{"id": "bedrock", "label": "AWS Bedrock", "iconKey": "cloud", "defaultModel": "us.amazon.nova-2-lite-v1:0"},
|
||||
{"id": "fireworks", "label": "Fireworks AI", "iconKey": "zap", "defaultModel": "accounts/fireworks/models/llama-v3p1-8b-instruct", "batchApi": true},
|
||||
{"id": "nous", "label": "Nous Portal", "iconKey": "openai-compatible", "defaultModel": "deepseek/deepseek-v4-flash"},
|
||||
|
||||
@@ -60,8 +60,8 @@ hindsight-embed configure --profile staging
|
||||
```
|
||||
|
||||
This will:
|
||||
- Let you choose an LLM provider (OpenAI, Groq, Google, Ollama)
|
||||
- Configure your API key
|
||||
- Let you choose an LLM provider (OpenAI, Groq, Google, Ollama, GitHub Copilot)
|
||||
- Configure credentials when the provider requires them
|
||||
- Set the model
|
||||
- Start the daemon with your configuration
|
||||
|
||||
@@ -151,7 +151,7 @@ Run `hindsight-embed configure` for a guided setup that saves to `~/.hindsight/e
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_EMBED_PROFILE` | Profile name to use (overrides active profile) | None (uses default profile) |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | LLM API key (or use `OPENAI_API_KEY`); required only when the selected provider uses an API key | Provider-dependent |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | LLM provider (`openai`, `groq`, `google`, `ollama`) | `openai` |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | LLM provider (`openai`, `groq`, `gemini`, `ollama`, `github-copilot`) | `openai` |
|
||||
| `HINDSIGHT_API_LLM_MODEL` | LLM model | `gpt-4o-mini` |
|
||||
| `HINDSIGHT_EMBED_API_URL` | Use external API server instead of starting local daemon | None (starts local daemon) |
|
||||
| `HINDSIGHT_EMBED_API_TOKEN` | Authentication token for external API (sent as Bearer token) | None |
|
||||
|
||||
@@ -126,9 +126,14 @@ def get_config():
|
||||
venvs (e.g. `uvx hindsight-embed`) where `hindsight-api` isn't installed.
|
||||
"""
|
||||
load_config_file()
|
||||
provider = os.environ.get("HINDSIGHT_API_LLM_PROVIDER", "openai")
|
||||
return {
|
||||
"llm_api_key": os.environ.get("HINDSIGHT_API_LLM_API_KEY") or os.environ.get("OPENAI_API_KEY"),
|
||||
"llm_provider": os.environ.get("HINDSIGHT_API_LLM_PROVIDER", "openai"),
|
||||
"llm_api_key": (
|
||||
None
|
||||
if provider in NO_API_KEY_PROVIDERS
|
||||
else os.environ.get("HINDSIGHT_API_LLM_API_KEY") or os.environ.get("OPENAI_API_KEY")
|
||||
),
|
||||
"llm_provider": provider,
|
||||
"llm_model": os.environ.get("HINDSIGHT_API_LLM_MODEL"),
|
||||
}
|
||||
|
||||
@@ -140,7 +145,9 @@ PROVIDER_API_KEYS = {
|
||||
"gemini": "GEMINI_API_KEY",
|
||||
"ollama": None,
|
||||
"vertexai": None,
|
||||
"github-copilot": None,
|
||||
}
|
||||
NO_API_KEY_PROVIDERS = frozenset(provider for provider, key_env in PROVIDER_API_KEYS.items() if key_env is None)
|
||||
|
||||
|
||||
def do_configure(args):
|
||||
@@ -194,29 +201,31 @@ def _has_non_interactive_env() -> bool:
|
||||
"""Whether the env vars required by _do_configure_from_env are already set.
|
||||
|
||||
Returns True when an API key is present, OR when the provider is one that
|
||||
doesn't need a key (ollama, vertexai — the latter authenticates via a
|
||||
service-account file path). Prevents the interactive prompt from kicking
|
||||
in when the user clearly wants CI/scripted behavior.
|
||||
doesn't need a key. Prevents the interactive prompt from kicking in when
|
||||
the user clearly wants CI/scripted behavior.
|
||||
"""
|
||||
if os.environ.get("HINDSIGHT_API_LLM_API_KEY") or os.environ.get("OPENAI_API_KEY"):
|
||||
return True
|
||||
return os.environ.get("HINDSIGHT_API_LLM_PROVIDER") in ("ollama", "vertexai")
|
||||
return os.environ.get("HINDSIGHT_API_LLM_PROVIDER") in NO_API_KEY_PROVIDERS
|
||||
|
||||
|
||||
def _do_configure_from_env():
|
||||
"""Non-interactive configuration from environment variables (for CI)."""
|
||||
# Check for required environment variables
|
||||
api_key = os.environ.get("HINDSIGHT_API_LLM_API_KEY") or os.environ.get("OPENAI_API_KEY")
|
||||
provider = os.environ.get("HINDSIGHT_API_LLM_PROVIDER", "openai")
|
||||
api_key = (
|
||||
None
|
||||
if provider in NO_API_KEY_PROVIDERS
|
||||
else os.environ.get("HINDSIGHT_API_LLM_API_KEY") or os.environ.get("OPENAI_API_KEY")
|
||||
)
|
||||
|
||||
# Don't gate on PROVIDER_API_KEYS — that's only the interactive-menu set
|
||||
# (5 entries). hindsight-api's PROVIDER_DEFAULT_MODELS supports ~18
|
||||
# Don't gate on PROVIDER_API_KEYS — that's only the small interactive-menu
|
||||
# set. hindsight-api's PROVIDER_DEFAULT_MODELS supports many more
|
||||
# providers (anthropic, claude-code, bedrock, openrouter, ...). Let the
|
||||
# daemon validate; rejecting here would block valid configurations.
|
||||
|
||||
# Check for API key (required for non-ollama and non-vertexai providers)
|
||||
# vertexai uses GCP service account credentials instead of an API key
|
||||
if not api_key and provider not in ("ollama", "vertexai"):
|
||||
# These providers authenticate locally rather than through an LLM API key.
|
||||
if not api_key and provider not in NO_API_KEY_PROVIDERS:
|
||||
print("Error: Cannot run interactive configuration without a terminal.", file=sys.stderr)
|
||||
print("", file=sys.stderr)
|
||||
print("For non-interactive (CI) mode, set environment variables:", file=sys.stderr)
|
||||
@@ -373,6 +382,7 @@ def _do_configure_interactive(profile_name: str | None = None, port: int | None
|
||||
("Groq (fast & free tier)", "groq"),
|
||||
("Google Gemini", "gemini"),
|
||||
("Ollama (local, no API key)", "ollama"),
|
||||
("GitHub Copilot (signed-in account, no LLM API key)", "github-copilot"),
|
||||
]
|
||||
|
||||
provider = _prompt_choice("Select your LLM provider:", providers, default=1)
|
||||
|
||||
@@ -2,11 +2,11 @@
|
||||
# Copy this file to .env and fill in your values
|
||||
|
||||
# LLM Configuration (Required)
|
||||
# Supported providers: openai, openai-responses, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, zai, atlas, volcano
|
||||
# Supported providers: openai, openai-responses, groq, ollama, gemini, anthropic, lmstudio, vertexai, minimax, deepseek, zai, atlas, volcano, openai-codex, claude-code, github-copilot
|
||||
HINDSIGHT_API_LLM_PROVIDER=openai
|
||||
HINDSIGHT_API_LLM_API_KEY=your-api-key-here
|
||||
HINDSIGHT_API_LLM_MODEL=gpt-4o-mini
|
||||
HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
|
||||
# HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
|
||||
# Reasoning effort for providers/models that support it. Examples: none, low, medium, high, xhigh.
|
||||
# Set it and the value is sent as given, whatever the model is called — use `none` to stop a
|
||||
# self-hosted reasoning model (vLLM, Ollama, llama.cpp, TGI) emitting thinking blocks. Unset,
|
||||
@@ -74,6 +74,11 @@ HINDSIGHT_API_LLM_BASE_URL=https://api.openai.com/v1
|
||||
# HINDSIGHT_API_LLM_API_KEY=your-anthropic-api-key
|
||||
# HINDSIGHT_API_LLM_MODEL=claude-sonnet-4-20250514
|
||||
|
||||
# Example: GitHub Copilot subscription via the official Copilot SDK
|
||||
# Sign in with Copilot CLI first; no HINDSIGHT_API_LLM_API_KEY is needed.
|
||||
# HINDSIGHT_API_LLM_PROVIDER=github-copilot
|
||||
# HINDSIGHT_API_LLM_MODEL=gpt-5.6-terra
|
||||
|
||||
# Example: Google Vertex AI configuration
|
||||
# HINDSIGHT_API_LLM_PROVIDER=vertexai
|
||||
# HINDSIGHT_API_LLM_MODEL=google/gemini-2.0-flash-001
|
||||
|
||||
@@ -594,7 +594,7 @@ def test_configure_from_env_accepts_providers_outside_interactive_menu(temp_home
|
||||
"""Regression test for issue #1360.
|
||||
|
||||
`_do_configure_from_env` previously rejected any provider not in the small
|
||||
interactive-menu set (`PROVIDER_API_KEYS` — 5 entries) with "Unknown
|
||||
interactive-menu set (`PROVIDER_API_KEYS`) with "Unknown
|
||||
provider". hindsight-api supports ~18 providers (anthropic, claude-code,
|
||||
bedrock, openrouter, ...), so the gate blocked valid CI configurations.
|
||||
Validation belongs in the daemon, not in the CLI's UX-only menu list.
|
||||
@@ -616,6 +616,29 @@ def test_configure_from_env_accepts_providers_outside_interactive_menu(temp_home
|
||||
assert "HINDSIGHT_API_LLM_PROVIDER=anthropic" in contents
|
||||
|
||||
|
||||
def test_configure_from_env_accepts_github_copilot_without_api_key(temp_home, monkeypatch):
|
||||
"""GitHub Copilot authenticates through Copilot CLI rather than an LLM API key."""
|
||||
from hindsight_embed import cli
|
||||
|
||||
config_dir = temp_home / ".hindsight"
|
||||
monkeypatch.setattr(cli, "CONFIG_DIR", config_dir)
|
||||
monkeypatch.setattr(cli, "CONFIG_FILE", config_dir / "embed")
|
||||
|
||||
monkeypatch.setenv("HINDSIGHT_API_LLM_PROVIDER", "github-copilot")
|
||||
monkeypatch.setenv("HINDSIGHT_API_LLM_MODEL", "gpt-5.6-terra")
|
||||
monkeypatch.delenv("HINDSIGHT_API_LLM_API_KEY", raising=False)
|
||||
monkeypatch.setenv("OPENAI_API_KEY", "unrelated-openai-key")
|
||||
|
||||
assert cli._has_non_interactive_env() is True
|
||||
assert cli._do_configure_from_env() == 0
|
||||
|
||||
contents = (config_dir / "embed").read_text()
|
||||
assert "HINDSIGHT_API_LLM_PROVIDER=github-copilot" in contents
|
||||
assert "HINDSIGHT_API_LLM_MODEL=gpt-5.6-terra" in contents
|
||||
active_lines = [line for line in contents.splitlines() if line and not line.startswith("#")]
|
||||
assert not any(line.startswith("HINDSIGHT_API_LLM_API_KEY=") for line in active_lines)
|
||||
|
||||
|
||||
def _windows_scripts_dir(tmp_path: Path, *, with_pythonw: bool) -> Path:
|
||||
"""Build a fake Windows venv Scripts dir with hindsight-api.exe.
|
||||
|
||||
|
||||
@@ -23,10 +23,11 @@ PROVIDER_DETECTION = [
|
||||
{"name": "ollama", "key_env": ""},
|
||||
{"name": "openai-codex", "key_env": ""},
|
||||
{"name": "claude-code", "key_env": ""},
|
||||
{"name": "github-copilot", "key_env": ""},
|
||||
]
|
||||
|
||||
# Providers that don't require an API key
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code"}
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code", "github-copilot"}
|
||||
|
||||
|
||||
def _find_provider(name):
|
||||
|
||||
@@ -23,10 +23,11 @@ PROVIDER_DETECTION = [
|
||||
{"name": "ollama", "key_env": ""},
|
||||
{"name": "openai-codex", "key_env": ""},
|
||||
{"name": "claude-code", "key_env": ""},
|
||||
{"name": "github-copilot", "key_env": ""},
|
||||
]
|
||||
|
||||
# Providers that don't require an API key
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code"}
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code", "github-copilot"}
|
||||
|
||||
|
||||
def _find_provider(name):
|
||||
|
||||
@@ -23,9 +23,10 @@ PROVIDER_DETECTION = [
|
||||
{"name": "ollama", "key_env": ""},
|
||||
{"name": "openai-codex", "key_env": ""},
|
||||
{"name": "claude-code", "key_env": ""},
|
||||
{"name": "github-copilot", "key_env": ""},
|
||||
]
|
||||
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code"}
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code", "github-copilot"}
|
||||
|
||||
|
||||
def _find_provider(name):
|
||||
|
||||
@@ -23,9 +23,10 @@ PROVIDER_DETECTION = [
|
||||
{"name": "ollama", "key_env": ""},
|
||||
{"name": "openai-codex", "key_env": ""},
|
||||
{"name": "claude-code", "key_env": ""},
|
||||
{"name": "github-copilot", "key_env": ""},
|
||||
]
|
||||
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code"}
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code", "github-copilot"}
|
||||
|
||||
|
||||
def _find_provider(name):
|
||||
|
||||
@@ -23,10 +23,11 @@ PROVIDER_DETECTION = [
|
||||
{"name": "ollama", "key_env": ""},
|
||||
{"name": "openai-codex", "key_env": ""},
|
||||
{"name": "claude-code", "key_env": ""},
|
||||
{"name": "github-copilot", "key_env": ""},
|
||||
]
|
||||
|
||||
# Providers that don't require an API key
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code"}
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code", "github-copilot"}
|
||||
|
||||
|
||||
def _find_provider(name):
|
||||
|
||||
@@ -1270,7 +1270,12 @@ export function formatMemories(results: MemoryResult[]): string {
|
||||
}
|
||||
|
||||
// Providers that authenticate via OAuth or run locally — no API key needed.
|
||||
const NO_KEY_REQUIRED_PROVIDERS = new Set(["ollama", "openai-codex", "claude-code"]);
|
||||
const NO_KEY_REQUIRED_PROVIDERS = new Set([
|
||||
"ollama",
|
||||
"openai-codex",
|
||||
"claude-code",
|
||||
"github-copilot",
|
||||
]);
|
||||
|
||||
export function detectLLMConfig(pluginConfig?: PluginConfig): {
|
||||
provider?: string;
|
||||
|
||||
@@ -352,6 +352,15 @@ describe("applyEmbeddedMode", () => {
|
||||
expect(pc.llmApiKey).toBeUndefined();
|
||||
});
|
||||
|
||||
it("omits llmApiKey for github-copilot", () => {
|
||||
const pc: Record<string, unknown> = {
|
||||
llmApiKey: { source: "env", provider: "default", id: "STALE" },
|
||||
};
|
||||
applyEmbeddedMode(pc, { llmProvider: "github-copilot" });
|
||||
expect(pc.llmProvider).toBe("github-copilot");
|
||||
expect(pc.llmApiKey).toBeUndefined();
|
||||
});
|
||||
|
||||
it("throws when a key-requiring provider is given without a key", () => {
|
||||
const pc: Record<string, unknown> = {};
|
||||
expect(() => applyEmbeddedMode(pc, { llmProvider: "openai" })).toThrow(
|
||||
|
||||
@@ -49,6 +49,7 @@ export type SetupMode = "cloud" | "api" | "embedded";
|
||||
|
||||
export const NO_KEY_PROVIDERS: ReadonlySet<string> = new Set([
|
||||
"claude-code",
|
||||
"github-copilot",
|
||||
"openai-codex",
|
||||
"ollama",
|
||||
]);
|
||||
|
||||
@@ -23,9 +23,10 @@ PROVIDER_DETECTION = [
|
||||
{"name": "ollama", "key_env": ""},
|
||||
{"name": "openai-codex", "key_env": ""},
|
||||
{"name": "claude-code", "key_env": ""},
|
||||
{"name": "github-copilot", "key_env": ""},
|
||||
]
|
||||
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code"}
|
||||
NO_KEY_REQUIRED = {"ollama", "openai-codex", "claude-code", "github-copilot"}
|
||||
|
||||
|
||||
def _find_provider(name):
|
||||
|
||||
@@ -213,8 +213,8 @@ For non-English banks (especially CJK) and the language/extraction-language trad
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | Provider: `openai`, `openai-responses`, `openai-codex`, `claude-code`, `anthropic`, `gemini`, `groq`, `minimax`, `deepseek`, `zai`, `opencode-go`, `nous`, `xai-oauth`, `fireworks`, `ollama`, `ollama-cloud`, `lmstudio`, `llamacpp`, `vertexai`, `bedrock`, `litellm`, `litellmrouter`, `volcano`, `openrouter`, `requesty`, `none` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | API key for LLM provider | - |
|
||||
| `HINDSIGHT_API_LLM_PROVIDER` | Provider: `openai`, `openai-responses`, `openai-codex`, `claude-code`, `github-copilot`, `anthropic`, `gemini`, `groq`, `minimax`, `deepseek`, `zai`, `opencode-go`, `nous`, `xai-oauth`, `fireworks`, `ollama`, `ollama-cloud`, `lmstudio`, `llamacpp`, `vertexai`, `bedrock`, `litellm`, `litellmrouter`, `volcano`, `openrouter`, `requesty`, `none` | `openai` |
|
||||
| `HINDSIGHT_API_LLM_API_KEY` | API key for providers that require one; unused by `github-copilot` | - |
|
||||
| `HINDSIGHT_API_LLM_MODEL` | Model name | `gpt-5-mini` |
|
||||
| `HINDSIGHT_API_LLM_BASE_URL` | Custom LLM endpoint | Provider default |
|
||||
| `HINDSIGHT_API_LLM_MAX_CONCURRENT` | Max concurrent LLM requests | `32` |
|
||||
|
||||
@@ -11,7 +11,7 @@ Guide to setting up a local development environment for contributing to Hindsigh
|
||||
- Python 3.11+
|
||||
- [uv](https://docs.astral.sh/uv/) - Fast Python package manager
|
||||
- Docker and Docker Compose
|
||||
- An LLM API key (OpenAI, Groq, or Ollama)
|
||||
- An LLM provider credential, or a local/subscription-backed provider such as Ollama or GitHub Copilot
|
||||
|
||||
## Local Development Setup
|
||||
|
||||
|
||||
@@ -39,6 +39,7 @@ Used for fact extraction, entity resolution, mental model consolidation, and ans
|
||||
- Requesty
|
||||
- OpenAI Codex
|
||||
- Claude Code
|
||||
- GitHub Copilot
|
||||
- AWS Bedrock
|
||||
- Fireworks AI
|
||||
- Nous Portal
|
||||
@@ -121,6 +122,7 @@ Beyond basic generation, some providers support optional features that lower cos
|
||||
| Requesty (`requesty`) | — | — |
|
||||
| OpenAI Codex (`openai-codex`) | — | — |
|
||||
| Claude Code (`claude-code`) | — | — |
|
||||
| GitHub Copilot (`github-copilot`) | — | — |
|
||||
| AWS Bedrock (`bedrock`) | — | — |
|
||||
| Fireworks AI (`fireworks`) | ✅ | — |
|
||||
| Nous Portal (`nous`) | — | — |
|
||||
@@ -186,6 +188,7 @@ Each provider has a recommended default model that's used when `HINDSIGHT_API_LL
|
||||
| `requesty` | `openai/gpt-4o-mini` |
|
||||
| `openai-codex` | `gpt-5.4-mini` |
|
||||
| `claude-code` | `claude-sonnet-4-5-20250929` |
|
||||
| `github-copilot` | `gpt-5.6-terra` |
|
||||
| `bedrock` | `us.amazon.nova-2-lite-v1:0` |
|
||||
| `fireworks` | `accounts/fireworks/models/llama-v3p1-8b-instruct` |
|
||||
| `nous` | `deepseek/deepseek-v4-flash` |
|
||||
@@ -332,6 +335,34 @@ export HINDSIGHT_API_LLM_VERTEXAI_PROJECT_ID=your-gcp-project-id
|
||||
|
||||
---
|
||||
|
||||
### GitHub Copilot Setup
|
||||
|
||||
Use a GitHub Copilot subscription for Hindsight's extraction, consolidation, and reflection calls through the official GitHub Copilot SDK.
|
||||
|
||||
**Prerequisites:**
|
||||
- An active GitHub Copilot entitlement
|
||||
- Copilot CLI signed in under the same operating-system user that runs Hindsight
|
||||
|
||||
**Configure Hindsight:**
|
||||
|
||||
```bash
|
||||
export HINDSIGHT_API_LLM_PROVIDER=github-copilot
|
||||
export HINDSIGHT_API_LLM_MODEL=gpt-5.6-terra
|
||||
# No HINDSIGHT_API_LLM_API_KEY is needed.
|
||||
```
|
||||
|
||||
The provider starts one shared Copilot runtime for all Hindsight LLM lanes. That runtime uses a hook-free temporary `COPILOT_HOME` containing only the signed-in account selection; the credential itself is resolved by the runtime from the system keychain or an existing `gh` CLI login. Each call then uses an isolated transient session with repository instructions, skills, Copilot memory, built-in tools, and the cross-session store disabled. This prevents the memory integration from recursively retaining its own extraction calls.
|
||||
|
||||
For automation, the Copilot SDK also accepts `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`. With one of those set, Copilot CLI never has to have been run on the host, so containers and CI images work with no `~/.copilot` at all. GitHub Actions and server-to-server deployments must have the appropriate Copilot organization policy and token permissions.
|
||||
|
||||
**Important notes:**
|
||||
- Usage counts against the authenticated account or organization's Copilot allowance and AI Credits.
|
||||
- `HINDSIGHT_API_LLM_BASE_URL` optionally points to an existing headless Copilot runtime, such as `http://127.0.0.1:4321`; it is not an LLM-provider endpoint for this provider.
|
||||
- GitHub-hosted Copilot sessions do not expose temperature or maximum-output-token controls through the SDK, so those Hindsight settings are not applied.
|
||||
- Embeddings and reranking continue to use Hindsight's separately configured providers; their local defaults require no API key.
|
||||
|
||||
---
|
||||
|
||||
### OpenAI Codex Setup (ChatGPT Plus/Pro)
|
||||
|
||||
Use your ChatGPT Plus or Pro subscription for Hindsight without separate OpenAI Platform API costs.
|
||||
|
||||
@@ -89,6 +89,7 @@ Browse all supported integrations in the Integrations Hub.
|
||||
- Requesty
|
||||
- OpenAI Codex
|
||||
- Claude Code
|
||||
- GitHub Copilot
|
||||
- AWS Bedrock
|
||||
- Fireworks AI
|
||||
- Nous Portal
|
||||
|
||||
@@ -1448,6 +1448,19 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl", hash = "sha256:67073e15955400952c6565cc3e707c554a4eea2e428946f7a4c162fab9bd9bcf", size = 62794, upload-time = "2025-01-02T07:20:43.624Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "github-copilot-sdk"
|
||||
version = "1.0.11"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "httpx" },
|
||||
{ name = "pydantic" },
|
||||
{ name = "python-dateutil" },
|
||||
]
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/67/ac/175cbb71fe637d963a885248a421040c9d500ea6390ed3b88bc68ecf51ca/github_copilot_sdk-1.0.11-py3-none-any.whl", hash = "sha256:6f664c7b843c34ab5a7455f7effb4d339eae75969d2b72f43c2e6214c9c637dc", size = 486719, upload-time = "2026-08-14T16:12:20.01Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "gitpython"
|
||||
version = "3.1.59"
|
||||
@@ -1688,6 +1701,7 @@ dependencies = [
|
||||
{ name = "fastapi", extra = ["standard"] },
|
||||
{ name = "fastmcp" },
|
||||
{ name = "filelock" },
|
||||
{ name = "github-copilot-sdk" },
|
||||
{ name = "google-auth" },
|
||||
{ name = "google-genai" },
|
||||
{ name = "greenlet" },
|
||||
@@ -1819,6 +1833,7 @@ requires-dist = [
|
||||
{ name = "filelock", specifier = ">=3.20.1" },
|
||||
{ name = "filelock", marker = "extra == 'test'", specifier = ">=3.20.1" },
|
||||
{ name = "flashrank", marker = "extra == 'local-ml'", specifier = ">=0.2.0" },
|
||||
{ name = "github-copilot-sdk", specifier = ">=1.0.11" },
|
||||
{ name = "google-auth", specifier = ">=2.0.0" },
|
||||
{ name = "google-genai", specifier = ">=1.72.0" },
|
||||
{ name = "greenlet", specifier = ">=3.2.4,<3.4.0" },
|
||||
|
||||
Reference in New Issue
Block a user