mirror of
https://github.com/lingzhi227/agent-research-skills.git
synced 2026-09-20 09:33:06 +08:00
Add github-research skill (v1.1.0)
6-phase pipeline for discovering, evaluating, and deeply analyzing GitHub repositories related to a research topic. Reads deep-research output, discovers repos from multiple sources (GitHub search, code search, Papers With Code), scores and ranks them, clones and analyzes code, then produces cross-repo comparison and integration blueprints. 13 Python scripts (stdlib-only), 1 SKILL.md, 1 phase-guide reference.
This commit is contained in:
Lingzhi Yang
@@ -1,3 +1,4 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
github/
|
||||
|
||||
@@ -2,6 +2,20 @@
|
||||
|
||||
All notable changes to this project will be documented in this file.
|
||||
|
||||
## [1.1.0] - 2026-02-23
|
||||
|
||||
### Added
|
||||
|
||||
- **github-research skill**: 6-phase pipeline for discovering, evaluating, and deeply analyzing GitHub repositories related to a research topic
|
||||
- Phase 1: Intake — extract refs, URLs, keywords from deep-research output
|
||||
- Phase 2: Discovery — multi-source GitHub search (repos, code, Papers With Code)
|
||||
- Phase 3: Filtering — score, rank, and select top repos
|
||||
- Phase 4: Deep Dive — clone and deeply analyze code structure, dependencies, implementations
|
||||
- Phase 5: Analysis — cross-repo comparison matrix, technique mapping, gap analysis
|
||||
- Phase 6: Blueprint — integration plan, reusable components catalog
|
||||
- **13 Python scripts** for github-research: `extract_research_refs.py`, `search_github.py`, `search_github_code.py`, `search_paperswithcode.py`, `repo_db.py`, `repo_metadata.py`, `repo_readme_fetch.py`, `clone_repo.py`, `analyze_repo_structure.py`, `extract_dependencies.py`, `find_implementations.py`, `compare_repos.py`, `compile_github_report.py`
|
||||
- **Phase guide reference** (`references/phase-guide.md`) with search strategy matrix and per-repo analysis templates
|
||||
|
||||
## [1.0.0] - 2026-02-20
|
||||
|
||||
### Added
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# agent-research-skills
|
||||
|
||||
30 skills for [Claude Code](https://claude.ai/code) covering the full academic research paper lifecycle — from literature search to slide generation.
|
||||
31 skills for [Claude Code](https://claude.ai/code) covering the full academic research paper lifecycle — from literature search to slide generation — plus GitHub repository analysis for research topics.
|
||||
|
||||
Extracted from 17 GitHub repos studying LLM-agent-driven research automation. See [SKILLS_DESIGN.md](/Users/lingzhi/Code/research-engine/SKILLS_DESIGN.md) for the original design specifications.
|
||||
|
||||
@@ -40,12 +40,13 @@ This installs slash commands, checks Python dependencies, and verifies script sy
|
||||
|
||||
2. **Output directory**: Deep research outputs go to `~/deep-research-output/` by default.
|
||||
|
||||
## Available Skills (30)
|
||||
## Available Skills (31)
|
||||
|
||||
### Phase 0: Research Discovery & Planning
|
||||
|
||||
| Skill | Description | Scripts |
|
||||
|-------|-------------|---------|
|
||||
| **[github-research](skills/github-research/)** | 6-phase GitHub repo discovery, analysis, and integration planning for research topics | 13 scripts: search, clone, analyze structure/deps/impls, compare, compile report |
|
||||
| **[deep-research](skills/deep-research/)** | 6-phase systematic literature survey (frontier → survey → deep dive → code → synthesis → report) | 7 scripts: search APIs, PDF extraction, paper DB, BibTeX, report compilation |
|
||||
| **[literature-search](skills/literature-search/)** | Multi-source academic search (Semantic Scholar, arXiv, OpenAlex, CrossRef) with ranking | 4 scripts: `search_crossref.py`, `download_arxiv_source.py`, `search_openalex.py` + shared |
|
||||
| **[literature-review](skills/literature-review/)** | Multi-perspective dialogue simulation with expert personas for grounded literature review | Shares search scripts |
|
||||
@@ -118,6 +119,7 @@ This installs slash commands, checks Python dependencies, and verifies script sy
|
||||
### Natural language (skills activate automatically)
|
||||
|
||||
```
|
||||
"Analyze GitHub repos for multi-agent coordination research"
|
||||
"Do a literature review on protein folding with LLMs"
|
||||
"Write the Methods section of my paper"
|
||||
"Generate a comparison table from results.json"
|
||||
@@ -130,6 +132,12 @@ This installs slash commands, checks Python dependencies, and verifies script sy
|
||||
### Direct script usage
|
||||
|
||||
```bash
|
||||
# Search GitHub repos for a research topic
|
||||
python ~/.claude/skills/github-research/scripts/search_github.py --query "multi-agent LLM coordination" --max-results 50 --output repos.jsonl
|
||||
|
||||
# Analyze a cloned repo's structure
|
||||
python ~/.claude/skills/github-research/scripts/analyze_repo_structure.py --repo-dir ./my-repo --output analysis.json
|
||||
|
||||
# Search literature
|
||||
python ~/.claude/skills/literature-search/scripts/search_crossref.py --query "attention mechanism" --rows 10
|
||||
|
||||
|
||||
@@ -0,0 +1,414 @@
|
||||
---
|
||||
name: github-research
|
||||
description: Explore and analyze GitHub repositories related to a research topic. Reads deep-research output, discovers repos from multiple sources, deeply analyzes code, and produces integration blueprints.
|
||||
argument-hint: [deep-research-output-dir]
|
||||
---
|
||||
|
||||
# GitHub Research Skill
|
||||
|
||||
## Trigger
|
||||
|
||||
Activate this skill when the user wants to:
|
||||
- "Find repos for [topic]", "GitHub research on [topic]"
|
||||
- "Analyze open-source code for [topic]"
|
||||
- "Find implementations of [paper/technique]"
|
||||
- "Which repos implement [algorithm]?"
|
||||
- Uses `/github-research <deep-research-output-dir>` slash command
|
||||
|
||||
## Overview
|
||||
|
||||
This skill systematically discovers, evaluates, and deeply analyzes GitHub repositories related to a research topic. It reads **deep-research** output (paper database, phase reports, code references) and produces an actionable integration blueprint for reusing open-source code.
|
||||
|
||||
**Installation**: `~/.claude/skills/github-research/` — scripts, references, and this skill definition.
|
||||
**Output**: `./github-research-output/{slug}/` relative to the current working directory.
|
||||
**Input**: A deep-research output directory (containing `paper_db.jsonl`, phase reports, `code_repos.md`, etc.)
|
||||
|
||||
## 6-Phase Pipeline
|
||||
|
||||
```
|
||||
Phase 1: Intake → Extract refs, URLs, keywords from deep-research output
|
||||
Phase 2: Discovery → Multi-source broad GitHub search (50-200 repos)
|
||||
Phase 3: Filtering → Score & rank → select top 15-30 repos
|
||||
Phase 4: Deep Dive → Clone & deeply analyze top 8-15 repos (code reading)
|
||||
Phase 5: Analysis → Per-repo reports + cross-repo comparison
|
||||
Phase 6: Blueprint → Integration/reuse plan for research topic
|
||||
```
|
||||
|
||||
## Output Directory Structure
|
||||
|
||||
```
|
||||
github-research-output/{slug}/
|
||||
├── repo_db.jsonl # Master repo database
|
||||
├── phase1_intake/
|
||||
│ ├── extracted_refs.jsonl # URLs, keywords, paper-repo links
|
||||
│ └── intake_summary.md
|
||||
├── phase2_discovery/
|
||||
│ ├── search_results/ # Raw JSONL from each search
|
||||
│ └── discovery_log.md
|
||||
├── phase3_filtering/
|
||||
│ ├── ranked_repos.jsonl # Scored & ranked subset
|
||||
│ └── filtering_report.md
|
||||
├── phase4_deep_dive/
|
||||
│ ├── repos/ # Cloned repos (shallow)
|
||||
│ ├── analyses/ # Per-repo analysis .md files
|
||||
│ └── deep_dive_summary.md
|
||||
├── phase5_analysis/
|
||||
│ ├── comparison_matrix.md # Cross-repo comparison
|
||||
│ ├── technique_map.md # Paper concept → code mapping
|
||||
│ └── analysis_report.md
|
||||
└── phase6_blueprint/
|
||||
├── integration_plan.md # How to combine repos
|
||||
├── reuse_catalog.md # Reusable components catalog
|
||||
├── final_report.md # Complete compiled report
|
||||
└── blueprint_summary.md
|
||||
```
|
||||
|
||||
## Scripts Reference
|
||||
|
||||
All scripts are Python 3, stdlib-only, located in `~/.claude/skills/github-research/scripts/`.
|
||||
|
||||
| Script | Purpose | Key Flags |
|
||||
|--------|---------|-----------|
|
||||
| `extract_research_refs.py` | Parse deep-research output for GitHub URLs, paper refs, keywords | `--research-dir`, `--output` |
|
||||
| `search_github.py` | Search GitHub repos via `gh api` | `--query`, `--language`, `--min-stars`, `--sort`, `--max-results`, `--topic`, `--output` |
|
||||
| `search_github_code.py` | Search GitHub code for implementations | `--query`, `--language`, `--filename`, `--max-results`, `--output` |
|
||||
| `search_paperswithcode.py` | Search Papers With Code for paper→repo mappings | `--paper-title`, `--arxiv-id`, `--query`, `--output` |
|
||||
| `repo_db.py` | JSONL repo database management | subcommands: `merge`, `filter`, `score`, `search`, `tag`, `stats`, `export`, `rank` |
|
||||
| `repo_metadata.py` | Fetch detailed metadata via `gh api` | `--repos`, `--input`, `--output`, `--delay` |
|
||||
| `clone_repo.py` | Shallow-clone repos for analysis | `--repo`, `--output-dir`, `--depth`, `--branch` |
|
||||
| `analyze_repo_structure.py` | Map file tree, key files, LOC stats | `--repo-dir`, `--output` |
|
||||
| `extract_dependencies.py` | Extract and parse dependency files | `--repo-dir`, `--output` |
|
||||
| `find_implementations.py` | Search cloned repo for specific code patterns | `--repo-dir`, `--patterns`, `--output` |
|
||||
| `repo_readme_fetch.py` | Fetch README without cloning | `--repos`, `--input`, `--output`, `--max-chars` |
|
||||
| `compare_repos.py` | Generate comparison matrix across repos | `--input`, `--output` |
|
||||
| `compile_github_report.py` | Assemble final report from all phases | `--topic-dir` |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Intake
|
||||
|
||||
**Goal**: Extract all relevant references, URLs, and keywords from the deep-research output.
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Create output directory structure**:
|
||||
```bash
|
||||
SLUG=$(echo "$TOPIC" | tr '[:upper:]' '[:lower:]' | tr ' ' '-' | tr -cd 'a-z0-9-')
|
||||
mkdir -p github-research-output/$SLUG/{phase1_intake,phase2_discovery/search_results,phase3_filtering,phase4_deep_dive/{repos,analyses},phase5_analysis,phase6_blueprint}
|
||||
```
|
||||
|
||||
2. **Extract references from deep-research output**:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/extract_research_refs.py \
|
||||
--research-dir <deep-research-output-dir> \
|
||||
--output github-research-output/$SLUG/phase1_intake/extracted_refs.jsonl
|
||||
```
|
||||
|
||||
3. **Review extracted refs**: Read the generated JSONL. Note:
|
||||
- GitHub URLs found directly in reports
|
||||
- Paper titles and arxiv IDs (for Papers With Code lookup)
|
||||
- Research keywords and themes (for GitHub search queries)
|
||||
|
||||
4. **Write intake summary**: Create `phase1_intake/intake_summary.md` with:
|
||||
- Number of direct GitHub URLs found
|
||||
- Number of papers with potential code links
|
||||
- Key research themes extracted
|
||||
- Planned search queries for Phase 2
|
||||
|
||||
### Checkpoint
|
||||
- `extracted_refs.jsonl` exists with entries
|
||||
- `intake_summary.md` written
|
||||
- Search strategy documented
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Discovery
|
||||
|
||||
**Goal**: Cast a wide net to find 50-200 candidate repos from multiple sources.
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Search by direct URLs**: Any GitHub URLs from Phase 1 → fetch metadata:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_metadata.py \
|
||||
--repos owner1/name1 owner2/name2 ... \
|
||||
--output github-research-output/$SLUG/phase2_discovery/search_results/direct_urls.jsonl
|
||||
```
|
||||
|
||||
2. **Search Papers With Code**: For each paper with an arxiv ID:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/search_paperswithcode.py \
|
||||
--arxiv-id 2401.12345 \
|
||||
--output github-research-output/$SLUG/phase2_discovery/search_results/pwc_2401.12345.jsonl
|
||||
```
|
||||
|
||||
3. **Search GitHub by keywords** (3-8 queries based on research themes):
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/search_github.py \
|
||||
--query "multi-agent LLM coordination" \
|
||||
--min-stars 10 --sort stars --max-results 50 \
|
||||
--output github-research-output/$SLUG/phase2_discovery/search_results/gh_query1.jsonl
|
||||
```
|
||||
|
||||
4. **Search GitHub code** (for specific implementations):
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/search_github_code.py \
|
||||
--query "class MultiAgentOrchestrator" \
|
||||
--language python --max-results 30 \
|
||||
--output github-research-output/$SLUG/phase2_discovery/search_results/code_query1.jsonl
|
||||
```
|
||||
|
||||
5. **Fetch READMEs** for repos that lack descriptions:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_readme_fetch.py \
|
||||
--input <repos.jsonl> \
|
||||
--output github-research-output/$SLUG/phase2_discovery/search_results/readmes.jsonl
|
||||
```
|
||||
|
||||
6. **Merge all results** into master database:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_db.py merge \
|
||||
--inputs github-research-output/$SLUG/phase2_discovery/search_results/*.jsonl \
|
||||
--output github-research-output/$SLUG/repo_db.jsonl
|
||||
```
|
||||
|
||||
7. **Write discovery log**: Create `phase2_discovery/discovery_log.md` with search queries used, results per source, total unique repos found.
|
||||
|
||||
### Rate Limits
|
||||
- GitHub search API: 30 requests/minute (authenticated)
|
||||
- Papers With Code API: No strict limit but be respectful (1 req/sec)
|
||||
- Add `--delay 1.0` to batch operations when needed
|
||||
|
||||
### Checkpoint
|
||||
- `repo_db.jsonl` populated with 50-200 repos
|
||||
- `discovery_log.md` with search details
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Filtering
|
||||
|
||||
**Goal**: Score and rank repos, select top 15-30 for deeper analysis.
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Enrich metadata** for all repos:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_metadata.py \
|
||||
--input github-research-output/$SLUG/repo_db.jsonl \
|
||||
--output github-research-output/$SLUG/repo_db.jsonl \
|
||||
--delay 0.5
|
||||
```
|
||||
|
||||
2. **Score repos** (quality + activity scores):
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_db.py score \
|
||||
--input github-research-output/$SLUG/repo_db.jsonl \
|
||||
--output github-research-output/$SLUG/repo_db.jsonl
|
||||
```
|
||||
|
||||
3. **LLM relevance scoring**: Read through the top ~50 repos (by quality_score) and assign `relevance_score` (0.0-1.0) based on:
|
||||
- Direct relevance to research topic
|
||||
- Implementation completeness
|
||||
- Code quality signals (from README, description)
|
||||
- Update the relevance scores:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_db.py tag \
|
||||
--input github-research-output/$SLUG/repo_db.jsonl \
|
||||
--ids owner/name --tags "relevance:0.85"
|
||||
```
|
||||
|
||||
4. **Compute composite scores and rank**:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_db.py score \
|
||||
--input github-research-output/$SLUG/repo_db.jsonl \
|
||||
--output github-research-output/$SLUG/repo_db.jsonl
|
||||
python ~/.claude/skills/github-research/scripts/repo_db.py rank \
|
||||
--input github-research-output/$SLUG/repo_db.jsonl \
|
||||
--output github-research-output/$SLUG/phase3_filtering/ranked_repos.jsonl \
|
||||
--by composite_score
|
||||
```
|
||||
|
||||
5. **Select top repos**: Filter to top 15-30:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/repo_db.py filter \
|
||||
--input github-research-output/$SLUG/phase3_filtering/ranked_repos.jsonl \
|
||||
--output github-research-output/$SLUG/phase3_filtering/ranked_repos.jsonl \
|
||||
--max-repos 30 --not-archived
|
||||
```
|
||||
|
||||
6. **Write filtering report**: Create `phase3_filtering/filtering_report.md`:
|
||||
- Stats before/after filtering
|
||||
- Score distributions
|
||||
- Top 30 repos with scores and rationale
|
||||
|
||||
### Scoring Formula
|
||||
```
|
||||
activity_score = sigmoid((days_since_push < 90) * 0.4 + has_recent_commits * 0.3 + open_issues_ratio * 0.3)
|
||||
quality_score = normalize(log(stars+1) * 0.3 + log(forks+1) * 0.2 + has_license * 0.15 + has_readme * 0.15 + not_archived * 0.2)
|
||||
composite_score = relevance * 0.4 + quality * 0.35 + activity * 0.25
|
||||
```
|
||||
|
||||
### Checkpoint
|
||||
- `ranked_repos.jsonl` with 15-30 repos
|
||||
- `filtering_report.md` with scoring details
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Deep Dive
|
||||
|
||||
**Goal**: Clone and deeply analyze the top 8-15 repos.
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Select repos for deep dive**: Take top 8-15 from ranked list.
|
||||
|
||||
2. **Clone each repo** (shallow):
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/clone_repo.py \
|
||||
--repo owner/name \
|
||||
--output-dir github-research-output/$SLUG/phase4_deep_dive/repos/
|
||||
```
|
||||
|
||||
3. **Analyze structure** for each cloned repo:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/analyze_repo_structure.py \
|
||||
--repo-dir github-research-output/$SLUG/phase4_deep_dive/repos/name/ \
|
||||
--output github-research-output/$SLUG/phase4_deep_dive/analyses/name_structure.json
|
||||
```
|
||||
|
||||
4. **Extract dependencies**:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/extract_dependencies.py \
|
||||
--repo-dir github-research-output/$SLUG/phase4_deep_dive/repos/name/ \
|
||||
--output github-research-output/$SLUG/phase4_deep_dive/analyses/name_deps.json
|
||||
```
|
||||
|
||||
5. **Find implementations**: Search for key algorithms/concepts from research:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/find_implementations.py \
|
||||
--repo-dir github-research-output/$SLUG/phase4_deep_dive/repos/name/ \
|
||||
--patterns "class Transformer" "def forward" "attention" \
|
||||
--output github-research-output/$SLUG/phase4_deep_dive/analyses/name_impls.jsonl
|
||||
```
|
||||
|
||||
6. **Deep code reading**: For each repo, READ the key source files identified by structure analysis. Write a per-repo analysis in `phase4_deep_dive/analyses/{name}_analysis.md`:
|
||||
- Architecture overview
|
||||
- Key algorithms implemented
|
||||
- Code quality assessment
|
||||
- API / interface design
|
||||
- Dependencies and requirements
|
||||
- Strengths and limitations
|
||||
- Reusability assessment (how easy to extract components)
|
||||
|
||||
7. **Write deep dive summary**: `phase4_deep_dive/deep_dive_summary.md`
|
||||
|
||||
### IMPORTANT: Actually Read Code
|
||||
Do NOT just summarize READMEs. You must:
|
||||
- Read the main source files (entry points, core modules)
|
||||
- Understand the actual implementation approach
|
||||
- Identify specific functions/classes that implement research concepts
|
||||
- Note code patterns, design decisions, and trade-offs
|
||||
|
||||
### Checkpoint
|
||||
- Repos cloned in `repos/`
|
||||
- Per-repo analysis files in `analyses/`
|
||||
- `deep_dive_summary.md` written
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Analysis
|
||||
|
||||
**Goal**: Cross-repo comparison and technique-to-code mapping.
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Generate comparison matrix**:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/compare_repos.py \
|
||||
--input github-research-output/$SLUG/phase4_deep_dive/analyses/ \
|
||||
--output github-research-output/$SLUG/phase5_analysis/comparison.json
|
||||
```
|
||||
|
||||
2. **Write comparison matrix**: Create `phase5_analysis/comparison_matrix.md`:
|
||||
- Table comparing repos across dimensions (language, LOC, stars, framework, license, tests)
|
||||
- Dependency overlap analysis
|
||||
- Strengths/weaknesses per repo
|
||||
|
||||
3. **Write technique map**: Create `phase5_analysis/technique_map.md`:
|
||||
- Map each paper concept / research technique → specific repo + file + function
|
||||
- Identify gaps (techniques with no implementation found)
|
||||
- Note alternative implementations of the same concept
|
||||
|
||||
4. **Write analysis report**: `phase5_analysis/analysis_report.md`:
|
||||
- Executive summary of findings
|
||||
- Key insights from code analysis
|
||||
- Recommendations for which repos to use for which purposes
|
||||
|
||||
### Checkpoint
|
||||
- `comparison_matrix.md` with repo comparison table
|
||||
- `technique_map.md` mapping concepts to code
|
||||
- `analysis_report.md` with findings
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Blueprint
|
||||
|
||||
**Goal**: Produce an actionable integration and reuse plan.
|
||||
|
||||
### Steps
|
||||
|
||||
1. **Write integration plan**: `phase6_blueprint/integration_plan.md`:
|
||||
- Recommended architecture for combining repos
|
||||
- Step-by-step integration approach
|
||||
- Dependency resolution strategy
|
||||
- Potential conflicts and how to resolve them
|
||||
|
||||
2. **Write reuse catalog**: `phase6_blueprint/reuse_catalog.md`:
|
||||
- For each reusable component: source repo, file path, function/class, what it does, how to extract it
|
||||
- License compatibility matrix
|
||||
- Effort estimates (easy/medium/hard to integrate)
|
||||
|
||||
3. **Compile final report**:
|
||||
```bash
|
||||
python ~/.claude/skills/github-research/scripts/compile_github_report.py \
|
||||
--topic-dir github-research-output/$SLUG/
|
||||
```
|
||||
|
||||
4. **Write blueprint summary**: `phase6_blueprint/blueprint_summary.md`:
|
||||
- One-page executive summary
|
||||
- Top 5 repos and why
|
||||
- Recommended next steps
|
||||
|
||||
### Checkpoint
|
||||
- `integration_plan.md` complete
|
||||
- `reuse_catalog.md` with component catalog
|
||||
- `final_report.md` compiled
|
||||
- `blueprint_summary.md` as executive summary
|
||||
|
||||
---
|
||||
|
||||
## Quality Conventions
|
||||
|
||||
1. **Repos are ranked by composite score**: `relevance × 0.4 + quality × 0.35 + activity × 0.25`
|
||||
2. **Deep dive requires reading actual code**, not just READMEs
|
||||
3. **Integration blueprint must map paper concepts → specific code files/functions**
|
||||
4. **Incremental saves**: Each phase writes to disk immediately
|
||||
5. **Checkpoint recovery**: Can resume from any phase by checking what outputs exist
|
||||
6. **All scripts are stdlib-only Python** — no pip installs needed
|
||||
7. **`gh` CLI is required** for GitHub API access (must be authenticated)
|
||||
8. **Deduplication** by `repo_id` (owner/name) across all searches
|
||||
9. **Rate limit awareness**: Respect GitHub search API limits (30 req/min)
|
||||
|
||||
## Error Handling
|
||||
|
||||
- If `gh` is not installed: warn user and provide installation instructions
|
||||
- If a repo is archived/deleted: skip gracefully, note in log
|
||||
- If clone fails: skip, note in log, continue with remaining repos
|
||||
- If Papers With Code API is down: skip, rely on GitHub search only
|
||||
- Always write partial progress to disk so work is not lost
|
||||
|
||||
## References
|
||||
|
||||
- See `references/phase-guide.md` for detailed phase execution guidance
|
||||
- Deep-research skill: `~/.claude/skills/deep-research/SKILL.md`
|
||||
- Paper database pattern: `~/.claude/skills/deep-research/scripts/paper_db.py`
|
||||
@@ -0,0 +1,215 @@
|
||||
# GitHub Research — Phase Guide
|
||||
|
||||
Detailed methodology reference for the github-research skill.
|
||||
|
||||
## Phase 1: Intake — Detailed Guide
|
||||
|
||||
### Purpose
|
||||
Extract structured information from deep-research output to seed GitHub discovery.
|
||||
|
||||
### Input Requirements
|
||||
- Deep-research output directory containing:
|
||||
- `paper_db.jsonl` (required)
|
||||
- `phase4_code/code_repos.md` (optional but valuable)
|
||||
- `phase5_synthesis/synthesis.md` (optional)
|
||||
- `phase6_report/report.md` (optional)
|
||||
|
||||
### Keyword Extraction Strategy
|
||||
- **Primary keywords**: From paper titles — extract 2-3 word technical phrases
|
||||
- **Secondary keywords**: From paper tags in paper_db.jsonl
|
||||
- **Tertiary keywords**: Method names, algorithm names, architecture names from synthesis
|
||||
- **Author-based**: Search for prolific authors' GitHub profiles
|
||||
|
||||
### Expected Output
|
||||
- 5-20 GitHub URLs directly from papers
|
||||
- 10-30 search keywords of varying specificity
|
||||
- Clear mapping: which papers mention which repos
|
||||
|
||||
### Edge Cases
|
||||
- No code_repos.md: rely entirely on paper_db.jsonl keywords
|
||||
- No paper_db.jsonl: ask user for manual topic keywords
|
||||
- Non-English papers: extract English technical terms only
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Discovery — Detailed Guide
|
||||
|
||||
### Search Strategy Matrix
|
||||
| Strategy | Query Pattern | Sort | When to Use |
|
||||
|----------|--------------|------|-------------|
|
||||
| Broad topic | "multi-agent LLM framework" | stars | Always — establishes landscape |
|
||||
| Paper title | "{exact paper title}" | best-match | For each key paper |
|
||||
| Method name | "{algorithm name} implementation" | stars | For specific techniques |
|
||||
| Author search | "{author name}" + topic | updated | For prolific researchers |
|
||||
| Code pattern | "class {ClassName}" | - | For specific implementations |
|
||||
| Language-specific | topic + language:python | stars | When language matters |
|
||||
| Awesome list | "awesome-{topic}" | stars | To find curated lists |
|
||||
|
||||
### Rate Limiting
|
||||
- GitHub search API: 30 requests/minute (unauthenticated), 10 requests/minute (code search)
|
||||
- Papers With Code API: ~60 requests/minute
|
||||
- Always set GITHUB_TOKEN for higher limits (5000 req/hr)
|
||||
|
||||
### Deduplication
|
||||
- Primary key: `repo_id` (owner/name, case-insensitive)
|
||||
- When merging duplicates: keep record with more populated fields; merge paper_ids lists
|
||||
|
||||
### Target Numbers
|
||||
- Aim for 50-200 unique repos before filtering
|
||||
- Use at least 5 different search queries
|
||||
- Check Papers With Code for all papers with arxiv_ids
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Filtering — Detailed Guide
|
||||
|
||||
### Scoring Deep Dive
|
||||
|
||||
**Activity Score** (0-1):
|
||||
- Days since last push: <30d -> 0.9-1.0, 30-90d -> 0.6-0.8, 90-365d -> 0.3-0.5, >365d -> 0.0-0.2
|
||||
- Frequency weight: pushed_at recency matters most
|
||||
|
||||
**Quality Score** (0-1):
|
||||
- Stars (log-scaled, 30% weight): log(stars+1) normalized across set
|
||||
- Forks (log-scaled, 20%): log(forks+1) normalized
|
||||
- Has license (15%): any recognized license = 1.0
|
||||
- Not archived (20%): archived repos get 0
|
||||
- Has README (15%): non-empty readme_excerpt = 1.0
|
||||
|
||||
**Relevance Score** (0-1, manually assigned):
|
||||
- 0.9-1.0: Direct implementation of a paper in the literature review
|
||||
- 0.7-0.89: Closely related technique or framework
|
||||
- 0.5-0.69: Related but tangential (e.g., general ML framework used by papers)
|
||||
- 0.3-0.49: Loosely related (e.g., same domain, different approach)
|
||||
- 0.0-0.29: Unlikely useful
|
||||
|
||||
**Composite**: relevance x 0.4 + quality x 0.35 + activity x 0.25
|
||||
|
||||
### Selection Criteria
|
||||
- Always include: repos directly linked to papers
|
||||
- Prefer: repos with tests, documentation, active maintenance
|
||||
- Diversity: ensure mix of approaches, not just top-starred
|
||||
- Minimum: 15 repos; Maximum: 30 repos
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Deep Dive — Detailed Guide
|
||||
|
||||
### What "Deep Dive" Means
|
||||
This is NOT a README scan. You must:
|
||||
1. Clone the repo (shallow)
|
||||
2. Read the directory structure
|
||||
3. Open and read key source files (model definitions, training loops, core algorithms)
|
||||
4. Trace the execution flow from entry point to core logic
|
||||
5. Evaluate code quality, documentation, test coverage
|
||||
|
||||
### Per-Repo Analysis Template
|
||||
```markdown
|
||||
# {owner/name} — Deep Dive Analysis
|
||||
|
||||
## Overview
|
||||
- **Purpose**: {one-sentence}
|
||||
- **Stars**: {N} | **Language**: {lang} | **License**: {license}
|
||||
- **Last active**: {date} | **Composite score**: {score}
|
||||
|
||||
## Architecture
|
||||
- Entry point: `{file}` -> calls `{function}` -> uses `{module}`
|
||||
- Core modules: {list key files with purposes}
|
||||
- Data flow: {how data moves through the system}
|
||||
|
||||
## Key Algorithms
|
||||
- **{Algorithm 1}**: Implemented in `{file}:{lines}`, function `{name}`
|
||||
- Matches paper: {yes/no/partially} — {details}
|
||||
- **{Algorithm 2}**: ...
|
||||
|
||||
## Code Quality
|
||||
- Documentation: {poor/fair/good/excellent}
|
||||
- Test coverage: {none/minimal/moderate/comprehensive}
|
||||
- Code style: {consistent/inconsistent}, {patterns used}
|
||||
- Error handling: {minimal/adequate/thorough}
|
||||
|
||||
## Dependencies
|
||||
- Core: {list key deps}
|
||||
- ML framework: {pytorch/tensorflow/jax/none}
|
||||
- Hardware: {CPU only / GPU required / TPU supported}
|
||||
|
||||
## Reusability Assessment
|
||||
- Ease of extraction: {easy/moderate/difficult}
|
||||
- Tight couplings: {what's hard to separate}
|
||||
- API surface: {clean/messy}
|
||||
- Recommended components: {what to reuse}
|
||||
|
||||
## Limitations
|
||||
- {limitation 1}
|
||||
- {limitation 2}
|
||||
```
|
||||
|
||||
### Prioritization
|
||||
- Read model/algorithm files first (highest value)
|
||||
- Then training/evaluation scripts
|
||||
- Then configuration and entry points
|
||||
- Skip: CI config, linting config, CHANGELOG
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Analysis — Detailed Guide
|
||||
|
||||
### Comparison Matrix Dimensions
|
||||
Build a table comparing all deep-dived repos across:
|
||||
1. Primary language & ML framework
|
||||
2. Lines of code & file count
|
||||
3. Code quality rating (1-5)
|
||||
4. Paper fidelity (how closely it matches the paper)
|
||||
5. Reusability rating (1-5)
|
||||
6. Activity status (active/maintained/stale/archived)
|
||||
7. Dependency count & overlap
|
||||
8. License restrictiveness (permissive/copyleft/unknown)
|
||||
9. Hardware requirements (CPU/GPU/TPU)
|
||||
|
||||
### Technique Map Construction
|
||||
For each significant concept in the deep-research papers:
|
||||
1. Identify the concept (e.g., "multi-head attention", "reward shaping")
|
||||
2. Find implementations in analyzed repos
|
||||
3. Note the specific file, class/function, and line numbers
|
||||
4. Rate fidelity: faithful / modified / inspired-by / missing
|
||||
5. Note any improvements or deviations from the paper
|
||||
|
||||
### Gap Analysis
|
||||
Identify:
|
||||
- Paper concepts with NO implementation found
|
||||
- Implementations that deviate significantly from papers
|
||||
- Missing components needed for a complete system
|
||||
- Opportunities for novel contributions
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Blueprint — Detailed Guide
|
||||
|
||||
### Integration Plan Structure
|
||||
1. **Objective**: What system are we building?
|
||||
2. **Recommended Stack**: Best combination of repos
|
||||
3. **Architecture Diagram** (text-based): How repos fit together
|
||||
4. **Step-by-step Plan**:
|
||||
- Step 1: Start with {repo} as base
|
||||
- Step 2: Extract {component} from {repo}
|
||||
- Step 3: Adapt {module} to work with base
|
||||
- ...
|
||||
5. **Risk Assessment**: License conflicts, version incompatibilities, maintenance risk
|
||||
6. **Estimated Complexity**: Per-step effort estimate (trivial/moderate/significant)
|
||||
|
||||
### Reuse Catalog Format
|
||||
For each extractable component:
|
||||
- Source repo and file path
|
||||
- What it does
|
||||
- Dependencies required
|
||||
- How to extract (copy files, install deps, adapt imports)
|
||||
- API surface (key classes/functions, input/output types)
|
||||
- Caveats and gotchas
|
||||
|
||||
### Quality Checklist
|
||||
Before finalizing:
|
||||
- [ ] Every paper concept has a code mapping (or explicit "not found")
|
||||
- [ ] License compatibility checked for recommended stack
|
||||
- [ ] Dependency conflicts identified between repos
|
||||
- [ ] At least one "quick start" integration path identified
|
||||
- [ ] Gaps clearly documented with suggested approaches
|
||||
+243
@@ -0,0 +1,243 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Map repo file tree, identify key files, and compute LOC stats.
|
||||
|
||||
Usage:
|
||||
python analyze_repo_structure.py --repo-dir /tmp/repos/myrepo --output analysis.json [--repo-id owner/name]
|
||||
|
||||
Writes analysis JSON to the specified output file.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
# Directories to skip during traversal
|
||||
SKIP_DIRS = {
|
||||
".git", "node_modules", "__pycache__", ".venv", "venv",
|
||||
".tox", ".eggs", "dist", "build",
|
||||
}
|
||||
|
||||
# File extension to language mapping
|
||||
EXT_TO_LANG = {
|
||||
".py": "Python",
|
||||
".js": "JavaScript",
|
||||
".ts": "TypeScript",
|
||||
".rs": "Rust",
|
||||
".go": "Go",
|
||||
".java": "Java",
|
||||
".cpp": "C++",
|
||||
".cc": "C++",
|
||||
".cxx": "C++",
|
||||
".c": "C",
|
||||
".rb": "Ruby",
|
||||
".sh": "Shell",
|
||||
".yaml": "YAML",
|
||||
".yml": "YAML",
|
||||
".json": "JSON",
|
||||
".md": "Markdown",
|
||||
}
|
||||
|
||||
# Known config file names
|
||||
CONFIG_FILES = {
|
||||
"pyproject.toml", "setup.py", "setup.cfg", "package.json",
|
||||
"Cargo.toml", "go.mod", "Makefile", "Dockerfile",
|
||||
"docker-compose.yml", "docker-compose.yaml",
|
||||
".env.example", "tox.ini",
|
||||
}
|
||||
|
||||
# Known entry-point file names
|
||||
ENTRY_POINT_NAMES = {"main.py", "app.py", "run.py", "train.py", "setup.py", "Makefile", "Dockerfile"}
|
||||
|
||||
|
||||
def is_egg_info_dir(name: str) -> bool:
|
||||
return name.endswith(".egg-info")
|
||||
|
||||
|
||||
def count_loc(filepath: str) -> int:
|
||||
"""Count non-empty lines in a text file."""
|
||||
try:
|
||||
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
|
||||
return sum(1 for line in f if line.strip())
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return 0
|
||||
|
||||
|
||||
def has_shebang(filepath: str) -> bool:
|
||||
"""Check if file starts with a shebang line."""
|
||||
try:
|
||||
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
|
||||
first_line = f.readline()
|
||||
return first_line.startswith("#!")
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return False
|
||||
|
||||
|
||||
def has_main_guard(filepath: str) -> bool:
|
||||
"""Check if a Python file has `if __name__` guard."""
|
||||
try:
|
||||
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
if re.search(r'if\s+__name__\s*==\s*["\']__main__["\']', line):
|
||||
return True
|
||||
except (OSError, UnicodeDecodeError):
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def should_skip_dir(name: str) -> bool:
|
||||
return name in SKIP_DIRS or is_egg_info_dir(name)
|
||||
|
||||
|
||||
def build_tree(repo_dir: str, max_lines: int = 200) -> str:
|
||||
"""Generate an indented file tree string, truncated to max_lines."""
|
||||
lines = []
|
||||
base = os.path.basename(os.path.abspath(repo_dir))
|
||||
|
||||
for dirpath, dirnames, filenames in os.walk(repo_dir):
|
||||
# Filter out skipped dirs in-place
|
||||
dirnames[:] = sorted([d for d in dirnames if not should_skip_dir(d)])
|
||||
filenames = sorted(filenames)
|
||||
|
||||
rel = os.path.relpath(dirpath, repo_dir)
|
||||
depth = 0 if rel == "." else rel.count(os.sep) + 1
|
||||
indent = " " * depth
|
||||
|
||||
if rel == ".":
|
||||
lines.append(f"{base}/")
|
||||
else:
|
||||
lines.append(f"{indent}{os.path.basename(dirpath)}/")
|
||||
|
||||
if len(lines) >= max_lines:
|
||||
break
|
||||
|
||||
file_indent = " " * (depth + 1)
|
||||
for fname in filenames:
|
||||
lines.append(f"{file_indent}{fname}")
|
||||
if len(lines) >= max_lines:
|
||||
break
|
||||
if len(lines) >= max_lines:
|
||||
break
|
||||
|
||||
if len(lines) >= max_lines:
|
||||
lines.append(" ... (truncated)")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def analyze(repo_dir: str, repo_id: str) -> dict:
|
||||
"""Perform full repo structure analysis."""
|
||||
total_files = 0
|
||||
total_loc = 0
|
||||
languages: dict[str, int] = {}
|
||||
key_files: list[str] = []
|
||||
entry_points: list[str] = []
|
||||
config_files: list[str] = []
|
||||
test_files: list[str] = []
|
||||
doc_files: list[str] = []
|
||||
file_loc: list[tuple[str, int]] = []
|
||||
|
||||
print(f"[analyze] Scanning {repo_dir} ...", file=sys.stderr)
|
||||
|
||||
for dirpath, dirnames, filenames in os.walk(repo_dir):
|
||||
dirnames[:] = [d for d in dirnames if not should_skip_dir(d)]
|
||||
|
||||
rel_dir = os.path.relpath(dirpath, repo_dir)
|
||||
|
||||
for fname in filenames:
|
||||
filepath = os.path.join(dirpath, fname)
|
||||
rel_path = os.path.join(rel_dir, fname) if rel_dir != "." else fname
|
||||
# Normalize path separators
|
||||
rel_path = rel_path.replace(os.sep, "/")
|
||||
|
||||
total_files += 1
|
||||
|
||||
_, ext = os.path.splitext(fname)
|
||||
ext_lower = ext.lower()
|
||||
|
||||
# LOC counting for known languages
|
||||
lang = EXT_TO_LANG.get(ext_lower)
|
||||
loc = 0
|
||||
if lang:
|
||||
loc = count_loc(filepath)
|
||||
languages[lang] = languages.get(lang, 0) + loc
|
||||
total_loc += loc
|
||||
file_loc.append((rel_path, loc))
|
||||
|
||||
# Entry points detection
|
||||
if fname in ENTRY_POINT_NAMES:
|
||||
entry_points.append(rel_path)
|
||||
elif ext_lower == ".py" and has_main_guard(filepath):
|
||||
entry_points.append(rel_path)
|
||||
elif has_shebang(filepath):
|
||||
entry_points.append(rel_path)
|
||||
|
||||
# Config files
|
||||
if fname in CONFIG_FILES:
|
||||
config_files.append(rel_path)
|
||||
|
||||
# Test files
|
||||
if (fname.startswith("test_") or fname.endswith(("_test.py", "_test.js", "_test.ts", "_test.go"))
|
||||
or "/tests/" in f"/{rel_path}/" or "/test/" in f"/{rel_path}/"
|
||||
or "/spec/" in f"/{rel_path}/"):
|
||||
test_files.append(rel_path)
|
||||
|
||||
# Doc files
|
||||
if ext_lower == ".md" or "/docs/" in f"/{rel_path}/" or "/doc/" in f"/{rel_path}/":
|
||||
doc_files.append(rel_path)
|
||||
|
||||
# Sort files by LOC descending, take top 10
|
||||
file_loc.sort(key=lambda x: x[1], reverse=True)
|
||||
file_size_top10 = file_loc[:10]
|
||||
|
||||
# Key files = top LOC files + entry points
|
||||
key_set = set()
|
||||
for path, _ in file_size_top10:
|
||||
key_set.add(path)
|
||||
for path in entry_points:
|
||||
key_set.add(path)
|
||||
key_files = sorted(key_set)
|
||||
|
||||
# Sort languages by LOC descending
|
||||
languages = dict(sorted(languages.items(), key=lambda x: x[1], reverse=True))
|
||||
|
||||
tree = build_tree(repo_dir)
|
||||
|
||||
print(f"[analyze] Files: {total_files}, LOC: {total_loc}, Languages: {len(languages)}", file=sys.stderr)
|
||||
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"total_files": total_files,
|
||||
"total_loc": total_loc,
|
||||
"languages": languages,
|
||||
"key_files": key_files,
|
||||
"entry_points": sorted(set(entry_points)),
|
||||
"config_files": sorted(set(config_files)),
|
||||
"test_files": sorted(set(test_files)),
|
||||
"doc_files": sorted(set(doc_files)),
|
||||
"tree": tree,
|
||||
"file_size_top10": file_size_top10,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Analyze repository structure, files, and LOC stats.")
|
||||
parser.add_argument("--repo-dir", required=True, help="Path to the cloned repository")
|
||||
parser.add_argument("--output", required=True, help="Output JSON file path")
|
||||
parser.add_argument("--repo-id", default="unknown/unknown", help="Repository identifier (owner/name)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not os.path.isdir(args.repo_dir):
|
||||
print(f"Error: {args.repo_dir} is not a directory", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
result = analyze(args.repo_dir, args.repo_id)
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, indent=2, ensure_ascii=False)
|
||||
|
||||
print(f"[analyze] Output written to {args.output}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+109
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Shallow-clone GitHub repos for local code analysis.
|
||||
|
||||
Usage:
|
||||
python clone_repo.py --repo owner/name --output-dir /tmp/repos [--depth 1] [--branch main]
|
||||
|
||||
Outputs JSON to stdout with clone result; stats to stderr.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def get_dir_size(path: str) -> int:
|
||||
"""Walk directory and sum all file sizes in bytes."""
|
||||
total = 0
|
||||
for dirpath, _dirnames, filenames in os.walk(path):
|
||||
for f in filenames:
|
||||
fp = os.path.join(dirpath, f)
|
||||
try:
|
||||
total += os.path.getsize(fp)
|
||||
except OSError:
|
||||
pass
|
||||
return total
|
||||
|
||||
|
||||
def clone_repo(repo: str, output_dir: str, depth: int, branch: str | None) -> dict:
|
||||
"""Clone a GitHub repo and return result metadata."""
|
||||
parts = repo.split("/")
|
||||
if len(parts) != 2:
|
||||
return {
|
||||
"repo_id": repo,
|
||||
"local_path": None,
|
||||
"clone_success": False,
|
||||
"error": f"Invalid repo format '{repo}', expected 'owner/name'",
|
||||
}
|
||||
|
||||
owner, name = parts
|
||||
url = f"https://github.com/{owner}/{name}.git"
|
||||
target = os.path.join(output_dir, name)
|
||||
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
# Build clone command
|
||||
cmd = ["git", "clone", f"--depth={depth}", "--single-branch"]
|
||||
if branch:
|
||||
cmd += ["--branch", branch]
|
||||
cmd += [url, target]
|
||||
|
||||
print(f"[clone] Running: {' '.join(cmd)}", file=sys.stderr)
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
# If branch-specific clone fails, retry without --branch
|
||||
if result.returncode != 0 and branch:
|
||||
print(f"[clone] Branch '{branch}' failed, retrying with default branch...", file=sys.stderr)
|
||||
cmd_retry = ["git", "clone", f"--depth={depth}", "--single-branch", url, target]
|
||||
result = subprocess.run(cmd_retry, capture_output=True, text=True)
|
||||
|
||||
# Validate clone
|
||||
if result.returncode != 0:
|
||||
return {
|
||||
"repo_id": repo,
|
||||
"local_path": target,
|
||||
"clone_success": False,
|
||||
"error": result.stderr.strip(),
|
||||
}
|
||||
|
||||
if not os.path.isdir(target) or not os.listdir(target):
|
||||
return {
|
||||
"repo_id": repo,
|
||||
"local_path": target,
|
||||
"clone_success": False,
|
||||
"error": "Clone directory is empty or missing",
|
||||
}
|
||||
|
||||
# Compute clone size
|
||||
size_bytes = get_dir_size(target)
|
||||
size_mb = size_bytes / (1024 * 1024)
|
||||
file_count = sum(len(files) for _, _, files in os.walk(target))
|
||||
|
||||
print(f"[clone] Success: {target}", file=sys.stderr)
|
||||
print(f"[clone] Size: {size_mb:.1f} MB, Files: {file_count}", file=sys.stderr)
|
||||
|
||||
return {
|
||||
"repo_id": repo,
|
||||
"local_path": os.path.abspath(target),
|
||||
"clone_success": True,
|
||||
"error": None,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Shallow-clone a GitHub repo for local analysis.")
|
||||
parser.add_argument("--repo", required=True, help="Repository in owner/name format")
|
||||
parser.add_argument("--output-dir", required=True, help="Parent directory for the clone")
|
||||
parser.add_argument("--depth", type=int, default=1, help="Clone depth (default: 1)")
|
||||
parser.add_argument("--branch", default=None, help="Branch to clone (default: repo default branch)")
|
||||
args = parser.parse_args()
|
||||
|
||||
result = clone_repo(args.repo, args.output_dir, args.depth, args.branch)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+467
@@ -0,0 +1,467 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate comparison matrix across analyzed repositories.
|
||||
|
||||
Reads analysis JSON files and dependency files to produce a structured comparison.
|
||||
|
||||
Usage:
|
||||
python compare_repos.py --input phase4_deep_dive/analyses/ --output comparison.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from itertools import combinations
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# I/O helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def load_json(path: str) -> dict | None:
|
||||
"""Load a JSON file, returning None on failure."""
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except (OSError, json.JSONDecodeError) as exc:
|
||||
print(f" Warning: cannot load {path}: {exc}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
def find_analysis_files(analyses_dir: str) -> dict[str, dict[str, dict | None]]:
|
||||
"""Discover *_structure.json and *_deps.json files, grouped by repo name.
|
||||
|
||||
Returns {repo_name: {"structure": <dict|None>, "deps": <dict|None>}}.
|
||||
"""
|
||||
repos: dict[str, dict[str, dict | None]] = {}
|
||||
|
||||
for fname in sorted(os.listdir(analyses_dir)):
|
||||
fpath = os.path.join(analyses_dir, fname)
|
||||
if not os.path.isfile(fpath):
|
||||
continue
|
||||
|
||||
if fname.endswith("_structure.json"):
|
||||
name = fname.removesuffix("_structure.json")
|
||||
repos.setdefault(name, {"structure": None, "deps": None})
|
||||
repos[name]["structure"] = load_json(fpath)
|
||||
elif fname.endswith("_deps.json"):
|
||||
name = fname.removesuffix("_deps.json")
|
||||
repos.setdefault(name, {"structure": None, "deps": None})
|
||||
repos[name]["deps"] = load_json(fpath)
|
||||
|
||||
return repos
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dimension extractors
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _get(data: dict | None, *keys: str, default: object = None) -> object:
|
||||
"""Try multiple keys on a dict, returning the first truthy value."""
|
||||
if data is None:
|
||||
return default
|
||||
for key in keys:
|
||||
val = data.get(key)
|
||||
if val is not None and val != "" and val != [] and val != {}:
|
||||
return val
|
||||
return default
|
||||
|
||||
|
||||
def extract_repo_id(
|
||||
structure: dict | None, deps: dict | None, file_key: str,
|
||||
) -> str:
|
||||
"""Best-effort extraction of owner/name repo_id."""
|
||||
rid = _get(structure, "repo_id", "repo")
|
||||
if rid:
|
||||
return str(rid)
|
||||
rid = _get(deps, "repo_id", "repo")
|
||||
if rid:
|
||||
return str(rid)
|
||||
return file_key
|
||||
|
||||
|
||||
def extract_primary_language(structure: dict | None) -> str:
|
||||
"""Primary language from structure analysis."""
|
||||
lang = _get(structure, "primary_language", "language")
|
||||
if lang and isinstance(lang, str):
|
||||
return lang
|
||||
# Fall back: pick language with highest LOC
|
||||
langs = _get(structure, "languages", "language_stats", default={})
|
||||
if isinstance(langs, dict) and langs:
|
||||
return max(langs, key=lambda k: langs[k] if isinstance(langs[k], (int, float)) else 0)
|
||||
return "Unknown"
|
||||
|
||||
|
||||
def extract_total_loc(structure: dict | None) -> int:
|
||||
"""Total lines of code."""
|
||||
loc = _get(structure, "total_loc", "loc", default=0)
|
||||
if isinstance(loc, (int, float)):
|
||||
return int(loc)
|
||||
# Sum across languages
|
||||
langs = _get(structure, "languages", "language_stats", default={})
|
||||
if isinstance(langs, dict):
|
||||
return sum(v for v in langs.values() if isinstance(v, (int, float)))
|
||||
return 0
|
||||
|
||||
|
||||
def extract_total_files(structure: dict | None) -> int:
|
||||
"""Total file count."""
|
||||
count = _get(structure, "total_files", "file_count", default=0)
|
||||
if isinstance(count, (int, float)):
|
||||
return int(count)
|
||||
files = _get(structure, "files", "file_tree", default=[])
|
||||
if isinstance(files, list):
|
||||
return len(files)
|
||||
return 0
|
||||
|
||||
|
||||
def extract_stars(structure: dict | None) -> int:
|
||||
"""Star count."""
|
||||
return int(_get(structure, "stars", default=0) or 0)
|
||||
|
||||
|
||||
def extract_ml_frameworks(deps: dict | None) -> list[str]:
|
||||
"""Detect ML frameworks from dependency data."""
|
||||
if deps is None:
|
||||
return _get(deps, "ml_frameworks", default=[]) or [] # type: ignore[return-value]
|
||||
|
||||
# If the deps file already has ml_frameworks, use it
|
||||
existing = deps.get("ml_frameworks")
|
||||
if existing and isinstance(existing, list):
|
||||
return existing
|
||||
|
||||
return []
|
||||
|
||||
|
||||
def extract_has_tests(structure: dict | None) -> bool:
|
||||
"""Whether the repo has test files."""
|
||||
if structure is None:
|
||||
return False
|
||||
# Direct field
|
||||
test_files = _get(structure, "test_files", default=[])
|
||||
if isinstance(test_files, list) and test_files:
|
||||
return True
|
||||
# Check has_tests flag
|
||||
if structure.get("has_tests"):
|
||||
return True
|
||||
# Scan file list for test indicators
|
||||
files = _get(structure, "files", "file_tree", default=[])
|
||||
if isinstance(files, list):
|
||||
for f in files:
|
||||
name = f if isinstance(f, str) else (
|
||||
f.get("path", "") if isinstance(f, dict) else ""
|
||||
)
|
||||
name_lower = name.lower()
|
||||
if ("/test/" in name_lower or "/tests/" in name_lower
|
||||
or name_lower.startswith("test") or "test_" in name_lower):
|
||||
return True
|
||||
# Check key directories
|
||||
key_dirs = _get(structure, "key_dirs", "directories", default=[])
|
||||
if isinstance(key_dirs, list):
|
||||
for d in key_dirs:
|
||||
d_name = d if isinstance(d, str) else (
|
||||
d.get("name", "") if isinstance(d, dict) else ""
|
||||
)
|
||||
if d_name.lower() in ("test", "tests", "testing", "test_suite"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def extract_has_docker(structure: dict | None) -> bool:
|
||||
"""Whether the repo has Docker configuration."""
|
||||
if structure is None:
|
||||
return False
|
||||
config_files = _get(structure, "config_files", default=[])
|
||||
if isinstance(config_files, list):
|
||||
for f in config_files:
|
||||
name = f if isinstance(f, str) else (
|
||||
f.get("path", f.get("name", "")) if isinstance(f, dict) else ""
|
||||
)
|
||||
name_lower = name.lower()
|
||||
if name_lower in ("dockerfile", "docker-compose.yml",
|
||||
"docker-compose.yaml", ".dockerignore"):
|
||||
return True
|
||||
# Also check files list
|
||||
files = _get(structure, "files", "file_tree", default=[])
|
||||
if isinstance(files, list):
|
||||
for f in files:
|
||||
name = f if isinstance(f, str) else (
|
||||
f.get("path", "") if isinstance(f, dict) else ""
|
||||
)
|
||||
basename = os.path.basename(name).lower()
|
||||
if basename in ("dockerfile", "docker-compose.yml",
|
||||
"docker-compose.yaml"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def extract_license(structure: dict | None) -> str:
|
||||
"""License identifier."""
|
||||
return str(_get(structure, "license", default="") or "")
|
||||
|
||||
|
||||
def extract_entry_points(structure: dict | None) -> list[str]:
|
||||
"""Main entry points."""
|
||||
entries = _get(structure, "entry_points", "main_files", default=[])
|
||||
if isinstance(entries, list):
|
||||
return [str(e) for e in entries[:10]]
|
||||
return []
|
||||
|
||||
|
||||
def extract_config_format(structure: dict | None) -> str:
|
||||
"""Dominant config file format."""
|
||||
config_files = _get(structure, "config_files", default=[])
|
||||
if not isinstance(config_files, list):
|
||||
return ""
|
||||
|
||||
format_counts: dict[str, int] = {}
|
||||
ext_map: dict[str, str] = {
|
||||
".yaml": "yaml", ".yml": "yaml",
|
||||
".toml": "toml",
|
||||
".json": "json",
|
||||
".ini": "ini", ".cfg": "ini",
|
||||
}
|
||||
|
||||
for f in config_files:
|
||||
name = f if isinstance(f, str) else (
|
||||
f.get("path", f.get("name", "")) if isinstance(f, dict) else ""
|
||||
)
|
||||
_, ext = os.path.splitext(name.lower())
|
||||
fmt = ext_map.get(ext, "")
|
||||
if fmt:
|
||||
format_counts[fmt] = format_counts.get(fmt, 0) + 1
|
||||
|
||||
if not format_counts:
|
||||
return ""
|
||||
return max(format_counts, key=lambda k: format_counts[k])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dependency overlap
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def collect_all_packages(deps: dict | None) -> set[str]:
|
||||
"""Collect a flat set of normalized package names from a deps record."""
|
||||
if deps is None:
|
||||
return set()
|
||||
|
||||
# Prefer the pre-computed all_packages field
|
||||
all_pkgs = deps.get("all_packages")
|
||||
if isinstance(all_pkgs, list) and all_pkgs:
|
||||
return {p.lower() for p in all_pkgs}
|
||||
|
||||
# Otherwise, gather from per-ecosystem sections
|
||||
names: set[str] = set()
|
||||
|
||||
# Python
|
||||
py = deps.get("python")
|
||||
if isinstance(py, dict):
|
||||
for spec in py.get("requirements", []):
|
||||
if isinstance(spec, str):
|
||||
bare = spec.split(">=")[0].split("<=")[0].split("==")[0]
|
||||
bare = bare.split("~=")[0].split("!=")[0].split("[")[0].strip()
|
||||
if bare:
|
||||
names.add(bare.lower())
|
||||
|
||||
# Node
|
||||
node = deps.get("node")
|
||||
if isinstance(node, dict):
|
||||
for key in ("dependencies", "devDependencies"):
|
||||
val = node.get(key, {})
|
||||
if isinstance(val, dict):
|
||||
names.update(k.lower() for k in val)
|
||||
|
||||
# Rust
|
||||
rust = deps.get("rust")
|
||||
if isinstance(rust, dict):
|
||||
for key in ("dependencies",):
|
||||
val = rust.get(key, {})
|
||||
if isinstance(val, dict):
|
||||
names.update(k.lower() for k in val)
|
||||
|
||||
# Go
|
||||
go = deps.get("go")
|
||||
if isinstance(go, dict):
|
||||
for mod in go.get("modules", []):
|
||||
if isinstance(mod, str):
|
||||
parts = mod.rstrip("/").split("/")
|
||||
names.add(parts[-1].lower())
|
||||
|
||||
return names
|
||||
|
||||
|
||||
def compute_dependency_overlap(
|
||||
repo_packages: dict[str, set[str]],
|
||||
min_shared: int = 3,
|
||||
) -> list[dict]:
|
||||
"""Compute shared packages for each pair of repos.
|
||||
|
||||
Only includes pairs with at least min_shared common packages.
|
||||
"""
|
||||
overlaps: list[dict] = []
|
||||
repo_ids = sorted(repo_packages.keys())
|
||||
for a, b in combinations(repo_ids, 2):
|
||||
shared = sorted(repo_packages[a] & repo_packages[b])
|
||||
if len(shared) >= min_shared:
|
||||
overlaps.append({
|
||||
"repos": [a, b],
|
||||
"shared": shared,
|
||||
})
|
||||
return overlaps
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Generate comparison matrix across analyzed repositories",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--input", required=True,
|
||||
help="Directory containing *_structure.json and *_deps.json files",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output", required=True,
|
||||
help="Output comparison JSON file path",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repo-db", default=None,
|
||||
help="Optional repo_db.jsonl to enrich with stars/metadata",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not os.path.isdir(args.input):
|
||||
print(f"Error: analyses directory not found: {args.input}",
|
||||
file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Load optional repo_db for enrichment (stars, etc.)
|
||||
repo_db_map: dict[str, dict] = {}
|
||||
if args.repo_db and os.path.isfile(args.repo_db):
|
||||
with open(args.repo_db, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
rec = json.loads(line)
|
||||
rid = rec.get("repo_id", "")
|
||||
# Index by both full id and short name
|
||||
repo_db_map[rid] = rec
|
||||
repo_db_map[rid.split("/")[-1]] = rec
|
||||
|
||||
# Discover and load analysis files
|
||||
repos = find_analysis_files(args.input)
|
||||
if not repos:
|
||||
print("Error: no *_structure.json or *_deps.json files found",
|
||||
file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Found analysis files for {len(repos)} repo(s)", file=sys.stderr)
|
||||
|
||||
# --- Build dimension maps ---
|
||||
repo_ids: list[str] = []
|
||||
dim_primary_language: dict[str, str] = {}
|
||||
dim_total_loc: dict[str, int] = {}
|
||||
dim_total_files: dict[str, int] = {}
|
||||
dim_stars: dict[str, int] = {}
|
||||
dim_ml_framework: dict[str, list[str]] = {}
|
||||
dim_has_tests: dict[str, bool] = {}
|
||||
dim_has_docker: dict[str, bool] = {}
|
||||
dim_license: dict[str, str] = {}
|
||||
dim_entry_points: dict[str, list[str]] = {}
|
||||
dim_config_format: dict[str, str] = {}
|
||||
|
||||
repo_packages: dict[str, set[str]] = {}
|
||||
|
||||
for file_key, data in sorted(repos.items()):
|
||||
structure = data.get("structure")
|
||||
deps = data.get("deps")
|
||||
rid = extract_repo_id(structure, deps, file_key)
|
||||
repo_ids.append(rid)
|
||||
|
||||
dim_primary_language[rid] = extract_primary_language(structure)
|
||||
dim_total_loc[rid] = extract_total_loc(structure)
|
||||
dim_total_files[rid] = extract_total_files(structure)
|
||||
# Stars from structure.json, fallback to repo_db
|
||||
stars = extract_stars(structure)
|
||||
if stars == 0 and rid in repo_db_map:
|
||||
stars = int(repo_db_map[rid].get("stars", 0))
|
||||
if stars == 0:
|
||||
# Try matching by short name
|
||||
short = rid.split("/")[-1]
|
||||
if short in repo_db_map:
|
||||
stars = int(repo_db_map[short].get("stars", 0))
|
||||
dim_stars[rid] = stars
|
||||
dim_ml_framework[rid] = extract_ml_frameworks(deps)
|
||||
dim_has_tests[rid] = extract_has_tests(structure)
|
||||
dim_has_docker[rid] = extract_has_docker(structure)
|
||||
dim_license[rid] = extract_license(structure)
|
||||
dim_entry_points[rid] = extract_entry_points(structure)
|
||||
dim_config_format[rid] = extract_config_format(structure)
|
||||
|
||||
repo_packages[rid] = collect_all_packages(deps)
|
||||
|
||||
# --- Compute pairwise dependency overlap ---
|
||||
dependency_overlap = compute_dependency_overlap(repo_packages)
|
||||
|
||||
# --- Assemble output ---
|
||||
comparison: dict = {
|
||||
"repos": repo_ids,
|
||||
"dimensions": {
|
||||
"primary_language": dim_primary_language,
|
||||
"total_loc": dim_total_loc,
|
||||
"total_files": dim_total_files,
|
||||
"stars": dim_stars,
|
||||
"ml_framework": dim_ml_framework,
|
||||
"has_tests": dim_has_tests,
|
||||
"has_docker": dim_has_docker,
|
||||
"license": dim_license,
|
||||
"entry_points": dim_entry_points,
|
||||
"config_format": dim_config_format,
|
||||
},
|
||||
"dependency_overlap": dependency_overlap,
|
||||
}
|
||||
|
||||
# --- Write output ---
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
json.dump(comparison, f, indent=2, ensure_ascii=False)
|
||||
f.write("\n")
|
||||
|
||||
# --- Summary to stderr ---
|
||||
n_dims = len(comparison["dimensions"])
|
||||
lang_counts: dict[str, int] = {}
|
||||
for lang in dim_primary_language.values():
|
||||
lang_counts[lang] = lang_counts.get(lang, 0) + 1
|
||||
lang_summary = ", ".join(
|
||||
f"{l}({c})" for l, c in sorted(lang_counts.items(), key=lambda x: -x[1])
|
||||
)
|
||||
|
||||
fw_counts: dict[str, int] = {}
|
||||
for fws in dim_ml_framework.values():
|
||||
for fw in fws:
|
||||
fw_counts[fw] = fw_counts.get(fw, 0) + 1
|
||||
fw_summary = ", ".join(
|
||||
f"{f}({c})" for f, c in sorted(fw_counts.items(), key=lambda x: -x[1])
|
||||
) if fw_counts else "none"
|
||||
|
||||
test_count = sum(1 for v in dim_has_tests.values() if v)
|
||||
docker_count = sum(1 for v in dim_has_docker.values() if v)
|
||||
|
||||
print(f"Compared {len(repo_ids)} repos across {n_dims} dimensions",
|
||||
file=sys.stderr)
|
||||
print(f" Languages: {lang_summary}", file=sys.stderr)
|
||||
print(f" ML frameworks: {fw_summary}", file=sys.stderr)
|
||||
print(f" With tests: {test_count}/{len(repo_ids)}", file=sys.stderr)
|
||||
print(f" With Docker: {docker_count}/{len(repo_ids)}", file=sys.stderr)
|
||||
print(f" Dependency overlaps: {len(dependency_overlap)} pair(s)",
|
||||
file=sys.stderr)
|
||||
print(f" Output: {args.output}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+524
@@ -0,0 +1,524 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Assemble final GitHub research report from all phase outputs.
|
||||
|
||||
Reads repo_db.jsonl and phase outputs to generate statistics and a comprehensive report.
|
||||
|
||||
Usage:
|
||||
python compile_github_report.py --topic-dir ./github-research-output/my-topic
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# -- I/O helpers --------------------------------------------------------------
|
||||
|
||||
def load_jsonl(path: str) -> list[dict]:
|
||||
"""Load records from a JSONL file. Returns empty list if file missing."""
|
||||
records: list[dict] = []
|
||||
if not os.path.isfile(path):
|
||||
return records
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
try:
|
||||
records.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
return records
|
||||
|
||||
|
||||
def load_text(path: str) -> str:
|
||||
"""Load a text file. Returns empty string if file missing."""
|
||||
if not os.path.isfile(path):
|
||||
return ""
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return f.read()
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def format_table(headers: list[str], rows: list[list[str]]) -> str:
|
||||
"""Generate a markdown table from headers and row data."""
|
||||
if not headers:
|
||||
return ""
|
||||
lines: list[str] = []
|
||||
lines.append("| " + " | ".join(headers) + " |")
|
||||
# Alignment: right-align columns that look numeric in first data row
|
||||
seps: list[str] = []
|
||||
for i, h in enumerate(headers):
|
||||
if rows and i < len(rows[0]):
|
||||
try:
|
||||
float(str(rows[0][i]).replace(",", ""))
|
||||
seps.append("------:")
|
||||
except (ValueError, IndexError):
|
||||
seps.append("------")
|
||||
else:
|
||||
seps.append("------")
|
||||
lines.append("| " + " | ".join(seps) + " |")
|
||||
for row in rows:
|
||||
cells = [str(c) for c in row]
|
||||
while len(cells) < len(headers):
|
||||
cells.append("")
|
||||
lines.append("| " + " | ".join(cells) + " |")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def truncate(text: str, max_chars: int = 200) -> str:
|
||||
"""Truncate text with ellipsis if it exceeds max_chars."""
|
||||
if len(text) <= max_chars:
|
||||
return text
|
||||
return text[: max_chars - 3].rstrip() + "..."
|
||||
|
||||
|
||||
# -- Statistics ---------------------------------------------------------------
|
||||
|
||||
def compute_stats(repos: list[dict]) -> dict:
|
||||
"""Compute all statistics from repo records."""
|
||||
if not repos:
|
||||
return {
|
||||
"total_discovered": 0, "total_filtered": 0, "total_analyzed": 0,
|
||||
"sources": {}, "languages": {}, "stars": {},
|
||||
"activity": {}, "score_buckets": {}, "ml_frameworks": {}, "licenses": {},
|
||||
}
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# -- By source --
|
||||
sources: dict[str, int] = {}
|
||||
for rec in repos:
|
||||
src = rec.get("source", "unknown")
|
||||
sources[src] = sources.get(src, 0) + 1
|
||||
|
||||
# -- By language --
|
||||
languages: dict[str, int] = {}
|
||||
for rec in repos:
|
||||
lang = rec.get("language") or "Unknown"
|
||||
languages[lang] = languages.get(lang, 0) + 1
|
||||
languages = dict(sorted(languages.items(), key=lambda x: -x[1]))
|
||||
|
||||
# -- Stars distribution --
|
||||
star_values = [rec.get("stars", 0) or 0 for rec in repos]
|
||||
star_values_sorted = sorted(star_values)
|
||||
n = len(star_values_sorted)
|
||||
stars_info: dict[str, float | int] = {
|
||||
"min": star_values_sorted[0],
|
||||
"max": star_values_sorted[-1],
|
||||
"median": star_values_sorted[n // 2] if n % 2 else
|
||||
(star_values_sorted[n // 2 - 1] + star_values_sorted[n // 2]) / 2,
|
||||
"mean": round(sum(star_values) / n, 1),
|
||||
"p25": star_values_sorted[max(n // 4 - 1, 0)],
|
||||
"p75": star_values_sorted[min(3 * n // 4, n - 1)],
|
||||
}
|
||||
|
||||
# -- Activity --
|
||||
activity: dict[str, int] = {"last_90d": 0, "last_year": 0, "older": 0}
|
||||
for rec in repos:
|
||||
pushed_str = rec.get("pushed_at") or rec.get("updated_at") or ""
|
||||
if not pushed_str:
|
||||
activity["older"] += 1
|
||||
continue
|
||||
try:
|
||||
pushed = datetime.fromisoformat(pushed_str.replace("Z", "+00:00"))
|
||||
days = (now - pushed).days
|
||||
if days <= 90:
|
||||
activity["last_90d"] += 1
|
||||
elif days <= 365:
|
||||
activity["last_year"] += 1
|
||||
else:
|
||||
activity["older"] += 1
|
||||
except (ValueError, TypeError):
|
||||
activity["older"] += 1
|
||||
|
||||
# -- Composite score histogram --
|
||||
score_buckets: dict[str, int] = {
|
||||
"0.0-0.2": 0, "0.2-0.4": 0, "0.4-0.6": 0, "0.6-0.8": 0, "0.8-1.0": 0,
|
||||
}
|
||||
for rec in repos:
|
||||
cs = rec.get("composite_score", 0.0) or 0.0
|
||||
if cs >= 0.8:
|
||||
score_buckets["0.8-1.0"] += 1
|
||||
elif cs >= 0.6:
|
||||
score_buckets["0.6-0.8"] += 1
|
||||
elif cs >= 0.4:
|
||||
score_buckets["0.4-0.6"] += 1
|
||||
elif cs >= 0.2:
|
||||
score_buckets["0.2-0.4"] += 1
|
||||
else:
|
||||
score_buckets["0.0-0.2"] += 1
|
||||
|
||||
# -- ML frameworks --
|
||||
ml_keywords = {
|
||||
"torch": "PyTorch", "pytorch": "PyTorch",
|
||||
"tensorflow": "TensorFlow", "tf-": "TensorFlow",
|
||||
"jax": "JAX", "flax": "JAX",
|
||||
"keras": "Keras", "sklearn": "scikit-learn",
|
||||
"transformers": "HuggingFace Transformers",
|
||||
"lightning": "PyTorch Lightning", "paddle": "PaddlePaddle",
|
||||
}
|
||||
ml_frameworks: dict[str, int] = {}
|
||||
for rec in repos:
|
||||
topics = rec.get("topics", []) or []
|
||||
desc = (rec.get("description") or "").lower()
|
||||
readme = (rec.get("readme_excerpt") or "").lower()
|
||||
combined = desc + " " + readme + " " + " ".join(str(t) for t in topics)
|
||||
found_for_repo: set[str] = set()
|
||||
for kw, framework in ml_keywords.items():
|
||||
if kw in combined:
|
||||
found_for_repo.add(framework)
|
||||
for fw in found_for_repo:
|
||||
ml_frameworks[fw] = ml_frameworks.get(fw, 0) + 1
|
||||
ml_frameworks = dict(sorted(ml_frameworks.items(), key=lambda x: -x[1]))
|
||||
|
||||
# -- Licenses --
|
||||
licenses: dict[str, int] = {}
|
||||
for rec in repos:
|
||||
lic = rec.get("license") or "None/Unknown"
|
||||
if isinstance(lic, dict):
|
||||
lic = lic.get("spdx_id") or lic.get("name") or "None/Unknown"
|
||||
licenses[lic] = licenses.get(lic, 0) + 1
|
||||
licenses = dict(sorted(licenses.items(), key=lambda x: -x[1]))
|
||||
|
||||
# -- Counts for filtered / analyzed --
|
||||
scored_repos = [r for r in repos if r.get("composite_score")]
|
||||
analyzed_repos = [r for r in repos if r.get("tags") and "deep-dived" in r.get("tags", [])]
|
||||
# Fallback: count repos with rank field as proxy for filtered
|
||||
ranked_repos = [r for r in repos if r.get("rank")]
|
||||
|
||||
return {
|
||||
"total_discovered": len(repos),
|
||||
"total_filtered": len(ranked_repos) if ranked_repos else len(scored_repos),
|
||||
"total_analyzed": len(analyzed_repos),
|
||||
"num_sources": len(sources),
|
||||
"sources": sources,
|
||||
"languages": languages,
|
||||
"stars": stars_info,
|
||||
"activity": activity,
|
||||
"score_buckets": score_buckets,
|
||||
"ml_frameworks": ml_frameworks,
|
||||
"licenses": licenses,
|
||||
}
|
||||
|
||||
|
||||
# -- Report sections ----------------------------------------------------------
|
||||
|
||||
def build_header(topic_dir: str) -> str:
|
||||
"""Build the report header with topic name and date."""
|
||||
slug = os.path.basename(topic_dir.rstrip("/"))
|
||||
topic = slug.replace("-", " ").title()
|
||||
date = datetime.now().strftime("%Y-%m-%d %H:%M")
|
||||
return f"# GitHub Research Report: {topic}\n\nGenerated: {date}\n"
|
||||
|
||||
|
||||
def build_executive_summary(stats: dict, blueprint_summary: str) -> str:
|
||||
"""Build the executive summary section."""
|
||||
lines = ["## Executive Summary\n"]
|
||||
if blueprint_summary:
|
||||
lines.append(blueprint_summary.strip())
|
||||
else:
|
||||
total = stats["total_discovered"]
|
||||
ns = stats["num_sources"]
|
||||
filtered = stats["total_filtered"]
|
||||
analyzed = stats["total_analyzed"]
|
||||
top_lang = next(iter(stats["languages"]), "N/A")
|
||||
active = stats["activity"].get("last_90d", 0)
|
||||
lines.append(
|
||||
f"This report covers a systematic GitHub research effort that discovered "
|
||||
f"**{total} repositories** from **{ns} source(s)**. "
|
||||
f"After scoring and filtering, **{filtered}** repos were shortlisted "
|
||||
f"and **{analyzed or 'several'}** received deep code analysis. "
|
||||
f"The dominant language is **{top_lang}** and "
|
||||
f"**{active}** repos have been active in the last 90 days."
|
||||
)
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_discovery_stats(stats: dict) -> str:
|
||||
"""Build the discovery statistics section."""
|
||||
lines = ["## 1. Discovery Statistics\n"]
|
||||
lines.append(
|
||||
f"- Repos discovered: **{stats['total_discovered']}** "
|
||||
f"from **{stats['num_sources']}** sources"
|
||||
)
|
||||
lines.append(
|
||||
f"- After filtering: **{stats['total_filtered']}** repos "
|
||||
f"(top by composite score)"
|
||||
)
|
||||
lines.append(f"- Deep-dived: **{stats['total_analyzed']}** repos\n")
|
||||
|
||||
# Source table
|
||||
if stats["sources"]:
|
||||
lines.append("### Repos by Source\n")
|
||||
rows = [[src, str(cnt)] for src, cnt in
|
||||
sorted(stats["sources"].items(), key=lambda x: -x[1])]
|
||||
lines.append(format_table(["Source", "Count"], rows))
|
||||
lines.append("")
|
||||
|
||||
# Language table
|
||||
if stats["languages"]:
|
||||
lines.append("### Repos by Language\n")
|
||||
rows = [[lang, str(cnt)] for lang, cnt in list(stats["languages"].items())[:15]]
|
||||
lines.append(format_table(["Language", "Count"], rows))
|
||||
lines.append("")
|
||||
|
||||
# Stars distribution
|
||||
stars = stats.get("stars", {})
|
||||
if stars:
|
||||
lines.append("### Stars Distribution\n")
|
||||
lines.append(f"- Min: **{stars.get('min', 0)}** | "
|
||||
f"Max: **{stars.get('max', 0)}** | "
|
||||
f"Median: **{stars.get('median', 0)}** | "
|
||||
f"Mean: **{stars.get('mean', 0)}**")
|
||||
lines.append(f"- P25: **{stars.get('p25', 0)}** | P75: **{stars.get('p75', 0)}**\n")
|
||||
|
||||
# Score buckets
|
||||
if stats["score_buckets"]:
|
||||
lines.append("### Composite Score Distribution\n")
|
||||
rows = [[bucket, str(cnt)] for bucket, cnt in stats["score_buckets"].items()]
|
||||
lines.append(format_table(["Score Range", "Count"], rows))
|
||||
lines.append("")
|
||||
|
||||
# Activity
|
||||
if stats["activity"]:
|
||||
lines.append("### Activity\n")
|
||||
act = stats["activity"]
|
||||
lines.append(f"- Pushed in last 90 days: **{act.get('last_90d', 0)}**")
|
||||
lines.append(f"- Pushed in last year: **{act.get('last_year', 0)}**")
|
||||
lines.append(f"- Older / unknown: **{act.get('older', 0)}**\n")
|
||||
|
||||
# ML frameworks
|
||||
if stats["ml_frameworks"]:
|
||||
lines.append("### ML Frameworks\n")
|
||||
rows = [[fw, str(cnt)] for fw, cnt in stats["ml_frameworks"].items()]
|
||||
lines.append(format_table(["Framework", "Repos"], rows))
|
||||
lines.append("")
|
||||
|
||||
# Licenses
|
||||
if stats["licenses"]:
|
||||
lines.append("### License Distribution\n")
|
||||
rows = [[lic, str(cnt)] for lic, cnt in list(stats["licenses"].items())[:10]]
|
||||
lines.append(format_table(["License", "Count"], rows))
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_top_repos(ranked_repos: list[dict]) -> str:
|
||||
"""Build the top repositories table from ranked_repos.jsonl."""
|
||||
lines = ["## 2. Top Repositories\n"]
|
||||
if not ranked_repos:
|
||||
lines.append("_No ranked repos available._\n")
|
||||
return "\n".join(lines)
|
||||
|
||||
headers = ["Rank", "Repo", "Stars", "Language", "Composite", "Description"]
|
||||
rows: list[list[str]] = []
|
||||
for i, rec in enumerate(ranked_repos[:30], 1):
|
||||
rid = rec.get("repo_id", "")
|
||||
url = rec.get("url", f"https://github.com/{rid}")
|
||||
stars = str(rec.get("stars", 0))
|
||||
lang = rec.get("language", "")
|
||||
score = f"{(rec.get('composite_score', 0.0) or 0.0):.3f}"
|
||||
desc = truncate((rec.get("description") or "").replace("|", "/"), 60)
|
||||
rows.append([str(i), f"[{rid}]({url})", stars, lang, score, desc])
|
||||
|
||||
lines.append(format_table(headers, rows))
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_deep_analysis_summaries(analyses_dir: str) -> str:
|
||||
"""Build condensed summaries from per-repo analyses/*.md files."""
|
||||
lines = ["## 3. Deep Analysis Summaries\n"]
|
||||
if not os.path.isdir(analyses_dir):
|
||||
lines.append("_No deep analysis files found._\n")
|
||||
return "\n".join(lines)
|
||||
|
||||
analysis_files = sorted(
|
||||
f for f in os.listdir(analyses_dir) if f.endswith("_analysis.md")
|
||||
)
|
||||
if not analysis_files:
|
||||
lines.append("_No per-repo analysis files found._\n")
|
||||
return "\n".join(lines)
|
||||
|
||||
for fname in analysis_files:
|
||||
content = load_text(os.path.join(analyses_dir, fname))
|
||||
if not content:
|
||||
continue
|
||||
# Extract repo name from filename
|
||||
repo_name = fname.removesuffix("_analysis.md")
|
||||
lines.append(f"### {repo_name}\n")
|
||||
# Include the first ~600 chars as a condensed summary
|
||||
lines.append(truncate(content.strip(), 600))
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_section(title: str, content: str) -> str:
|
||||
"""Wrap phase content in a section. Return empty section note if no content."""
|
||||
lines = [f"{title}\n"]
|
||||
if content.strip():
|
||||
lines.append(content.strip())
|
||||
else:
|
||||
heading_text = title.lstrip("#").strip()
|
||||
lines.append(f"_{heading_text} not available._")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_appendix(repos: list[dict]) -> str:
|
||||
"""Build the full repo database appendix table."""
|
||||
lines = ["## Appendix A: Full Repo Database\n"]
|
||||
if not repos:
|
||||
lines.append("_No repos in database._\n")
|
||||
return "\n".join(lines)
|
||||
|
||||
sorted_repos = sorted(repos, key=lambda r: -(r.get("composite_score", 0) or 0))
|
||||
headers = ["Repo", "Stars", "Language", "Score", "Source", "Description"]
|
||||
rows: list[list[str]] = []
|
||||
for rec in sorted_repos:
|
||||
rid = rec.get("repo_id", "")
|
||||
url = rec.get("url", f"https://github.com/{rid}")
|
||||
stars = str(rec.get("stars", 0))
|
||||
lang = rec.get("language", "")
|
||||
score = f"{(rec.get('composite_score', 0.0) or 0.0):.3f}"
|
||||
src = rec.get("source", "")
|
||||
desc = truncate((rec.get("description") or "").replace("|", "/"), 60)
|
||||
rows.append([f"[{rid}]({url})", stars, lang, score, src, desc])
|
||||
|
||||
lines.append(format_table(headers, rows))
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def build_methodology() -> str:
|
||||
"""Build the methodology appendix."""
|
||||
return """## Appendix B: Methodology
|
||||
|
||||
- **Phase 1: Intake** -- Parse deep-research output for GitHub URLs, paper references, and search keywords
|
||||
- **Phase 2: Discovery** -- Multi-source search (GitHub repo search, Papers With Code, GitHub code search, direct URLs)
|
||||
- **Phase 3: Scoring & Filtering** -- Composite score = relevance x 0.4 + quality x 0.35 + activity x 0.25; select top repos
|
||||
- **Phase 4: Deep Dive** -- Shallow clone + deep code reading of top repos (architecture, algorithms, quality, reusability)
|
||||
- **Phase 5: Cross-Repo Analysis** -- Comparison matrix, technique-to-code mapping, gap analysis
|
||||
- **Phase 6: Integration Blueprint** -- Recommended architecture, reuse catalog, license compatibility, effort estimates
|
||||
"""
|
||||
|
||||
|
||||
# -- Main ---------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Assemble final GitHub research report from all phase outputs."
|
||||
)
|
||||
parser.add_argument(
|
||||
"--topic-dir", required=True,
|
||||
help="Path to the github-research-output/{slug}/ directory",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
topic_dir = args.topic_dir.rstrip("/")
|
||||
if not os.path.isdir(topic_dir):
|
||||
print(f"[error] topic directory not found: {topic_dir}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
print(f"[compile] loading data from {topic_dir}", file=sys.stderr)
|
||||
|
||||
# -- Load repo database --
|
||||
repo_db_path = os.path.join(topic_dir, "repo_db.jsonl")
|
||||
all_repos = load_jsonl(repo_db_path)
|
||||
print(f" repo_db.jsonl: {len(all_repos)} repos", file=sys.stderr)
|
||||
|
||||
# -- Load ranked repos --
|
||||
ranked_path = os.path.join(topic_dir, "phase3_filtering", "ranked_repos.jsonl")
|
||||
ranked_repos = load_jsonl(ranked_path)
|
||||
print(f" ranked_repos.jsonl: {len(ranked_repos)} repos", file=sys.stderr)
|
||||
|
||||
# -- Load phase text outputs --
|
||||
phase_paths = {
|
||||
"intake_summary": os.path.join(topic_dir, "phase1_intake", "intake_summary.md"),
|
||||
"discovery_log": os.path.join(topic_dir, "phase2_discovery", "discovery_log.md"),
|
||||
"filtering_report": os.path.join(topic_dir, "phase3_filtering", "filtering_report.md"),
|
||||
"deep_dive_summary": os.path.join(topic_dir, "phase4_deep_dive", "deep_dive_summary.md"),
|
||||
"comparison_matrix": os.path.join(topic_dir, "phase5_analysis", "comparison_matrix.md"),
|
||||
"technique_map": os.path.join(topic_dir, "phase5_analysis", "technique_map.md"),
|
||||
"analysis_report": os.path.join(topic_dir, "phase5_analysis", "analysis_report.md"),
|
||||
"integration_plan": os.path.join(topic_dir, "phase6_blueprint", "integration_plan.md"),
|
||||
"reuse_catalog": os.path.join(topic_dir, "phase6_blueprint", "reuse_catalog.md"),
|
||||
"blueprint_summary": os.path.join(topic_dir, "phase6_blueprint", "blueprint_summary.md"),
|
||||
}
|
||||
|
||||
phases: dict[str, str] = {}
|
||||
for key, path in phase_paths.items():
|
||||
phases[key] = load_text(path)
|
||||
status = "loaded" if phases[key] else "not found"
|
||||
print(f" {key}: {status}", file=sys.stderr)
|
||||
|
||||
analyses_dir = os.path.join(topic_dir, "phase4_deep_dive", "analyses")
|
||||
|
||||
# -- Compute statistics --
|
||||
stats = compute_stats(all_repos)
|
||||
# Override total_filtered with actual ranked count if available
|
||||
if ranked_repos:
|
||||
stats["total_filtered"] = len(ranked_repos)
|
||||
# Count analysis files for total_analyzed
|
||||
if os.path.isdir(analyses_dir):
|
||||
analysis_count = sum(
|
||||
1 for f in os.listdir(analyses_dir) if f.endswith("_analysis.md")
|
||||
)
|
||||
if analysis_count > 0:
|
||||
stats["total_analyzed"] = analysis_count
|
||||
|
||||
print(f"[compile] assembling report...", file=sys.stderr)
|
||||
|
||||
# -- Assemble report --
|
||||
sections: list[str] = [
|
||||
build_header(topic_dir),
|
||||
"---\n",
|
||||
build_executive_summary(stats, phases["blueprint_summary"]),
|
||||
build_discovery_stats(stats),
|
||||
build_top_repos(ranked_repos if ranked_repos else all_repos),
|
||||
build_deep_analysis_summaries(analyses_dir),
|
||||
build_section("## 4. Cross-Repository Comparison", phases["comparison_matrix"]),
|
||||
build_section("## 5. Technique-to-Code Mapping", phases["technique_map"]),
|
||||
build_section("## 6. Integration Blueprint", phases["integration_plan"]),
|
||||
build_section("## 7. Reusable Components", phases["reuse_catalog"]),
|
||||
build_appendix(all_repos),
|
||||
build_methodology(),
|
||||
]
|
||||
|
||||
report = "\n".join(sections)
|
||||
|
||||
# -- Write outputs --
|
||||
output_dir = os.path.join(topic_dir, "phase6_blueprint")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
report_path = os.path.join(output_dir, "final_report.md")
|
||||
with open(report_path, "w", encoding="utf-8") as f:
|
||||
f.write(report)
|
||||
|
||||
stats_path = os.path.join(output_dir, "stats.json")
|
||||
with open(stats_path, "w", encoding="utf-8") as f:
|
||||
json.dump(stats, f, indent=2, ensure_ascii=False)
|
||||
|
||||
# -- Summary --
|
||||
section_count = sum(1 for s in sections if s.strip().startswith("##"))
|
||||
print(
|
||||
f"\nReport compiled: {section_count} sections, "
|
||||
f"{stats['total_discovered']} repos, "
|
||||
f"{stats['total_analyzed']} analyzed",
|
||||
file=sys.stderr,
|
||||
)
|
||||
print(f" -> {report_path}", file=sys.stderr)
|
||||
print(f" -> {stats_path}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+543
@@ -0,0 +1,543 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract and parse dependency files from a cloned repository.
|
||||
|
||||
Identifies Python, Node, Rust, Go, and system dependencies.
|
||||
Self-contained: stdlib only (regex-based parsing, no toml/yaml libraries).
|
||||
|
||||
Usage:
|
||||
python extract_dependencies.py --repo-dir ./repos/owner_name --output deps.json
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ML framework detection mapping
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
ML_FRAMEWORK_MAP: dict[str, str] = {
|
||||
# PyTorch ecosystem
|
||||
"torch": "pytorch", "torchvision": "pytorch", "torchaudio": "pytorch",
|
||||
"pytorch-lightning": "pytorch", "lightning": "pytorch",
|
||||
# TensorFlow ecosystem
|
||||
"tensorflow": "tensorflow", "tensorflow-gpu": "tensorflow",
|
||||
"tf-nightly": "tensorflow", "tf-estimator-nightly": "tensorflow",
|
||||
"keras": "tensorflow",
|
||||
# JAX ecosystem
|
||||
"jax": "jax", "jaxlib": "jax", "flax": "jax", "optax": "jax",
|
||||
# HuggingFace ecosystem
|
||||
"transformers": "huggingface", "datasets": "huggingface",
|
||||
"tokenizers": "huggingface", "accelerate": "huggingface",
|
||||
"diffusers": "huggingface", "peft": "huggingface",
|
||||
# scikit-learn
|
||||
"scikit-learn": "scikit-learn", "sklearn": "scikit-learn",
|
||||
# Other ML tools
|
||||
"onnx": "onnx", "onnxruntime": "onnx",
|
||||
"tensorrt": "tensorrt", "triton": "triton",
|
||||
"deepspeed": "deepspeed", "fairscale": "fairscale",
|
||||
}
|
||||
|
||||
SYSTEM_KEYWORDS: list[str] = [
|
||||
"cuda", "cudnn", "nvidia", "ffmpeg", "libsndfile", "sox",
|
||||
"opencv", "libgl", "libglib", "cmake", "gcc", "g++",
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _read_text(path: Path) -> str | None:
|
||||
"""Read a file as UTF-8, returning None on failure."""
|
||||
try:
|
||||
return path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def _strip_version(spec: str) -> str:
|
||||
"""Extract bare package name from a version specifier string."""
|
||||
return re.split(r"[><=!~;\[\s@]", spec.strip())[0].strip().lower()
|
||||
|
||||
|
||||
def _extract_quoted_strings(text: str) -> list[str]:
|
||||
"""Extract all single- or double-quoted strings from text."""
|
||||
return re.findall(r"""['"]([^'"]+)['"]""", text)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Python: requirements.txt
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_requirements_txt(repo_dir: Path) -> dict | None:
|
||||
"""Parse requirements*.txt files."""
|
||||
candidates = [
|
||||
"requirements.txt", "requirements-dev.txt", "requirements_dev.txt",
|
||||
"requirements-test.txt", "requirements_test.txt",
|
||||
"requirements/base.txt", "requirements/main.txt",
|
||||
]
|
||||
all_specs: list[str] = []
|
||||
source_files: list[str] = []
|
||||
|
||||
for name in candidates:
|
||||
path = repo_dir / name
|
||||
if not path.is_file():
|
||||
continue
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
continue
|
||||
source_files.append(name)
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or line.startswith("-"):
|
||||
continue
|
||||
all_specs.append(line)
|
||||
|
||||
if not all_specs:
|
||||
return None
|
||||
# Deduplicate while preserving order
|
||||
seen: set[str] = set()
|
||||
deduped: list[str] = []
|
||||
for spec in all_specs:
|
||||
if spec not in seen:
|
||||
seen.add(spec)
|
||||
deduped.append(spec)
|
||||
return {"requirements": deduped, "source_file": ", ".join(source_files)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Python: setup.py
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_setup_py(repo_dir: Path) -> dict | None:
|
||||
"""Parse setup.py for install_requires and extras_require."""
|
||||
path = repo_dir / "setup.py"
|
||||
if not path.is_file():
|
||||
return None
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
return None
|
||||
|
||||
specs: list[str] = []
|
||||
|
||||
# install_requires = [...]
|
||||
match = re.search(r"install_requires\s*=\s*\[([^\]]+)\]", text, re.DOTALL)
|
||||
if match:
|
||||
specs.extend(_extract_quoted_strings(match.group(1)))
|
||||
|
||||
# extras_require = { ... }
|
||||
match = re.search(r"extras_require\s*=\s*\{([^}]+)\}", text, re.DOTALL)
|
||||
if match:
|
||||
specs.extend(_extract_quoted_strings(match.group(1)))
|
||||
|
||||
if not specs:
|
||||
return None
|
||||
return {"requirements": specs, "source_file": "setup.py"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Python: pyproject.toml
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_pyproject_toml(repo_dir: Path) -> dict | None:
|
||||
"""Parse pyproject.toml for dependencies (regex-based, no toml lib)."""
|
||||
path = repo_dir / "pyproject.toml"
|
||||
if not path.is_file():
|
||||
return None
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
return None
|
||||
|
||||
specs: list[str] = []
|
||||
|
||||
# [project] dependencies = [...]
|
||||
match = re.search(r"dependencies\s*=\s*\[([^\]]*)\]", text, re.DOTALL)
|
||||
if match:
|
||||
specs.extend(_extract_quoted_strings(match.group(1)))
|
||||
|
||||
# [project.optional-dependencies] section -- grab all arrays
|
||||
section_match = re.search(
|
||||
r"\[project\.optional-dependencies\](.*?)(?=\n\[|\Z)", text, re.DOTALL,
|
||||
)
|
||||
if section_match:
|
||||
for array_match in re.finditer(r"=\s*\[([^\]]*)\]", section_match.group(1)):
|
||||
specs.extend(_extract_quoted_strings(array_match.group(1)))
|
||||
|
||||
if not specs:
|
||||
return None
|
||||
return {"requirements": specs, "source_file": "pyproject.toml"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Python: environment.yml / environment.yaml
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_environment_yml(repo_dir: Path) -> dict | None:
|
||||
"""Parse environment.yml / environment.yaml for conda dependencies."""
|
||||
source_name: str | None = None
|
||||
for name in ("environment.yml", "environment.yaml"):
|
||||
path = repo_dir / name
|
||||
if path.is_file():
|
||||
source_name = name
|
||||
break
|
||||
if source_name is None:
|
||||
return None
|
||||
|
||||
text = _read_text(repo_dir / source_name)
|
||||
if text is None:
|
||||
return None
|
||||
|
||||
specs: list[str] = []
|
||||
in_deps = False
|
||||
in_pip = False
|
||||
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
|
||||
# Detect dependencies: section
|
||||
if re.match(r"^dependencies\s*:", stripped):
|
||||
in_deps = True
|
||||
continue
|
||||
|
||||
if not in_deps:
|
||||
continue
|
||||
|
||||
# End of section on non-indented, non-list line
|
||||
if stripped and not stripped.startswith("-") and not stripped.startswith("#"):
|
||||
if not in_pip:
|
||||
in_deps = False
|
||||
continue
|
||||
|
||||
# pip sub-section
|
||||
if stripped == "- pip:" or stripped == "- pip":
|
||||
in_pip = True
|
||||
continue
|
||||
if in_pip:
|
||||
if stripped.startswith("- "):
|
||||
pkg = stripped[2:].strip()
|
||||
if pkg:
|
||||
specs.append(pkg)
|
||||
continue
|
||||
if not stripped.startswith("-") and stripped:
|
||||
in_pip = False
|
||||
|
||||
if stripped.startswith("- "):
|
||||
pkg = stripped[2:].strip()
|
||||
if pkg and pkg not in ("pip", "pip:"):
|
||||
specs.append(pkg)
|
||||
|
||||
if not specs:
|
||||
return None
|
||||
return {"requirements": specs, "source_file": source_name}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node: package.json
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_package_json(repo_dir: Path) -> dict | None:
|
||||
"""Parse package.json for dependencies and devDependencies."""
|
||||
path = repo_dir / "package.json"
|
||||
if not path.is_file():
|
||||
return None
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
return None
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
|
||||
deps = data.get("dependencies") or {}
|
||||
dev_deps = data.get("devDependencies") or {}
|
||||
if not deps and not dev_deps:
|
||||
return None
|
||||
|
||||
result: dict = {"source_file": "package.json"}
|
||||
if deps:
|
||||
result["dependencies"] = deps
|
||||
if dev_deps:
|
||||
result["devDependencies"] = dev_deps
|
||||
return result
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Rust: Cargo.toml
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_cargo_toml(repo_dir: Path) -> dict | None:
|
||||
"""Parse Cargo.toml for [dependencies] section."""
|
||||
path = repo_dir / "Cargo.toml"
|
||||
if not path.is_file():
|
||||
return None
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
return None
|
||||
|
||||
deps: dict[str, str] = {}
|
||||
match = re.search(
|
||||
r"\[dependencies\](.*?)(?=\n\[|\Z)", text, re.DOTALL,
|
||||
)
|
||||
if match:
|
||||
for line in match.group(1).splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
# name = "version"
|
||||
m = re.match(r'^(\S+)\s*=\s*"([^"]*)"', line)
|
||||
if m:
|
||||
deps[m.group(1)] = m.group(2)
|
||||
else:
|
||||
# name = { version = "...", ... }
|
||||
m = re.match(r"^(\S+)\s*=\s*\{", line)
|
||||
if m:
|
||||
ver = re.search(r'version\s*=\s*"([^"]*)"', line)
|
||||
deps[m.group(1)] = ver.group(1) if ver else "*"
|
||||
|
||||
if not deps:
|
||||
return None
|
||||
return {"dependencies": deps, "source_file": "Cargo.toml"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Go: go.mod
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_go_mod(repo_dir: Path) -> dict | None:
|
||||
"""Parse go.mod for require block."""
|
||||
path = repo_dir / "go.mod"
|
||||
if not path.is_file():
|
||||
return None
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
return None
|
||||
|
||||
modules: list[str] = []
|
||||
in_require = False
|
||||
|
||||
for line in text.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("require ("):
|
||||
in_require = True
|
||||
continue
|
||||
if in_require:
|
||||
if stripped == ")":
|
||||
in_require = False
|
||||
continue
|
||||
parts = stripped.split()
|
||||
if parts and not parts[0].startswith("//"):
|
||||
modules.append(parts[0])
|
||||
elif stripped.startswith("require "):
|
||||
parts = stripped.split()
|
||||
if len(parts) >= 2:
|
||||
modules.append(parts[1])
|
||||
|
||||
if not modules:
|
||||
return None
|
||||
return {"modules": modules, "source_file": "go.mod"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# System dependency detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def detect_system_deps(repo_dir: Path) -> list[str]:
|
||||
"""Detect system-level dependencies from Dockerfile, README, setup.py."""
|
||||
found: set[str] = set()
|
||||
scan_files = [
|
||||
"Dockerfile", "docker-compose.yml", "docker-compose.yaml",
|
||||
"README.md", "README.rst", "readme.md",
|
||||
"setup.py", "Makefile", "CMakeLists.txt",
|
||||
]
|
||||
|
||||
for name in scan_files:
|
||||
path = repo_dir / name
|
||||
if not path.is_file():
|
||||
continue
|
||||
text = _read_text(path)
|
||||
if text is None:
|
||||
continue
|
||||
text_lower = text.lower()
|
||||
|
||||
# Check for known system keywords
|
||||
for kw in SYSTEM_KEYWORDS:
|
||||
if kw in text_lower:
|
||||
found.add(kw)
|
||||
|
||||
# Extract apt-get install packages from Dockerfiles
|
||||
if name.lower().startswith("dockerfile") or name.lower() == "dockerfile":
|
||||
for m in re.finditer(r"apt-get\s+install[^&\n]*", text):
|
||||
tokens = m.group(0).split()
|
||||
for tok in tokens:
|
||||
tok = tok.strip().rstrip("\\")
|
||||
if tok and not tok.startswith("-") and tok not in (
|
||||
"apt-get", "install", "&&", "||", "RUN",
|
||||
):
|
||||
found.add(tok)
|
||||
|
||||
return sorted(found)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ML framework detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def detect_ml_frameworks(all_packages: list[str]) -> list[str]:
|
||||
"""Identify ML frameworks from the combined package list."""
|
||||
frameworks: set[str] = set()
|
||||
for pkg in all_packages:
|
||||
pkg_lower = pkg.lower()
|
||||
if pkg_lower in ML_FRAMEWORK_MAP:
|
||||
frameworks.add(ML_FRAMEWORK_MAP[pkg_lower])
|
||||
# Handle tf-* prefix patterns
|
||||
elif pkg_lower.startswith("tf-") or pkg_lower.startswith("tensorflow-"):
|
||||
frameworks.add("tensorflow")
|
||||
return sorted(frameworks)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Extract and parse dependency files from a cloned repository",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repo-dir", required=True,
|
||||
help="Path to cloned repository directory",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output", required=True,
|
||||
help="Output JSON file path",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repo-id", default=None,
|
||||
help="Repository identifier (owner/name); inferred from dir name if omitted",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
repo_dir = Path(args.repo_dir).resolve()
|
||||
if not repo_dir.is_dir():
|
||||
print(f"Error: repo directory not found: {repo_dir}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Infer repo_id from directory name if not provided
|
||||
repo_id: str = args.repo_id or ""
|
||||
if not repo_id:
|
||||
dir_name = repo_dir.name
|
||||
repo_id = dir_name.replace("_", "/", 1) if "_" in dir_name else dir_name
|
||||
|
||||
print(f"Extracting dependencies from {repo_dir} ...", file=sys.stderr)
|
||||
|
||||
# --- Python: merge results from all parsers ---
|
||||
python_info: dict | None = None
|
||||
for parser_fn in (parse_requirements_txt, parse_setup_py,
|
||||
parse_pyproject_toml, parse_environment_yml):
|
||||
result = parser_fn(repo_dir)
|
||||
if result is not None:
|
||||
if python_info is None:
|
||||
python_info = result
|
||||
else:
|
||||
# Merge: append specs, combine source_file
|
||||
existing = set(python_info["requirements"])
|
||||
for spec in result["requirements"]:
|
||||
if spec not in existing:
|
||||
python_info["requirements"].append(spec)
|
||||
existing.add(spec)
|
||||
python_info["source_file"] += f", {result['source_file']}"
|
||||
if python_info:
|
||||
print(f" Python: {len(python_info['requirements'])} packages "
|
||||
f"from {python_info['source_file']}", file=sys.stderr)
|
||||
|
||||
# --- Node ---
|
||||
node_info = parse_package_json(repo_dir)
|
||||
if node_info:
|
||||
n = len(node_info.get("dependencies", {})) + len(node_info.get("devDependencies", {}))
|
||||
print(f" Node: {n} packages from {node_info['source_file']}", file=sys.stderr)
|
||||
|
||||
# --- Rust ---
|
||||
rust_info = parse_cargo_toml(repo_dir)
|
||||
if rust_info:
|
||||
print(f" Rust: {len(rust_info['dependencies'])} crates "
|
||||
f"from {rust_info['source_file']}", file=sys.stderr)
|
||||
|
||||
# --- Go ---
|
||||
go_info = parse_go_mod(repo_dir)
|
||||
if go_info:
|
||||
print(f" Go: {len(go_info['modules'])} modules "
|
||||
f"from {go_info['source_file']}", file=sys.stderr)
|
||||
|
||||
# --- Collect all package names (flat, normalized) ---
|
||||
all_packages: set[str] = set()
|
||||
|
||||
if python_info:
|
||||
for spec in python_info["requirements"]:
|
||||
name = _strip_version(spec)
|
||||
if name:
|
||||
all_packages.add(name)
|
||||
|
||||
if node_info:
|
||||
for key in ("dependencies", "devDependencies"):
|
||||
for pkg_name in node_info.get(key, {}):
|
||||
all_packages.add(pkg_name.lower())
|
||||
|
||||
if rust_info:
|
||||
for crate_name in rust_info["dependencies"]:
|
||||
all_packages.add(crate_name.lower())
|
||||
|
||||
if go_info:
|
||||
for mod_path in go_info["modules"]:
|
||||
# Use last path segment as package name
|
||||
parts = mod_path.rstrip("/").split("/")
|
||||
all_packages.add(parts[-1].lower())
|
||||
|
||||
all_packages_sorted = sorted(all_packages)
|
||||
|
||||
# --- System dependencies ---
|
||||
system_deps = detect_system_deps(repo_dir)
|
||||
if system_deps:
|
||||
print(f" System: {len(system_deps)} system dependencies detected",
|
||||
file=sys.stderr)
|
||||
|
||||
# --- ML frameworks ---
|
||||
ml_frameworks = detect_ml_frameworks(all_packages_sorted)
|
||||
if ml_frameworks:
|
||||
print(f" ML frameworks: {', '.join(ml_frameworks)}", file=sys.stderr)
|
||||
|
||||
# --- Build output ---
|
||||
output: dict = {
|
||||
"repo_id": repo_id,
|
||||
"python": python_info,
|
||||
"node": node_info,
|
||||
"rust": rust_info,
|
||||
"go": go_info,
|
||||
"system": system_deps,
|
||||
"ml_frameworks": ml_frameworks,
|
||||
"all_packages": all_packages_sorted,
|
||||
"overlap_with": {},
|
||||
}
|
||||
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
json.dump(output, f, indent=2, ensure_ascii=False)
|
||||
f.write("\n")
|
||||
|
||||
ecosystems = sum(1 for x in (python_info, node_info, rust_info, go_info)
|
||||
if x is not None)
|
||||
print(
|
||||
f"Extracted {len(all_packages_sorted)} packages across "
|
||||
f"{ecosystems} ecosystem(s) -> {args.output}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+311
@@ -0,0 +1,311 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract GitHub URLs, paper refs, and keywords from deep-research output.
|
||||
|
||||
Scans markdown files for GitHub URLs (cleaning sub-paths to extract owner/name),
|
||||
parses paper_db.jsonl for paper metadata, extracts keywords from titles and tags,
|
||||
and outputs deduplicated JSONL records.
|
||||
|
||||
Usage:
|
||||
python extract_research_refs.py --research-dir ./deep-research-output/topic/ --output refs.jsonl
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
GITHUB_URL_RE = re.compile(r'https://github\.com/[^\s\)\]\>\"\']+')
|
||||
|
||||
# Sub-paths to strip to get base repo URL
|
||||
REPO_SUBPATH_RE = re.compile(
|
||||
r"/(?:tree|blob|issues|pull|releases|wiki|actions|commits|compare|archive|raw|discussions|pkgs|packages|security|stargazers|network)(?:/.*)?$"
|
||||
)
|
||||
|
||||
# Non-repo top-level GitHub pages
|
||||
NON_REPO_OWNERS = frozenset({
|
||||
"topics", "explore", "trending", "settings",
|
||||
"organizations", "sponsors", "features", "marketplace",
|
||||
"notifications", "new", "login", "signup", "pricing",
|
||||
})
|
||||
|
||||
# Common English stopwords for keyword filtering
|
||||
STOPWORDS = frozenset({
|
||||
"a", "an", "the", "and", "or", "but", "in", "on", "at", "to", "for",
|
||||
"of", "with", "by", "from", "is", "are", "was", "were", "be", "been",
|
||||
"being", "have", "has", "had", "do", "does", "did", "will", "would",
|
||||
"could", "should", "may", "might", "can", "shall", "not", "no", "nor",
|
||||
"so", "if", "then", "than", "that", "this", "these", "those", "it",
|
||||
"its", "we", "our", "they", "their", "them", "you", "your", "he",
|
||||
"she", "his", "her", "as", "up", "out", "about", "into", "over",
|
||||
"after", "before", "between", "through", "during", "each", "every",
|
||||
"all", "both", "few", "more", "most", "other", "some", "such", "only",
|
||||
"same", "also", "how", "what", "which", "who", "when", "where", "why",
|
||||
"very", "just", "because", "while", "using", "via", "based", "new",
|
||||
"one", "two", "first", "use", "used", "paper", "method", "approach",
|
||||
"proposed", "show", "results", "model", "models",
|
||||
})
|
||||
|
||||
|
||||
def clean_github_url(raw_url: str) -> tuple[str, str]:
|
||||
"""Clean a raw GitHub URL and extract owner/name repo_id.
|
||||
|
||||
Returns (clean_url, repo_id) or ("", "") if invalid.
|
||||
"""
|
||||
url = raw_url.rstrip(".,;:!?)]\">'/")
|
||||
url = url.rstrip("/")
|
||||
|
||||
# Remove sub-paths (tree, blob, issues, etc.) to get base repo URL
|
||||
url = REPO_SUBPATH_RE.sub("", url)
|
||||
|
||||
# Extract owner/name
|
||||
path = url.replace("https://github.com/", "")
|
||||
parts = path.split("/")
|
||||
if len(parts) >= 2 and parts[0] and parts[1]:
|
||||
owner, name = parts[0], parts[1]
|
||||
# Filter out non-repo pages
|
||||
if owner.lower() in NON_REPO_OWNERS:
|
||||
return "", ""
|
||||
# Strip .git suffix
|
||||
name = name.removesuffix(".git")
|
||||
repo_id = f"{owner}/{name}"
|
||||
clean_url = f"https://github.com/{repo_id}"
|
||||
return clean_url, repo_id
|
||||
return "", ""
|
||||
|
||||
|
||||
def scan_md_files_for_urls(research_dir: Path) -> list[dict]:
|
||||
"""Scan all .md files recursively for GitHub URLs."""
|
||||
results: list[dict] = []
|
||||
seen_repos: set[str] = set()
|
||||
|
||||
for md_file in sorted(research_dir.rglob("*.md")):
|
||||
try:
|
||||
text = md_file.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError as e:
|
||||
print(f"[warn] cannot read {md_file}: {e}", file=sys.stderr)
|
||||
continue
|
||||
|
||||
rel_path = str(md_file.relative_to(research_dir))
|
||||
|
||||
for match in GITHUB_URL_RE.finditer(text):
|
||||
raw_url = match.group(0)
|
||||
clean_url, repo_id = clean_github_url(raw_url)
|
||||
if not repo_id:
|
||||
continue
|
||||
|
||||
# Extract surrounding context (the line containing the URL)
|
||||
line_start = text.rfind("\n", 0, match.start()) + 1
|
||||
line_end = text.find("\n", match.end())
|
||||
if line_end == -1:
|
||||
line_end = len(text)
|
||||
context = text[line_start:line_end].strip()
|
||||
|
||||
# Deduplicate by repo_id (not raw URL) to merge /tree/... variants
|
||||
if repo_id in seen_repos:
|
||||
continue
|
||||
seen_repos.add(repo_id)
|
||||
|
||||
results.append({
|
||||
"type": "github_url",
|
||||
"url": clean_url,
|
||||
"repo_id": repo_id,
|
||||
"source_file": rel_path,
|
||||
"context": context[:200],
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def parse_paper_db(research_dir: Path) -> list[dict]:
|
||||
"""Parse paper_db.jsonl for paper titles, arxiv IDs, tags."""
|
||||
results: list[dict] = []
|
||||
paper_db = research_dir / "paper_db.jsonl"
|
||||
if not paper_db.exists():
|
||||
print("[info] paper_db.jsonl not found, skipping paper extraction", file=sys.stderr)
|
||||
return results
|
||||
|
||||
try:
|
||||
for line_num, line in enumerate(paper_db.read_text(encoding="utf-8").splitlines(), 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
paper = json.loads(line)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"[warn] paper_db.jsonl line {line_num}: {e}", file=sys.stderr)
|
||||
continue
|
||||
|
||||
title = paper.get("title", "")
|
||||
arxiv_id = paper.get("arxiv_id", paper.get("id", ""))
|
||||
tags = paper.get("tags", paper.get("keywords", paper.get("categories", [])))
|
||||
paper_id = paper.get("paperId", arxiv_id)
|
||||
|
||||
results.append({
|
||||
"type": "paper",
|
||||
"title": title,
|
||||
"arxiv_id": arxiv_id,
|
||||
"tags": tags if isinstance(tags, list) else [],
|
||||
"paper_id": paper_id,
|
||||
})
|
||||
except OSError as e:
|
||||
print(f"[warn] cannot read paper_db.jsonl: {e}", file=sys.stderr)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def extract_keywords(papers: list[dict]) -> list[dict]:
|
||||
"""Extract search keywords from paper titles and tags."""
|
||||
tag_counter: Counter[str] = Counter()
|
||||
title_term_counter: Counter[str] = Counter()
|
||||
|
||||
for paper in papers:
|
||||
# Count tags
|
||||
for tag in paper.get("tags", []):
|
||||
if isinstance(tag, str):
|
||||
tag_lower = tag.strip().lower()
|
||||
if tag_lower and len(tag_lower) > 2:
|
||||
tag_counter[tag_lower] += 1
|
||||
|
||||
# Extract terms from titles
|
||||
title = paper.get("title", "")
|
||||
if not title:
|
||||
continue
|
||||
words = re.findall(r"[A-Za-z][-A-Za-z]+", title)
|
||||
filtered = [w.lower() for w in words
|
||||
if w.lower() not in STOPWORDS and len(w) > 2]
|
||||
|
||||
# Unigrams
|
||||
for w in filtered:
|
||||
title_term_counter[w] += 1
|
||||
|
||||
# Bigrams (multi-word technical terms)
|
||||
for i in range(len(filtered) - 1):
|
||||
bigram = f"{filtered[i]} {filtered[i + 1]}"
|
||||
title_term_counter[bigram] += 1
|
||||
|
||||
keywords: list[dict] = []
|
||||
|
||||
# Tags with frequency
|
||||
for tag, freq in tag_counter.most_common():
|
||||
keywords.append({
|
||||
"type": "keyword",
|
||||
"value": tag,
|
||||
"frequency": freq,
|
||||
"source": "paper_tags",
|
||||
})
|
||||
|
||||
# Title terms (2+ occurrences, or bigrams with 1+ occurrence)
|
||||
for term, freq in title_term_counter.most_common():
|
||||
if freq >= 2 or (freq >= 1 and " " in term):
|
||||
# Skip if already covered by tags
|
||||
if term not in tag_counter:
|
||||
keywords.append({
|
||||
"type": "keyword",
|
||||
"value": term,
|
||||
"frequency": freq,
|
||||
"source": "paper_titles",
|
||||
})
|
||||
|
||||
return keywords
|
||||
|
||||
|
||||
def extract_synthesis_themes(research_dir: Path) -> list[dict]:
|
||||
"""Extract research themes from synthesis/report markdown headings."""
|
||||
results: list[dict] = []
|
||||
|
||||
# Check several possible locations for synthesis/report files
|
||||
candidates = [
|
||||
research_dir / "phase5_synthesis" / "synthesis.md",
|
||||
research_dir / "phase6_report" / "report.md",
|
||||
research_dir / "phase3_deep_dive" / "deep_dive.md",
|
||||
]
|
||||
|
||||
for filepath in candidates:
|
||||
if not filepath.exists():
|
||||
continue
|
||||
try:
|
||||
text = filepath.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
for match in re.finditer(r'^#{1,3}\s+(.+)$', text, re.MULTILINE):
|
||||
heading = match.group(1).strip()
|
||||
if len(heading) > 3:
|
||||
results.append({
|
||||
"type": "keyword",
|
||||
"value": heading.lower(),
|
||||
"frequency": 1,
|
||||
"source": "paper_titles",
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Extract GitHub URLs, paper refs, and keywords from deep-research output."
|
||||
)
|
||||
parser.add_argument("--research-dir", required=True,
|
||||
help="Path to deep-research output directory")
|
||||
parser.add_argument("--output", required=True,
|
||||
help="Output JSONL file path")
|
||||
args = parser.parse_args()
|
||||
|
||||
research_dir = Path(args.research_dir).resolve()
|
||||
if not research_dir.is_dir():
|
||||
print(f"[error] research dir not found: {research_dir}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
print(f"[info] scanning {research_dir} ...", file=sys.stderr)
|
||||
|
||||
# 1. Extract GitHub URLs from all markdown files
|
||||
github_refs = scan_md_files_for_urls(research_dir)
|
||||
print(f" GitHub URLs: {len(github_refs)}", file=sys.stderr)
|
||||
|
||||
# 2. Parse paper_db.jsonl
|
||||
papers = parse_paper_db(research_dir)
|
||||
print(f" Papers: {len(papers)}", file=sys.stderr)
|
||||
|
||||
# 3. Extract keywords from papers
|
||||
keywords = extract_keywords(papers)
|
||||
|
||||
# 4. Extract themes from synthesis/report headings
|
||||
themes = extract_synthesis_themes(research_dir)
|
||||
keywords.extend(themes)
|
||||
|
||||
# Deduplicate keywords by value
|
||||
seen_kw: set[str] = set()
|
||||
deduped_kw: list[dict] = []
|
||||
for kw in keywords:
|
||||
if kw["value"] not in seen_kw:
|
||||
seen_kw.add(kw["value"])
|
||||
deduped_kw.append(kw)
|
||||
keywords = deduped_kw
|
||||
|
||||
print(f" Keywords: {len(keywords)}", file=sys.stderr)
|
||||
|
||||
# Write output
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
for record in github_refs:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
for record in papers:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
for record in keywords:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
|
||||
# Summary stats
|
||||
unique_repos = len({r["repo_id"] for r in github_refs})
|
||||
print(
|
||||
f"Extracted {unique_repos} GitHub URLs, "
|
||||
f"{len(papers)} papers, {len(keywords)} keywords "
|
||||
f"-> {output_path}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+279
@@ -0,0 +1,279 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search cloned repo for specific implementations (classes, functions, algorithms).
|
||||
|
||||
Usage:
|
||||
python find_implementations.py --repo-dir ./repos/owner_name --patterns "class Transformer" "def train" --output matches.jsonl
|
||||
python find_implementations.py --repo-dir ./repos/owner_name --keywords "attention" "embedding" --output matches.jsonl
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SKIP_DIRS: frozenset[str] = frozenset({
|
||||
".git", "node_modules", "__pycache__", ".venv", "venv",
|
||||
".tox", ".mypy_cache", ".pytest_cache", ".eggs", "dist", "build",
|
||||
".next", ".nuxt",
|
||||
})
|
||||
|
||||
TEST_INDICATORS: tuple[str, ...] = (
|
||||
"/test/", "/tests/", "test_", "_test.py", ".test.", ".spec.",
|
||||
)
|
||||
|
||||
MAX_FILE_SIZE: int = 1_048_576 # 1 MB
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def is_binary(filepath: Path) -> bool:
|
||||
"""Check if a file is likely binary by reading the first 8 KB."""
|
||||
try:
|
||||
with open(filepath, "rb") as f:
|
||||
chunk = f.read(8192)
|
||||
return b"\x00" in chunk
|
||||
except OSError:
|
||||
return True
|
||||
|
||||
|
||||
def is_test_path(rel_path: str) -> bool:
|
||||
"""Return True if the relative path looks like a test file."""
|
||||
rel_lower = rel_path.lower()
|
||||
for indicator in TEST_INDICATORS:
|
||||
if indicator in rel_lower:
|
||||
return True
|
||||
# Also check basename prefix
|
||||
basename = os.path.basename(rel_lower)
|
||||
if basename.startswith("test_"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def classify_pattern(pattern: str) -> tuple[str, re.Pattern[str]]:
|
||||
"""Classify a search pattern and return (match_type, compiled_regex).
|
||||
|
||||
Patterns starting with "class " or "def " get special match types.
|
||||
Other patterns are compiled as-is (user-supplied regex).
|
||||
"""
|
||||
if pattern.startswith("class "):
|
||||
name = pattern[6:].strip()
|
||||
return "class", re.compile(r"class\s+" + re.escape(name))
|
||||
elif pattern.startswith("def "):
|
||||
name = pattern[4:].strip()
|
||||
return "function", re.compile(r"def\s+" + re.escape(name))
|
||||
else:
|
||||
try:
|
||||
return "keyword", re.compile(pattern)
|
||||
except re.error:
|
||||
# Fall back to escaped literal if pattern is not valid regex
|
||||
return "keyword", re.compile(re.escape(pattern))
|
||||
|
||||
|
||||
def read_lines(filepath: Path) -> list[str] | None:
|
||||
"""Read all lines from a text file. Returns None on decode failure."""
|
||||
try:
|
||||
with open(filepath, "r", encoding="utf-8", errors="strict") as f:
|
||||
return f.readlines()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Search logic
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def search_file(
|
||||
filepath: Path,
|
||||
rel_path: str,
|
||||
patterns: list[tuple[str, re.Pattern[str]]],
|
||||
keywords: list[str],
|
||||
context: int,
|
||||
repo_id: str,
|
||||
) -> list[dict]:
|
||||
"""Search a single file for patterns and keywords. Returns match dicts."""
|
||||
lines = read_lines(filepath)
|
||||
if lines is None:
|
||||
return []
|
||||
|
||||
results: list[dict] = []
|
||||
|
||||
# --- Pattern search (regex) ---
|
||||
for match_type, regex in patterns:
|
||||
for i, line in enumerate(lines):
|
||||
if regex.search(line):
|
||||
start = max(0, i - context)
|
||||
end = min(len(lines), i + context + 1)
|
||||
results.append({
|
||||
"repo_id": repo_id,
|
||||
"file_path": rel_path,
|
||||
"line_number": i + 1,
|
||||
"match_type": match_type,
|
||||
"matched_text": line.rstrip("\n"),
|
||||
"context_before": [
|
||||
l.rstrip("\n") for l in lines[start:i]
|
||||
],
|
||||
"context_after": [
|
||||
l.rstrip("\n") for l in lines[i + 1:end]
|
||||
],
|
||||
})
|
||||
|
||||
# --- Keyword search (case-insensitive substring) ---
|
||||
for kw in keywords:
|
||||
kw_lower = kw.lower()
|
||||
for i, line in enumerate(lines):
|
||||
if kw_lower in line.lower():
|
||||
start = max(0, i - context)
|
||||
end = min(len(lines), i + context + 1)
|
||||
results.append({
|
||||
"repo_id": repo_id,
|
||||
"file_path": rel_path,
|
||||
"line_number": i + 1,
|
||||
"match_type": "keyword",
|
||||
"matched_text": line.rstrip("\n"),
|
||||
"context_before": [
|
||||
l.rstrip("\n") for l in lines[start:i]
|
||||
],
|
||||
"context_after": [
|
||||
l.rstrip("\n") for l in lines[i + 1:end]
|
||||
],
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Search cloned repo for specific implementations "
|
||||
"(classes, functions, algorithms)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repo-dir", required=True,
|
||||
help="Path to the cloned repository",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--patterns", nargs="+", default=None,
|
||||
help='Regex patterns (e.g. "class Transformer" "def train")',
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keywords", nargs="+", default=None,
|
||||
help="Simple keyword search terms (case-insensitive substring match)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output", required=True,
|
||||
help="Output JSONL file path",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--context", type=int, default=5,
|
||||
help="Lines of context before and after each match (default: 5)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-tests", action="store_true",
|
||||
help="Skip test files and test directories",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--repo-id", default=None,
|
||||
help="Repository identifier (owner/name); inferred from dir name if omitted",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.patterns and not args.keywords:
|
||||
parser.error("Provide at least one of --patterns or --keywords")
|
||||
|
||||
repo_dir = Path(args.repo_dir).resolve()
|
||||
if not repo_dir.is_dir():
|
||||
print(f"Error: repo directory not found: {repo_dir}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Infer repo_id
|
||||
repo_id: str = args.repo_id or ""
|
||||
if not repo_id:
|
||||
dir_name = repo_dir.name
|
||||
repo_id = dir_name.replace("_", "/", 1) if "_" in dir_name else dir_name
|
||||
|
||||
# Classify patterns
|
||||
classified_patterns: list[tuple[str, re.Pattern[str]]] = []
|
||||
if args.patterns:
|
||||
for p in args.patterns:
|
||||
match_type, regex = classify_pattern(p)
|
||||
classified_patterns.append((match_type, regex))
|
||||
print(f" Pattern: '{p}' -> type={match_type}", file=sys.stderr)
|
||||
|
||||
keyword_list: list[str] = args.keywords or []
|
||||
if keyword_list:
|
||||
print(f" Keywords: {keyword_list}", file=sys.stderr)
|
||||
|
||||
# Walk and search
|
||||
total_matches = 0
|
||||
files_with_matches = 0
|
||||
files_scanned = 0
|
||||
files_skipped = 0
|
||||
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print(f"Searching {repo_dir} ...", file=sys.stderr)
|
||||
|
||||
with open(output_path, "w", encoding="utf-8") as out_f:
|
||||
for dirpath, dirnames, filenames in os.walk(repo_dir):
|
||||
# Filter directories in-place to prune traversal
|
||||
dirnames[:] = [d for d in dirnames if d not in SKIP_DIRS]
|
||||
|
||||
for fname in filenames:
|
||||
filepath = Path(dirpath) / fname
|
||||
rel_path = str(filepath.relative_to(repo_dir)).replace(os.sep, "/")
|
||||
|
||||
# Skip test files if requested
|
||||
if args.skip_tests and is_test_path(rel_path):
|
||||
files_skipped += 1
|
||||
continue
|
||||
|
||||
# Skip files that are too large
|
||||
try:
|
||||
size = filepath.stat().st_size
|
||||
except OSError:
|
||||
continue
|
||||
if size > MAX_FILE_SIZE:
|
||||
files_skipped += 1
|
||||
continue
|
||||
|
||||
# Skip binary files
|
||||
if is_binary(filepath):
|
||||
files_skipped += 1
|
||||
continue
|
||||
|
||||
files_scanned += 1
|
||||
matches = search_file(
|
||||
filepath, rel_path,
|
||||
classified_patterns, keyword_list,
|
||||
args.context, repo_id,
|
||||
)
|
||||
|
||||
if matches:
|
||||
files_with_matches += 1
|
||||
for m in matches:
|
||||
out_f.write(json.dumps(m, ensure_ascii=False) + "\n")
|
||||
total_matches += 1
|
||||
|
||||
print(
|
||||
f"Found {total_matches} matches across {files_with_matches} files "
|
||||
f"({files_scanned} scanned, {files_skipped} skipped) -> {args.output}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+470
@@ -0,0 +1,470 @@
|
||||
#!/usr/bin/env python3
|
||||
"""JSONL GitHub repo database management.
|
||||
|
||||
Subcommands: merge, filter, score, search, tag, stats, export, rank.
|
||||
Deduplication by exact repo_id (owner/name) match.
|
||||
|
||||
Usage:
|
||||
python repo_db.py merge --inputs a.jsonl b.jsonl --output merged.jsonl
|
||||
python repo_db.py filter --input db.jsonl --output filtered.jsonl --min-stars 100
|
||||
python repo_db.py score --input db.jsonl --output scored.jsonl
|
||||
python repo_db.py search --input db.jsonl --query "transformer"
|
||||
python repo_db.py tag --input db.jsonl --ids owner/name --tags impl baseline
|
||||
python repo_db.py stats --input db.jsonl
|
||||
python repo_db.py export --input db.jsonl --format markdown
|
||||
python repo_db.py rank --input db.jsonl --output ranked.jsonl --by composite_score
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
# -- I/O helpers --------------------------------------------------------------
|
||||
|
||||
def load_jsonl(path: str) -> list[dict]:
|
||||
"""Load records from a JSONL file."""
|
||||
records = []
|
||||
if not os.path.exists(path):
|
||||
return records
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
records.append(json.loads(line))
|
||||
return records
|
||||
|
||||
|
||||
def save_jsonl(records: list[dict], path: str):
|
||||
"""Save records to a JSONL file."""
|
||||
os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
for rec in records:
|
||||
f.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
def deduplicate(records: list[dict]) -> list[dict]:
|
||||
"""Remove duplicate repos by exact repo_id match. Later entries win."""
|
||||
seen: dict[str, int] = {}
|
||||
for i, rec in enumerate(records):
|
||||
rid = rec.get("repo_id", "")
|
||||
if rid:
|
||||
seen[rid] = i
|
||||
# Preserve order of last occurrence
|
||||
indices = sorted(seen.values())
|
||||
return [records[i] for i in indices]
|
||||
|
||||
|
||||
# -- Scoring ------------------------------------------------------------------
|
||||
|
||||
def _sigmoid(x: float) -> float:
|
||||
"""Sigmoid with steepness 5 centered at 0.5."""
|
||||
return 1.0 / (1.0 + math.exp(-5.0 * (x - 0.5)))
|
||||
|
||||
|
||||
def _parse_iso(dt_str: str | None) -> datetime | None:
|
||||
"""Parse an ISO-8601 datetime string."""
|
||||
if not dt_str:
|
||||
return None
|
||||
# Handle trailing Z and optional fractional seconds
|
||||
dt_str = dt_str.replace("Z", "+00:00")
|
||||
try:
|
||||
return datetime.fromisoformat(dt_str)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def compute_activity_score(rec: dict, now: datetime) -> float:
|
||||
"""Compute activity score in [0, 1]."""
|
||||
pushed = _parse_iso(rec.get("pushed_at"))
|
||||
if pushed is None:
|
||||
return _sigmoid(0.0)
|
||||
|
||||
days_since_push = max((now - pushed).days, 0)
|
||||
recent_push = 1.0 if days_since_push < 90 else 0.0
|
||||
has_recent_commits = 1.0 if days_since_push < 180 else 0.0
|
||||
|
||||
stars = max(rec.get("stars", 0), 1)
|
||||
open_issues = rec.get("open_issues", 0)
|
||||
ratio = min(open_issues / stars, 1.0)
|
||||
issues_component = 1.0 - ratio # lower ratio is better
|
||||
|
||||
raw = recent_push * 0.4 + has_recent_commits * 0.3 + issues_component * 0.3
|
||||
return _sigmoid(raw)
|
||||
|
||||
|
||||
def compute_quality_score(rec: dict) -> float:
|
||||
"""Compute quality score in [0, 1]."""
|
||||
stars = rec.get("stars", 0)
|
||||
forks = rec.get("forks", 0)
|
||||
|
||||
# Normalize log components: log(x+1) / log(100001) gives ~0-1 for 0-100k
|
||||
max_log = math.log(100001)
|
||||
star_comp = min(math.log(stars + 1) / max_log, 1.0) * 0.3
|
||||
fork_comp = min(math.log(forks + 1) / max_log, 1.0) * 0.2
|
||||
|
||||
has_license = 0.15 if rec.get("license") else 0.0
|
||||
has_readme = 0.15 if rec.get("readme_excerpt") else 0.0
|
||||
not_archived = 0.2 if not rec.get("archived", False) else 0.0
|
||||
|
||||
raw = star_comp + fork_comp + has_license + has_readme + not_archived
|
||||
return min(raw, 1.0)
|
||||
|
||||
|
||||
def score_record(rec: dict, now: datetime) -> dict:
|
||||
"""Compute all scores for a single record and return updated copy."""
|
||||
rec = dict(rec)
|
||||
rec["activity_score"] = round(compute_activity_score(rec, now), 4)
|
||||
rec["quality_score"] = round(compute_quality_score(rec), 4)
|
||||
|
||||
relevance = rec.get("relevance_score", 0.0) or 0.0
|
||||
quality = rec["quality_score"]
|
||||
activity = rec["activity_score"]
|
||||
rec["composite_score"] = round(
|
||||
relevance * 0.4 + quality * 0.35 + activity * 0.25, 4
|
||||
)
|
||||
return rec
|
||||
|
||||
|
||||
# -- Subcommand implementations -----------------------------------------------
|
||||
|
||||
def cmd_merge(args):
|
||||
"""Merge multiple JSONL files with deduplication by repo_id."""
|
||||
all_records = []
|
||||
for path in args.inputs:
|
||||
records = load_jsonl(path)
|
||||
print(f"Loaded {len(records)} from {path}", file=sys.stderr)
|
||||
all_records.extend(records)
|
||||
|
||||
merged = deduplicate(all_records)
|
||||
save_jsonl(merged, args.output)
|
||||
print(f"Merged: {len(all_records)} -> {len(merged)} unique repos -> {args.output}",
|
||||
file=sys.stderr)
|
||||
|
||||
|
||||
def cmd_filter(args):
|
||||
"""Filter repos by various criteria."""
|
||||
records = load_jsonl(args.input)
|
||||
kept = []
|
||||
|
||||
for rec in records:
|
||||
if args.min_stars is not None and rec.get("stars", 0) < args.min_stars:
|
||||
continue
|
||||
if args.min_score is not None and rec.get("composite_score", 0.0) < args.min_score:
|
||||
continue
|
||||
if args.language and rec.get("language", "").lower() != args.language.lower():
|
||||
continue
|
||||
if args.not_archived and rec.get("archived", False):
|
||||
continue
|
||||
kept.append(rec)
|
||||
|
||||
# Sort by composite_score descending
|
||||
kept.sort(key=lambda r: -(r.get("composite_score", 0.0) or 0.0))
|
||||
|
||||
if args.max_repos and args.max_repos > 0 and len(kept) > args.max_repos:
|
||||
kept = kept[:args.max_repos]
|
||||
|
||||
save_jsonl(kept, args.output)
|
||||
print(f"Filtered: {len(records)} -> {len(kept)} repos -> {args.output}",
|
||||
file=sys.stderr)
|
||||
|
||||
|
||||
def cmd_score(args):
|
||||
"""Compute composite scores for all repos."""
|
||||
records = load_jsonl(args.input)
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
scored = [score_record(rec, now) for rec in records]
|
||||
save_jsonl(scored, args.output)
|
||||
print(f"Scored {len(scored)} repos -> {args.output}", file=sys.stderr)
|
||||
|
||||
|
||||
def cmd_search(args):
|
||||
"""Search repos by keyword match in a field."""
|
||||
records = load_jsonl(args.input)
|
||||
query_lower = args.query.lower()
|
||||
results = []
|
||||
|
||||
for rec in records:
|
||||
value = rec.get(args.field, "")
|
||||
if isinstance(value, list):
|
||||
value = " ".join(str(v) for v in value)
|
||||
if query_lower in str(value).lower():
|
||||
results.append(rec)
|
||||
|
||||
for rec in results:
|
||||
print(json.dumps(rec, ensure_ascii=False))
|
||||
print(f"Found {len(results)} matches", file=sys.stderr)
|
||||
|
||||
|
||||
def cmd_tag(args):
|
||||
"""Add tags to specific repos. Supports 'relevance:0.85' format."""
|
||||
records = load_jsonl(args.input)
|
||||
id_set = set(args.ids)
|
||||
tagged = 0
|
||||
|
||||
# Separate relevance assignments from plain tags
|
||||
plain_tags = []
|
||||
relevance_val = None
|
||||
for t in args.tags:
|
||||
if t.startswith("relevance:"):
|
||||
try:
|
||||
relevance_val = float(t.split(":", 1)[1])
|
||||
except ValueError:
|
||||
plain_tags.append(t)
|
||||
else:
|
||||
plain_tags.append(t)
|
||||
|
||||
for rec in records:
|
||||
rid = rec.get("repo_id", "")
|
||||
if rid in id_set:
|
||||
if plain_tags:
|
||||
existing = rec.get("tags", [])
|
||||
rec["tags"] = sorted(set(existing + plain_tags))
|
||||
if relevance_val is not None:
|
||||
rec["relevance_score"] = relevance_val
|
||||
tagged += 1
|
||||
|
||||
save_jsonl(records, args.input)
|
||||
msg_parts = []
|
||||
if plain_tags:
|
||||
msg_parts.append(f"tags={plain_tags}")
|
||||
if relevance_val is not None:
|
||||
msg_parts.append(f"relevance_score={relevance_val}")
|
||||
print(f"Tagged {tagged} repos with {', '.join(msg_parts)}", file=sys.stderr)
|
||||
|
||||
|
||||
def cmd_stats(args):
|
||||
"""Compute and print JSON summary statistics."""
|
||||
records = load_jsonl(args.input)
|
||||
if not records:
|
||||
print(json.dumps({"total": 0}, indent=2))
|
||||
return
|
||||
|
||||
languages: dict[str, int] = {}
|
||||
sources: dict[str, int] = {}
|
||||
tags_dist: dict[str, int] = {}
|
||||
total_stars = 0
|
||||
total_forks = 0
|
||||
archived_count = 0
|
||||
with_readme = 0
|
||||
with_papers = 0
|
||||
scored_count = 0
|
||||
score_sum = 0.0
|
||||
|
||||
for rec in records:
|
||||
lang = rec.get("language") or "Unknown"
|
||||
languages[lang] = languages.get(lang, 0) + 1
|
||||
|
||||
src = rec.get("source", "unknown")
|
||||
sources[src] = sources.get(src, 0) + 1
|
||||
|
||||
total_stars += rec.get("stars", 0)
|
||||
total_forks += rec.get("forks", 0)
|
||||
|
||||
if rec.get("archived"):
|
||||
archived_count += 1
|
||||
if rec.get("readme_excerpt"):
|
||||
with_readme += 1
|
||||
if rec.get("paper_ids"):
|
||||
with_papers += 1
|
||||
|
||||
cs = rec.get("composite_score", 0.0) or 0.0
|
||||
if cs > 0:
|
||||
scored_count += 1
|
||||
score_sum += cs
|
||||
|
||||
for tag in rec.get("tags", []):
|
||||
tags_dist[tag] = tags_dist.get(tag, 0) + 1
|
||||
|
||||
stats = {
|
||||
"total": len(records),
|
||||
"archived": archived_count,
|
||||
"with_readme": with_readme,
|
||||
"with_papers": with_papers,
|
||||
"total_stars": total_stars,
|
||||
"total_forks": total_forks,
|
||||
"avg_stars": round(total_stars / len(records), 1),
|
||||
"avg_composite_score": round(score_sum / scored_count, 4) if scored_count else 0.0,
|
||||
"languages": dict(sorted(languages.items(), key=lambda x: -x[1])[:15]),
|
||||
"sources": sources,
|
||||
"tags": tags_dist,
|
||||
}
|
||||
print(json.dumps(stats, indent=2))
|
||||
|
||||
|
||||
def cmd_export(args):
|
||||
"""Export database in csv, jsonl, or markdown format."""
|
||||
records = load_jsonl(args.input)
|
||||
if not records:
|
||||
print("No records to export.", file=sys.stderr)
|
||||
return
|
||||
|
||||
fmt = args.format
|
||||
|
||||
if fmt == "csv":
|
||||
output = _export_csv(records)
|
||||
elif fmt == "jsonl":
|
||||
output = "\n".join(json.dumps(r, ensure_ascii=False) for r in records) + "\n"
|
||||
elif fmt == "markdown":
|
||||
output = _export_markdown(records)
|
||||
else:
|
||||
print(f"Unknown format: {fmt}", file=sys.stderr)
|
||||
return
|
||||
|
||||
if args.output:
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
f.write(output)
|
||||
print(f"Exported {len(records)} repos to {args.output}", file=sys.stderr)
|
||||
else:
|
||||
print(output, end="")
|
||||
|
||||
|
||||
def _export_csv(records: list[dict]) -> str:
|
||||
fields = [
|
||||
"repo_id", "name", "owner", "stars", "forks", "language", "license",
|
||||
"composite_score", "quality_score", "activity_score", "relevance_score",
|
||||
"topics", "tags", "archived", "source",
|
||||
]
|
||||
buf = io.StringIO()
|
||||
writer = csv.DictWriter(buf, fieldnames=fields, extrasaction="ignore")
|
||||
writer.writeheader()
|
||||
for rec in records:
|
||||
row = dict(rec)
|
||||
if isinstance(row.get("topics"), list):
|
||||
row["topics"] = "; ".join(row["topics"])
|
||||
if isinstance(row.get("tags"), list):
|
||||
row["tags"] = "; ".join(row["tags"])
|
||||
writer.writerow(row)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def _export_markdown(records: list[dict]) -> str:
|
||||
lines = ["| Repo | Stars | Language | Score | Description |",
|
||||
"|------|------:|----------|------:|-------------|"]
|
||||
for rec in records:
|
||||
rid = rec.get("repo_id", "")
|
||||
url = rec.get("url", f"https://github.com/{rid}")
|
||||
stars = rec.get("stars", 0)
|
||||
lang = rec.get("language", "")
|
||||
score = rec.get("composite_score", 0.0) or 0.0
|
||||
desc = (rec.get("description") or "")[:80]
|
||||
desc = desc.replace("|", "\\|")
|
||||
lines.append(f"| [{rid}]({url}) | {stars} | {lang} | {score:.3f} | {desc} |")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def cmd_rank(args):
|
||||
"""Rank repos by a given field descending."""
|
||||
records = load_jsonl(args.input)
|
||||
# Map CLI choice to actual record field name
|
||||
field_map = {
|
||||
"composite_score": "composite_score",
|
||||
"stars": "stars",
|
||||
"updated": "updated_at",
|
||||
}
|
||||
field = field_map.get(args.by, args.by)
|
||||
|
||||
def sort_key(r):
|
||||
val = r.get(field, 0)
|
||||
if val is None:
|
||||
return ""
|
||||
# For date strings, lexicographic sort works with ISO format
|
||||
if isinstance(val, str):
|
||||
return val
|
||||
return val
|
||||
|
||||
records.sort(key=sort_key, reverse=True)
|
||||
|
||||
for i, rec in enumerate(records):
|
||||
rec["rank"] = i + 1
|
||||
|
||||
save_jsonl(records, args.output)
|
||||
print(f"Ranked {len(records)} repos by {field} -> {args.output}", file=sys.stderr)
|
||||
|
||||
|
||||
# -- CLI -----------------------------------------------------------------------
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="JSONL GitHub repo database management tool"
|
||||
)
|
||||
sub = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
# merge
|
||||
p = sub.add_parser("merge", help="Merge multiple JSONL files with dedup by repo_id")
|
||||
p.add_argument("--inputs", nargs="+", required=True, help="Input JSONL files")
|
||||
p.add_argument("--output", required=True, help="Output JSONL file")
|
||||
|
||||
# filter
|
||||
p = sub.add_parser("filter", help="Filter repos by stars, score, language, etc.")
|
||||
p.add_argument("--input", required=True, help="Input JSONL file")
|
||||
p.add_argument("--output", required=True, help="Output JSONL file")
|
||||
p.add_argument("--min-stars", type=int, default=None, help="Minimum star count")
|
||||
p.add_argument("--min-score", type=float, default=None, help="Minimum composite score")
|
||||
p.add_argument("--max-repos", type=int, default=None, help="Max repos to keep (0=unlimited)")
|
||||
p.add_argument("--language", default=None, help="Filter by primary language")
|
||||
p.add_argument("--not-archived", action="store_true", help="Exclude archived repos")
|
||||
|
||||
# score
|
||||
p = sub.add_parser("score", help="Compute composite scores for all repos")
|
||||
p.add_argument("--input", required=True, help="Input JSONL file")
|
||||
p.add_argument("--output", required=True, help="Output scored JSONL file")
|
||||
|
||||
# search
|
||||
p = sub.add_parser("search", help="Search repos by keyword in a field")
|
||||
p.add_argument("--input", required=True, help="Input JSONL file")
|
||||
p.add_argument("--query", required=True, help="Search query string")
|
||||
p.add_argument("--field", default="description",
|
||||
choices=["description", "topics", "name"],
|
||||
help="Field to search (default: description)")
|
||||
|
||||
# tag
|
||||
p = sub.add_parser("tag", help="Add tags to repos; supports 'relevance:0.85' format")
|
||||
p.add_argument("--input", required=True, help="Input JSONL file (modified in-place)")
|
||||
p.add_argument("--ids", nargs="+", required=True, help="Repo IDs (owner/name)")
|
||||
p.add_argument("--tags", nargs="+", required=True,
|
||||
help="Tags to add; use 'relevance:0.85' to set relevance_score")
|
||||
|
||||
# stats
|
||||
p = sub.add_parser("stats", help="Print JSON summary statistics")
|
||||
p.add_argument("--input", required=True, help="Input JSONL file")
|
||||
|
||||
# export
|
||||
p = sub.add_parser("export", help="Export database to csv, jsonl, or markdown")
|
||||
p.add_argument("--input", required=True, help="Input JSONL file")
|
||||
p.add_argument("--format", choices=["csv", "jsonl", "markdown"], default="jsonl",
|
||||
help="Output format (default: jsonl)")
|
||||
p.add_argument("--output", "-o", default=None, help="Output file (default: stdout)")
|
||||
|
||||
# rank
|
||||
p = sub.add_parser("rank", help="Rank repos by a field descending")
|
||||
p.add_argument("--input", required=True, help="Input JSONL file")
|
||||
p.add_argument("--output", required=True, help="Output ranked JSONL file")
|
||||
p.add_argument("--by", default="composite_score",
|
||||
choices=["composite_score", "stars", "updated"],
|
||||
help="Field to rank by (default: composite_score)")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
dispatch = {
|
||||
"merge": cmd_merge,
|
||||
"filter": cmd_filter,
|
||||
"score": cmd_score,
|
||||
"search": cmd_search,
|
||||
"tag": cmd_tag,
|
||||
"stats": cmd_stats,
|
||||
"export": cmd_export,
|
||||
"rank": cmd_rank,
|
||||
}
|
||||
dispatch[args.command](args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+203
@@ -0,0 +1,203 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch detailed metadata for GitHub repos via ``gh api``.
|
||||
|
||||
Retrieves main repo info and language breakdown, then maps the response
|
||||
to the repo_db schema. Can enrich existing JSONL records or create new
|
||||
ones from scratch.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GitHub helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def gh_api(endpoint: str) -> dict | None:
|
||||
"""Call ``gh api`` and return parsed JSON, or *None* on failure."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["gh", "api", endpoint],
|
||||
capture_output=True, text=True, timeout=30,
|
||||
)
|
||||
if proc.returncode == 0:
|
||||
return json.loads(proc.stdout)
|
||||
stderr = proc.stderr.strip()
|
||||
if "404" in stderr or "Not Found" in stderr:
|
||||
print(f" 404 — repo not found", file=sys.stderr)
|
||||
elif "rate limit" in stderr.lower() or "403" in stderr:
|
||||
print(f" Rate limited. Consider increasing --delay.", file=sys.stderr)
|
||||
else:
|
||||
print(f" gh api error ({proc.returncode}): {stderr[:120]}", file=sys.stderr)
|
||||
except FileNotFoundError:
|
||||
print("Error: 'gh' CLI not found. Please install GitHub CLI.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except (subprocess.TimeoutExpired, json.JSONDecodeError) as exc:
|
||||
print(f" gh api exception: {exc}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Language breakdown
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def compute_languages_pct(lang_data: dict) -> dict[str, float]:
|
||||
"""Convert raw byte counts to percentages (0–100)."""
|
||||
total = sum(lang_data.values())
|
||||
if total == 0:
|
||||
return {}
|
||||
return {lang: round(count / total * 100, 2) for lang, count in lang_data.items()}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Record building
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def build_record(repo_id: str, meta: dict, lang_data: dict) -> dict:
|
||||
"""Build a repo_db-schema record from GitHub API responses."""
|
||||
license_info = meta.get("license") or {}
|
||||
languages_pct = compute_languages_pct(lang_data)
|
||||
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"github_url": meta.get("html_url", f"https://github.com/{repo_id}"),
|
||||
"description": meta.get("description", ""),
|
||||
"homepage": meta.get("homepage", ""),
|
||||
"stars": meta.get("stargazers_count", 0),
|
||||
"forks_count": meta.get("forks_count", 0),
|
||||
"open_issues_count": meta.get("open_issues_count", 0),
|
||||
"watchers_count": meta.get("watchers_count", 0),
|
||||
"default_branch": meta.get("default_branch", ""),
|
||||
"license": license_info.get("spdx_id") or license_info.get("name", ""),
|
||||
"language": meta.get("language", ""),
|
||||
"languages_pct": languages_pct,
|
||||
"topics": meta.get("topics", []),
|
||||
"is_fork": meta.get("fork", False),
|
||||
"is_archived": meta.get("archived", False),
|
||||
"size_kb": meta.get("size", 0),
|
||||
"created_at": meta.get("created_at", ""),
|
||||
"updated_at": meta.get("updated_at", ""),
|
||||
"pushed_at": meta.get("pushed_at", ""),
|
||||
"owner_type": (meta.get("owner") or {}).get("type", ""),
|
||||
"network_count": meta.get("network_count", 0),
|
||||
"subscribers_count": meta.get("subscribers_count", 0),
|
||||
}
|
||||
|
||||
|
||||
def enrich_record(existing: dict, fresh: dict) -> dict:
|
||||
"""Merge *fresh* metadata into an *existing* record.
|
||||
|
||||
Fresh values overwrite existing ones, except we preserve fields
|
||||
that exist only in the original (e.g. ``source``, ``paper_ids``).
|
||||
"""
|
||||
merged = dict(existing)
|
||||
merged.update(fresh)
|
||||
# Preserve provenance fields from the original record.
|
||||
for key in ("source", "paper_ids", "paper_titles", "is_official", "framework"):
|
||||
if key in existing:
|
||||
merged[key] = existing[key]
|
||||
return merged
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# I/O helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def load_jsonl(path: str) -> list[dict]:
|
||||
"""Load a JSONL file into a list of dicts."""
|
||||
records: list[dict] = []
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
records.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
return records
|
||||
|
||||
|
||||
def write_jsonl(records: list[dict], path: str) -> None:
|
||||
"""Write a list of dicts as JSONL."""
|
||||
with open(path, "w", encoding="utf-8") as fout:
|
||||
for rec in records:
|
||||
fout.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Fetch detailed GitHub repo metadata via gh api.",
|
||||
)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument("--repos", nargs="+", metavar="OWNER/NAME",
|
||||
help="One or more repos as owner/name")
|
||||
source.add_argument("--input", metavar="FILE",
|
||||
help="JSONL file with existing repo records to enrich")
|
||||
parser.add_argument("--output", required=True, help="Output JSONL file path")
|
||||
parser.add_argument("--delay", type=float, default=0.5,
|
||||
help="Seconds between API requests per repo (default: 0.5)")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Determine which repos to process.
|
||||
existing_records: dict[str, dict] = {} # keyed by repo_id
|
||||
|
||||
if args.input:
|
||||
for rec in load_jsonl(args.input):
|
||||
rid = rec.get("repo_id")
|
||||
if rid:
|
||||
existing_records[rid] = rec
|
||||
repo_ids = list(existing_records.keys())
|
||||
else:
|
||||
repo_ids = args.repos or []
|
||||
|
||||
if not repo_ids:
|
||||
print("No repos to process.", file=sys.stderr)
|
||||
sys.exit(0)
|
||||
|
||||
total = len(repo_ids)
|
||||
results: list[dict] = []
|
||||
|
||||
for idx, repo_id in enumerate(repo_ids, 1):
|
||||
print(f"Fetching metadata: {idx}/{total} — {repo_id}", file=sys.stderr)
|
||||
|
||||
meta = gh_api(f"/repos/{repo_id}")
|
||||
if meta is None:
|
||||
print(f" Skipping {repo_id} (metadata unavailable)", file=sys.stderr)
|
||||
# Keep the original record if enriching.
|
||||
if repo_id in existing_records:
|
||||
results.append(existing_records[repo_id])
|
||||
continue
|
||||
|
||||
time.sleep(args.delay)
|
||||
|
||||
lang_data = gh_api(f"/repos/{repo_id}/languages") or {}
|
||||
|
||||
fresh = build_record(repo_id, meta, lang_data)
|
||||
|
||||
if repo_id in existing_records:
|
||||
record = enrich_record(existing_records[repo_id], fresh)
|
||||
else:
|
||||
record = fresh
|
||||
|
||||
results.append(record)
|
||||
|
||||
if idx < total:
|
||||
time.sleep(args.delay)
|
||||
|
||||
write_jsonl(results, args.output)
|
||||
print(f"Done. {len(results)}/{total} record(s) written to {args.output}",
|
||||
file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fetch README content for GitHub repos without cloning.
|
||||
|
||||
Uses ``gh api`` to retrieve the README of one or more repositories,
|
||||
decodes the base64 payload, and writes the results as JSONL.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
|
||||
def gh_api(endpoint: str) -> dict | None:
|
||||
"""Call ``gh api`` and return parsed JSON, or *None* on failure."""
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["gh", "api", endpoint],
|
||||
capture_output=True, text=True, timeout=30,
|
||||
)
|
||||
if proc.returncode == 0:
|
||||
return json.loads(proc.stdout)
|
||||
# Distinguish 404 (no README) from other errors.
|
||||
if "404" in proc.stderr or "Not Found" in proc.stderr:
|
||||
return None
|
||||
print(f" gh api error ({proc.returncode}): {proc.stderr.strip()[:120]}",
|
||||
file=sys.stderr)
|
||||
except FileNotFoundError:
|
||||
print(" Error: 'gh' CLI not found. Please install GitHub CLI.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except (subprocess.TimeoutExpired, json.JSONDecodeError) as exc:
|
||||
print(f" gh api exception: {exc}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
def fetch_readme(repo_id: str, max_chars: int) -> dict | None:
|
||||
"""Return a dict with repo_id, readme_text, readme_length or *None*."""
|
||||
data = gh_api(f"/repos/{repo_id}/readme")
|
||||
if data is None:
|
||||
return None
|
||||
|
||||
content_b64 = data.get("content", "")
|
||||
try:
|
||||
raw = base64.b64decode(content_b64).decode("utf-8", errors="replace")
|
||||
except Exception:
|
||||
raw = ""
|
||||
|
||||
truncated = raw[:max_chars]
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"readme_text": truncated,
|
||||
"readme_length": len(raw),
|
||||
}
|
||||
|
||||
|
||||
def load_repo_ids_from_jsonl(path: str) -> list[str]:
|
||||
"""Read repo_id values from a JSONL file."""
|
||||
ids: list[str] = []
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
rid = obj.get("repo_id")
|
||||
if rid:
|
||||
ids.append(rid)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
return ids
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Fetch README content for GitHub repos without cloning.",
|
||||
)
|
||||
source = parser.add_mutually_exclusive_group(required=True)
|
||||
source.add_argument("--repos", nargs="+", metavar="OWNER/NAME",
|
||||
help="One or more repos as owner/name")
|
||||
source.add_argument("--input", metavar="FILE",
|
||||
help="JSONL file with repo_id fields")
|
||||
parser.add_argument("--output", required=True, help="Output JSONL file path")
|
||||
parser.add_argument("--max-chars", type=int, default=5000,
|
||||
help="Max characters to keep from README (default: 5000)")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
repo_ids: list[str] = args.repos if args.repos else load_repo_ids_from_jsonl(args.input)
|
||||
|
||||
if not repo_ids:
|
||||
print("No repos to process.", file=sys.stderr)
|
||||
sys.exit(0)
|
||||
|
||||
total = len(repo_ids)
|
||||
results: list[dict] = []
|
||||
|
||||
for idx, repo_id in enumerate(repo_ids, 1):
|
||||
print(f"Fetching README: {idx}/{total} — {repo_id}", file=sys.stderr)
|
||||
record = fetch_readme(repo_id, max_chars=args.max_chars)
|
||||
if record:
|
||||
results.append(record)
|
||||
else:
|
||||
print(f" No README found for {repo_id}", file=sys.stderr)
|
||||
if idx < total:
|
||||
time.sleep(0.5)
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as fout:
|
||||
for rec in results:
|
||||
fout.write(json.dumps(rec, ensure_ascii=False) + "\n")
|
||||
|
||||
print(f"Done. {len(results)}/{total} README(s) written to {args.output}",
|
||||
file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+265
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search GitHub repositories via `gh api` with multiple query strategies.
|
||||
|
||||
Builds qualified search queries and paginates through results, outputting
|
||||
repo_db-compatible JSONL records.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
RESULTS_PER_PAGE = 30
|
||||
DEFAULT_MAX_RESULTS = 100
|
||||
|
||||
|
||||
def check_gh_installed():
|
||||
"""Verify gh CLI is available."""
|
||||
if not shutil.which("gh"):
|
||||
print("[error] gh CLI not found. Install it: https://cli.github.com/", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def gh_api(endpoint: str, params: dict | None = None) -> dict:
|
||||
"""Call gh api and return parsed JSON."""
|
||||
if params:
|
||||
from urllib.parse import urlencode
|
||||
endpoint = f"{endpoint}?{urlencode(params)}"
|
||||
cmd = ["gh", "api", endpoint]
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
print("[error] gh CLI not found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except subprocess.TimeoutExpired:
|
||||
print("[error] gh api call timed out", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if result.returncode != 0:
|
||||
stderr = result.stderr.strip()
|
||||
# Check for rate limit
|
||||
if "rate limit" in stderr.lower() or "403" in stderr:
|
||||
print(f"[error] GitHub API rate limit hit: {stderr}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(f"[error] gh api failed (exit {result.returncode}): {stderr}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
return json.loads(result.stdout)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"[error] invalid JSON from gh api: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def build_query(query: str, language: str | None, min_stars: int | None, topic: str | None) -> str:
|
||||
"""Build GitHub search query string with qualifiers."""
|
||||
parts = [query]
|
||||
if language:
|
||||
parts.append(f"language:{language}")
|
||||
if min_stars is not None and min_stars > 0:
|
||||
parts.append(f"stars:>={min_stars}")
|
||||
if topic:
|
||||
parts.append(f"topic:{topic}")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def extract_license(license_obj) -> str:
|
||||
"""Extract license identifier from GitHub API license object."""
|
||||
if not license_obj or not isinstance(license_obj, dict):
|
||||
return ""
|
||||
return license_obj.get("spdx_id") or license_obj.get("name") or ""
|
||||
|
||||
|
||||
def map_repo(item: dict) -> dict:
|
||||
"""Map a GitHub API repository item to repo_db schema."""
|
||||
owner_obj = item.get("owner") or {}
|
||||
return {
|
||||
"repo_id": item.get("full_name", ""),
|
||||
"url": item.get("html_url", ""),
|
||||
"name": item.get("name", ""),
|
||||
"owner": owner_obj.get("login", ""),
|
||||
"description": item.get("description") or "",
|
||||
"stars": item.get("stargazers_count", 0),
|
||||
"forks": item.get("forks_count", 0),
|
||||
"language": item.get("language") or "",
|
||||
"license": extract_license(item.get("license")),
|
||||
"topics": item.get("topics", []),
|
||||
"created_at": item.get("created_at", ""),
|
||||
"updated_at": item.get("updated_at", ""),
|
||||
"pushed_at": item.get("pushed_at", ""),
|
||||
"open_issues": item.get("open_issues_count", 0),
|
||||
"default_branch": item.get("default_branch", "main"),
|
||||
"archived": item.get("archived", False),
|
||||
"source": "search_github",
|
||||
# Default fields for repo_db compatibility
|
||||
"languages_pct": {},
|
||||
"readme_excerpt": "",
|
||||
"paper_ids": [],
|
||||
"paper_titles": [],
|
||||
"relevance_score": 0.0,
|
||||
"quality_score": 0.0,
|
||||
"activity_score": 0.0,
|
||||
"composite_score": 0.0,
|
||||
"tags": [],
|
||||
"analyzed": False,
|
||||
"local_path": None,
|
||||
}
|
||||
|
||||
|
||||
def check_rate_limit():
|
||||
"""Check GitHub API rate limit and warn if approaching it."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["gh", "api", "rate_limit"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
data = json.loads(result.stdout)
|
||||
search = data.get("resources", {}).get("search", {})
|
||||
remaining = search.get("remaining", -1)
|
||||
limit = search.get("limit", -1)
|
||||
reset_ts = search.get("reset", 0)
|
||||
|
||||
if remaining >= 0:
|
||||
if remaining <= 3:
|
||||
reset_time = time.strftime("%H:%M:%S", time.localtime(reset_ts))
|
||||
print(
|
||||
f"[warn] GitHub search API rate limit: {remaining}/{limit} remaining, "
|
||||
f"resets at {reset_time}",
|
||||
file=sys.stderr,
|
||||
)
|
||||
else:
|
||||
print(f"[info] rate limit: {remaining}/{limit} search requests remaining", file=sys.stderr)
|
||||
except (subprocess.TimeoutExpired, json.JSONDecodeError, OSError):
|
||||
pass # Non-critical, skip
|
||||
|
||||
|
||||
def search_repos(
|
||||
query: str,
|
||||
sort: str,
|
||||
max_results: int,
|
||||
) -> list[dict]:
|
||||
"""Search GitHub repositories with pagination."""
|
||||
total_pages = math.ceil(max_results / RESULTS_PER_PAGE)
|
||||
all_repos = []
|
||||
seen_ids = set()
|
||||
|
||||
for page in range(1, total_pages + 1):
|
||||
print(f"[info] fetching page {page}/{total_pages} ...", file=sys.stderr)
|
||||
|
||||
params = {
|
||||
"q": query,
|
||||
"sort": sort,
|
||||
"order": "desc",
|
||||
"per_page": str(min(RESULTS_PER_PAGE, max_results - len(all_repos))),
|
||||
"page": str(page),
|
||||
}
|
||||
|
||||
data = gh_api("/search/repositories", params)
|
||||
|
||||
total_count = data.get("total_count", 0)
|
||||
if page == 1:
|
||||
print(f"[info] total matching repos: {total_count}", file=sys.stderr)
|
||||
|
||||
items = data.get("items", [])
|
||||
if not items:
|
||||
print("[info] no more results", file=sys.stderr)
|
||||
break
|
||||
|
||||
for item in items:
|
||||
repo = map_repo(item)
|
||||
if repo["repo_id"] not in seen_ids:
|
||||
seen_ids.add(repo["repo_id"])
|
||||
all_repos.append(repo)
|
||||
|
||||
if len(all_repos) >= max_results:
|
||||
break
|
||||
|
||||
if len(all_repos) >= max_results:
|
||||
break
|
||||
|
||||
# Incomplete results means GitHub truncated
|
||||
if data.get("incomplete_results", False):
|
||||
print("[warn] GitHub returned incomplete results (query too broad?)", file=sys.stderr)
|
||||
|
||||
# Respect rate limit: sleep 2s between pages for search API
|
||||
if page < total_pages:
|
||||
time.sleep(2)
|
||||
|
||||
return all_repos
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Search GitHub repositories via gh API with flexible query strategies."
|
||||
)
|
||||
parser.add_argument("--query", required=True, help="Search query string")
|
||||
parser.add_argument("--language", default=None, help="Filter by programming language")
|
||||
parser.add_argument("--min-stars", type=int, default=None, help="Minimum star count")
|
||||
parser.add_argument(
|
||||
"--sort",
|
||||
choices=["stars", "updated", "best-match"],
|
||||
default="best-match",
|
||||
help="Sort order (default: best-match)",
|
||||
)
|
||||
parser.add_argument("--max-results", type=int, default=DEFAULT_MAX_RESULTS, help="Max results to fetch (default: 100)")
|
||||
parser.add_argument("--topic", default=None, help="Filter by GitHub topic")
|
||||
parser.add_argument("--output", required=True, help="Output JSONL file path")
|
||||
args = parser.parse_args()
|
||||
|
||||
check_gh_installed()
|
||||
|
||||
# Build the qualified query
|
||||
full_query = build_query(args.query, args.language, args.min_stars, args.topic)
|
||||
print(f"[info] search query: {full_query}", file=sys.stderr)
|
||||
print(f"[info] sort: {args.sort}, max: {args.max_results}", file=sys.stderr)
|
||||
|
||||
# Check rate limit before starting
|
||||
check_rate_limit()
|
||||
|
||||
# Handle "best-match" -> empty sort for GitHub API (relevance is default)
|
||||
sort_param = "" if args.sort == "best-match" else args.sort
|
||||
|
||||
repos = search_repos(full_query, sort_param, args.max_results)
|
||||
|
||||
# Write output
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
for repo in repos:
|
||||
f.write(json.dumps(repo, ensure_ascii=False) + "\n")
|
||||
|
||||
# Summary stats
|
||||
print(f"\n[info] results summary:", file=sys.stderr)
|
||||
print(f" repos found: {len(repos)}", file=sys.stderr)
|
||||
if repos:
|
||||
languages = {}
|
||||
for r in repos:
|
||||
lang = r["language"] or "unknown"
|
||||
languages[lang] = languages.get(lang, 0) + 1
|
||||
top_langs = sorted(languages.items(), key=lambda x: -x[1])[:5]
|
||||
print(f" top languages: {', '.join(f'{l}({c})' for l, c in top_langs)}", file=sys.stderr)
|
||||
|
||||
star_counts = [r["stars"] for r in repos]
|
||||
print(f" stars range: {min(star_counts)} - {max(star_counts)}", file=sys.stderr)
|
||||
|
||||
print(f"[info] written to {output_path}", file=sys.stderr)
|
||||
|
||||
# Check rate limit after
|
||||
check_rate_limit()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+251
@@ -0,0 +1,251 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search GitHub code for specific function/class implementations via `gh api`.
|
||||
|
||||
Searches code using GitHub's code search API, groups results by repo, and
|
||||
outputs both code-level matches (stderr) and repo-level JSONL (--output).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
RESULTS_PER_PAGE = 30
|
||||
DEFAULT_MAX_RESULTS = 50
|
||||
|
||||
|
||||
def check_gh_installed():
|
||||
"""Verify gh CLI is available."""
|
||||
if not shutil.which("gh"):
|
||||
print("[error] gh CLI not found. Install it: https://cli.github.com/", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def gh_api(endpoint: str, params: dict | None = None) -> dict:
|
||||
"""Call gh api and return parsed JSON."""
|
||||
if params:
|
||||
from urllib.parse import urlencode
|
||||
endpoint = f"{endpoint}?{urlencode(params)}"
|
||||
cmd = ["gh", "api", endpoint]
|
||||
try:
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
print("[error] gh CLI not found", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
except subprocess.TimeoutExpired:
|
||||
print("[error] gh api call timed out", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if result.returncode != 0:
|
||||
stderr = result.stderr.strip()
|
||||
if "rate limit" in stderr.lower() or "403" in stderr:
|
||||
print(f"[error] GitHub API rate limit hit: {stderr}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(f"[error] gh api failed (exit {result.returncode}): {stderr}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
return json.loads(result.stdout)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"[error] invalid JSON from gh api: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def build_code_query(query: str, language: str | None, filename: str | None) -> str:
|
||||
"""Build GitHub code search query with qualifiers."""
|
||||
parts = [query]
|
||||
if language:
|
||||
parts.append(f"language:{language}")
|
||||
if filename:
|
||||
parts.append(f"filename:{filename}")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def search_code(query: str, max_results: int) -> list[dict]:
|
||||
"""Search GitHub code with pagination."""
|
||||
total_pages = math.ceil(max_results / RESULTS_PER_PAGE)
|
||||
all_items = []
|
||||
|
||||
for page in range(1, total_pages + 1):
|
||||
print(f"[info] fetching page {page}/{total_pages} ...", file=sys.stderr)
|
||||
|
||||
params = {
|
||||
"q": query,
|
||||
"per_page": str(min(RESULTS_PER_PAGE, max_results - len(all_items))),
|
||||
"page": str(page),
|
||||
}
|
||||
|
||||
data = gh_api("/search/code", params)
|
||||
|
||||
total_count = data.get("total_count", 0)
|
||||
if page == 1:
|
||||
print(f"[info] total matching files: {total_count}", file=sys.stderr)
|
||||
|
||||
items = data.get("items", [])
|
||||
if not items:
|
||||
print("[info] no more results", file=sys.stderr)
|
||||
break
|
||||
|
||||
all_items.extend(items)
|
||||
|
||||
if len(all_items) >= max_results:
|
||||
all_items = all_items[:max_results]
|
||||
break
|
||||
|
||||
if data.get("incomplete_results", False):
|
||||
print("[warn] GitHub returned incomplete results", file=sys.stderr)
|
||||
|
||||
# Code search is heavily rate limited: 10 req/min
|
||||
if page < total_pages:
|
||||
time.sleep(6)
|
||||
|
||||
return all_items
|
||||
|
||||
|
||||
def extract_code_match(item: dict) -> dict:
|
||||
"""Extract a code match record from a search result item."""
|
||||
repo = item.get("repository", {})
|
||||
owner_obj = repo.get("owner", {})
|
||||
full_name = repo.get("full_name", "")
|
||||
|
||||
# text_matches may be present if Accept header includes text-match
|
||||
text_fragments = []
|
||||
for tm in item.get("text_matches", []):
|
||||
fragment = tm.get("fragment", "")
|
||||
if fragment:
|
||||
text_fragments.append(fragment)
|
||||
|
||||
return {
|
||||
"repo_id": full_name,
|
||||
"file_path": item.get("path", ""),
|
||||
"file_url": item.get("html_url", ""),
|
||||
"file_name": item.get("name", ""),
|
||||
"matched_text": "\n---\n".join(text_fragments) if text_fragments else "",
|
||||
"context": {
|
||||
"sha": item.get("sha", ""),
|
||||
"score": item.get("score", 0),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def build_repo_record(repo_id: str, repo_data: dict) -> dict:
|
||||
"""Build a minimal repo_db-compatible record from code search results."""
|
||||
owner = repo_id.split("/")[0] if "/" in repo_id else ""
|
||||
name = repo_id.split("/")[1] if "/" in repo_id else repo_id
|
||||
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"url": f"https://github.com/{repo_id}",
|
||||
"name": name,
|
||||
"owner": owner,
|
||||
"description": repo_data.get("description", ""),
|
||||
"stars": 0,
|
||||
"forks": 0,
|
||||
"language": "",
|
||||
"license": "",
|
||||
"topics": [],
|
||||
"created_at": "",
|
||||
"updated_at": "",
|
||||
"pushed_at": "",
|
||||
"open_issues": 0,
|
||||
"default_branch": "main",
|
||||
"archived": False,
|
||||
"source": "code_search",
|
||||
"code_matches": repo_data.get("files", []),
|
||||
"languages_pct": {},
|
||||
"readme_excerpt": "",
|
||||
"paper_ids": [],
|
||||
"paper_titles": [],
|
||||
"relevance_score": 0.0,
|
||||
"quality_score": 0.0,
|
||||
"activity_score": 0.0,
|
||||
"composite_score": 0.0,
|
||||
"tags": [],
|
||||
"analyzed": False,
|
||||
"local_path": None,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Search GitHub code for function/class implementations via gh API."
|
||||
)
|
||||
parser.add_argument("--query", required=True, help="Code search query string")
|
||||
parser.add_argument("--language", default=None, help="Filter by programming language")
|
||||
parser.add_argument("--filename", default=None, help="Filter by filename pattern")
|
||||
parser.add_argument("--max-results", type=int, default=DEFAULT_MAX_RESULTS, help="Max code results to fetch (default: 50)")
|
||||
parser.add_argument("--output", required=True, help="Output JSONL file for repo-level records")
|
||||
args = parser.parse_args()
|
||||
|
||||
check_gh_installed()
|
||||
|
||||
full_query = build_code_query(args.query, args.language, args.filename)
|
||||
print(f"[info] code search query: {full_query}", file=sys.stderr)
|
||||
print(f"[info] max results: {args.max_results}", file=sys.stderr)
|
||||
|
||||
items = search_code(full_query, args.max_results)
|
||||
|
||||
# Process matches and group by repo
|
||||
code_matches = []
|
||||
repos: dict[str, dict] = {} # repo_id -> {description, file_count, files}
|
||||
|
||||
for item in items:
|
||||
match = extract_code_match(item)
|
||||
code_matches.append(match)
|
||||
|
||||
rid = match["repo_id"]
|
||||
if rid not in repos:
|
||||
repo_obj = item.get("repository", {})
|
||||
repos[rid] = {
|
||||
"description": repo_obj.get("description") or "",
|
||||
"file_count": 0,
|
||||
"files": [],
|
||||
}
|
||||
repos[rid]["file_count"] += 1
|
||||
repos[rid]["files"].append(match["file_path"])
|
||||
|
||||
# Print code matches to stderr, grouped by repo
|
||||
print(f"\n{'=' * 60}", file=sys.stderr)
|
||||
print(f"Code matches: {len(code_matches)} files in {len(repos)} repos", file=sys.stderr)
|
||||
print(f"{'=' * 60}", file=sys.stderr)
|
||||
|
||||
for rid in sorted(repos.keys()):
|
||||
info = repos[rid]
|
||||
print(f"\n {rid} ({info['file_count']} files)", file=sys.stderr)
|
||||
if info["description"]:
|
||||
print(f" {info['description'][:100]}", file=sys.stderr)
|
||||
for fpath in info["files"][:5]:
|
||||
print(f" - {fpath}", file=sys.stderr)
|
||||
if len(info["files"]) > 5:
|
||||
print(f" ... and {len(info['files']) - 5} more", file=sys.stderr)
|
||||
|
||||
# Write repo-level JSONL to --output (for feeding into repo_db merge)
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
for rid, rdata in sorted(repos.items()):
|
||||
record = build_repo_record(rid, rdata)
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
|
||||
print(f"\n[info] wrote {len(repos)} repo records to {output_path}", file=sys.stderr)
|
||||
|
||||
# Also write code-level matches alongside (with .code suffix)
|
||||
code_output = output_path.with_suffix(".code.jsonl")
|
||||
with open(code_output, "w", encoding="utf-8") as f:
|
||||
for match in code_matches:
|
||||
f.write(json.dumps(match, ensure_ascii=False) + "\n")
|
||||
|
||||
print(f"[info] wrote {len(code_matches)} code matches to {code_output}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+413
@@ -0,0 +1,413 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Search Papers With Code for paper-to-repo mappings.
|
||||
|
||||
Uses the Papers With Code public API (https://paperswithcode.com/api/v1/)
|
||||
to find GitHub repositories linked to research papers. Supports lookup by
|
||||
single arXiv ID, multiple IDs, a file of IDs, or keyword search.
|
||||
|
||||
Self-contained: stdlib only (urllib for HTTP).
|
||||
|
||||
Usage:
|
||||
python search_paperswithcode.py --arxiv-id 2301.12345 --output repos.jsonl
|
||||
python search_paperswithcode.py --query "multi-agent" --output repos.jsonl
|
||||
python search_paperswithcode.py --arxiv-ids-file papers.txt --output repos.jsonl
|
||||
python search_paperswithcode.py --arxiv-ids 2301.12345 2305.67890 --output repos.jsonl
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
PWC_API = "https://paperswithcode.com/api/v1"
|
||||
USER_AGENT = "github-research/1.0"
|
||||
REQUEST_DELAY = 1.0 # seconds between requests
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# HTTP helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def pwc_request(url: str) -> dict | list | None:
|
||||
"""Make a GET request to the Papers With Code API with retry logic.
|
||||
|
||||
Returns parsed JSON or None on failure.
|
||||
"""
|
||||
headers = {"User-Agent": USER_AGENT, "Accept": "application/json"}
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
|
||||
for attempt in range(3):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
return None
|
||||
if e.code == 429:
|
||||
wait = 2 ** (attempt + 1)
|
||||
print(f"[warn] rate limited, waiting {wait}s ...", file=sys.stderr)
|
||||
time.sleep(wait)
|
||||
continue
|
||||
print(f"[warn] HTTP {e.code} for {url}", file=sys.stderr)
|
||||
if attempt < 2:
|
||||
time.sleep(1)
|
||||
continue
|
||||
return None
|
||||
except (urllib.error.URLError, OSError) as e:
|
||||
print(f"[warn] request failed: {e}", file=sys.stderr)
|
||||
if attempt < 2:
|
||||
time.sleep(1)
|
||||
continue
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def paginate_results(url: str) -> list[dict]:
|
||||
"""Follow ``next`` links and collect all ``results`` items."""
|
||||
items: list[dict] = []
|
||||
current_url: str | None = url
|
||||
while current_url:
|
||||
data = pwc_request(current_url)
|
||||
if not data or not isinstance(data, dict):
|
||||
break
|
||||
items.extend(data.get("results", []))
|
||||
current_url = data.get("next")
|
||||
if current_url:
|
||||
time.sleep(REQUEST_DELAY)
|
||||
return items
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GitHub URL parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def parse_repo_url(url: str) -> tuple[str, str]:
|
||||
"""Parse a GitHub URL into (owner, name). Returns ("", "") if not GitHub."""
|
||||
if not url or "github.com" not in url:
|
||||
return "", ""
|
||||
url = url.rstrip("/")
|
||||
for prefix in ("https://github.com/", "http://github.com/"):
|
||||
if url.startswith(prefix):
|
||||
path = url[len(prefix):]
|
||||
parts = path.split("/")
|
||||
if len(parts) >= 2 and parts[0] and parts[1]:
|
||||
name = parts[1].removesuffix(".git")
|
||||
return parts[0], name
|
||||
return "", ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Repo record mapping
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def map_to_repo_record(
|
||||
repo: dict,
|
||||
paper_title: str = "",
|
||||
arxiv_id: str = "",
|
||||
) -> dict | None:
|
||||
"""Map a PWC repository entry to the repo_db schema.
|
||||
|
||||
Returns None if the URL is not a valid GitHub repo.
|
||||
"""
|
||||
url = repo.get("url", "")
|
||||
owner, name = parse_repo_url(url)
|
||||
if not owner or not name:
|
||||
return None
|
||||
|
||||
repo_id = f"{owner}/{name}"
|
||||
paper_ids = [arxiv_id] if arxiv_id else []
|
||||
paper_titles = [paper_title] if paper_title else []
|
||||
|
||||
return {
|
||||
"repo_id": repo_id,
|
||||
"url": f"https://github.com/{repo_id}",
|
||||
"name": name,
|
||||
"owner": owner,
|
||||
"description": repo.get("description") or "",
|
||||
"stars": repo.get("stars", 0) or 0,
|
||||
"forks": 0,
|
||||
"language": repo.get("framework") or "",
|
||||
"license": "",
|
||||
"topics": [],
|
||||
"created_at": "",
|
||||
"updated_at": "",
|
||||
"pushed_at": "",
|
||||
"open_issues": 0,
|
||||
"default_branch": "main",
|
||||
"archived": False,
|
||||
"source": "paperswithcode",
|
||||
"paper_ids": paper_ids,
|
||||
"paper_titles": paper_titles,
|
||||
"is_official": repo.get("is_official", False),
|
||||
# Default fields for repo_db compatibility
|
||||
"languages_pct": {},
|
||||
"readme_excerpt": "",
|
||||
"relevance_score": 0.0,
|
||||
"quality_score": 0.0,
|
||||
"activity_score": 0.0,
|
||||
"composite_score": 0.0,
|
||||
"tags": [],
|
||||
"analyzed": False,
|
||||
"local_path": None,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core lookup functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def lookup_paper_by_arxiv(arxiv_id: str) -> dict | None:
|
||||
"""Look up a paper on PWC by arXiv ID. Returns first matching paper or None."""
|
||||
clean_id = arxiv_id.strip()
|
||||
# Remove version suffix if present (e.g. 2301.12345v2 -> 2301.12345)
|
||||
if "v" in clean_id and clean_id[-1].isdigit():
|
||||
base, _, ver = clean_id.rpartition("v")
|
||||
if ver.isdigit():
|
||||
clean_id = base
|
||||
|
||||
url = f"{PWC_API}/papers/?arxiv_id={urllib.parse.quote(clean_id, safe='')}"
|
||||
data = pwc_request(url)
|
||||
if not data or not isinstance(data, dict):
|
||||
return None
|
||||
|
||||
results = data.get("results", [])
|
||||
return results[0] if results else None
|
||||
|
||||
|
||||
def fetch_paper_repos(paper_id: str) -> list[dict]:
|
||||
"""Fetch all repositories linked to a paper by its PWC paper ID."""
|
||||
url = f"{PWC_API}/papers/{urllib.parse.quote(paper_id, safe='')}/repositories/"
|
||||
return paginate_results(url)
|
||||
|
||||
|
||||
def search_papers_by_query(query: str, max_papers: int = 50) -> list[dict]:
|
||||
"""Search PWC papers by keyword query."""
|
||||
papers: list[dict] = []
|
||||
page = 1
|
||||
|
||||
while len(papers) < max_papers:
|
||||
params = urllib.parse.urlencode({"q": query, "page": page})
|
||||
url = f"{PWC_API}/papers/?{params}"
|
||||
data = pwc_request(url)
|
||||
if not data or not isinstance(data, dict):
|
||||
break
|
||||
|
||||
results = data.get("results", [])
|
||||
if not results:
|
||||
break
|
||||
|
||||
papers.extend(results)
|
||||
|
||||
if data.get("next"):
|
||||
page += 1
|
||||
time.sleep(REQUEST_DELAY)
|
||||
else:
|
||||
break
|
||||
|
||||
return papers[:max_papers]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Processing functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def process_arxiv_id(arxiv_id: str) -> tuple[list[dict], str]:
|
||||
"""Process a single arXiv ID: look up paper, fetch repos.
|
||||
|
||||
Returns (repo_records, paper_title).
|
||||
"""
|
||||
clean_id = arxiv_id.strip()
|
||||
if not clean_id:
|
||||
return [], ""
|
||||
|
||||
print(f" arXiv:{clean_id} ...", file=sys.stderr)
|
||||
paper = lookup_paper_by_arxiv(clean_id)
|
||||
if not paper:
|
||||
print(f" not found on PWC", file=sys.stderr)
|
||||
return [], ""
|
||||
|
||||
paper_id = paper.get("id", "")
|
||||
paper_title = paper.get("title", "")
|
||||
repo_count = paper.get("repository_count", 0)
|
||||
|
||||
print(f" {paper_title[:80]}", file=sys.stderr)
|
||||
print(f" repos: {repo_count}", file=sys.stderr)
|
||||
|
||||
if not paper_id or repo_count == 0:
|
||||
return [], paper_title
|
||||
|
||||
time.sleep(REQUEST_DELAY)
|
||||
raw_repos = fetch_paper_repos(paper_id)
|
||||
|
||||
records = []
|
||||
for repo in raw_repos:
|
||||
record = map_to_repo_record(repo, paper_title=paper_title, arxiv_id=clean_id)
|
||||
if record:
|
||||
records.append(record)
|
||||
|
||||
print(f" mapped {len(records)} GitHub repos", file=sys.stderr)
|
||||
return records, paper_title
|
||||
|
||||
|
||||
def process_query(query: str) -> list[dict]:
|
||||
"""Search PWC by keyword and fetch repos for matching papers."""
|
||||
print(f"[info] searching PWC for: {query}", file=sys.stderr)
|
||||
papers = search_papers_by_query(query)
|
||||
print(f"[info] found {len(papers)} papers", file=sys.stderr)
|
||||
|
||||
all_records: list[dict] = []
|
||||
papers_with_repos = 0
|
||||
|
||||
for i, paper in enumerate(papers):
|
||||
paper_id = paper.get("id", "")
|
||||
paper_title = paper.get("title", "")
|
||||
arxiv_id = paper.get("arxiv_id", "") or ""
|
||||
repo_count = paper.get("repository_count", 0)
|
||||
|
||||
if not paper_id or repo_count == 0:
|
||||
continue
|
||||
|
||||
print(
|
||||
f" [{i + 1}/{len(papers)}] {paper_title[:60]} ({repo_count} repos)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
time.sleep(REQUEST_DELAY)
|
||||
raw_repos = fetch_paper_repos(paper_id)
|
||||
|
||||
for repo in raw_repos:
|
||||
record = map_to_repo_record(
|
||||
repo, paper_title=paper_title, arxiv_id=arxiv_id
|
||||
)
|
||||
if record:
|
||||
all_records.append(record)
|
||||
|
||||
papers_with_repos += 1
|
||||
time.sleep(REQUEST_DELAY)
|
||||
|
||||
print(f"[info] {papers_with_repos} papers had repos", file=sys.stderr)
|
||||
return all_records
|
||||
|
||||
|
||||
def deduplicate_repos(records: list[dict]) -> list[dict]:
|
||||
"""Deduplicate repos by repo_id, merging paper_ids and paper_titles."""
|
||||
seen: dict[str, dict] = {}
|
||||
for record in records:
|
||||
rid = record["repo_id"]
|
||||
if rid in seen:
|
||||
existing = seen[rid]
|
||||
# Merge paper references
|
||||
for pid in record.get("paper_ids", []):
|
||||
if pid and pid not in existing["paper_ids"]:
|
||||
existing["paper_ids"].append(pid)
|
||||
for pt in record.get("paper_titles", []):
|
||||
if pt and pt not in existing["paper_titles"]:
|
||||
existing["paper_titles"].append(pt)
|
||||
# Keep higher star count
|
||||
if record.get("stars", 0) > existing.get("stars", 0):
|
||||
existing["stars"] = record["stars"]
|
||||
# Prefer official repos
|
||||
if record.get("is_official", False):
|
||||
existing["is_official"] = True
|
||||
else:
|
||||
seen[rid] = record
|
||||
return list(seen.values())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Search Papers With Code for paper-to-repo mappings."
|
||||
)
|
||||
parser.add_argument("--arxiv-id", default=None,
|
||||
help="Single arXiv ID to look up")
|
||||
parser.add_argument("--arxiv-ids", nargs="+", default=None,
|
||||
help="Multiple arXiv IDs")
|
||||
parser.add_argument("--arxiv-ids-file", default=None,
|
||||
help="File with one arXiv ID per line")
|
||||
parser.add_argument("--query", default=None,
|
||||
help="Search papers by keyword")
|
||||
parser.add_argument("--output", required=True,
|
||||
help="Output JSONL file path")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Validate: at least one input source required
|
||||
if not any([args.arxiv_id, args.arxiv_ids, args.arxiv_ids_file, args.query]):
|
||||
parser.error(
|
||||
"At least one of --arxiv-id, --arxiv-ids, --arxiv-ids-file, "
|
||||
"or --query is required"
|
||||
)
|
||||
|
||||
# Collect all arXiv IDs from all sources
|
||||
all_arxiv_ids: list[str] = []
|
||||
if args.arxiv_id:
|
||||
all_arxiv_ids.append(args.arxiv_id)
|
||||
if args.arxiv_ids:
|
||||
all_arxiv_ids.extend(args.arxiv_ids)
|
||||
if args.arxiv_ids_file:
|
||||
ids_file = Path(args.arxiv_ids_file)
|
||||
if not ids_file.exists():
|
||||
print(f"[error] arXiv IDs file not found: {ids_file}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
try:
|
||||
for line in ids_file.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if line and not line.startswith("#"):
|
||||
all_arxiv_ids.append(line)
|
||||
except OSError as e:
|
||||
print(f"[error] cannot read {ids_file}: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
all_records: list[dict] = []
|
||||
papers_found = 0
|
||||
|
||||
# Process arXiv IDs
|
||||
if all_arxiv_ids:
|
||||
print(
|
||||
f"[info] looking up {len(all_arxiv_ids)} arXiv IDs on Papers With Code ...",
|
||||
file=sys.stderr,
|
||||
)
|
||||
for arxiv_id in all_arxiv_ids:
|
||||
records, title = process_arxiv_id(arxiv_id)
|
||||
if title:
|
||||
papers_found += 1
|
||||
all_records.extend(records)
|
||||
time.sleep(REQUEST_DELAY)
|
||||
|
||||
# Process keyword query
|
||||
if args.query:
|
||||
query_records = process_query(args.query)
|
||||
all_records.extend(query_records)
|
||||
|
||||
# Deduplicate
|
||||
deduped = deduplicate_repos(all_records)
|
||||
|
||||
# Write output
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
for record in deduped:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
|
||||
# Summary
|
||||
total_paper_links = sum(len(r.get("paper_ids", [])) for r in deduped)
|
||||
official = sum(1 for r in deduped if r.get("is_official", False))
|
||||
|
||||
print(f"\n[info] results summary:", file=sys.stderr)
|
||||
print(f" repos found: {len(deduped)}", file=sys.stderr)
|
||||
print(f" linked to papers: {total_paper_links}", file=sys.stderr)
|
||||
if official:
|
||||
print(f" official implementations: {official}", file=sys.stderr)
|
||||
print(f"Found {len(deduped)} repos linked to {papers_found} papers", file=sys.stderr)
|
||||
print(f"[info] written to {output_path}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user