mirror of
https://github.com/rookie-ricardo/erduo-skills.git
synced 2026-09-19 06:00:10 +08:00
feat: Add daily news report skill with source configuration, agent definition, and comprehensive English documentation.
This commit is contained in:
rookie-ricardo
@@ -0,0 +1,36 @@
|
||||
---
|
||||
name: worker
|
||||
description: 无状态执行单元，完成单一任务后返回结构化结果
|
||||
tools: WebFetch, WebSearch, Read, Grep, Glob, mcp__chrome-devtools__*
|
||||
---
|
||||
|
||||
# Worker
|
||||
|
||||
无状态执行单元。完成任务，返回结果。
|
||||
|
||||
## 输入
|
||||
|
||||
```yaml
|
||||
task: fetch_and_extract | search_and_filter
|
||||
input: { urls: [...] } | { query: "..." }
|
||||
output_schema: { ... }
|
||||
constraints: { ... }
|
||||
```
|
||||
|
||||
## 输出
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success | partial | failed",
|
||||
"data": [...],
|
||||
"errors": [...],
|
||||
"metadata": { "processed": N, "failed": N }
|
||||
}
|
||||
```
|
||||
|
||||
## 规则
|
||||
|
||||
1. 只做被分配的任务
|
||||
2. 严格按 output_schema 格式输出
|
||||
3. 单个失败不中断整体
|
||||
4. 直接返回 JSON，不解释
|
||||
@@ -1,5 +1,4 @@
|
||||
.DS_Store
|
||||
.claude/
|
||||
node_modules/
|
||||
dist/
|
||||
.env
|
||||
|
||||
@@ -1,41 +1,105 @@
|
||||
# Erduo Skill 🚀
|
||||
# Erduo Skills / 耳朵技能库
|
||||
|
||||
> Empowering AI agents with advanced capabilities and structured knowledge.
|
||||
[English](README_EN.md)
|
||||
|
||||
## 📖 Overview
|
||||
> 为 AI Agent 赋能，提供结构化能力与智能工作流。
|
||||
|
||||
**Erduo Skill** is a specialized repository designed to house and manage intelligent skills and reporting mechanisms for AI agents. It serves as a knowledge base and execution framework, enabling agents to perform complex tasks such as news reporting and story planning.
|
||||
## 📖 简介
|
||||
|
||||
## ✨ Features
|
||||
|
||||
- **🤖 Agent Skills**: Modular capabilities designed for AI integration.
|
||||
- **📰 News Reports**: Automated or structured news generation context.
|
||||
- **🧩 Extensible Architecture**: Easy to add new skills and agent personas.
|
||||
|
||||
## 📂 Project Structure
|
||||
|
||||
```bash
|
||||
├── .claude/ # Agent configurations and definitions
|
||||
│ ├── agents/ # Agent personas (e.g., Story Planner)
|
||||
│ └── skills/ # Executable skills
|
||||
├── NewsReport/ # Generated reports and archives
|
||||
└── README.md # Project documentation
|
||||
```
|
||||
|
||||
## 🚀 Getting Started
|
||||
|
||||
1. **Clone the repository**
|
||||
```bash
|
||||
git clone https://github.com/your-username/Erduo-Skill.git
|
||||
```
|
||||
2. **Explore Agents**
|
||||
Check `.claude/agents` to see available personas.
|
||||
3. **Run Skills**
|
||||
Utilize the defined skills within your agent environment.
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please feel free to submit a Pull Request.
|
||||
**Erduo Skills** 是一个专门用于管理 AI Agent 智能技能的仓库。它作为一个知识库和执行框架，使 Agent 能够执行自动新闻报道、数据分析等复杂任务。
|
||||
|
||||
---
|
||||
|
||||
## ✨ 精选技能：每日日报
|
||||
|
||||
**每日日报** 是一个高级技能，旨在自动从多个来源抓取、筛选并总结高质量的技术新闻。
|
||||
|
||||
### 🏗 核心架构
|
||||
|
||||
该技能采用 **Master-Worker** 架构，包含智能调度器和专用子 Agent。
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
User((User)) -->|Start| Master[Master Agent<br>调度/监控/决策]
|
||||
|
||||
subgraph Execution Layer [SubAgent 执行层]
|
||||
WorkerA[Worker A<br>WebFetch]
|
||||
WorkerB[Worker B<br>WebFetch]
|
||||
Browser[Browser Worker<br>Headless Chrome]
|
||||
end
|
||||
|
||||
Master -->|Dispatch Tier 1| WorkerA
|
||||
Master -->|Dispatch Tier 2| WorkerB
|
||||
Master -->|Dispatch JS Pages| Browser
|
||||
|
||||
WorkerA -->|Result| Master
|
||||
WorkerB -->|Result| Master
|
||||
Browser -->|Result| Master
|
||||
|
||||
Master -->|Filter & Dedup| Report[Generate Report<br>生成日报]
|
||||
Master -->|Update| Cache[Smart Cache<br>智能缓存]
|
||||
```
|
||||
|
||||
### 🚀 核心特性
|
||||
|
||||
- **多源抓取**:
|
||||
- 聚合 HackerNews, HuggingFace Papers 等优质源。
|
||||
|
||||
- **智能筛选**:
|
||||
- 筛选高质量技术内容，排除营销软文。
|
||||
|
||||
- **动态调度**:
|
||||
- 采用“早停机制”：一旦抓取到足够的高质量条目（如 20 条），即停止抓取以节省资源。
|
||||
|
||||
- **无头浏览器支持**:
|
||||
- 使用 MCP Chrome DevTools 处理复杂的 JS 渲染页面（如 ProductHunt）。
|
||||
|
||||
### 📄 输出示例
|
||||
|
||||
日报以结构化 Markdown 格式生成，存储在 `NewsReport/` 目录下。
|
||||
|
||||
> **Daily News Report (2024-03-21)**
|
||||
>
|
||||
> **1. 文章标题**
|
||||
> - **摘要**: 文章内容的简要总结...
|
||||
> - **要点**:
|
||||
> 1. 要点一
|
||||
> 2. 要点二
|
||||
> - **来源**: [链接](...)
|
||||
> - **评分**: ⭐⭐⭐⭐⭐
|
||||
|
||||
---
|
||||
|
||||
## 📂 项目结构
|
||||
|
||||
```bash
|
||||
├── .claude/
|
||||
│ └── agents/ # Agent 定义 (Personas & Prompts)
|
||||
├── skills/ # 技能实现 (例如 daily-news-report)
|
||||
│ └── daily-news-report/ # 每日日报技能
|
||||
├── NewsReport/ # 生成的日报存档
|
||||
├── README.md # 项目文档 (默认为中文)
|
||||
└── README_EN.md # 英文项目文档
|
||||
```
|
||||
|
||||
## 🛠 使用方法
|
||||
|
||||
1. **克隆仓库**
|
||||
```bash
|
||||
git clone https://github.com/Start-to-DJ/erduo-skills.git
|
||||
cd erduo-skills
|
||||
```
|
||||
|
||||
2. **使用 Agent 运行**
|
||||
将此仓库加载到您的 Agent 环境中（例如 Claude Desktop 或支持 MCP 的 Zed）。Agent 将自动识别 `daily-news-report` 技能。
|
||||
|
||||
*提示词示例:*
|
||||
> “生成今天的日报。”
|
||||
|
||||
## 🤝 贡献指南
|
||||
|
||||
欢迎贡献！如果您有新的技能想法，请参考 `.claude/skills` 目录下的示例。
|
||||
|
||||
---
|
||||
|
||||
*Created with ❤️ by Erduo Team*
|
||||
|
||||
+105
@@ -0,0 +1,105 @@
|
||||
# Erduo Skills
|
||||
|
||||
[中文](README.md)
|
||||
|
||||
> Empowering AI Agents with structured capabilities and intelligent workflows.
|
||||
|
||||
## 📖 Overview
|
||||
|
||||
**Erduo Skills** is a specialized repository designed to house and manage intelligent skills for AI agents. It serves as a knowledge base and execution framework, enabling agents to perform complex tasks such as autonomous news reporting, data analysis, and more.
|
||||
|
||||
---
|
||||
|
||||
## ✨ Featured Skill: Daily News Report
|
||||
|
||||
The **Daily News Report** is a sophisticated skill designed to autonomously fetch, filter, and summarize high-quality technical news from multiple sources.
|
||||
|
||||
### 🏗 Architecture
|
||||
|
||||
This skill utilizes a **Master-Worker** architecture with a smart orchestrator and specialized sub-agents.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
User((User)) -->|Start| Master[Master Agent<br>Orchestrate/Monitor/Decide]
|
||||
|
||||
subgraph Execution Layer [SubAgent Layer]
|
||||
WorkerA[Worker A<br>WebFetch]
|
||||
WorkerB[Worker B<br>WebFetch]
|
||||
Browser[Browser Worker<br>Headless Chrome]
|
||||
end
|
||||
|
||||
Master -->|Dispatch Tier 1| WorkerA
|
||||
Master -->|Dispatch Tier 2| WorkerB
|
||||
Master -->|Dispatch JS Pages| Browser
|
||||
|
||||
WorkerA -->|Result| Master
|
||||
WorkerB -->|Result| Master
|
||||
Browser -->|Result| Master
|
||||
|
||||
Master -->|Filter & Dedup| Report[Generate Report]
|
||||
Master -->|Update| Cache[Smart Cache]
|
||||
```
|
||||
|
||||
### 🚀 Key Features
|
||||
|
||||
- **Multi-Source Fetching**:
|
||||
- Aggregates content from HackerNews, HuggingFace Papers, etc.
|
||||
|
||||
- **Smart Filtering**:
|
||||
- Filters for high-quality technical content, excluding marketing fluff.
|
||||
|
||||
- **Dynamic Scheduling**:
|
||||
- Uses an "Early Stopping" mechanism: if enough high-quality items are found (e.g., 20 items), it stops fetching to save resources.
|
||||
|
||||
- **Headless Browser Support**:
|
||||
- Handles complex, JS-rendered pages (e.g., ProductHunt) using MCP Chrome DevTools.
|
||||
|
||||
### 📄 Output Example
|
||||
|
||||
Reports are generated in structured Markdown format, stored in the `NewsReport/` directory.
|
||||
|
||||
> **Daily News Report (2024-03-21)**
|
||||
>
|
||||
> **1. Title of the Article**
|
||||
> - **Summary**: A concise summary of the article...
|
||||
> - **Key Points**:
|
||||
> 1. Point one
|
||||
> 2. Point two
|
||||
> - **Source**: [Link](...)
|
||||
> - **Rating**: ⭐⭐⭐⭐⭐
|
||||
|
||||
---
|
||||
|
||||
## 📂 Project Structure
|
||||
|
||||
```bash
|
||||
├── .claude/
|
||||
│ └── agents/ # Agent personas & prompts
|
||||
├── skills/ # Executable skill definitions
|
||||
│ └── daily-news-report/ # The Daily News Report skill
|
||||
├── NewsReport/ # Generated daily reports
|
||||
├── README.md # Project documentation (Chinese by default)
|
||||
└── README_EN.md # Project documentation (English)
|
||||
```
|
||||
|
||||
## 🛠 Usage
|
||||
|
||||
1. **Clone the repository**
|
||||
```bash
|
||||
git clone https://github.com/Start-to-DJ/erduo-skills.git
|
||||
cd erduo-skills
|
||||
```
|
||||
|
||||
2. **Run with Agent**
|
||||
Load this repository into your Agent environment (e.g., Claude Desktop, Zed with MCP). The Agent will automatically recognize the `daily-news-report` skill.
|
||||
|
||||
*Prompt Example:*
|
||||
> "Generate today's news report."
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! If you have a new skill idea, please check the `.claude/skills` directory for examples.
|
||||
|
||||
---
|
||||
|
||||
*Created with ❤️ by Erduo Team*
|
||||
@@ -0,0 +1,337 @@
|
||||
---
|
||||
name: daily-news-report
|
||||
description: 基于预设 URL 列表抓取内容，筛选高质量技术信息并生成每日 Markdown 报告。
|
||||
argument-hint: [可选: 日期]
|
||||
disable-model-invocation: false
|
||||
user-invocable: true
|
||||
allowed-tools: Task, WebFetch, Read, Write, Bash(mkdir*), Bash(date*), Bash(ls*), mcp__chrome-devtools__*
|
||||
---
|
||||
|
||||
# Daily News Report v3.0
|
||||
|
||||
> **架构升级**：主 Agent 调度 + SubAgent 执行 + 浏览器抓取 + 智能缓存
|
||||
|
||||
## 核心架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 主 Agent (Orchestrator) │
|
||||
│ 职责：调度、监控、评估、决策、汇总 │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ 1. 初始化 │ → │ 2. 调度 │ → │ 3. 监控 │ → │ 4. 评估 │ │
|
||||
│ │ 读取配置 │ │ 分发任务 │ │ 收集结果 │ │ 筛选排序 │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │ │ │ │ │
|
||||
│ ▼ ▼ ▼ ▼ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ 5. 决策 │ ← │ 够20条？ │ │ 6. 生成 │ → │ 7. 更新 │ │
|
||||
│ │ 继续/停止 │ │ Y/N │ │ 日报文件 │ │ 缓存统计 │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
↓ 调度 ↑ 返回结果
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ SubAgent 执行层 │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Worker A │ │ Worker B │ │ Browser │ │
|
||||
│ │ (WebFetch) │ │ (WebFetch) │ │ (Headless) │ │
|
||||
│ │ Tier1 Batch │ │ Tier2 Batch │ │ JS渲染页面 │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
│ ↓ ↓ ↓ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 结构化结果返回 │ │
|
||||
│ │ { status, data: [...], errors: [...], metadata: {...} } │ │
|
||||
│ └─────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 配置文件
|
||||
|
||||
本 Skill 使用以下配置文件：
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| `sources.json` | 信息源配置、优先级、抓取方法 |
|
||||
| `cache.json` | 缓存数据、历史统计、去重指纹 |
|
||||
|
||||
## 执行流程详解
|
||||
|
||||
### Phase 1: 初始化
|
||||
|
||||
```yaml
|
||||
步骤:
|
||||
1. 确定日期（用户参数或当前日期）
|
||||
2. 读取 sources.json 获取源配置
|
||||
3. 读取 cache.json 获取历史数据
|
||||
4. 创建输出目录 NewsReport/
|
||||
5. 检查今日是否已有部分报告（追加模式）
|
||||
```
|
||||
|
||||
### Phase 2: 调度 SubAgent
|
||||
|
||||
**策略**：并行调度，分批执行，早停机制
|
||||
|
||||
```yaml
|
||||
第1波 (并行):
|
||||
- Worker A: Tier1 Batch A (HN, HuggingFace Papers)
|
||||
- Worker B: Tier1 Batch B (OneUsefulThing, Paul Graham)
|
||||
|
||||
等待结果 → 评估数量
|
||||
|
||||
如果 < 15 条高质量:
|
||||
第2波 (并行):
|
||||
- Worker C: Tier2 Batch A (James Clear, FS Blog)
|
||||
- Worker D: Tier2 Batch B (HackerNoon, Scott Young)
|
||||
|
||||
如果仍 < 20 条:
|
||||
第3波 (浏览器):
|
||||
- Browser Worker: ProductHunt, Latent Space (需要JS渲染)
|
||||
```
|
||||
|
||||
### Phase 3: SubAgent 任务格式
|
||||
|
||||
每个 SubAgent 接收的任务格式：
|
||||
|
||||
```yaml
|
||||
task: fetch_and_extract
|
||||
sources:
|
||||
- id: hn
|
||||
url: https://news.ycombinator.com
|
||||
extract: top_10
|
||||
- id: hf_papers
|
||||
url: https://huggingface.co/papers
|
||||
extract: top_voted
|
||||
|
||||
output_schema:
|
||||
items:
|
||||
- source_id: string # 来源标识
|
||||
title: string # 标题
|
||||
summary: string # 2-4句摘要
|
||||
key_points: string[] # 最多3个要点
|
||||
url: string # 原文链接
|
||||
keywords: string[] # 关键词
|
||||
quality_score: 1-5 # 质量评分
|
||||
|
||||
constraints:
|
||||
filter: "前沿技术/高深技术/提效技术/实用资讯"
|
||||
exclude: "泛科普/营销软文/过度学术化/招聘帖"
|
||||
max_items_per_source: 10
|
||||
skip_on_error: true
|
||||
|
||||
return_format: JSON
|
||||
```
|
||||
|
||||
### Phase 4: 主 Agent 监控与反馈
|
||||
|
||||
主 Agent 职责：
|
||||
|
||||
```yaml
|
||||
监控:
|
||||
- 检查 SubAgent 返回状态 (success/partial/failed)
|
||||
- 统计收集到的条目数量
|
||||
- 记录每个源的成功率
|
||||
|
||||
反馈循环:
|
||||
- 如果某 SubAgent 失败，决定是否重试或跳过
|
||||
- 如果某源持续失败，标记为禁用
|
||||
- 动态调整后续批次的源选择
|
||||
|
||||
决策:
|
||||
- 条目数 >= 25 且高质量 >= 20 → 停止抓取
|
||||
- 条目数 < 15 → 继续下一批
|
||||
- 所有批次完成但 < 20 → 用现有内容生成（宁缺毋滥）
|
||||
```
|
||||
|
||||
### Phase 5: 评估与筛选
|
||||
|
||||
```yaml
|
||||
去重:
|
||||
- 基于 URL 完全匹配
|
||||
- 基于标题相似度 (>80% 视为重复)
|
||||
- 检查 cache.json 避免与历史重复
|
||||
|
||||
评分校准:
|
||||
- 统一各 SubAgent 的评分标准
|
||||
- 根据来源可信度调整权重
|
||||
- 手动标注的高质量源加分
|
||||
|
||||
排序:
|
||||
- 按 quality_score 降序
|
||||
- 同分按来源优先级排序
|
||||
- 截取 Top 20
|
||||
```
|
||||
|
||||
### Phase 6: 浏览器抓取 (MCP Chrome DevTools)
|
||||
|
||||
对于需要 JS 渲染的页面，使用无头浏览器：
|
||||
|
||||
```yaml
|
||||
流程:
|
||||
1. 调用 mcp__chrome-devtools__new_page 打开页面
|
||||
2. 调用 mcp__chrome-devtools__wait_for 等待内容加载
|
||||
3. 调用 mcp__chrome-devtools__take_snapshot 获取页面结构
|
||||
4. 解析 snapshot 提取所需内容
|
||||
5. 调用 mcp__chrome-devtools__close_page 关闭页面
|
||||
|
||||
适用场景:
|
||||
- ProductHunt (403 on WebFetch)
|
||||
- Latent Space (Substack JS 渲染)
|
||||
- 其他 SPA 应用
|
||||
```
|
||||
|
||||
### Phase 7: 生成日报
|
||||
|
||||
```yaml
|
||||
输出:
|
||||
- 目录: NewsReport/
|
||||
- 文件名: YYYY-MM-DD-news-report.md
|
||||
- 格式: 标准 Markdown
|
||||
|
||||
内容结构:
|
||||
- 标题 + 日期
|
||||
- 统计摘要（源数量、收录数量）
|
||||
- 20条高质量内容（按模板）
|
||||
- 生成信息（版本、时间戳）
|
||||
```
|
||||
|
||||
### Phase 8: 更新缓存
|
||||
|
||||
```yaml
|
||||
更新 cache.json:
|
||||
- last_run: 记录本次运行信息
|
||||
- source_stats: 更新各源统计数据
|
||||
- url_cache: 添加已处理的 URL
|
||||
- content_hashes: 添加内容指纹
|
||||
- article_history: 记录收录文章
|
||||
```
|
||||
|
||||
## SubAgent 调用示例
|
||||
|
||||
### 使用 general-purpose Agent
|
||||
|
||||
由于自定义 agent 需要 session 重启才能发现，可以使用 general-purpose 并注入 worker prompt：
|
||||
|
||||
```
|
||||
Task 调用:
|
||||
subagent_type: general-purpose
|
||||
model: haiku
|
||||
prompt: |
|
||||
你是一个无状态的执行单元。只做被分配的任务，返回结构化 JSON。
|
||||
|
||||
任务：抓取以下 URL 并提取内容
|
||||
|
||||
URLs:
|
||||
- https://news.ycombinator.com (提取 Top 10)
|
||||
- https://huggingface.co/papers (提取高投票论文)
|
||||
|
||||
输出格式：
|
||||
{
|
||||
"status": "success" | "partial" | "failed",
|
||||
"data": [
|
||||
{
|
||||
"source_id": "hn",
|
||||
"title": "...",
|
||||
"summary": "...",
|
||||
"key_points": ["...", "...", "..."],
|
||||
"url": "...",
|
||||
"keywords": ["...", "..."],
|
||||
"quality_score": 4
|
||||
}
|
||||
],
|
||||
"errors": [],
|
||||
"metadata": { "processed": 2, "failed": 0 }
|
||||
}
|
||||
|
||||
筛选标准：
|
||||
- 保留：前沿技术/高深技术/提效技术/实用资讯
|
||||
- 排除：泛科普/营销软文/过度学术化/招聘帖
|
||||
|
||||
直接返回 JSON，不要解释。
|
||||
```
|
||||
|
||||
### 使用 worker Agent（需重启 session）
|
||||
|
||||
```
|
||||
Task 调用:
|
||||
subagent_type: worker
|
||||
prompt: |
|
||||
task: fetch_and_extract
|
||||
input:
|
||||
urls:
|
||||
- https://news.ycombinator.com
|
||||
- https://huggingface.co/papers
|
||||
output_schema:
|
||||
- source_id: string
|
||||
- title: string
|
||||
- summary: string
|
||||
- key_points: string[]
|
||||
- url: string
|
||||
- keywords: string[]
|
||||
- quality_score: 1-5
|
||||
constraints:
|
||||
filter: 前沿技术/高深技术/提效技术/实用资讯
|
||||
exclude: 泛科普/营销软文/过度学术化
|
||||
```
|
||||
|
||||
## 输出模板
|
||||
|
||||
```markdown
|
||||
# Daily News Report（YYYY-MM-DD）
|
||||
|
||||
> 本日筛选自 N 个信息源，共收录 20 条高质量内容
|
||||
> 生成耗时: X 分钟 | 版本: v3.0
|
||||
|
||||
---
|
||||
|
||||
## 1. 标题
|
||||
|
||||
- **摘要**：2-4 行概述
|
||||
- **要点**：
|
||||
1. 要点一
|
||||
2. 要点二
|
||||
3. 要点三
|
||||
- **来源**：[链接](URL)
|
||||
- **关键词**：`keyword1` `keyword2` `keyword3`
|
||||
- **评分**：⭐⭐⭐⭐⭐ (5/5)
|
||||
|
||||
---
|
||||
|
||||
## 2. 标题
|
||||
...
|
||||
|
||||
---
|
||||
|
||||
*Generated by Daily News Report v3.0*
|
||||
*Sources: HN, HuggingFace, OneUsefulThing, ...*
|
||||
```
|
||||
|
||||
## 约束与原则
|
||||
|
||||
1. **宁缺毋滥**：低质量内容不进入日报
|
||||
2. **早停机制**：够 20 条高质量就停止抓取
|
||||
3. **并行优先**：同一批次的 SubAgent 并行执行
|
||||
4. **失败容错**：单个源失败不影响整体流程
|
||||
5. **缓存复用**：避免重复抓取相同内容
|
||||
6. **主 Agent 控制**：所有决策由主 Agent 做出
|
||||
|
||||
## 预期性能
|
||||
|
||||
| 场景 | 预期时间 | 说明 |
|
||||
|------|----------|------|
|
||||
| 最优情况 | ~2 分钟 | Tier1 足够，无需浏览器 |
|
||||
| 正常情况 | ~3-4 分钟 | 需要 Tier2 补充 |
|
||||
| 需要浏览器 | ~5-6 分钟 | 包含 JS 渲染页面 |
|
||||
|
||||
## 错误处理
|
||||
|
||||
| 错误类型 | 处理方式 |
|
||||
|----------|----------|
|
||||
| SubAgent 超时 | 记录错误，继续下一个 |
|
||||
| 源 403/404 | 标记禁用，更新 sources.json |
|
||||
| 内容提取失败 | 返回原始内容，主 Agent 决定 |
|
||||
| 浏览器崩溃 | 跳过该源，记录日志 |
|
||||
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"description": "Daily News Report 缓存文件，用于避免重复抓取和跟踪历史表现",
|
||||
|
||||
"last_run": {
|
||||
"date": "2026-01-21",
|
||||
"duration_seconds": 180,
|
||||
"items_collected": 20,
|
||||
"items_published": 20,
|
||||
"sources_used": ["hn", "hf_papers", "james_clear", "fs_blog", "scotthyoung"]
|
||||
},
|
||||
|
||||
"source_stats": {
|
||||
"_comment": "记录每个源的历史表现，用于动态调整优先级",
|
||||
"hn": {
|
||||
"total_fetches": 0,
|
||||
"success_count": 0,
|
||||
"avg_items_per_fetch": 0,
|
||||
"avg_quality_score": 0,
|
||||
"last_fetch": null,
|
||||
"last_success": null
|
||||
}
|
||||
},
|
||||
|
||||
"url_cache": {
|
||||
"_comment": "已处理的 URL 缓存，避免重复收录",
|
||||
"_ttl_hours": 168,
|
||||
"entries": {}
|
||||
},
|
||||
|
||||
"content_hashes": {
|
||||
"_comment": "内容指纹，用于去重",
|
||||
"_ttl_hours": 168,
|
||||
"entries": {}
|
||||
},
|
||||
|
||||
"article_history": {
|
||||
"_comment": "已收录文章的简要记录",
|
||||
"2026-01-21": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,183 @@
|
||||
{
|
||||
"version": "2.1",
|
||||
"last_updated": "2026-01-21",
|
||||
|
||||
"sources": {
|
||||
"tier1": {
|
||||
"description": "高命中率源，优先抓取",
|
||||
"batch_a": [
|
||||
{
|
||||
"id": "hn",
|
||||
"name": "Hacker News",
|
||||
"url": "https://news.ycombinator.com",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "top_10",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.5,
|
||||
"success_rate": 0.95
|
||||
},
|
||||
{
|
||||
"id": "hf_papers",
|
||||
"name": "HuggingFace Papers",
|
||||
"url": "https://huggingface.co/papers",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "top_voted",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.8,
|
||||
"success_rate": 0.98
|
||||
}
|
||||
],
|
||||
"batch_b": [
|
||||
{
|
||||
"id": "one_useful_thing",
|
||||
"name": "One Useful Thing",
|
||||
"url": "https://www.oneusefulthing.org",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "latest_3",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.7,
|
||||
"success_rate": 0.92
|
||||
},
|
||||
{
|
||||
"id": "paul_graham",
|
||||
"name": "Paul Graham Essays",
|
||||
"url": "https://paulgraham.com/articles.html",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "latest_5",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.6,
|
||||
"success_rate": 0.99
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
"tier2": {
|
||||
"description": "中等命中率，按需抓取",
|
||||
"batch_a": [
|
||||
{
|
||||
"id": "james_clear",
|
||||
"name": "James Clear 3-2-1",
|
||||
"url": "https://jamesclear.com/3-2-1",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "latest_issue",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.3,
|
||||
"success_rate": 0.90
|
||||
},
|
||||
{
|
||||
"id": "fs_blog",
|
||||
"name": "Farnam Street Brain Food",
|
||||
"url": "https://fs.blog/brain-food",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "latest_issue",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.4,
|
||||
"success_rate": 0.88
|
||||
}
|
||||
],
|
||||
"batch_b": [
|
||||
{
|
||||
"id": "hackernoon_pm",
|
||||
"name": "HackerNoon PM",
|
||||
"url": "https://hackernoon.com/c/product-management",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "latest_5",
|
||||
"enabled": true,
|
||||
"avg_quality": 3.8,
|
||||
"success_rate": 0.85
|
||||
},
|
||||
{
|
||||
"id": "scotthyoung",
|
||||
"name": "Scott Young Blog",
|
||||
"url": "https://scotthyoung.com/blog/articles",
|
||||
"fetch_method": "webfetch",
|
||||
"extract": "latest_3",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.0,
|
||||
"success_rate": 0.90
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
"tier3_browser": {
|
||||
"description": "需要浏览器渲染的源",
|
||||
"sources": [
|
||||
{
|
||||
"id": "producthunt",
|
||||
"name": "Product Hunt",
|
||||
"url": "https://www.producthunt.com",
|
||||
"fetch_method": "browser",
|
||||
"extract": "today_top_5",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.2,
|
||||
"success_rate": 0.75,
|
||||
"note": "需要无头浏览器，403 on WebFetch"
|
||||
},
|
||||
{
|
||||
"id": "latent_space",
|
||||
"name": "Latent Space",
|
||||
"url": "https://www.latent.space",
|
||||
"fetch_method": "browser",
|
||||
"extract": "latest_3",
|
||||
"enabled": true,
|
||||
"avg_quality": 4.6,
|
||||
"success_rate": 0.70,
|
||||
"note": "Substack 需要 JS 渲染"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
"disabled": {
|
||||
"description": "已禁用的源（失效或低质量）",
|
||||
"sources": [
|
||||
{
|
||||
"id": "tldr_ai",
|
||||
"name": "TLDR AI",
|
||||
"url": "https://tldr.tech/ai",
|
||||
"reason": "订阅页面，无文章列表",
|
||||
"disabled_date": "2026-01-21"
|
||||
},
|
||||
{
|
||||
"id": "bensbites",
|
||||
"name": "Ben's Bites",
|
||||
"url": "https://bensbites.com/archive",
|
||||
"reason": "需要登录/付费墙",
|
||||
"disabled_date": "2026-01-21"
|
||||
},
|
||||
{
|
||||
"id": "interconnects",
|
||||
"name": "Interconnects AI",
|
||||
"url": "https://interconnects.ai",
|
||||
"reason": "内容提取失败，Substack 结构问题",
|
||||
"disabled_date": "2026-01-21"
|
||||
},
|
||||
{
|
||||
"id": "beehiiv_rss",
|
||||
"name": "Beehiiv RSS feeds",
|
||||
"url": "https://rss.beehiiv.com",
|
||||
"reason": "RSS 抓取困难",
|
||||
"disabled_date": "2026-01-21"
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
|
||||
"fetch_config": {
|
||||
"webfetch": {
|
||||
"timeout_ms": 30000,
|
||||
"retry_count": 1,
|
||||
"cache_ttl_minutes": 60
|
||||
},
|
||||
"browser": {
|
||||
"timeout_ms": 45000,
|
||||
"wait_for_selector": "article, .post, .item",
|
||||
"screenshot_on_error": true
|
||||
}
|
||||
},
|
||||
|
||||
"quality_thresholds": {
|
||||
"min_score_to_include": 3,
|
||||
"target_items": 20,
|
||||
"early_stop_threshold": 25
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user