mirror of
https://github.com/shipshitdev/skills.git
synced 2026-09-19 06:04:15 +08:00
Update .gitignore, .markdownlint.json, and README.md for improved clarity and organization
- Added .agent/SESSIONS/ to .gitignore to exclude agent session files from version control. - Reverted the "default" setting in .markdownlint.json to true for consistent linting behavior. - Removed an empty line in .markdownlintignore for cleaner formatting. - Deleted obsolete .skill-validate.txt file to streamline project structure. - Enhanced README.md to clarify the purpose and contents of the project, including a new section on included capabilities. Total sessions today: 4
This commit is contained in:
vincentonchain
@@ -1,5 +1,7 @@
|
||||
node_modules/
|
||||
|
||||
# Agent sessions (ignored for open source projects)
|
||||
.agent/SESSIONS/
|
||||
|
||||
# macOS
|
||||
.DS_Store
|
||||
|
||||
+3
-4
@@ -1,5 +1,4 @@
|
||||
{
|
||||
"default": true,
|
||||
"MD001": false,
|
||||
"MD003": {
|
||||
"style": "atx"
|
||||
@@ -16,6 +15,6 @@
|
||||
"MD034": false,
|
||||
"MD036": false,
|
||||
"MD040": false,
|
||||
"MD041": false
|
||||
}
|
||||
|
||||
"MD041": false,
|
||||
"default": true
|
||||
}
|
||||
@@ -3,4 +3,3 @@ node_modules/
|
||||
.claude/
|
||||
.codex/
|
||||
.cursor/
|
||||
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
[0;34mValidating all skills...[0m
|
||||
|
||||
[0;34mValidating: accessibility[0m
|
||||
Claude version:
|
||||
Codex version:
|
||||
Drift check:
|
||||
[1;33m⚠[0m Significant content drift detected (774 line difference)
|
||||
Claude: 820 lines, Codex: 46 lines
|
||||
Consider syncing to ensure core content is identical
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
|
||||
|
||||
Centralized **global** skills and commands for Claude Code, OpenAI Codex, and Cursor.
|
||||
A comprehensive collection of agent capabilities—skills, commands, and workflows—for Claude Code, OpenAI Codex, and Cursor. Centralized and globally distributed via symlinks.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
@@ -33,6 +33,13 @@ library/
|
||||
| **Skills** | 43 | 43 | 48 | Base parity + Cursor extras |
|
||||
| **Commands** | 1 | 0 | 30 | Cursor-focused |
|
||||
|
||||
## What's Included
|
||||
|
||||
- **Skills**: Specialized agent capabilities for specific domains (e.g., `stripe-implementer`, `mongodb-migration-expert`)
|
||||
- **Commands**: Workflow commands for structured tasks (e.g., `code-review`, `deploy`, `mvp-plan`)
|
||||
- **Documentation**: Platform-specific adaptations and management guides
|
||||
- **Scripts**: Tooling for syncing, validation, and generation
|
||||
|
||||
## How It Works
|
||||
|
||||
This repo is symlinked from each agent's home directory:
|
||||
@@ -45,7 +52,7 @@ This repo is symlinked from each agent's home directory:
|
||||
~/.cursor/commands -> library/agents/.cursor/commands
|
||||
```
|
||||
|
||||
Edit skills/commands in `agents/`, changes are immediately available to all agents.
|
||||
Edit capabilities in `agents/`, changes are immediately available to all agents.
|
||||
|
||||
See `docs/SYMLINK-CONFIG.md` for full configuration details.
|
||||
|
||||
|
||||
@@ -0,0 +1,945 @@
|
||||
---
|
||||
name: ec2-backend-deployer
|
||||
description: Expert in deploying backends to EC2 instances using CI/CD pipelines, Docker containers, and GitHub Actions. This skill guides through the complete deployment workflow including Docker image building, container registry management, Tailscale integration, and automated deployment to EC2. Activates when users need to deploy backend services to EC2.
|
||||
---
|
||||
|
||||
# EC2 Backend Deployer
|
||||
|
||||
You are an expert in deploying backend applications to EC2 instances using CI/CD pipelines, Docker containers, and GitHub Actions. This skill provides comprehensive guidance for setting up automated deployments from GitHub to EC2, including Docker image building, container registry management, secure access via Tailscale, and service orchestration.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Setting up CI/CD for backend deployment to EC2
|
||||
- Configuring Docker-based deployments
|
||||
- Implementing automated deployment pipelines
|
||||
- Deploying NestJS, Next.js, or Express backends to EC2
|
||||
- Setting up container registries and image management
|
||||
- Configuring secure EC2 access for deployments
|
||||
- Implementing health checks and deployment verification
|
||||
- Setting up multi-service deployments with dependencies
|
||||
|
||||
## Project Context Discovery
|
||||
|
||||
**Before deploying, discover the project's context:**
|
||||
|
||||
1. **Identify Project Type:**
|
||||
- Scan for `package.json` to detect framework (NestJS, Next.js, Express)
|
||||
- Check for `nest-cli.json` (NestJS)
|
||||
- Check for `next.config.js` (Next.js)
|
||||
- Check for monorepo structure (workspaces)
|
||||
|
||||
2. **Check Existing Setup:**
|
||||
- Look for existing Dockerfiles
|
||||
- Check for docker-compose files
|
||||
- Review existing GitHub Actions workflows
|
||||
- Check for deployment scripts
|
||||
- Verify environment configuration files
|
||||
|
||||
3. **Identify Infrastructure:**
|
||||
- Check for EC2 instance details
|
||||
- Verify Tailscale setup (if using secure access)
|
||||
- Check for container registry configuration
|
||||
- Review security group and network setup
|
||||
|
||||
4. **Use Project-Specific Skills:**
|
||||
- Check for `[project]-ec2-backend-deployer` skill
|
||||
- Review project-specific deployment patterns
|
||||
- Follow project's infrastructure standards
|
||||
|
||||
## Docker Setup
|
||||
|
||||
### Multi-Stage Dockerfile Pattern
|
||||
|
||||
**Recommended structure for production deployments:**
|
||||
|
||||
```dockerfile
|
||||
# ==================================================
|
||||
# Stage 1: Base - Install dependencies
|
||||
# ==================================================
|
||||
FROM node:22.17.0 AS base
|
||||
|
||||
# Install bun (or use npm if preferred)
|
||||
RUN curl -fsSL https://bun.sh/install | bash && \
|
||||
cp /root/.bun/bin/bun /usr/local/bin/bun && \
|
||||
chmod +x /usr/local/bin/bun
|
||||
ENV PATH="/usr/local/bin:${PATH}"
|
||||
|
||||
# Set memory limits for builds
|
||||
ENV NODE_OPTIONS=--max-old-space-size=4096
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Copy package files
|
||||
COPY package.json package-lock.json* bun.lockb* ./
|
||||
COPY .npmrc ./
|
||||
|
||||
# For monorepos: create workspace structure
|
||||
RUN mkdir -p apps libs
|
||||
|
||||
# Install dependencies with secrets for private packages
|
||||
RUN --mount=type=secret,id=NPM_TOKEN \
|
||||
export NPM_TOKEN=$(cat /run/secrets/NPM_TOKEN 2>/dev/null || echo "") && \
|
||||
npm ci --frozen-lockfile || bun install --frozen-lockfile
|
||||
|
||||
# ==================================================
|
||||
# Stage 2: Builder - Build application
|
||||
# ==================================================
|
||||
FROM base AS builder
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Build application (with build secrets if needed)
|
||||
RUN --mount=type=secret,id=SENTRY_AUTH_TOKEN \
|
||||
export SENTRY_AUTH_TOKEN=$(cat /run/secrets/SENTRY_AUTH_TOKEN 2>/dev/null || echo "") && \
|
||||
npm run build:prod || bun run build:prod
|
||||
|
||||
# ==================================================
|
||||
# Stage 3: Production - Runtime image
|
||||
# ==================================================
|
||||
FROM node:22.17.0-slim AS production
|
||||
|
||||
# Install runtime dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
bash \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Copy built artifacts and production dependencies
|
||||
COPY --from=builder /usr/src/app/dist ./dist
|
||||
COPY --from=builder /usr/src/app/node_modules ./node_modules
|
||||
COPY --from=builder /usr/src/app/package.json ./package.json
|
||||
COPY --from=builder /usr/src/app/public ./public
|
||||
|
||||
# Create non-root user for security
|
||||
RUN groupadd -r appuser && useradd -r -g appuser -u 1001 appuser
|
||||
|
||||
# Set permissions
|
||||
RUN chown -R appuser:appuser /usr/src/app
|
||||
|
||||
# Switch to non-root user
|
||||
USER appuser
|
||||
|
||||
# Expose application port
|
||||
EXPOSE 3001
|
||||
|
||||
# Health check for container orchestration
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD curl -f http://localhost:3001/v1/health || exit 1
|
||||
|
||||
# Start application
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
**For NestJS specifically:**
|
||||
|
||||
```dockerfile
|
||||
# Use the pattern above but adjust:
|
||||
# - Build command: npm run build (creates dist/)
|
||||
# - Start command: node dist/main.js
|
||||
# - Health endpoint: /v1/health or /health
|
||||
```
|
||||
|
||||
**For Next.js API routes:**
|
||||
|
||||
```dockerfile
|
||||
# Adjust for Next.js:
|
||||
# - Build command: npm run build
|
||||
# - Start command: npm start
|
||||
# - Health endpoint: /api/health
|
||||
```
|
||||
|
||||
### Dockerfile Best Practices
|
||||
|
||||
- **Multi-stage builds**: Reduce final image size
|
||||
- **Non-root user**: Run containers as non-root for security
|
||||
- **Health checks**: Include HEALTHCHECK in Dockerfile
|
||||
- **Build secrets**: Use BuildKit secrets for sensitive data
|
||||
- **Layer caching**: Order COPY commands to maximize cache hits
|
||||
- **System dependencies**: Install only what's needed in production stage
|
||||
|
||||
## Container Registry Setup
|
||||
|
||||
### GitHub Container Registry (ghcr.io) - Recommended
|
||||
|
||||
**Advantages:**
|
||||
- Integrated with GitHub
|
||||
- Free for public repos, included with GitHub plans
|
||||
- Automatic authentication via GitHub tokens
|
||||
- Image versioning with tags
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Enable GitHub Container Registry:**
|
||||
- Go to repository Settings → Packages
|
||||
- Container registry is automatically enabled
|
||||
|
||||
2. **Image Naming Convention:**
|
||||
```
|
||||
ghcr.io/[owner]/[service-name]:[tag]
|
||||
```
|
||||
|
||||
3. **Image Tagging Strategy:**
|
||||
- `latest` - Most recent deployment
|
||||
- `production` - Production deployments
|
||||
- `[branch]-[sha]` - Branch and commit SHA
|
||||
- `[version]` - Semantic versioning
|
||||
|
||||
**Authentication in GitHub Actions:**
|
||||
|
||||
```yaml
|
||||
- name: Log in to Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
**Authentication on EC2:**
|
||||
|
||||
```bash
|
||||
# Login to registry
|
||||
echo "$GITHUB_TOKEN" | docker login ghcr.io -u USERNAME --password-stdin
|
||||
```
|
||||
|
||||
### AWS ECR (Alternative)
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Create ECR Repository:**
|
||||
```bash
|
||||
aws ecr create-repository --repository-name [service-name]
|
||||
```
|
||||
|
||||
2. **Get Login Token:**
|
||||
```bash
|
||||
aws ecr get-login-password --region [region] | \
|
||||
docker login --username AWS --password-stdin [account-id].dkr.ecr.[region].amazonaws.com
|
||||
```
|
||||
|
||||
3. **Push Image:**
|
||||
```bash
|
||||
docker tag [image]:[tag] [account-id].dkr.ecr.[region].amazonaws.com/[service-name]:[tag]
|
||||
docker push [account-id].dkr.ecr.[region].amazonaws.com/[service-name]:[tag]
|
||||
```
|
||||
|
||||
### Docker Hub (Alternative)
|
||||
|
||||
**Setup:**
|
||||
|
||||
```yaml
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
```
|
||||
|
||||
## CI/CD Pipeline (GitHub Actions)
|
||||
|
||||
### Main Deployment Workflow
|
||||
|
||||
**File:** `.github/workflows/deploy-production.yml`
|
||||
|
||||
```yaml
|
||||
name: Deploy Production
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'apps/**'
|
||||
- 'libs/**'
|
||||
- 'package.json'
|
||||
- 'Dockerfile*'
|
||||
- 'docker/**'
|
||||
workflow_dispatch:
|
||||
branches: [master]
|
||||
inputs:
|
||||
skip_tests:
|
||||
description: 'Skip pre-deployment tests'
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_PREFIX: ${{ github.repository_owner }}/[service-name]
|
||||
|
||||
jobs:
|
||||
# Branch safety check
|
||||
branch-check:
|
||||
name: Branch Safety Check
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Verify master branch
|
||||
run: |
|
||||
if [ "${{ github.ref_name }}" != "master" ]; then
|
||||
echo "❌ ERROR: Production deployment can only run from master branch"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pre-deployment checks
|
||||
pre-deployment-checks:
|
||||
name: Pre-Deployment Checks
|
||||
runs-on: ubuntu-latest
|
||||
needs: [branch-check]
|
||||
if: inputs.skip_tests != true
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Run linter
|
||||
run: npm run lint
|
||||
continue-on-error: true
|
||||
|
||||
- name: Run tests
|
||||
run: npm test
|
||||
continue-on-error: true
|
||||
|
||||
# Build and push image
|
||||
build-image:
|
||||
name: Build and Push Image
|
||||
needs: [branch-check, pre-deployment-checks]
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Log in to Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}
|
||||
tags: |
|
||||
type=sha,prefix={{branch}}-
|
||||
type=raw,value=latest
|
||||
type=raw,value=production
|
||||
|
||||
- name: Build and push image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:buildcache
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:buildcache,mode=max
|
||||
secrets: |
|
||||
NPM_TOKEN=${{ secrets.NPM_TOKEN }}
|
||||
SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }}
|
||||
platforms: linux/amd64
|
||||
|
||||
# Deploy to EC2
|
||||
deploy:
|
||||
name: Deploy to EC2
|
||||
needs: [build-image]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
|
||||
- name: Verify Tailscale connectivity
|
||||
run: |
|
||||
echo "⏳ Waiting for Tailscale to connect..."
|
||||
timeout 30 bash -c 'until tailscale status >/dev/null 2>&1; do sleep 1; done'
|
||||
echo "✅ Tailscale connected"
|
||||
|
||||
- name: Deploy to instance
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ vars.TAILSCALE_INSTANCE_IP || secrets.EC2_IP }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
port: 22
|
||||
script: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "🔐 Logging into container registry..."
|
||||
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
|
||||
|
||||
cd ~/[project-path] || mkdir -p ~/[project-path] && cd ~/[project-path]
|
||||
|
||||
# Verify Docker Compose v2
|
||||
if ! docker compose version &>/dev/null; then
|
||||
echo "❌ ERROR: Docker Compose v2 is not installed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Update docker-compose file
|
||||
cat > docker-compose.yml << 'EOF'
|
||||
version: '3.8'
|
||||
services:
|
||||
api:
|
||||
image: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:latest
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '3001:3001'
|
||||
env_file:
|
||||
- .env.production
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
EOF
|
||||
|
||||
# Pull latest images
|
||||
docker compose pull
|
||||
|
||||
# Deploy
|
||||
docker compose up -d --force-recreate
|
||||
|
||||
# Wait for health check
|
||||
echo "🏥 Waiting for service to be healthy..."
|
||||
sleep 10
|
||||
for i in {1..60}; do
|
||||
STATUS=$(docker inspect --format='{{.State.Health.Status}}' [container-name] 2>/dev/null || echo "none")
|
||||
if [ "$STATUS" = "healthy" ]; then
|
||||
echo "✅ Service is healthy"
|
||||
break
|
||||
fi
|
||||
if [ $i -eq 60 ]; then
|
||||
echo "❌ Service failed to become healthy"
|
||||
docker compose logs --tail=50
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting for service... ($i/60) [status: $STATUS]"
|
||||
sleep 3
|
||||
done
|
||||
|
||||
echo "✅ Deployment complete!"
|
||||
```
|
||||
|
||||
### Reusable Deployment Workflow
|
||||
|
||||
**File:** `.github/workflows/_deploy-service.yml`
|
||||
|
||||
For projects with multiple services, create a reusable workflow:
|
||||
|
||||
```yaml
|
||||
name: Deploy Service (Reusable)
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
service_name:
|
||||
required: true
|
||||
type: string
|
||||
instance_ip:
|
||||
required: true
|
||||
type: string
|
||||
docker_compose_file:
|
||||
required: true
|
||||
type: string
|
||||
health_check_services:
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
secrets:
|
||||
TAILSCALE_CLIENT_ID:
|
||||
required: true
|
||||
TAILSCALE_CLIENT_SECRET:
|
||||
required: true
|
||||
EC2_USER:
|
||||
required: true
|
||||
EC2_SSH_KEY:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
|
||||
- name: Deploy to instance
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ inputs.instance_ip }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
script: |
|
||||
# Deployment script (same as above)
|
||||
```
|
||||
|
||||
## EC2 Deployment Process
|
||||
|
||||
### Tailscale Integration (Recommended)
|
||||
|
||||
**Why Tailscale:**
|
||||
- Secure access without public IPs
|
||||
- No need to manage security groups for SSH
|
||||
- Easy connectivity from CI/CD runners
|
||||
- Works across networks
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Install Tailscale on EC2:**
|
||||
```bash
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
sudo tailscale up
|
||||
```
|
||||
|
||||
2. **Get Tailscale IP:**
|
||||
```bash
|
||||
tailscale ip -4
|
||||
```
|
||||
|
||||
3. **Configure GitHub Secrets:**
|
||||
- `TAILSCALE_CLIENT_ID` - OAuth client ID from Tailscale
|
||||
- `TAILSCALE_CLIENT_SECRET` - OAuth client secret
|
||||
- `TAILSCALE_INSTANCE_IP` - Tailscale IP of EC2 instance
|
||||
|
||||
4. **Use in GitHub Actions:**
|
||||
```yaml
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
```
|
||||
|
||||
### SSH Configuration (Alternative)
|
||||
|
||||
If not using Tailscale, use direct SSH:
|
||||
|
||||
**GitHub Secrets Required:**
|
||||
- `EC2_USER` - SSH username (e.g., `ubuntu`, `ec2-user`)
|
||||
- `EC2_SSH_KEY` - Private SSH key
|
||||
- `EC2_IP` - Public IP or hostname
|
||||
|
||||
**Security Group Configuration:**
|
||||
- Allow SSH (port 22) from GitHub Actions IPs
|
||||
- Or use a bastion host for additional security
|
||||
|
||||
### Docker Compose Deployment
|
||||
|
||||
**Requirements on EC2:**
|
||||
- Docker installed
|
||||
- Docker Compose v2 (not v1)
|
||||
- Sufficient disk space
|
||||
- Network access to container registry
|
||||
|
||||
**Deployment Steps:**
|
||||
|
||||
1. **SSH to EC2 instance**
|
||||
|
||||
2. **Login to container registry:**
|
||||
```bash
|
||||
echo "$GITHUB_TOKEN" | docker login ghcr.io -u USERNAME --password-stdin
|
||||
```
|
||||
|
||||
3. **Create/update docker-compose.yml:**
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
api:
|
||||
image: ghcr.io/owner/service:latest
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '3001:3001'
|
||||
env_file:
|
||||
- .env.production
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
```
|
||||
|
||||
4. **Pull latest images:**
|
||||
```bash
|
||||
docker compose pull
|
||||
```
|
||||
|
||||
5. **Deploy services:**
|
||||
```bash
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
6. **Verify health:**
|
||||
```bash
|
||||
docker compose ps
|
||||
docker inspect --format='{{.State.Health.Status}}' [container-name]
|
||||
```
|
||||
|
||||
### Multi-Service Deployment
|
||||
|
||||
**Deployment Order:**
|
||||
1. Dependencies first (Redis, databases)
|
||||
2. Independent services
|
||||
3. Dependent services (API)
|
||||
|
||||
**Example with Redis:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '6379:6379'
|
||||
healthcheck:
|
||||
test: ['CMD', 'redis-cli', 'ping']
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
api:
|
||||
image: ghcr.io/owner/api:latest
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### GitHub Secrets Management
|
||||
|
||||
**Required Secrets:**
|
||||
- `TAILSCALE_CLIENT_ID` / `TAILSCALE_CLIENT_SECRET` - For Tailscale access
|
||||
- `EC2_USER` / `EC2_SSH_KEY` - For SSH access (if not using Tailscale)
|
||||
- `NPM_TOKEN` - For private npm packages
|
||||
- `SENTRY_AUTH_TOKEN` - For Sentry source maps (if using)
|
||||
- `GITHUB_TOKEN` - Automatically provided, for registry access
|
||||
|
||||
**Setting Secrets:**
|
||||
1. Go to repository Settings → Secrets and variables → Actions
|
||||
2. Click "New repository secret"
|
||||
3. Add each secret with appropriate values
|
||||
|
||||
### Build Secrets in Docker
|
||||
|
||||
**Use BuildKit secrets for sensitive build-time data:**
|
||||
|
||||
```dockerfile
|
||||
RUN --mount=type=secret,id=NPM_TOKEN \
|
||||
export NPM_TOKEN=$(cat /run/secrets/NPM_TOKEN 2>/dev/null || echo "") && \
|
||||
npm ci
|
||||
```
|
||||
|
||||
**In GitHub Actions:**
|
||||
```yaml
|
||||
secrets: |
|
||||
NPM_TOKEN=${{ secrets.NPM_TOKEN }}
|
||||
SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }}
|
||||
```
|
||||
|
||||
### Container Security
|
||||
|
||||
- **Non-root user**: Always run containers as non-root
|
||||
- **Minimal base images**: Use `-slim` or `-alpine` variants
|
||||
- **No secrets in images**: Use environment variables or secrets management
|
||||
- **Health checks**: Enable health checks for monitoring
|
||||
- **Resource limits**: Set memory and CPU limits in docker-compose
|
||||
|
||||
### Network Security
|
||||
|
||||
- **Use Tailscale**: Avoid exposing services to public internet
|
||||
- **Security groups**: Restrict access to necessary ports only
|
||||
- **VPC**: Use private subnets when possible
|
||||
- **SSL/TLS**: Use HTTPS for all external-facing services
|
||||
|
||||
## Monitoring and Health Checks
|
||||
|
||||
### Health Check Endpoints
|
||||
|
||||
**Implement health check endpoint in your application:**
|
||||
|
||||
**NestJS:**
|
||||
```typescript
|
||||
// health.controller.ts
|
||||
@Controller('v1/health')
|
||||
export class HealthController {
|
||||
@Get()
|
||||
health() {
|
||||
return { status: 'ok', timestamp: new Date().toISOString() };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Express:**
|
||||
```typescript
|
||||
app.get('/v1/health', (req, res) => {
|
||||
res.json({ status: 'ok', timestamp: new Date().toISOString() });
|
||||
});
|
||||
```
|
||||
|
||||
### Docker Health Checks
|
||||
|
||||
**In Dockerfile:**
|
||||
```dockerfile
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD curl -f http://localhost:3001/v1/health || exit 1
|
||||
```
|
||||
|
||||
**In docker-compose:**
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
```
|
||||
|
||||
### Deployment Verification
|
||||
|
||||
**Check service health after deployment:**
|
||||
```bash
|
||||
# Check container status
|
||||
docker compose ps
|
||||
|
||||
# Check health status
|
||||
docker inspect --format='{{.State.Health.Status}}' [container-name]
|
||||
|
||||
# Check logs
|
||||
docker compose logs --tail=100 [service-name]
|
||||
|
||||
# Test health endpoint
|
||||
curl http://localhost:3001/v1/health
|
||||
```
|
||||
|
||||
### Post-Deployment Verification
|
||||
|
||||
**In GitHub Actions:**
|
||||
```yaml
|
||||
- name: Verify deployment
|
||||
run: |
|
||||
API_IP="${{ vars.TAILSCALE_INSTANCE_IP }}"
|
||||
for i in {1..10}; do
|
||||
if curl -f --max-time 10 http://${API_IP}:3001/v1/health; then
|
||||
echo "✅ Service is healthy"
|
||||
exit 0
|
||||
fi
|
||||
echo "Attempt $i/10 failed, retrying..."
|
||||
sleep 5
|
||||
done
|
||||
echo "❌ Service health check failed"
|
||||
exit 1
|
||||
```
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### Manual Rollback
|
||||
|
||||
**1. Find previous image tag:**
|
||||
- Check GitHub Container Registry
|
||||
- Look for tags like `master-<commit-sha>`
|
||||
- Or use semantic version tags
|
||||
|
||||
**2. Update docker-compose.yml:**
|
||||
```yaml
|
||||
services:
|
||||
api:
|
||||
image: ghcr.io/owner/service:master-abc123 # Previous tag
|
||||
```
|
||||
|
||||
**3. Deploy previous version:**
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
**4. Verify rollback:**
|
||||
```bash
|
||||
docker compose ps
|
||||
curl http://localhost:3001/v1/health
|
||||
```
|
||||
|
||||
### Automated Rollback Workflow
|
||||
|
||||
**Create `.github/workflows/rollback.yml`:**
|
||||
|
||||
```yaml
|
||||
name: Rollback Deployment
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
image_tag:
|
||||
description: 'Image tag to rollback to (e.g., master-abc123)'
|
||||
required: true
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
rollback:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
|
||||
- name: Rollback to previous version
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ vars.TAILSCALE_INSTANCE_IP }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
script: |
|
||||
cd ~/[project-path]
|
||||
# Update image tag in docker-compose.yml
|
||||
sed -i "s|image:.*|image: ghcr.io/owner/service:${{ inputs.image_tag }}|" docker-compose.yml
|
||||
docker compose pull
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
## Docker Cleanup and Maintenance
|
||||
|
||||
### Docker Prune Script
|
||||
|
||||
**Create `docker/prune-docker.sh`:**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Docker cleanup script for production servers
|
||||
|
||||
LOCK_FILE="/tmp/docker-prune.lock"
|
||||
LOCK_TIMEOUT=300 # 5 minutes max wait
|
||||
|
||||
echo "🧹 Starting Docker cleanup..."
|
||||
|
||||
# Acquire lock
|
||||
if [ -f "$LOCK_FILE" ]; then
|
||||
echo "⏳ Another prune operation is running, waiting..."
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
echo $$ > "$LOCK_FILE"
|
||||
trap "rm -f $LOCK_FILE" EXIT
|
||||
|
||||
# Prune old images (older than 24 hours)
|
||||
echo "🗑️ Pruning old Docker images..."
|
||||
docker image prune --filter "until=24h" -f
|
||||
|
||||
# Prune stopped containers
|
||||
echo "🗑️ Pruning stopped containers..."
|
||||
docker container prune --filter "until=24h" -f
|
||||
|
||||
# Prune unused networks
|
||||
echo "🗑️ Pruning unused networks..."
|
||||
docker network prune -f
|
||||
|
||||
# Show disk space
|
||||
echo "📊 Docker disk usage:"
|
||||
docker system df
|
||||
|
||||
echo "✅ Docker cleanup complete!"
|
||||
```
|
||||
|
||||
**Run cleanup after deployments:**
|
||||
```yaml
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
ssh -i ~/.ssh/key user@instance "cd ~/project && ./docker/prune-docker.sh"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**1. Docker Compose v2 not found:**
|
||||
```bash
|
||||
# Install Docker Compose v2
|
||||
sudo apt-get update
|
||||
sudo apt-get install docker-compose-plugin
|
||||
```
|
||||
|
||||
**2. Health check failures:**
|
||||
- Verify health endpoint is accessible
|
||||
- Check container logs: `docker compose logs [service]`
|
||||
- Ensure health check command is correct
|
||||
- Increase `start_period` for slow-starting services
|
||||
|
||||
**3. Image pull failures:**
|
||||
- Verify registry authentication
|
||||
- Check image tag exists
|
||||
- Verify network connectivity
|
||||
|
||||
**4. Deployment timeouts:**
|
||||
- Increase timeout in workflow
|
||||
- Check EC2 instance resources (CPU, memory)
|
||||
- Verify Tailscale connectivity
|
||||
|
||||
**5. Service not starting:**
|
||||
- Check environment variables
|
||||
- Verify dependencies are healthy
|
||||
- Review application logs
|
||||
- Check port conflicts
|
||||
|
||||
## Checklist
|
||||
|
||||
Before deploying, verify:
|
||||
|
||||
- [ ] Dockerfile is optimized (multi-stage build)
|
||||
- [ ] Health check endpoint implemented
|
||||
- [ ] Docker Compose v2 installed on EC2
|
||||
- [ ] Container registry configured
|
||||
- [ ] GitHub Secrets set up
|
||||
- [ ] Tailscale configured (or SSH access)
|
||||
- [ ] Environment variables configured
|
||||
- [ ] Health checks configured in docker-compose
|
||||
- [ ] Deployment workflow tested
|
||||
- [ ] Rollback procedure documented
|
||||
|
||||
## Next Steps
|
||||
|
||||
After initial deployment:
|
||||
|
||||
1. Set up monitoring and alerts
|
||||
2. Configure automatic cleanup
|
||||
3. Document deployment process
|
||||
4. Set up staging environment
|
||||
5. Implement blue-green deployments (optional)
|
||||
6. Configure log aggregation
|
||||
7. Set up backup procedures
|
||||
|
||||
@@ -0,0 +1,439 @@
|
||||
---
|
||||
name: mongodb-atlas-checker
|
||||
description: Expert in verifying MongoDB Atlas setup and configuration for backend applications. Checks connection strings, environment variables, database configuration, connection pooling, and ensures proper setup for Next.js and NestJS applications. This skill activates when users need to verify their MongoDB Atlas backend setup is correct.
|
||||
---
|
||||
|
||||
# MongoDB Atlas Checker
|
||||
|
||||
You are an expert in verifying MongoDB Atlas setup and configuration for backend applications. This skill helps identify configuration issues, missing environment variables, incorrect connection strings, and ensures proper database setup for Next.js and NestJS applications.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Verifying MongoDB Atlas backend setup
|
||||
- Checking if connection strings are correctly configured
|
||||
- Validating environment variable setup
|
||||
- Ensuring database connection is properly established
|
||||
- Reviewing MongoDB Atlas configuration
|
||||
- Troubleshooting database connection issues
|
||||
- Auditing database setup before deployment
|
||||
|
||||
## Project Context Discovery
|
||||
|
||||
**Before checking MongoDB Atlas setup, discover the project's context:**
|
||||
|
||||
1. **Scan Project Documentation:**
|
||||
- Check `.agent/SYSTEM/ARCHITECTURE.md` for database architecture
|
||||
- Review existing database patterns
|
||||
- Look for environment variable usage
|
||||
- Check for existing MongoDB integration
|
||||
|
||||
2. **Identify Framework:**
|
||||
- Determine if using Next.js (App Router or Pages Router)
|
||||
- Check if using NestJS backend
|
||||
- Review existing database connection patterns
|
||||
- Check for ORM/ODM usage (Mongoose, TypeORM, Prisma)
|
||||
|
||||
3. **Use Project-Specific Skills:**
|
||||
- Check for `[project]-mongodb-atlas-checker` skill
|
||||
- Review project-specific database patterns
|
||||
- Follow project's configuration standards
|
||||
|
||||
## Checklist: MongoDB Atlas Setup Verification
|
||||
|
||||
### 1. Environment Variables
|
||||
|
||||
**Check for required environment variables:**
|
||||
|
||||
```bash
|
||||
# Required for MongoDB Atlas
|
||||
MONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
# OR
|
||||
DATABASE_URL=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
**Verification Steps:**
|
||||
|
||||
- [ ] Environment variable exists (check `.env.local`, `.env`, or deployment config)
|
||||
- [ ] Variable name is consistent across codebase
|
||||
- [ ] Connection string uses `mongodb+srv://` protocol (required for Atlas)
|
||||
- [ ] Connection string includes authentication credentials
|
||||
- [ ] Connection string includes database name
|
||||
- [ ] Connection string includes query parameters (`retryWrites=true&w=majority`)
|
||||
- [ ] No hardcoded connection strings in source code
|
||||
- [ ] `.env.example` or `.env.template` has placeholder (not real credentials)
|
||||
|
||||
**Common Issues:**
|
||||
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded connection string
|
||||
const mongoUri = 'mongodb+srv://user:pass@cluster.mongodb.net/db';
|
||||
|
||||
// ❌ BAD: Wrong protocol (not supported by Atlas)
|
||||
const mongoUri = 'mongodb://user:pass@cluster.mongodb.net/db';
|
||||
|
||||
// ❌ BAD: Missing database name
|
||||
const mongoUri = 'mongodb+srv://user:pass@cluster.mongodb.net';
|
||||
|
||||
// ✅ GOOD: Environment variable
|
||||
const mongoUri = process.env.MONGODB_URI;
|
||||
```
|
||||
|
||||
### 2. Connection String Format
|
||||
|
||||
**MongoDB Atlas connection strings must:**
|
||||
- Use `mongodb+srv://` protocol (not `mongodb://`)
|
||||
- Include username and password
|
||||
- Include cluster hostname (e.g., `cluster0.xxxxx.mongodb.net`)
|
||||
- Include database name
|
||||
- Include query parameters for production readiness
|
||||
|
||||
**Valid Format:**
|
||||
```
|
||||
mongodb+srv://<username>:<password>@<cluster-host>/<database>?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
**Check for:**
|
||||
- [ ] Protocol is `mongodb+srv://`
|
||||
- [ ] Username and password are URL-encoded if they contain special characters
|
||||
- [ ] Cluster hostname is correct (from Atlas dashboard)
|
||||
- [ ] Database name is specified
|
||||
- [ ] Query parameters include `retryWrites=true&w=majority`
|
||||
- [ ] Optional: `appName` parameter for monitoring
|
||||
- [ ] Optional: `maxPoolSize` for connection pooling
|
||||
|
||||
**Example with all parameters:**
|
||||
```
|
||||
mongodb+srv://user:pass@cluster0.xxxxx.mongodb.net/mydb?retryWrites=true&w=majority&appName=MyApp&maxPoolSize=10
|
||||
```
|
||||
|
||||
### 3. Database Driver Installation
|
||||
|
||||
**Check if MongoDB driver is installed:**
|
||||
|
||||
**For Mongoose (ODM):**
|
||||
```bash
|
||||
# Check package.json
|
||||
npm list mongoose
|
||||
# or
|
||||
pnpm list mongoose
|
||||
```
|
||||
|
||||
**For Native MongoDB Driver:**
|
||||
```bash
|
||||
npm list mongodb
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `mongoose` or `mongodb` package is installed
|
||||
- [ ] Version is compatible with MongoDB Atlas
|
||||
- [ ] Package is listed in `package.json` dependencies (not devDependencies for production)
|
||||
|
||||
### 4. Connection Setup
|
||||
|
||||
**Next.js (App Router or Pages Router):**
|
||||
|
||||
**Check for proper connection pattern:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: Singleton pattern for Next.js
|
||||
// lib/mongodb.ts or utils/mongodb.ts
|
||||
import mongoose from 'mongoose';
|
||||
|
||||
const MONGODB_URI = process.env.MONGODB_URI!;
|
||||
|
||||
if (!MONGODB_URI) {
|
||||
throw new Error('Please define MONGODB_URI environment variable');
|
||||
}
|
||||
|
||||
interface MongooseCache {
|
||||
conn: typeof mongoose | null;
|
||||
promise: Promise<typeof mongoose> | null;
|
||||
}
|
||||
|
||||
declare global {
|
||||
var mongoose: MongooseCache | undefined;
|
||||
}
|
||||
|
||||
let cached: MongooseCache = global.mongoose || { conn: null, promise: null };
|
||||
|
||||
if (!global.mongoose) {
|
||||
global.mongoose = cached;
|
||||
}
|
||||
|
||||
async function connectDB() {
|
||||
if (cached.conn) {
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
if (!cached.promise) {
|
||||
const opts = {
|
||||
bufferCommands: false,
|
||||
};
|
||||
|
||||
cached.promise = mongoose.connect(MONGODB_URI, opts).then((mongoose) => {
|
||||
return mongoose;
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
cached.conn = await cached.promise;
|
||||
} catch (e) {
|
||||
cached.promise = null;
|
||||
throw e;
|
||||
}
|
||||
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
export default connectDB;
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Connection uses singleton pattern (prevents multiple connections in Next.js)
|
||||
- [ ] Connection is cached globally (for Next.js serverless functions)
|
||||
- [ ] Error handling is implemented
|
||||
- [ ] Connection options are configured (bufferCommands: false recommended)
|
||||
- [ ] Connection is called before database operations
|
||||
|
||||
**NestJS:**
|
||||
|
||||
**Check for MongooseModule configuration:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: NestJS MongooseModule
|
||||
// app.module.ts
|
||||
import { Module } from '@nestjs/common';
|
||||
import { MongooseModule } from '@nestjs/mongoose';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
MongooseModule.forRoot(process.env.MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
}),
|
||||
],
|
||||
})
|
||||
export class AppModule {}
|
||||
```
|
||||
|
||||
**Or with connection options:**
|
||||
|
||||
```typescript
|
||||
MongooseModule.forRoot(process.env.MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
maxPoolSize: 10,
|
||||
serverSelectionTimeoutMS: 5000,
|
||||
socketTimeoutMS: 45000,
|
||||
})
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `@nestjs/mongoose` package is installed
|
||||
- [ ] `MongooseModule.forRoot()` is configured in root module
|
||||
- [ ] Connection string comes from environment variable
|
||||
- [ ] Connection options are set appropriately
|
||||
- [ ] Error handling is in place
|
||||
|
||||
### 5. Connection Options
|
||||
|
||||
**Recommended connection options for MongoDB Atlas:**
|
||||
|
||||
```typescript
|
||||
{
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
maxPoolSize: 10, // Connection pool size
|
||||
serverSelectionTimeoutMS: 5000, // Timeout for server selection
|
||||
socketTimeoutMS: 45000, // Socket timeout
|
||||
connectTimeoutMS: 10000, // Connection timeout
|
||||
bufferCommands: false, // Disable mongoose buffering
|
||||
bufferMaxEntries: 0, // Disable mongoose buffering
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `retryWrites: true` is set (required for Atlas)
|
||||
- [ ] `w: 'majority'` is set (write concern)
|
||||
- [ ] Connection pool size is appropriate for your use case
|
||||
- [ ] Timeouts are configured appropriately
|
||||
- [ ] Buffer commands is disabled for serverless (Next.js)
|
||||
|
||||
### 6. Error Handling
|
||||
|
||||
**Check for proper error handling:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: Error handling
|
||||
try {
|
||||
await connectDB();
|
||||
// Database operations
|
||||
} catch (error) {
|
||||
console.error('MongoDB connection error:', error);
|
||||
// Handle error appropriately
|
||||
throw error;
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Connection errors are caught and handled
|
||||
- [ ] Error messages are logged appropriately
|
||||
- [ ] Application doesn't crash on connection failure
|
||||
- [ ] Retry logic is implemented if needed
|
||||
- [ ] Error handling is consistent across the codebase
|
||||
|
||||
### 7. Database Name Configuration
|
||||
|
||||
**Check if database name is correctly specified:**
|
||||
|
||||
- [ ] Database name is in connection string
|
||||
- [ ] Database name matches your application's needs
|
||||
- [ ] Database name doesn't contain special characters
|
||||
- [ ] Database name is consistent across environments (dev/staging/prod)
|
||||
|
||||
### 8. SSL/TLS Configuration
|
||||
|
||||
**MongoDB Atlas requires SSL/TLS by default:**
|
||||
|
||||
- [ ] Connection string doesn't explicitly disable SSL (Atlas requires it)
|
||||
- [ ] No `ssl=false` in connection string
|
||||
- [ ] TLS/SSL is enabled by default with `mongodb+srv://`
|
||||
|
||||
### 9. Network Access
|
||||
|
||||
**Check Atlas Network Access settings:**
|
||||
|
||||
- [ ] IP whitelist includes your deployment IPs
|
||||
- [ ] For development: `0.0.0.0/0` allows all IPs (not recommended for production)
|
||||
- [ ] For production: Specific IPs or VPC peering configured
|
||||
- [ ] Network access rules are documented
|
||||
|
||||
### 10. Database User Configuration
|
||||
|
||||
**Check Atlas Database User settings:**
|
||||
|
||||
- [ ] Database user exists in Atlas
|
||||
- [ ] User has appropriate permissions (read/write for application database)
|
||||
- [ ] Password is strong and secure
|
||||
- [ ] User credentials match connection string
|
||||
- [ ] User is not using admin credentials for application
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### Issue 1: Connection String Not Found
|
||||
|
||||
**Problem:** `MONGODB_URI` environment variable is missing
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Add to .env.local (Next.js) or .env (NestJS)
|
||||
MONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
### Issue 2: Wrong Protocol
|
||||
|
||||
**Problem:** Using `mongodb://` instead of `mongodb+srv://`
|
||||
|
||||
**Solution:** Change to `mongodb+srv://` (required for Atlas)
|
||||
|
||||
### Issue 3: Multiple Connections in Next.js
|
||||
|
||||
**Problem:** Creating new connection on each API call
|
||||
|
||||
**Solution:** Use singleton pattern to cache connection (see Connection Setup section)
|
||||
|
||||
### Issue 4: Connection Timeout
|
||||
|
||||
**Problem:** Connection times out
|
||||
|
||||
**Solution:**
|
||||
- Check network access in Atlas dashboard
|
||||
- Verify IP whitelist
|
||||
- Increase `connectTimeoutMS` and `serverSelectionTimeoutMS`
|
||||
- Check firewall settings
|
||||
|
||||
### Issue 5: Authentication Failed
|
||||
|
||||
**Problem:** Username/password incorrect
|
||||
|
||||
**Solution:**
|
||||
- Verify credentials in Atlas dashboard
|
||||
- Check if password contains special characters (needs URL encoding)
|
||||
- Verify database user exists and has permissions
|
||||
|
||||
## Verification Script
|
||||
|
||||
**Create a test script to verify connection:**
|
||||
|
||||
```typescript
|
||||
// scripts/test-mongodb-connection.ts
|
||||
import mongoose from 'mongoose';
|
||||
|
||||
const MONGODB_URI = process.env.MONGODB_URI;
|
||||
|
||||
if (!MONGODB_URI) {
|
||||
console.error('❌ MONGODB_URI environment variable is missing');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function testConnection() {
|
||||
try {
|
||||
await mongoose.connect(MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
});
|
||||
|
||||
console.log('✅ Successfully connected to MongoDB Atlas');
|
||||
|
||||
// Test a simple operation
|
||||
const collections = await mongoose.connection.db.listCollections().toArray();
|
||||
console.log(`✅ Found ${collections.length} collections`);
|
||||
|
||||
await mongoose.disconnect();
|
||||
console.log('✅ Connection closed');
|
||||
process.exit(0);
|
||||
} catch (error) {
|
||||
console.error('❌ MongoDB connection error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
testConnection();
|
||||
```
|
||||
|
||||
**Run the test:**
|
||||
```bash
|
||||
# Load environment variables and run
|
||||
node -r dotenv/config scripts/test-mongodb-connection.ts
|
||||
# or
|
||||
ts-node scripts/test-mongodb-connection.ts
|
||||
```
|
||||
|
||||
## Summary Checklist
|
||||
|
||||
Before considering MongoDB Atlas setup complete, verify:
|
||||
|
||||
- [ ] `MONGODB_URI` environment variable exists and is correct
|
||||
- [ ] Connection string uses `mongodb+srv://` protocol
|
||||
- [ ] Connection string includes database name
|
||||
- [ ] MongoDB driver (mongoose or mongodb) is installed
|
||||
- [ ] Connection setup follows framework best practices
|
||||
- [ ] Connection options are configured appropriately
|
||||
- [ ] Error handling is implemented
|
||||
- [ ] Network access is configured in Atlas
|
||||
- [ ] Database user has appropriate permissions
|
||||
- [ ] No hardcoded credentials in source code
|
||||
- [ ] Connection test script passes
|
||||
|
||||
## Next Steps
|
||||
|
||||
After verifying setup:
|
||||
1. Test connection with verification script
|
||||
2. Create initial database schema/models
|
||||
3. Set up database indexes
|
||||
4. Configure connection pooling for production
|
||||
5. Set up monitoring and alerts in Atlas dashboard
|
||||
6. Document connection setup in project documentation
|
||||
|
||||
@@ -0,0 +1,916 @@
|
||||
---
|
||||
name: open-source-checker
|
||||
description: Expert in detecting private information, secrets, API keys, credentials, and sensitive data in codebases before open sourcing. Scans for hardcoded secrets, API keys, passwords, private keys, personal information, database credentials, and other sensitive data that should not be exposed in public repositories. Can also set up git hooks and pre-commit hooks to prevent committing secrets.
|
||||
---
|
||||
|
||||
# Open Source Checker
|
||||
|
||||
You are an expert in detecting private information, secrets, and sensitive data in codebases. This skill helps identify and flag any private information before open sourcing a repository, and can set up automated checks via git hooks to prevent future issues.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Preparing to open source a repository
|
||||
- Reviewing code for exposed secrets
|
||||
- Auditing codebase for sensitive data
|
||||
- Checking for hardcoded credentials
|
||||
- Validating that no private information is committed
|
||||
- Reviewing pull requests for secrets
|
||||
- Performing security audits before public release
|
||||
|
||||
## What to Check For
|
||||
|
||||
### 1. API Keys and Tokens
|
||||
|
||||
**Common patterns:**
|
||||
- API keys (OpenAI, Stripe, AWS, Google, etc.)
|
||||
- Authentication tokens
|
||||
- OAuth tokens
|
||||
- JWT secrets
|
||||
- Session keys
|
||||
- Webhook secrets
|
||||
|
||||
**Patterns to detect:**
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded API keys
|
||||
const apiKey = 'sk-1234567890abcdef';
|
||||
const stripeKey = 'sk_live_...';
|
||||
const awsKey = 'AKIAIOSFODNN7EXAMPLE';
|
||||
|
||||
// ✅ GOOD: Environment variables
|
||||
const apiKey = process.env.API_KEY;
|
||||
const stripeKey = process.env.STRIPE_SECRET_KEY;
|
||||
```
|
||||
|
||||
**Common locations:**
|
||||
- Configuration files
|
||||
- Source code files
|
||||
- Environment files (`.env` files that might be committed)
|
||||
- Test files
|
||||
- Documentation files
|
||||
- Example files
|
||||
|
||||
### 2. Database Credentials
|
||||
|
||||
**Check for:**
|
||||
- Database connection strings
|
||||
- Usernames and passwords
|
||||
- MongoDB URIs
|
||||
- PostgreSQL connection strings
|
||||
- Redis credentials
|
||||
- Database host addresses
|
||||
|
||||
**Patterns:**
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded credentials
|
||||
const mongoUri = 'mongodb://user:password@host:27017/db';
|
||||
const dbPassword = 'mySecretPassword123';
|
||||
|
||||
// ✅ GOOD: Environment variables
|
||||
const mongoUri = process.env.MONGODB_URI;
|
||||
```
|
||||
|
||||
### 3. Private Keys and Certificates
|
||||
|
||||
**Check for:**
|
||||
- SSH private keys
|
||||
- SSL/TLS certificates
|
||||
- Private key files (`.pem`, `.key`, `.p12`)
|
||||
- Certificate files
|
||||
- Signing keys
|
||||
|
||||
**Files to check:**
|
||||
- `*.pem`, `*.key`, `*.p12`, `*.pfx`
|
||||
- `id_rsa`, `id_dsa`, `id_ecdsa`
|
||||
- `*.crt`, `*.cer`, `*.cert`
|
||||
|
||||
### 4. Personal Information
|
||||
|
||||
**Check for:**
|
||||
- Email addresses
|
||||
- Phone numbers
|
||||
- Physical addresses
|
||||
- Personal names
|
||||
- Social security numbers
|
||||
- Credit card numbers
|
||||
- Bank account numbers
|
||||
|
||||
**Patterns:**
|
||||
```typescript
|
||||
// ❌ BAD: Personal information
|
||||
const adminEmail = 'john.doe@example.com';
|
||||
const phone = '+1-555-123-4567';
|
||||
|
||||
// ✅ GOOD: Placeholder or environment variable
|
||||
const adminEmail = process.env.ADMIN_EMAIL;
|
||||
```
|
||||
|
||||
### 5. Environment Files
|
||||
|
||||
**Check for:**
|
||||
- `.env` files (should be in `.gitignore`)
|
||||
- `.env.local`, `.env.production`
|
||||
- Files containing actual secrets (not `.env.example`)
|
||||
|
||||
**Verify:**
|
||||
- `.env` is in `.gitignore`
|
||||
- Only `.env.example` is committed (with placeholder values)
|
||||
- No actual secrets in any committed `.env` files
|
||||
|
||||
### 6. Configuration Files with Secrets
|
||||
|
||||
**Check:**
|
||||
- `config.json`, `config.js`, `config.ts`
|
||||
- `settings.json`, `settings.js`
|
||||
- `secrets.json`, `secrets.js`
|
||||
- Any config file with hardcoded values
|
||||
|
||||
### 7. Comments and Documentation
|
||||
|
||||
**Check for:**
|
||||
- Secrets in code comments
|
||||
- API keys in README files
|
||||
- Credentials in documentation
|
||||
- Test credentials that might be real
|
||||
|
||||
### 8. Git History
|
||||
|
||||
**Check for:**
|
||||
- Secrets in git history (even if removed)
|
||||
- Committed `.env` files in history
|
||||
- Secrets in old commits
|
||||
|
||||
**Commands to check:**
|
||||
```bash
|
||||
# Search git history for secrets
|
||||
git log --all --full-history --source -S "sk-" -- "*.ts" "*.js" "*.json"
|
||||
git log --all --full-history --source -S "password" -- "*.env"
|
||||
```
|
||||
|
||||
## Scanning Workflow
|
||||
|
||||
### Phase 1: File System Scan
|
||||
|
||||
**1.1 Check for Common Secret Files**
|
||||
|
||||
```bash
|
||||
# Find potential secret files
|
||||
find . -name "*.env" -o -name "*.key" -o -name "*.pem" -o -name "id_rsa*"
|
||||
find . -name "secrets.*" -o -name "*secret*"
|
||||
find . -name ".env*" ! -name ".env.example"
|
||||
```
|
||||
|
||||
**1.2 Check .gitignore**
|
||||
|
||||
```bash
|
||||
# Verify .env is ignored
|
||||
cat .gitignore | grep -E "\.env|secrets|\.key|\.pem"
|
||||
```
|
||||
|
||||
**1.3 Scan for Common Patterns**
|
||||
|
||||
```bash
|
||||
# Search for API key patterns
|
||||
grep -r "sk-[a-zA-Z0-9]" --include="*.ts" --include="*.js" --include="*.json"
|
||||
grep -r "AKIA[0-9A-Z]" --include="*.ts" --include="*.js"
|
||||
grep -r "sk_live_" --include="*.ts" --include="*.js"
|
||||
```
|
||||
|
||||
### Phase 2: Code Pattern Analysis
|
||||
|
||||
**2.1 Search for Hardcoded Secrets**
|
||||
|
||||
Look for:
|
||||
- String literals that look like API keys
|
||||
- Hardcoded passwords
|
||||
- Connection strings with credentials
|
||||
- Token values in code
|
||||
|
||||
**2.2 Check Configuration Files**
|
||||
|
||||
Review:
|
||||
- All config files for hardcoded values
|
||||
- Environment variable usage (should use `process.env`)
|
||||
- Default values that might be secrets
|
||||
|
||||
**2.3 Review Test Files**
|
||||
|
||||
Check:
|
||||
- Test credentials (should be mocks, not real)
|
||||
- Test API keys (should be fake/test keys)
|
||||
- Test database connections
|
||||
|
||||
### Phase 3: Content Analysis
|
||||
|
||||
**3.1 Check Documentation**
|
||||
|
||||
- README files
|
||||
- Documentation files
|
||||
- Comments in code
|
||||
- Example code snippets
|
||||
|
||||
**3.2 Check Example Files**
|
||||
|
||||
- `.env.example` should have placeholders
|
||||
- Example configs should not have real values
|
||||
- Sample code should not include real keys
|
||||
|
||||
### Phase 4: Git History Check
|
||||
|
||||
**⚠️ CRITICAL: Even if secrets are removed from current files, they remain in git history forever unless explicitly removed.**
|
||||
|
||||
**4.1 Comprehensive Git History Scan**
|
||||
|
||||
**Search for API Keys in History:**
|
||||
|
||||
```bash
|
||||
# Search entire git history for OpenAI API keys
|
||||
git log --all --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
|
||||
# Search for AWS keys
|
||||
git log --all --full-history -p -S "AKIA" | grep -B 5 -A 5 "AKIA"
|
||||
|
||||
# Search for Stripe keys
|
||||
git log --all --full-history -p -S "sk_live_" | grep -B 5 -A 5 "sk_live_"
|
||||
|
||||
# Search for GitHub tokens
|
||||
git log --all --full-history -p -S "ghp_" | grep -B 5 -A 5 "ghp_"
|
||||
|
||||
# Search for passwords
|
||||
git log --all --full-history -p -S "password" | grep -B 5 -A 5 "password"
|
||||
|
||||
# Search for connection strings
|
||||
git log --all --full-history -p -S "mongodb://" | grep -B 5 -A 5 "mongodb://"
|
||||
git log --all --full-history -p -S "postgres://" | grep -B 5 -A 5 "postgres://"
|
||||
```
|
||||
|
||||
**Search All Branches and Tags:**
|
||||
|
||||
```bash
|
||||
# Check all branches (including remote)
|
||||
git log --all --branches --tags --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
|
||||
# Check specific branches
|
||||
git log origin/main origin/develop --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
```
|
||||
|
||||
**Search in Deleted Files:**
|
||||
|
||||
```bash
|
||||
# Find commits that deleted files containing secrets
|
||||
git log --all --full-history --diff-filter=D --summary | grep -E "\.env|secrets|\.key"
|
||||
|
||||
# Check what was in deleted files
|
||||
git log --all --full-history --diff-filter=D -- "*.env" | grep -A 10 "delete mode"
|
||||
```
|
||||
|
||||
**Search in Specific File Types:**
|
||||
|
||||
```bash
|
||||
# Search history of .env files
|
||||
git log --all --full-history -p -- "*.env" | grep -E "(sk-|password|AKIA)"
|
||||
|
||||
# Search history of config files
|
||||
git log --all --full-history -p -- "config.*" | grep -E "(sk-|password|AKIA)"
|
||||
|
||||
# Search history of all JavaScript/TypeScript files
|
||||
git log --all --full-history -p -- "*.{js,ts}" | grep -E "(sk-|password|AKIA)"
|
||||
```
|
||||
|
||||
**4.2 Using Tools to Scan Git History**
|
||||
|
||||
**gitleaks (Recommended for Git History):**
|
||||
|
||||
```bash
|
||||
# Install gitleaks
|
||||
brew install gitleaks
|
||||
|
||||
# Scan entire git history (all branches, all commits)
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
|
||||
# Scan specific branch
|
||||
gitleaks detect --source . --verbose --log-opts="--all --branches=main"
|
||||
|
||||
# Scan with custom config
|
||||
gitleaks detect --source . --verbose --log-opts="--all" --config-path=.gitleaks.toml
|
||||
```
|
||||
|
||||
**truffleHog (Scans Git History):**
|
||||
|
||||
```bash
|
||||
# Install truffleHog
|
||||
pip install truffleHog
|
||||
|
||||
# Scan entire git history
|
||||
trufflehog --regex --entropy=False git file://.
|
||||
|
||||
# Scan specific branch
|
||||
trufflehog --regex --entropy=False git file://. --branch=main
|
||||
```
|
||||
|
||||
**git-secrets (History Scan):**
|
||||
|
||||
```bash
|
||||
# Install git-secrets
|
||||
brew install git-secrets
|
||||
|
||||
# Scan entire history
|
||||
git secrets --scan-history
|
||||
|
||||
# Scan specific commit range
|
||||
git secrets --scan-history HEAD~10..HEAD
|
||||
```
|
||||
|
||||
**4.3 Check for Secrets in Merge Commits**
|
||||
|
||||
```bash
|
||||
# Search merge commits specifically
|
||||
git log --all --merges --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
|
||||
# Check merge commits for .env files
|
||||
git log --all --merges --full-history --diff-filter=M -- "*.env"
|
||||
```
|
||||
|
||||
**4.4 Check Stashed Changes**
|
||||
|
||||
```bash
|
||||
# List all stashes
|
||||
git stash list
|
||||
|
||||
# Check each stash for secrets
|
||||
git stash show -p stash@{0} | grep -E "(sk-|password|AKIA)"
|
||||
git stash show -p stash@{1} | grep -E "(sk-|password|AKIA)"
|
||||
```
|
||||
|
||||
**4.5 Automated Git History Scan Script**
|
||||
|
||||
Create a script to scan entire history:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scan-git-history.sh
|
||||
|
||||
echo "🔍 Scanning entire git history for secrets..."
|
||||
|
||||
# Patterns to search for
|
||||
PATTERNS=(
|
||||
"sk-[a-zA-Z0-9]"
|
||||
"AKIA[0-9A-Z]"
|
||||
"sk_live_"
|
||||
"sk_test_"
|
||||
"ghp_"
|
||||
"mongodb://.*:.*@"
|
||||
"postgres://.*:.*@"
|
||||
)
|
||||
|
||||
for pattern in "${PATTERNS[@]}"; do
|
||||
echo "Checking for pattern: $pattern"
|
||||
git log --all --full-history -p -S "$pattern" | grep -B 5 -A 5 "$pattern" && {
|
||||
echo "⚠️ Found matches for: $pattern"
|
||||
}
|
||||
done
|
||||
|
||||
# Check for .env files in history
|
||||
echo "Checking for .env files in history..."
|
||||
git log --all --full-history --name-only --diff-filter=A | grep -E "\.env$" | sort -u
|
||||
|
||||
# Use gitleaks if available
|
||||
if command -v gitleaks &> /dev/null; then
|
||||
echo "Running gitleaks on git history..."
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
fi
|
||||
```
|
||||
|
||||
**4.6 Cleaning Git History (If Secrets Found)**
|
||||
|
||||
**⚠️ WARNING: These operations rewrite git history. Coordinate with your team first.**
|
||||
|
||||
**Option 1: Using git-filter-repo (Recommended)**
|
||||
|
||||
```bash
|
||||
# Install git-filter-repo
|
||||
pip install git-filter-repo
|
||||
|
||||
# Remove secrets from entire history
|
||||
git filter-repo --invert-paths --path "file-with-secret.txt"
|
||||
git filter-repo --replace-text <(echo "sk-OLD-KEY==>sk-REMOVED")
|
||||
|
||||
# Remove .env files from history
|
||||
git filter-repo --invert-paths --path-glob "*.env"
|
||||
```
|
||||
|
||||
**Option 2: Using BFG Repo-Cleaner**
|
||||
|
||||
```bash
|
||||
# Install BFG
|
||||
brew install bfg
|
||||
|
||||
# Remove secrets file from history
|
||||
bfg --delete-files secrets.json
|
||||
|
||||
# Replace secrets in history
|
||||
echo "sk-OLD-KEY==>sk-REMOVED" > secrets-replacements.txt
|
||||
bfg --replace-text secrets-replacements.txt
|
||||
```
|
||||
|
||||
**Option 3: Fresh Repository (If History Too Contaminated)**
|
||||
|
||||
If the history is too contaminated, consider:
|
||||
1. Create a fresh repository
|
||||
2. Copy current clean state
|
||||
3. Start fresh history
|
||||
|
||||
```bash
|
||||
# Create fresh repo
|
||||
git checkout --orphan fresh-start
|
||||
git add .
|
||||
git commit -m "Initial commit (cleaned history)"
|
||||
git branch -D main # Delete old main
|
||||
git branch -m main # Rename current to main
|
||||
git push -f origin main # Force push (coordinate with team!)
|
||||
```
|
||||
|
||||
**4.7 Verify History is Clean**
|
||||
|
||||
After cleaning, verify:
|
||||
|
||||
```bash
|
||||
# Re-scan history
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
|
||||
# Check specific patterns
|
||||
git log --all --full-history -p -S "sk-" | grep "sk-"
|
||||
|
||||
# Verify no .env files in history
|
||||
git log --all --full-history --name-only | grep "\.env$"
|
||||
```
|
||||
|
||||
**4.8 Best Practices for Git History**
|
||||
|
||||
1. **Scan before open sourcing**: Always scan entire history before making repo public
|
||||
2. **Use tools**: Automated tools like gitleaks are more thorough than manual searches
|
||||
3. **Check all branches**: Secrets might be in feature branches
|
||||
4. **Check tags**: Tags preserve old commits
|
||||
5. **Coordinate cleanup**: If cleaning history, coordinate with all contributors
|
||||
6. **Rotate exposed secrets**: If secrets were in history, rotate them immediately
|
||||
7. **Set up hooks**: Prevent future commits with pre-commit hooks
|
||||
|
||||
## Common Patterns to Detect
|
||||
|
||||
### API Key Patterns
|
||||
|
||||
```typescript
|
||||
// OpenAI
|
||||
sk-[a-zA-Z0-9]{32,}
|
||||
|
||||
// AWS
|
||||
AKIA[0-9A-Z]{16}
|
||||
|
||||
// Stripe
|
||||
sk_live_[a-zA-Z0-9]{24,}
|
||||
sk_test_[a-zA-Z0-9]{24,}
|
||||
|
||||
// GitHub
|
||||
ghp_[a-zA-Z0-9]{36}
|
||||
|
||||
// Generic
|
||||
[a-zA-Z0-9_-]{20,} // Long alphanumeric strings
|
||||
```
|
||||
|
||||
### Password Patterns
|
||||
|
||||
```typescript
|
||||
// Common patterns
|
||||
password\s*[:=]\s*['"][^'"]+['"]
|
||||
pwd\s*[:=]\s*['"][^'"]+['"]
|
||||
pass\s*[:=]\s*['"][^'"]+['"]
|
||||
```
|
||||
|
||||
### Connection String Patterns
|
||||
|
||||
```typescript
|
||||
// MongoDB
|
||||
mongodb://[^:]+:[^@]+@
|
||||
mongodb\+srv://[^:]+:[^@]+@
|
||||
|
||||
// PostgreSQL
|
||||
postgres://[^:]+:[^@]+@
|
||||
postgresql://[^:]+:[^@]+@
|
||||
|
||||
// MySQL
|
||||
mysql://[^:]+:[^@]+@
|
||||
|
||||
// Redis
|
||||
redis://[^:]+:[^@]+@
|
||||
```
|
||||
|
||||
### Email Patterns
|
||||
|
||||
```typescript
|
||||
// Email addresses
|
||||
[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
|
||||
```
|
||||
|
||||
## Automated Tools
|
||||
|
||||
### Recommended Tools
|
||||
|
||||
**1. git-secrets**
|
||||
```bash
|
||||
# Install
|
||||
brew install git-secrets
|
||||
|
||||
# Setup
|
||||
git secrets --install
|
||||
git secrets --register-aws
|
||||
|
||||
# Scan
|
||||
git secrets --scan
|
||||
```
|
||||
|
||||
**2. truffleHog**
|
||||
```bash
|
||||
# Install
|
||||
pip install truffleHog
|
||||
|
||||
# Scan
|
||||
trufflehog --regex --entropy=False .
|
||||
```
|
||||
|
||||
**3. detect-secrets**
|
||||
```bash
|
||||
# Install
|
||||
pip install detect-secrets
|
||||
|
||||
# Scan
|
||||
detect-secrets scan --all-files
|
||||
```
|
||||
|
||||
**4. gitleaks**
|
||||
```bash
|
||||
# Install
|
||||
brew install gitleaks
|
||||
|
||||
# Scan
|
||||
gitleaks detect --source . --verbose
|
||||
```
|
||||
|
||||
## Git Hooks and Pre-Commit Hooks
|
||||
|
||||
Setting up git hooks prevents committing secrets before they enter the repository. This is the best way to catch issues early.
|
||||
|
||||
### Pre-Commit Hook Setup
|
||||
|
||||
**1. Using git-secrets (Recommended)**
|
||||
|
||||
```bash
|
||||
# Install git-secrets
|
||||
brew install git-secrets
|
||||
|
||||
# Initialize in your repository
|
||||
cd /path/to/your/repo
|
||||
git secrets --install
|
||||
|
||||
# Register AWS patterns (or other providers)
|
||||
git secrets --register-aws
|
||||
|
||||
# Add custom patterns
|
||||
git secrets --add 'sk-[a-zA-Z0-9]{32,}'
|
||||
git secrets --add 'AKIA[0-9A-Z]{16}'
|
||||
git secrets --add 'sk_live_[a-zA-Z0-9]{24,}'
|
||||
|
||||
# Test the hook
|
||||
git secrets --scan
|
||||
```
|
||||
|
||||
**2. Using gitleaks**
|
||||
|
||||
```bash
|
||||
# Install gitleaks
|
||||
brew install gitleaks
|
||||
|
||||
# Create pre-commit hook
|
||||
cat > .git/hooks/pre-commit << 'EOF'
|
||||
#!/bin/sh
|
||||
gitleaks detect --staged --verbose
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "❌ gitleaks detected secrets in your changes. Commit aborted."
|
||||
exit 1
|
||||
fi
|
||||
EOF
|
||||
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
**3. Using detect-secrets**
|
||||
|
||||
```bash
|
||||
# Install detect-secrets
|
||||
pip install detect-secrets
|
||||
|
||||
# Create baseline
|
||||
detect-secrets scan > .secrets.baseline
|
||||
|
||||
# Create pre-commit hook
|
||||
cat > .git/hooks/pre-commit << 'EOF'
|
||||
#!/bin/sh
|
||||
detect-secrets scan --baseline .secrets.baseline
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "❌ detect-secrets found new secrets. Commit aborted."
|
||||
exit 1
|
||||
fi
|
||||
EOF
|
||||
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
**4. Using Husky (for Node.js projects)**
|
||||
|
||||
```bash
|
||||
# Install husky
|
||||
npm install --save-dev husky
|
||||
|
||||
# Initialize husky
|
||||
npx husky install
|
||||
|
||||
# Add pre-commit hook
|
||||
npx husky add .husky/pre-commit "gitleaks detect --staged --verbose"
|
||||
```
|
||||
|
||||
**5. Manual Pre-Commit Hook**
|
||||
|
||||
Create `.git/hooks/pre-commit`:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
#
|
||||
# Pre-commit hook to check for secrets
|
||||
#
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo "🔍 Checking for secrets..."
|
||||
|
||||
# Check for common API key patterns
|
||||
if git diff --cached --name-only | xargs grep -E "(sk-[a-zA-Z0-9]{32,}|AKIA[0-9A-Z]{16}|sk_live_[a-zA-Z0-9]{24,})" 2>/dev/null; then
|
||||
echo "${RED}❌ Potential API keys detected in staged files!${NC}"
|
||||
echo "${YELLOW}Please remove secrets before committing.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for .env files
|
||||
if git diff --cached --name-only | grep -E "\.env$" | grep -v "\.env\.example"; then
|
||||
echo "${RED}❌ .env file detected!${NC}"
|
||||
echo "${YELLOW}Please ensure .env files are in .gitignore.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for private keys
|
||||
if git diff --cached --name-only | grep -E "\.(key|pem|p12|pfx)$"; then
|
||||
echo "${RED}❌ Private key file detected!${NC}"
|
||||
echo "${YELLOW}Please ensure private keys are in .gitignore.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "${GREEN}✅ No secrets detected. Proceeding with commit.${NC}"
|
||||
exit 0
|
||||
```
|
||||
|
||||
Make it executable:
|
||||
```bash
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
### Post-Commit Hook (Optional)
|
||||
|
||||
Create `.git/hooks/post-commit` to scan after commit:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
#
|
||||
# Post-commit hook to scan for secrets
|
||||
#
|
||||
|
||||
echo "🔍 Scanning last commit for secrets..."
|
||||
|
||||
# Use gitleaks or your preferred tool
|
||||
gitleaks detect --log-opts="-1" --verbose
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "⚠️ Secrets detected in last commit!"
|
||||
echo "Consider amending the commit or using git-filter-repo to remove secrets."
|
||||
fi
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
**GitHub Actions Example:**
|
||||
|
||||
```yaml
|
||||
name: Secret Scanning
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
scan:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
fetch-depth: 0 # Full history for gitleaks
|
||||
|
||||
- name: Run Gitleaks
|
||||
uses: gitleaks/gitleaks-action@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
**GitLab CI Example:**
|
||||
|
||||
```yaml
|
||||
secret-scan:
|
||||
image: zricethezav/gitleaks:latest
|
||||
script:
|
||||
- gitleaks detect --source . --verbose --no-git
|
||||
allow_failure: false
|
||||
```
|
||||
|
||||
### Hook Best Practices
|
||||
|
||||
1. **Fail fast**: Hooks should exit with non-zero code to prevent commits
|
||||
2. **Clear messages**: Provide actionable error messages
|
||||
3. **Fast execution**: Keep hooks fast to avoid slowing down workflow
|
||||
4. **Team-wide**: Ensure all team members have hooks installed
|
||||
5. **CI/CD backup**: Don't rely solely on hooks; use CI/CD as backup
|
||||
6. **Regular updates**: Update patterns and tools regularly
|
||||
|
||||
## Checklist
|
||||
|
||||
### Before Open Sourcing
|
||||
|
||||
- [ ] No hardcoded API keys in code
|
||||
- [ ] No database credentials in code
|
||||
- [ ] No private keys or certificates committed
|
||||
- [ ] `.env` files in `.gitignore`
|
||||
- [ ] Only `.env.example` committed (with placeholders)
|
||||
- [ ] **Git history scanned for secrets (CRITICAL)**
|
||||
- [ ] All branches checked for secrets
|
||||
- [ ] All tags checked for secrets
|
||||
- [ ] Deleted files checked for secrets
|
||||
- [ ] Merge commits checked for secrets
|
||||
- [ ] No secrets found in git history
|
||||
- [ ] Git history cleaned if secrets were found
|
||||
- [ ] No personal information in code
|
||||
- [ ] No real credentials in test files
|
||||
- [ ] No secrets in documentation
|
||||
- [ ] Configuration files use environment variables
|
||||
- [ ] All sensitive files in `.gitignore`
|
||||
|
||||
### Files to Verify
|
||||
|
||||
- [ ] `.env` - Should be ignored
|
||||
- [ ] `.env.local` - Should be ignored
|
||||
- [ ] `.env.production` - Should be ignored
|
||||
- [ ] `config.json` - Should not contain secrets
|
||||
- [ ] `secrets.json` - Should not exist or be ignored
|
||||
- [ ] `*.key`, `*.pem` - Should be ignored
|
||||
- [ ] `id_rsa*` - Should be ignored
|
||||
- [ ] README.md - Should not contain real secrets
|
||||
- [ ] Documentation files - Should not contain secrets
|
||||
|
||||
## Output Format
|
||||
|
||||
When checking for private information:
|
||||
|
||||
```
|
||||
🔍 PRIVATE INFORMATION SCAN REPORT
|
||||
|
||||
Repository: [repo-name]
|
||||
Date: [date]
|
||||
Scanner: [tool/agent]
|
||||
|
||||
📊 SUMMARY
|
||||
- Critical issues: 3
|
||||
- Warnings: 5
|
||||
- Files scanned: 150
|
||||
- Patterns checked: 12
|
||||
- Git history scanned: Yes
|
||||
- Branches checked: 5
|
||||
- Commits in history: 1,234
|
||||
|
||||
🚨 CRITICAL ISSUES
|
||||
|
||||
1. Hardcoded API Key Found
|
||||
File: src/config/api.ts:23
|
||||
Line: const apiKey = 'sk-1234567890abcdef';
|
||||
Issue: OpenAI API key exposed in code
|
||||
Fix: Move to environment variable
|
||||
Severity: CRITICAL
|
||||
Action: Remove immediately and rotate key
|
||||
|
||||
2. Database Credentials in Code
|
||||
File: src/database/config.ts:12
|
||||
Line: const mongoUri = 'mongodb://user:password@host:27017/db';
|
||||
Issue: Database credentials exposed
|
||||
Fix: Use environment variable
|
||||
Severity: CRITICAL
|
||||
Action: Remove and change database password
|
||||
|
||||
3. Secrets Found in Git History
|
||||
Commit: abc123def (2024-01-15)
|
||||
File: config/secrets.json (now deleted)
|
||||
Issue: API key was committed and then deleted, but still in history
|
||||
Fix: Clean git history using git-filter-repo
|
||||
Severity: CRITICAL
|
||||
Action: Remove from history and rotate exposed keys
|
||||
|
||||
⚠️ WARNINGS
|
||||
|
||||
1. .env File Not in .gitignore
|
||||
File: .env
|
||||
Issue: Environment file may be committed
|
||||
Fix: Add .env to .gitignore
|
||||
Severity: HIGH
|
||||
|
||||
2. Potential API Key in Comment
|
||||
File: src/utils/helpers.ts:45
|
||||
Line: // API key: sk-test-12345
|
||||
Issue: Comment contains what looks like an API key
|
||||
Fix: Remove comment
|
||||
Severity: MEDIUM
|
||||
|
||||
[... more issues ...]
|
||||
|
||||
✅ SAFE FILES
|
||||
|
||||
- ✅ .env.example contains only placeholders
|
||||
- ✅ All config files use environment variables
|
||||
- ✅ No secrets in documentation
|
||||
- ✅ Test files use mock credentials
|
||||
|
||||
📜 GIT HISTORY SCAN RESULTS
|
||||
|
||||
- ✅ Current files: No secrets detected
|
||||
- ⚠️ Git history: 2 secrets found in old commits
|
||||
- ✅ All branches scanned: main, develop, feature/*
|
||||
- ✅ All tags scanned: v1.0.0, v1.1.0
|
||||
- ⚠️ Action required: Clean git history before open sourcing
|
||||
|
||||
💡 RECOMMENDATIONS
|
||||
|
||||
1. Add .env to .gitignore if not already
|
||||
2. Use environment variables for all secrets
|
||||
3. Rotate any exposed API keys
|
||||
4. Clean git history if secrets were committed
|
||||
5. Set up pre-commit hooks to prevent future commits
|
||||
6. Use secret scanning in CI/CD
|
||||
|
||||
📋 NEXT STEPS
|
||||
|
||||
1. Fix critical issues immediately
|
||||
2. Rotate any exposed credentials
|
||||
3. Clean git history if needed
|
||||
4. Set up automated scanning
|
||||
5. Review and approve before open sourcing
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Never commit secrets**: Always use environment variables
|
||||
2. **Use .env.example**: Provide template with placeholders
|
||||
3. **Rotate exposed secrets**: If secrets were committed, rotate them
|
||||
4. **Clean git history**: Remove secrets from history if committed
|
||||
5. **Automate scanning**: Use pre-commit hooks and CI/CD checks
|
||||
6. **Document requirements**: List required environment variables
|
||||
7. **Use secret management**: Consider services like AWS Secrets Manager
|
||||
8. **Regular audits**: Scan before each release
|
||||
|
||||
## Resources
|
||||
|
||||
### Tools
|
||||
- git-secrets: https://github.com/awslabs/git-secrets
|
||||
- truffleHog: https://github.com/trufflesecurity/trufflehog
|
||||
- detect-secrets: https://github.com/Yelp/detect-secrets
|
||||
- gitleaks: https://github.com/gitleaks/gitleaks
|
||||
|
||||
### Guides
|
||||
- GitHub: Removing sensitive data from a repository
|
||||
- OWASP: Secrets Management Cheat Sheet
|
||||
- Git: Rewriting History
|
||||
|
||||
---
|
||||
|
||||
**When this skill is active**, you will:
|
||||
1. Scan the codebase for private information patterns
|
||||
2. Check for hardcoded secrets and credentials
|
||||
3. Verify .gitignore includes sensitive files
|
||||
4. Review git history for exposed secrets
|
||||
5. Provide actionable recommendations
|
||||
6. Generate a comprehensive report
|
||||
7. Help clean up any found issues before open sourcing
|
||||
|
||||
@@ -0,0 +1,945 @@
|
||||
---
|
||||
name: ec2-backend-deployer
|
||||
description: Expert in deploying backends to EC2 instances using CI/CD pipelines, Docker containers, and GitHub Actions. This skill guides through the complete deployment workflow including Docker image building, container registry management, Tailscale integration, and automated deployment to EC2. Activates when users need to deploy backend services to EC2.
|
||||
---
|
||||
|
||||
# EC2 Backend Deployer
|
||||
|
||||
You are an expert in deploying backend applications to EC2 instances using CI/CD pipelines, Docker containers, and GitHub Actions. This skill provides comprehensive guidance for setting up automated deployments from GitHub to EC2, including Docker image building, container registry management, secure access via Tailscale, and service orchestration.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Setting up CI/CD for backend deployment to EC2
|
||||
- Configuring Docker-based deployments
|
||||
- Implementing automated deployment pipelines
|
||||
- Deploying NestJS, Next.js, or Express backends to EC2
|
||||
- Setting up container registries and image management
|
||||
- Configuring secure EC2 access for deployments
|
||||
- Implementing health checks and deployment verification
|
||||
- Setting up multi-service deployments with dependencies
|
||||
|
||||
## Project Context Discovery
|
||||
|
||||
**Before deploying, discover the project's context:**
|
||||
|
||||
1. **Identify Project Type:**
|
||||
- Scan for `package.json` to detect framework (NestJS, Next.js, Express)
|
||||
- Check for `nest-cli.json` (NestJS)
|
||||
- Check for `next.config.js` (Next.js)
|
||||
- Check for monorepo structure (workspaces)
|
||||
|
||||
2. **Check Existing Setup:**
|
||||
- Look for existing Dockerfiles
|
||||
- Check for docker-compose files
|
||||
- Review existing GitHub Actions workflows
|
||||
- Check for deployment scripts
|
||||
- Verify environment configuration files
|
||||
|
||||
3. **Identify Infrastructure:**
|
||||
- Check for EC2 instance details
|
||||
- Verify Tailscale setup (if using secure access)
|
||||
- Check for container registry configuration
|
||||
- Review security group and network setup
|
||||
|
||||
4. **Use Project-Specific Skills:**
|
||||
- Check for `[project]-ec2-backend-deployer` skill
|
||||
- Review project-specific deployment patterns
|
||||
- Follow project's infrastructure standards
|
||||
|
||||
## Docker Setup
|
||||
|
||||
### Multi-Stage Dockerfile Pattern
|
||||
|
||||
**Recommended structure for production deployments:**
|
||||
|
||||
```dockerfile
|
||||
# ==================================================
|
||||
# Stage 1: Base - Install dependencies
|
||||
# ==================================================
|
||||
FROM node:22.17.0 AS base
|
||||
|
||||
# Install bun (or use npm if preferred)
|
||||
RUN curl -fsSL https://bun.sh/install | bash && \
|
||||
cp /root/.bun/bin/bun /usr/local/bin/bun && \
|
||||
chmod +x /usr/local/bin/bun
|
||||
ENV PATH="/usr/local/bin:${PATH}"
|
||||
|
||||
# Set memory limits for builds
|
||||
ENV NODE_OPTIONS=--max-old-space-size=4096
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Copy package files
|
||||
COPY package.json package-lock.json* bun.lockb* ./
|
||||
COPY .npmrc ./
|
||||
|
||||
# For monorepos: create workspace structure
|
||||
RUN mkdir -p apps libs
|
||||
|
||||
# Install dependencies with secrets for private packages
|
||||
RUN --mount=type=secret,id=NPM_TOKEN \
|
||||
export NPM_TOKEN=$(cat /run/secrets/NPM_TOKEN 2>/dev/null || echo "") && \
|
||||
npm ci --frozen-lockfile || bun install --frozen-lockfile
|
||||
|
||||
# ==================================================
|
||||
# Stage 2: Builder - Build application
|
||||
# ==================================================
|
||||
FROM base AS builder
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Build application (with build secrets if needed)
|
||||
RUN --mount=type=secret,id=SENTRY_AUTH_TOKEN \
|
||||
export SENTRY_AUTH_TOKEN=$(cat /run/secrets/SENTRY_AUTH_TOKEN 2>/dev/null || echo "") && \
|
||||
npm run build:prod || bun run build:prod
|
||||
|
||||
# ==================================================
|
||||
# Stage 3: Production - Runtime image
|
||||
# ==================================================
|
||||
FROM node:22.17.0-slim AS production
|
||||
|
||||
# Install runtime dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
bash \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Copy built artifacts and production dependencies
|
||||
COPY --from=builder /usr/src/app/dist ./dist
|
||||
COPY --from=builder /usr/src/app/node_modules ./node_modules
|
||||
COPY --from=builder /usr/src/app/package.json ./package.json
|
||||
COPY --from=builder /usr/src/app/public ./public
|
||||
|
||||
# Create non-root user for security
|
||||
RUN groupadd -r appuser && useradd -r -g appuser -u 1001 appuser
|
||||
|
||||
# Set permissions
|
||||
RUN chown -R appuser:appuser /usr/src/app
|
||||
|
||||
# Switch to non-root user
|
||||
USER appuser
|
||||
|
||||
# Expose application port
|
||||
EXPOSE 3001
|
||||
|
||||
# Health check for container orchestration
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD curl -f http://localhost:3001/v1/health || exit 1
|
||||
|
||||
# Start application
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
**For NestJS specifically:**
|
||||
|
||||
```dockerfile
|
||||
# Use the pattern above but adjust:
|
||||
# - Build command: npm run build (creates dist/)
|
||||
# - Start command: node dist/main.js
|
||||
# - Health endpoint: /v1/health or /health
|
||||
```
|
||||
|
||||
**For Next.js API routes:**
|
||||
|
||||
```dockerfile
|
||||
# Adjust for Next.js:
|
||||
# - Build command: npm run build
|
||||
# - Start command: npm start
|
||||
# - Health endpoint: /api/health
|
||||
```
|
||||
|
||||
### Dockerfile Best Practices
|
||||
|
||||
- **Multi-stage builds**: Reduce final image size
|
||||
- **Non-root user**: Run containers as non-root for security
|
||||
- **Health checks**: Include HEALTHCHECK in Dockerfile
|
||||
- **Build secrets**: Use BuildKit secrets for sensitive data
|
||||
- **Layer caching**: Order COPY commands to maximize cache hits
|
||||
- **System dependencies**: Install only what's needed in production stage
|
||||
|
||||
## Container Registry Setup
|
||||
|
||||
### GitHub Container Registry (ghcr.io) - Recommended
|
||||
|
||||
**Advantages:**
|
||||
- Integrated with GitHub
|
||||
- Free for public repos, included with GitHub plans
|
||||
- Automatic authentication via GitHub tokens
|
||||
- Image versioning with tags
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Enable GitHub Container Registry:**
|
||||
- Go to repository Settings → Packages
|
||||
- Container registry is automatically enabled
|
||||
|
||||
2. **Image Naming Convention:**
|
||||
```
|
||||
ghcr.io/[owner]/[service-name]:[tag]
|
||||
```
|
||||
|
||||
3. **Image Tagging Strategy:**
|
||||
- `latest` - Most recent deployment
|
||||
- `production` - Production deployments
|
||||
- `[branch]-[sha]` - Branch and commit SHA
|
||||
- `[version]` - Semantic versioning
|
||||
|
||||
**Authentication in GitHub Actions:**
|
||||
|
||||
```yaml
|
||||
- name: Log in to Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
**Authentication on EC2:**
|
||||
|
||||
```bash
|
||||
# Login to registry
|
||||
echo "$GITHUB_TOKEN" | docker login ghcr.io -u USERNAME --password-stdin
|
||||
```
|
||||
|
||||
### AWS ECR (Alternative)
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Create ECR Repository:**
|
||||
```bash
|
||||
aws ecr create-repository --repository-name [service-name]
|
||||
```
|
||||
|
||||
2. **Get Login Token:**
|
||||
```bash
|
||||
aws ecr get-login-password --region [region] | \
|
||||
docker login --username AWS --password-stdin [account-id].dkr.ecr.[region].amazonaws.com
|
||||
```
|
||||
|
||||
3. **Push Image:**
|
||||
```bash
|
||||
docker tag [image]:[tag] [account-id].dkr.ecr.[region].amazonaws.com/[service-name]:[tag]
|
||||
docker push [account-id].dkr.ecr.[region].amazonaws.com/[service-name]:[tag]
|
||||
```
|
||||
|
||||
### Docker Hub (Alternative)
|
||||
|
||||
**Setup:**
|
||||
|
||||
```yaml
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
```
|
||||
|
||||
## CI/CD Pipeline (GitHub Actions)
|
||||
|
||||
### Main Deployment Workflow
|
||||
|
||||
**File:** `.github/workflows/deploy-production.yml`
|
||||
|
||||
```yaml
|
||||
name: Deploy Production
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'apps/**'
|
||||
- 'libs/**'
|
||||
- 'package.json'
|
||||
- 'Dockerfile*'
|
||||
- 'docker/**'
|
||||
workflow_dispatch:
|
||||
branches: [master]
|
||||
inputs:
|
||||
skip_tests:
|
||||
description: 'Skip pre-deployment tests'
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_PREFIX: ${{ github.repository_owner }}/[service-name]
|
||||
|
||||
jobs:
|
||||
# Branch safety check
|
||||
branch-check:
|
||||
name: Branch Safety Check
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Verify master branch
|
||||
run: |
|
||||
if [ "${{ github.ref_name }}" != "master" ]; then
|
||||
echo "❌ ERROR: Production deployment can only run from master branch"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pre-deployment checks
|
||||
pre-deployment-checks:
|
||||
name: Pre-Deployment Checks
|
||||
runs-on: ubuntu-latest
|
||||
needs: [branch-check]
|
||||
if: inputs.skip_tests != true
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Run linter
|
||||
run: npm run lint
|
||||
continue-on-error: true
|
||||
|
||||
- name: Run tests
|
||||
run: npm test
|
||||
continue-on-error: true
|
||||
|
||||
# Build and push image
|
||||
build-image:
|
||||
name: Build and Push Image
|
||||
needs: [branch-check, pre-deployment-checks]
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Log in to Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}
|
||||
tags: |
|
||||
type=sha,prefix={{branch}}-
|
||||
type=raw,value=latest
|
||||
type=raw,value=production
|
||||
|
||||
- name: Build and push image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:buildcache
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:buildcache,mode=max
|
||||
secrets: |
|
||||
NPM_TOKEN=${{ secrets.NPM_TOKEN }}
|
||||
SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }}
|
||||
platforms: linux/amd64
|
||||
|
||||
# Deploy to EC2
|
||||
deploy:
|
||||
name: Deploy to EC2
|
||||
needs: [build-image]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
|
||||
- name: Verify Tailscale connectivity
|
||||
run: |
|
||||
echo "⏳ Waiting for Tailscale to connect..."
|
||||
timeout 30 bash -c 'until tailscale status >/dev/null 2>&1; do sleep 1; done'
|
||||
echo "✅ Tailscale connected"
|
||||
|
||||
- name: Deploy to instance
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ vars.TAILSCALE_INSTANCE_IP || secrets.EC2_IP }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
port: 22
|
||||
script: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "🔐 Logging into container registry..."
|
||||
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
|
||||
|
||||
cd ~/[project-path] || mkdir -p ~/[project-path] && cd ~/[project-path]
|
||||
|
||||
# Verify Docker Compose v2
|
||||
if ! docker compose version &>/dev/null; then
|
||||
echo "❌ ERROR: Docker Compose v2 is not installed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Update docker-compose file
|
||||
cat > docker-compose.yml << 'EOF'
|
||||
version: '3.8'
|
||||
services:
|
||||
api:
|
||||
image: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:latest
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '3001:3001'
|
||||
env_file:
|
||||
- .env.production
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
EOF
|
||||
|
||||
# Pull latest images
|
||||
docker compose pull
|
||||
|
||||
# Deploy
|
||||
docker compose up -d --force-recreate
|
||||
|
||||
# Wait for health check
|
||||
echo "🏥 Waiting for service to be healthy..."
|
||||
sleep 10
|
||||
for i in {1..60}; do
|
||||
STATUS=$(docker inspect --format='{{.State.Health.Status}}' [container-name] 2>/dev/null || echo "none")
|
||||
if [ "$STATUS" = "healthy" ]; then
|
||||
echo "✅ Service is healthy"
|
||||
break
|
||||
fi
|
||||
if [ $i -eq 60 ]; then
|
||||
echo "❌ Service failed to become healthy"
|
||||
docker compose logs --tail=50
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting for service... ($i/60) [status: $STATUS]"
|
||||
sleep 3
|
||||
done
|
||||
|
||||
echo "✅ Deployment complete!"
|
||||
```
|
||||
|
||||
### Reusable Deployment Workflow
|
||||
|
||||
**File:** `.github/workflows/_deploy-service.yml`
|
||||
|
||||
For projects with multiple services, create a reusable workflow:
|
||||
|
||||
```yaml
|
||||
name: Deploy Service (Reusable)
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
service_name:
|
||||
required: true
|
||||
type: string
|
||||
instance_ip:
|
||||
required: true
|
||||
type: string
|
||||
docker_compose_file:
|
||||
required: true
|
||||
type: string
|
||||
health_check_services:
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
secrets:
|
||||
TAILSCALE_CLIENT_ID:
|
||||
required: true
|
||||
TAILSCALE_CLIENT_SECRET:
|
||||
required: true
|
||||
EC2_USER:
|
||||
required: true
|
||||
EC2_SSH_KEY:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
|
||||
- name: Deploy to instance
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ inputs.instance_ip }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
script: |
|
||||
# Deployment script (same as above)
|
||||
```
|
||||
|
||||
## EC2 Deployment Process
|
||||
|
||||
### Tailscale Integration (Recommended)
|
||||
|
||||
**Why Tailscale:**
|
||||
- Secure access without public IPs
|
||||
- No need to manage security groups for SSH
|
||||
- Easy connectivity from CI/CD runners
|
||||
- Works across networks
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Install Tailscale on EC2:**
|
||||
```bash
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
sudo tailscale up
|
||||
```
|
||||
|
||||
2. **Get Tailscale IP:**
|
||||
```bash
|
||||
tailscale ip -4
|
||||
```
|
||||
|
||||
3. **Configure GitHub Secrets:**
|
||||
- `TAILSCALE_CLIENT_ID` - OAuth client ID from Tailscale
|
||||
- `TAILSCALE_CLIENT_SECRET` - OAuth client secret
|
||||
- `TAILSCALE_INSTANCE_IP` - Tailscale IP of EC2 instance
|
||||
|
||||
4. **Use in GitHub Actions:**
|
||||
```yaml
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
```
|
||||
|
||||
### SSH Configuration (Alternative)
|
||||
|
||||
If not using Tailscale, use direct SSH:
|
||||
|
||||
**GitHub Secrets Required:**
|
||||
- `EC2_USER` - SSH username (e.g., `ubuntu`, `ec2-user`)
|
||||
- `EC2_SSH_KEY` - Private SSH key
|
||||
- `EC2_IP` - Public IP or hostname
|
||||
|
||||
**Security Group Configuration:**
|
||||
- Allow SSH (port 22) from GitHub Actions IPs
|
||||
- Or use a bastion host for additional security
|
||||
|
||||
### Docker Compose Deployment
|
||||
|
||||
**Requirements on EC2:**
|
||||
- Docker installed
|
||||
- Docker Compose v2 (not v1)
|
||||
- Sufficient disk space
|
||||
- Network access to container registry
|
||||
|
||||
**Deployment Steps:**
|
||||
|
||||
1. **SSH to EC2 instance**
|
||||
|
||||
2. **Login to container registry:**
|
||||
```bash
|
||||
echo "$GITHUB_TOKEN" | docker login ghcr.io -u USERNAME --password-stdin
|
||||
```
|
||||
|
||||
3. **Create/update docker-compose.yml:**
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
api:
|
||||
image: ghcr.io/owner/service:latest
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '3001:3001'
|
||||
env_file:
|
||||
- .env.production
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
```
|
||||
|
||||
4. **Pull latest images:**
|
||||
```bash
|
||||
docker compose pull
|
||||
```
|
||||
|
||||
5. **Deploy services:**
|
||||
```bash
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
6. **Verify health:**
|
||||
```bash
|
||||
docker compose ps
|
||||
docker inspect --format='{{.State.Health.Status}}' [container-name]
|
||||
```
|
||||
|
||||
### Multi-Service Deployment
|
||||
|
||||
**Deployment Order:**
|
||||
1. Dependencies first (Redis, databases)
|
||||
2. Independent services
|
||||
3. Dependent services (API)
|
||||
|
||||
**Example with Redis:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '6379:6379'
|
||||
healthcheck:
|
||||
test: ['CMD', 'redis-cli', 'ping']
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
api:
|
||||
image: ghcr.io/owner/api:latest
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### GitHub Secrets Management
|
||||
|
||||
**Required Secrets:**
|
||||
- `TAILSCALE_CLIENT_ID` / `TAILSCALE_CLIENT_SECRET` - For Tailscale access
|
||||
- `EC2_USER` / `EC2_SSH_KEY` - For SSH access (if not using Tailscale)
|
||||
- `NPM_TOKEN` - For private npm packages
|
||||
- `SENTRY_AUTH_TOKEN` - For Sentry source maps (if using)
|
||||
- `GITHUB_TOKEN` - Automatically provided, for registry access
|
||||
|
||||
**Setting Secrets:**
|
||||
1. Go to repository Settings → Secrets and variables → Actions
|
||||
2. Click "New repository secret"
|
||||
3. Add each secret with appropriate values
|
||||
|
||||
### Build Secrets in Docker
|
||||
|
||||
**Use BuildKit secrets for sensitive build-time data:**
|
||||
|
||||
```dockerfile
|
||||
RUN --mount=type=secret,id=NPM_TOKEN \
|
||||
export NPM_TOKEN=$(cat /run/secrets/NPM_TOKEN 2>/dev/null || echo "") && \
|
||||
npm ci
|
||||
```
|
||||
|
||||
**In GitHub Actions:**
|
||||
```yaml
|
||||
secrets: |
|
||||
NPM_TOKEN=${{ secrets.NPM_TOKEN }}
|
||||
SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }}
|
||||
```
|
||||
|
||||
### Container Security
|
||||
|
||||
- **Non-root user**: Always run containers as non-root
|
||||
- **Minimal base images**: Use `-slim` or `-alpine` variants
|
||||
- **No secrets in images**: Use environment variables or secrets management
|
||||
- **Health checks**: Enable health checks for monitoring
|
||||
- **Resource limits**: Set memory and CPU limits in docker-compose
|
||||
|
||||
### Network Security
|
||||
|
||||
- **Use Tailscale**: Avoid exposing services to public internet
|
||||
- **Security groups**: Restrict access to necessary ports only
|
||||
- **VPC**: Use private subnets when possible
|
||||
- **SSL/TLS**: Use HTTPS for all external-facing services
|
||||
|
||||
## Monitoring and Health Checks
|
||||
|
||||
### Health Check Endpoints
|
||||
|
||||
**Implement health check endpoint in your application:**
|
||||
|
||||
**NestJS:**
|
||||
```typescript
|
||||
// health.controller.ts
|
||||
@Controller('v1/health')
|
||||
export class HealthController {
|
||||
@Get()
|
||||
health() {
|
||||
return { status: 'ok', timestamp: new Date().toISOString() };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Express:**
|
||||
```typescript
|
||||
app.get('/v1/health', (req, res) => {
|
||||
res.json({ status: 'ok', timestamp: new Date().toISOString() });
|
||||
});
|
||||
```
|
||||
|
||||
### Docker Health Checks
|
||||
|
||||
**In Dockerfile:**
|
||||
```dockerfile
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD curl -f http://localhost:3001/v1/health || exit 1
|
||||
```
|
||||
|
||||
**In docker-compose:**
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
```
|
||||
|
||||
### Deployment Verification
|
||||
|
||||
**Check service health after deployment:**
|
||||
```bash
|
||||
# Check container status
|
||||
docker compose ps
|
||||
|
||||
# Check health status
|
||||
docker inspect --format='{{.State.Health.Status}}' [container-name]
|
||||
|
||||
# Check logs
|
||||
docker compose logs --tail=100 [service-name]
|
||||
|
||||
# Test health endpoint
|
||||
curl http://localhost:3001/v1/health
|
||||
```
|
||||
|
||||
### Post-Deployment Verification
|
||||
|
||||
**In GitHub Actions:**
|
||||
```yaml
|
||||
- name: Verify deployment
|
||||
run: |
|
||||
API_IP="${{ vars.TAILSCALE_INSTANCE_IP }}"
|
||||
for i in {1..10}; do
|
||||
if curl -f --max-time 10 http://${API_IP}:3001/v1/health; then
|
||||
echo "✅ Service is healthy"
|
||||
exit 0
|
||||
fi
|
||||
echo "Attempt $i/10 failed, retrying..."
|
||||
sleep 5
|
||||
done
|
||||
echo "❌ Service health check failed"
|
||||
exit 1
|
||||
```
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### Manual Rollback
|
||||
|
||||
**1. Find previous image tag:**
|
||||
- Check GitHub Container Registry
|
||||
- Look for tags like `master-<commit-sha>`
|
||||
- Or use semantic version tags
|
||||
|
||||
**2. Update docker-compose.yml:**
|
||||
```yaml
|
||||
services:
|
||||
api:
|
||||
image: ghcr.io/owner/service:master-abc123 # Previous tag
|
||||
```
|
||||
|
||||
**3. Deploy previous version:**
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
**4. Verify rollback:**
|
||||
```bash
|
||||
docker compose ps
|
||||
curl http://localhost:3001/v1/health
|
||||
```
|
||||
|
||||
### Automated Rollback Workflow
|
||||
|
||||
**Create `.github/workflows/rollback.yml`:**
|
||||
|
||||
```yaml
|
||||
name: Rollback Deployment
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
image_tag:
|
||||
description: 'Image tag to rollback to (e.g., master-abc123)'
|
||||
required: true
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
rollback:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
|
||||
- name: Rollback to previous version
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ vars.TAILSCALE_INSTANCE_IP }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
script: |
|
||||
cd ~/[project-path]
|
||||
# Update image tag in docker-compose.yml
|
||||
sed -i "s|image:.*|image: ghcr.io/owner/service:${{ inputs.image_tag }}|" docker-compose.yml
|
||||
docker compose pull
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
## Docker Cleanup and Maintenance
|
||||
|
||||
### Docker Prune Script
|
||||
|
||||
**Create `docker/prune-docker.sh`:**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Docker cleanup script for production servers
|
||||
|
||||
LOCK_FILE="/tmp/docker-prune.lock"
|
||||
LOCK_TIMEOUT=300 # 5 minutes max wait
|
||||
|
||||
echo "🧹 Starting Docker cleanup..."
|
||||
|
||||
# Acquire lock
|
||||
if [ -f "$LOCK_FILE" ]; then
|
||||
echo "⏳ Another prune operation is running, waiting..."
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
echo $$ > "$LOCK_FILE"
|
||||
trap "rm -f $LOCK_FILE" EXIT
|
||||
|
||||
# Prune old images (older than 24 hours)
|
||||
echo "🗑️ Pruning old Docker images..."
|
||||
docker image prune --filter "until=24h" -f
|
||||
|
||||
# Prune stopped containers
|
||||
echo "🗑️ Pruning stopped containers..."
|
||||
docker container prune --filter "until=24h" -f
|
||||
|
||||
# Prune unused networks
|
||||
echo "🗑️ Pruning unused networks..."
|
||||
docker network prune -f
|
||||
|
||||
# Show disk space
|
||||
echo "📊 Docker disk usage:"
|
||||
docker system df
|
||||
|
||||
echo "✅ Docker cleanup complete!"
|
||||
```
|
||||
|
||||
**Run cleanup after deployments:**
|
||||
```yaml
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
ssh -i ~/.ssh/key user@instance "cd ~/project && ./docker/prune-docker.sh"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**1. Docker Compose v2 not found:**
|
||||
```bash
|
||||
# Install Docker Compose v2
|
||||
sudo apt-get update
|
||||
sudo apt-get install docker-compose-plugin
|
||||
```
|
||||
|
||||
**2. Health check failures:**
|
||||
- Verify health endpoint is accessible
|
||||
- Check container logs: `docker compose logs [service]`
|
||||
- Ensure health check command is correct
|
||||
- Increase `start_period` for slow-starting services
|
||||
|
||||
**3. Image pull failures:**
|
||||
- Verify registry authentication
|
||||
- Check image tag exists
|
||||
- Verify network connectivity
|
||||
|
||||
**4. Deployment timeouts:**
|
||||
- Increase timeout in workflow
|
||||
- Check EC2 instance resources (CPU, memory)
|
||||
- Verify Tailscale connectivity
|
||||
|
||||
**5. Service not starting:**
|
||||
- Check environment variables
|
||||
- Verify dependencies are healthy
|
||||
- Review application logs
|
||||
- Check port conflicts
|
||||
|
||||
## Checklist
|
||||
|
||||
Before deploying, verify:
|
||||
|
||||
- [ ] Dockerfile is optimized (multi-stage build)
|
||||
- [ ] Health check endpoint implemented
|
||||
- [ ] Docker Compose v2 installed on EC2
|
||||
- [ ] Container registry configured
|
||||
- [ ] GitHub Secrets set up
|
||||
- [ ] Tailscale configured (or SSH access)
|
||||
- [ ] Environment variables configured
|
||||
- [ ] Health checks configured in docker-compose
|
||||
- [ ] Deployment workflow tested
|
||||
- [ ] Rollback procedure documented
|
||||
|
||||
## Next Steps
|
||||
|
||||
After initial deployment:
|
||||
|
||||
1. Set up monitoring and alerts
|
||||
2. Configure automatic cleanup
|
||||
3. Document deployment process
|
||||
4. Set up staging environment
|
||||
5. Implement blue-green deployments (optional)
|
||||
6. Configure log aggregation
|
||||
7. Set up backup procedures
|
||||
|
||||
@@ -0,0 +1,439 @@
|
||||
---
|
||||
name: mongodb-atlas-checker
|
||||
description: Expert in verifying MongoDB Atlas setup and configuration for backend applications. Checks connection strings, environment variables, database configuration, connection pooling, and ensures proper setup for Next.js and NestJS applications. This skill activates when users need to verify their MongoDB Atlas backend setup is correct.
|
||||
---
|
||||
|
||||
# MongoDB Atlas Checker
|
||||
|
||||
You are an expert in verifying MongoDB Atlas setup and configuration for backend applications. This skill helps identify configuration issues, missing environment variables, incorrect connection strings, and ensures proper database setup for Next.js and NestJS applications.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Verifying MongoDB Atlas backend setup
|
||||
- Checking if connection strings are correctly configured
|
||||
- Validating environment variable setup
|
||||
- Ensuring database connection is properly established
|
||||
- Reviewing MongoDB Atlas configuration
|
||||
- Troubleshooting database connection issues
|
||||
- Auditing database setup before deployment
|
||||
|
||||
## Project Context Discovery
|
||||
|
||||
**Before checking MongoDB Atlas setup, discover the project's context:**
|
||||
|
||||
1. **Scan Project Documentation:**
|
||||
- Check `.agent/SYSTEM/ARCHITECTURE.md` for database architecture
|
||||
- Review existing database patterns
|
||||
- Look for environment variable usage
|
||||
- Check for existing MongoDB integration
|
||||
|
||||
2. **Identify Framework:**
|
||||
- Determine if using Next.js (App Router or Pages Router)
|
||||
- Check if using NestJS backend
|
||||
- Review existing database connection patterns
|
||||
- Check for ORM/ODM usage (Mongoose, TypeORM, Prisma)
|
||||
|
||||
3. **Use Project-Specific Skills:**
|
||||
- Check for `[project]-mongodb-atlas-checker` skill
|
||||
- Review project-specific database patterns
|
||||
- Follow project's configuration standards
|
||||
|
||||
## Checklist: MongoDB Atlas Setup Verification
|
||||
|
||||
### 1. Environment Variables
|
||||
|
||||
**Check for required environment variables:**
|
||||
|
||||
```bash
|
||||
# Required for MongoDB Atlas
|
||||
MONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
# OR
|
||||
DATABASE_URL=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
**Verification Steps:**
|
||||
|
||||
- [ ] Environment variable exists (check `.env.local`, `.env`, or deployment config)
|
||||
- [ ] Variable name is consistent across codebase
|
||||
- [ ] Connection string uses `mongodb+srv://` protocol (required for Atlas)
|
||||
- [ ] Connection string includes authentication credentials
|
||||
- [ ] Connection string includes database name
|
||||
- [ ] Connection string includes query parameters (`retryWrites=true&w=majority`)
|
||||
- [ ] No hardcoded connection strings in source code
|
||||
- [ ] `.env.example` or `.env.template` has placeholder (not real credentials)
|
||||
|
||||
**Common Issues:**
|
||||
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded connection string
|
||||
const mongoUri = 'mongodb+srv://user:pass@cluster.mongodb.net/db';
|
||||
|
||||
// ❌ BAD: Wrong protocol (not supported by Atlas)
|
||||
const mongoUri = 'mongodb://user:pass@cluster.mongodb.net/db';
|
||||
|
||||
// ❌ BAD: Missing database name
|
||||
const mongoUri = 'mongodb+srv://user:pass@cluster.mongodb.net';
|
||||
|
||||
// ✅ GOOD: Environment variable
|
||||
const mongoUri = process.env.MONGODB_URI;
|
||||
```
|
||||
|
||||
### 2. Connection String Format
|
||||
|
||||
**MongoDB Atlas connection strings must:**
|
||||
- Use `mongodb+srv://` protocol (not `mongodb://`)
|
||||
- Include username and password
|
||||
- Include cluster hostname (e.g., `cluster0.xxxxx.mongodb.net`)
|
||||
- Include database name
|
||||
- Include query parameters for production readiness
|
||||
|
||||
**Valid Format:**
|
||||
```
|
||||
mongodb+srv://<username>:<password>@<cluster-host>/<database>?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
**Check for:**
|
||||
- [ ] Protocol is `mongodb+srv://`
|
||||
- [ ] Username and password are URL-encoded if they contain special characters
|
||||
- [ ] Cluster hostname is correct (from Atlas dashboard)
|
||||
- [ ] Database name is specified
|
||||
- [ ] Query parameters include `retryWrites=true&w=majority`
|
||||
- [ ] Optional: `appName` parameter for monitoring
|
||||
- [ ] Optional: `maxPoolSize` for connection pooling
|
||||
|
||||
**Example with all parameters:**
|
||||
```
|
||||
mongodb+srv://user:pass@cluster0.xxxxx.mongodb.net/mydb?retryWrites=true&w=majority&appName=MyApp&maxPoolSize=10
|
||||
```
|
||||
|
||||
### 3. Database Driver Installation
|
||||
|
||||
**Check if MongoDB driver is installed:**
|
||||
|
||||
**For Mongoose (ODM):**
|
||||
```bash
|
||||
# Check package.json
|
||||
npm list mongoose
|
||||
# or
|
||||
pnpm list mongoose
|
||||
```
|
||||
|
||||
**For Native MongoDB Driver:**
|
||||
```bash
|
||||
npm list mongodb
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `mongoose` or `mongodb` package is installed
|
||||
- [ ] Version is compatible with MongoDB Atlas
|
||||
- [ ] Package is listed in `package.json` dependencies (not devDependencies for production)
|
||||
|
||||
### 4. Connection Setup
|
||||
|
||||
**Next.js (App Router or Pages Router):**
|
||||
|
||||
**Check for proper connection pattern:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: Singleton pattern for Next.js
|
||||
// lib/mongodb.ts or utils/mongodb.ts
|
||||
import mongoose from 'mongoose';
|
||||
|
||||
const MONGODB_URI = process.env.MONGODB_URI!;
|
||||
|
||||
if (!MONGODB_URI) {
|
||||
throw new Error('Please define MONGODB_URI environment variable');
|
||||
}
|
||||
|
||||
interface MongooseCache {
|
||||
conn: typeof mongoose | null;
|
||||
promise: Promise<typeof mongoose> | null;
|
||||
}
|
||||
|
||||
declare global {
|
||||
var mongoose: MongooseCache | undefined;
|
||||
}
|
||||
|
||||
let cached: MongooseCache = global.mongoose || { conn: null, promise: null };
|
||||
|
||||
if (!global.mongoose) {
|
||||
global.mongoose = cached;
|
||||
}
|
||||
|
||||
async function connectDB() {
|
||||
if (cached.conn) {
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
if (!cached.promise) {
|
||||
const opts = {
|
||||
bufferCommands: false,
|
||||
};
|
||||
|
||||
cached.promise = mongoose.connect(MONGODB_URI, opts).then((mongoose) => {
|
||||
return mongoose;
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
cached.conn = await cached.promise;
|
||||
} catch (e) {
|
||||
cached.promise = null;
|
||||
throw e;
|
||||
}
|
||||
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
export default connectDB;
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Connection uses singleton pattern (prevents multiple connections in Next.js)
|
||||
- [ ] Connection is cached globally (for Next.js serverless functions)
|
||||
- [ ] Error handling is implemented
|
||||
- [ ] Connection options are configured (bufferCommands: false recommended)
|
||||
- [ ] Connection is called before database operations
|
||||
|
||||
**NestJS:**
|
||||
|
||||
**Check for MongooseModule configuration:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: NestJS MongooseModule
|
||||
// app.module.ts
|
||||
import { Module } from '@nestjs/common';
|
||||
import { MongooseModule } from '@nestjs/mongoose';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
MongooseModule.forRoot(process.env.MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
}),
|
||||
],
|
||||
})
|
||||
export class AppModule {}
|
||||
```
|
||||
|
||||
**Or with connection options:**
|
||||
|
||||
```typescript
|
||||
MongooseModule.forRoot(process.env.MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
maxPoolSize: 10,
|
||||
serverSelectionTimeoutMS: 5000,
|
||||
socketTimeoutMS: 45000,
|
||||
})
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `@nestjs/mongoose` package is installed
|
||||
- [ ] `MongooseModule.forRoot()` is configured in root module
|
||||
- [ ] Connection string comes from environment variable
|
||||
- [ ] Connection options are set appropriately
|
||||
- [ ] Error handling is in place
|
||||
|
||||
### 5. Connection Options
|
||||
|
||||
**Recommended connection options for MongoDB Atlas:**
|
||||
|
||||
```typescript
|
||||
{
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
maxPoolSize: 10, // Connection pool size
|
||||
serverSelectionTimeoutMS: 5000, // Timeout for server selection
|
||||
socketTimeoutMS: 45000, // Socket timeout
|
||||
connectTimeoutMS: 10000, // Connection timeout
|
||||
bufferCommands: false, // Disable mongoose buffering
|
||||
bufferMaxEntries: 0, // Disable mongoose buffering
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `retryWrites: true` is set (required for Atlas)
|
||||
- [ ] `w: 'majority'` is set (write concern)
|
||||
- [ ] Connection pool size is appropriate for your use case
|
||||
- [ ] Timeouts are configured appropriately
|
||||
- [ ] Buffer commands is disabled for serverless (Next.js)
|
||||
|
||||
### 6. Error Handling
|
||||
|
||||
**Check for proper error handling:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: Error handling
|
||||
try {
|
||||
await connectDB();
|
||||
// Database operations
|
||||
} catch (error) {
|
||||
console.error('MongoDB connection error:', error);
|
||||
// Handle error appropriately
|
||||
throw error;
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Connection errors are caught and handled
|
||||
- [ ] Error messages are logged appropriately
|
||||
- [ ] Application doesn't crash on connection failure
|
||||
- [ ] Retry logic is implemented if needed
|
||||
- [ ] Error handling is consistent across the codebase
|
||||
|
||||
### 7. Database Name Configuration
|
||||
|
||||
**Check if database name is correctly specified:**
|
||||
|
||||
- [ ] Database name is in connection string
|
||||
- [ ] Database name matches your application's needs
|
||||
- [ ] Database name doesn't contain special characters
|
||||
- [ ] Database name is consistent across environments (dev/staging/prod)
|
||||
|
||||
### 8. SSL/TLS Configuration
|
||||
|
||||
**MongoDB Atlas requires SSL/TLS by default:**
|
||||
|
||||
- [ ] Connection string doesn't explicitly disable SSL (Atlas requires it)
|
||||
- [ ] No `ssl=false` in connection string
|
||||
- [ ] TLS/SSL is enabled by default with `mongodb+srv://`
|
||||
|
||||
### 9. Network Access
|
||||
|
||||
**Check Atlas Network Access settings:**
|
||||
|
||||
- [ ] IP whitelist includes your deployment IPs
|
||||
- [ ] For development: `0.0.0.0/0` allows all IPs (not recommended for production)
|
||||
- [ ] For production: Specific IPs or VPC peering configured
|
||||
- [ ] Network access rules are documented
|
||||
|
||||
### 10. Database User Configuration
|
||||
|
||||
**Check Atlas Database User settings:**
|
||||
|
||||
- [ ] Database user exists in Atlas
|
||||
- [ ] User has appropriate permissions (read/write for application database)
|
||||
- [ ] Password is strong and secure
|
||||
- [ ] User credentials match connection string
|
||||
- [ ] User is not using admin credentials for application
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### Issue 1: Connection String Not Found
|
||||
|
||||
**Problem:** `MONGODB_URI` environment variable is missing
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Add to .env.local (Next.js) or .env (NestJS)
|
||||
MONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
### Issue 2: Wrong Protocol
|
||||
|
||||
**Problem:** Using `mongodb://` instead of `mongodb+srv://`
|
||||
|
||||
**Solution:** Change to `mongodb+srv://` (required for Atlas)
|
||||
|
||||
### Issue 3: Multiple Connections in Next.js
|
||||
|
||||
**Problem:** Creating new connection on each API call
|
||||
|
||||
**Solution:** Use singleton pattern to cache connection (see Connection Setup section)
|
||||
|
||||
### Issue 4: Connection Timeout
|
||||
|
||||
**Problem:** Connection times out
|
||||
|
||||
**Solution:**
|
||||
- Check network access in Atlas dashboard
|
||||
- Verify IP whitelist
|
||||
- Increase `connectTimeoutMS` and `serverSelectionTimeoutMS`
|
||||
- Check firewall settings
|
||||
|
||||
### Issue 5: Authentication Failed
|
||||
|
||||
**Problem:** Username/password incorrect
|
||||
|
||||
**Solution:**
|
||||
- Verify credentials in Atlas dashboard
|
||||
- Check if password contains special characters (needs URL encoding)
|
||||
- Verify database user exists and has permissions
|
||||
|
||||
## Verification Script
|
||||
|
||||
**Create a test script to verify connection:**
|
||||
|
||||
```typescript
|
||||
// scripts/test-mongodb-connection.ts
|
||||
import mongoose from 'mongoose';
|
||||
|
||||
const MONGODB_URI = process.env.MONGODB_URI;
|
||||
|
||||
if (!MONGODB_URI) {
|
||||
console.error('❌ MONGODB_URI environment variable is missing');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function testConnection() {
|
||||
try {
|
||||
await mongoose.connect(MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
});
|
||||
|
||||
console.log('✅ Successfully connected to MongoDB Atlas');
|
||||
|
||||
// Test a simple operation
|
||||
const collections = await mongoose.connection.db.listCollections().toArray();
|
||||
console.log(`✅ Found ${collections.length} collections`);
|
||||
|
||||
await mongoose.disconnect();
|
||||
console.log('✅ Connection closed');
|
||||
process.exit(0);
|
||||
} catch (error) {
|
||||
console.error('❌ MongoDB connection error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
testConnection();
|
||||
```
|
||||
|
||||
**Run the test:**
|
||||
```bash
|
||||
# Load environment variables and run
|
||||
node -r dotenv/config scripts/test-mongodb-connection.ts
|
||||
# or
|
||||
ts-node scripts/test-mongodb-connection.ts
|
||||
```
|
||||
|
||||
## Summary Checklist
|
||||
|
||||
Before considering MongoDB Atlas setup complete, verify:
|
||||
|
||||
- [ ] `MONGODB_URI` environment variable exists and is correct
|
||||
- [ ] Connection string uses `mongodb+srv://` protocol
|
||||
- [ ] Connection string includes database name
|
||||
- [ ] MongoDB driver (mongoose or mongodb) is installed
|
||||
- [ ] Connection setup follows framework best practices
|
||||
- [ ] Connection options are configured appropriately
|
||||
- [ ] Error handling is implemented
|
||||
- [ ] Network access is configured in Atlas
|
||||
- [ ] Database user has appropriate permissions
|
||||
- [ ] No hardcoded credentials in source code
|
||||
- [ ] Connection test script passes
|
||||
|
||||
## Next Steps
|
||||
|
||||
After verifying setup:
|
||||
1. Test connection with verification script
|
||||
2. Create initial database schema/models
|
||||
3. Set up database indexes
|
||||
4. Configure connection pooling for production
|
||||
5. Set up monitoring and alerts in Atlas dashboard
|
||||
6. Document connection setup in project documentation
|
||||
|
||||
@@ -0,0 +1,637 @@
|
||||
---
|
||||
name: open-source-checker
|
||||
description: Expert in detecting private information, secrets, API keys, credentials, and sensitive data in codebases before open sourcing. Scans for hardcoded secrets, API keys, passwords, private keys, personal information, database credentials, and other sensitive data that should not be exposed in public repositories. Can also set up git hooks and pre-commit hooks to prevent committing secrets.
|
||||
metadata:
|
||||
short-description: Check for private info before open sourcing
|
||||
---
|
||||
|
||||
# Open Source Checker
|
||||
|
||||
Expert in detecting private information, secrets, and sensitive data in codebases. Use when preparing to open source a repository or auditing for exposed secrets. Can set up automated checks via git hooks to prevent future issues.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Preparing to open source a repository
|
||||
- Reviewing code for exposed secrets
|
||||
- Auditing codebase for sensitive data
|
||||
- Checking for hardcoded credentials
|
||||
- Validating that no private information is committed
|
||||
- Reviewing pull requests for secrets
|
||||
- Performing security audits before public release
|
||||
|
||||
## What to Check For
|
||||
|
||||
### 1. API Keys and Tokens
|
||||
|
||||
**Common patterns:**
|
||||
- API keys (OpenAI, Stripe, AWS, Google, etc.)
|
||||
- Authentication tokens
|
||||
- OAuth tokens
|
||||
- JWT secrets
|
||||
- Session keys
|
||||
- Webhook secrets
|
||||
|
||||
**Patterns to detect:**
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded API keys
|
||||
const apiKey = 'sk-1234567890abcdef';
|
||||
const stripeKey = 'sk_live_...';
|
||||
const awsKey = 'AKIAIOSFODNN7EXAMPLE';
|
||||
|
||||
// ✅ GOOD: Environment variables
|
||||
const apiKey = process.env.API_KEY;
|
||||
const stripeKey = process.env.STRIPE_SECRET_KEY;
|
||||
```
|
||||
|
||||
**Common locations:**
|
||||
- Configuration files
|
||||
- Source code files
|
||||
- Environment files (`.env` files that might be committed)
|
||||
- Test files
|
||||
- Documentation files
|
||||
- Example files
|
||||
|
||||
### 2. Database Credentials
|
||||
|
||||
**Check for:**
|
||||
- Database connection strings
|
||||
- Usernames and passwords
|
||||
- MongoDB URIs
|
||||
- PostgreSQL connection strings
|
||||
- Redis credentials
|
||||
- Database host addresses
|
||||
|
||||
**Patterns:**
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded credentials
|
||||
const mongoUri = 'mongodb://user:password@host:27017/db';
|
||||
const dbPassword = 'mySecretPassword123';
|
||||
|
||||
// ✅ GOOD: Environment variables
|
||||
const mongoUri = process.env.MONGODB_URI;
|
||||
```
|
||||
|
||||
### 3. Private Keys and Certificates
|
||||
|
||||
**Check for:**
|
||||
- SSH private keys
|
||||
- SSL/TLS certificates
|
||||
- Private key files (`.pem`, `.key`, `.p12`)
|
||||
- Certificate files
|
||||
- Signing keys
|
||||
|
||||
**Files to check:**
|
||||
- `*.pem`, `*.key`, `*.p12`, `*.pfx`
|
||||
- `id_rsa`, `id_dsa`, `id_ecdsa`
|
||||
- `*.crt`, `*.cer`, `*.cert`
|
||||
|
||||
### 4. Personal Information
|
||||
|
||||
**Check for:**
|
||||
- Email addresses
|
||||
- Phone numbers
|
||||
- Physical addresses
|
||||
- Personal names
|
||||
- Social security numbers
|
||||
- Credit card numbers
|
||||
- Bank account numbers
|
||||
|
||||
**Patterns:**
|
||||
```typescript
|
||||
// ❌ BAD: Personal information
|
||||
const adminEmail = 'john.doe@example.com';
|
||||
const phone = '+1-555-123-4567';
|
||||
|
||||
// ✅ GOOD: Placeholder or environment variable
|
||||
const adminEmail = process.env.ADMIN_EMAIL;
|
||||
```
|
||||
|
||||
### 5. Environment Files
|
||||
|
||||
**Check for:**
|
||||
- `.env` files (should be in `.gitignore`)
|
||||
- `.env.local`, `.env.production`
|
||||
- Files containing actual secrets (not `.env.example`)
|
||||
|
||||
**Verify:**
|
||||
- `.env` is in `.gitignore`
|
||||
- Only `.env.example` is committed (with placeholder values)
|
||||
- No actual secrets in any committed `.env` files
|
||||
|
||||
## Scanning Workflow
|
||||
|
||||
### Phase 1: File System Scan
|
||||
|
||||
**1.1 Check for Common Secret Files**
|
||||
|
||||
```bash
|
||||
# Find potential secret files
|
||||
find . -name "*.env" -o -name "*.key" -o -name "*.pem" -o -name "id_rsa*"
|
||||
find . -name "secrets.*" -o -name "*secret*"
|
||||
find . -name ".env*" ! -name ".env.example"
|
||||
```
|
||||
|
||||
**1.2 Check .gitignore**
|
||||
|
||||
```bash
|
||||
# Verify .env is ignored
|
||||
cat .gitignore | grep -E "\.env|secrets|\.key|\.pem"
|
||||
```
|
||||
|
||||
**1.3 Scan for Common Patterns**
|
||||
|
||||
```bash
|
||||
# Search for API key patterns
|
||||
grep -r "sk-[a-zA-Z0-9]" --include="*.ts" --include="*.js" --include="*.json"
|
||||
grep -r "AKIA[0-9A-Z]" --include="*.ts" --include="*.js"
|
||||
grep -r "sk_live_" --include="*.ts" --include="*.js"
|
||||
```
|
||||
|
||||
### Phase 2: Code Pattern Analysis
|
||||
|
||||
**2.1 Search for Hardcoded Secrets**
|
||||
|
||||
Look for:
|
||||
- String literals that look like API keys
|
||||
- Hardcoded passwords
|
||||
- Connection strings with credentials
|
||||
- Token values in code
|
||||
|
||||
**2.2 Check Configuration Files**
|
||||
|
||||
Review:
|
||||
- All config files for hardcoded values
|
||||
- Environment variable usage (should use `process.env`)
|
||||
- Default values that might be secrets
|
||||
|
||||
**2.3 Review Test Files**
|
||||
|
||||
Check:
|
||||
- Test credentials (should be mocks, not real)
|
||||
- Test API keys (should be fake/test keys)
|
||||
- Test database connections
|
||||
|
||||
### Phase 3: Content Analysis
|
||||
|
||||
**3.1 Check Documentation**
|
||||
|
||||
- README files
|
||||
- Documentation files
|
||||
- Comments in code
|
||||
- Example code snippets
|
||||
|
||||
**3.2 Check Example Files**
|
||||
|
||||
- `.env.example` should have placeholders
|
||||
- Example configs should not have real values
|
||||
- Sample code should not include real keys
|
||||
|
||||
### Phase 4: Git History Check
|
||||
|
||||
**⚠️ CRITICAL: Secrets remain in git history even if removed from current files.**
|
||||
|
||||
**4.1 Comprehensive Git History Scan**
|
||||
|
||||
```bash
|
||||
# Search entire history for API keys
|
||||
git log --all --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
git log --all --full-history -p -S "AKIA" | grep -B 5 -A 5 "AKIA"
|
||||
git log --all --full-history -p -S "sk_live_" | grep -B 5 -A 5 "sk_live_"
|
||||
|
||||
# Search all branches and tags
|
||||
git log --all --branches --tags --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
|
||||
# Search deleted files
|
||||
git log --all --full-history --diff-filter=D --summary | grep -E "\.env|secrets"
|
||||
|
||||
# Search specific file types
|
||||
git log --all --full-history -p -- "*.env" | grep -E "(sk-|password|AKIA)"
|
||||
```
|
||||
|
||||
**4.2 Using Tools for Git History**
|
||||
|
||||
**gitleaks (Recommended):**
|
||||
|
||||
```bash
|
||||
# Scan entire git history
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
|
||||
# Scan specific branch
|
||||
gitleaks detect --source . --verbose --log-opts="--all --branches=main"
|
||||
```
|
||||
|
||||
**truffleHog:**
|
||||
|
||||
```bash
|
||||
# Scan entire git history
|
||||
trufflehog --regex --entropy=False git file://.
|
||||
```
|
||||
|
||||
**git-secrets:**
|
||||
|
||||
```bash
|
||||
# Scan entire history
|
||||
git secrets --scan-history
|
||||
```
|
||||
|
||||
**4.3 Cleaning Git History (If Secrets Found)**
|
||||
|
||||
**⚠️ WARNING: Rewrites history. Coordinate with team first.**
|
||||
|
||||
**Using git-filter-repo:**
|
||||
|
||||
```bash
|
||||
pip install git-filter-repo
|
||||
git filter-repo --invert-paths --path-glob "*.env"
|
||||
git filter-repo --replace-text <(echo "sk-OLD-KEY==>sk-REMOVED")
|
||||
```
|
||||
|
||||
**Using BFG Repo-Cleaner:**
|
||||
|
||||
```bash
|
||||
brew install bfg
|
||||
bfg --delete-files secrets.json
|
||||
echo "sk-OLD-KEY==>sk-REMOVED" > replacements.txt
|
||||
bfg --replace-text replacements.txt
|
||||
```
|
||||
|
||||
**Fresh Repository (if history too contaminated):**
|
||||
|
||||
```bash
|
||||
git checkout --orphan fresh-start
|
||||
git add .
|
||||
git commit -m "Initial commit (cleaned history)"
|
||||
git branch -D main
|
||||
git branch -m main
|
||||
git push -f origin main # Coordinate with team!
|
||||
```
|
||||
|
||||
**4.4 Verify History is Clean**
|
||||
|
||||
```bash
|
||||
# Re-scan after cleaning
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
git log --all --full-history -p -S "sk-" | grep "sk-"
|
||||
```
|
||||
|
||||
## Common Patterns to Detect
|
||||
|
||||
### API Key Patterns
|
||||
|
||||
```typescript
|
||||
// OpenAI
|
||||
sk-[a-zA-Z0-9]{32,}
|
||||
|
||||
// AWS
|
||||
AKIA[0-9A-Z]{16}
|
||||
|
||||
// Stripe
|
||||
sk_live_[a-zA-Z0-9]{24,}
|
||||
sk_test_[a-zA-Z0-9]{24,}
|
||||
|
||||
// GitHub
|
||||
ghp_[a-zA-Z0-9]{36}
|
||||
|
||||
// Generic
|
||||
[a-zA-Z0-9_-]{20,} // Long alphanumeric strings
|
||||
```
|
||||
|
||||
### Password Patterns
|
||||
|
||||
```typescript
|
||||
// Common patterns
|
||||
password\s*[:=]\s*['"][^'"]+['"]
|
||||
pwd\s*[:=]\s*['"][^'"]+['"]
|
||||
pass\s*[:=]\s*['"][^'"]+['"]
|
||||
```
|
||||
|
||||
### Connection String Patterns
|
||||
|
||||
```typescript
|
||||
// MongoDB
|
||||
mongodb://[^:]+:[^@]+@
|
||||
mongodb\+srv://[^:]+:[^@]+@
|
||||
|
||||
// PostgreSQL
|
||||
postgres://[^:]+:[^@]+@
|
||||
postgresql://[^:]+:[^@]+@
|
||||
|
||||
// MySQL
|
||||
mysql://[^:]+:[^@]+@
|
||||
|
||||
// Redis
|
||||
redis://[^:]+:[^@]+@
|
||||
```
|
||||
|
||||
### Email Patterns
|
||||
|
||||
```typescript
|
||||
// Email addresses
|
||||
[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
|
||||
```
|
||||
|
||||
## Automated Tools
|
||||
|
||||
### Recommended Tools
|
||||
|
||||
**1. git-secrets**
|
||||
```bash
|
||||
# Install
|
||||
brew install git-secrets
|
||||
|
||||
# Setup
|
||||
git secrets --install
|
||||
git secrets --register-aws
|
||||
|
||||
# Scan
|
||||
git secrets --scan
|
||||
```
|
||||
|
||||
**2. truffleHog**
|
||||
```bash
|
||||
# Install
|
||||
pip install truffleHog
|
||||
|
||||
# Scan
|
||||
trufflehog --regex --entropy=False .
|
||||
```
|
||||
|
||||
**3. detect-secrets**
|
||||
```bash
|
||||
# Install
|
||||
pip install detect-secrets
|
||||
|
||||
# Scan
|
||||
detect-secrets scan --all-files
|
||||
```
|
||||
|
||||
**4. gitleaks**
|
||||
```bash
|
||||
# Install
|
||||
brew install gitleaks
|
||||
|
||||
# Scan
|
||||
gitleaks detect --source . --verbose
|
||||
```
|
||||
|
||||
## Git Hooks and Pre-Commit Hooks
|
||||
|
||||
Set up git hooks to prevent committing secrets before they enter the repository.
|
||||
|
||||
### Pre-Commit Hook Setup
|
||||
|
||||
**1. Using git-secrets**
|
||||
|
||||
```bash
|
||||
# Install and initialize
|
||||
brew install git-secrets
|
||||
cd /path/to/your/repo
|
||||
git secrets --install
|
||||
git secrets --register-aws
|
||||
|
||||
# Add custom patterns
|
||||
git secrets --add 'sk-[a-zA-Z0-9]{32,}'
|
||||
git secrets --add 'AKIA[0-9A-Z]{16}'
|
||||
```
|
||||
|
||||
**2. Using gitleaks**
|
||||
|
||||
```bash
|
||||
# Install and create hook
|
||||
brew install gitleaks
|
||||
cat > .git/hooks/pre-commit << 'EOF'
|
||||
#!/bin/sh
|
||||
gitleaks detect --staged --verbose
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "❌ gitleaks detected secrets. Commit aborted."
|
||||
exit 1
|
||||
fi
|
||||
EOF
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
**3. Using detect-secrets**
|
||||
|
||||
```bash
|
||||
# Install and create baseline
|
||||
pip install detect-secrets
|
||||
detect-secrets scan > .secrets.baseline
|
||||
|
||||
# Create hook
|
||||
cat > .git/hooks/pre-commit << 'EOF'
|
||||
#!/bin/sh
|
||||
detect-secrets scan --baseline .secrets.baseline
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "❌ New secrets detected. Commit aborted."
|
||||
exit 1
|
||||
fi
|
||||
EOF
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
**4. Using Husky (Node.js)**
|
||||
|
||||
```bash
|
||||
npm install --save-dev husky
|
||||
npx husky install
|
||||
npx husky add .husky/pre-commit "gitleaks detect --staged --verbose"
|
||||
```
|
||||
|
||||
**5. Manual Pre-Commit Hook**
|
||||
|
||||
Create `.git/hooks/pre-commit`:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
# Check for API keys
|
||||
if git diff --cached --name-only | xargs grep -E "(sk-[a-zA-Z0-9]{32,}|AKIA[0-9A-Z]{16})" 2>/dev/null; then
|
||||
echo "❌ API keys detected!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for .env files
|
||||
if git diff --cached --name-only | grep -E "\.env$" | grep -v "\.env\.example"; then
|
||||
echo "❌ .env file detected!"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exit 0
|
||||
```
|
||||
|
||||
Make executable: `chmod +x .git/hooks/pre-commit`
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
**GitHub Actions:**
|
||||
|
||||
```yaml
|
||||
name: Secret Scanning
|
||||
on: [push, pull_request]
|
||||
jobs:
|
||||
scan:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: gitleaks/gitleaks-action@v2
|
||||
```
|
||||
|
||||
## Checklist
|
||||
|
||||
### Before Open Sourcing
|
||||
|
||||
- [ ] No hardcoded API keys in code
|
||||
- [ ] No database credentials in code
|
||||
- [ ] No private keys or certificates committed
|
||||
- [ ] `.env` files in `.gitignore`
|
||||
- [ ] Only `.env.example` committed (with placeholders)
|
||||
- [ ] **Git history scanned for secrets (CRITICAL)**
|
||||
- [ ] All branches checked for secrets
|
||||
- [ ] All tags checked for secrets
|
||||
- [ ] Deleted files checked for secrets
|
||||
- [ ] Merge commits checked for secrets
|
||||
- [ ] No secrets found in git history
|
||||
- [ ] Git history cleaned if secrets were found
|
||||
- [ ] No personal information in code
|
||||
- [ ] No real credentials in test files
|
||||
- [ ] No secrets in documentation
|
||||
- [ ] Configuration files use environment variables
|
||||
- [ ] All sensitive files in `.gitignore`
|
||||
|
||||
### Files to Verify
|
||||
|
||||
- [ ] `.env` - Should be ignored
|
||||
- [ ] `.env.local` - Should be ignored
|
||||
- [ ] `.env.production` - Should be ignored
|
||||
- [ ] `config.json` - Should not contain secrets
|
||||
- [ ] `secrets.json` - Should not exist or be ignored
|
||||
- [ ] `*.key`, `*.pem` - Should be ignored
|
||||
- [ ] `id_rsa*` - Should be ignored
|
||||
- [ ] README.md - Should not contain real secrets
|
||||
- [ ] Documentation files - Should not contain secrets
|
||||
|
||||
## Output Format
|
||||
|
||||
When checking for private information:
|
||||
|
||||
```
|
||||
🔍 PRIVATE INFORMATION SCAN REPORT
|
||||
|
||||
Repository: [repo-name]
|
||||
Date: [date]
|
||||
Scanner: [tool/agent]
|
||||
|
||||
📊 SUMMARY
|
||||
- Critical issues: 3
|
||||
- Warnings: 5
|
||||
- Files scanned: 150
|
||||
- Patterns checked: 12
|
||||
- Git history scanned: Yes
|
||||
- Branches checked: 5
|
||||
- Commits in history: 1,234
|
||||
|
||||
🚨 CRITICAL ISSUES
|
||||
|
||||
1. Hardcoded API Key Found
|
||||
File: src/config/api.ts:23
|
||||
Line: const apiKey = 'sk-1234567890abcdef';
|
||||
Issue: OpenAI API key exposed in code
|
||||
Fix: Move to environment variable
|
||||
Severity: CRITICAL
|
||||
Action: Remove immediately and rotate key
|
||||
|
||||
2. Database Credentials in Code
|
||||
File: src/database/config.ts:12
|
||||
Line: const mongoUri = 'mongodb://user:password@host:27017/db';
|
||||
Issue: Database credentials exposed
|
||||
Fix: Use environment variable
|
||||
Severity: CRITICAL
|
||||
Action: Remove and change database password
|
||||
|
||||
3. Secrets Found in Git History
|
||||
Commit: abc123def (2024-01-15)
|
||||
File: config/secrets.json (now deleted)
|
||||
Issue: API key was committed and then deleted, but still in history
|
||||
Fix: Clean git history using git-filter-repo
|
||||
Severity: CRITICAL
|
||||
Action: Remove from history and rotate exposed keys
|
||||
|
||||
⚠️ WARNINGS
|
||||
|
||||
1. .env File Not in .gitignore
|
||||
File: .env
|
||||
Issue: Environment file may be committed
|
||||
Fix: Add .env to .gitignore
|
||||
Severity: HIGH
|
||||
|
||||
[... more issues ...]
|
||||
|
||||
✅ SAFE FILES
|
||||
|
||||
- ✅ .env.example contains only placeholders
|
||||
- ✅ All config files use environment variables
|
||||
- ✅ No secrets in documentation
|
||||
- ✅ Test files use mock credentials
|
||||
|
||||
📜 GIT HISTORY SCAN RESULTS
|
||||
|
||||
- ✅ Current files: No secrets detected
|
||||
- ⚠️ Git history: 2 secrets found in old commits
|
||||
- ✅ All branches scanned: main, develop, feature/*
|
||||
- ✅ All tags scanned: v1.0.0, v1.1.0
|
||||
- ⚠️ Action required: Clean git history before open sourcing
|
||||
|
||||
💡 RECOMMENDATIONS
|
||||
|
||||
1. Add .env to .gitignore if not already
|
||||
2. Use environment variables for all secrets
|
||||
3. Rotate any exposed API keys
|
||||
4. Clean git history if secrets were committed
|
||||
5. Set up pre-commit hooks to prevent future commits
|
||||
6. Use secret scanning in CI/CD
|
||||
|
||||
📋 NEXT STEPS
|
||||
|
||||
1. Fix critical issues immediately
|
||||
2. Rotate any exposed credentials
|
||||
3. Clean git history if needed
|
||||
4. Set up automated scanning
|
||||
5. Review and approve before open sourcing
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Never commit secrets**: Always use environment variables
|
||||
2. **Use .env.example**: Provide template with placeholders
|
||||
3. **Rotate exposed secrets**: If secrets were committed, rotate them
|
||||
4. **Clean git history**: Remove secrets from history if committed
|
||||
5. **Automate scanning**: Use pre-commit hooks and CI/CD checks
|
||||
6. **Document requirements**: List required environment variables
|
||||
7. **Use secret management**: Consider services like AWS Secrets Manager
|
||||
8. **Regular audits**: Scan before each release
|
||||
|
||||
## Resources
|
||||
|
||||
### Tools
|
||||
- git-secrets: https://github.com/awslabs/git-secrets
|
||||
- truffleHog: https://github.com/trufflesecurity/trufflehog
|
||||
- detect-secrets: https://github.com/Yelp/detect-secrets
|
||||
- gitleaks: https://github.com/gitleaks/gitleaks
|
||||
|
||||
### Guides
|
||||
- GitHub: Removing sensitive data from a repository
|
||||
- OWASP: Secrets Management Cheat Sheet
|
||||
- Git: Rewriting History
|
||||
|
||||
---
|
||||
|
||||
**When this skill is active**, you will:
|
||||
1. Scan the codebase for private information patterns
|
||||
2. Check for hardcoded secrets and credentials
|
||||
3. Verify .gitignore includes sensitive files
|
||||
4. Review git history for exposed secrets
|
||||
5. Provide actionable recommendations
|
||||
6. Generate a comprehensive report
|
||||
7. Help clean up any found issues before open sourcing
|
||||
|
||||
@@ -0,0 +1,554 @@
|
||||
# Co-Founder
|
||||
|
||||
**Purpose:** Act as a strategic business partner to an indie developer, providing data-driven advice on business growth, product strategy, competitive positioning, and tech trends. Reads project context from sessions, tasks, and business conversations to provide informed guidance.
|
||||
|
||||
## When to Use This Command
|
||||
|
||||
- Planning strategic initiatives or major decisions
|
||||
- Analyzing business metrics and growth opportunities
|
||||
- Evaluating feature prioritization and product roadmap
|
||||
- Discussing pricing, positioning, or go-to-market strategy
|
||||
- Reviewing weekly/monthly business performance
|
||||
- Challenging assumptions or seeking alternative perspectives
|
||||
- Staying current with AI/tech trends and competitive landscape
|
||||
- Optimizing resource allocation and development priorities
|
||||
|
||||
## Co-Founder Personas
|
||||
|
||||
Adapt your response style based on the question type and context:
|
||||
|
||||
### 1. Strategic Partner (Default)
|
||||
|
||||
- **When:** General business questions, roadmap planning, high-level decisions
|
||||
- **Tone:** Collaborative, thoughtful, forward-thinking
|
||||
- **Depth:** Balanced - actionable insights with supporting rationale
|
||||
- **Format:** 3-5 paragraphs with clear recommendations
|
||||
|
||||
### 2. Data Analyst
|
||||
|
||||
- **When:** Metrics-focused questions, performance reviews, tracking requests
|
||||
- **Tone:** Analytical, precise, evidence-based
|
||||
- **Depth:** Deep - comprehensive data analysis with visualizations when helpful
|
||||
- **Format:** Structured reports with data points, trends, and recommendations
|
||||
|
||||
### 3. Devil's Advocate
|
||||
|
||||
- **When:** User asks to be challenged, presents new idea, or requests critique
|
||||
- **Tone:** Constructively critical, probing, risk-aware
|
||||
- **Depth:** Thorough - examine assumptions, risks, alternatives
|
||||
- **Format:** Question-driven with counterpoints and alternative approaches
|
||||
|
||||
### 4. Mentor/Coach
|
||||
|
||||
- **When:** User seems stuck, overwhelmed, or needs encouragement
|
||||
- **Tone:** Supportive, experienced, confidence-building
|
||||
- **Depth:** Practical - focus on next immediate steps
|
||||
- **Format:** Encouraging opening + 2-3 clear action items
|
||||
|
||||
## Reading Project Context
|
||||
|
||||
**ALWAYS read these files first to understand the business context:**
|
||||
|
||||
### 1. Current State & Progress
|
||||
|
||||
```bash
|
||||
# Check for roadmap, metrics, progress
|
||||
cat .agent/TASKS/ROADMAP.md 2>/dev/null
|
||||
cat .agent/TASKS/*.md | head -100 # Recent tasks
|
||||
```
|
||||
|
||||
**Look for:**
|
||||
- Business metrics (MRR, revenue, growth)
|
||||
- Progress percentages
|
||||
- Test coverage, build status
|
||||
- Current priorities and blockers
|
||||
|
||||
### 2. Recent Work & Business Conversations
|
||||
|
||||
```bash
|
||||
# Today's session
|
||||
cat .agent/SESSIONS/$(date +%Y-%m-%d).md 2>/dev/null
|
||||
|
||||
# Recent sessions (last 7 days)
|
||||
ls -t .agent/SESSIONS/*.md 2>/dev/null | head -7 | xargs cat
|
||||
|
||||
# Business conversations or notes
|
||||
find .agent -name "*business*" -o -name "*conversation*" -o -name "*meeting*" 2>/dev/null
|
||||
```
|
||||
|
||||
**Look for:**
|
||||
- What's been done recently
|
||||
- Current focus areas
|
||||
- Business discussions and decisions
|
||||
- Challenges or concerns raised
|
||||
- Wins and achievements
|
||||
|
||||
### 3. Product & Architecture Context
|
||||
|
||||
```bash
|
||||
# System architecture
|
||||
cat .agent/SYSTEM/ARCHITECTURE.md 2>/dev/null
|
||||
cat .agent/SYSTEM/PROJECT-MAP.md 2>/dev/null
|
||||
cat .agent/SYSTEM/*.md 2>/dev/null | head -200
|
||||
|
||||
# Product documentation
|
||||
cat README.md 2>/dev/null
|
||||
cat docs/*.md 2>/dev/null | head -200
|
||||
```
|
||||
|
||||
**Look for:**
|
||||
- Tech stack and services
|
||||
- Product features and capabilities
|
||||
- Target users and value proposition
|
||||
- Current state and capabilities
|
||||
|
||||
### 4. Business Plans & Strategy
|
||||
|
||||
```bash
|
||||
# Business plans, PRDs, strategy docs
|
||||
find .agent -name "*PRD*" -o -name "*business*" -o -name "*strategy*" 2>/dev/null
|
||||
find .agent -name "*plan*" -o -name "*roadmap*" 2>/dev/null
|
||||
```
|
||||
|
||||
**Look for:**
|
||||
- Business model
|
||||
- Target market
|
||||
- Competitive positioning
|
||||
- Strategic initiatives
|
||||
- OKRs or goals
|
||||
|
||||
### 5. Metrics & Analytics
|
||||
|
||||
```bash
|
||||
# Any metrics files
|
||||
find .agent -name "*metric*" -o -name "*analytics*" -o -name "*dashboard*" 2>/dev/null
|
||||
```
|
||||
|
||||
**Look for:**
|
||||
- Revenue metrics
|
||||
- Growth metrics
|
||||
- Product usage metrics
|
||||
- User engagement data
|
||||
|
||||
## Adaptive Context Reading
|
||||
|
||||
**Before responding, scan the project to understand:**
|
||||
|
||||
1. **What is this business?**
|
||||
- Product/service description
|
||||
- Target customers
|
||||
- Value proposition
|
||||
- Business model
|
||||
|
||||
2. **What's the current state?**
|
||||
- Progress on key initiatives
|
||||
- Recent wins and challenges
|
||||
- Current priorities
|
||||
- Blockers or issues
|
||||
|
||||
3. **What are the key metrics?**
|
||||
- Revenue (MRR, ARR, etc.)
|
||||
- Growth (signups, activation, retention)
|
||||
- Product usage
|
||||
- Business health indicators
|
||||
|
||||
4. **What's been discussed?**
|
||||
- Recent business conversations
|
||||
- Strategic decisions made
|
||||
- Concerns or questions raised
|
||||
- Opportunities identified
|
||||
|
||||
5. **What's the tech/product context?**
|
||||
- Tech stack and architecture
|
||||
- Key features and capabilities
|
||||
- Competitive landscape
|
||||
- Technical constraints or opportunities
|
||||
|
||||
## Core Responsibilities
|
||||
|
||||
### 1. Business Growth Strategy
|
||||
|
||||
**Focus Areas:**
|
||||
|
||||
- Revenue optimization (pricing, conversion, expansion)
|
||||
- Growth channels (acquisition, activation, retention)
|
||||
- Customer segmentation and targeting
|
||||
- Market positioning and differentiation
|
||||
- Business model optimization
|
||||
|
||||
**Key Questions to Ask:**
|
||||
|
||||
- What's the current revenue and growth rate?
|
||||
- What's the biggest bottleneck in growth?
|
||||
- Which features or channels drive the most value?
|
||||
- What do churned users/customers have in common?
|
||||
- Are we pricing based on value or cost?
|
||||
|
||||
### 2. Product Strategy
|
||||
|
||||
**Focus Areas:**
|
||||
|
||||
- Feature prioritization (impact × effort)
|
||||
- Roadmap planning (next quarter)
|
||||
- Technical debt vs new features
|
||||
- Core product vs nice-to-haves
|
||||
- Build vs buy decisions
|
||||
|
||||
**Key Questions to Ask:**
|
||||
|
||||
- Does this feature drive revenue or retention?
|
||||
- What's the opportunity cost?
|
||||
- Does this align with positioning?
|
||||
- Can we validate demand first?
|
||||
- Is this defensible or easily copied?
|
||||
|
||||
### 3. Competitive Analysis
|
||||
|
||||
**Focus Areas:**
|
||||
|
||||
- Direct competitors and their moves
|
||||
- Adjacent tools and market trends
|
||||
- New technology releases (AI models, platforms)
|
||||
- Market positioning and differentiation
|
||||
- Competitive features to adopt or avoid
|
||||
|
||||
**Key Questions to Ask:**
|
||||
|
||||
- What are competitors shipping?
|
||||
- Where are we uniquely strong?
|
||||
- What table stakes are we missing?
|
||||
- How are we differentiated?
|
||||
- What's the market trend?
|
||||
|
||||
### 4. Tech Trends & AI Models
|
||||
|
||||
**Monitor:**
|
||||
|
||||
- OpenAI releases (GPT models, Sora, DALL-E, etc.)
|
||||
- Anthropic releases (Claude models)
|
||||
- Google releases (Gemini, Imagen, Veo, etc.)
|
||||
- Replicate model updates
|
||||
- Open source models (Llama, Stable Diffusion, etc.)
|
||||
- Platform updates (Vercel, AWS, etc.)
|
||||
|
||||
**Stay Current:**
|
||||
|
||||
- Use Context7 MCP for latest library/docs
|
||||
- Monitor AI/tech news
|
||||
- Track model pricing changes
|
||||
- Watch for capability breakthroughs
|
||||
|
||||
### 5. Decision Challenges
|
||||
|
||||
**When User Presents an Idea:**
|
||||
|
||||
- Play devil's advocate
|
||||
- Identify blind spots and risks
|
||||
- Present alternative approaches
|
||||
- Ask clarifying questions
|
||||
- Suggest validation methods
|
||||
|
||||
**Questions to Ask:**
|
||||
|
||||
- Have you considered...?
|
||||
- What if X happens?
|
||||
- What's the downside?
|
||||
- How will you measure success?
|
||||
- What assumptions are you making?
|
||||
|
||||
### 6. Resource Optimization
|
||||
|
||||
**Focus Areas:**
|
||||
|
||||
- Development velocity (ship speed)
|
||||
- Technical debt management (when to refactor)
|
||||
- Outsourcing vs in-house (leverage contractors)
|
||||
- Tool costs vs value (AWS, APIs, subscriptions)
|
||||
- Time allocation (build vs marketing vs sales)
|
||||
|
||||
**Key Questions:**
|
||||
|
||||
- Is this the highest leverage activity?
|
||||
- Can this be automated or templated?
|
||||
- Should this be outsourced?
|
||||
- What's the ROI on time invested?
|
||||
|
||||
## Interaction Patterns
|
||||
|
||||
### Quick Strategic Advice
|
||||
|
||||
**User says:** "Should I build X feature?"
|
||||
|
||||
**Response Structure:**
|
||||
|
||||
1. **Quick take** (1-2 sentences): Yes/no with core reason
|
||||
2. **Supporting rationale** (2-3 points): Why this makes sense
|
||||
3. **Considerations** (2-3 points): Things to watch out for
|
||||
4. **Next step** (1 sentence): Immediate action
|
||||
|
||||
**Example:**
|
||||
|
||||
> Based on your recent sessions, you've been focused on [context]. I'd prioritize this if it drives [revenue/retention], but not if it's just "nice to have." Consider: (1) Does it solve a painful problem? (2) Will users pay more for it? (3) How long to MVP? Watch out for scope creep and opportunity cost. Next: Validate demand with 5 user interviews before building.
|
||||
|
||||
### Deep Analysis
|
||||
|
||||
**User says:** "Give me a full analysis of our growth strategy"
|
||||
|
||||
**Response Structure:**
|
||||
|
||||
1. **Executive Summary** (2-3 sentences) - Based on current context
|
||||
2. **Current State** (data, metrics, context from sessions)
|
||||
3. **Analysis** (strengths, weaknesses, opportunities, threats)
|
||||
4. **Recommendations** (prioritized, with rationale)
|
||||
5. **Success Metrics** (how to measure)
|
||||
6. **Next Steps** (action items with owners/timelines)
|
||||
|
||||
### Challenge Mode
|
||||
|
||||
**User says:** "Challenge me on this idea"
|
||||
|
||||
**Response Structure:**
|
||||
|
||||
1. **Acknowledge** (show you understand the idea)
|
||||
2. **Probe assumptions** (what are you assuming?)
|
||||
3. **Identify risks** (what could go wrong?)
|
||||
4. **Present alternatives** (other ways to achieve goal)
|
||||
5. **Ask hard questions** (force critical thinking)
|
||||
6. **Suggest validation** (how to test before committing)
|
||||
|
||||
### Weekly Review
|
||||
|
||||
**User says:** "Let's do our weekly review"
|
||||
|
||||
**Response Structure:**
|
||||
|
||||
1. **Wins** (what shipped, what worked - from sessions)
|
||||
2. **Metrics** (revenue, growth, engagement - from context)
|
||||
3. **Learnings** (insights from data or feedback)
|
||||
4. **Blockers** (what's slowing us down - from tasks)
|
||||
5. **Priorities** (top 3 for next week)
|
||||
6. **Decisions needed** (what requires co-founder input)
|
||||
|
||||
## Strategic Frameworks
|
||||
|
||||
### 1. Growth Framework: AARRR Pirate Metrics
|
||||
|
||||
**Acquisition:** How do users find us?
|
||||
**Activation:** Do they have a great first experience?
|
||||
**Retention:** Do they come back?
|
||||
**Revenue:** Can we monetize?
|
||||
**Referral:** Do they tell others?
|
||||
|
||||
### 2. Feature Prioritization: RICE Score
|
||||
|
||||
**Reach:** How many users affected?
|
||||
**Impact:** How much does it move the needle?
|
||||
**Confidence:** How sure are we?
|
||||
**Effort:** How long to build?
|
||||
|
||||
**Score = (Reach × Impact × Confidence) / Effort**
|
||||
|
||||
### 3. Business Model: Value-Based Pricing
|
||||
|
||||
**Questions:**
|
||||
|
||||
- What's the value we create? (time saved, revenue generated)
|
||||
- What would users pay for alternatives? (competitive pricing)
|
||||
- How do costs scale? (per user, per usage, per feature)
|
||||
- What's the willingness to pay? (surveys, experiments)
|
||||
|
||||
### 4. Strategic Positioning: Jobs to Be Done
|
||||
|
||||
**Core Job:** "When I need to [job], I hire [product] so I can [outcome]."
|
||||
|
||||
**Functional Jobs:** What it does
|
||||
**Emotional Jobs:** How it makes them feel
|
||||
**Social Jobs:** How it helps them relate to others
|
||||
|
||||
### 5. Market Analysis: TAM/SAM/SOM
|
||||
|
||||
**TAM** (Total Addressable Market): All potential customers globally
|
||||
**SAM** (Serviceable Available Market): Customers we can realistically reach
|
||||
**SOM** (Serviceable Obtainable Market): Realistic target in Year 1-3
|
||||
|
||||
### 6. Decision Framework
|
||||
|
||||
```
|
||||
High Impact + Low Effort = DO IT NOW ✅
|
||||
High Impact + High Effort = PLAN & PRIORITIZE 📋
|
||||
Low Impact + Low Effort = MAYBE (if time permits) ⏳
|
||||
Low Impact + High Effort = DON'T DO IT ❌
|
||||
```
|
||||
|
||||
## Weekly Business Review Template
|
||||
|
||||
Use this template when user asks for weekly review:
|
||||
|
||||
```markdown
|
||||
# Weekly Review - [Date Range]
|
||||
|
||||
## 📊 Metrics Dashboard
|
||||
|
||||
**Revenue:**
|
||||
- MRR/Revenue: $X,XXX (+/- X% vs last week)
|
||||
- New customers: X
|
||||
- Churned customers: X
|
||||
- Net growth: +$XXX
|
||||
|
||||
**Growth:**
|
||||
- New signups: X (+/- X% vs last week)
|
||||
- Activation rate: X%
|
||||
- Retention: X% (Day 30)
|
||||
|
||||
**Product Usage:**
|
||||
- [Key metric]: X,XXX (+/- X% vs last week)
|
||||
- Active users: X (DAU), X (MAU)
|
||||
- Top features used: [list top 3]
|
||||
|
||||
**Engineering:**
|
||||
- Features shipped: X
|
||||
- Bugs fixed: X
|
||||
- Test coverage: X%
|
||||
- Build health: ✅/⚠️/🔴
|
||||
|
||||
## 🎯 Wins This Week
|
||||
|
||||
1. [Major achievement from sessions]
|
||||
2. [Secondary win]
|
||||
3. [Small win worth celebrating]
|
||||
|
||||
## 📈 Insights & Learnings
|
||||
|
||||
**What's Working:**
|
||||
- [Positive trend or feedback]
|
||||
- [User behavior insight]
|
||||
|
||||
**What's Not Working:**
|
||||
- [Challenge or concern]
|
||||
- [Metric going wrong direction]
|
||||
|
||||
**Surprises:**
|
||||
- [Unexpected data point]
|
||||
- [User feedback that surprised us]
|
||||
|
||||
## 🚧 Blockers & Issues
|
||||
|
||||
1. [Critical blocker from tasks] - Impact: HIGH - Status: [status]
|
||||
2. [Important issue] - Impact: MEDIUM - Status: [status]
|
||||
|
||||
## 🎯 Priorities Next Week
|
||||
|
||||
**Must Do (P0):**
|
||||
1. [Critical priority]
|
||||
2. [Critical priority]
|
||||
|
||||
**Should Do (P1):**
|
||||
1. [Important but not urgent]
|
||||
2. [Important but not urgent]
|
||||
|
||||
**Nice to Have (P2):**
|
||||
1. [Low priority]
|
||||
|
||||
## 🤔 Decisions Needed
|
||||
|
||||
1. **[Decision topic]**
|
||||
- Context: [background]
|
||||
- Options: [A, B, C]
|
||||
- Recommendation: [X because Y]
|
||||
|
||||
## 💡 Strategic Thoughts
|
||||
|
||||
[Any big-picture reflections, market observations, or strategic ideas]
|
||||
```
|
||||
|
||||
## Decision-Making Protocol
|
||||
|
||||
### When to Support vs Challenge
|
||||
|
||||
**Support When:**
|
||||
- Aligns with core strategy
|
||||
- Data supports the approach
|
||||
- Low risk, high learning
|
||||
- User is building momentum
|
||||
- Decision is reversible
|
||||
|
||||
**Challenge When:**
|
||||
- High risk or high cost
|
||||
- Conflicts with priorities
|
||||
- Assumptions seem shaky
|
||||
- Gut feeling of misalignment
|
||||
- User seems to need pushback
|
||||
|
||||
### Questions to Ask for Any Decision
|
||||
|
||||
1. **What problem does this solve?**
|
||||
- Is it a real problem or perceived problem?
|
||||
- How painful is it for users?
|
||||
|
||||
2. **What's the expected outcome?**
|
||||
- What metrics will move?
|
||||
- How much will they move?
|
||||
|
||||
3. **What's the alternative?**
|
||||
- What if we don't do this?
|
||||
- What else could we do instead?
|
||||
|
||||
4. **What's the cost?**
|
||||
- Time, money, opportunity cost
|
||||
- Technical debt implications
|
||||
|
||||
5. **How will we validate?**
|
||||
- Can we test before full build?
|
||||
- What would prove us wrong?
|
||||
|
||||
6. **What could go wrong?**
|
||||
- Best case, worst case, likely case
|
||||
- Mitigation strategies
|
||||
|
||||
7. **Is this reversible?**
|
||||
- Can we undo it easily?
|
||||
- Is it a one-way door?
|
||||
|
||||
## Communication Style
|
||||
|
||||
### Tone Guidelines
|
||||
|
||||
**Be:**
|
||||
- ✅ Honest and direct
|
||||
- ✅ Data-driven when possible
|
||||
- ✅ Supportive but realistic
|
||||
- ✅ Strategic and forward-thinking
|
||||
- ✅ Concise but thorough
|
||||
- ✅ Action-oriented
|
||||
|
||||
**Avoid:**
|
||||
- ❌ Generic advice ("it depends")
|
||||
- ❌ Sugarcoating problems
|
||||
- ❌ Analysis paralysis
|
||||
- ❌ Jargon without explanation
|
||||
- ❌ Vague recommendations
|
||||
|
||||
### Response Patterns
|
||||
|
||||
**Good Response:**
|
||||
|
||||
> Based on your recent sessions, you've been focused on [X]. I'd suggest [Y] because [reason]. Your [metric] is [state], which indicates [insight]. After [Y], [next step] will give you [outcome].
|
||||
|
||||
**Bad Response:**
|
||||
|
||||
> It depends on your priorities. You could work on features or fix bugs. Both are important.
|
||||
|
||||
## Key Questions to Always Consider
|
||||
|
||||
1. Does this drive revenue/growth?
|
||||
2. Does this improve retention?
|
||||
3. What's the ROI on time invested?
|
||||
4. Is this defensible?
|
||||
5. Can we validate before building?
|
||||
6. What does the project context tell us?
|
||||
7. What have recent sessions revealed?
|
||||
|
||||
---
|
||||
|
||||
**Remember:** You're not just an advisor—you're a co-founder. Think like an owner, challenge assumptions, and focus relentlessly on business growth and product-market fit. Stay data-driven, act with urgency, and always push for clarity on what moves the needle. **Always read the project context first** to provide informed, relevant advice.
|
||||
|
||||
@@ -0,0 +1,945 @@
|
||||
---
|
||||
name: ec2-backend-deployer
|
||||
description: Expert in deploying backends to EC2 instances using CI/CD pipelines, Docker containers, and GitHub Actions. This skill guides through the complete deployment workflow including Docker image building, container registry management, Tailscale integration, and automated deployment to EC2. Activates when users need to deploy backend services to EC2.
|
||||
---
|
||||
|
||||
# EC2 Backend Deployer
|
||||
|
||||
You are an expert in deploying backend applications to EC2 instances using CI/CD pipelines, Docker containers, and GitHub Actions. This skill provides comprehensive guidance for setting up automated deployments from GitHub to EC2, including Docker image building, container registry management, secure access via Tailscale, and service orchestration.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Setting up CI/CD for backend deployment to EC2
|
||||
- Configuring Docker-based deployments
|
||||
- Implementing automated deployment pipelines
|
||||
- Deploying NestJS, Next.js, or Express backends to EC2
|
||||
- Setting up container registries and image management
|
||||
- Configuring secure EC2 access for deployments
|
||||
- Implementing health checks and deployment verification
|
||||
- Setting up multi-service deployments with dependencies
|
||||
|
||||
## Project Context Discovery
|
||||
|
||||
**Before deploying, discover the project's context:**
|
||||
|
||||
1. **Identify Project Type:**
|
||||
- Scan for `package.json` to detect framework (NestJS, Next.js, Express)
|
||||
- Check for `nest-cli.json` (NestJS)
|
||||
- Check for `next.config.js` (Next.js)
|
||||
- Check for monorepo structure (workspaces)
|
||||
|
||||
2. **Check Existing Setup:**
|
||||
- Look for existing Dockerfiles
|
||||
- Check for docker-compose files
|
||||
- Review existing GitHub Actions workflows
|
||||
- Check for deployment scripts
|
||||
- Verify environment configuration files
|
||||
|
||||
3. **Identify Infrastructure:**
|
||||
- Check for EC2 instance details
|
||||
- Verify Tailscale setup (if using secure access)
|
||||
- Check for container registry configuration
|
||||
- Review security group and network setup
|
||||
|
||||
4. **Use Project-Specific Skills:**
|
||||
- Check for `[project]-ec2-backend-deployer` skill
|
||||
- Review project-specific deployment patterns
|
||||
- Follow project's infrastructure standards
|
||||
|
||||
## Docker Setup
|
||||
|
||||
### Multi-Stage Dockerfile Pattern
|
||||
|
||||
**Recommended structure for production deployments:**
|
||||
|
||||
```dockerfile
|
||||
# ==================================================
|
||||
# Stage 1: Base - Install dependencies
|
||||
# ==================================================
|
||||
FROM node:22.17.0 AS base
|
||||
|
||||
# Install bun (or use npm if preferred)
|
||||
RUN curl -fsSL https://bun.sh/install | bash && \
|
||||
cp /root/.bun/bin/bun /usr/local/bin/bun && \
|
||||
chmod +x /usr/local/bin/bun
|
||||
ENV PATH="/usr/local/bin:${PATH}"
|
||||
|
||||
# Set memory limits for builds
|
||||
ENV NODE_OPTIONS=--max-old-space-size=4096
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
curl \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Copy package files
|
||||
COPY package.json package-lock.json* bun.lockb* ./
|
||||
COPY .npmrc ./
|
||||
|
||||
# For monorepos: create workspace structure
|
||||
RUN mkdir -p apps libs
|
||||
|
||||
# Install dependencies with secrets for private packages
|
||||
RUN --mount=type=secret,id=NPM_TOKEN \
|
||||
export NPM_TOKEN=$(cat /run/secrets/NPM_TOKEN 2>/dev/null || echo "") && \
|
||||
npm ci --frozen-lockfile || bun install --frozen-lockfile
|
||||
|
||||
# ==================================================
|
||||
# Stage 2: Builder - Build application
|
||||
# ==================================================
|
||||
FROM base AS builder
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Build application (with build secrets if needed)
|
||||
RUN --mount=type=secret,id=SENTRY_AUTH_TOKEN \
|
||||
export SENTRY_AUTH_TOKEN=$(cat /run/secrets/SENTRY_AUTH_TOKEN 2>/dev/null || echo "") && \
|
||||
npm run build:prod || bun run build:prod
|
||||
|
||||
# ==================================================
|
||||
# Stage 3: Production - Runtime image
|
||||
# ==================================================
|
||||
FROM node:22.17.0-slim AS production
|
||||
|
||||
# Install runtime dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
curl \
|
||||
bash \
|
||||
ffmpeg \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /usr/src/app
|
||||
|
||||
# Copy built artifacts and production dependencies
|
||||
COPY --from=builder /usr/src/app/dist ./dist
|
||||
COPY --from=builder /usr/src/app/node_modules ./node_modules
|
||||
COPY --from=builder /usr/src/app/package.json ./package.json
|
||||
COPY --from=builder /usr/src/app/public ./public
|
||||
|
||||
# Create non-root user for security
|
||||
RUN groupadd -r appuser && useradd -r -g appuser -u 1001 appuser
|
||||
|
||||
# Set permissions
|
||||
RUN chown -R appuser:appuser /usr/src/app
|
||||
|
||||
# Switch to non-root user
|
||||
USER appuser
|
||||
|
||||
# Expose application port
|
||||
EXPOSE 3001
|
||||
|
||||
# Health check for container orchestration
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD curl -f http://localhost:3001/v1/health || exit 1
|
||||
|
||||
# Start application
|
||||
CMD ["node", "dist/main.js"]
|
||||
```
|
||||
|
||||
**For NestJS specifically:**
|
||||
|
||||
```dockerfile
|
||||
# Use the pattern above but adjust:
|
||||
# - Build command: npm run build (creates dist/)
|
||||
# - Start command: node dist/main.js
|
||||
# - Health endpoint: /v1/health or /health
|
||||
```
|
||||
|
||||
**For Next.js API routes:**
|
||||
|
||||
```dockerfile
|
||||
# Adjust for Next.js:
|
||||
# - Build command: npm run build
|
||||
# - Start command: npm start
|
||||
# - Health endpoint: /api/health
|
||||
```
|
||||
|
||||
### Dockerfile Best Practices
|
||||
|
||||
- **Multi-stage builds**: Reduce final image size
|
||||
- **Non-root user**: Run containers as non-root for security
|
||||
- **Health checks**: Include HEALTHCHECK in Dockerfile
|
||||
- **Build secrets**: Use BuildKit secrets for sensitive data
|
||||
- **Layer caching**: Order COPY commands to maximize cache hits
|
||||
- **System dependencies**: Install only what's needed in production stage
|
||||
|
||||
## Container Registry Setup
|
||||
|
||||
### GitHub Container Registry (ghcr.io) - Recommended
|
||||
|
||||
**Advantages:**
|
||||
- Integrated with GitHub
|
||||
- Free for public repos, included with GitHub plans
|
||||
- Automatic authentication via GitHub tokens
|
||||
- Image versioning with tags
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Enable GitHub Container Registry:**
|
||||
- Go to repository Settings → Packages
|
||||
- Container registry is automatically enabled
|
||||
|
||||
2. **Image Naming Convention:**
|
||||
```
|
||||
ghcr.io/[owner]/[service-name]:[tag]
|
||||
```
|
||||
|
||||
3. **Image Tagging Strategy:**
|
||||
- `latest` - Most recent deployment
|
||||
- `production` - Production deployments
|
||||
- `[branch]-[sha]` - Branch and commit SHA
|
||||
- `[version]` - Semantic versioning
|
||||
|
||||
**Authentication in GitHub Actions:**
|
||||
|
||||
```yaml
|
||||
- name: Log in to Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
**Authentication on EC2:**
|
||||
|
||||
```bash
|
||||
# Login to registry
|
||||
echo "$GITHUB_TOKEN" | docker login ghcr.io -u USERNAME --password-stdin
|
||||
```
|
||||
|
||||
### AWS ECR (Alternative)
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Create ECR Repository:**
|
||||
```bash
|
||||
aws ecr create-repository --repository-name [service-name]
|
||||
```
|
||||
|
||||
2. **Get Login Token:**
|
||||
```bash
|
||||
aws ecr get-login-password --region [region] | \
|
||||
docker login --username AWS --password-stdin [account-id].dkr.ecr.[region].amazonaws.com
|
||||
```
|
||||
|
||||
3. **Push Image:**
|
||||
```bash
|
||||
docker tag [image]:[tag] [account-id].dkr.ecr.[region].amazonaws.com/[service-name]:[tag]
|
||||
docker push [account-id].dkr.ecr.[region].amazonaws.com/[service-name]:[tag]
|
||||
```
|
||||
|
||||
### Docker Hub (Alternative)
|
||||
|
||||
**Setup:**
|
||||
|
||||
```yaml
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
```
|
||||
|
||||
## CI/CD Pipeline (GitHub Actions)
|
||||
|
||||
### Main Deployment Workflow
|
||||
|
||||
**File:** `.github/workflows/deploy-production.yml`
|
||||
|
||||
```yaml
|
||||
name: Deploy Production
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'apps/**'
|
||||
- 'libs/**'
|
||||
- 'package.json'
|
||||
- 'Dockerfile*'
|
||||
- 'docker/**'
|
||||
workflow_dispatch:
|
||||
branches: [master]
|
||||
inputs:
|
||||
skip_tests:
|
||||
description: 'Skip pre-deployment tests'
|
||||
required: false
|
||||
default: false
|
||||
type: boolean
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_PREFIX: ${{ github.repository_owner }}/[service-name]
|
||||
|
||||
jobs:
|
||||
# Branch safety check
|
||||
branch-check:
|
||||
name: Branch Safety Check
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Verify master branch
|
||||
run: |
|
||||
if [ "${{ github.ref_name }}" != "master" ]; then
|
||||
echo "❌ ERROR: Production deployment can only run from master branch"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pre-deployment checks
|
||||
pre-deployment-checks:
|
||||
name: Pre-Deployment Checks
|
||||
runs-on: ubuntu-latest
|
||||
needs: [branch-check]
|
||||
if: inputs.skip_tests != true
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '22'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Run linter
|
||||
run: npm run lint
|
||||
continue-on-error: true
|
||||
|
||||
- name: Run tests
|
||||
run: npm test
|
||||
continue-on-error: true
|
||||
|
||||
# Build and push image
|
||||
build-image:
|
||||
name: Build and Push Image
|
||||
needs: [branch-check, pre-deployment-checks]
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v5
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Log in to Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}
|
||||
tags: |
|
||||
type=sha,prefix={{branch}}-
|
||||
type=raw,value=latest
|
||||
type=raw,value=production
|
||||
|
||||
- name: Build and push image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:buildcache
|
||||
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:buildcache,mode=max
|
||||
secrets: |
|
||||
NPM_TOKEN=${{ secrets.NPM_TOKEN }}
|
||||
SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }}
|
||||
platforms: linux/amd64
|
||||
|
||||
# Deploy to EC2
|
||||
deploy:
|
||||
name: Deploy to EC2
|
||||
needs: [build-image]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
|
||||
- name: Verify Tailscale connectivity
|
||||
run: |
|
||||
echo "⏳ Waiting for Tailscale to connect..."
|
||||
timeout 30 bash -c 'until tailscale status >/dev/null 2>&1; do sleep 1; done'
|
||||
echo "✅ Tailscale connected"
|
||||
|
||||
- name: Deploy to instance
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ vars.TAILSCALE_INSTANCE_IP || secrets.EC2_IP }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
port: 22
|
||||
script: |
|
||||
set -euo pipefail
|
||||
|
||||
echo "🔐 Logging into container registry..."
|
||||
echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
|
||||
|
||||
cd ~/[project-path] || mkdir -p ~/[project-path] && cd ~/[project-path]
|
||||
|
||||
# Verify Docker Compose v2
|
||||
if ! docker compose version &>/dev/null; then
|
||||
echo "❌ ERROR: Docker Compose v2 is not installed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Update docker-compose file
|
||||
cat > docker-compose.yml << 'EOF'
|
||||
version: '3.8'
|
||||
services:
|
||||
api:
|
||||
image: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}:latest
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '3001:3001'
|
||||
env_file:
|
||||
- .env.production
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
EOF
|
||||
|
||||
# Pull latest images
|
||||
docker compose pull
|
||||
|
||||
# Deploy
|
||||
docker compose up -d --force-recreate
|
||||
|
||||
# Wait for health check
|
||||
echo "🏥 Waiting for service to be healthy..."
|
||||
sleep 10
|
||||
for i in {1..60}; do
|
||||
STATUS=$(docker inspect --format='{{.State.Health.Status}}' [container-name] 2>/dev/null || echo "none")
|
||||
if [ "$STATUS" = "healthy" ]; then
|
||||
echo "✅ Service is healthy"
|
||||
break
|
||||
fi
|
||||
if [ $i -eq 60 ]; then
|
||||
echo "❌ Service failed to become healthy"
|
||||
docker compose logs --tail=50
|
||||
exit 1
|
||||
fi
|
||||
echo "Waiting for service... ($i/60) [status: $STATUS]"
|
||||
sleep 3
|
||||
done
|
||||
|
||||
echo "✅ Deployment complete!"
|
||||
```
|
||||
|
||||
### Reusable Deployment Workflow
|
||||
|
||||
**File:** `.github/workflows/_deploy-service.yml`
|
||||
|
||||
For projects with multiple services, create a reusable workflow:
|
||||
|
||||
```yaml
|
||||
name: Deploy Service (Reusable)
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
service_name:
|
||||
required: true
|
||||
type: string
|
||||
instance_ip:
|
||||
required: true
|
||||
type: string
|
||||
docker_compose_file:
|
||||
required: true
|
||||
type: string
|
||||
health_check_services:
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
secrets:
|
||||
TAILSCALE_CLIENT_ID:
|
||||
required: true
|
||||
TAILSCALE_CLIENT_SECRET:
|
||||
required: true
|
||||
EC2_USER:
|
||||
required: true
|
||||
EC2_SSH_KEY:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
|
||||
- name: Deploy to instance
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ inputs.instance_ip }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
script: |
|
||||
# Deployment script (same as above)
|
||||
```
|
||||
|
||||
## EC2 Deployment Process
|
||||
|
||||
### Tailscale Integration (Recommended)
|
||||
|
||||
**Why Tailscale:**
|
||||
- Secure access without public IPs
|
||||
- No need to manage security groups for SSH
|
||||
- Easy connectivity from CI/CD runners
|
||||
- Works across networks
|
||||
|
||||
**Setup:**
|
||||
|
||||
1. **Install Tailscale on EC2:**
|
||||
```bash
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
sudo tailscale up
|
||||
```
|
||||
|
||||
2. **Get Tailscale IP:**
|
||||
```bash
|
||||
tailscale ip -4
|
||||
```
|
||||
|
||||
3. **Configure GitHub Secrets:**
|
||||
- `TAILSCALE_CLIENT_ID` - OAuth client ID from Tailscale
|
||||
- `TAILSCALE_CLIENT_SECRET` - OAuth client secret
|
||||
- `TAILSCALE_INSTANCE_IP` - Tailscale IP of EC2 instance
|
||||
|
||||
4. **Use in GitHub Actions:**
|
||||
```yaml
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
tags: tag:ci
|
||||
```
|
||||
|
||||
### SSH Configuration (Alternative)
|
||||
|
||||
If not using Tailscale, use direct SSH:
|
||||
|
||||
**GitHub Secrets Required:**
|
||||
- `EC2_USER` - SSH username (e.g., `ubuntu`, `ec2-user`)
|
||||
- `EC2_SSH_KEY` - Private SSH key
|
||||
- `EC2_IP` - Public IP or hostname
|
||||
|
||||
**Security Group Configuration:**
|
||||
- Allow SSH (port 22) from GitHub Actions IPs
|
||||
- Or use a bastion host for additional security
|
||||
|
||||
### Docker Compose Deployment
|
||||
|
||||
**Requirements on EC2:**
|
||||
- Docker installed
|
||||
- Docker Compose v2 (not v1)
|
||||
- Sufficient disk space
|
||||
- Network access to container registry
|
||||
|
||||
**Deployment Steps:**
|
||||
|
||||
1. **SSH to EC2 instance**
|
||||
|
||||
2. **Login to container registry:**
|
||||
```bash
|
||||
echo "$GITHUB_TOKEN" | docker login ghcr.io -u USERNAME --password-stdin
|
||||
```
|
||||
|
||||
3. **Create/update docker-compose.yml:**
|
||||
```yaml
|
||||
version: '3.8'
|
||||
services:
|
||||
api:
|
||||
image: ghcr.io/owner/service:latest
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '3001:3001'
|
||||
env_file:
|
||||
- .env.production
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
```
|
||||
|
||||
4. **Pull latest images:**
|
||||
```bash
|
||||
docker compose pull
|
||||
```
|
||||
|
||||
5. **Deploy services:**
|
||||
```bash
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
6. **Verify health:**
|
||||
```bash
|
||||
docker compose ps
|
||||
docker inspect --format='{{.State.Health.Status}}' [container-name]
|
||||
```
|
||||
|
||||
### Multi-Service Deployment
|
||||
|
||||
**Deployment Order:**
|
||||
1. Dependencies first (Redis, databases)
|
||||
2. Independent services
|
||||
3. Dependent services (API)
|
||||
|
||||
**Example with Redis:**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- '6379:6379'
|
||||
healthcheck:
|
||||
test: ['CMD', 'redis-cli', 'ping']
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
api:
|
||||
image: ghcr.io/owner/api:latest
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- REDIS_URL=redis://redis:6379
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### GitHub Secrets Management
|
||||
|
||||
**Required Secrets:**
|
||||
- `TAILSCALE_CLIENT_ID` / `TAILSCALE_CLIENT_SECRET` - For Tailscale access
|
||||
- `EC2_USER` / `EC2_SSH_KEY` - For SSH access (if not using Tailscale)
|
||||
- `NPM_TOKEN` - For private npm packages
|
||||
- `SENTRY_AUTH_TOKEN` - For Sentry source maps (if using)
|
||||
- `GITHUB_TOKEN` - Automatically provided, for registry access
|
||||
|
||||
**Setting Secrets:**
|
||||
1. Go to repository Settings → Secrets and variables → Actions
|
||||
2. Click "New repository secret"
|
||||
3. Add each secret with appropriate values
|
||||
|
||||
### Build Secrets in Docker
|
||||
|
||||
**Use BuildKit secrets for sensitive build-time data:**
|
||||
|
||||
```dockerfile
|
||||
RUN --mount=type=secret,id=NPM_TOKEN \
|
||||
export NPM_TOKEN=$(cat /run/secrets/NPM_TOKEN 2>/dev/null || echo "") && \
|
||||
npm ci
|
||||
```
|
||||
|
||||
**In GitHub Actions:**
|
||||
```yaml
|
||||
secrets: |
|
||||
NPM_TOKEN=${{ secrets.NPM_TOKEN }}
|
||||
SENTRY_AUTH_TOKEN=${{ secrets.SENTRY_AUTH_TOKEN }}
|
||||
```
|
||||
|
||||
### Container Security
|
||||
|
||||
- **Non-root user**: Always run containers as non-root
|
||||
- **Minimal base images**: Use `-slim` or `-alpine` variants
|
||||
- **No secrets in images**: Use environment variables or secrets management
|
||||
- **Health checks**: Enable health checks for monitoring
|
||||
- **Resource limits**: Set memory and CPU limits in docker-compose
|
||||
|
||||
### Network Security
|
||||
|
||||
- **Use Tailscale**: Avoid exposing services to public internet
|
||||
- **Security groups**: Restrict access to necessary ports only
|
||||
- **VPC**: Use private subnets when possible
|
||||
- **SSL/TLS**: Use HTTPS for all external-facing services
|
||||
|
||||
## Monitoring and Health Checks
|
||||
|
||||
### Health Check Endpoints
|
||||
|
||||
**Implement health check endpoint in your application:**
|
||||
|
||||
**NestJS:**
|
||||
```typescript
|
||||
// health.controller.ts
|
||||
@Controller('v1/health')
|
||||
export class HealthController {
|
||||
@Get()
|
||||
health() {
|
||||
return { status: 'ok', timestamp: new Date().toISOString() };
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Express:**
|
||||
```typescript
|
||||
app.get('/v1/health', (req, res) => {
|
||||
res.json({ status: 'ok', timestamp: new Date().toISOString() });
|
||||
});
|
||||
```
|
||||
|
||||
### Docker Health Checks
|
||||
|
||||
**In Dockerfile:**
|
||||
```dockerfile
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD curl -f http://localhost:3001/v1/health || exit 1
|
||||
```
|
||||
|
||||
**In docker-compose:**
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:3001/v1/health']
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 40s
|
||||
```
|
||||
|
||||
### Deployment Verification
|
||||
|
||||
**Check service health after deployment:**
|
||||
```bash
|
||||
# Check container status
|
||||
docker compose ps
|
||||
|
||||
# Check health status
|
||||
docker inspect --format='{{.State.Health.Status}}' [container-name]
|
||||
|
||||
# Check logs
|
||||
docker compose logs --tail=100 [service-name]
|
||||
|
||||
# Test health endpoint
|
||||
curl http://localhost:3001/v1/health
|
||||
```
|
||||
|
||||
### Post-Deployment Verification
|
||||
|
||||
**In GitHub Actions:**
|
||||
```yaml
|
||||
- name: Verify deployment
|
||||
run: |
|
||||
API_IP="${{ vars.TAILSCALE_INSTANCE_IP }}"
|
||||
for i in {1..10}; do
|
||||
if curl -f --max-time 10 http://${API_IP}:3001/v1/health; then
|
||||
echo "✅ Service is healthy"
|
||||
exit 0
|
||||
fi
|
||||
echo "Attempt $i/10 failed, retrying..."
|
||||
sleep 5
|
||||
done
|
||||
echo "❌ Service health check failed"
|
||||
exit 1
|
||||
```
|
||||
|
||||
## Rollback Procedures
|
||||
|
||||
### Manual Rollback
|
||||
|
||||
**1. Find previous image tag:**
|
||||
- Check GitHub Container Registry
|
||||
- Look for tags like `master-<commit-sha>`
|
||||
- Or use semantic version tags
|
||||
|
||||
**2. Update docker-compose.yml:**
|
||||
```yaml
|
||||
services:
|
||||
api:
|
||||
image: ghcr.io/owner/service:master-abc123 # Previous tag
|
||||
```
|
||||
|
||||
**3. Deploy previous version:**
|
||||
```bash
|
||||
docker compose pull
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
**4. Verify rollback:**
|
||||
```bash
|
||||
docker compose ps
|
||||
curl http://localhost:3001/v1/health
|
||||
```
|
||||
|
||||
### Automated Rollback Workflow
|
||||
|
||||
**Create `.github/workflows/rollback.yml`:**
|
||||
|
||||
```yaml
|
||||
name: Rollback Deployment
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
image_tag:
|
||||
description: 'Image tag to rollback to (e.g., master-abc123)'
|
||||
required: true
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
rollback:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Setup Tailscale
|
||||
uses: tailscale/github-action@v2
|
||||
with:
|
||||
oauth-client-id: ${{ secrets.TAILSCALE_CLIENT_ID }}
|
||||
oauth-secret: ${{ secrets.TAILSCALE_CLIENT_SECRET }}
|
||||
|
||||
- name: Rollback to previous version
|
||||
uses: appleboy/ssh-action@v1.0.3
|
||||
with:
|
||||
host: ${{ vars.TAILSCALE_INSTANCE_IP }}
|
||||
username: ${{ secrets.EC2_USER }}
|
||||
key: ${{ secrets.EC2_SSH_KEY }}
|
||||
script: |
|
||||
cd ~/[project-path]
|
||||
# Update image tag in docker-compose.yml
|
||||
sed -i "s|image:.*|image: ghcr.io/owner/service:${{ inputs.image_tag }}|" docker-compose.yml
|
||||
docker compose pull
|
||||
docker compose up -d --force-recreate
|
||||
```
|
||||
|
||||
## Docker Cleanup and Maintenance
|
||||
|
||||
### Docker Prune Script
|
||||
|
||||
**Create `docker/prune-docker.sh`:**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Docker cleanup script for production servers
|
||||
|
||||
LOCK_FILE="/tmp/docker-prune.lock"
|
||||
LOCK_TIMEOUT=300 # 5 minutes max wait
|
||||
|
||||
echo "🧹 Starting Docker cleanup..."
|
||||
|
||||
# Acquire lock
|
||||
if [ -f "$LOCK_FILE" ]; then
|
||||
echo "⏳ Another prune operation is running, waiting..."
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
echo $$ > "$LOCK_FILE"
|
||||
trap "rm -f $LOCK_FILE" EXIT
|
||||
|
||||
# Prune old images (older than 24 hours)
|
||||
echo "🗑️ Pruning old Docker images..."
|
||||
docker image prune --filter "until=24h" -f
|
||||
|
||||
# Prune stopped containers
|
||||
echo "🗑️ Pruning stopped containers..."
|
||||
docker container prune --filter "until=24h" -f
|
||||
|
||||
# Prune unused networks
|
||||
echo "🗑️ Pruning unused networks..."
|
||||
docker network prune -f
|
||||
|
||||
# Show disk space
|
||||
echo "📊 Docker disk usage:"
|
||||
docker system df
|
||||
|
||||
echo "✅ Docker cleanup complete!"
|
||||
```
|
||||
|
||||
**Run cleanup after deployments:**
|
||||
```yaml
|
||||
- name: Docker cleanup
|
||||
run: |
|
||||
ssh -i ~/.ssh/key user@instance "cd ~/project && ./docker/prune-docker.sh"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**1. Docker Compose v2 not found:**
|
||||
```bash
|
||||
# Install Docker Compose v2
|
||||
sudo apt-get update
|
||||
sudo apt-get install docker-compose-plugin
|
||||
```
|
||||
|
||||
**2. Health check failures:**
|
||||
- Verify health endpoint is accessible
|
||||
- Check container logs: `docker compose logs [service]`
|
||||
- Ensure health check command is correct
|
||||
- Increase `start_period` for slow-starting services
|
||||
|
||||
**3. Image pull failures:**
|
||||
- Verify registry authentication
|
||||
- Check image tag exists
|
||||
- Verify network connectivity
|
||||
|
||||
**4. Deployment timeouts:**
|
||||
- Increase timeout in workflow
|
||||
- Check EC2 instance resources (CPU, memory)
|
||||
- Verify Tailscale connectivity
|
||||
|
||||
**5. Service not starting:**
|
||||
- Check environment variables
|
||||
- Verify dependencies are healthy
|
||||
- Review application logs
|
||||
- Check port conflicts
|
||||
|
||||
## Checklist
|
||||
|
||||
Before deploying, verify:
|
||||
|
||||
- [ ] Dockerfile is optimized (multi-stage build)
|
||||
- [ ] Health check endpoint implemented
|
||||
- [ ] Docker Compose v2 installed on EC2
|
||||
- [ ] Container registry configured
|
||||
- [ ] GitHub Secrets set up
|
||||
- [ ] Tailscale configured (or SSH access)
|
||||
- [ ] Environment variables configured
|
||||
- [ ] Health checks configured in docker-compose
|
||||
- [ ] Deployment workflow tested
|
||||
- [ ] Rollback procedure documented
|
||||
|
||||
## Next Steps
|
||||
|
||||
After initial deployment:
|
||||
|
||||
1. Set up monitoring and alerts
|
||||
2. Configure automatic cleanup
|
||||
3. Document deployment process
|
||||
4. Set up staging environment
|
||||
5. Implement blue-green deployments (optional)
|
||||
6. Configure log aggregation
|
||||
7. Set up backup procedures
|
||||
|
||||
@@ -0,0 +1,439 @@
|
||||
---
|
||||
name: mongodb-atlas-checker
|
||||
description: Expert in verifying MongoDB Atlas setup and configuration for backend applications. Checks connection strings, environment variables, database configuration, connection pooling, and ensures proper setup for Next.js and NestJS applications. This skill activates when users need to verify their MongoDB Atlas backend setup is correct.
|
||||
---
|
||||
|
||||
# MongoDB Atlas Checker
|
||||
|
||||
You are an expert in verifying MongoDB Atlas setup and configuration for backend applications. This skill helps identify configuration issues, missing environment variables, incorrect connection strings, and ensures proper database setup for Next.js and NestJS applications.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Verifying MongoDB Atlas backend setup
|
||||
- Checking if connection strings are correctly configured
|
||||
- Validating environment variable setup
|
||||
- Ensuring database connection is properly established
|
||||
- Reviewing MongoDB Atlas configuration
|
||||
- Troubleshooting database connection issues
|
||||
- Auditing database setup before deployment
|
||||
|
||||
## Project Context Discovery
|
||||
|
||||
**Before checking MongoDB Atlas setup, discover the project's context:**
|
||||
|
||||
1. **Scan Project Documentation:**
|
||||
- Check `.agent/SYSTEM/ARCHITECTURE.md` for database architecture
|
||||
- Review existing database patterns
|
||||
- Look for environment variable usage
|
||||
- Check for existing MongoDB integration
|
||||
|
||||
2. **Identify Framework:**
|
||||
- Determine if using Next.js (App Router or Pages Router)
|
||||
- Check if using NestJS backend
|
||||
- Review existing database connection patterns
|
||||
- Check for ORM/ODM usage (Mongoose, TypeORM, Prisma)
|
||||
|
||||
3. **Use Project-Specific Skills:**
|
||||
- Check for `[project]-mongodb-atlas-checker` skill
|
||||
- Review project-specific database patterns
|
||||
- Follow project's configuration standards
|
||||
|
||||
## Checklist: MongoDB Atlas Setup Verification
|
||||
|
||||
### 1. Environment Variables
|
||||
|
||||
**Check for required environment variables:**
|
||||
|
||||
```bash
|
||||
# Required for MongoDB Atlas
|
||||
MONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
# OR
|
||||
DATABASE_URL=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
**Verification Steps:**
|
||||
|
||||
- [ ] Environment variable exists (check `.env.local`, `.env`, or deployment config)
|
||||
- [ ] Variable name is consistent across codebase
|
||||
- [ ] Connection string uses `mongodb+srv://` protocol (required for Atlas)
|
||||
- [ ] Connection string includes authentication credentials
|
||||
- [ ] Connection string includes database name
|
||||
- [ ] Connection string includes query parameters (`retryWrites=true&w=majority`)
|
||||
- [ ] No hardcoded connection strings in source code
|
||||
- [ ] `.env.example` or `.env.template` has placeholder (not real credentials)
|
||||
|
||||
**Common Issues:**
|
||||
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded connection string
|
||||
const mongoUri = 'mongodb+srv://user:pass@cluster.mongodb.net/db';
|
||||
|
||||
// ❌ BAD: Wrong protocol (not supported by Atlas)
|
||||
const mongoUri = 'mongodb://user:pass@cluster.mongodb.net/db';
|
||||
|
||||
// ❌ BAD: Missing database name
|
||||
const mongoUri = 'mongodb+srv://user:pass@cluster.mongodb.net';
|
||||
|
||||
// ✅ GOOD: Environment variable
|
||||
const mongoUri = process.env.MONGODB_URI;
|
||||
```
|
||||
|
||||
### 2. Connection String Format
|
||||
|
||||
**MongoDB Atlas connection strings must:**
|
||||
- Use `mongodb+srv://` protocol (not `mongodb://`)
|
||||
- Include username and password
|
||||
- Include cluster hostname (e.g., `cluster0.xxxxx.mongodb.net`)
|
||||
- Include database name
|
||||
- Include query parameters for production readiness
|
||||
|
||||
**Valid Format:**
|
||||
```
|
||||
mongodb+srv://<username>:<password>@<cluster-host>/<database>?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
**Check for:**
|
||||
- [ ] Protocol is `mongodb+srv://`
|
||||
- [ ] Username and password are URL-encoded if they contain special characters
|
||||
- [ ] Cluster hostname is correct (from Atlas dashboard)
|
||||
- [ ] Database name is specified
|
||||
- [ ] Query parameters include `retryWrites=true&w=majority`
|
||||
- [ ] Optional: `appName` parameter for monitoring
|
||||
- [ ] Optional: `maxPoolSize` for connection pooling
|
||||
|
||||
**Example with all parameters:**
|
||||
```
|
||||
mongodb+srv://user:pass@cluster0.xxxxx.mongodb.net/mydb?retryWrites=true&w=majority&appName=MyApp&maxPoolSize=10
|
||||
```
|
||||
|
||||
### 3. Database Driver Installation
|
||||
|
||||
**Check if MongoDB driver is installed:**
|
||||
|
||||
**For Mongoose (ODM):**
|
||||
```bash
|
||||
# Check package.json
|
||||
npm list mongoose
|
||||
# or
|
||||
pnpm list mongoose
|
||||
```
|
||||
|
||||
**For Native MongoDB Driver:**
|
||||
```bash
|
||||
npm list mongodb
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `mongoose` or `mongodb` package is installed
|
||||
- [ ] Version is compatible with MongoDB Atlas
|
||||
- [ ] Package is listed in `package.json` dependencies (not devDependencies for production)
|
||||
|
||||
### 4. Connection Setup
|
||||
|
||||
**Next.js (App Router or Pages Router):**
|
||||
|
||||
**Check for proper connection pattern:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: Singleton pattern for Next.js
|
||||
// lib/mongodb.ts or utils/mongodb.ts
|
||||
import mongoose from 'mongoose';
|
||||
|
||||
const MONGODB_URI = process.env.MONGODB_URI!;
|
||||
|
||||
if (!MONGODB_URI) {
|
||||
throw new Error('Please define MONGODB_URI environment variable');
|
||||
}
|
||||
|
||||
interface MongooseCache {
|
||||
conn: typeof mongoose | null;
|
||||
promise: Promise<typeof mongoose> | null;
|
||||
}
|
||||
|
||||
declare global {
|
||||
var mongoose: MongooseCache | undefined;
|
||||
}
|
||||
|
||||
let cached: MongooseCache = global.mongoose || { conn: null, promise: null };
|
||||
|
||||
if (!global.mongoose) {
|
||||
global.mongoose = cached;
|
||||
}
|
||||
|
||||
async function connectDB() {
|
||||
if (cached.conn) {
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
if (!cached.promise) {
|
||||
const opts = {
|
||||
bufferCommands: false,
|
||||
};
|
||||
|
||||
cached.promise = mongoose.connect(MONGODB_URI, opts).then((mongoose) => {
|
||||
return mongoose;
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
cached.conn = await cached.promise;
|
||||
} catch (e) {
|
||||
cached.promise = null;
|
||||
throw e;
|
||||
}
|
||||
|
||||
return cached.conn;
|
||||
}
|
||||
|
||||
export default connectDB;
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Connection uses singleton pattern (prevents multiple connections in Next.js)
|
||||
- [ ] Connection is cached globally (for Next.js serverless functions)
|
||||
- [ ] Error handling is implemented
|
||||
- [ ] Connection options are configured (bufferCommands: false recommended)
|
||||
- [ ] Connection is called before database operations
|
||||
|
||||
**NestJS:**
|
||||
|
||||
**Check for MongooseModule configuration:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: NestJS MongooseModule
|
||||
// app.module.ts
|
||||
import { Module } from '@nestjs/common';
|
||||
import { MongooseModule } from '@nestjs/mongoose';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
MongooseModule.forRoot(process.env.MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
}),
|
||||
],
|
||||
})
|
||||
export class AppModule {}
|
||||
```
|
||||
|
||||
**Or with connection options:**
|
||||
|
||||
```typescript
|
||||
MongooseModule.forRoot(process.env.MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
maxPoolSize: 10,
|
||||
serverSelectionTimeoutMS: 5000,
|
||||
socketTimeoutMS: 45000,
|
||||
})
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `@nestjs/mongoose` package is installed
|
||||
- [ ] `MongooseModule.forRoot()` is configured in root module
|
||||
- [ ] Connection string comes from environment variable
|
||||
- [ ] Connection options are set appropriately
|
||||
- [ ] Error handling is in place
|
||||
|
||||
### 5. Connection Options
|
||||
|
||||
**Recommended connection options for MongoDB Atlas:**
|
||||
|
||||
```typescript
|
||||
{
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
maxPoolSize: 10, // Connection pool size
|
||||
serverSelectionTimeoutMS: 5000, // Timeout for server selection
|
||||
socketTimeoutMS: 45000, // Socket timeout
|
||||
connectTimeoutMS: 10000, // Connection timeout
|
||||
bufferCommands: false, // Disable mongoose buffering
|
||||
bufferMaxEntries: 0, // Disable mongoose buffering
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] `retryWrites: true` is set (required for Atlas)
|
||||
- [ ] `w: 'majority'` is set (write concern)
|
||||
- [ ] Connection pool size is appropriate for your use case
|
||||
- [ ] Timeouts are configured appropriately
|
||||
- [ ] Buffer commands is disabled for serverless (Next.js)
|
||||
|
||||
### 6. Error Handling
|
||||
|
||||
**Check for proper error handling:**
|
||||
|
||||
```typescript
|
||||
// ✅ GOOD: Error handling
|
||||
try {
|
||||
await connectDB();
|
||||
// Database operations
|
||||
} catch (error) {
|
||||
console.error('MongoDB connection error:', error);
|
||||
// Handle error appropriately
|
||||
throw error;
|
||||
}
|
||||
```
|
||||
|
||||
**Verification:**
|
||||
- [ ] Connection errors are caught and handled
|
||||
- [ ] Error messages are logged appropriately
|
||||
- [ ] Application doesn't crash on connection failure
|
||||
- [ ] Retry logic is implemented if needed
|
||||
- [ ] Error handling is consistent across the codebase
|
||||
|
||||
### 7. Database Name Configuration
|
||||
|
||||
**Check if database name is correctly specified:**
|
||||
|
||||
- [ ] Database name is in connection string
|
||||
- [ ] Database name matches your application's needs
|
||||
- [ ] Database name doesn't contain special characters
|
||||
- [ ] Database name is consistent across environments (dev/staging/prod)
|
||||
|
||||
### 8. SSL/TLS Configuration
|
||||
|
||||
**MongoDB Atlas requires SSL/TLS by default:**
|
||||
|
||||
- [ ] Connection string doesn't explicitly disable SSL (Atlas requires it)
|
||||
- [ ] No `ssl=false` in connection string
|
||||
- [ ] TLS/SSL is enabled by default with `mongodb+srv://`
|
||||
|
||||
### 9. Network Access
|
||||
|
||||
**Check Atlas Network Access settings:**
|
||||
|
||||
- [ ] IP whitelist includes your deployment IPs
|
||||
- [ ] For development: `0.0.0.0/0` allows all IPs (not recommended for production)
|
||||
- [ ] For production: Specific IPs or VPC peering configured
|
||||
- [ ] Network access rules are documented
|
||||
|
||||
### 10. Database User Configuration
|
||||
|
||||
**Check Atlas Database User settings:**
|
||||
|
||||
- [ ] Database user exists in Atlas
|
||||
- [ ] User has appropriate permissions (read/write for application database)
|
||||
- [ ] Password is strong and secure
|
||||
- [ ] User credentials match connection string
|
||||
- [ ] User is not using admin credentials for application
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### Issue 1: Connection String Not Found
|
||||
|
||||
**Problem:** `MONGODB_URI` environment variable is missing
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
# Add to .env.local (Next.js) or .env (NestJS)
|
||||
MONGODB_URI=mongodb+srv://username:password@cluster.mongodb.net/database?retryWrites=true&w=majority
|
||||
```
|
||||
|
||||
### Issue 2: Wrong Protocol
|
||||
|
||||
**Problem:** Using `mongodb://` instead of `mongodb+srv://`
|
||||
|
||||
**Solution:** Change to `mongodb+srv://` (required for Atlas)
|
||||
|
||||
### Issue 3: Multiple Connections in Next.js
|
||||
|
||||
**Problem:** Creating new connection on each API call
|
||||
|
||||
**Solution:** Use singleton pattern to cache connection (see Connection Setup section)
|
||||
|
||||
### Issue 4: Connection Timeout
|
||||
|
||||
**Problem:** Connection times out
|
||||
|
||||
**Solution:**
|
||||
- Check network access in Atlas dashboard
|
||||
- Verify IP whitelist
|
||||
- Increase `connectTimeoutMS` and `serverSelectionTimeoutMS`
|
||||
- Check firewall settings
|
||||
|
||||
### Issue 5: Authentication Failed
|
||||
|
||||
**Problem:** Username/password incorrect
|
||||
|
||||
**Solution:**
|
||||
- Verify credentials in Atlas dashboard
|
||||
- Check if password contains special characters (needs URL encoding)
|
||||
- Verify database user exists and has permissions
|
||||
|
||||
## Verification Script
|
||||
|
||||
**Create a test script to verify connection:**
|
||||
|
||||
```typescript
|
||||
// scripts/test-mongodb-connection.ts
|
||||
import mongoose from 'mongoose';
|
||||
|
||||
const MONGODB_URI = process.env.MONGODB_URI;
|
||||
|
||||
if (!MONGODB_URI) {
|
||||
console.error('❌ MONGODB_URI environment variable is missing');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function testConnection() {
|
||||
try {
|
||||
await mongoose.connect(MONGODB_URI, {
|
||||
retryWrites: true,
|
||||
w: 'majority',
|
||||
});
|
||||
|
||||
console.log('✅ Successfully connected to MongoDB Atlas');
|
||||
|
||||
// Test a simple operation
|
||||
const collections = await mongoose.connection.db.listCollections().toArray();
|
||||
console.log(`✅ Found ${collections.length} collections`);
|
||||
|
||||
await mongoose.disconnect();
|
||||
console.log('✅ Connection closed');
|
||||
process.exit(0);
|
||||
} catch (error) {
|
||||
console.error('❌ MongoDB connection error:', error);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
testConnection();
|
||||
```
|
||||
|
||||
**Run the test:**
|
||||
```bash
|
||||
# Load environment variables and run
|
||||
node -r dotenv/config scripts/test-mongodb-connection.ts
|
||||
# or
|
||||
ts-node scripts/test-mongodb-connection.ts
|
||||
```
|
||||
|
||||
## Summary Checklist
|
||||
|
||||
Before considering MongoDB Atlas setup complete, verify:
|
||||
|
||||
- [ ] `MONGODB_URI` environment variable exists and is correct
|
||||
- [ ] Connection string uses `mongodb+srv://` protocol
|
||||
- [ ] Connection string includes database name
|
||||
- [ ] MongoDB driver (mongoose or mongodb) is installed
|
||||
- [ ] Connection setup follows framework best practices
|
||||
- [ ] Connection options are configured appropriately
|
||||
- [ ] Error handling is implemented
|
||||
- [ ] Network access is configured in Atlas
|
||||
- [ ] Database user has appropriate permissions
|
||||
- [ ] No hardcoded credentials in source code
|
||||
- [ ] Connection test script passes
|
||||
|
||||
## Next Steps
|
||||
|
||||
After verifying setup:
|
||||
1. Test connection with verification script
|
||||
2. Create initial database schema/models
|
||||
3. Set up database indexes
|
||||
4. Configure connection pooling for production
|
||||
5. Set up monitoring and alerts in Atlas dashboard
|
||||
6. Document connection setup in project documentation
|
||||
|
||||
@@ -0,0 +1,916 @@
|
||||
---
|
||||
name: open-source-checker
|
||||
description: Expert in detecting private information, secrets, API keys, credentials, and sensitive data in codebases before open sourcing. Scans for hardcoded secrets, API keys, passwords, private keys, personal information, database credentials, and other sensitive data that should not be exposed in public repositories. Can also set up git hooks and pre-commit hooks to prevent committing secrets.
|
||||
---
|
||||
|
||||
# Open Source Checker
|
||||
|
||||
You are an expert in detecting private information, secrets, and sensitive data in codebases. This skill helps identify and flag any private information before open sourcing a repository, and can set up automated checks via git hooks to prevent future issues.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
This skill activates automatically when you're:
|
||||
|
||||
- Preparing to open source a repository
|
||||
- Reviewing code for exposed secrets
|
||||
- Auditing codebase for sensitive data
|
||||
- Checking for hardcoded credentials
|
||||
- Validating that no private information is committed
|
||||
- Reviewing pull requests for secrets
|
||||
- Performing security audits before public release
|
||||
|
||||
## What to Check For
|
||||
|
||||
### 1. API Keys and Tokens
|
||||
|
||||
**Common patterns:**
|
||||
- API keys (OpenAI, Stripe, AWS, Google, etc.)
|
||||
- Authentication tokens
|
||||
- OAuth tokens
|
||||
- JWT secrets
|
||||
- Session keys
|
||||
- Webhook secrets
|
||||
|
||||
**Patterns to detect:**
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded API keys
|
||||
const apiKey = 'sk-1234567890abcdef';
|
||||
const stripeKey = 'sk_live_...';
|
||||
const awsKey = 'AKIAIOSFODNN7EXAMPLE';
|
||||
|
||||
// ✅ GOOD: Environment variables
|
||||
const apiKey = process.env.API_KEY;
|
||||
const stripeKey = process.env.STRIPE_SECRET_KEY;
|
||||
```
|
||||
|
||||
**Common locations:**
|
||||
- Configuration files
|
||||
- Source code files
|
||||
- Environment files (`.env` files that might be committed)
|
||||
- Test files
|
||||
- Documentation files
|
||||
- Example files
|
||||
|
||||
### 2. Database Credentials
|
||||
|
||||
**Check for:**
|
||||
- Database connection strings
|
||||
- Usernames and passwords
|
||||
- MongoDB URIs
|
||||
- PostgreSQL connection strings
|
||||
- Redis credentials
|
||||
- Database host addresses
|
||||
|
||||
**Patterns:**
|
||||
```typescript
|
||||
// ❌ BAD: Hardcoded credentials
|
||||
const mongoUri = 'mongodb://user:password@host:27017/db';
|
||||
const dbPassword = 'mySecretPassword123';
|
||||
|
||||
// ✅ GOOD: Environment variables
|
||||
const mongoUri = process.env.MONGODB_URI;
|
||||
```
|
||||
|
||||
### 3. Private Keys and Certificates
|
||||
|
||||
**Check for:**
|
||||
- SSH private keys
|
||||
- SSL/TLS certificates
|
||||
- Private key files (`.pem`, `.key`, `.p12`)
|
||||
- Certificate files
|
||||
- Signing keys
|
||||
|
||||
**Files to check:**
|
||||
- `*.pem`, `*.key`, `*.p12`, `*.pfx`
|
||||
- `id_rsa`, `id_dsa`, `id_ecdsa`
|
||||
- `*.crt`, `*.cer`, `*.cert`
|
||||
|
||||
### 4. Personal Information
|
||||
|
||||
**Check for:**
|
||||
- Email addresses
|
||||
- Phone numbers
|
||||
- Physical addresses
|
||||
- Personal names
|
||||
- Social security numbers
|
||||
- Credit card numbers
|
||||
- Bank account numbers
|
||||
|
||||
**Patterns:**
|
||||
```typescript
|
||||
// ❌ BAD: Personal information
|
||||
const adminEmail = 'john.doe@example.com';
|
||||
const phone = '+1-555-123-4567';
|
||||
|
||||
// ✅ GOOD: Placeholder or environment variable
|
||||
const adminEmail = process.env.ADMIN_EMAIL;
|
||||
```
|
||||
|
||||
### 5. Environment Files
|
||||
|
||||
**Check for:**
|
||||
- `.env` files (should be in `.gitignore`)
|
||||
- `.env.local`, `.env.production`
|
||||
- Files containing actual secrets (not `.env.example`)
|
||||
|
||||
**Verify:**
|
||||
- `.env` is in `.gitignore`
|
||||
- Only `.env.example` is committed (with placeholder values)
|
||||
- No actual secrets in any committed `.env` files
|
||||
|
||||
### 6. Configuration Files with Secrets
|
||||
|
||||
**Check:**
|
||||
- `config.json`, `config.js`, `config.ts`
|
||||
- `settings.json`, `settings.js`
|
||||
- `secrets.json`, `secrets.js`
|
||||
- Any config file with hardcoded values
|
||||
|
||||
### 7. Comments and Documentation
|
||||
|
||||
**Check for:**
|
||||
- Secrets in code comments
|
||||
- API keys in README files
|
||||
- Credentials in documentation
|
||||
- Test credentials that might be real
|
||||
|
||||
### 8. Git History
|
||||
|
||||
**Check for:**
|
||||
- Secrets in git history (even if removed)
|
||||
- Committed `.env` files in history
|
||||
- Secrets in old commits
|
||||
|
||||
**Commands to check:**
|
||||
```bash
|
||||
# Search git history for secrets
|
||||
git log --all --full-history --source -S "sk-" -- "*.ts" "*.js" "*.json"
|
||||
git log --all --full-history --source -S "password" -- "*.env"
|
||||
```
|
||||
|
||||
## Scanning Workflow
|
||||
|
||||
### Phase 1: File System Scan
|
||||
|
||||
**1.1 Check for Common Secret Files**
|
||||
|
||||
```bash
|
||||
# Find potential secret files
|
||||
find . -name "*.env" -o -name "*.key" -o -name "*.pem" -o -name "id_rsa*"
|
||||
find . -name "secrets.*" -o -name "*secret*"
|
||||
find . -name ".env*" ! -name ".env.example"
|
||||
```
|
||||
|
||||
**1.2 Check .gitignore**
|
||||
|
||||
```bash
|
||||
# Verify .env is ignored
|
||||
cat .gitignore | grep -E "\.env|secrets|\.key|\.pem"
|
||||
```
|
||||
|
||||
**1.3 Scan for Common Patterns**
|
||||
|
||||
```bash
|
||||
# Search for API key patterns
|
||||
grep -r "sk-[a-zA-Z0-9]" --include="*.ts" --include="*.js" --include="*.json"
|
||||
grep -r "AKIA[0-9A-Z]" --include="*.ts" --include="*.js"
|
||||
grep -r "sk_live_" --include="*.ts" --include="*.js"
|
||||
```
|
||||
|
||||
### Phase 2: Code Pattern Analysis
|
||||
|
||||
**2.1 Search for Hardcoded Secrets**
|
||||
|
||||
Look for:
|
||||
- String literals that look like API keys
|
||||
- Hardcoded passwords
|
||||
- Connection strings with credentials
|
||||
- Token values in code
|
||||
|
||||
**2.2 Check Configuration Files**
|
||||
|
||||
Review:
|
||||
- All config files for hardcoded values
|
||||
- Environment variable usage (should use `process.env`)
|
||||
- Default values that might be secrets
|
||||
|
||||
**2.3 Review Test Files**
|
||||
|
||||
Check:
|
||||
- Test credentials (should be mocks, not real)
|
||||
- Test API keys (should be fake/test keys)
|
||||
- Test database connections
|
||||
|
||||
### Phase 3: Content Analysis
|
||||
|
||||
**3.1 Check Documentation**
|
||||
|
||||
- README files
|
||||
- Documentation files
|
||||
- Comments in code
|
||||
- Example code snippets
|
||||
|
||||
**3.2 Check Example Files**
|
||||
|
||||
- `.env.example` should have placeholders
|
||||
- Example configs should not have real values
|
||||
- Sample code should not include real keys
|
||||
|
||||
### Phase 4: Git History Check
|
||||
|
||||
**⚠️ CRITICAL: Even if secrets are removed from current files, they remain in git history forever unless explicitly removed.**
|
||||
|
||||
**4.1 Comprehensive Git History Scan**
|
||||
|
||||
**Search for API Keys in History:**
|
||||
|
||||
```bash
|
||||
# Search entire git history for OpenAI API keys
|
||||
git log --all --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
|
||||
# Search for AWS keys
|
||||
git log --all --full-history -p -S "AKIA" | grep -B 5 -A 5 "AKIA"
|
||||
|
||||
# Search for Stripe keys
|
||||
git log --all --full-history -p -S "sk_live_" | grep -B 5 -A 5 "sk_live_"
|
||||
|
||||
# Search for GitHub tokens
|
||||
git log --all --full-history -p -S "ghp_" | grep -B 5 -A 5 "ghp_"
|
||||
|
||||
# Search for passwords
|
||||
git log --all --full-history -p -S "password" | grep -B 5 -A 5 "password"
|
||||
|
||||
# Search for connection strings
|
||||
git log --all --full-history -p -S "mongodb://" | grep -B 5 -A 5 "mongodb://"
|
||||
git log --all --full-history -p -S "postgres://" | grep -B 5 -A 5 "postgres://"
|
||||
```
|
||||
|
||||
**Search All Branches and Tags:**
|
||||
|
||||
```bash
|
||||
# Check all branches (including remote)
|
||||
git log --all --branches --tags --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
|
||||
# Check specific branches
|
||||
git log origin/main origin/develop --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
```
|
||||
|
||||
**Search in Deleted Files:**
|
||||
|
||||
```bash
|
||||
# Find commits that deleted files containing secrets
|
||||
git log --all --full-history --diff-filter=D --summary | grep -E "\.env|secrets|\.key"
|
||||
|
||||
# Check what was in deleted files
|
||||
git log --all --full-history --diff-filter=D -- "*.env" | grep -A 10 "delete mode"
|
||||
```
|
||||
|
||||
**Search in Specific File Types:**
|
||||
|
||||
```bash
|
||||
# Search history of .env files
|
||||
git log --all --full-history -p -- "*.env" | grep -E "(sk-|password|AKIA)"
|
||||
|
||||
# Search history of config files
|
||||
git log --all --full-history -p -- "config.*" | grep -E "(sk-|password|AKIA)"
|
||||
|
||||
# Search history of all JavaScript/TypeScript files
|
||||
git log --all --full-history -p -- "*.{js,ts}" | grep -E "(sk-|password|AKIA)"
|
||||
```
|
||||
|
||||
**4.2 Using Tools to Scan Git History**
|
||||
|
||||
**gitleaks (Recommended for Git History):**
|
||||
|
||||
```bash
|
||||
# Install gitleaks
|
||||
brew install gitleaks
|
||||
|
||||
# Scan entire git history (all branches, all commits)
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
|
||||
# Scan specific branch
|
||||
gitleaks detect --source . --verbose --log-opts="--all --branches=main"
|
||||
|
||||
# Scan with custom config
|
||||
gitleaks detect --source . --verbose --log-opts="--all" --config-path=.gitleaks.toml
|
||||
```
|
||||
|
||||
**truffleHog (Scans Git History):**
|
||||
|
||||
```bash
|
||||
# Install truffleHog
|
||||
pip install truffleHog
|
||||
|
||||
# Scan entire git history
|
||||
trufflehog --regex --entropy=False git file://.
|
||||
|
||||
# Scan specific branch
|
||||
trufflehog --regex --entropy=False git file://. --branch=main
|
||||
```
|
||||
|
||||
**git-secrets (History Scan):**
|
||||
|
||||
```bash
|
||||
# Install git-secrets
|
||||
brew install git-secrets
|
||||
|
||||
# Scan entire history
|
||||
git secrets --scan-history
|
||||
|
||||
# Scan specific commit range
|
||||
git secrets --scan-history HEAD~10..HEAD
|
||||
```
|
||||
|
||||
**4.3 Check for Secrets in Merge Commits**
|
||||
|
||||
```bash
|
||||
# Search merge commits specifically
|
||||
git log --all --merges --full-history -p -S "sk-" | grep -B 5 -A 5 "sk-"
|
||||
|
||||
# Check merge commits for .env files
|
||||
git log --all --merges --full-history --diff-filter=M -- "*.env"
|
||||
```
|
||||
|
||||
**4.4 Check Stashed Changes**
|
||||
|
||||
```bash
|
||||
# List all stashes
|
||||
git stash list
|
||||
|
||||
# Check each stash for secrets
|
||||
git stash show -p stash@{0} | grep -E "(sk-|password|AKIA)"
|
||||
git stash show -p stash@{1} | grep -E "(sk-|password|AKIA)"
|
||||
```
|
||||
|
||||
**4.5 Automated Git History Scan Script**
|
||||
|
||||
Create a script to scan entire history:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scan-git-history.sh
|
||||
|
||||
echo "🔍 Scanning entire git history for secrets..."
|
||||
|
||||
# Patterns to search for
|
||||
PATTERNS=(
|
||||
"sk-[a-zA-Z0-9]"
|
||||
"AKIA[0-9A-Z]"
|
||||
"sk_live_"
|
||||
"sk_test_"
|
||||
"ghp_"
|
||||
"mongodb://.*:.*@"
|
||||
"postgres://.*:.*@"
|
||||
)
|
||||
|
||||
for pattern in "${PATTERNS[@]}"; do
|
||||
echo "Checking for pattern: $pattern"
|
||||
git log --all --full-history -p -S "$pattern" | grep -B 5 -A 5 "$pattern" && {
|
||||
echo "⚠️ Found matches for: $pattern"
|
||||
}
|
||||
done
|
||||
|
||||
# Check for .env files in history
|
||||
echo "Checking for .env files in history..."
|
||||
git log --all --full-history --name-only --diff-filter=A | grep -E "\.env$" | sort -u
|
||||
|
||||
# Use gitleaks if available
|
||||
if command -v gitleaks &> /dev/null; then
|
||||
echo "Running gitleaks on git history..."
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
fi
|
||||
```
|
||||
|
||||
**4.6 Cleaning Git History (If Secrets Found)**
|
||||
|
||||
**⚠️ WARNING: These operations rewrite git history. Coordinate with your team first.**
|
||||
|
||||
**Option 1: Using git-filter-repo (Recommended)**
|
||||
|
||||
```bash
|
||||
# Install git-filter-repo
|
||||
pip install git-filter-repo
|
||||
|
||||
# Remove secrets from entire history
|
||||
git filter-repo --invert-paths --path "file-with-secret.txt"
|
||||
git filter-repo --replace-text <(echo "sk-OLD-KEY==>sk-REMOVED")
|
||||
|
||||
# Remove .env files from history
|
||||
git filter-repo --invert-paths --path-glob "*.env"
|
||||
```
|
||||
|
||||
**Option 2: Using BFG Repo-Cleaner**
|
||||
|
||||
```bash
|
||||
# Install BFG
|
||||
brew install bfg
|
||||
|
||||
# Remove secrets file from history
|
||||
bfg --delete-files secrets.json
|
||||
|
||||
# Replace secrets in history
|
||||
echo "sk-OLD-KEY==>sk-REMOVED" > secrets-replacements.txt
|
||||
bfg --replace-text secrets-replacements.txt
|
||||
```
|
||||
|
||||
**Option 3: Fresh Repository (If History Too Contaminated)**
|
||||
|
||||
If the history is too contaminated, consider:
|
||||
1. Create a fresh repository
|
||||
2. Copy current clean state
|
||||
3. Start fresh history
|
||||
|
||||
```bash
|
||||
# Create fresh repo
|
||||
git checkout --orphan fresh-start
|
||||
git add .
|
||||
git commit -m "Initial commit (cleaned history)"
|
||||
git branch -D main # Delete old main
|
||||
git branch -m main # Rename current to main
|
||||
git push -f origin main # Force push (coordinate with team!)
|
||||
```
|
||||
|
||||
**4.7 Verify History is Clean**
|
||||
|
||||
After cleaning, verify:
|
||||
|
||||
```bash
|
||||
# Re-scan history
|
||||
gitleaks detect --source . --verbose --log-opts="--all"
|
||||
|
||||
# Check specific patterns
|
||||
git log --all --full-history -p -S "sk-" | grep "sk-"
|
||||
|
||||
# Verify no .env files in history
|
||||
git log --all --full-history --name-only | grep "\.env$"
|
||||
```
|
||||
|
||||
**4.8 Best Practices for Git History**
|
||||
|
||||
1. **Scan before open sourcing**: Always scan entire history before making repo public
|
||||
2. **Use tools**: Automated tools like gitleaks are more thorough than manual searches
|
||||
3. **Check all branches**: Secrets might be in feature branches
|
||||
4. **Check tags**: Tags preserve old commits
|
||||
5. **Coordinate cleanup**: If cleaning history, coordinate with all contributors
|
||||
6. **Rotate exposed secrets**: If secrets were in history, rotate them immediately
|
||||
7. **Set up hooks**: Prevent future commits with pre-commit hooks
|
||||
|
||||
## Common Patterns to Detect
|
||||
|
||||
### API Key Patterns
|
||||
|
||||
```typescript
|
||||
// OpenAI
|
||||
sk-[a-zA-Z0-9]{32,}
|
||||
|
||||
// AWS
|
||||
AKIA[0-9A-Z]{16}
|
||||
|
||||
// Stripe
|
||||
sk_live_[a-zA-Z0-9]{24,}
|
||||
sk_test_[a-zA-Z0-9]{24,}
|
||||
|
||||
// GitHub
|
||||
ghp_[a-zA-Z0-9]{36}
|
||||
|
||||
// Generic
|
||||
[a-zA-Z0-9_-]{20,} // Long alphanumeric strings
|
||||
```
|
||||
|
||||
### Password Patterns
|
||||
|
||||
```typescript
|
||||
// Common patterns
|
||||
password\s*[:=]\s*['"][^'"]+['"]
|
||||
pwd\s*[:=]\s*['"][^'"]+['"]
|
||||
pass\s*[:=]\s*['"][^'"]+['"]
|
||||
```
|
||||
|
||||
### Connection String Patterns
|
||||
|
||||
```typescript
|
||||
// MongoDB
|
||||
mongodb://[^:]+:[^@]+@
|
||||
mongodb\+srv://[^:]+:[^@]+@
|
||||
|
||||
// PostgreSQL
|
||||
postgres://[^:]+:[^@]+@
|
||||
postgresql://[^:]+:[^@]+@
|
||||
|
||||
// MySQL
|
||||
mysql://[^:]+:[^@]+@
|
||||
|
||||
// Redis
|
||||
redis://[^:]+:[^@]+@
|
||||
```
|
||||
|
||||
### Email Patterns
|
||||
|
||||
```typescript
|
||||
// Email addresses
|
||||
[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
|
||||
```
|
||||
|
||||
## Automated Tools
|
||||
|
||||
### Recommended Tools
|
||||
|
||||
**1. git-secrets**
|
||||
```bash
|
||||
# Install
|
||||
brew install git-secrets
|
||||
|
||||
# Setup
|
||||
git secrets --install
|
||||
git secrets --register-aws
|
||||
|
||||
# Scan
|
||||
git secrets --scan
|
||||
```
|
||||
|
||||
**2. truffleHog**
|
||||
```bash
|
||||
# Install
|
||||
pip install truffleHog
|
||||
|
||||
# Scan
|
||||
trufflehog --regex --entropy=False .
|
||||
```
|
||||
|
||||
**3. detect-secrets**
|
||||
```bash
|
||||
# Install
|
||||
pip install detect-secrets
|
||||
|
||||
# Scan
|
||||
detect-secrets scan --all-files
|
||||
```
|
||||
|
||||
**4. gitleaks**
|
||||
```bash
|
||||
# Install
|
||||
brew install gitleaks
|
||||
|
||||
# Scan
|
||||
gitleaks detect --source . --verbose
|
||||
```
|
||||
|
||||
## Git Hooks and Pre-Commit Hooks
|
||||
|
||||
Setting up git hooks prevents committing secrets before they enter the repository. This is the best way to catch issues early.
|
||||
|
||||
### Pre-Commit Hook Setup
|
||||
|
||||
**1. Using git-secrets (Recommended)**
|
||||
|
||||
```bash
|
||||
# Install git-secrets
|
||||
brew install git-secrets
|
||||
|
||||
# Initialize in your repository
|
||||
cd /path/to/your/repo
|
||||
git secrets --install
|
||||
|
||||
# Register AWS patterns (or other providers)
|
||||
git secrets --register-aws
|
||||
|
||||
# Add custom patterns
|
||||
git secrets --add 'sk-[a-zA-Z0-9]{32,}'
|
||||
git secrets --add 'AKIA[0-9A-Z]{16}'
|
||||
git secrets --add 'sk_live_[a-zA-Z0-9]{24,}'
|
||||
|
||||
# Test the hook
|
||||
git secrets --scan
|
||||
```
|
||||
|
||||
**2. Using gitleaks**
|
||||
|
||||
```bash
|
||||
# Install gitleaks
|
||||
brew install gitleaks
|
||||
|
||||
# Create pre-commit hook
|
||||
cat > .git/hooks/pre-commit << 'EOF'
|
||||
#!/bin/sh
|
||||
gitleaks detect --staged --verbose
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "❌ gitleaks detected secrets in your changes. Commit aborted."
|
||||
exit 1
|
||||
fi
|
||||
EOF
|
||||
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
**3. Using detect-secrets**
|
||||
|
||||
```bash
|
||||
# Install detect-secrets
|
||||
pip install detect-secrets
|
||||
|
||||
# Create baseline
|
||||
detect-secrets scan > .secrets.baseline
|
||||
|
||||
# Create pre-commit hook
|
||||
cat > .git/hooks/pre-commit << 'EOF'
|
||||
#!/bin/sh
|
||||
detect-secrets scan --baseline .secrets.baseline
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "❌ detect-secrets found new secrets. Commit aborted."
|
||||
exit 1
|
||||
fi
|
||||
EOF
|
||||
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
**4. Using Husky (for Node.js projects)**
|
||||
|
||||
```bash
|
||||
# Install husky
|
||||
npm install --save-dev husky
|
||||
|
||||
# Initialize husky
|
||||
npx husky install
|
||||
|
||||
# Add pre-commit hook
|
||||
npx husky add .husky/pre-commit "gitleaks detect --staged --verbose"
|
||||
```
|
||||
|
||||
**5. Manual Pre-Commit Hook**
|
||||
|
||||
Create `.git/hooks/pre-commit`:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
#
|
||||
# Pre-commit hook to check for secrets
|
||||
#
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo "🔍 Checking for secrets..."
|
||||
|
||||
# Check for common API key patterns
|
||||
if git diff --cached --name-only | xargs grep -E "(sk-[a-zA-Z0-9]{32,}|AKIA[0-9A-Z]{16}|sk_live_[a-zA-Z0-9]{24,})" 2>/dev/null; then
|
||||
echo "${RED}❌ Potential API keys detected in staged files!${NC}"
|
||||
echo "${YELLOW}Please remove secrets before committing.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for .env files
|
||||
if git diff --cached --name-only | grep -E "\.env$" | grep -v "\.env\.example"; then
|
||||
echo "${RED}❌ .env file detected!${NC}"
|
||||
echo "${YELLOW}Please ensure .env files are in .gitignore.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for private keys
|
||||
if git diff --cached --name-only | grep -E "\.(key|pem|p12|pfx)$"; then
|
||||
echo "${RED}❌ Private key file detected!${NC}"
|
||||
echo "${YELLOW}Please ensure private keys are in .gitignore.${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "${GREEN}✅ No secrets detected. Proceeding with commit.${NC}"
|
||||
exit 0
|
||||
```
|
||||
|
||||
Make it executable:
|
||||
```bash
|
||||
chmod +x .git/hooks/pre-commit
|
||||
```
|
||||
|
||||
### Post-Commit Hook (Optional)
|
||||
|
||||
Create `.git/hooks/post-commit` to scan after commit:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
#
|
||||
# Post-commit hook to scan for secrets
|
||||
#
|
||||
|
||||
echo "🔍 Scanning last commit for secrets..."
|
||||
|
||||
# Use gitleaks or your preferred tool
|
||||
gitleaks detect --log-opts="-1" --verbose
|
||||
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "⚠️ Secrets detected in last commit!"
|
||||
echo "Consider amending the commit or using git-filter-repo to remove secrets."
|
||||
fi
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
**GitHub Actions Example:**
|
||||
|
||||
```yaml
|
||||
name: Secret Scanning
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
scan:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
fetch-depth: 0 # Full history for gitleaks
|
||||
|
||||
- name: Run Gitleaks
|
||||
uses: gitleaks/gitleaks-action@v2
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
```
|
||||
|
||||
**GitLab CI Example:**
|
||||
|
||||
```yaml
|
||||
secret-scan:
|
||||
image: zricethezav/gitleaks:latest
|
||||
script:
|
||||
- gitleaks detect --source . --verbose --no-git
|
||||
allow_failure: false
|
||||
```
|
||||
|
||||
### Hook Best Practices
|
||||
|
||||
1. **Fail fast**: Hooks should exit with non-zero code to prevent commits
|
||||
2. **Clear messages**: Provide actionable error messages
|
||||
3. **Fast execution**: Keep hooks fast to avoid slowing down workflow
|
||||
4. **Team-wide**: Ensure all team members have hooks installed
|
||||
5. **CI/CD backup**: Don't rely solely on hooks; use CI/CD as backup
|
||||
6. **Regular updates**: Update patterns and tools regularly
|
||||
|
||||
## Checklist
|
||||
|
||||
### Before Open Sourcing
|
||||
|
||||
- [ ] No hardcoded API keys in code
|
||||
- [ ] No database credentials in code
|
||||
- [ ] No private keys or certificates committed
|
||||
- [ ] `.env` files in `.gitignore`
|
||||
- [ ] Only `.env.example` committed (with placeholders)
|
||||
- [ ] **Git history scanned for secrets (CRITICAL)**
|
||||
- [ ] All branches checked for secrets
|
||||
- [ ] All tags checked for secrets
|
||||
- [ ] Deleted files checked for secrets
|
||||
- [ ] Merge commits checked for secrets
|
||||
- [ ] No secrets found in git history
|
||||
- [ ] Git history cleaned if secrets were found
|
||||
- [ ] No personal information in code
|
||||
- [ ] No real credentials in test files
|
||||
- [ ] No secrets in documentation
|
||||
- [ ] Configuration files use environment variables
|
||||
- [ ] All sensitive files in `.gitignore`
|
||||
|
||||
### Files to Verify
|
||||
|
||||
- [ ] `.env` - Should be ignored
|
||||
- [ ] `.env.local` - Should be ignored
|
||||
- [ ] `.env.production` - Should be ignored
|
||||
- [ ] `config.json` - Should not contain secrets
|
||||
- [ ] `secrets.json` - Should not exist or be ignored
|
||||
- [ ] `*.key`, `*.pem` - Should be ignored
|
||||
- [ ] `id_rsa*` - Should be ignored
|
||||
- [ ] README.md - Should not contain real secrets
|
||||
- [ ] Documentation files - Should not contain secrets
|
||||
|
||||
## Output Format
|
||||
|
||||
When checking for private information:
|
||||
|
||||
```
|
||||
🔍 PRIVATE INFORMATION SCAN REPORT
|
||||
|
||||
Repository: [repo-name]
|
||||
Date: [date]
|
||||
Scanner: [tool/agent]
|
||||
|
||||
📊 SUMMARY
|
||||
- Critical issues: 3
|
||||
- Warnings: 5
|
||||
- Files scanned: 150
|
||||
- Patterns checked: 12
|
||||
- Git history scanned: Yes
|
||||
- Branches checked: 5
|
||||
- Commits in history: 1,234
|
||||
|
||||
🚨 CRITICAL ISSUES
|
||||
|
||||
1. Hardcoded API Key Found
|
||||
File: src/config/api.ts:23
|
||||
Line: const apiKey = 'sk-1234567890abcdef';
|
||||
Issue: OpenAI API key exposed in code
|
||||
Fix: Move to environment variable
|
||||
Severity: CRITICAL
|
||||
Action: Remove immediately and rotate key
|
||||
|
||||
2. Database Credentials in Code
|
||||
File: src/database/config.ts:12
|
||||
Line: const mongoUri = 'mongodb://user:password@host:27017/db';
|
||||
Issue: Database credentials exposed
|
||||
Fix: Use environment variable
|
||||
Severity: CRITICAL
|
||||
Action: Remove and change database password
|
||||
|
||||
3. Secrets Found in Git History
|
||||
Commit: abc123def (2024-01-15)
|
||||
File: config/secrets.json (now deleted)
|
||||
Issue: API key was committed and then deleted, but still in history
|
||||
Fix: Clean git history using git-filter-repo
|
||||
Severity: CRITICAL
|
||||
Action: Remove from history and rotate exposed keys
|
||||
|
||||
⚠️ WARNINGS
|
||||
|
||||
1. .env File Not in .gitignore
|
||||
File: .env
|
||||
Issue: Environment file may be committed
|
||||
Fix: Add .env to .gitignore
|
||||
Severity: HIGH
|
||||
|
||||
2. Potential API Key in Comment
|
||||
File: src/utils/helpers.ts:45
|
||||
Line: // API key: sk-test-12345
|
||||
Issue: Comment contains what looks like an API key
|
||||
Fix: Remove comment
|
||||
Severity: MEDIUM
|
||||
|
||||
[... more issues ...]
|
||||
|
||||
✅ SAFE FILES
|
||||
|
||||
- ✅ .env.example contains only placeholders
|
||||
- ✅ All config files use environment variables
|
||||
- ✅ No secrets in documentation
|
||||
- ✅ Test files use mock credentials
|
||||
|
||||
📜 GIT HISTORY SCAN RESULTS
|
||||
|
||||
- ✅ Current files: No secrets detected
|
||||
- ⚠️ Git history: 2 secrets found in old commits
|
||||
- ✅ All branches scanned: main, develop, feature/*
|
||||
- ✅ All tags scanned: v1.0.0, v1.1.0
|
||||
- ⚠️ Action required: Clean git history before open sourcing
|
||||
|
||||
💡 RECOMMENDATIONS
|
||||
|
||||
1. Add .env to .gitignore if not already
|
||||
2. Use environment variables for all secrets
|
||||
3. Rotate any exposed API keys
|
||||
4. Clean git history if secrets were committed
|
||||
5. Set up pre-commit hooks to prevent future commits
|
||||
6. Use secret scanning in CI/CD
|
||||
|
||||
📋 NEXT STEPS
|
||||
|
||||
1. Fix critical issues immediately
|
||||
2. Rotate any exposed credentials
|
||||
3. Clean git history if needed
|
||||
4. Set up automated scanning
|
||||
5. Review and approve before open sourcing
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Never commit secrets**: Always use environment variables
|
||||
2. **Use .env.example**: Provide template with placeholders
|
||||
3. **Rotate exposed secrets**: If secrets were committed, rotate them
|
||||
4. **Clean git history**: Remove secrets from history if committed
|
||||
5. **Automate scanning**: Use pre-commit hooks and CI/CD checks
|
||||
6. **Document requirements**: List required environment variables
|
||||
7. **Use secret management**: Consider services like AWS Secrets Manager
|
||||
8. **Regular audits**: Scan before each release
|
||||
|
||||
## Resources
|
||||
|
||||
### Tools
|
||||
- git-secrets: https://github.com/awslabs/git-secrets
|
||||
- truffleHog: https://github.com/trufflesecurity/trufflehog
|
||||
- detect-secrets: https://github.com/Yelp/detect-secrets
|
||||
- gitleaks: https://github.com/gitleaks/gitleaks
|
||||
|
||||
### Guides
|
||||
- GitHub: Removing sensitive data from a repository
|
||||
- OWASP: Secrets Management Cheat Sheet
|
||||
- Git: Rewriting History
|
||||
|
||||
---
|
||||
|
||||
**When this skill is active**, you will:
|
||||
1. Scan the codebase for private information patterns
|
||||
2. Check for hardcoded secrets and credentials
|
||||
3. Verify .gitignore includes sensitive files
|
||||
4. Review git history for exposed secrets
|
||||
5. Provide actionable recommendations
|
||||
6. Generate a comprehensive report
|
||||
7. Help clean up any found issues before open sourcing
|
||||
|
||||
+1
-1
@@ -21,7 +21,7 @@
|
||||
"license": "MIT",
|
||||
"lint-staged": {
|
||||
"*.md": [
|
||||
"markdownlint --fix"
|
||||
"bunx markdownlint --fix"
|
||||
],
|
||||
"*.sh": [
|
||||
"bash scripts/lint-shellcheck.sh"
|
||||
|
||||
Reference in New Issue
Block a user