mirror of
https://github.com/Jeffallan/claude-skills.git
synced 2026-09-14 18:43:38 +08:00
feat(skills): add django-storages-s3 skill (#218)
Production S3 file storage for Django: STORAGES dict (4.2+), custom public/private backends, presigned URLs, CloudFront, IAM, and testing with moto/InMemoryStorage. Squash-merged locally to resolve count and changelog conflicts with today's main; counts recomputed via update-docs.py (67 skills, 371 references). Co-authored-by: Awais Qureshi <awais.qureshi@arbisoft.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01E6LA4sndtVGyvoXqwYeHoB
This commit is contained in:
Jeff Smolinski
@@ -11,7 +11,7 @@
|
||||
{
|
||||
"name": "fullstack-dev-skills",
|
||||
"source": "./",
|
||||
"description": "66 specialized skills for full-stack development: 12 language experts (Python, TypeScript, Go, Rust, C++, Swift, Kotlin, C#, PHP, Java, SQL, JavaScript), 10 backend frameworks, 6 frontend/mobile, plus infrastructure, DevOps, security, and testing skills. Includes 9 project workflow commands for epic planning, discovery, execution, and retrospectives.",
|
||||
"description": "67 specialized skills for full-stack development: 12 language experts (Python, TypeScript, Go, Rust, C++, Swift, Kotlin, C#, PHP, Java, SQL, JavaScript), 10 backend frameworks, 6 frontend/mobile, plus infrastructure, DevOps, security, and testing skills. Includes 9 project workflow commands for epic planning, discovery, execution, and retrospectives.",
|
||||
"version": "0.4.15",
|
||||
"author": {
|
||||
"name": "jeffallan",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "fullstack-dev-skills",
|
||||
"version": "0.4.15",
|
||||
"description": "Comprehensive skill pack with 66 specialized skills for full-stack developers: 12 language experts (Python, TypeScript, Go, Rust, C++, Swift, Kotlin, C#, PHP, Java, SQL, JavaScript), 10 backend frameworks, 6 frontend/mobile, plus infrastructure, DevOps, security, and testing. Features progressive disclosure architecture for 50% faster loading.",
|
||||
"description": "Comprehensive skill pack with 67 specialized skills for full-stack developers: 12 language experts (Python, TypeScript, Go, Rust, C++, Swift, Kotlin, C#, PHP, Java, SQL, JavaScript), 10 backend frameworks, 6 frontend/mobile, plus infrastructure, DevOps, security, and testing. Features progressive disclosure architecture for 50% faster loading.",
|
||||
"author": {
|
||||
"name": "Jeffallan"
|
||||
},
|
||||
|
||||
@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- New skill: `django-storages-s3` — production AWS S3 file storage for Django with `django-storages` and `boto3`. Covers the Django 4.2+ `STORAGES` dict, public/private custom backends, presigned GET/POST URLs, CloudFront integration, a least-privilege IAM policy, and S3 mocking (`InMemoryStorage`/`moto`). Cross-referenced with `django-expert` (#218)
|
||||
- `ReferencePathChecker` in `scripts/validate-skills.py`: validates that file paths cited in skill markdown (backtick paths and markdown links) resolve relative to the containing file or the skill root. Broken paths previously failed silently when an agent tried to load deferred reference content; this class of bug has now recurred across several releases and is guarded automatically in CI and `make validate`
|
||||
- `devops-engineer`: new `references/gitlab-ci.md` covering GitLab CI/CD best practices (pipeline dedup via `workflow:rules`, `needs:` DAG, cache vs artifacts, CI/CD components, environments, OIDC secrets, runner isolation, MR-widget reporting) plus a routing-table row; the GitLab counterpart to the existing GitHub Actions reference (#219)
|
||||
|
||||
@@ -26,6 +27,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- @vasugarg09 — Fixed broken relative reference paths in `vue-expert-js` and `react-expert` (#225)
|
||||
- @chgreer1070 — Patched docs-site dependency vulnerabilities via `npm audit fix` (#220)
|
||||
- @kasymovpost — GitLab CI/CD best-practices reference for devops-engineer (#219)
|
||||
- @awais786 — New `django-storages-s3` skill: production S3 file storage for Django (#218)
|
||||
|
||||
## [0.4.15] - 2026-05-20
|
||||
|
||||
|
||||
+1
-1
@@ -62,7 +62,7 @@ Verify skills are working:
|
||||
|
||||
### 1. What's Included
|
||||
|
||||
<!-- SKILL_COUNT -->66<!-- /SKILL_COUNT --> skills covering:
|
||||
<!-- SKILL_COUNT -->67<!-- /SKILL_COUNT --> skills covering:
|
||||
- 12 Language Experts (Python, TypeScript, Go, Rust, C++, Swift, Kotlin, C#, PHP, Java, SQL, JavaScript)
|
||||
- 7 Backend Framework Experts (NestJS, Django, FastAPI, Spring Boot, Laravel, Rails, .NET Core)
|
||||
- 7 Frontend & Mobile Experts (React, Next.js, Vue, Angular, React Native, Flutter)
|
||||
|
||||
@@ -33,7 +33,7 @@ For all installation methods and first steps, see the [**Quick Start Guide**](QU
|
||||
|
||||
## Skills
|
||||
|
||||
<!-- SKILL_COUNT -->66<!-- /SKILL_COUNT --> specialized skills across 12 categories covering languages, backend/frontend frameworks, infrastructure, APIs, testing, DevOps, security, data/ML, and platform specialists.
|
||||
<!-- SKILL_COUNT -->67<!-- /SKILL_COUNT --> specialized skills across 12 categories covering languages, backend/frontend frameworks, infrastructure, APIs, testing, DevOps, security, data/ML, and platform specialists.
|
||||
|
||||
See [**Skills Guide**](SKILLS_GUIDE.md) for the full list, decision trees, and workflow combinations.
|
||||
|
||||
@@ -122,4 +122,4 @@ Fullstack engineering, security engineering, compliance, and technical due dilig
|
||||
|
||||
---
|
||||
|
||||
**Built for Claude Code** | **<!-- WORKFLOW_COUNT -->9<!-- /WORKFLOW_COUNT --> Workflows** | **<!-- REFERENCE_COUNT -->367<!-- /REFERENCE_COUNT --> Reference Files** | **<!-- SKILL_COUNT -->66<!-- /SKILL_COUNT --> Skills**
|
||||
**Built for Claude Code** | **<!-- WORKFLOW_COUNT -->9<!-- /WORKFLOW_COUNT --> Workflows** | **<!-- REFERENCE_COUNT -->371<!-- /REFERENCE_COUNT --> Reference Files** | **<!-- SKILL_COUNT -->67<!-- /SKILL_COUNT --> Skills**
|
||||
|
||||
+2
-2
@@ -4,8 +4,8 @@
|
||||
|
||||
**Version:** v<!-- VERSION -->0.4.15<!-- /VERSION --> (Released January 2026)
|
||||
|
||||
- **<!-- SKILL_COUNT -->66<!-- /SKILL_COUNT --> Skills** across 12 domains
|
||||
- **<!-- REFERENCE_COUNT -->367<!-- /REFERENCE_COUNT --> Reference Files** with progressive disclosure architecture
|
||||
- **<!-- SKILL_COUNT -->67<!-- /SKILL_COUNT --> Skills** across 12 domains
|
||||
- **<!-- REFERENCE_COUNT -->371<!-- /REFERENCE_COUNT --> Reference Files** with progressive disclosure architecture
|
||||
- **30+ Frameworks** and technologies covered
|
||||
- **<!-- WORKFLOW_COUNT -->9<!-- /WORKFLOW_COUNT --> Project Workflow Commands** for epic planning, discovery, execution, and retrospectives
|
||||
- **50% Token Reduction** through selective disclosure architecture
|
||||
|
||||
@@ -23,6 +23,7 @@ A guide for choosing the right skill for your task. For installation, see [Quick
|
||||
### Backend Frameworks
|
||||
- **[NestJS Expert](https://jeffallan.github.io/claude-skills/skills/backend/nestjs-expert/)**: Building TypeScript backend APIs with NestJS
|
||||
- **[Django Expert](https://jeffallan.github.io/claude-skills/skills/backend/django-expert/)**: Building Python web apps with Django/DRF
|
||||
- **[Django Storages S3](https://jeffallan.github.io/claude-skills/skills/backend/django-storages-s3/)**: Production AWS S3 file storage with django-storages and boto3
|
||||
- **[FastAPI Expert](https://jeffallan.github.io/claude-skills/skills/backend/fastapi-expert/)**: Building async Python APIs with FastAPI
|
||||
- **[Spring Boot Engineer](https://jeffallan.github.io/claude-skills/skills/backend/spring-boot-engineer/)**: Java enterprise applications with Spring Boot
|
||||
- **[Laravel Specialist](https://jeffallan.github.io/claude-skills/skills/backend/laravel-specialist/)**: PHP web applications with Laravel
|
||||
|
||||
@@ -262,7 +262,7 @@
|
||||
</div>
|
||||
|
||||
<!-- Subtitle -->
|
||||
<h2 class="subtitle"><!-- SKILL_COUNT -->66<!-- /SKILL_COUNT --> Specialized Skills and <!-- WORKFLOW_COUNT -->9<!-- /WORKFLOW_COUNT --> workflows for Full-Stack Developers</h2>
|
||||
<h2 class="subtitle"><!-- SKILL_COUNT -->67<!-- /SKILL_COUNT --> Specialized Skills and <!-- WORKFLOW_COUNT -->9<!-- /WORKFLOW_COUNT --> workflows for Full-Stack Developers</h2>
|
||||
|
||||
<!-- Tagline -->
|
||||
<p class="tagline">Transform Claude Code into your expert pair programmer</p>
|
||||
@@ -277,7 +277,7 @@
|
||||
<div class="stat">
|
||||
<div class="stat-content">
|
||||
<span class="stat-icon">📚</span>
|
||||
<span><!-- REFERENCE_COUNT -->367<!-- /REFERENCE_COUNT --> Reference Files</span>
|
||||
<span><!-- REFERENCE_COUNT -->371<!-- /REFERENCE_COUNT --> Reference Files</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="stat">
|
||||
|
||||
@@ -8,7 +8,7 @@ export default defineConfig({
|
||||
starlight({
|
||||
title: 'Claude Skills',
|
||||
description:
|
||||
'66 specialized skills for Claude Code — progressive disclosure, context engineering, and full-stack coverage.',
|
||||
'67 specialized skills for Claude Code — progressive disclosure, context engineering, and full-stack coverage.',
|
||||
customCss: ['./src/styles/custom.css'],
|
||||
head: [
|
||||
{
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Claude Skills
|
||||
description: 66 specialized skills for Claude Code — progressive disclosure, context engineering, and full-stack coverage.
|
||||
description: 67 specialized skills for Claude Code — progressive disclosure, context engineering, and full-stack coverage.
|
||||
template: splash
|
||||
hero:
|
||||
tagline: Transform Claude Code into your expert pair programmer across the entire development stack.
|
||||
@@ -17,14 +17,14 @@ hero:
|
||||
import { Card, CardGrid } from '@astrojs/starlight/components';
|
||||
|
||||
<CardGrid stagger>
|
||||
<Card title="66 Skills" icon="puzzle">
|
||||
<Card title="67 Skills" icon="puzzle">
|
||||
Specialized skills covering 12 domains — languages, frameworks, infrastructure, security, DevOps, data/ML, and more.
|
||||
</Card>
|
||||
<Card title="9 Workflows" icon="rocket">
|
||||
Project workflow commands managing epics from discovery through retrospectives, with Jira and Confluence
|
||||
integration.
|
||||
</Card>
|
||||
<Card title="367 References" icon="open-book">
|
||||
<Card title="371 References" icon="open-book">
|
||||
Deep-dive reference files loaded on-demand for surgical precision when context requires it.
|
||||
</Card>
|
||||
<Card title="Progressive Disclosure" icon="magnifier">
|
||||
|
||||
@@ -10,7 +10,7 @@ metadata:
|
||||
role: specialist
|
||||
scope: implementation
|
||||
output-format: code
|
||||
related-skills: fullstack-guardian, fastapi-expert, test-master
|
||||
related-skills: fullstack-guardian, fastapi-expert, test-master, django-storages-s3
|
||||
---
|
||||
|
||||
# Django Expert
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
---
|
||||
name: django-storages-s3
|
||||
description: "Use when configuring Django to store static and media files on AWS S3 with django-storages. Invoke when working with the STORAGES setting, S3 buckets, presigned URLs, CloudFront, or boto3-backed file storage in settings.py. Configures the Django 4.2+ STORAGES dict, public/private custom backends, presigned GET/POST URLs, IAM policies, and S3 mocking for tests. Trigger terms: django-storages, S3, boto3, S3Boto3Storage, STORAGES, presigned URL, CloudFront, media files, collectstatic, AWS_STORAGE_BUCKET_NAME."
|
||||
license: MIT
|
||||
metadata:
|
||||
author: https://github.com/awais786
|
||||
version: "1.0.0"
|
||||
domain: backend
|
||||
triggers: django-storages, S3, boto3, S3Boto3Storage, STORAGES, presigned URL, CloudFront, media files, collectstatic
|
||||
role: specialist
|
||||
scope: implementation
|
||||
output-format: code
|
||||
related-skills: django-expert
|
||||
---
|
||||
|
||||
# Django Storages S3
|
||||
|
||||
Senior Django specialist for production-grade file storage on AWS S3 via `django-storages` and `boto3` — public and private media, static files, presigned URLs, and CloudFront.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
- Serving static and/or media files from AWS S3 instead of the local filesystem
|
||||
- Configuring the Django 4.2+ `STORAGES` dict or legacy `DEFAULT_FILE_STORAGE`
|
||||
- Separating public (CDN-served) and private (presigned) file backends
|
||||
- Generating presigned download or direct browser-to-S3 upload URLs
|
||||
- Fronting S3 with CloudFront and writing a least-privilege IAM policy
|
||||
- Migrating local `FileField`/`ImageField` storage to S3 without code changes
|
||||
- Testing storage code without hitting S3
|
||||
|
||||
## Core Workflow
|
||||
|
||||
1. **Install & register** — `pip install django-storages[s3] boto3`; add `"storages"` to `INSTALLED_APPS`
|
||||
2. **Configure credentials** — Load from env vars or rely on an attached IAM role; never hardcode
|
||||
3. **Wire the `STORAGES` dict** — Set `default` (media) and `staticfiles` backends with separate `location` prefixes
|
||||
4. **Add named backends** — Split public vs. private buckets/ACLs as additional `STORAGES` entries when needed
|
||||
5. **Verify & test** — Run `collectstatic`, confirm uploads land in S3, and mock S3 in tests with `InMemoryStorage` or `moto`
|
||||
|
||||
## Reference Guide
|
||||
|
||||
Load detailed guidance based on context:
|
||||
|
||||
| Topic | Reference | Load When |
|
||||
|-------|-----------|-----------|
|
||||
| Settings & STORAGES | `references/configuration.md` | Core settings, 4.2+ vs legacy, CloudFront |
|
||||
| Custom backends | `references/custom-backends.md` | Public vs. private buckets, per-field storage |
|
||||
| Presigned URLs | `references/presigned-urls.md` | Download links, direct browser uploads |
|
||||
| Testing & IAM | `references/testing-storages.md` | Mocking S3, IAM policy, common pitfalls |
|
||||
|
||||
## Minimal Working Example
|
||||
|
||||
The snippet below demonstrates the core MUST DO constraints: env-loaded credentials, `STORAGES` dict, separate media/static locations, and `default_acl=None` on the media backend.
|
||||
|
||||
```python
|
||||
# settings.py
|
||||
import os
|
||||
|
||||
AWS_STORAGE_BUCKET_NAME = os.environ["AWS_STORAGE_BUCKET_NAME"]
|
||||
AWS_S3_REGION_NAME = os.environ.get("AWS_S3_REGION_NAME", "us-east-1")
|
||||
AWS_S3_CUSTOM_DOMAIN = f"{AWS_STORAGE_BUCKET_NAME}.s3.{AWS_S3_REGION_NAME}.amazonaws.com"
|
||||
# On EC2/ECS/Lambda, omit keys entirely — boto3 uses the attached IAM role.
|
||||
|
||||
STORAGES = {
|
||||
"default": { # media uploads
|
||||
"BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": AWS_STORAGE_BUCKET_NAME,
|
||||
"location": "media",
|
||||
"default_acl": None, # rely on bucket policy, not per-object ACLs
|
||||
"file_overwrite": False,
|
||||
"querystring_auth": False, # public objects → clean URLs
|
||||
},
|
||||
},
|
||||
"staticfiles": {
|
||||
"BACKEND": "storages.backends.s3boto3.S3StaticStorage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": AWS_STORAGE_BUCKET_NAME,
|
||||
"location": "static",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
MEDIA_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/media/"
|
||||
STATIC_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/static/"
|
||||
```
|
||||
|
||||
```python
|
||||
# models.py — uploads go straight to S3 on save()
|
||||
from django.db import models
|
||||
|
||||
class Document(models.Model):
|
||||
file = models.FileField(upload_to="docs/") # uses STORAGES["default"]
|
||||
```
|
||||
|
||||
## Auditing an Existing Configuration
|
||||
|
||||
When reviewing a project that already uses S3 (not greenfield), walk this
|
||||
checklist — each item is a constraint below rephrased as "find X, confirm Y":
|
||||
|
||||
1. **Credentials** — `grep -rn "AWS_SECRET_ACCESS_KEY\|aws_secret" settings/` → confirm values come from `os.environ`/`django-environ` or an IAM role, never literals committed to the repo.
|
||||
2. **ACLs** — `grep -rn "default_acl\|AWS_DEFAULT_ACL" .` → on buckets created after April 2023, every value must be `None`. Any `"public-read"`/`"private"` will raise `AccessControlListNotSupported`; public access belongs in a bucket policy.
|
||||
3. **Storage backend** — confirm Django 4.2+ uses the `STORAGES` dict, not `DEFAULT_FILE_STORAGE`/`STATICFILES_STORAGE` (removed in Django 5.1, so silently ignored on 5.1/5.2/6.0); confirm the static class is `S3StaticStorage`, not a fabricated name.
|
||||
4. **Locations** — confirm `default` (media) and `staticfiles` have distinct `location` prefixes or buckets so `collectstatic` never collides with uploads.
|
||||
5. **Region** — confirm `region_name` (or the global `AWS_S3_REGION_NAME`) matches the bucket's real region and that `AWS_S3_CUSTOM_DOMAIN` includes the region segment for non-`us-east-1` buckets.
|
||||
6. **Presigning** — for private backends, confirm `querystring_auth=True` **and** `custom_domain=None`; confirm presigned `.url()` results aren't cached past `AWS_QUERYSTRING_EXPIRE`.
|
||||
7. **Overwrite cleanup** — where `file_overwrite=False`, confirm replaced files are explicitly deleted (otherwise superseded objects leak).
|
||||
8. **IAM** — confirm the policy grants only `Get/Put/Delete/ListBucket` on the bucket ARN, not broader S3 access.
|
||||
|
||||
## Constraints
|
||||
|
||||
### MUST DO
|
||||
- Load AWS credentials from environment variables or an attached IAM role
|
||||
- Set `default_acl=None` so bucket policies (not object ACLs) control access
|
||||
- Give static and media files separate `location` prefixes or separate buckets
|
||||
- Use the `STORAGES` dict on Django 4.2+ (same config through 5.2 LTS and 6.0); `DEFAULT_FILE_STORAGE`/`STATICFILES_STORAGE` were removed in 5.1, so reserve them for < 4.2 only
|
||||
- Set `custom_domain=None` on any backend that issues presigned URLs
|
||||
- Mock S3 (`InMemoryStorage` or `moto`) in tests instead of hitting real buckets
|
||||
|
||||
### MUST NOT DO
|
||||
- Hardcode `AWS_SECRET_ACCESS_KEY` in `settings.py` or commit it
|
||||
- Mix `querystring_auth=True` with a `custom_domain` (presigning breaks)
|
||||
- Mix static and media files under the same prefix
|
||||
- Grant the IAM user broader than `Get/Put/Delete/ListBucket` on the bucket ARN
|
||||
- Rely on per-object ACLs on buckets created after April 2023 (ACLs disabled by default)
|
||||
|
||||
## Knowledge Reference
|
||||
|
||||
django-storages, S3Boto3Storage, S3StaticStorage, boto3, STORAGES dict, presigned URLs, generate_presigned_post, CloudFront, IAM policy, InMemoryStorage, moto
|
||||
|
||||
## Related Skills
|
||||
|
||||
- `django-expert` — core Django models, DRF, and ORM that produce the files this skill persists to S3
|
||||
- `fullstack-guardian` — secure end-to-end upload flows and access control around stored files
|
||||
- `devops-engineer` — provisioning the S3 buckets, IAM roles, and CloudFront distributions this skill targets
|
||||
|
||||
[Documentation](https://jeffallan.github.io/claude-skills/skills/backend/django-storages-s3/)
|
||||
@@ -0,0 +1,181 @@
|
||||
# Configuration & Settings
|
||||
|
||||
## Package Installation
|
||||
|
||||
```bash
|
||||
pip install django-storages[s3] boto3
|
||||
```
|
||||
|
||||
```python
|
||||
# settings.py
|
||||
INSTALLED_APPS = [
|
||||
# ...
|
||||
"storages",
|
||||
]
|
||||
```
|
||||
|
||||
## Credentials
|
||||
|
||||
Load credentials from environment variables (via `os.environ` or `django-environ`):
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID")
|
||||
AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY")
|
||||
AWS_STORAGE_BUCKET_NAME = os.environ.get("AWS_STORAGE_BUCKET_NAME")
|
||||
AWS_S3_REGION_NAME = os.environ.get("AWS_S3_REGION_NAME", "us-east-1")
|
||||
|
||||
AWS_S3_CUSTOM_DOMAIN = f"{AWS_STORAGE_BUCKET_NAME}.s3.{AWS_S3_REGION_NAME}.amazonaws.com"
|
||||
AWS_DEFAULT_ACL = None # Recommended: let bucket policy control access
|
||||
AWS_S3_FILE_OVERWRITE = False # Avoid overwriting files with the same name
|
||||
AWS_QUERYSTRING_AUTH = False # Global default. Private backends must set
|
||||
# querystring_auth=True in their STORAGES
|
||||
# OPTIONS (below) to get presigned .url() links.
|
||||
```
|
||||
|
||||
**When not to set keys:** On AWS infrastructure (EC2, ECS, Lambda), omit
|
||||
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` entirely — boto3 picks up the
|
||||
attached IAM role automatically. This is preferred over long-lived keys.
|
||||
|
||||
## Django 4.2+ — the `STORAGES` dict (recommended)
|
||||
|
||||
```python
|
||||
STORAGES = {
|
||||
"default": {
|
||||
"BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": AWS_STORAGE_BUCKET_NAME,
|
||||
"location": "media",
|
||||
"file_overwrite": False,
|
||||
},
|
||||
},
|
||||
"staticfiles": {
|
||||
"BACKEND": "storages.backends.s3boto3.S3StaticStorage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": AWS_STORAGE_BUCKET_NAME,
|
||||
"location": "static",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
MEDIA_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/media/"
|
||||
STATIC_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/static/"
|
||||
```
|
||||
|
||||
The `STORAGES` dict is unchanged across Django 4.2, 5.x, and 6.0 — the same
|
||||
config above is correct on every release from 4.2 onward. Nothing extra is
|
||||
needed for Django 5.2 LTS or 6.0.
|
||||
|
||||
## Django < 4.2 (legacy)
|
||||
|
||||
```python
|
||||
DEFAULT_FILE_STORAGE = "storages.backends.s3boto3.S3Boto3Storage"
|
||||
STATICFILES_STORAGE = "storages.backends.s3boto3.S3StaticStorage"
|
||||
MEDIA_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/media/"
|
||||
STATIC_URL = f"https://{AWS_S3_CUSTOM_DOMAIN}/static/"
|
||||
```
|
||||
|
||||
`DEFAULT_FILE_STORAGE` and `STATICFILES_STORAGE` were deprecated in Django 4.2
|
||||
and **removed in Django 5.1**. They still function on 4.2 and 5.0 only — on 5.1,
|
||||
5.2 LTS, and 6.0 they are gone and silently ignored, so the `STORAGES` dict is
|
||||
mandatory there. Use `STORAGES` on any project running 4.2 or newer; only reach
|
||||
for these settings on Django < 4.2.
|
||||
|
||||
**Always** use separate `location` prefixes (e.g. `media/` and `static/`) or
|
||||
separate buckets so that static and media files are never mixed — otherwise
|
||||
`collectstatic` can overwrite or collide with user uploads.
|
||||
|
||||
## CloudFront CDN Integration
|
||||
|
||||
For production, serve files via CloudFront instead of directly from S3:
|
||||
|
||||
```python
|
||||
AWS_S3_CUSTOM_DOMAIN = os.environ.get("CLOUDFRONT_DOMAIN") # e.g. "d1234abcdef.cloudfront.net"
|
||||
|
||||
# Only if using signed CloudFront URLs (private distribution):
|
||||
AWS_CLOUDFRONT_KEY_ID = os.environ.get("AWS_CLOUDFRONT_KEY_ID")
|
||||
AWS_CLOUDFRONT_KEY = os.environ.get("AWS_CLOUDFRONT_KEY") # PEM private key string
|
||||
```
|
||||
|
||||
Apply the custom domain per backend in the `STORAGES` dict:
|
||||
|
||||
```python
|
||||
STORAGES = {
|
||||
"default": {
|
||||
"BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": AWS_STORAGE_BUCKET_NAME,
|
||||
"custom_domain": AWS_S3_CUSTOM_DOMAIN,
|
||||
"location": "media",
|
||||
},
|
||||
},
|
||||
"staticfiles": {
|
||||
"BACKEND": "storages.backends.s3boto3.S3StaticStorage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": AWS_STORAGE_BUCKET_NAME,
|
||||
"custom_domain": AWS_S3_CUSTOM_DOMAIN,
|
||||
"location": "static",
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
> A backend that serves through CloudFront should **not** also set
|
||||
> `querystring_auth=True` for plain presigned S3 URLs — see
|
||||
> `presigned-urls.md` and the conflict note in `testing-storages.md`.
|
||||
|
||||
### Signed CloudFront URLs (private distributions)
|
||||
|
||||
When `AWS_CLOUDFRONT_KEY_ID` and `AWS_CLOUDFRONT_KEY` are set and `custom_domain`
|
||||
points at the CloudFront domain, django-storages signs URLs **automatically** —
|
||||
`.url()` returns a signed CloudFront URL, no manual boto3 call:
|
||||
|
||||
```python
|
||||
AWS_CLOUDFRONT_KEY_ID = os.environ["AWS_CLOUDFRONT_KEY_ID"] # public key ID
|
||||
AWS_CLOUDFRONT_KEY = os.environ["AWS_CLOUDFRONT_KEY"] # PEM private key
|
||||
AWS_QUERYSTRING_EXPIRE = 3600 # signature lifetime
|
||||
|
||||
doc.contract.url # → https://d123.cloudfront.net/...?Expires=...&Signature=...&Key-Pair-Id=...
|
||||
```
|
||||
|
||||
Signing requires the `cryptography` package (`pip install django-storages[cloudfront]`);
|
||||
without it django-storages cannot build the signature and falls back to an
|
||||
unsigned URL, so private objects return `403`.
|
||||
|
||||
If you do **not** need signed access (public distribution), omit both
|
||||
`AWS_CLOUDFRONT_KEY*` vars entirely — setting them only matters for private
|
||||
distributions. This is distinct from S3 presigning (`presigned-urls.md`), which
|
||||
signs against S3 directly rather than CloudFront.
|
||||
|
||||
## Per-Environment Backends
|
||||
|
||||
Layer storage by environment so tests and local dev never touch S3. With
|
||||
split settings modules, override `STORAGES` per environment.
|
||||
|
||||
**Every override must define _both_ `default` and `staticfiles`.** Django does
|
||||
not merge your `STORAGES` with the defaults — it uses your dict verbatim
|
||||
(`settings.STORAGES.copy()`), so omitting `staticfiles` makes the `{% static %}`
|
||||
tag, `collectstatic`, and admin CSS raise
|
||||
`InvalidStorageError: Could not find config for 'staticfiles'`.
|
||||
|
||||
```python
|
||||
_STATIC = {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"}
|
||||
|
||||
# settings/dev.py — local filesystem
|
||||
STORAGES = {
|
||||
"default": {"BACKEND": "django.core.files.storage.FileSystemStorage"},
|
||||
"staticfiles": _STATIC,
|
||||
}
|
||||
|
||||
# settings/test.py — in-memory, nothing persists
|
||||
STORAGES = {
|
||||
"default": {"BACKEND": "django.core.files.storage.InMemoryStorage"},
|
||||
"staticfiles": _STATIC,
|
||||
}
|
||||
|
||||
# settings/prod.py — S3 (the full STORAGES dict shown above, with both keys)
|
||||
```
|
||||
|
||||
Keep the S3 backend confined to production (and staging); see
|
||||
`testing-storages.md` for the `override_settings` equivalent in individual tests.
|
||||
@@ -0,0 +1,156 @@
|
||||
# Custom Storage Backends
|
||||
|
||||
Split storage by access level: public images served over the CDN, private
|
||||
documents accessed only through presigned URLs. There are two idiomatic ways to
|
||||
do this.
|
||||
|
||||
## Option A — Named entries in the `STORAGES` dict (Django 4.2+)
|
||||
|
||||
Preferred on modern Django. Each named backend is just another key:
|
||||
|
||||
```python
|
||||
STORAGES = {
|
||||
"default": { ... }, # media uploads
|
||||
"staticfiles": { ... }, # static files
|
||||
"public_images": {
|
||||
"BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": "mybucket-public",
|
||||
"region_name": AWS_S3_REGION_NAME,
|
||||
"default_acl": None, # public access via bucket policy, not ACLs
|
||||
"querystring_auth": False, # clean, unsigned URLs
|
||||
"file_overwrite": False,
|
||||
"location": "media/public",
|
||||
},
|
||||
},
|
||||
"private_files": {
|
||||
"BACKEND": "storages.backends.s3boto3.S3Boto3Storage",
|
||||
"OPTIONS": {
|
||||
"bucket_name": "mybucket-private",
|
||||
"region_name": AWS_S3_REGION_NAME,
|
||||
"default_acl": None, # access controlled by presigning, not ACLs
|
||||
"querystring_auth": True, # .url() returns presigned URLs automatically
|
||||
"custom_domain": None, # MUST be None for presigning to work
|
||||
"file_overwrite": False,
|
||||
"location": "media/private",
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
> **Do not use `default_acl="public-read"` / `"private"` here.** Since April 2023,
|
||||
> new S3 buckets ship with Object Ownership = *Bucket owner enforced*, which
|
||||
> **disables ACLs** — any `default_acl` value other than `None` raises
|
||||
> `AccessControlListNotSupported` at upload time. Make the *public* bucket public
|
||||
> with a bucket policy (`s3:GetObject` on `media/public/*`) and keep the *private*
|
||||
> bucket locked down, gating access through presigned URLs (`querystring_auth=True`).
|
||||
> ACL string values only work on legacy buckets that explicitly re-enable ACLs
|
||||
> (Object Ownership = *ACLs enabled*).
|
||||
|
||||
Reference a named backend on a model field:
|
||||
|
||||
```python
|
||||
from django.core.files.storage import storages
|
||||
from django.db import models
|
||||
|
||||
class Document(models.Model):
|
||||
image = models.ImageField(storage=storages["public_images"])
|
||||
contract = models.FileField(storage=storages["private_files"])
|
||||
```
|
||||
|
||||
`storages["..."]` is the Django 4.2+ accessor; it resolves lazily, so it is
|
||||
safe to reference at class-definition time.
|
||||
|
||||
## Option B — Manual helper (all Django versions)
|
||||
|
||||
Useful on Django < 4.2 or when you want a single source of truth that is not the
|
||||
`STORAGES` dict. Define backends as plain dicts:
|
||||
|
||||
```python
|
||||
# settings.py
|
||||
PUBLIC_IMAGE_BACKEND = {
|
||||
"class": "storages.backends.s3boto3.S3Boto3Storage",
|
||||
"options": {
|
||||
"bucket_name": "mybucket-public",
|
||||
"region_name": AWS_S3_REGION_NAME,
|
||||
"default_acl": None, # public access via bucket policy, not ACLs
|
||||
"querystring_auth": False,
|
||||
"file_overwrite": False,
|
||||
"location": "media/public",
|
||||
},
|
||||
}
|
||||
|
||||
PRIVATE_FILE_BACKEND = {
|
||||
"class": "storages.backends.s3boto3.S3Boto3Storage",
|
||||
"options": {
|
||||
"bucket_name": "mybucket-private",
|
||||
"region_name": AWS_S3_REGION_NAME,
|
||||
"default_acl": None, # access controlled by presigning, not ACLs
|
||||
"querystring_auth": True, # .url() returns presigned URLs automatically
|
||||
"custom_domain": None, # Must be None for presigned URLs to work
|
||||
"file_overwrite": False,
|
||||
"location": "media/private",
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
Instantiate a backend from a settings dict:
|
||||
|
||||
```python
|
||||
# myapp/storages.py
|
||||
from django.conf import settings
|
||||
from django.utils.module_loading import import_string
|
||||
|
||||
def get_storage(setting_name):
|
||||
config = getattr(settings, setting_name)
|
||||
storage_class = import_string(config["class"])
|
||||
return storage_class(**config.get("options", {}))
|
||||
```
|
||||
|
||||
```python
|
||||
# models.py
|
||||
from myapp.storages import get_storage
|
||||
|
||||
class Document(models.Model):
|
||||
image = models.ImageField(storage=get_storage("PUBLIC_IMAGE_BACKEND"))
|
||||
contract = models.FileField(storage=get_storage("PRIVATE_FILE_BACKEND"))
|
||||
```
|
||||
|
||||
## When to use which
|
||||
|
||||
| Situation | Use |
|
||||
|-----------|-----|
|
||||
| Django 4.2+, backend is a fixed config | Option A (`STORAGES` dict) |
|
||||
| Django < 4.2 | Option B (manual helper) |
|
||||
| Backend chosen dynamically at runtime | Option B |
|
||||
|
||||
## File Upload in Views
|
||||
|
||||
Saving a model with a populated file field uploads to S3 automatically — no
|
||||
explicit boto3 call:
|
||||
|
||||
```python
|
||||
# forms.py
|
||||
from django import forms
|
||||
|
||||
class UploadForm(forms.Form):
|
||||
file = forms.FileField()
|
||||
|
||||
# views.py
|
||||
from django.shortcuts import render, redirect
|
||||
|
||||
def upload_view(request):
|
||||
if request.method == "POST":
|
||||
form = UploadForm(request.POST, request.FILES)
|
||||
if form.is_valid():
|
||||
instance = Document(contract=form.cleaned_data["file"])
|
||||
instance.save() # uploads to S3 via the field's storage backend
|
||||
return redirect("success")
|
||||
else:
|
||||
form = UploadForm()
|
||||
return render(request, "upload.html", {"form": form})
|
||||
```
|
||||
|
||||
For large files (> 100 MB), prefer direct browser-to-S3 uploads with a
|
||||
presigned POST so the file never transits your Django server — see
|
||||
`presigned-urls.md`.
|
||||
@@ -0,0 +1,104 @@
|
||||
# Presigned URLs
|
||||
|
||||
Presigned URLs grant time-limited access to private S3 objects without making
|
||||
the bucket public. Two directions: **download** (presigned GET) and **upload**
|
||||
(presigned POST).
|
||||
|
||||
## Automatic download URLs (recommended)
|
||||
|
||||
For a backend with `querystring_auth=True` (e.g. the `private_files` backend in
|
||||
`custom-backends.md`), calling `.url` on a file field returns a presigned URL —
|
||||
no manual boto3 call:
|
||||
|
||||
```python
|
||||
doc = Document.objects.get(pk=1)
|
||||
download_link = doc.contract.url # presigned GET URL, expires per AWS_QUERYSTRING_EXPIRE
|
||||
```
|
||||
|
||||
`AWS_QUERYSTRING_EXPIRE` (default `3600` seconds) controls expiry. The backend
|
||||
**must** have `custom_domain=None`, or presigning silently breaks.
|
||||
|
||||
## Manual download URL (custom expiry / non-model objects)
|
||||
|
||||
```python
|
||||
import boto3
|
||||
from django.conf import settings
|
||||
|
||||
def get_presigned_url(s3_key: str, expiry_seconds: int = 3600) -> str:
|
||||
"""Generate a time-limited GET URL for a private S3 object."""
|
||||
client = boto3.client("s3", region_name=settings.AWS_S3_REGION_NAME)
|
||||
return client.generate_presigned_url(
|
||||
"get_object",
|
||||
Params={"Bucket": settings.AWS_STORAGE_BUCKET_NAME, "Key": s3_key},
|
||||
ExpiresIn=expiry_seconds,
|
||||
)
|
||||
```
|
||||
|
||||
On AWS infrastructure with an IAM role, `boto3.client("s3")` picks up
|
||||
credentials automatically — do not pass keys explicitly.
|
||||
|
||||
## Presigned upload URL (direct browser-to-S3)
|
||||
|
||||
Let the client upload straight to S3, bypassing your Django server — essential
|
||||
for large files and to avoid request timeouts:
|
||||
|
||||
```python
|
||||
import boto3
|
||||
from django.conf import settings
|
||||
|
||||
def get_presigned_upload_url(
|
||||
s3_key: str,
|
||||
content_type: str = "application/octet-stream",
|
||||
expiry: int = 3600,
|
||||
) -> dict:
|
||||
"""Generate a presigned POST for direct browser-to-S3 uploads.
|
||||
|
||||
Returns {"url": ..., "fields": {...}} — POST these as multipart/form-data
|
||||
from the browser with the file appended last.
|
||||
"""
|
||||
client = boto3.client("s3", region_name=settings.AWS_S3_REGION_NAME)
|
||||
return client.generate_presigned_post(
|
||||
Bucket=settings.AWS_STORAGE_BUCKET_NAME,
|
||||
Key=s3_key,
|
||||
Fields={"Content-Type": content_type},
|
||||
Conditions=[{"Content-Type": content_type}],
|
||||
ExpiresIn=expiry,
|
||||
)
|
||||
```
|
||||
|
||||
### Returning it from a view
|
||||
|
||||
```python
|
||||
from django.http import JsonResponse
|
||||
|
||||
def upload_url_view(request):
|
||||
key = f"uploads/{request.GET['filename']}"
|
||||
content_type = request.GET.get("type", "application/octet-stream")
|
||||
return JsonResponse(get_presigned_upload_url(key, content_type))
|
||||
```
|
||||
|
||||
### Client-side upload (sketch)
|
||||
|
||||
```js
|
||||
const { url, fields } = await fetch("/upload-url/?filename=report.pdf").then(r => r.json());
|
||||
const form = new FormData();
|
||||
Object.entries(fields).forEach(([k, v]) => form.append(k, v));
|
||||
form.append("file", fileInput.files[0]); // file MUST be appended last
|
||||
await fetch(url, { method: "POST", body: form });
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`querystring_auth=True` + `custom_domain` set** → presigning breaks. Set
|
||||
`custom_domain=None` on private backends.
|
||||
- **Clock skew** → presigned URLs are time-sensitive; ensure server time (NTP)
|
||||
is correct or signatures fail with `403`.
|
||||
- **Wrong region** → a URL signed for the wrong region returns
|
||||
`AuthorizationHeaderMalformed`. Pass the bucket's actual region. Set
|
||||
`region_name` in each backend's `OPTIONS` rather than relying only on the
|
||||
global `AWS_S3_REGION_NAME`, so per-backend signing always uses the right region.
|
||||
- **Caching a presigned URL past its expiry** → a presigned GET is only valid for
|
||||
`AWS_QUERYSTRING_EXPIRE` seconds. Serializing `.url()` into a DRF/HTML response
|
||||
that is cached (CDN, `cache_page`, client) longer than that window means the
|
||||
embedded links silently start returning `403` once they expire. Generate the URL
|
||||
per request, or keep the response TTL shorter than the signature lifetime.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Testing, IAM & Pitfalls
|
||||
|
||||
## Testing without hitting S3
|
||||
|
||||
### `InMemoryStorage` (Django 4.2+)
|
||||
|
||||
Override the storage backend per test so nothing touches AWS:
|
||||
|
||||
```python
|
||||
from django.test import TestCase, override_settings
|
||||
from django.core.files.uploadedfile import SimpleUploadedFile
|
||||
|
||||
@override_settings(
|
||||
STORAGES={
|
||||
"default": {"BACKEND": "django.core.files.storage.InMemoryStorage"},
|
||||
"staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"},
|
||||
}
|
||||
)
|
||||
class FileUploadTests(TestCase):
|
||||
def test_upload(self):
|
||||
f = SimpleUploadedFile("test.txt", b"hello", content_type="text/plain")
|
||||
obj = Document.objects.create(file=f)
|
||||
self.assertIn("test", obj.file.name)
|
||||
```
|
||||
|
||||
The `override_settings(STORAGES=...)` form above works on Django 4.2 through 6.0.
|
||||
For Django < 4.2 only, use
|
||||
`@override_settings(DEFAULT_FILE_STORAGE="django.core.files.storage.FileSystemStorage")`
|
||||
— that setting was removed in Django 5.1, so it has no effect on 5.1+.
|
||||
|
||||
### `moto` — mock the S3 API itself
|
||||
|
||||
Use when the code under test calls boto3 directly (e.g. presigned URLs):
|
||||
|
||||
```bash
|
||||
pip install moto[s3]
|
||||
```
|
||||
|
||||
```python
|
||||
import boto3
|
||||
from moto import mock_aws
|
||||
|
||||
@mock_aws
|
||||
def test_presigned_url():
|
||||
conn = boto3.client("s3", region_name="us-east-1")
|
||||
conn.create_bucket(Bucket="test-bucket")
|
||||
conn.put_object(Bucket="test-bucket", Key="test.txt", Body=b"data")
|
||||
|
||||
url = get_presigned_url("test.txt")
|
||||
assert "test.txt" in url
|
||||
```
|
||||
|
||||
| Use | When |
|
||||
|-----|------|
|
||||
| `InMemoryStorage` / `override_settings` | Testing model/field/upload behavior |
|
||||
| `moto` | Testing direct boto3 calls (presigned URLs, custom clients) |
|
||||
|
||||
## IAM Policy (minimum required)
|
||||
|
||||
Grant the app's IAM user/role only object-level access plus `ListBucket`:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
|
||||
"Resource": "arn:aws:s3:::your-bucket-name/*"
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": ["s3:ListBucket"],
|
||||
"Resource": "arn:aws:s3:::your-bucket-name"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Note the two different ARNs: object actions target `bucket/*`, while
|
||||
`ListBucket` targets the bucket itself (`bucket`, no `/*`). Prefer attaching
|
||||
this to a role over creating long-lived access keys.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **`querystring_auth=True` + `custom_domain`** — These conflict. Presigned URLs
|
||||
require the default S3 domain, so set `custom_domain=None` on private backends.
|
||||
- **ACL errors on ACL-disabled buckets** — Set `AWS_DEFAULT_ACL=None` and rely on
|
||||
bucket policies. Since April 2023, new S3 buckets have ACLs disabled by default,
|
||||
so `default_acl="public-read"` raises `AccessControlListNotSupported`.
|
||||
- **`collectstatic` uploading to the wrong location** — Ensure the `staticfiles`
|
||||
backend has `location="static"` so it never mixes with media.
|
||||
- **Credentials in `settings.py`** — Always load via env vars or IAM roles; never
|
||||
hardcode or commit secrets.
|
||||
- **Removed storage settings** — `DEFAULT_FILE_STORAGE` and `STATICFILES_STORAGE`
|
||||
were deprecated in Django 4.2 and **removed in Django 5.1**. They are silently
|
||||
ignored on 5.1, 5.2 LTS, and 6.0. Use the `STORAGES` dict on 4.2+.
|
||||
- **Large uploads timing out** — For files > 100 MB, use presigned upload URLs for
|
||||
direct browser-to-S3 uploads to bypass the Django server.
|
||||
- **`file_overwrite=False` orphans replaced files** — With overwrite disabled, a
|
||||
re-upload to the same field writes a *new* suffixed key (`avatar_a1b2c3.png`)
|
||||
and the previous object is **not** deleted — S3 grows unbounded. You own the
|
||||
cleanup: capture the old name before saving and delete it after, e.g.
|
||||
`old = instance.avatar.name; ...; instance.avatar.storage.delete(old)`. This
|
||||
is a silent storage-cost leak, not an error.
|
||||
- **Missing `Content-Type`** — S3 may default to `binary/octet-stream`.
|
||||
django-storages detects content type automatically by default; only override
|
||||
via `AWS_S3_OBJECT_PARAMETERS` if you need to force it.
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"version": "0.4.15",
|
||||
"skillCount": 66,
|
||||
"skillCount": 67,
|
||||
"workflowCount": 9,
|
||||
"referenceFileCount": 367
|
||||
"referenceFileCount": 371
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user