feat(schema-design): Access pattern analysis MCP-476 (#37)

2026-09-18 21:15:11 +08:00 · 2026-07-01 15:06:19 +02:00
parent 9ea7387c7a
commit be846b4deb
8 changed files with 265 additions and 71 deletions
@@ -53,6 +53,44 @@ Reference these guidelines when:
 - [pattern-schema-versioning](references/pattern-schema-versioning.md) - Schema evolution, preventing drift, and safe online migrations. Consult when encountering inconsistent document structures, or when planning a schema change that cannot be applied atomically.
 - [pattern-time-series-collections](references/pattern-time-series-collections.md) - Use native time series collections for high-frequency time series data

+### Access Pattern Analysis
+
+Do not immediately recommend a pattern or schema change without understanding the broader context. Together with the user, analyze access patterns to identify pain points and opportunities for optimization.
+
+#### Workflow
+
+**Step 1: Assess the environment**
+Ask the user:
+  - Is this a new design or is there a production database with existing access patterns to analyze?
+  - If there is production data, is it on Atlas? If yes, what tier? (M0/M2/M5 vs M10+)
+
+**Step 2: Determine workload type**
+Is the workload read-heavy, write-heavy, or balanced? This will influence which diagnostic sources are most relevant.
+Ask the user:
+- What's the primary workload for these collections — read-heavy (analytics, reports, searches), write-heavy (logging, IoT ingestion, frequent updates), or balanced?
+
+Verify with `db.serverStatus().opcounters`.
+
+**Step 3: Work with the user to choose the best source(s)**
+  Recommend the best source(s) for their situation, explaining the tradeoffs. For schema design decisions, we often need to combine multiple sources for a complete picture.
+
+**Step 4: Proceed with analysis**
+  Only after source selection, fetch data or guide the user through analysis.
+
+#### Sources
+
+- [Query statistics](references/source-query-stats.md) - Returns runtime statistics for recorded queries showing query shapes and frequency. **Limitation**: Currently only captures read operations (pair with other sources for write patterns). Requires Atlas M10+ tier.
+- [Atlas Slow Query Logs](references/source-slow-query-logs.md) - Review slow queries (actual queries, not shapes) to identify performance bottlenecks. Captures all reads and writes. Requires Atlas M10+ tier.
+- Codebase - Examine actual queries in application code to understand access patterns, especially for new applications or with changing workloads. Can be used in conjunction with query stats for a more complete picture.
+- Natural language input - Ask the user to describe their typical queries and access patterns in natural language. Can be used as the only source or to supplement and validate other sources - the user might have contextual knowledge that is not reflected in the data or codebase.
+
+**Combining Query Stats and Slow Query Logs:**
+
+Use both together for comprehensive analysis:
+1. Query Stats → identify frequent access patterns (which queries run most often)
+2. Slow Query Logs → identify performance bottlenecks (which queries are slow)
+3. Focus schema optimization on queries that are both frequent AND slow (highest impact)
+
 ## Key Principle

 > **"Data that is accessed together should be stored together."**
@@ -102,7 +140,7 @@ Each reference file contains:

 For automatic verification, connect the [MongoDB MCP Server](https://github.com/mongodb-js/mongodb-mcp-server).

-If the MCP server is running and connected, I can automatically run verification commands to check your actual schema, document sizes, array lengths, index usage, and more. This allows me to provide tailored recommendations based on your real data, not just code patterns.
+If the MCP server is running and connected, I can automatically run verification commands to check your actual schema, document sizes, array lengths, index usage, slow query logs, and more. This allows me to provide tailored recommendations based on your real data, not just code patterns.

 **⚠️ Security**: Use `--readOnly` for safety. Remove only if you need write operations.

@@ -52,21 +52,24 @@ When category data changes (a rare event), use `updateMany` to update all produc

 ## Verify with

+####  Find pipelines with $lookup stages 
+
+For Atlas M10+ use $queryStats. See [Query Stats](references/source-query-stats.md)
+Use codebase if available, ask the user.
+
 ```javascript
-// Find pipelines with multiple $lookup stages
-db.setProfilingLevel(1, { slowms: 50 }) // Disable afterwards
-db.system.profile.find({
-  "command.aggregate": { $exists: true },
-  "command.pipeline.$lookup": {
-    $exists: true
-  }
-}).sort({ millis: -1 })

 // Check if $lookup foreign fields are indexed
-db.reviews.aggregate([
+
+// Example A - $indexStats
+db.categories.aggregate([
  { $indexStats: {} }
 ])
-// Look for index supporting the query in result
+
+// Example B - getIndexes()
+db.categories.getIndexes()
+
+// Look for index supporting the query (either a direct index on the foreign field or a compound index that has the foreign field as a prefix, note the collation)

 // Measure $lookup impact
 db.products.aggregate([
@@ -79,18 +79,13 @@ for (const d of db.adminCommand({ listDatabases: 1 }).databases) {
  print(`${d.name}: ${colls} collections`)
 }
 // Count alone is not sufficient: combine with access and index/storage evidence
-
-// Check if collections are always accessed together
-// If orders always needs customer, items, addresses
-// → they should be embedded
-db.system.profile.aggregate([
-  { $match: { op: "query" } },
-  { $group: { _id: "$ns", count: { $sum: 1 } } },
-  { $sort: { count: -1 } }
-])
-// Collections with similar access patterns should be combined
 ```

+### Check if collections are always accessed together.
+
+For Atlas M10+ use $queryStats. See [Query Stats](references/source-query-stats.md)
+Use codebase if available, ask the user.
+
 Atlas Schema Suggestions flags: "Reduce number of collections"

 Reference: [Reduce the Number of Collections](https://mongodb.com/docs/manual/data-modeling/design-antipatterns/reduce-collections/)
@@ -56,25 +56,23 @@ Use Approximation when staleness is acceptable. Use Computed when exact values a

 ## Verify with

-```javascript
-// Check write frequency on counter fields
-db.setProfilingLevel(1, { slowms: 0 })
-db.system.profile.find({
-  "command.update": "articles",
-  "command.updates.u.$inc.viewCount": { $exists: true }
-}).count()
-// High count relative to read count suggests approximation would help
+### Check write frequency on counter fields

+Use codebase if available, ask the user.
+
+High count relative to read count on a specific field suggests approximation would help
+
+```javascript
 // Compare counter staleness
 db.articles.aggregate([
+  { $sort: { lastSyncedAt: 1 } },
+  { $limit: 10 },
  { $project: {
    title: 1,
    viewCount: 1,
    lastSyncedAt: 1,
    staleness: { $subtract: ["$$NOW", "$lastSyncedAt"] }
-  }},
-  { $sort: { staleness: -1 } },
-  { $limit: 10 }
+  }}
 ])
 // Verify staleness is within acceptable bounds for your use case
 ```
@@ -147,25 +147,16 @@ On-demand materialized views are not automatically refreshed—you control when

 ## Verify with

-```javascript
-// Find expensive aggregations that should be pre-computed
-db.setProfilingLevel(1, { slowms: 100 }) // Disable afterwards
-db.system.profile.find({
-  "command.aggregate": { $exists: true },
-  millis: { $gt: 100 }
-}).sort({ millis: -1 })
+### Find expensive aggregations that should be pre-computed

-// Check if same aggregation runs repeatedly
-db.system.profile.aggregate([
-  { $match: { "command.aggregate": { $exists: true } } },
-  { $group: {
-    _id: "$command.pipeline",
-    count: { $sum: 1 },
-    avgMs: { $avg: "$millis" }
-  }},
-  { $match: { count: { $gt: 100 } } }  // Repeated 100+ times
-])
-// High count + high avgMs = candidate for computed pattern
-```
+For Atlas M10+ use slow query logs to find the slowest aggregations. See [Slow query logs](references/source-slow-query-logs.md). 
+Use codebase if available, ask the user.
+
+### Check if same aggregation runs repeatedly
+
+For Atlas M10+ use $queryStats. See [Query Stats](references/source-query-stats.md)
+Use codebase if available, ask the user.
+
+High count + high avgMs on an aggregation that computes a result = candidate for computed pattern

 Reference: [Computed Schema Pattern](https://mongodb.com/docs/manual/data-modeling/design-patterns/computed-values/computed-schema-pattern/)
@@ -64,26 +64,9 @@ Keep both a bare reference (`customerId`) and an optional cache subdocument (`cu

 ## Verify with

-```javascript
-// Find $lookup-heavy aggregations in profile
-db.setProfilingLevel(1, { slowms: 20 }) // Disable afterwards
-db.system.profile.find({
-  "command.aggregate": { $exists: true },
-  "command.pipeline.$lookup": {
-    $exists: true
-  }
-}).sort({ millis: -1 }).limit(10)
+Find lookup-heavy aggregations. See how often lookups hit the same collection. High count = candidate for extended reference

-// Check how often lookups hit same collections
-db.system.profile.aggregate([
-  { $match: { "command.pipeline.$lookup": { $exists: true } } },
-  { $project: { pipeline: "$command.pipeline" } },
-  { $unwind: "$pipeline" },
-  { $project: { lookup: { $getField: { field: { $literal: '$lookup' }, input: '$pipeline' } } } },
-  { $match: { "lookup": { $exists: true } } },
-  { $group: { _id: "$lookup.from", count: { $sum: 1 } } }
-])
-// High count = candidate for extended reference
-```
+For Atlas M10+ use $queryStats. See [Query Stats](references/source-query-stats.md) and [Slow query logs](references/source-slow-query-logs.md)
+Use codebase if available, ask the user.

 Reference: [Reduce $lookup Operations](https://mongodb.com/docs/manual/data-modeling/design-antipatterns/reduce-lookup-operations/)
@@ -0,0 +1,123 @@
+# Query Stats
+
+## When to use
+
+Analyzes query access patterns with minimal performance overhead. Use for identifying co-accessed fields, collection relationships, and query frequencies. Only supports `find`, `aggregate`, and `distinct` operations.
+
+## Requirements
+
+Atlas M10+ tier.
+
+## How to use
+
+Aggregate on the admin database.
+
+With mcp-server, use the `mcp__mongodb__aggregateDB` tool with database set to `admin`.
+
+```javascript
+db.getSiblingDB("admin").aggregate([{ $queryStats: {} }])
+```
+
+**Example 1: Find collections frequently queried together with others (embedding candidates)**
+```javascript
+db.aggregate([
+  { $queryStats: {} },
+  {
+    $match: {
+      "key.queryShape.cmdNs.db": "databaseName",
+      "key.queryShape.command": "aggregate",
+      "key.queryShape.pipeline.$lookup": { $exists: true }
+    }
+  },
+  { $unwind: "$key.queryShape.pipeline" },
+  { 
+    $match: { "key.queryShape.pipeline.$lookup": { $exists: true } } 
+  },
+  { 
+    $set: { 
+      stageKeyValue: { 
+        $first: { $objectToArray: "$key.queryShape.pipeline" } 
+      } 
+    }
+  },
+  { 
+    $group: {
+      _id: { 
+        source: "$key.queryShape.cmdNs.coll",
+        target: "$stageKeyValue.v.from"
+      },
+      totalLookupHits: { $sum: "$metrics.execCount" },
+      avgPipelineMs: {
+        $avg: { $divide: [
+          { $divide: ["$metrics.totalExecMicros.sum", 1000] },
+          "$metrics.execCount"
+        ]}
+      }
+    }
+  },
+  { $sort: { totalLookupHits: -1 } }
+])
+
+// High totalLookupHits = frequently joined 
+// High avgPipelineMS = lookup is part of slow queries (does not automatically mean that the $lookup is slow, could be the whole pipeline - see the full query shapes)
+// High scores on both - consider embedding to avoid $lookup 
+```
+
+**Example 1.1: Find query shapes that use $lookup on specific collections**
+```javascript
+db.aggregate([
+  { $queryStats: {} },
+  {
+    $match: {
+      "key.queryShape.cmdNs.db": "databaseName",
+      "key.queryShape.command": "aggregate",
+      "key.queryShape.cmdNs.coll": "sourceCollectionName",
+      "key.queryShape.pipeline.$lookup.from": "targetCollectionName"
+    }
+  },
+  {
+    $project: {
+      database: "$key.queryShape.cmdNs.db",
+      collection: "$key.queryShape.cmdNs.coll",
+      pipeline: "$key.queryShape.pipeline",
+      execCount: "$metrics.execCount",
+      avgMs: {
+          $divide: [
+            { $divide: ["$metrics.totalExecMicros.sum", 1000] },
+            "$metrics.execCount"
+          ]
+      }
+    },
+  },
+  { $sort: { execCount: -1 } },
+  { $limit: 10 }
+])
+```
+
+**Example 2: Find top most frequent query shapes (optimize hot paths)**
+```javascript
+db.getSiblingDB("admin").aggregate([
+  { $queryStats: {} },
+  { $sort: { "metrics.execCount": -1 } },
+  { $limit: 10 },
+  {
+    $project: {
+      command: "$key.queryShape.command",
+      database: "$key.queryShape.cmdNs.db",
+      collection: "$key.queryShape.cmdNs.coll",
+      queryShape: "$key.queryShape",
+      execCount: "$metrics.execCount",
+      avgMs: {
+        $divide: [
+          { $divide: ["$metrics.totalExecMicros.sum", 1000] },
+          "$metrics.execCount"
+        ]
+      }
+    }
+  }
+])
+
+// High execCount = hot path → design your schema for these queries first
+// Cross reference with avgMS or [slow query logs](references/source-slow-query-logs.md) to find queries that are both frequent and slow
+// Note: Query stats do not include write patterns (update, insert)
+```
@@ -0,0 +1,63 @@
+# Atlas Slow Query Logs
+
+## When to use
+
+Retrieves log lines for slow queries as determined by the Performance Advisor. Use to identify slow queries and performance bottlenecks. Provides actual query examples (not shapes) with execution times. Captures all operation types including writes, unlike Query Stats which currently only covers find/aggregate/distinct.
+
+## Requirements
+
+- Atlas M10+ cluster
+- Atlas API credentials configured
+- Performance Advisor enabled (enabled by default on M10+)
+
+If the API call returns auth or access errors, see the [Performance Advisor docs](https://www.mongodb.com/docs/atlas/performance-advisor/).
+
+## How to use
+
+Atlas Admin API endpoint ([query parameters reference](https://www.mongodb.com/docs/ops-manager/current/reference/api/performance-advisor/get-slow-queries/#request-query-parameters)):
+```
+GET /groups/{PROJECT-ID}/hosts/{HOST-ID}/performanceAdvisor/slowQueryLogs
+```
+
+With MongoDB MCP server:
+```javascript
+mcp__plugin_mongodb_mongodb__atlas-get-performance-advisor({
+  projectId: "507f1f77bcf86cd799439011",
+  clusterName: "MyCluster",
+  operations: ["slowQueryLogs"]
+})
+```
+
+Performance Advisor analyzes up to 200,000 of the cluster's most recent log lines.
+
+**Example response structure:**
+```javascript
+{
+  "slowQueries": [
+    {
+      "line": "2026-05-06T10:23:45.447+0000 I COMMAND [conn10614] command mydb.orders appName: \"MongoDB Shell\" command: find { find: \"orders\", filter: { status: \"pending\", customerId: 12345 }, sort: { createdAt: -1 } } planSummary: COLLSCAN keysExamined:0 docsExamined:50000 nreturned:100 executionTimeMillis:1247 ...",
+      "namespace": "mydb.orders"
+    }
+  ]
+}
+```
+
+The response contains raw log lines. Parse the log line to extract:
+- Timestamp (beginning of line)
+- Operation type (command: find, aggregate, update, etc.)
+- Query details (filter, pipeline, etc.)
+- Execution metrics (executionTimeMillis, docsExamined, planSummary, etc.)
+
+## What to Look For
+
+When analyzing slow query logs, focus on:
+
+**Slow $lookup operations:**
+- Look for `$lookup` in the log line
+- Consider embedding to reduce slow $lookup operations
+- Cross-reference with Query Stats to identify frequent lookups
+- High executionTimeMillis + high frequency = urgent schema redesign
+
+**Other slow aggregations:**
+- Consider the Computed Pattern to avoid slow aggregations
+