Scaling MCP Servers

Introduction

A single MCP server process can handle a limited number of concurrent sessions. As usage grows, you need strategies for horizontal scaling, session management, connection pooling, and rate limiting. The key challenge is that MCP sessions can be stateful, which complicates traditional load balancing approaches.

Key Concepts

• Stateless vs Stateful Servers: Stateless servers treat each request independently and scale trivially. Stateful servers maintain per-session data and require session affinity or external state stores.
• Session Affinity (Sticky Sessions): A load balancer configuration that routes all requests from a given session to the same backend instance.
• Connection Pooling: Reusing database connections across tool invocations instead of creating new ones for each request.
• Rate Limiting: Constraining the number of tool invocations per user, per tool, or globally to prevent abuse and ensure fair resource distribution.
• Backpressure: A mechanism for handling slow consumers by slowing down producers, preventing memory exhaustion from unbounded queues.

Real World Context

A popular MCP server providing code analysis tools handles 10,000 sessions during peak hours. Without connection pooling, each tool invocation opens a new database connection, exhausting the database's connection limit at 500. Without rate limiting, a single user running automated scripts can starve other users. Without session affinity, a multi-step analysis workflow breaks because follow-up requests hit a different server instance that doesn't have the workflow state.

Deep Dive

Stateless Server Design

Prefer stateless design when possible. Store all session state externally:

typescript
import { Redis } from "ioredis";

const redis = new Redis(process.env.REDIS_URL);

// Store session state in Redis instead of in-memory
async function getSessionState(sessionId: string): Promise<SessionState> {
  const data = await redis.get(`mcp:session:${sessionId}`);
  if (!data) throw new Error("Session not found");
  return JSON.parse(data);
}

async function setSessionState(sessionId: string, state: SessionState): Promise<void> {
  await redis.set(
    `mcp:session:${sessionId}`,
    JSON.stringify(state),
    "EX",
    3600 // 1 hour TTL
  );
}

// Tools access session state from Redis
server.tool("continue_analysis", "Continue a multi-step analysis",
  { sessionId: z.string().uuid() },
  async ({ sessionId }) => {
    const state = await getSessionState(sessionId);
    // Process next step using stored state
    state.currentStep++;
    await setSessionState(sessionId, state);
    return { content: [{ type: "text", text: `Step ${state.currentStep} complete` }] };
  }
);

With Redis-backed state, any server instance can handle any session, enabling simple round-robin load balancing.

Session Affinity with Load Balancers

When stateless design is not feasible, configure session affinity:

nginx
# Nginx configuration for sticky sessions
upstream mcp_servers {
    hash $http_mcp_session_id consistent;
    server mcp-1:3100;
    server mcp-2:3100;
    server mcp-3:3100;
}

server {
    listen 443 ssl;
    server_name mcp.example.com;

    location /mcp {
        proxy_pass http://mcp_servers;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_read_timeout 300s;  # Long timeout for streaming
    }

    location /health {
        proxy_pass http://mcp_servers;
    }
}

The hash $http_mcp_session_id consistent directive routes all requests with the same MCP session ID to the same backend. The consistent keyword minimizes session disruption when servers are added or removed.

Connection Pooling

Reuse database connections across tool invocations:

typescript
import { Pool } from "pg";

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  max: 20,           // Maximum connections in pool
  min: 5,            // Minimum idle connections
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
  maxUses: 7500      // Close connection after N uses (prevents leaks)
});

// Monitor pool health
setInterval(() => {
  console.log({
    total: pool.totalCount,
    idle: pool.idleCount,
    waiting: pool.waitingCount
  });
}, 60000);

server.tool("query", "Run a database query",
  { sql: z.string(), params: z.array(z.unknown()).optional() },
  async (args) => {
    const client = await pool.connect();
    try {
      const result = await client.query(args.sql, args.params);
      return { content: [{ type: "text", text: JSON.stringify(result.rows) }] };
    } finally {
      client.release(); // Return connection to pool
    }
  }
);

The pool maintains 5-20 connections, reusing them across all tool invocations. The maxUses setting prevents connection leaks from accumulating over time.

Rate Limiting

Implement multi-tier rate limiting to prevent abuse:

typescript
import { RateLimiterRedis } from "rate-limiter-flexible";

const rateLimiters = {
  // Per-user: 100 requests per minute
  perUser: new RateLimiterRedis({
    storeClient: redis,
    keyPrefix: "rl:user",
    points: 100,
    duration: 60
  }),
  // Per-tool: 20 requests per minute for expensive tools
  perTool: new RateLimiterRedis({
    storeClient: redis,
    keyPrefix: "rl:tool",
    points: 20,
    duration: 60
  }),
  // Global: 1000 requests per minute
  global: new RateLimiterRedis({
    storeClient: redis,
    keyPrefix: "rl:global",
    points: 1000,
    duration: 60
  })
};

async function checkRateLimit(userId: string, toolName: string): Promise<void> {
  try {
    await Promise.all([
      rateLimiters.global.consume("global"),
      rateLimiters.perUser.consume(userId),
      rateLimiters.perTool.consume(`${userId}:${toolName}`)
    ]);
  } catch (rejection) {
    const retryAfter = Math.ceil(rejection.msBeforeNext / 1000);
    throw new Error(`Rate limit exceeded. Retry after ${retryAfter} seconds.`);
  }
}

Three tiers ensure that no single user or tool can overwhelm the system, while the global limit protects against distributed attacks.

Common Pitfalls

1. In-memory session state with multiple instances: If session state lives in server memory and the load balancer routes a request to a different instance, the session is lost. Use external state stores.
2. No connection pooling: Creating a new database connection per request is slow (50-100ms overhead) and can exhaust database connection limits under load.
3. Rate limiting only at the API gateway: Gateway-level rate limiting protects the gateway but not the MCP server. Apply rate limits at both layers.

Best Practices

1. Design for statelessness: Store session state in Redis or a database. This enables horizontal scaling with simple round-robin load balancing.
2. Pool all external connections: Database, Redis, and HTTP client connections should all use pooling to minimize overhead and prevent resource exhaustion.
3. Implement tiered rate limiting: Combine per-user, per-tool, and global rate limits to provide fair access while preventing abuse.

Summary

• Stateless server design with external state stores enables trivial horizontal scaling
• Session affinity routes same-session requests to the same instance when statefulness is unavoidable
• Connection pooling prevents database exhaustion and reduces per-request latency
• Multi-tier rate limiting (per-user, per-tool, global) ensures fair access and abuse prevention
• Backpressure mechanisms prevent memory exhaustion from slow consumers

import { RateLimiterRedis } from "rate-limiter-flexible";
import { Redis } from "ioredis";

const redis = new Redis(process.env.REDIS_URL!);

const limiter = new RateLimiterRedis({
  storeClient: redis,
  keyPrefix: "rl:mcp",
  points: 100,     // 100 invocations
  duration: 60     // per 60 seconds
});

async function checkRateLimit(userId: string): Promise<void> {
  try {
    await limiter.consume(userId);
  } catch (err) {
    throw new Error(`Rate limit exceeded. Retry after ${Math.ceil(err.msBeforeNext / 1000)}s`);
  }
}