Alerting & Debugging

Introduction

Monitoring tells you what happened. Alerting tells you when something is wrong before users notice. Debugging tells you why. This lesson covers setting up alerts for MCP server anomalies, debugging failed tool invocations with structured error logs, tracing slow requests through latency analysis, and using the MCP Inspector for production debugging.

Key Concepts

• Error Rate Alerting: Set thresholds for error rate spikes per tool. A tool that normally fails 0.1% of the time spiking to 5% warrants immediate investigation. Alert on rate of change, not just absolute values.
• Latency Alerting: Alert on P95 and P99 latency, not averages. A P99 breach means the slowest 1% of requests are degraded, which often indicates resource contention or downstream issues.
• Client-Side vs Server-Side Errors: Distinguish between errors caused by invalid client input (4xx equivalent) and errors caused by server-side failures (5xx equivalent). Client errors indicate bad tool usage. Server errors indicate bugs or infrastructure problems.
• MCP Inspector: The official debugging tool for MCP servers, invoked via npx @modelcontextprotocol/inspector. It connects to your server, lists available tools and resources, and lets you invoke them interactively with full request/response visibility.
• Structured Error Logs: Every error log should include the tool name, input parameters (sanitized), error type, error message, stack trace, and duration. This gives debuggers everything they need without asking follow-up questions.

Real World Context

At 3 AM, your MCP server's deploy_service tool starts returning errors. Without alerting, the team discovers the problem at 9 AM when a developer complains. With proper alerting, PagerDuty fires at 3:02 AM, the on-call engineer opens the structured error logs, sees "connection refused" errors to the deployment API, checks the downstream service status page, and finds a scheduled maintenance window that was not communicated. Total incident response time: 8 minutes instead of 6 hours.

Deep Dive

Setting Up Error Rate Alerts

Define alert rules based on the metrics you are exporting to Prometheus. Here is a Prometheus alerting rule for MCP tool error rate:

yaml
# prometheus-alerts.yml
groups:
  - name: mcp-server-alerts
    rules:
      - alert: MCPToolHighErrorRate
        expr: |
          rate(mcp_tool_errors_total[5m])
          / rate(mcp_tool_invocations_total[5m])
          > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "MCP tool {{ $labels.tool_name }} error rate above 5%"
          description: "Tool {{ $labels.tool_name }} has a {{ $value | humanizePercentage }} error rate over the last 5 minutes."

      - alert: MCPToolHighLatency
        expr: |
          histogram_quantile(0.99,
            rate(mcp_tool_duration_seconds_bucket[5m])
          ) > 5
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "MCP tool P99 latency above 5 seconds"

The first rule fires when any tool's error rate exceeds 5% for more than 2 minutes. The second fires when P99 latency exceeds 5 seconds for 3 minutes. Adjust thresholds based on your tool's expected behavior.

Debugging Failed Tool Invocations

When an alert fires, structured error logs are your first line of investigation. Emit comprehensive error context:

typescript
server.tool(
  "deploy_service",
  "Deploy a service to production",
  { service: z.string(), version: z.string() },
  async ({ service, version }) => {
    try {
      const result = await deploymentAPI.deploy(service, version);
      return {
        content: [{ type: "text", text: `Deployed ${service}@${version}` }]
      };
    } catch (error) {
      server.sendLoggingMessage({
        level: "error",
        logger: "tool-handler",
        data: {
          tool: "deploy_service",
          input: { service, version },
          error_type: error.constructor.name,
          error_message: error.message,
          error_code: error.code || "UNKNOWN",
          stack: error.stack,
          timestamp: new Date().toISOString()
        }
      });

      return {
        content: [{
          type: "text",
          text: `Deployment failed: ${error.message}`
        }],
        isError: true
      };
    }
  }
);

Every field in the error log is searchable. You can query for all errors with error_type: "ConnectionRefused" or all failures for a specific service.

Using MCP Inspector for Production Debugging

The MCP Inspector is an interactive debugging tool that connects to your server and lets you test tools manually:

bash
npx @modelcontextprotocol/inspector node ./dist/server.js

The Inspector opens a web UI where you can see all registered tools, resources, and prompts. You can invoke any tool with custom parameters and see the full JSON-RPC request and response. This is invaluable for reproducing production issues in a controlled environment.

Use the Inspector to verify that your server handles edge cases correctly: empty inputs, missing required fields, oversized payloads, and concurrent invocations.

Client-Side vs Server-Side Error Attribution

Not all errors are your server's fault. Distinguish between the two:

text
Client-Side Errors             Server-Side Errors
──────────────────────────────  ──────────────────────────────
Invalid input parameters        Database connection failures
Missing required fields         Timeout on downstream API
Schema validation failures      Out of memory
Unsupported tool name           Unhandled exceptions
Malformed JSON-RPC              Disk full

Client-side errors should be logged at warning level and should not trigger alerts. Server-side errors should be logged at error level and should trigger alerts. Mixing them inflates your error rate metrics and creates alert fatigue.

Common Pitfalls

1. Alerting on every error — Some tools have inherent failure modes (network timeouts, rate limits). Set alert thresholds above baseline error rates to avoid alert fatigue. A 0% error rate target is unrealistic for tools that call external services.
2. Insufficient context in error logs — An error log that says "tool failed" without the tool name, input, or stack trace is useless. Always include enough context to reproduce the issue without access to the original request.
3. Ignoring slow requests in favor of errors — A tool that never errors but takes 30 seconds to respond is still broken from the user's perspective. Monitor latency as aggressively as you monitor errors.

Best Practices

1. Use the MCP Inspector as part of your deployment checklist — Before deploying a new server version, run it through the Inspector to verify all tools respond correctly. This catches regressions that automated tests might miss.
2. Separate alert channels by severity — Critical alerts (error rate spikes) go to PagerDuty. Warning alerts (latency increases) go to Slack. Informational alerts (traffic changes) go to email. This prevents desensitization.
3. Build runbooks for common alert scenarios — When an alert fires at 3 AM, the on-call engineer should have a runbook that says: check these dashboards, run these queries, escalate to this team if the issue is in a downstream service.

Summary

• Set up error rate and latency alerts with meaningful thresholds and reasonable evaluation windows.
• Emit structured error logs with tool name, input, error type, message, and stack trace for every failure.
• Distinguish client-side errors (invalid input) from server-side errors (infrastructure failures) to avoid alert noise.
• Use npx @modelcontextprotocol/inspector for interactive debugging and pre-deployment verification.
• Monitor P95/P99 latency alongside error rates to catch performance degradation before it becomes an outage.

npx @modelcontextprotocol/inspector node ./dist/server.js

server.sendLoggingMessage({
  level: "error",
  logger: "tool-handler",
  data: {
    tool: "deploy_service",
    input: { service: "api-gateway", version: "2.4.1" },
    error_type: "ConnectionRefused",
    error_message: "connect ECONNREFUSED 10.0.1.50:443",
    timestamp: new Date().toISOString()
  }
});