How We Built a Scalable,
Production-Ready 6-Agent LangGraph System

At Rainbow Intelligence, we recently completed a 6-agent LangGraph system for automated B2B sales outreach. This is a technical walkthrough of the architecture, the decisions we made, the problems we hit, and what we would do differently.

If you are building with LangGraph in production — or planning to — this is the article I wish I had when I started.

---

What we built and why

The system is called ProspAI — an autonomous B2B sales outreach engine. It does the following without human intervention:

• Accepts a plain-English description of an ideal customer profile
• Extracts structured search parameters using an LLM
• Discovers matching prospects via Apollo.io API
• Generates a unique personalised email for each prospect
• Sends emails from the client's Gmail or Outlook account
• Runs a multi-step follow-up sequence over 25 days
• Tracks opens, clicks, and replies and notifies the client of hot leads

This is not a chatbot. This is a workflow — a sequence of decisions, tool calls, and side effects that must run reliably, handle failures gracefully, and scale to hundreds of prospects per client.

---

Why LangGraph and not something else

Before we committed to LangGraph, we evaluated three options:

LangGraph is harder to learn than CrewAI. But in production, you want explicit control — not magic.

---

The architecture — 6 agents, one graph

---

The state management design

The most critical design decision in any LangGraph system is the state schema. Get this wrong and you will be refactoring everything in week 4. Here is the schema we settled on after two iterations:

from typing import TypedDict, List, Optional
from langgraph.graph import StateGraph

class CampaignState(TypedDict):
    # Input
    tenant_id: str
    raw_icp: str           # plain English from client

    # ICP Extraction
    structured_icp: Optional[dict]   # structured JSON
    icp_confirmed: bool              # client confirmed?
    clarification_needed: bool       # needs more info?

    # Lead Discovery
    leads: List[dict]         # raw from Apollo
    scored_leads: List[dict]  # after ICP scoring

    # Outreach
    emails_generated: List[dict]  # personalised emails
    emails_sent: int              # count sent

    # Control
    current_step: str    # which agent is active
    errors: List[str]    # error log
    retry_count: int     # retry tracking

Key lessons from this schema design:

• Keep state flat — deeply nested state is hard to debug. Keep everything at the top level.
• Track errors explicitly — have a dedicated errors list. Do not let failures disappear silently.
• Always track tenant_id — critical for multi-tenant isolation. Never forget this field.
• Add retry_count from the start — you will need it. LLM calls fail. APIs time out. Build retry logic in from day one.

---

The email generation problem — and how we solved it

The hardest engineering problem in the whole system was email generation at scale. We tried three approaches:

Approach 1 — One call per email (rejected)

600 individual GPT-4o calls. Each call: ~1,000 tokens. Total: ~600,000 tokens. Cost: ~₹300 per campaign. Time: ~60 minutes. Unacceptable on both dimensions.

Approach 2 — One call for all 600 (rejected)

One massive prompt with all 600 prospect records. The LLM gets confused between records at this scale — wrong names in wrong emails. Hallucination rate too high. Completely unusable.

Approach 3 — Batches of 10 with validation (adopted ✅)

Send 10 prospects per LLM call. Ask for a JSON array of 10 emails back. Validate each email — does it contain the correct name? Correct company? Is it above 100 characters? Is the similarity score between emails below 0.85?

# Validation function (simplified)
def validate_email_batch(prospects, emails):
    valid = []
    retry = []
    for prospect, email in zip(prospects, emails):
        if prospect['name'] not in email['body']:
            retry.append(prospect)       # wrong name
        elif len(email['body']) < 100:
            retry.append(prospect)       # too short
        elif similarity_score(email, valid) > 0.85:
            retry.append(prospect)       # too similar
        else:
            valid.append(email)
    return valid, retry

Batch size 10 is the sweet spot. Small enough to avoid model confusion. Large enough to be cost-efficient.
Failed emails are retried individually — bringing error rate below 2% and cost to ₹50 per 600-email campaign.

---

The 5 production problems nobody warns you about

Problem 1 — LangGraph state persistence

LangGraph's default in-memory state disappears if your server restarts. For a multi-step workflow running over hours, this is catastrophic. Solution: use PostgreSQL checkpointing from the start.

from langgraph.checkpoint.postgres import PostgresSaver

checkpointer = PostgresSaver.from_conn_string(DATABASE_URL)
graph = workflow.compile(checkpointer=checkpointer)

Problem 2 — API rate limits at scale

Apollo.io, OpenAI, and Gmail all have rate limits. When running campaigns for 10 clients simultaneously, you hit them. Solution: implement exponential backoff and per-client rate limit queues using Celery.

Problem 3 — Multi-tenant data isolation

Every database query must filter by tenant_id. Miss even one and you have a data leak between clients. Solution: Row-Level Security (RLS) in PostgreSQL — it enforces isolation at the database level, not just application level.

-- PostgreSQL RLS policy
CREATE POLICY tenant_isolation ON campaigns
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

Problem 4 — LLM non-determinism in production

The same prompt can produce different outputs on different runs. For ICP extraction where we need consistent JSON structure, this caused failures. Solution: use temperature=0.1 for structured output tasks and always validate with Pydantic.

Problem 5 — Celery task visibility

When 600 emails are being sent over 3 days, you need to know which ones were sent, which failed, and which are pending. Default Celery gives you almost no visibility. Solution: use Flower for monitoring and log every task state change to PostgreSQL.

---

The tech stack — final production version

---

What we would do differently

• Set up LangSmith on day 1 — we added it in week 5. Every LLM debugging session before that was painful. Add it first.
• Use PostgreSQL checkpointing from the start — we migrated from in-memory state in week 3. The migration was painful.
• Write integration tests for every agent — not just unit tests. We found 3 critical bugs only when agents interacted that passed individual testing.
• Build the monitoring dashboard before showing to clients — our first client asked 'how many emails have been sent?' and we had no UI to show them.
• Cap batch size at 10, not 15 — we tried 15 initially. Quality dropped noticeably. 10 is the sweet spot for GPT-4o.

---

Key takeaways for engineers building with LangGraph