Chapter 18: Feature Releases: A/B Testing & Feature Flags

Welcome to Part Seven: Advanced Integration & Architecture Practices. In Part Six, we built a complete DevOps and observability stack. We now have:

1. A secure CI/CD pipeline that automatically runs tests before code merges (Chapter 14).
2. A comprehensive observability stack that tells us about performance, errors, and user behavior in production (Chapter 17).

We've solved the "how to safely deploy code" problem. But a new, more advanced question emerges: how do we safely release features?

"Deployment" and "Release" are two different concepts:

• Deployment: Pushing new code to production servers. This is a technical action.
• Release: Making new features available to real users. This is a business action.

In traditional Python/Django deployment workflows, these two actions are typically bundled: once code is git pulled and Gunicorn restarted, new features are immediately visible to 100% of users. This is a "Big Bang" release approach with extremely high risk. If the new feature has hidden bugs, performance bottlenecks, or simply isn't well-received by users, your only option is an emergency rollback.

The core principle of modern SAAS architecture is decoupling deployment from release. We want to be able to deploy new code to 100% of production, but only release it to 0%, 1%, 10%, or 50% of users. This enables us to "move fast and break things" in a controlled scope, testing new features and deciding whether to fully roll them out or urgently "turn them off" based on data from Sentry and Plausible/Umami (Chapter 17).

In this chapter, we'll explore two key techniques to achieve this: A/B Testing (for validating hypotheses) and Feature Flags (for safe releases).

18.1. [Skill Practice]: Implementing A/B Testing with Middleware Rewrite

A/B Testing is a controlled experiment to compare two (or more) versions of a feature to determine which performs better.

Scenario: Our marketing team isn't sure if the pricing page design at src/app/[locale]/pricing/page.tsx (Chapter 12) is optimal. They've designed a completely new page at src/app/[locale]/pricing-variant-b/page.tsx and hypothesize that this new page will improve the "click upgrade button" conversion rate by 10%.

How do we validate this hypothesis?

We can't simply replace the old page with the new one. We need to:

1. Randomly assign 50% of site visitors to "Group A" (Control), who see the old page.
2. Randomly assign the other 50% to "Group B" (Variant), who see the new page.
3. Both groups must see the exact same URL (e.g., .../pricing) to avoid confusion and SEO issues.
4. Use our analytics tools (Chapter 17) to track conversion rates for both groups.

In Next.js App Router, the best place to achieve this is Middleware.

[Skill Practice: claude-nextjs-skills/nextjs-advanced-routing/SKILL.md]

This Skill demonstrates how to use NextResponse.rewrite() for advanced routing. rewrite is a powerful mechanism that allows you to internally forward user requests to another page on the server while keeping the URL in the browser address bar unchanged.

Let's see how to apply this to A/B testing.

src/middleware.ts Code Analysis

// src/middleware.ts
import { NextRequest, NextResponse } from 'next/server';
import { get } from '@vercel/edge-config'; // (Optional) Use Vercel Edge Config for dynamic control

// Define our A/B test configuration
const AB_TEST_COOKIE = 'ab_test_pricing_page';
const PAGE_TO_TEST = '/pricing';

export async function middleware(request: NextRequest) {
  const { pathname } = request.nextUrl;

  // 1. Check if this is the page we want to test
  // (Note: need to handle [locale] i18n paths)
  if (!pathname.endsWith(PAGE_TO_TEST)) {
    return NextResponse.next();
  }

  // (Optional) We can dynamically read from Vercel Edge Config whether to enable A/B testing
  // const abTestEnabled = await get('enablePricingAbTest');
  // if (!abTestEnabled) return NextResponse.next();

  // 2. Check if user already has a bucket cookie
  let bucket = request.cookies.get(AB_TEST_COOKIE)?.value;
  let hasBucket = !!bucket;

  // 3. If not, randomly assign one
  if (!bucket) {
    const random = Math.random();
    bucket = random < 0.5 ? 'control' : 'variant';
  }

  // 4. Use NextResponse.rewrite() to show the variant
  // (Assuming i18n paths are `/en/pricing`, `/jp/pricing`, etc.)
  const [_, locale] = pathname.match(/\/([a-z]{2})\/pricing/) || [];
  const url = request.nextUrl.clone(); // Clone the URL object

  if (bucket === 'variant') {
    // Key! Rewrite to Group B page
    url.pathname = `/${locale}/pricing-variant-b`;
  } else {
    // Group A users, do nothing (or explicitly rewrite to original page)
    // url.pathname = `/${locale}/pricing`; // This is default behavior
  }

  // 5. Create response
  // NextResponse.rewrite() returns a special response telling Next.js to render another page
  const response = NextResponse.rewrite(url);

  // 6. If new user, set cookie so they stay in same bucket on next visit
  if (!hasBucket) {
    response.cookies.set(AB_TEST_COOKIE, bucket, {
      httpOnly: true,
      maxAge: 60 * 60 * 24 * 7, // 7 days
    });
  }

  return response;
}

// Edge runtime configuration
export const config = {
  matcher: [
    /*
     * Match all request paths except:
     * - /api (API routes)
     * - /_next/static (static files)
     * - /_next/image (image optimization)
     * - /favicon.ico (favicon)
     */
    '/((?!api|_next/static|_next/image|favicon.ico).*)',
  ],
};

File Structure

Our App Router directory now looks like this:

src/app/[locale]/
├── pricing/
│   └── page.tsx      # Group A (Control) sees this
├── pricing-variant-b/
│   └── page.tsx      # Group B (Variant) sees this
└── ...

With this setup, both groups of users only see https://yourdomain.com/en/pricing in their browsers, but Middleware decides on the server whether to render pricing/page.tsx or pricing-variant-b/page.tsx based on the cookie.

Finally, in pricing-variant-b/page.tsx, we'll trigger a specific analytics event to compare performance of both groups in Plausible/Umami:

// src/app/[locale]/pricing-variant-b/page.tsx
'use client';
import { useEffect } from 'react';
import { trackEvent } from '@/analytics/events';
import { PricingTable } from '@/components/payment/PricingTable'; // (Maybe new pricing table)

export default function PricingVariantPage() {
  useEffect(() => {
    // Report that this user saw Group B page
    trackEvent('view_pricing_page', { variant: 'b' });
  }, []);

  // ... Render new pricing page UI
  return (
    <div>
      <h1>Our New Pricing!</h1>
      {/* <NewPricingTable /> */}
    </div>
  );
}

18.2. [Code Analysis]: Feature Flags

Feature Flags are a simpler, more direct release control technique. They're not for "testing" but for "safety".

A feature flag is a boolean value (true / false) that allows you to wrap a new feature in your code, turning it on or off without redeploying.

Scenario: We've developed a brand new AI-powered "Dashboard Summary" feature (<DashboardSummary />). This feature relies on an expensive third-party AI API, and we're unsure about its performance and cost in production.

We want to:

1. Deploy code containing the <DashboardSummary /> component but keep it "off" by default.
2. First "turn it on" for internal employees (or specific test users) for "dogfooding".
3. After monitoring for no errors in Sentry (Chapter 17), turn it on for 10% of users.
4. If Sentry reports a spike in errors, immediately "turn it off" without needing an emergency code rollback.

"Static" vs "Dynamic" Flags

1. Static Flags - Book Practice

This is the simplest implementation. Flags are defined in a config file in the codebase.

src/config/website.tsx (or features.ts)

// src/config/website.tsx

export const featureFlags = {
  // Description: Enable new AI dashboard summary feature
  // Status: Off by default, awaiting production validation
  enableAiDashboard: process.env.NEXT_PUBLIC_FLAG_AI_DASHBOARD === 'true' || false,

  // Description: Enable new navbar design
  // Status: Rolled out to 100% of users
  enableNewNavbar: true,

  // Description: Allow users to purchase "Lifetime Deal"
  // Status: Only true in .env.local for local development
  enableLifetimeDeal: process.env.NODE_ENV === 'development',
};

How to use it?

• In Server Components (RSC):

  // src/app/[locale]/dashboard/page.tsx
  import { featureFlags } from '@/config/website';
  import { DashboardSummary } from '@/components/dashboard/DashboardSummary';
  
  export default async function DashboardPage() {
    // Read config directly on server
    const showAiSummary = featureFlags.enableAiDashboard;
  
    return (
      <div>
        <h1>Your Dashboard</h1>
  
        {/* Feature flag wraps new component */}
        {showAiSummary && <DashboardSummary />}
  
        {/* Old component */}
        {!showAiSummary && <OldDashboardMetrics />}
      </div>
    );
  }

• In Client Components ("use client"):

  // src/components/layout/Navbar.tsx
  'use client';
  import { featureFlags } from '@/config/website';
  import { OldNavbar } from './OldNavbar';
  import { NewNavbar } from './NewNavbar';
  
  export function Navbar() {
    // Client components can directly import and read
    // (assuming flag comes from NEXT_PUBLIC_ or is determined at build time)
    if (featureFlags.enableNewNavbar) {
      return <NewNavbar />;
    }
    return <OldNavbar />;
  }

This "static" flag approach has the advantage of zero cost, zero dependencies, and simple implementation. The disadvantage is that "turning off" a feature (changing process.env.NEXT_PUBLIC_FLAG_AI_DASHBOARD) still requires you to rebuild and redeploy the application (on Vercel). This doesn't achieve "instant off".

2. Dynamic Flags - Vercel Flags / LaunchDarkly

To achieve "instant off", you need a dynamic feature flag system.

• Vercel Flags: Vercel-provided feature integrated with Vercel Edge Config (an ultra-low-latency key-value store).
• LaunchDarkly / PostHog: Third-party professional feature flagging platforms.

With these systems, the featureFlags object no longer reads from a local file but fetches in real-time from an external SDK:

// Pseudo-code: Using Vercel Flags (dynamic)
import { checkFlag } from '@vercel/flags';

export default async function DashboardPage() {
  // Check Edge Config in real-time on every request
  const showAiSummary = await checkFlag('enableAiDashboard');

  return (
    <div>
      {showAiSummary && <DashboardSummary />}
      {/* ... */}
    </div>
  );
}

Now, if you see a spike in errors in Sentry, you just log into Vercel (or LaunchDarkly) dashboard and flip the enableAiDashboard switch to "off". The next time users refresh, the feature disappears immediately.

For our SAAS template, static flags starting from src/config/website.tsx are a perfect starting point, already solving 90% of safe release problems.

Chapter 18 Summary

In this chapter, we mastered two core techniques for safe, progressive feature releases, completely decoupling "deployment" from "release".

• A/B Testing (Middleware): We leveraged Next.js Middleware and NextResponse.rewrite() to implement a powerful A/B testing system. This enables us to show different versions of a page (while keeping the URL the same) to different user groups and drive design and business decisions with analytics data.
• Feature Flags: We analyzed both "static" (based on config.ts) and "dynamic" (based on Vercel Flags/LaunchDarkly) feature flags. By using simple if (featureFlags.myFeature) checks in our code (whether in server or client components), we gained granular control over feature rollouts.

This provides the final piece of our DevOps puzzle. We can now not only Continuously Integrate (CI, Ch 14), Continuously Deploy (CD, Ch 14), and Continuously Monitor (Observability, Ch 17), but also Continuously Release & Validate (Flags & A/B Tests, Ch 18). This complete cycle enables our SAAS team to innovate with high speed and low risk.