How to Use Graph RAG and Specialized Agents to Repair Playwright Tests

|

Modern end-to-end test suites rarely fail for just one reason. A Playwright test can break because a button was renamed, a component moved, an API response changed, a fixture became stale, authentication behavior was modified, or a seemingly harmless UI refactor invalidated a locator. In a large repository, identifying the actual cause can require understanding relationships among tests, source code, selectors, routes, API contracts, fixtures, commits, and previous failures.

This is where Graph RAG (Graph Retrieval-Augmented Generation) combined with specialized agents can provide a more systematic approach to automated test repair.

Instead of asking a single AI agent to inspect a failing test and guess a fix, we can build a graph representing the relationships inside the application and test suite. A set of specialized agents can then investigate different dimensions of the failure: locator changes, application behavior, API contracts, historical fixes, and test execution. The agents collaborate around the graph and produce a repair that can be validated automatically.

The result is not simply an AI-generated code modification. It is a workflow in which retrieval, reasoning, code editing, and verification are separated into controlled stages.

What Is Graph RAG?

Traditional RAG retrieves documents that are semantically similar to a question. For example, given the failing test:

await page.getByRole('button', { name: 'Submit order' }).click();

a conventional vector search might retrieve documentation containing the words “Submit”, “order”, and “button”.

That can be useful, but software repositories contain relationships that plain similarity search does not represent particularly well.

Consider these relationships:

Test
 ├── uses → Locator
 ├── navigates_to → Route
 ├── depends_on → Fixture
 └── validates → API

Locator
 └── targets → Component

Component
 ├── calls → API
 └── rendered_on → Page

API
 └── defined_by → OpenAPI schema

A graph database can represent these relationships explicitly.

Graph RAG combines this graph representation with retrieval and language-model reasoning.

Instead of asking:

“What documents look similar to this failing test?”

we can ask:

“What application components, routes, APIs, fixtures, historical changes, and tests are connected to this failing test?”

That difference is extremely important for test repair.

Why Playwright Tests Are Good Candidates for Graph RAG

Playwright tests contain many structured relationships.

For example:

test('user can create an order', async ({ page }) => {
  await page.goto('/checkout');

  await page.getByRole('textbox', { name: 'Address' })
    .fill('123 Main Street');

  await page.getByRole('button', { name: 'Place order' })
    .click();

  await expect(page.getByText('Order confirmed'))
    .toBeVisible();
});

This test is connected to:

  • the /checkout route,
  • the address input,
  • the order button,
  • the confirmation message,
  • the checkout page component,
  • the order API,
  • authentication fixtures,
  • potentially a database fixture,
  • and other tests covering the same functionality.

If the test fails, the important information may not be contained inside the test file itself.

Suppose the application changed:

<button aria-label="Confirm purchase">
  Place order
</button>

The visible text may remain similar while the accessible name changes.

A graph can connect the failed locator to the component responsible for rendering it and to the Git history showing that the component recently changed.

Designing the Test Knowledge Graph

A useful graph starts with carefully chosen entities.

A simple model might contain:

(:Test)
(:TestStep)
(:Locator)
(:Page)
(:Component)
(:Route)
(:ApiEndpoint)
(:Fixture)
(:Failure)
(:Commit)
(:File)
(:Symbol)
(:Selector)

Relationships could include:

(Test)-[:CONTAINS]->(TestStep)
(TestStep)-[:USES]->(Locator)
(Test)-[:NAVIGATES_TO]->(Route)
(Route)-[:RENDERS]->(Component)
(Component)-[:DEFINED_IN]->(File)
(Component)-[:CALLS]->(ApiEndpoint)
(Test)-[:USES_FIXTURE]->(Fixture)
(Test)-[:FAILED_WITH]->(Failure)
(File)-[:CHANGED_IN]->(Commit)
(Commit)-[:MODIFIED]->(Component)

The graph does not need to contain every line of code. Its purpose is to preserve the relationships that are useful for diagnosis.

Building Graph Nodes From Playwright Tests

You can begin with a simple TypeScript parser.

For example:

import { readFileSync } from 'node:fs';

const source = readFileSync(
  'tests/checkout.spec.ts',
  'utf8'
);

const locatorPattern =
  /getByRole\(['"]([^'"]+)['"](?:,\s*\{\s*name:\s*['"]([^'"]+)['"])?/g;

let match;

while ((match = locatorPattern.exec(source))) {
  console.log({
    role: match[1],
    name: match[2]
  });
}

This is deliberately simple. A production implementation should preferably use an AST parser rather than regular expressions because Playwright tests can construct locators dynamically.

The extracted information could become graph nodes such as:

{
  "type": "Locator",
  "role": "button",
  "name": "Place order",
  "file": "tests/checkout.spec.ts"
}

The next step is connecting that locator to the test and application component.

Adding Application-Code Relationships

Suppose the checkout page contains:

export function CheckoutPage() {
  return (
    <section>
      <AddressForm />
      <OrderSummary />
      <button aria-label="Confirm purchase">
        Place order
      </button>
    </section>
  );
}

A repository-analysis agent can identify that the button belongs to CheckoutPage.

The resulting graph can contain:

checkout.spec.ts
       |
       | USES
       v
"Place order" locator
       |
       | TARGETS
       v
CheckoutPage
       |
       | DEFINED_IN
       v
CheckoutPage.tsx

This gives the repair system a route into the application’s implementation.

Adding Git History to the Graph

Git history is particularly valuable for automated repair.

Imagine the failing locator was previously valid.

The graph can represent:

CheckoutPage.tsx
       |
       | CHANGED_IN
       v
Commit abc123
       |
       | CHANGED
       v
aria-label:
"Place order"
        ↓
"Confirm purchase"

Now the repair system has evidence that the locator changed because of a specific commit.

A specialized history agent can inspect:

git log --all -- tests/checkout.spec.ts

and:

git log --all -S'Place order' -- src/components/CheckoutPage.tsx

It can also inspect the relevant diff:

git show abc123

This is much more reliable than asking an LLM to invent a plausible replacement.

Introducing Specialized Agents

The next important part is dividing the repair task among specialized agents.

A useful architecture might contain six agents:

                    ┌──────────────────┐
                    │ Repair Coordinator│
                    └─────────┬────────┘
                              │
       ┌──────────────┬───────┼────────┬───────────────┐
       │              │       │        │               │
       ▼              ▼       ▼        ▼               ▼
   Failure        Locator   Source    History         API
   Agent          Agent     Agent     Agent           Agent
       │              │       │        │               │
       └──────────────┴───────┼────────┴───────────────┘
                              ▼
                     Validation Agent

Each agent has a narrow responsibility.

That specialization is important because a single general-purpose agent tends to mix diagnosis, speculation, editing, and validation together.

The Failure Analysis Agent

The failure agent receives Playwright output such as:

Error: locator.click: Timeout exceeded

waiting for getByRole('button', { name: 'Place order' })

It extracts structured information:

{
  "test": "user can create an order",
  "operation": "click",
  "locator": {
    "role": "button",
    "name": "Place order"
  },
  "error": "timeout",
  "timeoutMs": 30000
}

The agent should not immediately modify the test.

Its job is to classify the failure.

Possible categories include:

LOCATOR_CHANGED
PAGE_NOT_REACHED
AUTHENTICATION_FAILURE
API_FAILURE
TIMING_FAILURE
FIXTURE_FAILURE
APPLICATION_REGRESSION
ENVIRONMENT_FAILURE
UNKNOWN

This classification controls which specialized agents should investigate next.

The Locator Agent

The locator agent focuses exclusively on selectors.

For example, it can search the DOM or source code for alternatives:

const candidates = [
  page.getByRole('button', { name: 'Confirm purchase' }),
  page.getByText('Place order'),
  page.locator('[aria-label="Confirm purchase"]')
];

However, the agent should prefer robust selectors.

A repair proposal might therefore be represented as:

{
  "old": "getByRole('button', { name: 'Place order' })",
  "new": "getByRole('button', { name: 'Confirm purchase' })",
  "reason": "Accessible name changed in CheckoutPage",
  "confidence": 0.94
}

The important principle is that confidence should be based on evidence rather than generated merely because the model “feels certain.”

The Source-Code Agent

The source agent investigates what the application currently does.

For example, it may retrieve:

<button
  aria-label="Confirm purchase"
  disabled={isSubmitting}
>
  Place order
</button>

It can determine that the button exists but has a different accessible name.

It might also discover a more significant problem:

{isAuthenticated && (
  <button>Place order</button>
)}

If authentication is missing, changing the locator would be the wrong repair.

The source agent therefore helps distinguish test breakage from application breakage.

The History Agent

The history agent investigates when and why the relevant behavior changed.

For example:

Commit: 4f92c1a
Message: Improve accessibility labels in checkout

Changed:
aria-label="Place order"
→
aria-label="Confirm purchase"

The agent can correlate this with the failing test.

This creates a strong causal chain:

Failure
  ↓
Broken locator
  ↓
Current component
  ↓
Recent source change
  ↓
Changed accessible name
  ↓
Updated locator

That chain is much stronger than semantic similarity alone.

The API Agent

Not every Playwright failure is caused by the UI.

Consider:

await page.getByRole('button', {
  name: 'Place order'
}).click();

await expect(
  page.getByText('Order confirmed')
).toBeVisible();

The locator may be completely correct while the backend returns:

500 Internal Server Error

An API-focused agent can inspect network logs, API definitions, mocks, fixtures, and recent backend changes.

For example:

page.on('response', async response => {
  if (response.url().includes('/api/orders')) {
    console.log(
      response.status(),
      response.url()
    );
  }
});

The agent can then distinguish:

UI locator failure

from:

Order API failure

That distinction prevents dangerous test edits.

The Validation Agent

No repair should be considered complete until it has been executed.

The validation agent can run a targeted Playwright command:

npx playwright test tests/checkout.spec.ts \
  --project=chromium

If that succeeds, it can expand validation:

npx playwright test tests/checkout.spec.ts

and eventually:

npx playwright test

A staged validation strategy is useful because a one-test repair should not immediately require the entire suite.

A Simple Agent Coordinator

A coordinator can combine the findings.

For example:

type Investigation = {
  failure: unknown;
  locator?: unknown;
  source?: unknown;
  history?: unknown;
  api?: unknown;
};

async function repairTest(
  failure: unknown
) {
  const investigation: Investigation = {
    failure
  };

  investigation.locator =
    await locatorAgent(failure);

  investigation.source =
    await sourceAgent(investigation.locator);

  investigation.history =
    await historyAgent(investigation.source);

  if (looksLikeApiFailure(failure)) {
    investigation.api =
      await apiAgent(failure);
  }

  return coordinatorAgent(investigation);
}

The coordinator should produce a repair proposal rather than blindly editing files.

For example:

{
  "action": "edit",
  "file": "tests/checkout.spec.ts",
  "old": "getByRole('button', { name: 'Place order' })",
  "new": "getByRole('button', { name: 'Confirm purchase' })",
  "evidence": [
    "Current DOM exposes Confirm purchase",
    "CheckoutPage changed accessible label",
    "Failure targets old accessible name"
  ]
}

Applying the Repair Safely

A repair agent can then modify the test.

For example, before:

await page.getByRole(
  'button',
  { name: 'Place order' }
).click();

After:

await page.getByRole(
  'button',
  { name: 'Confirm purchase' }
).click();

The important part is that the agent should make the smallest possible change.

A good automated repair system follows a principle such as:

minimum necessary modification

Rather than:

rewrite the entire test

This reduces unintended changes and makes code review easier.

Using Graph Traversal for Better Retrieval

Graph retrieval becomes particularly powerful when a failure requires multiple hops.

Suppose the failing test is:

checkout.spec.ts

The system can traverse:

Test
 ↓
Locator
 ↓
Component
 ↓
File
 ↓
Commit
 ↓
Related Component
 ↓
API Endpoint

The retrieved context might become:

{
  "test": "checkout.spec.ts",
  "locator": "Place order",
  "component": "CheckoutPage",
  "sourceFile": "CheckoutPage.tsx",
  "recentCommit": "4f92c1a",
  "changedProperty": "aria-label",
  "currentValue": "Confirm purchase",
  "api": "POST /api/orders"
}

This context can then be given to the reasoning model.

The LLM is no longer operating on a random collection of retrieved files. It receives a structured explanation of how those files relate.

Combining Vector Search and Graph Search

Graph RAG does not mean abandoning vector search.

In practice, the strongest architecture often combines several retrieval mechanisms.

For example:

Failure
   │
   ├── Vector retrieval
   │      └── similar historical failures
   │
   ├── Graph traversal
   │      └── related components and tests
   │
   ├── Keyword retrieval
   │      └── exact selector/API names
   │
   └── Git retrieval
          └── relevant historical changes

The results can be merged before being passed to the reasoning agent.

This provides both semantic and structural retrieval.

Learning From Previous Repairs

One of the biggest advantages of this architecture is that successful repairs can become new knowledge.

Suppose an agent repairs:

getByText("Save")

to:

getByRole("button", { name: "Save changes" })

and the repair passes.

The system can store:

{
  "failureType": "LOCATOR_CHANGED",
  "oldLocator": "getByText('Save')",
  "newLocator": "getByRole('button', { name: 'Save changes' })",
  "component": "ProfileForm",
  "validated": true
}

The next time a similar failure occurs, Graph RAG can retrieve this historical repair.

Over time, the repair system develops a repository-specific knowledge base.

Avoiding the Most Common AI Repair Mistakes

Automated test repair can create new problems if the system optimizes only for passing tests.

For example, an agent might replace:

page.getByRole('button', {
  name: 'Place order'
});

with:

page.locator('button').first();

The test may pass today, but the locator is significantly less expressive and more fragile.

The repair system should therefore enforce rules such as:

Prefer role-based locators.
Prefer accessible names.
Avoid positional selectors.
Avoid arbitrary CSS selectors.
Avoid XPath unless necessary.
Do not remove assertions merely to make a test pass.
Do not increase timeouts without evidence.
Do not modify application code unless explicitly authorized.

These constraints turn test repair into controlled engineering rather than unrestricted code generation.

A Practical End-to-End Workflow

A production pipeline can look like this:

1. Playwright test fails
        ↓
2. Failure parser extracts structured error
        ↓
3. Graph retrieves connected repository context
        ↓
4. Specialized agents investigate
        ↓
5. Coordinator builds repair hypothesis
        ↓
6. Repair agent proposes minimal patch
        ↓
7. Patch is applied in an isolated workspace
        ↓
8. Playwright test is rerun
        ↓
9. Related tests are executed
        ↓
10. Repair is accepted or rejected
        ↓
11. Successful repair enters knowledge graph

This separation is important because every stage can be inspected.

Example Repair Report

A useful system could produce a report like:

Test:
tests/checkout.spec.ts

Failure:
Timeout waiting for:
getByRole('button', { name: 'Place order' })

Diagnosis:
LOCATOR_CHANGED

Evidence:
1. CheckoutPage currently exposes accessible name "Confirm purchase".
2. Commit 4f92c1a changed the aria-label.
3. The test was not updated by that commit.
4. The button remains associated with the same checkout action.

Proposed repair:
Replace "Place order" with "Confirm purchase".

Validation:
Targeted test: PASS
Checkout suite: PASS

This makes the AI’s decision auditable.

Measuring Repair Quality

A mature system should measure more than the number of repaired tests.

Useful metrics include:

Repair success rate
Validation success rate
False repair rate
Average repair time
Human acceptance rate
Regression rate
Number of unnecessary edits
Locator robustness

One particularly important metric is validated repair rate.

A generated patch should not count as successful simply because an LLM produced syntactically valid code.

It should count only when the repaired test passes and the change survives relevant regression tests.

Integrating the System Into CI

The architecture can run automatically when CI detects a failure.

For example:

steps:
  - name: Run Playwright
    run: npx playwright test

  - name: Analyze failures
    if: failure()
    run: npm run ai:analyze

  - name: Generate repair
    if: failure()
    run: npm run ai:repair

  - name: Validate repair
    if: failure()
    run: npm run ai:validate

The repair system could create a patch rather than committing directly.

That allows a developer to inspect:

git diff

before merging the repair.

For sensitive repositories, this human-in-the-loop model is preferable to allowing an autonomous agent to modify production tests without review.

A Recommended Architecture

A practical implementation can be organized into these layers:

┌─────────────────────────────────────┐
│           Playwright Suite          │
└─────────────────┬───────────────────┘
                  │
                  ▼
┌─────────────────────────────────────┐
│        Failure Extraction           │
└─────────────────┬───────────────────┘
                  │
                  ▼
┌─────────────────────────────────────┐
│        Graph + Vector Retrieval     │
└─────────────────┬───────────────────┘
                  │
        ┌─────────┼──────────┐
        ▼         ▼          ▼
     Locator    Source     History
      Agent      Agent      Agent
        │         │          │
        └─────────┼──────────┘
                  ▼
          Repair Coordinator
                  │
                  ▼
            Patch Generator
                  │
                  ▼
          Validation Agent
                  │
          ┌───────┴────────┐
          ▼                ▼
        PASS              FAIL
          │                │
          ▼                ▼
     Store repair       Escalate

The architecture can begin small. You do not need a sophisticated graph platform on day one.

A useful first version can consist of:

  1. Playwright failure extraction.
  2. Repository indexing.
  3. A graph containing tests, components, routes, and locators.
  4. Three specialized agents.
  5. A patch generator.
  6. Automated Playwright validation.

Additional relationships can be added as the system proves useful.

Why Specialized Agents Matter

The main advantage of specialized agents is not simply that there are multiple LLM calls.

It is that each agent operates under a different responsibility.

A locator agent asks:

“What selector currently represents this UI element?”

A source agent asks:

“What does the application currently render?”

A history agent asks:

“What changed and when?”

An API agent asks:

“Is the application behavior failing independently of the test?”

A validation agent asks:

“Does the proposed repair actually work?”

These are different questions. Separating them makes the overall reasoning process more reliable and easier to debug.

Security and Governance Considerations

An autonomous repair system should have strict boundaries.

Repository access should be read-only during investigation whenever possible.

Code modification should happen in an isolated workspace.

The system should also avoid exposing secrets to language models.

For example, Playwright traces may contain:

authorization headers
cookies
tokens
customer information

These should be filtered before entering the retrieval pipeline.

Similarly, generated patches should be reviewed before merging when the repair system has write access to a production repository.

The Future of Automated Playwright Repair

The broader idea extends beyond Playwright.

The same architecture can repair:

  • API integration tests,
  • Cypress tests,
  • Selenium suites,
  • unit tests,
  • contract tests,
  • mobile UI tests,
  • visual regression tests.

The common pattern is:

software system
      +
relationship graph
      +
specialized reasoning agents
      +
automated validation

Graph RAG supplies context, specialized agents supply focused reasoning, and automated execution supplies objective feedback.

That combination is much more powerful than asking an LLM to “fix this failing test.”

Conclusion

Repairing Playwright tests with AI becomes substantially more reliable when the problem is treated as an evidence-driven software-engineering workflow rather than a simple code-generation task. A failing test rarely exists in isolation. Its selectors are connected to components, components are connected to routes and APIs, tests depend on fixtures, and all of these artifacts evolve through commits and refactoring. Graph RAG provides a mechanism for representing and retrieving those relationships so that an AI system can understand the broader context surrounding a failure.

The graph should capture the most useful entities and relationships in the repository: tests, test steps, locators, pages, components, routes, API endpoints, fixtures, source files, failures, and commits. Once those relationships are available, a failing locator can be traced back to the component that renders it, the source file that defines that component, and the commit that changed its behavior. This gives the repair process evidence that ordinary semantic search may not provide.

Specialized agents add another important layer. Instead of assigning one general-purpose agent to diagnose, modify, and validate everything, separate agents can investigate different aspects of the problem. A failure agent can classify the error, a locator agent can investigate selector changes, a source agent can inspect current application behavior, a history agent can identify relevant changes, and an API agent can determine whether the failure originates outside the browser UI. A validation agent can then execute the proposed repair and determine whether it actually solves the problem.

The most important principle is that the AI should not optimize merely for making the test pass. A repair that changes a semantic role-based locator into locator('button').first() might make a test pass temporarily while making the suite considerably more fragile. Likewise, deleting an assertion, increasing a timeout, or replacing a meaningful locator with an arbitrary CSS selector can hide the underlying problem instead of repairing it.

A robust repair system should therefore prefer minimal, evidence-backed modifications. It should preserve meaningful assertions, favor accessible Playwright locators, avoid positional selectors, and require execution-based validation before considering a repair successful. The resulting patch should also be explainable: developers should be able to see what failed, what evidence was retrieved, why a particular repair was proposed, and which tests validated the change.

The combination of Graph RAG and specialized agents also creates a valuable feedback loop. Every validated repair can become another piece of repository knowledge. Historical failures, successful locator replacements, recurring component changes, and previous debugging decisions can be stored in the graph. Future failures can then retrieve those experiences and use them as additional evidence. In this way, the system gradually becomes more familiar with the specific architecture and conventions of the codebase.

The strongest implementation is therefore not an autonomous agent that blindly edits Playwright files.

It is a controlled pipeline:

Failure
   ↓
Structured diagnosis
   ↓
Graph and semantic retrieval
   ↓
Specialized investigation
   ↓
Evidence-based repair hypothesis
   ↓
Minimal patch
   ↓
Automated validation
   ↓
Human review when appropriate
   ↓
Validated repair becomes new knowledge

This architecture balances automation with engineering discipline. Graph RAG provides the context that a language model needs, specialized agents divide complex reasoning into manageable responsibilities, and Playwright itself provides the feedback required to determine whether the proposed change actually works.

Ultimately, the goal is not simply to have AI write replacement selectors faster. The larger opportunity is to build a repository-aware test maintenance system that understands how tests, application code, APIs, fixtures, and historical changes interact. With careful graph modeling, focused agents, conservative patch generation, and rigorous validation, many repetitive Playwright maintenance tasks can move from manual debugging toward a repeatable and auditable repair workflow.