Corpus Design Principles for AI Systems

Summary

Many AI discussions focus on model selection, prompting techniques, vector databases, and agent frameworks.

While those topics are important, retrieval-based systems often succeed or fail based on a different factor:

The quality of the underlying corpus.

For systems that rely on Retrieval-Augmented Generation (RAG), the corpus becomes the foundation upon which retrieval, reasoning, recommendations, and responses are built.

A sophisticated model operating against poor information will frequently produce poor outcomes.

A well-designed corpus can often improve system quality more than upgrading to a larger or more capable model.

What Is a Corpus?

A corpus is the collection of information available to an AI system.

Examples include:

• documentation
• runbooks
• architecture articles
• source code
• issue tracking systems
• knowledge bases
• operating procedures
• incident reports
• design documents
• meeting notes

In retrieval-based systems, the corpus serves as the authoritative knowledge source.

The model does not become the source of truth.

The corpus remains the source of truth.

The Corpus Is Often the Product

Many organizations focus heavily on selecting models while spending comparatively little time evaluating the quality of their knowledge sources.

This can create a mismatch between expectations and outcomes.

Organizations frequently assume that stronger models will compensate for weak information.

In practice, the opposite is often true.

A highly capable model operating against poor information may produce answers that sound convincing while remaining incomplete, outdated, or incorrect.

As models become increasingly capable, corpus quality often becomes one of the primary factors limiting system performance.

Models change.

Vendors change.

Embedding techniques change.

The corpus often remains.

For many organizations, the corpus ultimately becomes one of the most valuable assets in the entire AI architecture.

Characteristics of a High-Quality Corpus

High-quality corpora tend to share several characteristics.

Accuracy

Information should be factually correct and reflect operational reality.

Incorrect information leads to incorrect outputs regardless of model quality.

Completeness

Knowledge should include sufficient detail to explain not only what was done, but also why it was done.

Missing context often creates incomplete answers.

Consistency

Conflicting information creates uncertainty for both humans and AI systems.

Consistency improves trust and predictability.

Ownership

Content should have clear ownership.

Someone should be responsible for maintaining accuracy and relevance.

Freshness

Knowledge should be reviewed and updated regularly.

Outdated information can be as harmful as incorrect information.

Characteristics of a Poor Corpus

Many organizations unknowingly accumulate knowledge that is difficult for both humans and AI systems to use effectively.

Common issues include:

• outdated documentation
• abandoned wiki pages
• conflicting procedures
• undocumented decisions
• missing context
• tribal knowledge
• inconsistent terminology
• duplicate content

These problems frequently appear long before AI systems are introduced.

AI systems simply make the weaknesses more visible.

The State of the Source Material Matters

Many organizations approach AI adoption by focusing on model selection.

Questions often include:

• Which model should we use?
• Which coding assistant is best?
• Which vector database should we deploy?

These are reasonable questions.

However, an equally important question is often overlooked:

What is the state of the information we are providing to the system?

AI systems frequently rely on information from:

• Confluence
• Jira
• Google Drive
• SharePoint
• source code repositories
• internal documentation
• runbooks
• architecture diagrams

If those sources are incomplete, outdated, contradictory, poorly organized, or inaccurate, the resulting AI system will inherit many of the same problems.

Retrieval systems can only retrieve information that exists.

Language models can only reason about the information they receive.

Organizations sometimes assume that sufficiently advanced models will compensate for weak source material.

In practice, the opposite is often true.

AI can make poor information easier to consume, but it cannot reliably transform poor information into trustworthy knowledge.

Content Designed for Humans vs Content Designed for Retrieval

Not all documentation is equally useful for retrieval systems.

Consider a Jira ticket containing:

• Fixed.
• Resolved.
• Updated configuration.
• See screenshot.

The author may understand exactly what happened.

Future engineers and retrieval systems often do not.

Now consider:

Issue: Karpenter failed to provision replacement nodes.

Root Cause: NodePool constraints excluded all eligible instance types.

Resolution: Expanded instance family requirements and redeployed the NodePool.

Verification: Confirmed successful node provisioning and pod scheduling.

Both examples describe a completed task.

Only one preserves the reasoning and operational context behind the change.

Retrieval systems benefit from content that explains:

• what happened
• why it happened
• how it was resolved
• how success was validated

The same information also improves future troubleshooting, onboarding, audits, and knowledge transfer.

The Screenshot Problem

A common challenge in enterprise documentation is the overuse of screenshots.

Examples include:

• command output screenshots
• configuration screenshots
• architecture screenshots
• embedded diagrams
• screenshots containing procedures

While screenshots may be useful to human readers, they often reduce searchability and retrieval quality.

Important information can become trapped inside images rather than existing as searchable text.

Where practical, screenshots should supplement documentation rather than replace written explanations.

The Tribal Knowledge Problem

Some of the most valuable organizational knowledge never reaches documentation.

It exists in conversations, experience, habits, and institutional memory.

This creates risk.

Knowledge that exists only in people's heads cannot be searched, reviewed, governed, or retrieved.

When individuals leave teams or organizations, that knowledge frequently leaves with them.

Some organizations are also exploring structured approaches for preserving institutional knowledge from retiring employees and long-tenured subject matter experts before that knowledge is lost.

AI systems cannot retrieve knowledge that was never documented.

Documentation vs Knowledge

Documentation and knowledge are not always the same thing.

Many organizations have large volumes of documentation.

The more important question is whether people trust it.

Documentation that is inaccurate, outdated, or incomplete may technically exist while providing little operational value.

Trustworthy knowledge requires:

• maintenance
• validation
• ownership
• review

The goal is not merely to create documentation.

The goal is to create knowledge that remains useful over time.

Governance and Ownership

Effective corpora require governance.

Important questions include:

• Who owns this content?
• Who reviews it?
• How frequently is it updated?
• When should it be archived?
• How is accuracy verified?

Without governance, corpus quality tends to degrade over time.

Retrieval quality eventually follows.

Training Matters

Building a high-quality corpus is not solely a technical challenge.

It is also a people and process challenge.

Engineers, quality assurance teams, operations staff, architects, managers, and subject matter experts all contribute knowledge to the organization.

Many organizations invest heavily in AI platforms while investing comparatively little in teaching people how to create knowledge that remains useful over time.

Training should include topics such as:

• writing clear documentation
• explaining decisions and rationale
• documenting troubleshooting steps
• using consistent terminology
• creating meaningful Jira updates
• designing searchable knowledge base articles
• structuring spreadsheets and tabular data
• documenting diagrams and screenshots with supporting text

The objective is not to teach people how to write for AI.

The objective is to teach people how to create knowledge that is understandable, searchable, maintainable, and reusable by both humans and AI systems.

Why This Project Uses Markdown

This project uses Markdown as the canonical content format.

The decision was driven by several factors:

• portability
• version control
• readability
• automation
• simplicity
• long-term maintainability

Markdown works well for both humans and machines.

It can be stored in Git repositories, reviewed through pull requests, tracked through version history, and processed through automated workflows.

The objective is not to select the most sophisticated format.

The objective is to select a format that remains accessible and maintainable over time.

Content Structure Matters

Corpus quality is influenced not only by what information exists, but also by how that information is structured.

Poorly organized content can be difficult for both humans and retrieval systems to interpret.

Examples include:

• screenshots containing critical information with no supporting text
• architecture diagrams without explanations
• spreadsheets with unclear column names
• documents with inconsistent terminology
• tickets containing only status updates
• tables without context or definitions

Well-structured content improves retrieval quality, searchability, maintainability, and long-term usefulness.

The goal is not to eliminate diagrams, screenshots, spreadsheets, or visual content.

The goal is to ensure that important information also exists in forms that can be searched, reviewed, governed, and understood over time.

Corpus Maintenance Is Continuous

A corpus is never truly finished.

Content must be:

• reviewed
• updated
• corrected
• archived
• removed

Knowledge quality naturally degrades when maintenance stops.

Like software systems, knowledge systems require ongoing stewardship.

Modern AI systems can assist with this process.

During ingestion, governance, review, and maintenance workflows, AI can help identify duplicate content, improve structure, rewrite unclear documentation, suggest metadata, identify outdated information, and improve consistency across knowledge sources.

While AI cannot replace human ownership and validation, it can significantly reduce the effort required to maintain large corpora over time.

Should Organizations Fix the Corpus Before Building AI?

A common question is whether organizations should delay AI initiatives until their knowledge sources have been fully cleaned up and optimized.

In most cases, the answer is no.

Organizations should improve corpus quality and build AI capabilities in parallel.

Many enterprises have accumulated years or decades of documentation, tickets, source code, spreadsheets, presentations, diagrams, and operational knowledge.

Attempting to fully optimize all of that information before beginning an AI initiative may delay progress indefinitely.

A more practical approach is iterative improvement.

Build the platform.

Identify knowledge gaps.

Improve content quality.

Measure retrieval performance.

Continue improving the corpus over time.

Corpus improvement should be viewed as an ongoing operational responsibility rather than a one-time project.

The most successful organizations often treat knowledge stewardship as a continuous process that continues long after the initial AI platform has been deployed.

Key Design Principle

Retrieval systems are often blamed for poor answers when the underlying problem is poor knowledge quality.

A vector database can only retrieve the information that exists.

It cannot retrieve context, reasoning, or decisions that were never documented.

A retrieval system cannot consistently outperform the quality of the knowledge it retrieves.

Trustworthy AI systems begin with trustworthy knowledge.