Enterprises have spent years cataloguing tables, measuring data quality and setting permissions in structured systems. Then generative AI arrived and began searching through documents, presentations, policies, contracts, emails and shared folders. The surface area changed. A US data leadership discussion dedicated to unstructured data governance highlighted why familiar controls can be inadequate when knowledge becomes the raw material for AI responses.
For vendors, this is more than a retrieval problem. The buyer needs to know whether the information is current, who owns it, whether the user is permitted to see it and how the model should behave when two documents disagree. A brilliant semantic search experience can still become an expensive governance failure.
The document repository is not a trusted knowledge base
An organisation may have thousands of files that look official but serve different purposes. A draft policy may sit beside the approved version. A sales deck may contain obsolete product claims. A contract may have been superseded by an amendment. A team may copy a sensitive spreadsheet into an open collaboration space without understanding the consequences.
Traditional access permissions help, but they do not establish reliability. When an AI assistant retrieves a document, the user may treat the answer as endorsed organisational knowledge. The US roundtable material discussed unstructured-data management, discoverability, ownership and the need to bring governance into the places where people actually create and share content.
A vendor should therefore distinguish retrievable from authoritative. What marks the approved version? How does the platform recognise an expired policy? Can it identify conflicting documents? Is the answer linked to a verifiable passage and an accountable owner? If not, the customer may simply automate the distribution of stale or unsupported information.
Access control becomes more complicated when AI joins the workflow
AI retrieval can assemble fragments from multiple sources in ways that human users would not ordinarily connect. Even when each individual system enforces access, a cross-system assistant can create a new route for information exposure if identities, permissions or indexing rules are inconsistent.
US enterprise discussions about operational governance and AI-ready data repeatedly raised concerns about who may use which information, how controls can be maintained and how business functions adopt AI tools. These concerns matter acutely for unstructured content, because documents often contain personal, legal, commercial or employee information not represented in a neat row-level schema.
Vendors should show permission-aware indexing, source-level entitlements, prompt and response controls, auditability and deletion handling. The buyer will ask whether withdrawing a user’s access immediately affects the assistant, how cached content is treated and whether sensitive passages can be excluded from retrieval or output. ‘We don’t train on your data’ does not answer those questions.
Metadata is an operating discipline, not just a tagging exercise
A document title, upload date and owner are a starting point. Effective governance may also require classification, retention rules, jurisdiction, approved-use status, version, business domain and relationships to other records. Those attributes help determine whether a retrieved answer should be trusted and who can correct it.
Yet requiring employees to populate dozens of fields manually creates its own failure mode. The enterprise needs an approach that fits existing document workflows, automates appropriate classification and makes exceptions visible. AI can assist with metadata extraction, but the organisation still needs agreed policies and human accountability for high-consequence decisions.
This is where vendors can make a defensible contribution. Show how content becomes discoverable and controlled, how the platform handles ambiguity and how it supports an information owner correcting errors at the source. A vendor that measures success solely by the number of indexed pages is tracking coverage, not trust.
Retrieval evaluation needs realistic failure cases
A search demo usually asks questions where the answer is known to be present. A production evaluation should include the hard cases: two conflicting policies, a revoked permission, an outdated contract, a missing source, an ambiguous entity name and a question that combines restricted and unrestricted material.
The buyer should understand whether the assistant cites sources accurately, refuses unsupported answers and routes uncertain issues to a person. They should also see what happens when a document is replaced, moved, reclassified or deleted. Reliability requires a content lifecycle, not one successful retrieval test.
Unstructured data governance also connects to the larger economics of AI. Poorly curated content increases retrieval noise, review effort and downstream rework. The operational cost of a wrong answer may be much larger than the token cost of generating it. Suppliers should be prepared to model both.
A document can be accessible and still be the wrong evidence
The unstructured-data challenge begins long before an AI system answers a question. Enterprise repositories contain drafts, expired policies, customer correspondence, scanned agreements, conflicting versions and documents whose owners have left. A search interface may retrieve the most relevant-sounding passage without knowing whether it remains authoritative. A retrieval-augmented generation application can therefore look more trustworthy than the underlying knowledge estate warrants.
The US discussions on unstructured data management covered the practical need to classify content, assign responsibility and integrate information from disconnected repositories. These concerns connected with broader conversations about data ownership, governance and emerging AI. The commercial lesson is not simply that every document needs more metadata. It is that buyers need a repeatable way to determine which information should be used for which task, by whom and under what conditions.
Consider a hypothetical insurance workflow that searches underwriting guidance. A retrieved document may be authentic but superseded; another may be current yet apply to a different customer segment. A good solution needs to preserve version, effective date, jurisdiction, ownership and approval status where these are relevant. Retrieval relevance alone does not establish authority. Vendors should show how the system treats two plausible sources that disagree and whether it alerts the user to that disagreement.
The best demonstrations include an answer that the system should refuse to give confidently. Supply an outdated policy alongside the current version, then ask a question for which the older document would be tempting. Does the solution privilege the approved source, cite its provenance and make uncertainty visible? A product that gracefully flags missing evidence may be safer and ultimately more useful than one optimised to answer every prompt.
Permission-aware retrieval is a living control
A knowledge assistant inherits the security problems of the systems it can access. Group permissions are frequently inherited, exceptions accumulate and users move between teams. A search result can expose a sensitive title or summary even where the full document remains protected. Once information is pulled into an index, cached or summarised, access rules may no longer follow the original repository automatically. These issues are not hypothetical claims about any single vendor; they are architectural failure modes buyers should test.
An enterprise evaluation should start with identities rather than model quality. Create users with different privileges, including one whose access has just been revoked. Confirm that each can retrieve only permitted sources and that restricted material is absent from citations, snippets and intermediate outputs. Then change a source permission and test how quickly that change reaches the retrieval system. A platform that rechecks permissions at query time may have a different risk profile from one relying on infrequent indexing cycles.
The process should include non-human identities. If an agent retrieves a document to support another workflow, whose rights does it inherit? Can it join information across repositories in a way no individual user was authorised to do? Can administrators reconstruct the retrieval chain after a complaint or incident? US governance discussions stressed the need to connect AI access with business policy, not simply grant a broad licence to an enterprise repository.
For vendors, this creates a differentiated proposition around access integrity, not just search accuracy. The product should demonstrate source-level controls, permission propagation, auditing, retention and defensible deletion. Where capabilities depend on third-party connectors, suppliers need to document those dependencies rather than implying they control every downstream permission event.
What a trustworthy enterprise knowledge pilot should test
A sensible pilot works with a deliberately messy but bounded collection rather than a pristine set of hand-picked PDFs. Include valid and obsolete versions, varying formats, multiple owners, unclear metadata and a few access restrictions. Choose questions from people who actually rely on the content. Score not only whether a response sounds helpful, but whether its supporting sources are correct, current and accessible to the questioner.
The evaluation should separate three failures. A retrieval failure happens when the right document was eligible but not found. A grounding failure happens when the document was found but the generated answer misrepresents it. A governance failure happens when an answer uses a source that should not have been available or whose authority was not established. Each failure requires a different remedy, and a vendor should not bundle them under a vague ‘accuracy’ percentage.
Operationally, buyers should ask who maintains the knowledge base after launch. Documents change ownership; policies are replaced; records must sometimes be retained while others must be deleted. A technology that requires an unsustainable manual tagging exercise will struggle even if the proof of concept succeeds. The product’s ingestion workflows, exception queues and stale-content detection deserve as much scrutiny as the chatbot.
The commercial result worth measuring is not documents indexed or questions answered. It is fewer unnecessary escalations, faster access to approved information and a demonstrable reduction in unsupported or unauthorised answers within a defined process. That connects the roundtable’s governance concerns with a buyer outcome the vendor can defend without promising that unstructured enterprise knowledge will ever be perfectly clean.
Why a ‘single source of truth’ may be the wrong promise
Enterprise knowledge is rarely a single coherent document library. Legal, operations, product and customer teams maintain different authoritative records for different decisions. The correct answer may depend on the date, jurisdiction, product or user’s role. Trying to collapse all of that material into one undifferentiated index can erase the distinctions that make it useful. A vendor should therefore explain how its system recognises competing authorities and which rules determine precedence.
A more defensible promise is a trusted route to the right source for the question being asked. That may require document-level permissions, version history, explicit ownership and the ability to abstain when evidence conflicts. The US governance conversations highlighted the organisational difficulty of keeping definitions and access consistent. A knowledge-AI product should help the buyer manage that difficulty rather than claim it disappears once documents become searchable. In a buyer meeting, an honest example of the assistant refusing an unsupported answer may be the strongest proof of control.
A further test concerns deletion. A source document may be removed for a legitimate legal or operational reason, but its content could remain in search indexes, caches or generated summaries. Ask the supplier to demonstrate how a deletion request propagates through the full retrieval path and how completion is verified. This is rarely as visually exciting as a perfect chatbot answer, but it speaks directly to the lifecycle governance that enterprise data teams must operate after the initial launch.
The vendor checklist for enterprise knowledge AI
- Can the system distinguish approved content from drafts and superseded versions?
- Are source permissions enforced at retrieval and response time?
- How quickly do permission and document-lifecycle changes propagate?
- Can users inspect the source passage and its owner?
- How does the system respond to conflicting, missing or restricted documents?
- Are classifications, retention policies and audit logs integrated into existing workflows?
More access without more control is not progress
The US discussions show why governance of unstructured knowledge deserves a place beside structured data quality in enterprise AI procurement. The winning product will not merely find more documents. It will help the organisation control which knowledge is trusted, who can use it and how errors are corrected. That is a more credible promise than ‘chat with all your company data’.
For vendors: The Data Leadership Board facilitates conversations with senior enterprise data leaders responsible for trusted AI and knowledge management. If your platform solves the control problem behind enterprise search and retrieval, connect with us to explore relevant buyer access.