Skip to content
PartnerinAI
AI Governance Tools

CTGT Concordance: Trace Inconsistent AI Agent Decisions

Explore CTGT Concordance, an open-source tool for testing inconsistent AI agent decisions, with a protocol, reference implementation, and data.

PartnerinAI3 min read529 words
CTGT Concordance: Trace Inconsistent AI Agent Decisions
Table of Contents

Quick Answer

CTGT Concordance is an open-source release that helps teams investigate whether an AI application produces different decisions across repeated runs. It includes a testing protocol, a reference implementation, and experimental data for examining and reproducing inconsistent results.

The launch addresses a practical challenge for organizations adopting AI agents: a system may return different answers or recommendations across repeated runs, even when the input has not changed. When those outputs inform compliance work or other consequential processes, teams need more than a final answer. They need a way to investigate how a discrepancy arose and assess whether the system is reliable enough for its intended use.

What the Open-Source Concordance Release Includes

Concordance is intended to support the investigation of inconsistent AI decisions. Its testing protocol provides a structured way to examine application behavior, while the reference implementation gives developers a working example. CTGT has also released experimental data alongside the software.

Together, these components make the project a reproducibility and auditability resource, rather than simply a tool for checking whether an answer looks right. Teams can use the materials to test whether a model-driven application responds consistently and to document differences for further review. Concordance does not, by itself, establish why every discrepancy occurred or certify an AI system as compliant; it is a means of helping teams examine the issue.

A Banking Manual Produced Four Different Sets of Obligations

CTGT says the work was prompted in part by a result from its compliance application. When the application processed the same banking manual four times, it generated four different sets of obligations.

That example makes the problem concrete. In compliance work, an AI-generated list of obligations can affect what a team investigates, records or treats as a requirement. If repeated processing of the same source material produces different lists, reviewers must be able to spot the differences and determine what needs human attention. A single plausible-looking response may not reveal that the system behaved differently on another run.

The broader concern applies wherever organizations use AI agents to approve, flag or record decisions. Inconsistent outputs can complicate review and make it harder to explain how a result was reached. Testing for repeatability can help teams identify such behavior before relying on an application in a consequential workflow.

Apache 2.0 Licensing Makes Concordance Available for Commercial Use

CTGT released Concordance under the Apache 2.0 license. The company says businesses can use and modify the software commercially. It also says it does not plan to offer a paid or hosted edition of Concordance.

For organizations evaluating the tool, the open license allows them to consider using or adapting the release within their own testing processes. Teams should still assess whether the tool and its materials fit their particular systems, risks and governance requirements; an open-source license does not replace internal validation or compliance review.

CTGT Concordance gives AI teams a practical starting point for testing whether repeated runs produce consistent decisions. Its release puts the protocol, implementation and experimental data in the open for organizations seeking to investigate discrepancies in their own AI applications.

Frequently Asked Questions

What is CTGT Concordance?
CTGT Concordance is an open-source release for investigating inconsistent decisions from AI applications. It includes a testing protocol, a reference implementation, and experimental data. Teams can use these materials to examine and reproduce differences across repeated runs.
How does Concordance help test AI agent consistency?
Concordance provides materials for testing how an AI application responds when it receives the same task repeatedly. Teams can compare the outputs and document discrepancies for review. The release supports investigation but does not establish the cause of every difference.
Why can inconsistent AI decisions matter in compliance work?
Inconsistent decisions can change which obligations or issues a team identifies and reviews. CTGT says its compliance application produced four different sets of obligations after processing the same banking manual four times. This illustrates why repeatability checks can be useful before relying on AI in consequential workflows.
Can businesses use CTGT Concordance commercially?
Yes, CTGT released Concordance under the Apache 2.0 license and says businesses can use and modify it commercially. The license does not replace internal validation or compliance review. Organizations should assess whether the tool fits their systems and governance requirements.

Key Takeaways

  • Concordance is designed to help organizations investigate inconsistent AI application decisions.
  • The release includes a testing protocol, reference implementation, and experimental data.
  • CTGT says its compliance application produced four different sets of obligations when processing the same banking manual four times.
  • Concordance is released under the Apache 2.0 license, which CTGT says permits commercial use and modification.
  • The tool can support testing and review, but it does not itself explain every discrepancy or certify compliance.