In the age of AI, traditional methods of software procurement are no longer fit for purpose. Often viewed as a commercial exercise—securing the best software at the best price—buying teams are now making decisions about highly complex technology systems that can shape an organization’s management posture, regulatory exposure, security and operational resilience.
Businesses are interacting with AI in many ways, from standalone software to features embedded in existing platforms. No matter how AI enters the business, its presence immediately raises management questions. Who owns the data? How is this program trained? Who is responsible for mistakes? Most procurement teams lack the technical expertise to analyze these complexities and evaluate systems before making purchasing decisions, creating governance risks from the outset.
Exacerbating these complexities is the balance of power between procurement teams and AI vendors. A small number of prominent AI providers can set implementation goals and implement changes down the line without consulting customers. Managers who encourage organizational adoption of AI only intensify this pressure, leaving procurement officers to navigate outdated processes without the benefit of AI-specific training or guidance.
Together, these factors created a new frontier for governance. Procurement has evolved from a commercial activity to a more important—and less mature—aspects of enterprise AI governance.
Emerging risks of AI procurement
Unlike traditional software purchases, AI purchases present management risks that continue to emerge after the contract is signed. AI models supply chains, infrastructure providers, APIs and application layers, making it difficult to identify who is responsible for what when things go wrong. Even if liability is determined at the time of purchase, renewals and exclusions may complicate liability structures and introduce new risks after the contract is signed.
The pace of technological and regulatory change is shortening procurement and contract review cycles. The advantage is that short cycles offer flexibility; the downside is that they present ongoing demands for procurement and governance that organizations must manage on an ongoing basis.
These challenges are exacerbated by market focus. OpenAI, Anthropic and Google jointly account 88 percent of business LLM useleaving consumers exposed to downward changes in prices, product features and contract terms. On top of this, the nature of AI is driving “lock-in” to individual providers because models evolve by interacting with user data and workflows. Switching suppliers can therefore be operationally disruptive, expensive and technically difficult.
Data management is one of the best limited risk areas in AI procurement. Key questions regarding data collection, maintenance and model training are often left unanswered during the contract. As a result, organizations may improperly disclose confidential information, proprietary business data or customer records to external model training processes.
AI presents IP risks, too; models trained on data dumped on the web may produce output that includes copyrighted or unauthorized content, exposing organizations to legal and reputational risks. Other IP considerations, such as ownership of AI results and metadata, must be addressed at the time of purchase.
Agent AI is the next frontier of consumer risk. Capable of independently traversing multiple platforms and data sets, these systems present risks of a very different order of magnitude. They are increasing, sellers without significant AI-related damage from liability clauses within AI contracts, leaving consumers exposed. As AI agents grow in complexity and popularity, all of these concerns—liability, accountability, dependency and data management—will need to be continually addressed and negotiated.
The four pillars of responsible AI procurement
Because purchasing AI presents unique risks not associated with conventional software, organizations need procurement frameworks designed for AI Based on our experience consulting companies on responsible AI procurement, we recommend building such frameworks on four pillars.
First, teams must develop skills. Procurement teams must also learn about AI functionality, security and compliance. This does not mean that every procurement officer must be a technical expert, but they need to have sufficient understanding to assess governance implications.
Organizations must support this through certification programs, diverse procurement models and close collaboration between procurement, legal, compliance, cybersecurity and technology teams. Extensive staff training in responsible AI use is also important, especially as workers increasingly adopt AI tools independently.
Emerging vendor engagement models, where procurement staff interact with AI vendors forward deployment engineers to make tools suitable for operational needs, ensure a deep understanding of technical elements. While this approach can improve results, organizations also need to invest time in internal change management, governance review and process development.
Second, optimize purchasing processes with the type of AI system that is available. Procurement teams deal with a range of AI products, including AI-powered tools, AI-enabled features and basic models. Each stage presents novel governance, compliance and operational risks that require complementary approaches to procurement.
For AI-enabled products, procurement teams must focus on use case fit, data sovereignty and hosting plans. Consider a bad example: when Workday releases Applicant Tracking System powered by AIit was billed as a ready-to-use tool for HR teams. However, the brand violated the Age Discrimination in the Workplace Act in favor of applicants under 40 years of age. Organizations that adopted the tool without adequate procurement testing faced compliance exposure under employment law.
When vendors introduce AI features into existing software products, procurement teams must renegotiate contracts that do not have AI-specific clauses. This is an increasingly common situation that presents complex governance and compliance risks. For example, when GitHub has updated its training requirements for early 2026organizations with low-level subscriptions discovered that their private data was being used to train AI models. Situations like these undermine an organization’s privacy, data protection and security controls.
When purchasing a basic model or platform, organizations should focus on technical capabilities and strategic implications. For example, the UK’s NHS has recently faced criticism over its procurement process allowed the US company Palantir to access identifiable patient data while creating a unified data platform. The platform aims to deploy AI across patient records to improve efficiency. However, the approach to procurement has eroded public trust in the NHS and its services.
Third, the procurement of AI should be focused on established governance structures, standards and emerging regulatory requirements. Even in areas where AI regulation remains underdeveloped, compliance with a comprehensive regulatory framework allows organizations to demonstrate that their use of AI is safe, responsible and reliable. It also creates defensible evidence that management responsibilities were considered prior to deployment.
The standards are the same ISO 42001 of AI management systems and ISO 23894 of AI risk management can help organizations establish documented management processes and create auditable evidence trails from the start. Other methods, such as IEEE standards and compatibility testing again IAPP AI Governance Vendor Reportscan further support procurement due diligence.
Finally, procurement should be included in ongoing AI governance processes. When an organization buys an AI system, it makes governance choices about data sovereignty, liability, regulatory compliance and long-term vendor dependence. These options require ongoing review with governance stakeholders as models change, regulations change and vendors revise products or service policies.
Therefore, organizations should integrate procurement directly into broader AI governance structures through recurring audits, compliance reviews, performance evaluations and various oversight processes. Where possible, organizations should pursue short contracts and procurement cycles to avoid making long-term commitments in the face of rapidly changing technology.
The dominance of AI starts in shopping
For many organizations, procurement remains one of the most mature dimensions of AI governance. But purchasing decisions increasingly determine how data is managed, where accountability resides, where vendors gain influence over performance and how resilient organizations remain as AI systems evolve.
As AI adoption accelerates, procurement will no longer function as a sales function. Every AI contract now embeds decisions about governance, risk, compliance, security and strategic dependencies. It has become a management practice, which will play a defining role in whether business AI programs are deployed responsibly, compliantly and successfully.
Amelia Williams is a Senior Research Impact Officer at Trilateral Research with expertise in science communication at the intersection of emerging technologies, environmental issues, ethics, and policy. At Trilateral, he supports the development and implementation of policy-related research projects, media and industry collaboration.
