To evaluate AI in a project management tool, score it on nine criteria: hierarchy awareness, agentic versus assistive, data scope, governance and audit, explainability, human-in-the-loop controls, integration depth, time-to-value and total cost. Test each one in the live product, not the slides. Then run a 30-day pilot against two delivery outcomes you already measure.
Key takeaways
- Most demos show features. Score what the AI does with your data, at your scale, with a person accountable.
- Hierarchy awareness, the ability to reason from portfolio down to task, separates enterprise AI from team-level AI.
- Autonomy is a setting. Accountability does not transfer to an agent. Check audit trails and approval controls before you grant autonomy.
- Judge a pilot by whether a delivery outcome you already measure moves, not by whether the AI “works”.
Most AI demos in project management tools show features, not outcomes
Every project management tool you evaluate claims AI features. Some work. Others are a chatbot added to a task list, a summary that restates the screen, or a prediction that breaks when your data replaces the demo data.
Adoption is outrunning oversight. Digital.ai’s 18th State of Agile Report, published in October 2025, found that AI adoption among surveyed Agile practitioners rose from 68% to 84% in a year. Only 49% had guardrails in place. The survey covered nearly 350 participants, mostly Agile coaches and consultants at enterprises with more than 20,000 employees. Read it as a signal from large enterprises, not a market-wide count.
The risk is cost without value. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027. The reasons it gives are escalating costs, unclear business value and inadequate risk controls, according to w.media’s report on the release. Gartner also describes “agent washing”: vendors rebranding assistants, chatbots and robotic process automation as agentic. It estimates only about 130 of the thousands of agentic AI vendors are real, as RCR Wireless reports.
Expect every vendor to say “agentic”. Gartner predicts that up to 40% of enterprise applications will include task-specific agents by 2026, up from less than 5% in 2025. The nine criteria below help you check the claim.
This guide includes:
- A 9-criteria framework
- 12 vendor demo questions
- A 30-day pilot plan
- 7 red flags
- A scoring rubric and a downloadable scorecard
If you’re shopping for an enterprise PM platform with serious AI capabilities, you need a framework that , goes beyond the marketing page. This will give you a scoring rubric too that you can run every vendor through.
The 9 criteria on how to evaluate AI in PM tools
Every AI capability in a PM tool should be evaluated on these nine dimensions. We’ve ordered them roughly by how often vendors fail them — hierarchy awareness and explainability are the two most common gaps; pricing transparency is where the surprise bill usually shows up.
| # | Criterion | What good looks like |
|---|---|---|
|
1 |
Hierarchy awareness |
AI understands the work item in context — task is part of a project, project is part of a program, program rolls up to a portfolio. It can reason across levels |
|
2 |
Agentic vs assistive |
AI can act on your behalf (create, assign, schedule, escalate). |
|
3 |
Data scope |
Trained / grounded on your actual project data — not generic web data dressed up as project intelligence. |
|
4 |
Governance and audit |
Every AI action is logged with who triggered it, what changed, and why. Audit trail is exportable. |
|
5 |
Explainability |
AI can justify its recommendation: what data it used, what assumptions, what confidence level. |
|
6 |
Human-in-the-loop controls |
Reviewer queues, approval gates, configurable autonomy levels — humans can override at every step. |
|
7 |
Integration depth |
AI works across PM, PSA, reporting, and resource data — not isolated to one slice (just chat, just summaries). |
|
8 |
Time-to-value |
Pilot to production in weeks, not quarters. Pre-built use cases out of the box |
|
9 |
Total cost of AI |
Pricing is transparent — credits, seats, inference costs. No surprise bill at month three. |
1. Hierarchy awareness
Does the AI understand portfolio → program → project → task?
Hierarchy awareness is the AI’s ability to reason across portfolio, program, project and task, not only inside one task list. Many tools started as flat task lists, and AI added to a flat structure inherits it.
How to test it: Ask “What’s the impact on the Apollo program if the Backend API project slips two weeks?” An AI that only sees tasks will return a list of overdue cards. An AI with hierarchy awareness will trace dependencies up to the program, identify which downstream projects are affected, surface the resource conflicts that follow, and suggest which scope to defer.
This single capability separates enterprise-grade AI from team-level AI. If your PMO needs portfolio visibility, hierarchy awareness is non-negotiable.
2. Agentic vs assistive
Can it act, or only summarise?
An assistive AI describes the world. An agentic AI changes it.
How to test it: ask the AI to do something irreversible (with your permission) — create a project from a brief, rebalance a sprint, draft a status report and send it. If the vendor can only show you summaries, smart search, and “AI-suggested” content the user has to act on manually, you are looking at an assistant, not an agent.
Gartner predicts that 40% of enterprise apps will feature task-specific AI agents by 2026, up from less than 5% in 2025. The split between assistive and agentic AI is the next major dividing line in this category — and it’s the one most vendors are quietly behind on.
3. Data scope
Is the AI grounded on your project data, or generic web data?
Data scope is what the AI sees when it answers, and what it was trained on.
How to test it: Ask “What data was your model trained on?” and “What data does it see at inference time?” A good answer names your tenant’s project, resource, and historical delivery data.
A bad answer is some variant of “we use a leading large language model.” That tells you the AI knows about the world; it doesn’t tell you whether it knows about your business. For enterprise PMOs, the latter is what matters.
4. Governance and audit
Can you see what the AI did and why?
Governance and audit means every AI action leaves a record: who or what triggered it, what changed, when and with what input.
How to test it. Ask to see the audit trail for one AI action, and ask whether it can be exported.
Why it matters. Regulated industries such as financial services, healthcare and the public sector need this, and IT governance teams in every industry increasingly expect it. It is the practical answer to the Digital.ai finding that only 49% of surveyed organizations had guardrails in place.
5. Explainability
Can it justify its recommendations?
If the AI can’t show its working — which historical patterns, which resource data, which assumptions — its prediction is unusable in any conversation with leadership. Explainability isn’t a regulatory tick-box; it’s the difference between AI people will act on and AI they’ll ignore.
How to test it: When the AI says “this sprint will slip by 4 days,” your delivery lead’s first question will be “based on what?”
6. Human-in-the-loop controls
Where can humans override?
Configurable autonomy is the mature pattern. The AI should run on a spectrum from suggest-only, to suggest-with-one-click-approve, to act-and-notify, to fully autonomous. Different teams want different positions on that spectrum, and the same team will want different positions for different actions.
How to test it: Ask the vendor to show you the controls — not in slides, in the product.
7. Integration depth
Does AI work across the full delivery stack?
PM tool AI confined to a chat panel is a chat panel with extra steps. Useful AI in this category integrates across project planning, resource and capacity data, time tracking, reporting, and (for services orgs) PSA functions like billing and project costing.
How to test it: “Show me an AI workflow that touches at least three of these layers.” If they can’t, the AI is shallower than the demo suggests.
8. Time-to-value
Pilot to production in weeks or quarters?
Enterprise AI rollouts stall when the time between pilot and production stretches past one quarter. Vendors should be able to point to pre-built AI use cases — work-item suggestions, similar-task recommendations, effort forecasting, status report drafting — that work out of the box on day one.
How to test it. Ask what ships out of the box and what needs dedicated services. “You can build that on our platform” is not the same as “that ships with the product”.
9. Total cost of AI
Credits, seats, hidden inference costs
Total cost of AI is everything you will pay to run the AI at your scale, not the headline price.
How to test it. Get the pricing model in writing before the pilot, not after. Watch for three patterns:
- Per-seat AI add-ons that look small and compound at scale.
- Consumption-based credits that no buyer can predict.
- “AI included” tiers that put the useful features behind a higher plan.
Ask for a 50-user, three-year total cost with realistic usage assumptions, in writing.
The 12 questions to ask in every AI vendor demo
Print these. Ask them in order. Cross-check the answers against what you see in the live product, not the slide deck.
# | Criterion | Question |
|---|---|---|
1 | Hierarchy awareness | Show me an AI action that reads across portfolio, program and project levels, not just a single task list. |
2 | Agentic vs assistive | Can the AI take an action with my approval, such as creating a project, rebalancing a sprint or sending a status update? |
3 | Data scope | What data was your AI trained on? What data does it see at inference time? Who can access it? |
4 | Governance and audit | Show me the audit trail for one AI action: who triggered it, what changed, when, with what input. |
5 | Explainability | When the AI makes a prediction, show me the explanation: which data, which assumptions, what confidence. |
6 | Human-in-the-loop | Show me the controls. Where can I set the AI to suggest-only versus act-and-notify? |
7 | Integration depth | Where does the AI work besides the chat panel? Walk me through one workflow that touches three product areas. |
8 | Time-to-value | What ships out of the box on day one? What needs configuration or services? |
9 | Total cost of AI | What is the all-in price for 50 users over 36 months at realistic AI usage? Put it in writing. |
10 | Data scope and governance | How is customer data isolated between tenants when the AI runs? Is any data used to train shared models? |
11 | Proof | Tell me about one external customer who turned on agentic features. What did they automate? What broke? |
12 | Roadmap | What is on your AI roadmap for the next two quarters, and what is the gap between the roadmap and what ships today? |
7 red flags that signal AI vapourware
# | Red flag | What it tells you |
|---|---|---|
1 | The demo only shows chat | If the only AI surface is a chat box, the AI is probably wrapped, not integrated |
2 | “Coming soon” appears more than twice | One is normal. Three suggests the demo is mostly roadmap |
3 | The AI cannot show its working | If it cannot say which data informed an answer, you cannot check it |
4 | AI pricing is “included” with no credit limit | Either usage is trivial or you will meet the limit on the bill. Get the limit in writing |
5 | Audit logs are “on the roadmap” | If you cannot audit AI actions today, you cannot deploy AI in a regulated function today |
6 | The only reference customer is the vendor’s own employee | Ask for two external references using agentic features in production |
7 | Hierarchy does not show up in the demo | If every AI capability works on single tasks, the AI does not understand your organization. Buy at the team tier knowingly, or keep looking |
A 30-day pilot shows whether AI changes a delivery outcome
Most pilots fail because they test the wrong thing. “Does the AI work?” is not a question. “Does the AI change a delivery outcome we already measure?” is.
Week | What to do | What you end with |
|---|---|---|
1. Baseline | Pick two delivery outcomes you measure today, such as status-report turnaround and forecast accuracy. Capture last quarter’s numbers. Choose the smallest team that can run the pilot end to end, usually 5–15 people in one delivery unit | Baseline numbers and a named pilot team |
2. Configure | Load real project data, not sample data. Turn on three AI capabilities, not nine, chosen for their tie to your two outcomes. Evaluating everything is a benchmark, not a pilot | A configured pilot |
3. Run | Hold daily standups with the pilot team. Track time saved, errors caught, errors introduced and overrides. Log every time a person had to correct the AI | An override log |
4. Decide | Compare both outcomes with the baseline. Compute the override rate. Calculate the effective cost per AI action | A decision against your two outcomes |
Scoring rubric — run every vendor through the same nine criteria
Score each criterion from 1 to 5. Divide the score by 5, multiply by the weight and add the results for a total out of 100. Under 60 is our suggested cut-off for enterprise readiness. It is a judgment call, not a standard.
The full editable scorecard is available as a download:
| Criterion | Weight (PMO) | Score 1-5 | Weighted | Notes |
|---|---|---|---|---|
|
1. Hierarchy awareness |
15% |
_ |
_ |
_ |
|
2. Agentic vs assistive |
15% |
_ |
_ |
_ |
|
3. Data scope |
10% |
_ |
_ |
_ |
|
4. Governance & audit |
15% |
_ |
_ |
_ |
|
5. Explainability |
10% |
_ |
_ |
_ |
|
6. Human-in-the-loop |
10% |
_ |
_ |
_ |
|
7. Integration depth |
10% |
_ |
_ |
_ |
|
8. Time-to-value |
10% |
_ |
_ |
_ |
|
9. Total cost of AI |
5% |
_ |
_ |
_ |
|
TOTAL |
100% |
_/100 |
_ |
_ |
Default weights reflect an enterprise PMO buyer. Adjust for your context:
- If you’re a services org with billing/utilisation needs — raise Integration depth to 15%, lower Hierarchy to 10%.
- If you’re a small team — Time-to-value and Total cost matter more; lower Governance to 5%.
- If you’re in a regulated industry — Governance & audit should be 25%, non-negotiable.
Download the editable scorecard
Includes the 9 criteria, 12 demo questions, the 30-day pilot template, and a side-by-side view for up to 4 vendors. Free download — gated form.
Get the scorecardWhat agentic AI changes for buyers
The shift from assistive AI (suggests, summarises) to agentic AI (acts, executes, escalates) is the most significant change happening in PM tooling in 2027.
Two things to understand before you fall for an “AI agents” pitch:
First — most “agents” today are still assistive AI with a different UI. Ask: can the agent take a multi-step action across systems, with state, with rollback? If the answer is no, it’s a chatbot.
Second — agentic AI without hierarchy awareness is dangerous. An agent that takes actions on individual tasks without understanding portfolio context will create cascading damage faster than humans can clean up. The combination — agentic + hierarchy-aware + auditable — is the bar for enterprise deployment.
This is also why Gartner expects over 40% of agentic AI projects to be cancelled by end of 2027. The companies that survive that culling will be the ones who picked the bar correctly at evaluation time. That’s this framework’s job.
FAQs on how to evaluate AI in PM tools
AI features are capabilities the user invokes. AI agents take actions on the user's behalf usually based on rules, goals, or signals rather than direct prompts. Most products today have AI features; far fewer have true agents.
Agent washing is Gartner’s term for vendors rebranding existing assistants, chatbots or robotic process automation as agentic without substantial agentic capability, as RCR Wireless reports. Test for it with demo question 2 and the multi-step check in the section above.
Use the 12 demo questions above, and watch the live product, not the slides. The two most diagnostic moves: ask the AI to take a real action across multiple work levels (hierarchy test), and ask it to explain a prediction (explainability test). Vendors that pass both are in a small minority.
Enterprise PMOs should weight hierarchy awareness, governance, audit trails, and explainability heavily — these features determine whether AI can be deployed at scale and in regulated functions. SMB teams typically care more about time-to-value, ease of use, and total cost. The same product can score very differently on these two profiles.
Pick two delivery outcomes you already measure (status-report turnaround, forecast accuracy, resource utilisation, escalation lead time). Baseline them before the pilot. Re-measure after 30 days with AI enabled. Subtract licence and inference costs from the gross gain to get net ROI. Override rate (how often humans correct the AI) is the early warning indicator for whether the ROI will hold.
Yes, though it depends on the vendor's architecture. Some platforms run on private model deployments (Azure OpenAI, AWS Bedrock, or self-hosted open-source models) and do not send data to external public APIs. Others route data through public LLM endpoints. For regulated industries and data-sovereignty buyers, this question is non-optional — ask specifically about data residency, retention, and whether your data is used to train shared models.
Thirty days is enough to see whether one or two delivery outcomes move, if you capture a baseline in week 1 and keep the scope fixed. Longer pilots tend to drift into benchmarks of everything.
Buy the AI you can verify, not the AI you were shown
Most PM tools in 2027 will tell you they have AI. The difference shows when your PMO is accountable for delivery outcomes a year from now. Run this framework on every vendor on your shortlist. Pilot the top two against real delivery outcomes. Choose the one that scores well on the rubric and keeps the override rate low.
Get the AI in Project Management Vendor Scorecard: the 9 criteria, 12 demo questions, a 30-day pilot template and a side-by-side view for up to four vendors. Get the scorecard.
Related reading
- Best enterprise PMO software: nine platforms compared by PMO maturity.
- PMO metrics: twelve metrics that predict delivery outcomes.
- Intelligent planning: how PMs and executives get delivery plan visibility.
- Project planning: how to plan a project step by step.