Why most AI demos in PM tools are theatre
Every PM tool you’re evaluating in 2026 has “AI features.”
Some of it works. Most of it is a chatbot bolted onto a task list, a summary generator that re-states what’s on the screen, or a prediction that breaks the moment your data leaves the demo environment.
Digital.ai’s 18th State of Agile Report (2026) found that AI adoption in delivery teams has surged from 68% to 84% in one year, but only 49% of those organisations have guardrails in place. Gartner forecasts that over 40% of agentic AI projects will be cancelled by end of 2027 — the cited reasons are escalating costs, unclear business value, and inadequate risk controls.
What to expect from this piece:
- A 9-criteria framework,
- 12 vendor demo questions,
- a 30-day pilot plan,
- 7 red flags to watch for, and
- a downloadable scorecard.
If you’re shopping for an enterprise PM platform with serious AI capabilities, you need a framework that , goes beyond the marketing page. This will give you a scoring rubric too that you can run every vendor through.
The 9-criteria evaluation framework
Every AI capability in a PM tool should be evaluated on these nine dimensions. We’ve ordered them roughly by how often vendors fail them — hierarchy awareness and explainability are the two most common gaps; pricing transparency is where the surprise bill usually shows up.
| # | Criterion | What good looks like |
|---|---|---|
|
1 |
Hierarchy awareness |
AI understands the work item in context — task is part of a project, project is part of a program, program rolls up to a portfolio. It can reason across levels |
|
2 |
Agentic vs assistive |
AI can act on your behalf (create, assign, schedule, escalate). |
|
3 |
Data scope |
Trained / grounded on your actual project data — not generic web data dressed up as project intelligence. |
|
4 |
Governance and audit |
Every AI action is logged with who triggered it, what changed, and why. Audit trail is exportable. |
|
5 |
Explainability |
AI can justify its recommendation: what data it used, what assumptions, what confidence level. |
|
6 |
Human-in-the-loop controls |
Reviewer queues, approval gates, configurable autonomy levels — humans can override at every step. |
|
7 |
Integration depth |
AI works across PM, PSA, reporting, and resource data — not isolated to one slice (just chat, just summaries). |
|
8 |
Time-to-value |
Pilot to production in weeks, not quarters. Pre-built use cases out of the box |
|
9 |
Total cost of AI |
Pricing is transparent — credits, seats, inference costs. No surprise bill at month three. |
1. Hierarchy awareness
Does the AI understand portfolio → program → project → task?
Most PM tools were built around flat task lists. Their AI inherits that flatness. Ask the vendor to demonstrate a query like: “What’s the impact on the Apollo program if the Backend API project slips two weeks?” An AI that only sees tasks will return a list of overdue cards. An AI with hierarchy awareness will trace dependencies up to the program, identify which downstream projects are affected, surface the resource conflicts that follow, and suggest which scope to defer.
This single capability separates enterprise-grade AI from team-level AI. If your PMO needs portfolio visibility, hierarchy awareness is non-negotiable.
2. Agentic vs assistive
Can it act, or only summarise?
An assistive AI describes the world. An agentic AI changes it. The cleanest test: ask the AI to do something irreversible (with your permission) — create a project from a brief, rebalance a sprint, draft a status report and send it. If the vendor can only show you summaries, smart search, and “AI-suggested” content the user has to act on manually, you are looking at an assistant, not an agent.
Gartner predicts that 40% of enterprise apps will feature task-specific AI agents by 2026, up from less than 5% in 2025. The split between assistive and agentic AI is the next major dividing line in this category — and it’s the one most vendors are quietly behind on.
3. Data scope
Is the AI grounded on your project data, or generic web data?
Ask: “What data was your model trained on?” and “What data does it see at inference time?” A good answer names your tenant’s project, resource, and historical delivery data. A bad answer is some variant of “we use a leading large language model.” That tells you the AI knows about the world; it doesn’t tell you whether it knows about your business. For enterprise PMOs, the latter is what matters.
4. Governance and audit
Can you see what the AI did and why?
Every AI action should generate an audit entry: who or what triggered it, what changed, when, with what input. This is non-negotiable for regulated industries — Financial Services, Healthcare, Public Sector — and increasingly expected by IT governance teams in every industry. The Digital.ai 2026 report’s finding that only 49% of organisations have AI guardrails in place is the practical version of this problem. Don’t be one of them.
5. Explainability
Can it justify its recommendations?
When the AI says “this sprint will slip by 4 days,” your delivery lead’s first question will be “based on what?” If the AI can’t show its working — which historical patterns, which resource data, which assumptions — its prediction is unusable in any conversation with leadership. Explainability isn’t a regulatory tick-box; it’s the difference between AI people will act on and AI they’ll ignore.
6. Human-in-the-loop controls
Where can humans override?
Configurable autonomy is the mature pattern. The AI should run on a spectrum from suggest-only, to suggest-with-one-click-approve, to act-and-notify, to fully autonomous. Different teams want different positions on that spectrum, and the same team will want different positions for different actions. Ask the vendor to show you the controls — not in slides, in the product.
7. Integration depth
Does AI work across the full delivery stack?
PM tool AI confined to a chat panel is a chat panel with extra steps. Useful AI in this category integrates across project planning, resource and capacity data, time tracking, reporting, and (for services orgs) PSA functions like billing and project costing. Ask: “Show me an AI workflow that touches at least three of these layers.” If they can’t, the AI is shallower than the demo suggests.
8. Time-to-value
Pilot to production in weeks or quarters?
Enterprise AI rollouts stall when the time between pilot and production stretches past one quarter. Vendors should be able to point to pre-built AI use cases — work-item suggestions, similar-task recommendations, effort forecasting, status report drafting — that work out of the box on day one. “You can build that with our platform” is not the same as “that ships with the product.”
9. Total cost of AI
Credits, seats, hidden inference costs
Get the pricing model in writing before the pilot, not after. The patterns to watch: per-seat AI add-ons that look small but compound at scale; consumption-based credits that no buyer can predict; “AI included” tiers that gate the actually useful features behind a higher plan. Ask for a 50-user, three-year TCO with realistic usage assumptions. Most vendors won’t put this in writing voluntarily. Make them.
Need a comparison view?
See how Nimble AI, Asana AI, and Monday AI stack up against these criteria — side by side
AI compatibility comparisonThe 12 questions to ask in every AI vendor demo
Print these. Ask them in order. Cross-check the answers against what you see in the live product, not the slide deck.
Q1. Show me an AI action that reads across portfolio, program, and project levels — not just a single task list.
Q2. Can the AI take an irreversible action with my approval — create a project, rebalance a sprint, send a status update?
Q3. What data was your AI trained on? What data does it see at inference time? Who can access it?
Q4. Show me the audit trail for one AI action — who triggered it, what changed, when, with what input.
Q5. When the AI makes a prediction, show me the explanation — which data, which assumptions, what confidence.
Q6. Show me the human-in-the-loop controls. Where can I set the AI to suggest-only vs act-and-notify?
Q7. Where does the AI work besides the chat panel? Walk me through one workflow that touches three product areas.
Q8. What ships out of the box on day one? What requires configuration or services to enable?
Q9. What’s the all-in price for 50 users over 36 months at realistic AI usage? Put it in writing.
Q10. How is customer data isolated between tenants when the AI is running? Is any data used to train shared models?
Q11. Tell me about one customer who turned on agentic features. What did they automate? What broke?
Q12. What’s on your AI roadmap for the next two quarters — and what’s the gap between roadmap and shipped today?
How to run a 30-day AI pilot that actually tells you something
Most pilots fail because they test the wrong thing. “Does the AI work?” is not a question. “Does the AI change a delivery outcome we already measure?” is.
A 30-day pilot done well looks like this:
Week 1 — Baseline. Pick two delivery outcomes you measure today: e.g. status-report turnaround time and forecast accuracy. Capture last quarter’s numbers as the baseline. Identify the smallest team that can run the pilot end-to-end — usually 5–15 people in one delivery unit.
Week 2 — Configure. Load real (not sample) project data. Turn on three AI capabilities, not nine. Choose the three most directly tied to your two outcomes. Resist the urge to evaluate “everything” — that’s a benchmark, not a pilot.
Week 3 — Run. Daily standups with the pilot team. Track time saved, errors caught, errors introduced, and overrides — every time a human had to correct the AI, log it. This becomes your trust calibration data.
Week 4 — Decide. Compare your two outcomes against baseline. Compute the override rate. Calculate effective cost per AI action. If status-report time dropped and override rate is under 15%, the AI is real. If override rate is over 30% or outcomes didn’t move, the AI is theatre.
One discipline matters more than the rest: do not change the pilot scope mid-pilot. Vendors will offer to “customise” things when they see the AI is missing the mark. That’s a sales tactic, not an evaluation. Customisation in week three is a fail signal, not a feature
7 red flags that signal AI vapourware
The demo only shows chat.
If the only AI surface you see is a Copilot-style chat box, the AI is probably wrapped, not integrated.
“Coming soon” appears more than twice in the demo.
Vapourware roadmap features tend to cluster. One “coming soon” is normal. Three is the product strategy.
The AI can’t show its working.
If the AI gives an answer but can’t tell you which data informed it, it’s running blind — and so are you.
Pricing for AI is “included” — without a credit limit.
Either the AI usage is trivial, or you’ll see the credit limit when the bill arrives. Get the limit in writing.
Audit logs are “on the roadmap.”
If you can’t audit AI actions today, you can’t deploy AI in any regulated function today.
The reference customer is the vendor’s own employee.
Ask for two external references using agentic features in production. If you get one internal user and a slide, that’s the truth.
Hierarchy doesn’t show up in the demo.
If every AI capability operates on individual tasks, the AI doesn’t understand your org. Walk away — or buy at the team tier knowingly.
Scoring rubric — run every vendor through this
Score each vendor on the nine criteria. Multiply by the weight that reflects your buying context (enterprise PMO vs team-level use case vs services org). Total out of 100. Anything under 60 is not enterprise-ready.
The full editable scorecard is available as a download:
| Criterion | Weight (PMO) | Score 1-5 | Weighted | Notes |
|---|---|---|---|---|
|
1. Hierarchy awareness |
15% |
_ |
_ |
_ |
|
2. Agentic vs assistive |
15% |
_ |
_ |
_ |
|
3. Data scope |
10% |
_ |
_ |
_ |
|
4. Governance & audit |
15% |
_ |
_ |
_ |
|
5. Explainability |
10% |
_ |
_ |
_ |
|
6. Human-in-the-loop |
10% |
_ |
_ |
_ |
|
7. Integration depth |
10% |
_ |
_ |
_ |
|
8. Time-to-value |
10% |
_ |
_ |
_ |
|
9. Total cost of AI |
5% |
_ |
_ |
_ |
|
TOTAL |
100% |
_/100 |
_ |
_ |
Default weights reflect an enterprise PMO buyer. Adjust for your context:
- If you’re a services org with billing/utilisation needs — raise Integration depth to 15%, lower Hierarchy to 10%.
- If you’re a small team — Time-to-value and Total cost matter more; lower Governance to 5%.
- If you’re in a regulated industry — Governance & audit should be 25%, non-negotiable.
Download the editable scorecard
Includes the 9 criteria, 12 demo questions, the 30-day pilot template, and a side-by-side view for up to 4 vendors. Free download — gated form.
Get the scorecardA note on agentic AI
The shift from assistive AI (suggests, summarises) to agentic AI (acts, executes, escalates) is the most significant change happening in PM tooling in 2026. Gartner forecasts that 40% of enterprise applications will feature task-specific AI agents by 2026, up from under 5% in 2025.
Two things to understand before you fall for an “AI agents” pitch:
First — most “agents” today are still assistive AI with a different UI. Ask: can the agent take a multi-step action across systems, with state, with rollback? If the answer is no, it’s a chatbot.
Second — agentic AI without hierarchy awareness is dangerous. An agent that takes actions on individual tasks without understanding portfolio context will create cascading damage faster than humans can clean up. The combination — agentic + hierarchy-aware + auditable — is the bar for enterprise deployment.
This is also why Gartner expects over 40% of agentic AI projects to be cancelled by end of 2027. The companies that survive that culling will be the ones who picked the bar correctly at evaluation time. That’s this framework’s job.
FAQs
AI features are capabilities the user invokes. AI agents take actions on the user's behalf usually based on rules, goals, or signals rather than direct prompts. Most products today have AI features; far fewer have true agents.
Use the 12 demo questions above, and watch the live product, not the slides. The two most diagnostic moves: ask the AI to take a real action across multiple work levels (hierarchy test), and ask it to explain a prediction (explainability test). Vendors that pass both are in a small minority.
Enterprise PMOs should weight hierarchy awareness, governance, audit trails, and explainability heavily — these features determine whether AI can be deployed at scale and in regulated functions. SMB teams typically care more about time-to-value, ease of use, and total cost. The same product can score very differently on these two profiles.
Pick two delivery outcomes you already measure (status-report turnaround, forecast accuracy, resource utilisation, escalation lead time). Baseline them before the pilot. Re-measure after 30 days with AI enabled. Subtract licence and inference costs from the gross gain to get net ROI. Override rate (how often humans correct the AI) is the early warning indicator for whether the ROI will hold.
Yes, though it depends on the vendor's architecture. Some platforms run on private model deployments (Azure OpenAI, AWS Bedrock, or self-hosted open-source models) and do not send data to external public APIs. Others route data through public LLM endpoints. For regulated industries and data-sovereignty buyers, this question is non-optional — ask specifically about data residency, retention, and whether your data is used to train shared models.
Conclusion — buy the AI you can verify, not the AI you were shown
Most PM tools in 2026 will tell you they have AI. Some will. Most will have GPT pasted onto a task list. The difference matters when your PMO is on the line for delivery outcomes a year from now.
Run this framework on every vendor in your shortlist. Pilot the top two against real delivery outcomes. Buy the one that scores above 60 on the rubric and below 15% on the override rate.
Two next steps:
- Download the AI in PM Vendor Scorecard — the 9-criteria framework, 12 demo questions, and a 4-vendor side-by-side view.
- See how Nimble’s agentic AI (kAIron) handles enterprise hierarchies — book a 30-minute architecture walkthrough.