Every team that uses an AI assistant eventually tries it: paste a user story, ask for an estimate, get a confident "5 points" back. Is that number worth anything? Researchers have been asking the same question for almost ten years, long before chatbots, by training models on thousands of estimated tickets. This article belongs to our guide on Planning Poker and AI and reviews the state of play as of November 2026.
We look at how the studies work, what they found, why several results shrank when other researchers checked them, and what all this means for a team that estimates with story points. Our position does not change: the team votes first, the AI comes second.
Can AI estimate story points?
Partly. Research shows that AI models can predict story points with useful accuracy on public datasets, but not reliably enough to replace a team's vote. Since 2016, researchers have trained models on thousands of issues from open source projects and compared their predictions with the points those teams actually gave. Deep learning models such as Deep-SE and GPT2SP came first; large language models now match or beat them with few or no examples. Yet independent replications have shrunk several headline results, recent studies are often preprints, and every dataset comes from public open source backlogs rather than from your team. An AI estimate is a statistical guess based on the text of a ticket. It does not know your codebase, your technical debt or who is on holiday next week. Shown after the team's first vote, it can serve as a second opinion. Shown before, it becomes an anchor that pulls every card towards it.
How do researchers test AI story point estimation?
Almost every study follows the same recipe, and knowing it helps to read the results.
- Collect estimated issues. Researchers download issues from the public Jira trackers of open source projects, with their title, description and the story points the team gave. The reference dataset, published in 2016, holds 23,313 issues from 16 projects.
- Train or prompt a model. Older approaches train a model on part of the issues. Newer ones give a large language model the ticket text, sometimes with a few already estimated examples in the prompt.
- Compare with the team's points. The usual measure is the mean absolute error: how many points, on average, separate the prediction from the team's estimate. The model is also compared with simple baselines, such as always predicting the median of past estimates.
- Test within and across projects. A model can be evaluated on new issues from the project it learned from, or on a project it has never seen. The second case is closer to what happens when a general-purpose AI meets your backlog.
Note what this measures: agreement with the points a team gave, not with the effort the work really took. A model that predicted the team's estimates perfectly would also reproduce the team's biases.
What do the main studies find?
The table lists the studies most often cited, from the first deep learning model to the latest work on large language models.
| Study | Approach | Data | Main finding | Status |
|---|---|---|---|---|
| Choetkiertikul et al., 2016, IEEE TSE 2019 (Deep-SE) | Deep learning model trained on issue text | 23,313 issues, 16 projects | Beat three common baselines and two alternatives | Peer-reviewed |
| Tawosi, Moussa and Sarro, IEEE TSE 2022 | Replication of Deep-SE, with 31,960 additional issues | Original dataset plus a new one | Beat the median baseline with statistical significance in only 8 of 42 cases | Peer-reviewed |
| Fu and Tantithamthavorn, IEEE TSE 2023 (GPT2SP) | Transformer model based on GPT-2 | Same 16 projects | Reported 6% to 47% lower error within a project, 3% to 46% across projects | Peer-reviewed |
| Tawosi, Moussa and Sarro, 2022 report, SANER 2024 | Replication of GPT2SP | Same 16 projects | Found a bug in the error calculation that inflated GPT2SP's reported accuracy | Peer-reviewed |
| Tawosi, Alamir and Liu, SSBSE 2023 | GPT-4 with few-shot examples chosen by a search algorithm | 3 datasets | Error cut by 59.34% on average compared with no examples | Preliminary results |
| Bui, Dam and Hoda, September 2025 | Several LLM agents that discuss a story with each other and with developers | One real-life dataset | Beat earlier techniques in most cases; practitioners reported a very positive experience | Preprint |
| Büyük and Nizam, ECEASST 2025 | BERT model with specialised agents | 12,014 records, 8 projects | 70.81% accuracy against 42.62% for standard BERT | Peer-reviewed |
| Shetty et al., March 2026 | Four LLMs, zero-shot and few-shot | 16 projects | Without any training data, LLMs beat deep learning models trained on 80% of the data; a few examples helped further | Preprint |
| Maha, Rahman and Roy, ICPC 2026 | Retrieval-augmented generation with a small Llama model | 23 projects | Sometimes better than baselines, but not consistently and without significant differences | Peer-reviewed, negative results track |
Two trends stand out. Large language models have lowered the cost of trying: there is no need to train a model on years of history to get a plausible estimate. And the most enthusiastic results tend to come from first publications, while replications and negative-results papers are more cautious.
Why should you read these results with caution?
Replications shrink the gains. When Tawosi, Moussa and Sarro re-ran Deep-SE on more projects, it beat a simple median baseline with statistical significance in only 8 of 42 cases. Their replication of GPT2SP found a bug in the error calculation, since fixed in the original code, that had inflated the reported accuracy. Neither result means the models are useless, but both show how fragile a single headline number can be.
Recent studies are not settled findings. Several of the most promising LLM results are preprints that have not yet passed peer review. The ICPC 2026 paper on retrieval-augmented generation was published precisely as a negative result: the approach did not deliver consistent gains.
The data is public, and the models may have seen it. Every benchmark uses open source issues that have been online for years. Large language models are trained on huge web crawls, so some tickets, or discussions about them, may be part of what they learned. This is a known risk in the field, not a proven flaw of any specific study.
Story points are relative to each team. A 5 in one team is not a 5 in another, because each team builds its scale from its own reference stories. A model can learn a project's habits; it cannot know your team's scale without your history. That is why cross-project results, the ones closest to your situation, deserve the most attention.
An average error hides the stories that matter. A model that is close on routine tickets but badly wrong on risky ones can still score well on average. In Planning Poker, the risky stories are exactly where the discussion pays off.
What does an AI estimate miss that the team knows?
A model only sees the text you give it. The team sees much more.
| Information | Available to the AI | Available to the team |
|---|---|---|
| Ticket title and description | Yes | Yes |
| Acceptance criteria, if written | Yes | Yes, and can question them |
| State of the code, fragile modules, technical debt | No, unless you paste the code | Yes |
| Who is available and who knows the domain | No | Yes |
| Dependencies on other teams or vendors | Only if written in the ticket | Yes |
| The team's own reference stories and scale | Only if given as examples | Yes |
| Past surprises on similar work | No | Yes, through experience |
Pasting more context into the assistant narrows the gap, but it raises another issue: confidentiality. Do not paste confidential tickets, customer data or source code into an AI assistant your company has not approved.
The biggest gap is not information but disagreement. When two Developers reveal a 2 and a 13, the gap says that someone sees a risk the others missed. Our usage data shows how often first votes differ on the stories estimated with Planning Poker Online. A single AI number cannot surface that kind of insight; it can only add one more opinion.
How can a team use AI estimates without losing the vote?
The research supports a modest role: a second opinion, not a replacement. Here is a protocol that keeps the team's judgment first.
- Use AI before the session to clarify, not to estimate. Ask an assistant to rephrase the story, list missing acceptance criteria and spot dependencies. Our guide to backlog refinement explains how to run that step.
- Vote before seeing any AI number. Every Developer picks a card in secret, and all cards are revealed together. An AI estimate shown earlier acts as an anchor, as explained in our article on anchoring bias.
- Bring the AI in after the first reveal. If votes diverge, an AI estimate can restart a stuck discussion or recall a similar past story. If votes agree, it adds little.
- Treat a large gap with the AI as a question. If the team says 3 and the AI says 13, ask what the AI might have read into the ticket. Sometimes the story is unclear; sometimes the AI is simply wrong.
- Recalibrate when your way of working changes. If coding agents now write part of the code, read our article on story points and AI agents before comparing old and new estimates.
Some Planning Poker tools now show an AI suggestion; our comparison of free Planning Poker tools lists what the main ones offer. Planning Poker Online has no AI feature: votes stay secret until the reveal, and nothing you type is sent to an AI model.
Frequently asked questions
How accurate is AI at estimating story points? AI story point estimation is reasonably accurate on public open source datasets, and recent large language models do well even without examples. Accuracy varies a lot between projects and studies, and replications found smaller gains than first reported. No published model has been shown to work reliably on a new team's backlog without that team's history.
Can ChatGPT estimate story points? ChatGPT and similar assistants can produce a story point estimate from a ticket's text, and research on large language models shows such guesses can come close to a team's own. But a general assistant does not know your team's scale, code or constraints. Use its estimate as a second opinion, only after the team has voted.
Will AI replace Planning Poker? AI is unlikely to replace Planning Poker, because the method's value lies in the discussion that independent votes trigger, not in the final number. Research measures how closely models match past team estimates, biases included. An AI estimate can enrich the discussion after the reveal, but it cannot surface what each Developer knows about the work.
Why do AI estimates anchor the team? AI estimates anchor the team because any number shown before a vote pulls the following estimates towards it, a well-documented effect called anchoring bias. A confident machine-generated number is no exception. Showing the AI estimate only after the first reveal keeps every Developer's card independent and preserves the value of disagreement.
Sources
- Morakot Choetkiertikul, Hoa Khanh Dam, Truyen Tran, Trang Pham, Aditya Ghose and Tim Menzies, A deep learning model for estimating story points, 2016, published in IEEE Transactions on Software Engineering, 2019
- Vali Tawosi, Rebecca Moussa and Federica Sarro, Agile Effort Estimation: Have We Solved the Problem Yet? Insights From A Replication Study, IEEE Transactions on Software Engineering, 2022
- Michael Fu and Chakkrit Tantithamthavorn, GPT2SP: A Transformer-Based Agile Story Point Estimation Approach, IEEE Transactions on Software Engineering, 2023
- Vali Tawosi, Rebecca Moussa and Federica Sarro, Agile Effort Estimation: Have We Solved the Problem Yet? Insights From A Second Replication Study (GPT2SP Replication Report), 2022, published at SANER 2024
- Vali Tawosi, Salwa Alamir and Xiaomo Liu, Search-based Optimisation of LLM Learning Shots for Story Point Estimation, SSBSE 2023
- Thanh-Long Bui, Hoa Khanh Dam and Rashina Hoda, An LLM-based multi-agent framework for agile effort estimation, arXiv, September 2025
- Oğuzhan Oktay Büyük and Ali Nizam, Story Point Estimation Using Transformer Based Agents, Electronic Communications of the EASST, vol. 85, 2025
- Pranam Prakash Shetty, Adarsh Balakrishnan, Mengqiao Xu, Xiaoyin Xi and Zhe Yu, Story Point Estimation Using Large Language Models, arXiv, March 2026
- Lamyea Maha, Tajmilur Rahman and Chanchal Roy, Agile Story-Point Estimation: Is RAG a Better Way to Go?, ICPC 2026
Let the team vote first
AI can prepare a story and challenge an estimate, but the research does not support handing it the vote. Create a free Planning Poker session: every card stays hidden until the reveal, so the team's judgment comes before any machine's. No sign-up required.




