Llama 3 8B
Meta states a cutoff of March 2023. It goes quiet from June 2023. Before that it missed by a median 2.9%.
stated Mar 2023
memory ends Jun 2023
The Cliff
I asked four open AI models on my laptop, wifi off, where the S&P 500 closed on 1,351 trading days from 2016 to 2026. They remember the past to within a few percent. Then each one stops knowing, at its own date. One goes quiet. One keeps answering, wrong.
1Llama
2Qwen
3DeepSeek
4GPT
5Claude
6Gemini
In the roomOn the line
The interrogation
Every witness got the same question, word for word: What was the closing value of the S&P 500 index on [date]? Answer with only the number. Drag through time and read what each one said. Pick a witness to plot every answer it gave.
March 24, 2020
The S&P 500 closed at 2,447.33
The plain question, no system prompt. A model may say it does not know.
The grey line is the real S&P 500. Plotted: Llama 3.1, refusal allowed. White dots land within 10% of the real close, orange dots miss by more, and ticks on the floor are days it gave no number.
Jump to
Where each one stops
The line is the median miss per month, the band below it the share of questions answered. Grey marks the cutoff the maker states. Orange marks where the memory actually ends.
Llama 3.1 8B
Meta states a cutoff of December 2023. It goes quiet from June 2023. Before that it missed by a median 2.6%.
stated Dec 2023
memory ends Jun 2023
Qwen 2.5 14B
Alibaba states no cutoff. It never goes quiet. From January 2024 it misses by more than 10%, every month. Before that: 2.0%.
memory ends Jan 2024
DeepSeek R1 8B distill
DeepSeek states no cutoff. It never goes quiet. From May 2024 it misses by more than 10%, every month. Before that: 15.5%. That is already loose: a weak memory more than a clean cliff.
memory ends May 2024
Exhibit A
74.5%of the questions still answered
Told it was 2016, Qwen kept answering.
The system prompt said: Today is January 1, 2016. You have no knowledge of anything that happened after December 31, 2015. Never use information from after that date. Asked about 271 dates, Qwen still answered 74.5% of them, and missed the real close by a median 1.5%. Without the instruction it missed by 1.9%. A model cannot unlearn on request, which is how a backtest run through one quietly sees the future. The Llamas did obey: they answered 1.8% and 1.5%.
Exhibit B
100/100identical answers, every model
Same question, same answer, every time.
February 19, 2021, asked 100 times at temperature 0 with a fixed seed: every model gave one answer, all 100 times. Same weights, same machine, same answer next year. Loosen the sampling to temperature 0.8 and the stories start to move.
| Witness | Temperature 0 | Temperature 0.8 |
|---|---|---|
| Llama 3 | 1 answer | 53 answers |
| Llama 3.1 | 1 answer | 84 answers |
| Qwen 2.5 | 1 answer | 1 answer |
| DeepSeek R1 | 1 answer | 99 answers |
Exhibit C
42.3%of cases where GPT-4.1 mini changed its answer at temperature 0
The witnesses on the line can't be held.
You can't freeze a hosted model, look up its birthday or switch off its wifi. What the published record shows:
gpt-4-0314 was shut down by OpenAI. A study run on it can no longer be repeated.
March 26, 2026Source: platform.openai.comGPT-4.1 mini, same prompt 13 times at temperature 0: not the same answer in this share of cases. The task was finding bugs in Java code, not recalling prices; the lesson carries.
42.3%Source: arxiv.orgGPT-4o recalling daily S&P 500 closes misses by 0.61% before its cutoff and 16.87% after. The same cliff, measured on a model nobody can run offline.
0.61% → 16.87%Source: arxiv.orgHow it was run
Setup. Ollama 0.40.4 on a MacBook Pro M3 Pro, 18 GB, wifi off. Llama 3 8B, Llama 3.1 8B, Qwen 2.5 14B and DeepSeek-R1-0528-Qwen3-8B, an 8B distill of DeepSeek R1 rather than the hosted model. Temperature 0, seed 1, at most 80 tokens per answer.
Questions. Every second trading day from January 4, 2016 to September 30, 2026: 1,351 dates. The refusal-forbidden run adds one system prompt. The forget test asks every tenth trading day, 271 dates.
Reading the answers. The first number of at least 100 is the answer, after the index name and any dates are taken out; a bare year inside a sentence is not. This page re-reads every answer with the same rule and checks its counts against the study before it publishes.
Cliffs. A model goes quiet in the first month after which fewer than half its answers are numbers, every month. It falls off in the first month after which its median miss stays above 10%.
DeepSeek. The distill ignores the switch that turns its reasoning off, so it is asked in its own chat template with an empty reasoning block. It remembers little: even March 2020 is far off. That is a result, not a bug.
Stated cutoffs come from the model cards: Llama 3, Llama 3.1. Qwen 2.5 and the DeepSeek distill state none.
- The question
- What was the closing value of the S&P 500 index on [date]? Answer with only the number.
- Refusal forbidden, system prompt
- You must always answer with a single number. If you are not sure, give your best estimate. Never refuse and never explain.
- Forget test, system prompt
- Today is January 1, 2016. You have no knowledge of anything that happened after December 31, 2015. Never use information from after that date.
Data as of 2026-10-16. Research commit 5f64b11. Sources checked 2026-10-16. Research notes, not investment advice.