A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models
Using TextFlint across 21 datasets, the study finds RLHF improves human-like dialogue but not overall NLU accuracy or robustness.
Key Findings
Methodology
The authors query six OpenAI API models—davinci, text-davinci-001, code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo. They cover nine NLU tasks and 21 English/Chinese datasets, comparing zero-shot, one-shot, and three-shot prompting. TextFlint supplies transformations such as AddSent, back-translation, and target reversal. Accuracy, F1, Exact Match, analyzability, and robustness are reported.
Key Results
- Capability growth is non-monotonic. In zero-shot evaluation, code-davinci-002 leads ABSA, MRC, and sentiment classification; text-davinci-003 is strongest on POS, relation extraction, and semantic matching; gpt-3.5-turbo performs best on NLI and WSC. On SQuAD1.1, however, code-davinci-002 obtains about 83.58 micro-F1 versus 55.26 for gpt-3.5-turbo.
- RLHF does not automatically add task knowledge. text-davinci-003 is broadly comparable to text-davinci-002, but is worse on sentiment classification and semantic matching. gpt-3.5-turbo is similar to text-davinci-003 overall, yet weaker on MRC, POS, and relation extraction, possibly reflecting chat optimization or smaller scale.
- Few-shot prompting is useful but conditional. Examples strongly improve davinci’s analyzability on NER and POS, while gains on inference tasks are limited. On Restaurant ABSA, code-davinci-002 reaches 100% original accuracy in a three-shot ReverseNonTarget setting, but transformed inputs still cause substantial degradation.
Significance
The study separates pretraining knowledge, instruction following, conversational alignment, and robustness instead of treating “better chat” as universal intelligence. This matters academically because model-generation narratives often rely on isolated benchmarks. It matters industrially because a fluent answer may still be unusable for extraction, reading comprehension, or exact classification. The near-unchanged robustness across most iterations also shows that larger or newer models require dedicated stress testing rather than assumed reliability.
Technical Contribution
Its main contribution is a unified black-box evaluation protocol spanning model lineage, tasks, languages, prompting regimes, and perturbations. Promptsource templates and manually designed prompts are narrowed to three strong prompts per dataset; label phrases are mapped for NER, POS, and RE. Separating analyzability from task score distinguishes failure to understand a task from failure to emit the expected format. TextFlint transformations then expose whether apparent performance survives surface and semantic changes.
Novelty
Unlike studies that compare ChatGPT with fine-tuned systems on selected tasks, this work analyzes capability evolution from GPT-3 to GPT-3.5 and relates changes to FeedME, code training, and PPO/RLHF. The novelty is empirical and diagnostic rather than architectural: a 21-dataset longitudinal comparison demonstrates non-monotonic progress, prompt sensitivity, and the alignment tax within one model family.
Limitations
- Evaluation budgets differ: davinci and code-davinci-002 use 100 samples, text-davinci-001 and text-davinci-002 use 1,000, while text-davinci-003 and gpt-3.5-turbo use full datasets. Sampling therefore complicates direct comparisons.
- OpenAI does not disclose training data, supervised task mixtures, parameter counts, or complete optimization details. Claims that chat models are weaker because of smaller size remain hypotheses, not controlled causal conclusions.
- Only three strong prompts per dataset are selected, and API models can change over time. Results may therefore be prompt- and date-dependent rather than universal.
Future Work
Future studies should use matched samples, repeated trials, fixed decoding controls, and fully released prompts. Controlled training ablations could isolate supervision, code data, PPO, and reward modeling. Broader languages, model scales, calibration, factuality, safety, cost, and robustness should be evaluated jointly. A useful direction is multi-objective alignment that preserves structured-task accuracy while improving human preference and interaction quality.
AI Executive Summary
Progress in large language models is often portrayed as a steadily rising curve. This paper complicates that story. The Fudan University team evaluates six representatives of the GPT-3/GPT-3.5 lineage—davinci, text-davinci-001, code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo—on nine NLU tasks and 21 English and Chinese datasets. Using TextFlint transformations, the authors also ask whether models understand meaning or merely exploit familiar wording.
The answer is decidedly non-monotonic. In zero-shot tests, code-davinci-002 leads ABSA, MRC, and sentiment classification; text-davinci-003 performs best on POS, relation extraction, and semantic matching; and gpt-3.5-turbo is strongest on NLI and WSC. Yet on SQuAD1.1, code-davinci-002 reaches about 83.58 micro-F1, compared with 55.26 for gpt-3.5-turbo. PPO-based RLHF makes text-davinci-003 more human-like, but does not consistently raise NLU scores and even hurts sentiment classification and semantic matching relative to text-davinci-002.
The broader message is that conversational quality, task competence, and robustness are distinct properties. Few-shot examples can make davinci’s NER and POS outputs analyzable, but do little for some inference tasks. Across most tasks, perturbations cause large drops and newer models do not reliably resist them. The paper therefore recommends longitudinal, task-specific evaluation: before deployment, organizations should test exact correctness, output format, prompt sensitivity, and stability under realistic rewrites—not merely whether an answer sounds persuasive.
Deep Analysis
Background
GPT-3, represented by the 175-billion-parameter davinci, established the idea that many NLU problems could be expressed as generation. OpenAI then pursued instruction tuning through FeedME, code training through Codex, and preference optimization through PPO/RLHF, producing the GPT-3.5 family and ChatGPT. Prior work mainly contrasted an individual GPT model with fine-tuned systems; the evolution of capabilities within the GPT lineage remained insufficiently mapped.
Core Problem
The paper asks how pretraining, supervised instruction tuning, code data, and RLHF alter different NLU abilities. Does more natural dialogue imply better classification, extraction, or reading comprehension? It also examines whether zero-shot and few-shot gains are stable and whether models retain accuracy after lexical, syntactic, or semantic perturbations. These questions are difficult because model sizes, data, prompts, and API versions are not fully disclosed.
Innovation
- �� Longitudinally compares six GPT models rather than treating ChatGPT as a single endpoint.
- �� Covers nine tasks, 21 datasets, and English/Chinese settings.
- �� Separates analyzability—whether an answer follows the requested format—from correctness.
- �� Combines promptsource, manual prompts, and TextFlint transformations for stress testing.
- �� Provides evidence that FeedME and PPO/RLHF change the capability profile, including an alignment tax in structured tasks.
Methodology
- �� Models: official OpenAI APIs for davinci, text-davinci-001, code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo.
- �� Tasks and data: ABSA, MRC, NER, NLI, POS, RE, sentiment classification, semantic matching, and WSC; examples include SQuAD1.1/2.0, MNLI, SNLI, CoNLL2003, Tacred, IMDB, MRPC, and QQP.
- �� Prompts: three selected templates per dataset, expanded to zero-, one-, and three-shot settings; labels are verbalized for NER, POS, and RE.
- �� Metrics: Accuracy for classification, F1 for sequence tasks and RE, and micro-F1 plus EM for MRC.
- �� Robustness: TextFlint generates AddDiff, AddSent, ReverseTarget, back-translation, MLM perturbation, and question modifications; original and transformed scores are compared.
Experiments
The benchmark includes English and Chinese datasets such as SemEval2014-Laptop/Restaurant, SQuAD1.1/2.0, ACE2005, CoNLL2003, OntoNotes v5, HONOR, MSRANER, MNLI-m/mm, SNLI, WSJ, PKU-SEGPOS, Tacred, IMDB, MRPC, QQP, and WSC273. davinci and code-davinci-002 are evaluated on 100 samples, text-davinci-001/002 on 1,000, and the final two models on full datasets. Zero-shot and few-shot performance, analyzability, and TextFlint robustness are compared.
Results
There is no universal version-by-version improvement. On SQuAD1.1 zero-shot, code-davinci-002 scores 83.58 micro-F1 versus 55.26 for gpt-3.5-turbo; on three-shot SemEval2014-Laptop original accuracy, the values are 97.00 and 90.75. Few-shot examples markedly improve davinci’s NER/POS analyzability but not necessarily inference accuracy. Except for ABSA, robustness differences are generally small, while transformations often produce substantial score drops.
Applications
The findings support task-specific model selection. Code-trained models may be strong on structured understanding, instruction-tuned models may better follow explicit formats, and chat models should not automatically replace MRC or extraction systems. Production teams can use SQuAD, IMDB, Tacred, and Chinese NER/POS sets to test Accuracy, F1, EM, and analyzability. TextFlint-style rewrites are useful for pre-launch stress tests in search, customer service, education, and data annotation.
Limitations & Outlook
This is a black-box API study: training mixtures, parameter counts, supervision, decoding details, and version changes are opaque, limiting causal attribution. Unequal sample sizes and selection of only three strong prompts complicate statistical generalization. The paper also does not provide every raw score or a comprehensive significance analysis. Future work should conduct controlled training ablations, evaluate more languages and scales, and jointly measure calibration, factuality, safety, cost, and robustness.
Plain Language Accessible to non-experts
Imagine six students taking the same school examinations. The school prepares nine subjects and 21 papers: reading passages, emotion judgments, name finding, word labels, and sentence comparisons. Each student first answers alone, then receives a few solved examples. Finally, the teacher changes the wording, reverses parts of a sentence, or adds distracting information to see whether the student understands the lesson rather than memorizing the paper.
The newest student is not automatically best in every subject. code-davinci-002 does especially well on several reading and sentiment tests, while text-davinci-003 is stronger on word and relation tasks. The chat-oriented student can be good at reasoning conversations but weak at structured reading. On SQuAD1.1, code-davinci-002 reaches about 83.58 F1, while gpt-3.5-turbo reaches only 55.26.
RLHF is like coaching a student to be polite, helpful, and pleasant to talk with. That can improve conversation without teaching every examination skill, and may even reduce performance on some exact tests. Examples help davinci understand how to format NER and POS answers, but not every reasoning problem. The lesson is simple: an AI that sounds confident still needs testing for correctness, format, and resistance to changed wording.
ELI14 Explained like you're 14
Think of the GPT models as different gaming characters. They all trained in a huge world of text, but then received different upgrades. davinci is the original general character. FeedME teaches a character to follow instructions. Code training adds lots of programming examples. PPO and RLHF are like coaches who reward answers that people find helpful, polite, and natural.
Now send six characters into nine kinds of school challenges across 21 datasets. Some ask for emotions in product reviews, some ask who or what is mentioned, and some ask whether two sentences match. The surprising part? The newest chat character is not the champion everywhere! code-davinci-002 gets about 83.58 F1 on SQuAD1.1, while gpt-3.5-turbo gets about 55.26. That is like having the best social-media skills but a lower reading-test score.
The researchers also remix questions: they translate them back and forth, shuffle wording, or add distractions. Most characters lose points. Giving examples usually helps, but the result depends on the examples and the exact prompt. davinci especially learns how to format name-finding and word-label answers after seeing examples.
So what should we remember? “Sounds human” and “is correct” are different stats! Before trusting an AI for homework, customer service, or sorting documents, test several wordings, check its format, and compare answers with a trusted source. Even a super-smart game assistant needs a proper exam before joining the team!
Glossary
Natural Language Understanding (NLU)
The ability to interpret text and perform classification, extraction, matching, or inference. Technically, it evaluates task-level semantic behavior rather than fluent generation alone.
The paper evaluates nine NLU task families.
In-context learning
Learning a task from examples placed in the prompt without changing model parameters. Few-shot prompting is the standard implementation.
The study compares zero-shot, one-shot, and three-shot conditions.
RLHF
Reinforcement learning from human feedback, usually using a reward model trained from human preference comparisons. PPO is the optimization algorithm discussed here.
text-davinci-003 uses PPO/RLHF.
FeedME
A supervised fine-tuning strategy using human demonstrations and high-quality model samples rated by people. It improves instruction following but does not guarantee broad task knowledge.
text-davinci-001 and text-davinci-002 are described as FeedME models.
Robustness
The ability to preserve performance when inputs undergo reasonable changes. High original accuracy alone does not establish robustness.
TextFlint transformations measure score degradation.
Alignment tax
A potential loss in task performance when optimization for human preferences or interaction behavior conflicts with exact problem solving. It is a capability trade-off, not a universal law.
The paper relates it to text-davinci-002 versus text-davinci-003.
Open Questions Unanswered questions from this research
- 1 Because training data, supervised task mixtures, and parameter counts are undisclosed, the study cannot separate effects of scale, data, instruction tuning, and RLHF.
- 2 Few-shot gains depend strongly on prompt and example selection; a general theory of prompt stability across tasks and languages is still missing.
- 3 Large degradation under perturbations may indicate reliance on surface patterns rather than semantics, but the mechanism remains unresolved.
Applications
Immediate Applications
Task-specific model selection
Organizations should benchmark candidate models on representative datasets such as SQuAD, IMDB, Tacred, and Chinese NER/POS data. Compare Accuracy, F1, EM, and analyzability rather than relying on conversational impressions, then select models per workflow.
Prompt and robustness monitoring
Production teams can freeze validated templates, retain a few high-quality demonstrations, and replay tests under back-translation, reordering, spelling, and distractor changes. Structured tasks should enforce machine-readable labels and reject malformed outputs.
Long-term Vision
Multi-objective alignment
Future systems could optimize human preference, task accuracy, format compliance, and perturbation robustness together. This may preserve structured reasoning while retaining the helpful conversational behavior sought by RLHF.
Abstract
GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on, have gained considerable attention due to their exceptional natural language processing capabilities. However, despite the abundance of research on the difference in capabilities between GPT series models and fine-tuned models, there has been limited attention given to the evolution of GPT series models' capabilities over time. To conduct a comprehensive analysis of the capabilities of GPT series models, we select six representative models, comprising two GPT-3 series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo). We evaluate their performance on nine natural language understanding (NLU) tasks using 21 datasets. In particular, we compare the performance and robustness of different models for each task under zero-shot and few-shot scenarios. Our extensive experiments reveal that the overall ability of GPT series models on NLU tasks does not increase gradually as the models evolve, especially with the introduction of the RLHF training strategy. While this strategy enhances the models' ability to generate human-like responses, it also compromises their ability to solve some tasks. Furthermore, our findings indicate that there is still room for improvement in areas such as model robustness.