Skip to content

GPT-3

Quick Facts

Attribute Value
Released June 2020
Organization OpenAI
Size 175 billion parameters
Training Data 570GB (Common Crawl, WebText2, Books, Wikipedia)
Context Window 2,048 tokens
Key Innovation Few-shot learning without fine-tuning
License API only (proprietary)

Historical Significance

GPT-3 was the breakthrough moment for LLMs:

  • First model to show true few-shot learning
  • Demonstrated emergent abilities at scale
  • Made LLMs practically useful beyond research
  • Sparked the current AI revolution

Architecture

Decoder-only Transformer
 - 96 transformer layers
 - 12,288 hidden dimensions
 - 96 attention heads
 - Trained on 300 billion tokens
 - Estimated $4.6M training cost

Few-Shot Learning Example

import openai

# GPT-3's magic
prompt = """
Translate English to French:

English: "Hello, how are you?"
French: "Bonjour, comment allez-vous?"

English: "Good morning"
French: "Bon matin"

English: "What is your name?"
French:"""

response = openai.Completion.create(
 engine="text-davinci-003", # GPT-3 variant
 prompt=prompt,
 temperature=0.7,
 max_tokens=100
)

print(response.choices[0].text)
# Output

Performance

Task Accuracy Notes
MMLU 71.3% Good for knowledge tasks
HumanEval 48.1% Weaker on code
Few-shot Excellent Core strength

Why It Mattered

Before GPT-3:
- ML required task-specific training
- Fine-tuning on labeled data necessary
- Separate model per task

After GPT-3:
- Few-shot learning works
- One model for many tasks
- Prompt engineering replaces fine-tuning

Real-World Applications

  1. Text Summarization
prompt = """
Summarize this text in one sentence:
The quick brown fox jumps over the lazy dog. 
This sentence contains all letters of the alphabet.
"""
# Result
  1. Question Answering
prompt = """
Q: What is the capital of France?
A: Paris
Q: What is the capital of Germany?
A:"""
# Result
  1. Code Generation
prompt = """
# Python function to calculate factorial
def factorial(n):"""
# Result

Strengths & Weaknesses

Strengths

  • Revolutionary few-shot learning
  • Excellent at diverse tasks
  • Natural language understanding
  • Good generalization

Weaknesses

  • Can hallucinate facts
  • Doesn't learn from instructions perfectly
  • Slower than newer models
  • Expensive for high volume
  • 2,048 token limit (short context)

Current Status (2024)

  • Deprecated (replaced by GPT-3.5, GPT-4)
  • Historical significance remains
  • Available via API for backwards compatibility
  • Paved way for modern LLMs

Legacy

GPT-3 fundamentally changed AI:

  • Proved scale leads to emergent abilities
  • Showed practical applications beyond research
  • Sparked multi-billion dollar AI industry
  • Influenced architecture of all modern LLMs

-

Last Updated: 2026-08-09 Status: Superseded but historically crucial