Analysis from GPT-5 (Thinking extended):

Here’s a clean way to answer your question using the Epoch simulator post as the anchor, then cross-checking *supply* (revenue/shipments) and *per-GPU FLOPs* by year.

# Bottom line (what year?)

* **If you mean “plausibly doable by a single well-funded lab with hardware actually available to buy,” the first realistic window is ~*****2020–2021***** (A100 era).** That’s when the global flow of datacenter GPUs was big enough that one actor could feasibly assemble the ~50k-GPU cluster needed to deliver **~10²⁶ training FLOP** in ~90 days at high utilization. ([StockAnalysis][1])
* **If you mean “in principle, ignoring cost and supply,”** Epoch’s simulator shows that **2012-era GTX 580s** could reach **~10²⁶ training FLOP in ~3 months**—but only by marshalling **~16 million GTX 580s**, which drastically **exceeds what anyone could have procured**, and the post **doesn’t** claim it was actually feasible in 2012. ([Epoch AI][2])

# What counts as “GPT-4-equivalent” compute?

Epoch’s dataset treats **GPT-4** as the **first model above ~10²⁵ training FLOP** (with credible estimates putting it somewhere in the **10²⁵–10²⁶** range). I’ll use **10²⁶ FLOP** as the conservative “GPT-4-equivalent” target. ([Epoch AI][3])

# How many GPUs would you need (by generation)?

Assuming a 90-day run at ~80% utilization:

| GPU (year)            |         Peak training throughput (approx) |                                                  GPUs to reach 10²⁶ FLOP |
| --------------------- | ----------------------------------------: | -----------------------------------------------------------------------: |
| **GTX 580 (2010–12)** |                        ~1.58 TFLOP/s FP32 | **~10.2M** (Epoch’s simulator quotes **~16M** under its own assumptions) |
| **V100 (2017–19)**    |      **~125 TFLOP/s FP16 (Tensor Cores)** |                                                                **~129k** |
| **A100 (2020–22)**    | **~312 TFLOP/s FP16/BF16 (Tensor Cores)** |                                                                 **~51k** |
| **H100 (2023–25)**    |     **~1000 TFLOP/s BF16 (Tensor Cores)** |                                                                 **~16k** |

Sources for per-GPU throughput: NVIDIA V100 whitepaper (125 TFLOP/s), NVIDIA A100 datasheet (FP16/BF16 312 TFLOP/s), NVIDIA H100 product page (BF16 ~989–1000 TFLOP/s), GTX 580 datasheet (1.58 TFLOP/s). Calculations are simple: **GPUs ≈ (10²⁶) / (throughput · 0.8 · seconds in 90d)**. ([NVIDIA Images][4])

# Did each year have enough **compute in the market** to make that possible?

Below I combine **NVIDIA’s revenue/shipments** with **flops per GPU** to check whether enough hardware *existed* to support a single 90-day 10²⁶-FLOP run.

## 2012 (GTX 580 era)

* **Epoch simulator result:** 10²⁶ FLOP in ~3 months is *theoretically* possible on GTX 580s—but it needs **~16 million GPUs** (they also mention ~$300 per GPU ⇒ ~$5 B just for cards). The blog explicitly models perfect scaling/ideal networks, **not** real-world supply chains. ([Epoch AI][2])
* **Supply reality:** JPR reports **~14.8 M discrete add-in boards shipped in Q2 2012** total (all vendors, all models). Getting **16 M units of a single high-end SKU** at once was fantastical; the post’s language can mislead here, but it **does not** imply 2012 was *actually feasible*. ([Jon Peddie Research][5])

**Verdict for 2012:** *Not practically possible.* There wasn’t remotely enough of the right GPU, networking, memory, or datacenter capacity in one place—despite the simulator’s “physics-only” result.

## 2017–2019 (V100 era)

* **Throughput needed:** ~**129k V100s** for a 90-day 10²⁶-FLOP run.
* **NVIDIA Data Center revenue FY2018:** **~$1.9 B** (Volta ramp). If you (roughly) convert that to V100 units at ~$8k/board you get **~240k V100s** shipped *globally* that year. One actor would need **~54%** of that entire annual flow—still unrealistic, though *in principle* the market had enough *total* V100s to meet the number. ([Market Realist][6])

**Verdict for 2018/19:** *Borderline in aggregate*, but still not something a single lab could likely assemble.

## 2020–2021 (A100 era) — **first realistic window**

* **Throughput needed:** ~**51k A100s** for 10²⁶ FLOP in 90 days.
* **NVIDIA Data Center revenue FY2021:** **~$6.7 B** (TTM around Jan 31, 2021). Using an **A100 ASP of ~$11k–$20k** (SXM/PCIe range), that’s a ballpark **~330k–610k A100-equivalent units** shipped globally across the year. A single 10²⁶ run would require **~8–15%** of that—**finally within reach** for the largest buyers. ([StockAnalysis][1])
* This lines up with what we actually saw soon after: credible reporting puts **GPT-4’s training** on the order of **~20–25k A100s for ~90–100 days** (that’s more consistent with **~10²⁵-scale** runs), with **GPT-4 released March 2023**. ([SemiAnalysis][7])

**Verdict for 2020–21:** **Yes—this is the first period when a single org could realistically source enough datacenter GPUs** to do a **10²⁶-FLOP** run (or a 10²⁵-FLOP run with far fewer).

## 2023–2024 (H100 era) — abundant headroom

* **Throughput needed:** **~16k H100s** for 10²⁶ FLOP.
* **Observed supply:** Meta publicly targeted **~350k H100s** and **~600k H100-equivalents** by end-2024; clearly enough for multiple 10²⁶ runs. ([Engineering at Meta][8])
* **NVIDIA FY2024/25 revenue jumps** (full-year **$60.9 B** in FY2024; **$130.5 B** in FY2025), dominated by **Data Center**, further underscore the supply surge. ([NVIDIA Investor Relations][9])

**Verdict for 2023–24:** Trivially feasible for multiple actors.

---

## Why the 2012 wording can mislead

Epoch’s simulator asks: *“What’s the largest training run **if** you had as many 2012 GPUs as you like and networking scaled ideally?”* It finds **10²⁶ FLOP in ~3 months** with GTX 580s—**but** that requires **~16 M boards**, while quarterly *total* discrete shipments (all vendors/models) were ~**14.8 M**. The post **does not** say a 2012 run was practicable; it’s a theoretical upper bound. ([Epoch AI][2])

---

## A quick “compute-per-year” sanity check (method you asked for)

Two complementary lenses give consistent answers about *when* it first became possible:

1. **Per-GPU math (above table)** → see when a single org could plausibly buy the required GPU count in that year’s market. This points to **A100 era (~2020–21)** for a **10²⁶** run, **V100 era (~2018–19)** for a **10²⁵** run. ([NVIDIA Images][4])

2. **Revenue → units → aggregate capacity (per year):**

   * **FY2018 Data Center ≈ $1.9 B** → ~240k V100s (at ~$8k) existed globally; a single **10²⁶** run would need **~129k** V100s → **>50%** of that year’s flow. Borderline at best. ([Market Realist][6])
   * **FY2021 Data Center ≈ $6.7 B** → **~330k–610k A100s** (at ~$11–20k) existed globally; a single **10²⁶** run needs **~51k** → **~8–15%** share—**plausible** for the largest buyer. ([StockAnalysis][1])
   * **FY2024–25** revenues show the floodgates fully open (ample H100). ([NVIDIA Investor Relations][9])

---

## Takeaways

* **First practically feasible year for a GPT-4-scale (10²⁶ FLOP) training run:** **~2020–2021** (A100 era).
* **2012** is **not** a practical answer (even though the simulator shows a physics-only path with ~16 M GTX 580s). Your interpretation is correct: the post does **not** imply feasibility in 2012. ([Epoch AI][2])

---

### Citations

* Epoch simulator blog (GTX 580 → **10²⁶ FLOP**, ~16 M GPUs, ~$5B at ~$300 each). ([Epoch AI][2])
* Epoch data insight (GPT-4 = first model ≥10²⁵ FLOP, released Mar 2023). ([Epoch AI][3])
* GPU throughputs: **V100** (125 TFLOP/s FP16) whitepaper; **A100** datasheet (312 TFLOP/s FP16/BF16); **H100** product page (~1 PFLOP/s BF16); **GTX 580** datasheet (1.58 TFLOP/s FP32). ([NVIDIA Images][4])
* 2012 shipments context (AIB shipments **Q2 2012 ≈ 14.8 M** total). ([Jon Peddie Research][5])
* NVIDIA revenue ramps: **FY2018 total $9.71 B**; **FY2024 total $60.9 B**; **FY2025 total $130.5 B**. ([NVIDIA Newsroom][10])
* NVIDIA Data Center revenue history incl. **FY2021 ≈ $6.7 B** (TTM): shows the inflection where a single-actor 10²⁶ run becomes plausible. ([StockAnalysis][1])
* A100/H100 pricing ranges for ASP sanity checks. ([Cyfuture Cloud][11])
* Meta public plan for **~350k H100** and **~600k H100-equivalents** by end-2024. ([Engineering at Meta][8])
* Widely reported (non-official) GPT-4 training scale (~20–25k A100s for ~90–100 days). ([SemiAnalysis][7])

If you want, I can drop this into a small spreadsheet with the formulas so you can tweak the utilization (e.g., 70% vs 85%) or the target FLOPs (10²⁵ vs 10²⁶) and see how the “first feasible year” moves.

[1]: https://stockanalysis.com/stocks/nvda/metrics/data-center-revenue/?utm_source=chatgpt.com "NVIDIA (NVDA) Data Center Revenue - Stock Analysis"
[2]: https://epoch.ai/blog/introducing-the-distributed-training-interactive-simulator?utm_source=chatgpt.com "Introducing the LLM Distributed Training Simulator | Epoch AI"
[3]: https://epoch.ai/data-insights/models-over-1e25-flop?utm_source=chatgpt.com "Over 30 AI models have been trained at the scale of GPT-4 - epoch.ai"
[4]: https://images.nvidia.com/content/volta-architecture/pdf/volta-architecture-whitepaper.pdf?utm_source=chatgpt.com "NVIDIA TESLA V100 GPU ARCHITECTURE"
[5]: https://www.jonpeddie.com/news/graphics-add-in-board-shipments-seasonally-down-from-last-quarter/?utm_source=chatgpt.com "Graphics Add-in Board Shipments seasonally down from Last Quarter – Jon ..."
[6]: https://marketrealist.com/2018/05/data-center-a-50-billion-market-for-nvidia/?utm_source=chatgpt.com "Data Center: A $50 Billion Market for NVIDIA - Market Realist"
[7]: https://semianalysis.com/2023/07/10/gpt-4-architecture-infrastructure/?utm_source=chatgpt.com "GPT-4 Architecture, Infrastructure, Training Dataset, Costs, Vision ..."
[8]: https://engineering.fb.com/2024/03/12/data-center-engineering/building-metas-genai-infrastructure/?utm_source=chatgpt.com "Building Meta’s GenAI Infrastructure - Engineering at Meta"
[9]: https://investor.nvidia.com/news/press-release-details/2024/NVIDIA-Announces-Financial-Results-for-Fourth-Quarter-and-Fiscal-2024/?utm_source=chatgpt.com "NVIDIA Announces Financial Results for Fourth Quarter and Fiscal 2024"
[10]: https://nvidianews.nvidia.com/news/nvidia-announces-financial-results-for-fourth-quarter-and-fiscal-2018?utm_source=chatgpt.com "NVIDIA Announces Financial Results for Fourth Quarter and Fiscal 2018"
[11]: https://cyfuture.cloud/kb/gpu/a100-gpu-cost-guide-for-ai-training-and-scientific-workloads?utm_source=chatgpt.com "A100 GPU Cost Guide for AI Training and Scientific Workloads"