microgpt

Learning AI from Scratch, Part 1, Edited by Yoshio Okamoto / Written by Claude (Anthropic) September 28, 2026

Part 2 is here.

1. Introduction

Amazed by the progress of LLMs (large language models), I have timidly begun studying them, hoping to put them to use in my work in education.

The natural path would be Machine Learning ⇒ Deep Learning ⇒ LLMs, but I have started by sampling whatever catches my interest.
There are many references, but these are the series I recommend:

(Sorry in Japanese sites)
The YouTube video series by 3Blue1BrownJapan.
The deep learning video series by Yobinori

In particular, I recommend Chapter5, Chapter6 and Chapter7 of the former.

2. Running microgpt in a Python environment

Compared with the more full-scale GPT-2 described in the next chapter, microgpt is a Python program of about 200 lines, released by Andrej Karpathy in February 2026, that trains and runs a GPT. It uses only the standard libraries os, math and random, with neither NumPy nor PyTorch. Loading the data, the tokenizer, automatic differentiation, the Transformer itself, optimization, and text generation are all in one file. (From the document Claude made for me,11_microgpt_en.pdf; click the link below to read the PDF.)

https://seagull.stars.ne.jp/2026_LLM/microgpt/11_microgpt_en.pdf

The document says the following.

1. microgpt is a GPT written in about 200 lines using only the standard library: data, tokenizer, automatic differentiation, Transformer, Adam, and generation.
2. Automatic differentiation breaks every calculation into basic operations with known local derivatives, records them, and accumulates gradients from the output side by the chain rule. It performs the backpropagation of Chapter 04 mechanically, one scalar at a time.
3. The model processes one token at a time and appends keys and values to lists. This makes a causal mask unnecessary, and at the same time takes the form of a KV cache.
4. For each name, it computes the cross-entropy loss and updates the parameters with Adam. Over 1000 steps the loss falls from ln 27 ≈ 3.30 to about 2.3, and random strings turn into name-like strings.
5. The differences from real LLMs are mainly scale and how the computation is done; the framework of the model and the principle of learning are the same.

For setting up the Python environment, see the next chapter.

- Where to get the program and the data

Program: https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95

Example data: https://raw.githubusercontent.com/karpathy/makemore/988aa59/names.txt
This URL is written inside the program above.

Author's explanation: https://karpathy.github.io/2026/02/12/microgpt/

Reference: microgptを勉強する：１

0. What does this program do?

It learns, from the input text of 32,033 English names (input.txt), which letters each letter tends to connect to. In other words, it studies over and over how letters are joined in names, and stores those features as the weights of matrices (numerical data); this is the "learning". Then it combines what it has learned to output names that might plausibly exist. That is all the program does.
(In a real LLM, this massive learning produces what is called a model — GPT-3, for example, adjusts 175 billion parameters — and the model is then used in a system that answers our input in the form of a chat.) This program shows the process of making such a model, and then uses what it has learned to generate names.

The input file input.txt looks like this:

emma
olivia
ava
isabella
sophia
charlotte
mia
amelia
harper
evelyn
abigail
emily
elizabeth
mila
ella
avery
sofia
camila

(omitted)

zyheem
zykeem
zylas
zyran
zyrie
zyron
zzyzx

Near the end, I can't help wondering whether such names really exist.

<Running it and the results>

Run it from a terminal as shown below (an example of a terminal on my Linux Mint). On Windows, search for cmd and run it; a black command terminal window for Windows will appear.
Type python microgpt.py and press Enter to run it.

The picture below shows the terminal in my Linux Mint environment, but the same processing takes place in a Command Prompt terminal on Windows.
On both Linux and Windows, however, you need to make a directory (folder) to run it in, and put the program itself (microgpt.py) and the data input.txt there.
You move to that directory with the cd (change directory) command (this makes the folder containing the program and data the current directory).
There are many guide sites on setting up an environment for running these Python programs, so please refer to them.

After a few minutes of reading input.txt and computing the weight matrices, it outputs, as the result, 20 generated names like those shown above. In other words, it has produced words that could be names.
If you change the kind of data, it should be able to do the same kind of calculation.

<A version that shows the internal calculations>

Encouraged by this, I had Claude make a version that prints every internal calculation step by step (click either link; both are the same file).

microgpt_trace_en_v2.py

https://seagull.stars.ne.jp/2026_LLM/microgpt/Python/microgpt_trace_en_v2.py

Manual
https://seagull.stars.ne.jp/2026_LLM/microgpt/microgpt_trace_en_v2.py-User Guide.pdf

Run it from a terminal as shown below. On Windows, search for cmd and run it; a black command terminal window for Windows will appear.

In that terminal,
each time you press Enter, the processing proceeds one stage and the matrix variables are printed. This makes it possible to infer what kind of calculations go on inside the AI.
As for the initial values of each matrix, according to Claude:

Every element of the weight matrices is set, one by one, to a random number from a normal distribution with mean 0 and standard deviation 0.08 (random.gauss(0, 0.08)). Most of the values (about 95%) fall in the range −0.16 to +0.16.

This seems to be the starting point of the calculation. It looks as though you have to press Enter quite a few times before you see any change, so I would like to improve that in the future.

Actually, there is more:

With this, all the variables have finally been computed and Step 1 is finished; the program then enters the next loop of calculation.

The complete console output is collected here:

https://seagull.stars.ne.jp/2026_LLM/microgpt/1step_en.txt

I would like to consider an even more detailed explanatory version later.

As for the role of each parameter in the calculation:

Two kinds of "weights"

	The matrices W_Q, W_K, W_V, W_O, the MLP's W₁, W₂, etc.	The attention weights a_ij
What they are	Parameters of the model	Coefficients computed from the input
When they are set	Set when the model is made (during training), then fixed	Computed every time, for each chat context
Do they change with the input?	No	Yes

In other words,

The matrices themselves have already been fixed by gradient descent on an enormous amount of text when the LLM was built, and they do not change at all during a chat.
What is computed each time during a chat is the result of passing the input through those fixed matrices: vectors such as q_i = W_Qx_i, and the a_ij obtained from them. by Claude

3. Trying GPT-2 on a PC. First, setting up the Python environment (this was my first attempt; I plan to put it on a separate page later)

Even after watching the introductory videos and the like, the content is hard! The Transformer and Attention in particular leave me puzzled no matter how many times I watch. As someone involved in science education, it is very frustrating to use something every day without knowing how it works or what is inside. So I started installing simple models of these LLMs on my own PC, to check part of their calculations for myself.

Of course, doing this alone would be very hard, so I have been working through it in discussion with Claude (Anthropic), my ever-reliable partner. Here I introduce a little of that.

Setting up the Python environment. (Please refer to the many guide sites for setting up Python on your own OS.)

I already have a Python environment set up on Linux Mint, but on Windows too you should be able to install it by following one of the many guide sites. (Unlike Chapter 2, this needs dedicated libraries, which must be installed with the command below.)

If the basics are already installed, the additional libraries needed to run the Python code can be installed with

pip install torch transformers numpy

On Linux open a terminal, and on Windows open cmd — that is, a black command-line window — then copy and paste the command above and run it.

Now the GPT-2 Python scripts will run. Next, we move on to the individual Python scripts to run.

(Under construction below)