python3 microgpt_trace_en.py num docs: 32033 vocab size: 27 num params: 4192 ====================================================================== Showing the computation of training step 1 (learning rate etc. unchanged) ====================================================================== ■ 0. Input: the name used in this step = 'yuheng' Token ids : [26, 24, 20, 7, 4, 13, 6, 26] Input -> target (the next character): pos 0: BOS -> y pos 1: y -> u pos 2: u -> h pos 3: h -> e pos 4: e -> n pos 5: n -> g pos 6: g -> BOS Each row stands for one position (position number:character input at that position). ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 1a. Rows of the token embedding wte (a vector for each character) (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS +0.082 -0.036 -0.120 -0.033 -0.016 +0.096 +0.071 -0.045 -0.075 -0.058 +0.107 -0.012 +0.020 -0.002 +0.192 +0.151 1:y -0.029 -0.026 +0.041 -0.004 -0.058 -0.031 +0.000 +0.046 -0.095 +0.133 +0.052 +0.027 +0.049 -0.027 +0.102 +0.034 2:u +0.123 +0.157 +0.008 -0.061 +0.148 -0.114 -0.020 +0.059 +0.016 +0.023 -0.013 +0.029 +0.085 +0.034 +0.036 -0.004 3:h +0.075 -0.019 -0.077 +0.135 -0.015 +0.135 -0.036 +0.030 +0.038 -0.158 -0.111 -0.008 -0.059 +0.034 -0.015 +0.139 4:e -0.020 +0.053 +0.046 +0.034 +0.023 +0.045 -0.156 +0.148 +0.014 -0.045 -0.007 -0.061 +0.041 -0.072 -0.009 +0.041 5:n -0.119 -0.042 -0.041 -0.015 -0.012 -0.104 +0.164 +0.150 +0.080 -0.011 +0.045 +0.129 +0.018 -0.085 +0.046 +0.105 6:g -0.114 +0.034 -0.114 -0.114 +0.006 +0.118 -0.107 -0.035 +0.040 +0.081 +0.009 +0.081 +0.011 -0.071 +0.035 +0.071 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 1b. Rows of the position embedding wpe (a vector for each position) (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS -0.022 -0.070 -0.181 +0.057 +0.047 -0.027 -0.062 +0.072 +0.118 -0.102 +0.046 +0.104 +0.040 +0.024 -0.111 +0.077 1:y -0.049 +0.058 -0.009 +0.163 -0.117 +0.017 +0.137 +0.079 -0.036 -0.053 +0.004 -0.007 +0.040 -0.087 +0.080 -0.023 2:u +0.010 +0.021 -0.027 -0.083 +0.058 -0.153 +0.114 +0.002 +0.032 -0.076 -0.097 -0.070 +0.144 -0.011 +0.035 +0.025 3:h +0.057 +0.070 +0.046 +0.082 +0.057 +0.048 +0.013 -0.046 +0.013 +0.101 +0.043 +0.093 -0.021 +0.097 -0.158 +0.029 4:e -0.074 -0.012 -0.085 -0.067 +0.063 +0.013 +0.129 +0.032 -0.012 +0.108 +0.058 +0.051 +0.025 -0.122 +0.068 -0.029 5:n +0.154 +0.029 +0.008 -0.052 +0.188 -0.039 +0.043 -0.021 -0.067 -0.084 +0.009 -0.005 +0.060 -0.028 -0.120 -0.144 6:g -0.059 +0.072 +0.107 -0.013 -0.058 +0.013 +0.069 +0.072 +0.010 -0.014 -0.035 +0.106 +0.107 -0.068 -0.032 +0.059 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 1c. RMSNorm of the sum = input x to the block (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS +0.495 -0.873 -2.488 +0.196 +0.261 +0.572 +0.073 +0.217 +0.355 -1.325 +1.265 +0.763 +0.494 +0.180 +0.664 +1.878 1:y -0.739 +0.309 +0.296 +1.496 -1.645 -0.126 +1.294 +1.175 -1.236 +0.760 +0.533 +0.191 +0.837 -1.071 +1.720 +0.106 2:u +1.012 +1.358 -0.145 -1.100 +1.576 -2.042 +0.718 +0.469 +0.361 -0.401 -0.842 -0.316 +1.747 +0.181 +0.540 +0.168 3:h +1.170 +0.449 -0.271 +1.925 +0.373 +1.624 -0.209 -0.141 +0.453 -0.508 -0.604 +0.753 -0.705 +1.160 -1.529 +1.491 4:e -1.138 +0.497 -0.463 -0.391 +1.036 +0.698 -0.315 +2.183 +0.015 +0.766 +0.618 -0.115 +0.792 -2.345 +0.717 +0.142 5:n +0.337 -0.124 -0.321 -0.648 +1.699 -1.382 +2.004 +1.247 +0.128 -0.912 +0.521 +1.203 +0.749 -1.090 -0.712 -0.379 6:g -1.665 +1.018 -0.060 -1.221 -0.503 +1.259 -0.358 +0.356 +0.484 +0.645 -0.244 +1.800 +1.132 -1.338 +0.021 +1.253 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 2a. Query q = Wq·RMSNorm(x) (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS +0.035 +0.395 -0.323 +0.216 +0.083 -0.268 +0.343 -0.550 +0.362 -0.588 +0.366 +0.182 -0.156 -0.581 -0.290 -0.661 1:y +0.132 +0.551 +0.067 +0.017 -0.189 +0.463 -0.589 +0.234 +0.276 +0.275 +0.213 +0.142 -0.308 +0.198 -0.196 -0.235 2:u -0.032 -0.030 +0.082 +0.126 +0.059 -0.607 -0.226 +0.150 -0.070 +0.360 -0.136 +0.264 +0.571 +0.132 +0.344 +0.195 3:h +0.217 +0.075 -0.293 +0.179 -0.085 -0.075 +0.381 -0.129 -0.125 -0.551 +0.408 -0.398 +0.228 -0.386 -0.068 +0.029 4:e +0.143 -0.016 +0.235 -0.193 -0.212 -0.546 -0.475 +0.179 -0.069 +0.234 -0.121 +0.656 +0.150 +0.494 -0.066 -0.447 5:n -0.226 -0.162 -0.062 -0.037 -0.294 -0.396 -0.037 -0.227 +0.120 +0.005 -0.390 +0.313 +0.174 +0.059 -0.128 +0.106 6:g +0.272 -0.135 +0.317 +0.141 -0.019 -0.712 -0.082 +0.198 -0.155 +0.037 -0.127 +0.594 -0.411 +0.694 +0.622 -0.499 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 2b. Key k = Wk·RMSNorm(x) (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS -0.068 -0.047 +0.223 -0.447 +0.054 -0.055 +0.330 -0.325 -0.152 +0.272 +0.302 -0.130 -0.217 -0.083 -0.337 +0.211 1:y +0.128 +0.489 -0.099 +0.665 +0.309 -0.226 +0.241 +0.163 +0.128 -0.047 +0.227 +0.603 +0.750 -0.167 -0.207 -0.180 2:u -0.323 -0.361 +0.576 -0.010 -0.495 +0.021 +0.018 -0.600 +0.531 -0.594 -0.339 -0.352 +0.214 +0.121 +0.110 +0.277 3:h +0.249 -0.491 -0.128 +0.508 +0.223 -0.105 +0.244 +0.333 -0.253 +0.498 +0.351 +0.184 -0.783 -0.038 -0.359 +0.280 4:e -0.334 +0.330 -0.330 -0.313 -0.274 +0.516 +0.053 -0.337 +0.097 +0.191 -0.838 -0.210 +0.279 -0.031 +0.090 -0.019 5:n -0.210 -0.202 +0.028 +0.000 -0.772 +0.396 +0.436 -0.086 +0.487 -0.586 -0.447 +0.024 +0.054 +0.186 +0.063 +0.101 6:g -0.524 -0.455 -0.295 -0.567 -0.081 -0.083 -0.140 +0.238 -0.017 +0.162 -0.257 +0.112 -0.484 +0.170 +0.150 +0.183 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 2c. Value v = Wv·RMSNorm(x) (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS -0.201 -0.116 +0.652 +0.144 +0.106 +0.078 -0.505 +0.018 +0.125 -0.252 -0.444 +0.005 +0.656 -0.496 -0.583 +0.077 1:y -0.396 -0.302 -0.390 -0.181 +0.353 +0.102 -0.759 -0.391 -0.213 -0.368 -0.191 +0.078 -0.010 +0.203 +0.009 -0.151 2:u -0.298 -0.205 -0.042 +0.490 +0.085 +0.430 +0.588 -0.250 -0.284 +0.123 +0.032 +0.140 -0.149 +0.065 +0.265 -0.416 3:h -0.150 +0.242 +0.360 +0.656 -0.178 +0.460 -0.620 +0.439 -0.373 -0.268 -0.130 -0.144 +0.178 -0.170 +0.013 +0.010 4:e +0.153 +0.388 -0.066 -0.023 +0.311 -0.094 -0.392 +0.143 +0.058 -0.037 -0.490 +0.382 +0.544 +0.146 -0.054 +0.179 5:n -0.044 +0.193 -0.061 +0.508 +0.649 +0.266 -0.154 -0.009 +0.025 +0.036 -0.433 +0.032 +0.122 +0.204 +0.167 -0.172 6:g -0.086 +0.650 +0.435 +0.086 -0.245 -0.116 -0.500 +0.290 +0.031 +0.024 -0.357 +0.659 +0.116 +0.412 -0.201 +0.255 * The 16 dimensions are split into groups of 4: d0–d3 are used by head 0, d4–d7 by head 1, ... ---- [Enter] next / [a+Enter] show the rest without stopping ---- --- Head 0 (dimensions d0–d3) --- ■ 3a. Attention scores q·k/√4 (row = query position, col = key position) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS -0.095 · · · · · · 1:y -0.014 +0.146 · · · · · 2:u -0.017 +0.028 +0.034 · · · · 3:h -0.082 +0.106 -0.134 +0.073 · · · 4:e +0.065 -0.071 +0.049 -0.042 -0.035 · · 5:n +0.013 -0.063 +0.048 +0.006 +0.027 +0.039 · 6:g -0.002 +0.016 +0.071 +0.083 -0.142 -0.010 -0.127 ■ 3b. Attention weights softmax(scores) (each row sums to 1) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS 1.000 · · · · · · 1:y 0.460 0.540 · · · · · 2:u 0.323 0.338 0.340 · · · · 3:h 0.231 0.279 0.220 0.270 · · · 4:e 0.215 0.187 0.211 0.193 0.194 · · 5:n 0.167 0.155 0.173 0.166 0.169 0.171 · 6:g 0.144 0.147 0.155 0.157 0.126 0.143 0.127 * '·' marks future positions not yet computed (their keys do not exist yet). ---- [Enter] next / [a+Enter] show the rest without stopping ---- --- Head 1 (dimensions d4–d7) --- ■ 3a. Attention scores q·k/√4 (row = query position, col = key position) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS +0.155 · · · · · · 1:y -0.153 -0.134 · · · · · 2:u -0.043 +0.063 -0.068 · · · · 3:h +0.084 +0.031 +0.062 +0.020 · · · 4:e -0.098 -0.014 -0.011 -0.023 -0.154 · · 5:n +0.034 -0.023 +0.136 -0.054 -0.025 +0.036 · 6:g -0.026 +0.084 -0.063 +0.058 -0.216 -0.160 +0.060 ■ 3b. Attention weights softmax(scores) (each row sums to 1) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS 1.000 · · · · · · 1:y 0.495 0.505 · · · · · 2:u 0.324 0.360 0.316 · · · · 3:h 0.259 0.245 0.253 0.243 · · · 4:e 0.192 0.209 0.210 0.207 0.182 · · 5:n 0.169 0.160 0.187 0.155 0.159 0.170 · 6:g 0.144 0.160 0.139 0.156 0.119 0.126 0.157 * '·' marks future positions not yet computed (their keys do not exist yet). ---- [Enter] next / [a+Enter] show the rest without stopping ---- --- Head 2 (dimensions d8–d11) --- ■ 3a. Attention scores q·k/√4 (row = query position, col = key position) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS -0.064 · · · · · · 1:y +0.039 +0.078 · · · · · 2:u +0.017 +0.051 -0.149 · · · · 3:h +0.022 -0.069 +0.131 -0.086 · · · 4:e -0.024 +0.174 -0.183 +0.106 +0.001 · · 5:n -0.088 +0.058 +0.041 -0.053 +0.137 +0.118 · 6:g -0.041 +0.154 -0.135 +0.061 -0.013 -0.013 +0.054 ■ 3b. Attention weights softmax(scores) (each row sums to 1) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS 1.000 · · · · · · 1:y 0.490 0.510 · · · · · 2:u 0.347 0.359 0.294 · · · · 3:h 0.255 0.233 0.284 0.229 · · · 4:e 0.191 0.233 0.163 0.217 0.196 · · 5:n 0.147 0.170 0.167 0.152 0.184 0.180 · 6:g 0.135 0.164 0.123 0.150 0.139 0.139 0.149 * '·' marks future positions not yet computed (their keys do not exist yet). ---- [Enter] next / [a+Enter] show the rest without stopping ---- --- Head 3 (dimensions d12–d15) --- ■ 3a. Attention scores q·k/√4 (row = query position, col = key position) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS +0.020 · · · · · · 1:y +0.033 -0.091 · · · · · 2:u -0.105 +0.150 +0.115 · · · · 3:h +0.006 +0.122 +0.001 -0.066 · · · 4:e -0.073 +0.062 -0.019 -0.119 +0.014 · · 5:n +0.011 +0.064 +0.030 -0.031 +0.017 +0.011 · 6:g -0.142 -0.231 -0.037 -0.034 -0.035 +0.048 +0.160 ■ 3b. Attention weights softmax(scores) (each row sums to 1) (7 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g 0:BOS 1.000 · · · · · · 1:y 0.531 0.469 · · · · · 2:u 0.283 0.365 0.352 · · · · 3:h 0.247 0.277 0.246 0.230 · · · 4:e 0.191 0.218 0.201 0.182 0.208 · · 5:n 0.166 0.175 0.169 0.159 0.167 0.166 · 6:g 0.128 0.117 0.142 0.143 0.142 0.155 0.173 * '·' marks future positions not yet computed (their keys do not exist yet). ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 4a. Outputs of the 4 heads concatenated (values mixed by the weights) (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS -0.201 -0.116 +0.652 +0.144 +0.106 +0.078 -0.505 +0.018 +0.125 -0.252 -0.444 +0.005 +0.656 -0.496 -0.583 +0.077 1:y -0.306 -0.216 +0.089 -0.032 +0.231 +0.090 -0.633 -0.188 -0.047 -0.311 -0.315 +0.042 +0.344 -0.168 -0.305 -0.030 2:u -0.299 -0.209 +0.064 +0.152 +0.189 +0.198 -0.251 -0.214 -0.117 -0.183 -0.213 +0.071 +0.129 -0.043 -0.068 -0.180 3:h -0.263 -0.091 +0.130 +0.267 +0.093 +0.266 -0.318 -0.048 -0.184 -0.176 -0.178 +0.026 +0.163 -0.089 -0.073 -0.123 4:e -0.179 -0.003 +0.114 +0.222 +0.132 +0.205 -0.332 -0.014 -0.142 -0.179 -0.248 +0.085 +0.238 -0.038 -0.065 -0.063 5:n -0.153 +0.037 +0.079 +0.272 +0.223 +0.211 -0.281 -0.017 -0.107 -0.120 -0.280 +0.091 +0.221 -0.005 -0.030 -0.081 6:g -0.155 +0.104 +0.125 +0.254 +0.136 +0.163 -0.354 +0.035 -0.093 -0.116 -0.289 +0.165 +0.203 +0.069 -0.051 -0.022 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 4b. After multiplying by Wo (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS +0.119 -0.015 -0.068 -0.096 -0.143 -0.161 -0.023 +0.196 -0.078 -0.085 -0.064 +0.100 -0.002 +0.015 -0.144 +0.106 1:y +0.097 +0.020 -0.006 -0.081 -0.042 -0.058 +0.035 +0.140 -0.025 -0.071 -0.035 +0.070 +0.023 +0.033 -0.007 +0.044 2:u +0.045 +0.012 +0.031 -0.008 -0.003 -0.032 -0.025 +0.105 -0.001 +0.006 -0.011 +0.038 +0.038 +0.028 -0.011 +0.026 3:h +0.036 +0.021 -0.005 +0.001 -0.003 -0.031 -0.035 +0.133 -0.031 -0.012 -0.002 +0.040 +0.015 +0.013 -0.036 +0.019 4:e +0.018 +0.009 -0.007 -0.011 -0.016 -0.052 -0.056 +0.138 -0.034 -0.030 -0.025 +0.044 -0.011 +0.027 -0.067 -0.000 5:n +0.004 +0.016 -0.004 -0.005 -0.021 -0.062 -0.074 +0.131 -0.026 -0.016 -0.040 +0.035 -0.001 +0.033 -0.077 -0.012 6:g -0.010 -0.002 -0.006 -0.017 -0.030 -0.069 -0.087 +0.151 -0.044 -0.029 -0.034 +0.043 -0.028 +0.033 -0.107 -0.025 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 4c. Residual connection: 4b + block input x (1c) = output of the attention block (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS +0.614 -0.887 -2.556 +0.100 +0.118 +0.410 +0.050 +0.413 +0.277 -1.410 +1.202 +0.863 +0.492 +0.195 +0.520 +1.984 1:y -0.643 +0.329 +0.290 +1.415 -1.687 -0.183 +1.328 +1.314 -1.260 +0.689 +0.498 +0.261 +0.860 -1.038 +1.712 +0.150 2:u +1.057 +1.370 -0.114 -1.108 +1.573 -2.073 +0.693 +0.574 +0.360 -0.394 -0.853 -0.278 +1.785 +0.209 +0.530 +0.193 3:h +1.206 +0.471 -0.276 +1.926 +0.370 +1.593 -0.244 -0.009 +0.422 -0.520 -0.606 +0.793 -0.690 +1.173 -1.565 +1.510 4:e -1.120 +0.506 -0.470 -0.403 +1.020 +0.646 -0.371 +2.321 -0.019 +0.736 +0.594 -0.070 +0.781 -2.318 +0.650 +0.142 5:n +0.342 -0.108 -0.325 -0.653 +1.679 -1.444 +1.930 +1.378 +0.102 -0.929 +0.481 +1.238 +0.748 -1.057 -0.789 -0.391 6:g -1.675 +1.016 -0.066 -1.238 -0.533 +1.190 -0.445 +0.507 +0.440 +0.616 -0.278 +1.843 +1.104 -1.305 -0.086 +1.228 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 5a. MLP hidden layer ReLU(W1·RMSNorm(x)) (64 dims, so transposed: row = component, col = position) (64 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g h0 0.423 0.404 0.320 · · 0.189 · h1 · · 0.142 · · 0.112 0.294 h2 0.566 0.078 0.152 0.193 0.353 0.450 0.165 h3 0.029 · · 0.392 0.051 · 0.280 h4 · 0.160 0.138 · · 0.063 · h5 · 0.017 0.531 · 0.763 0.666 0.369 h6 · · 0.380 · · 0.136 · h7 · 0.210 0.177 · · 0.402 · h8 0.199 0.549 0.268 · 0.102 0.521 0.138 h9 0.442 0.025 · 0.233 0.268 · 0.159 h10 0.216 · 0.272 · 0.091 · 0.278 h11 0.140 0.008 0.153 · 0.387 · · h12 0.476 · 0.066 0.240 · 0.017 · h13 0.158 · 0.124 0.045 · · · h14 · · · 0.353 · 0.072 · h15 · 0.034 · · 0.372 · 0.043 h16 0.027 0.085 0.219 · 0.491 0.211 · h17 · · 0.231 · 0.191 0.537 0.315 h18 0.175 0.315 0.414 · 0.624 0.293 · h19 0.008 · 0.301 · 0.053 0.236 · h20 · · · 0.748 · · · h21 0.160 0.326 · · · · 0.286 h22 · · 0.738 · 0.282 0.366 0.547 h23 0.052 0.276 0.330 · 0.059 · · h24 · · · · · · 0.225 h25 0.080 0.395 0.047 · 0.524 0.231 0.275 h26 0.340 0.446 0.351 · 0.245 0.250 0.475 h27 · · 0.244 0.233 · · · h28 0.053 · · 0.099 · · · h29 · · · · 0.543 0.023 0.912 h30 · 0.463 · · · · · h31 · 0.077 · 0.066 0.206 0.058 · h32 · · · 0.452 0.161 · 0.337 h33 0.576 0.237 · 0.415 0.461 0.464 0.261 h34 0.307 · · 0.655 0.045 · 0.310 h35 0.024 0.227 · · 0.165 0.618 · h36 · · 0.392 0.153 · 0.031 · h37 · · · 0.085 0.029 · 0.086 h38 0.252 0.364 · 0.195 · · 0.104 h39 · · 0.194 · 0.278 0.207 0.446 h40 · · 0.069 0.416 · · 0.148 h41 · · 0.165 0.004 · · · h42 · · · 0.501 · · · h43 0.338 0.091 0.296 0.172 0.589 0.203 0.049 h44 · · 0.893 0.616 · 0.746 · h45 · · 1.116 · · 0.074 · h46 · · 0.682 · 0.002 0.311 0.219 h47 0.073 · 0.852 · 0.191 0.476 · h48 0.095 · 0.041 0.160 · 0.079 0.042 h49 · · · · · 0.015 · h50 · · · · · 0.044 · h51 · · · · · 0.084 · h52 0.050 0.507 0.435 · 0.034 0.151 · h53 0.262 0.156 · 0.003 · · · h54 0.283 · · · 0.626 0.151 0.664 h55 0.023 · 0.277 0.244 0.216 · · h56 0.063 0.491 · 0.348 · · · h57 0.589 0.381 · 0.166 · 0.093 · h58 · 0.126 · · 0.106 · · h59 0.232 0.526 · · 0.331 0.274 0.112 h60 · 0.019 0.101 · · · · h61 · 0.422 · 0.067 0.127 · 0.255 h62 0.101 0.421 · 0.307 0.578 · 0.238 h63 · 0.291 · · 0.205 0.193 0.238 * '·' marks components set to 0 by ReLU. ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 5b. After multiplying by W2 (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS -0.134 -0.122 +0.040 +0.179 +0.308 -0.046 +0.049 -0.004 -0.124 -0.037 -0.048 +0.119 -0.217 +0.007 +0.030 +0.009 1:y -0.067 +0.045 -0.149 -0.033 +0.056 -0.304 +0.176 -0.075 +0.064 -0.221 -0.106 +0.051 -0.106 -0.085 +0.136 -0.031 2:u +0.112 -0.031 -0.074 -0.228 -0.013 -0.061 -0.296 +0.048 -0.062 -0.139 +0.184 -0.329 +0.015 +0.049 -0.232 +0.145 3:h +0.079 -0.130 -0.094 +0.032 +0.376 +0.077 -0.045 -0.263 -0.101 +0.067 -0.048 +0.159 -0.060 +0.091 -0.064 +0.029 4:e -0.036 +0.056 -0.060 +0.068 +0.267 +0.256 -0.033 -0.181 -0.157 -0.090 -0.126 -0.038 +0.015 -0.010 +0.107 +0.062 5:n +0.048 +0.154 -0.139 -0.128 +0.062 -0.044 +0.211 +0.045 -0.094 -0.100 +0.053 -0.195 -0.126 -0.047 -0.002 +0.015 6:g +0.039 -0.016 -0.180 +0.081 +0.344 +0.044 +0.017 -0.064 -0.065 -0.104 -0.035 -0.092 -0.048 +0.016 +0.113 -0.010 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 5c. Residual connection: 5b + 4c = output of the MLP block (output of the last layer) (7 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 0:BOS +0.480 -1.009 -2.516 +0.279 +0.426 +0.365 +0.099 +0.409 +0.153 -1.447 +1.154 +0.982 +0.274 +0.202 +0.550 +1.994 1:y -0.710 +0.374 +0.141 +1.383 -1.631 -0.488 +1.504 +1.239 -1.196 +0.468 +0.392 +0.311 +0.754 -1.124 +1.849 +0.118 2:u +1.169 +1.338 -0.188 -1.336 +1.560 -2.135 +0.397 +0.622 +0.297 -0.533 -0.669 -0.607 +1.800 +0.259 +0.298 +0.338 3:h +1.286 +0.340 -0.371 +1.958 +0.746 +1.670 -0.289 -0.271 +0.321 -0.453 -0.654 +0.953 -0.750 +1.264 -1.629 +1.539 4:e -1.157 +0.562 -0.529 -0.335 +1.288 +0.902 -0.404 +2.139 -0.176 +0.646 +0.468 -0.108 +0.796 -2.328 +0.757 +0.204 5:n +0.390 +0.046 -0.464 -0.780 +1.740 -1.488 +2.141 +1.423 +0.008 -1.029 +0.534 +1.043 +0.622 -1.104 -0.791 -0.376 6:g -1.636 +1.000 -0.246 -1.157 -0.189 +1.234 -0.428 +0.443 +0.375 +0.512 -0.313 +1.751 +1.056 -1.288 +0.027 +1.218 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 6a. Logits (scores for the next character) (transposed: row = character, col = position) (27 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g a +0.005 +0.195 -0.097 -0.018 +0.242 -0.122 +0.209 b +0.239 +0.076 -0.028 -0.245 +0.392 +0.249 -0.181 c +0.543 -0.061 +0.379 +0.443 +0.122 +0.028 +0.131 d -0.477 +0.373 -0.632 +0.071 +0.271 -0.067 -0.031 e +0.351 +0.075 +0.631 -0.109 +0.516 +0.562 +0.263 f -0.007 -0.399 -0.099 +0.243 -0.087 -0.217 -0.310 g -0.075 -0.152 +0.512 +0.180 +0.169 +0.105 +0.173 h +0.178 -0.229 -0.234 +0.234 -0.165 -0.156 -0.133 i +0.037 +0.095 +0.365 +0.169 -0.146 +0.048 +0.146 j +0.439 -0.361 -0.015 +0.276 -0.188 -0.373 -0.260 k +0.459 -0.229 -0.117 +0.871 +0.051 -0.140 +0.235 l +0.384 -0.217 -0.235 +0.607 -0.460 -0.105 -0.206 m -0.351 -0.846 -0.315 +0.357 -0.606 -0.754 +0.129 n +0.085 +0.150 +0.104 +0.133 +0.329 +0.389 +0.102 o +0.674 +0.066 +0.106 +0.209 +0.210 +0.124 +0.195 p -0.094 -0.024 -0.478 -0.090 +0.292 -0.412 +0.240 q +0.022 +0.511 +0.663 -0.408 +0.326 +0.445 +0.325 r -0.536 +0.263 -0.144 -0.326 -0.032 -0.187 -0.041 s +0.552 -0.378 -0.228 +0.204 -0.232 +0.055 -0.412 t -0.064 -0.160 +0.113 -0.027 +0.155 +0.270 +0.363 u +0.306 +0.060 -0.476 -0.101 -0.237 -0.326 -0.036 v -0.152 +0.186 +0.077 -0.120 -0.245 +0.278 -0.234 w -0.068 -0.100 -0.521 +0.226 -0.080 -0.597 -0.003 x +0.369 +0.079 +0.204 -0.109 +0.163 +0.075 +0.280 y -0.168 -0.063 -0.047 -0.270 +0.222 +0.308 -0.265 z +0.477 -0.227 -0.025 +0.103 +0.781 +0.176 +0.590 BOS +0.037 -0.198 -0.117 +0.097 -0.076 +0.347 +0.069 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 6b. Probabilities softmax(logits) (transposed: row = character, col = position, each column sums to 1) (27 rows x 7 cols) 0:BOS 1:y 2:u 3:h 4:e 5:n 6:g a 0.032 0.046 0.033 0.032 0.042 0.031 0.042 b 0.040 0.041 0.035 0.025 0.049 0.045 0.029 c 0.054 0.036 0.052 0.050 0.038 0.036 0.039 d 0.019 0.055 0.019 0.035 0.044 0.033 0.033 e 0.045 0.041 0.067 0.029 0.056 0.062 0.045 f 0.031 0.025 0.033 0.041 0.030 0.028 0.025 g 0.029 0.032 0.060 0.039 0.039 0.039 0.041 h 0.038 0.030 0.028 0.041 0.028 0.030 0.030 i 0.033 0.042 0.052 0.038 0.029 0.037 0.040 j 0.049 0.026 0.035 0.043 0.028 0.024 0.026 k 0.050 0.030 0.032 0.077 0.035 0.031 0.043 l 0.046 0.030 0.028 0.059 0.021 0.032 0.028 m 0.022 0.016 0.026 0.046 0.018 0.017 0.039 n 0.034 0.044 0.040 0.037 0.046 0.052 0.038 o 0.062 0.040 0.040 0.040 0.041 0.040 0.042 p 0.029 0.037 0.022 0.030 0.045 0.023 0.044 q 0.032 0.063 0.070 0.021 0.046 0.055 0.047 r 0.018 0.049 0.031 0.023 0.032 0.029 0.033 s 0.055 0.026 0.029 0.040 0.026 0.037 0.023 t 0.029 0.032 0.040 0.031 0.039 0.046 0.049 u 0.043 0.040 0.022 0.029 0.026 0.026 0.033 v 0.027 0.046 0.039 0.029 0.026 0.047 0.027 w 0.029 0.034 0.021 0.040 0.031 0.019 0.034 x 0.045 0.041 0.044 0.029 0.039 0.038 0.045 y 0.027 0.036 0.034 0.025 0.042 0.048 0.026 z 0.051 0.030 0.035 0.036 0.073 0.042 0.062 BOS 0.033 0.031 0.032 0.036 0.031 0.050 0.037 ■ 6c. Probability of the correct character and the loss −ln p pos 0: target y p = 0.0266 −ln p = 3.6283 pos 1: target u p = 0.0402 −ln p = 3.2146 pos 2: target h p = 0.0284 −ln p = 3.5608 pos 3: target e p = 0.0290 −ln p = 3.5420 pos 4: target n p = 0.0462 −ln p = 3.0743 pos 5: target g p = 0.0393 −ln p = 3.2375 pos 6: target BOS p = 0.0367 −ln p = 3.3043 Loss (mean) L = 3.3660 (reference: random guessing gives ln 27 = 3.2958) ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 7. Backpropagation loss.backward() is done. Each parameter's .grad now holds ∂L/∂w. ■ 7a. Gradient of Wq ∂L/∂Wq (row = component of q, col = component of the input) (16 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 q0 +0.0003 +0.0001 -0.0001 -0.0005 +0.0005 -0.0001 -0.0003 -0.0003 +0.0004 -0.0002 -0.0003 +0.0000 -0.0000 +0.0003 -0.0005 +0.0001 q1 +0.0005 +0.0004 -0.0002 -0.0014 +0.0011 +0.0002 -0.0014 -0.0010 +0.0011 -0.0003 -0.0008 -0.0002 -0.0002 +0.0010 -0.0010 +0.0003 q2 -0.0006 -0.0001 +0.0002 +0.0008 -0.0014 +0.0004 +0.0002 +0.0002 -0.0006 +0.0005 +0.0003 +0.0001 +0.0000 -0.0003 +0.0008 +0.0001 q3 +0.0012 +0.0005 -0.0005 -0.0028 +0.0028 +0.0001 -0.0023 -0.0018 +0.0022 -0.0010 -0.0015 -0.0002 -0.0004 +0.0017 -0.0024 +0.0005 q4 -0.0005 -0.0005 +0.0002 -0.0007 -0.0004 -0.0008 +0.0001 -0.0002 -0.0003 +0.0002 +0.0004 -0.0005 +0.0000 -0.0003 +0.0007 -0.0009 q5 +0.0001 +0.0001 -0.0000 +0.0002 +0.0000 +0.0002 -0.0001 +0.0000 +0.0001 -0.0000 -0.0001 +0.0001 -0.0001 +0.0001 -0.0001 +0.0002 q6 -0.0003 -0.0001 +0.0001 -0.0002 -0.0003 -0.0000 -0.0001 -0.0001 -0.0001 +0.0002 +0.0001 -0.0001 -0.0000 -0.0001 +0.0002 -0.0002 q7 -0.0004 -0.0005 +0.0002 -0.0008 -0.0005 -0.0009 -0.0000 -0.0004 -0.0003 +0.0002 +0.0003 -0.0007 +0.0000 -0.0001 +0.0008 -0.0010 q8 +0.0002 +0.0001 -0.0001 -0.0000 +0.0003 +0.0001 -0.0001 -0.0001 +0.0002 -0.0001 -0.0002 +0.0001 -0.0000 +0.0002 -0.0003 +0.0002 q9 -0.0002 -0.0002 +0.0001 +0.0000 -0.0002 -0.0002 +0.0002 +0.0001 -0.0002 +0.0001 +0.0002 -0.0001 -0.0000 -0.0002 +0.0003 -0.0003 q10 -0.0005 +0.0000 +0.0001 -0.0001 -0.0005 +0.0004 -0.0004 -0.0001 -0.0001 +0.0003 +0.0000 -0.0000 -0.0000 -0.0002 +0.0003 +0.0001 q11 +0.0000 -0.0002 -0.0000 -0.0002 +0.0001 +0.0003 -0.0005 -0.0003 +0.0002 -0.0001 -0.0000 -0.0000 -0.0004 +0.0002 -0.0004 -0.0000 q12 +0.0005 -0.0004 -0.0001 +0.0005 +0.0001 +0.0005 -0.0003 -0.0005 +0.0002 -0.0003 -0.0001 -0.0001 -0.0008 +0.0007 -0.0008 +0.0001 q13 +0.0000 -0.0001 +0.0000 +0.0002 -0.0001 +0.0001 +0.0000 -0.0000 -0.0000 -0.0000 +0.0000 -0.0000 -0.0001 +0.0001 -0.0000 -0.0000 q14 +0.0001 -0.0002 +0.0000 +0.0004 -0.0002 +0.0003 -0.0001 -0.0002 -0.0000 -0.0001 +0.0000 -0.0000 -0.0004 +0.0003 -0.0003 +0.0000 q15 -0.0001 +0.0000 +0.0000 +0.0000 -0.0001 -0.0001 +0.0002 +0.0002 -0.0001 +0.0001 +0.0001 +0.0000 +0.0001 -0.0002 +0.0002 -0.0001 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 8. Update by Adam (learning rate for this step 0.01000) ■ 8a. Wq before the update (16 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 q0 +0.045 +0.062 -0.039 +0.044 +0.002 +0.117 +0.012 -0.095 -0.016 +0.078 -0.019 -0.030 +0.103 -0.070 -0.012 +0.020 q1 -0.042 -0.090 -0.009 +0.084 -0.104 -0.032 +0.038 +0.142 +0.002 -0.009 +0.018 -0.071 +0.082 +0.125 +0.058 +0.121 q2 -0.093 +0.044 -0.037 -0.049 +0.070 +0.033 -0.002 -0.028 -0.069 +0.148 -0.046 +0.049 +0.046 +0.037 +0.030 -0.085 q3 +0.065 +0.165 -0.101 -0.023 -0.164 +0.029 +0.063 -0.095 +0.006 -0.057 +0.102 -0.116 +0.135 -0.044 -0.176 +0.035 q4 +0.038 -0.012 +0.068 -0.065 +0.027 +0.032 -0.013 -0.116 +0.047 -0.025 +0.005 -0.028 -0.031 +0.012 +0.134 +0.067 q5 -0.014 -0.063 +0.042 +0.112 -0.167 +0.012 +0.072 -0.016 -0.084 -0.131 -0.031 -0.081 -0.132 +0.026 +0.071 -0.121 q6 +0.153 -0.107 -0.046 -0.095 -0.077 +0.012 -0.021 -0.083 +0.011 -0.080 -0.064 +0.019 -0.100 -0.080 -0.113 +0.109 q7 +0.047 +0.131 +0.110 +0.011 -0.032 +0.035 +0.029 -0.058 +0.077 +0.092 -0.032 -0.069 +0.021 -0.110 +0.049 -0.013 q8 +0.055 -0.090 +0.004 -0.060 -0.009 -0.009 +0.122 +0.016 -0.059 +0.008 +0.037 +0.012 -0.114 +0.005 +0.132 +0.115 q9 -0.063 +0.116 +0.025 +0.017 -0.045 -0.170 -0.017 +0.010 +0.079 +0.017 -0.118 -0.047 -0.075 -0.135 +0.091 -0.058 q10 +0.117 -0.135 +0.020 +0.067 -0.126 -0.006 -0.107 +0.132 -0.031 -0.022 -0.082 -0.039 +0.051 +0.033 +0.008 +0.170 q11 -0.083 -0.003 -0.105 -0.117 -0.026 -0.053 +0.091 +0.136 +0.218 +0.142 -0.044 +0.001 -0.046 -0.070 +0.059 +0.077 q12 -0.037 +0.092 -0.015 +0.167 +0.195 -0.139 -0.042 +0.001 +0.146 +0.002 -0.049 -0.121 -0.009 -0.014 -0.020 +0.016 q13 -0.044 +0.125 +0.017 -0.068 -0.087 -0.054 -0.029 +0.047 +0.102 +0.128 -0.094 +0.068 -0.021 -0.163 -0.018 -0.050 q14 +0.017 +0.170 +0.099 -0.145 -0.093 +0.086 +0.027 -0.047 +0.174 +0.014 +0.021 +0.008 +0.140 +0.048 -0.025 -0.016 q15 -0.037 +0.014 +0.190 +0.004 +0.060 -0.078 +0.094 +0.027 +0.058 -0.006 -0.020 -0.061 -0.084 +0.142 -0.041 -0.030 ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 8b. Amount of update (after − before) (16 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 q0 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 -0.0100 +0.0100 -0.0100 q1 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 -0.0100 q2 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 q3 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 -0.0100 q4 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 -0.0100 +0.0100 -0.0100 +0.0100 q5 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 -0.0100 +0.0100 -0.0100 q6 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 +0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 q7 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 +0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 -0.0100 +0.0100 -0.0100 +0.0100 q8 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 -0.0100 +0.0100 -0.0100 q9 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 q10 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 +0.0100 -0.0100 -0.0100 q11 -0.0099 +0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 q12 -0.0100 +0.0100 +0.0100 -0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 -0.0100 q13 -0.0100 +0.0100 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 q14 -0.0100 +0.0100 -0.0100 -0.0100 +0.0100 -0.0100 +0.0100 +0.0100 +0.0100 +0.0100 -0.0100 +0.0100 +0.0100 -0.0100 +0.0100 -0.0100 q15 +0.0100 -0.0100 -0.0100 -0.0100 +0.0100 +0.0100 -0.0100 -0.0100 +0.0100 -0.0100 -0.0100 -0.0100 -0.0100 +0.0100 -0.0100 +0.0100 * In the first step, the bias-corrected value m̂/√v̂ equals the sign of the gradient, so every update is ±learning rate (only the direction matters, not the size of the gradient). ---- [Enter] next / [a+Enter] show the rest without stopping ---- ■ 8c. Wq after the update (16 rows x 16 cols) d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d10 d11 d12 d13 d14 d15 q0 +0.035 +0.052 -0.029 +0.054 -0.008 +0.127 +0.022 -0.085 -0.026 +0.088 -0.009 -0.040 +0.113 -0.080 -0.002 +0.010 q1 -0.052 -0.100 +0.001 +0.094 -0.114 -0.042 +0.048 +0.152 -0.008 +0.001 +0.028 -0.061 +0.092 +0.115 +0.068 +0.111 q2 -0.083 +0.054 -0.047 -0.059 +0.080 +0.023 -0.012 -0.038 -0.059 +0.138 -0.056 +0.039 +0.036 +0.047 +0.020 -0.095 q3 +0.055 +0.155 -0.091 -0.013 -0.174 +0.019 +0.073 -0.085 -0.004 -0.047 +0.112 -0.106 +0.145 -0.054 -0.166 +0.025 q4 +0.048 -0.002 +0.058 -0.055 +0.037 +0.042 -0.023 -0.106 +0.057 -0.035 -0.005 -0.018 -0.041 +0.022 +0.124 +0.077 q5 -0.024 -0.073 +0.052 +0.102 -0.177 +0.002 +0.082 -0.026 -0.094 -0.121 -0.021 -0.091 -0.122 +0.016 +0.081 -0.131 q6 +0.163 -0.097 -0.056 -0.085 -0.067 +0.022 -0.011 -0.073 +0.021 -0.090 -0.074 +0.029 -0.090 -0.070 -0.123 +0.119 q7 +0.057 +0.141 +0.100 +0.021 -0.022 +0.045 +0.039 -0.048 +0.087 +0.082 -0.042 -0.059 +0.011 -0.100 +0.039 -0.003 q8 +0.045 -0.100 +0.014 -0.050 -0.019 -0.019 +0.132 +0.026 -0.069 +0.018 +0.047 +0.002 -0.104 -0.005 +0.142 +0.105 q9 -0.053 +0.126 +0.015 +0.007 -0.035 -0.160 -0.027 -0.000 +0.089 +0.007 -0.128 -0.037 -0.065 -0.125 +0.081 -0.048 q10 +0.127 -0.145 +0.010 +0.077 -0.116 -0.016 -0.097 +0.142 -0.021 -0.032 -0.092 -0.029 +0.061 +0.043 -0.002 +0.160 q11 -0.092 +0.007 -0.095 -0.107 -0.036 -0.063 +0.101 +0.146 +0.208 +0.152 -0.034 +0.011 -0.036 -0.080 +0.069 +0.087 q12 -0.047 +0.102 -0.005 +0.157 +0.185 -0.149 -0.032 +0.011 +0.136 +0.012 -0.039 -0.111 +0.001 -0.024 -0.010 +0.006 q13 -0.054 +0.135 +0.007 -0.078 -0.077 -0.064 -0.039 +0.057 +0.112 +0.138 -0.104 +0.078 -0.011 -0.173 -0.008 -0.040 q14 +0.007 +0.180 +0.089 -0.155 -0.083 +0.076 +0.037 -0.037 +0.184 +0.024 +0.011 +0.018 +0.150 +0.038 -0.015 -0.026 q15 -0.027 +0.004 +0.180 -0.006 +0.070 -0.068 +0.084 +0.017 +0.068 -0.016 -0.030 -0.071 -0.094 +0.152 -0.051 -0.020 This completes one training step. From here on, training continues as in the original program, without display. ---- [Enter] next / [a+Enter] show the rest without stopping ---- step 1000 / 1000 | loss 2.6497 --- inference (new, hallucinated names) --- sample 1: kamon sample 2: ann sample 3: karai sample 4: jaire sample 5: vialan sample 6: karia sample 7: yeran sample 8: anna sample 9: areli sample 10: kaina sample 11: konna sample 12: keylen sample 13: liole sample 14: alerin sample 15: earan sample 16: lenne sample 17: kana sample 18: lara sample 19: alela sample 20: anton