Byte-pair tokenizer

Training stops when the most frequent remaining pair occurs only once. A pair seen a single time is not a pattern, so this text supports 62 merges and no more.

24 merges applied. 235 tokens over 352 characters, a ratio of 1.50 characters per token, from a vocabulary of 54 symbols.

Vocabulary is the 30 distinct characters this text starts with, plus one entry per merge. Characters counts the normalised stream, in which every run of whitespace has become a single ▁ boundary marker.

A production tokenizer trained on billions of characters gets English to roughly four characters per token. A few hundred characters cannot reach that, and the gap is the lesson: the ratio is a property of the corpus the vocabulary was trained on, not of the text you are counting. It is also why a bill quoted in tokens moves when you write in a language, or a codebase, the vocabulary never saw.

1. ▁T
2. o
3. k
4. en
5. i
6. z
7. e
8. r
9. s
10. ▁
11. d
12. o
13. ▁n
14. o
15. t
16. ▁
17. re
18. a
19. d
20. ▁words
21. .
22. ▁T
23. he
24. y
25. ▁
26. re
27. a
28. d
29. ▁p
30. i
31. e
32. ce
33. s
34. ▁o
35. f
36. ▁words
37. ,
38. ▁a
39. n
40. d
41. ▁the
42. ▁p
43. i
44. e
45. ce
46. s
47. ▁a
48. re
49. ▁
50. c
51. h
52. o
53. s
54. en
55. ▁
56. b
57. y
58. ▁co
59. u
60. n
61. t
62. i
63. n
64. g
65. .
66. ▁T
67. he
68. ▁mo
69. st
70. ▁co
71. mmon
72. ▁p
73. a
74. i
75. r
76. ▁o
77. f
78. ▁s
79. y
80. m
81. b
82. o
83. l
84. s
85. ▁
86. b
87. e
88. co
89. m
90. e
91. s
92. ▁o
93. n
94. e
95. ▁s
96. y
97. m
98. b
99. o
100. l
101. ,
102. ▁t
103. h
104. en
105. ▁the
106. ▁n
107. e
108. x
109. t
110. ▁mo
111. st
112. ▁co
113. mmon
114. ▁p
115. a
116. i
117. r
118. ,
119. ▁a
120. n
121. d
122. ▁s
123. o
124. ▁o
125. n
126. ▁
127. u
128. n
129. t
130. i
131. l
132. ▁the
133. ▁
134. v
135. o
136. c
137. a
138. b
139. u
140. l
141. a
142. r
143. y
144. ▁
145. i
146. s
147. ▁
148. f
149. u
150. l
151. l
152. .
153. ▁
154. C
155. o
156. mmon
157. ▁words
158. ▁s
159. u
160. r
161. v
162. i
163. v
164. e
165. ▁w
166. h
167. o
168. l
169. e
170. .
171. ▁
172. U
173. n
174. co
175. mmon
176. ▁words
177. ▁a
178. re
179. ▁s
180. p
181. en
182. t
183. ▁a
184. s
185. ▁s
186. e
187. v
188. e
189. r
190. a
191. l
192. ▁t
193. o
194. k
195. en
196. s
197. ,
198. ▁w
199. h
200. i
201. c
202. h
203. ▁
204. i
205. s
206. ▁w
207. h
208. y
209. ▁a
210. ▁
211. r
212. a
213. re
214. ▁n
215. a
216. m
217. e
218. ▁co
219. st
220. s
221. ▁mo
222. re
223. ▁t
224. h
225. a
226. n
227. ▁a
228. ▁co
229. mmon
230. ▁s
231. en
232. t
233. en
234. ce
235. .