Compiler
Lexer Token Reference
Current token IDs from the active Raz lexer.
This table exists to make the current lexer inspectable. Token IDs are private compiler implementation details until explicitly stabilized; source-language specifications should refer to spellings and semantic roles.
Special and literal tokens
| Kind | ID | Details |
|---|---|---|
| End of file | 0 |
Terminates the token stream. |
| Identifier | 1 |
Also the fallback kind for unrecognized words that are not keywords. |
| Numeric literal | 2 |
Integer or decimal spelling. |
| String literal | 3 |
Double-quoted; selected escapes decoded during lexing. |
| Character literal | 4 |
Single-quoted; currently stored as a numeric character code represented as text. |
| Unknown single character | 99 |
Fallback from operator recognition; diagnostic policy needs specification. |
Keyword tokens
| Spelling | ID |
|---|---|
function |
10 |
struct |
11 |
let |
12 |
return |
13 |
if |
14 |
else |
15 |
while |
16 |
true |
17 |
false |
18 |
new |
19 |
use |
46 |
include |
47 |
unsafe |
48 |
nullptr |
49 |
const |
51 |
define |
52 |
Operator and punctuation tokens
| Spelling | ID |
|---|---|
( / ) |
20 / 21 |
{ / } |
22 / 23 |
;, :, ,,
.
|
24, 25, 26, 27 |
[ / ] |
28 / 29 |
= |
30 |
+, -, *,
/, %, !
|
31–36 |
==, !=, <,
<=, >, >=
|
37–42 |
&&, ||, ->
|
43, 44, 45 |
Known inconsistency: token ID
50 is
not assigned by the lexer, while a parser branch refers to it in an
extern-related check. Do not allocate this ID to a new
feature until that branch is resolved.