c_lex.pl
The C lexer (milestone 4a): it turns preprocessed C text into a list of tokens, each with its file and line.
Input
The input is preprocessed text. A line marker # N "file" or #line N "file" sets the location of the next line, and #pragma weak and #pragma pack are tokens; any other directive line, such as another #pragma or #ident, is skipped. Comments are skipped too, so hand-written test input works without a preprocessor.
Tokens
A token is tokT, File:Line, with T one of:
| T | Is |
|---|---|
idAtom | an identifier |
kwAtom | a keyword; GNU spellings are folded, so __restrict__ is kwrestrict and __asm__ is kwasm |
intValue, Suffix−Radix | an integer constant: Suffix is none, u, l, ul, ll or ull; Radix is dec, or other for hex and octal, which type differently |
floatChars | a floating constant, its characters as written |
chrValue | a character constant, as an int |
strBytes | a string literal, its bytes without the terminator (u8"..." too, as UTF-8 bytes) |
wstrType, Codes | L"...", u"..." or U"...": Type is the element type, int (wchar_t), ushort (char16_t) or uint (char32_t), and Codes the elements without the 0 |
pAtom | a punctuator, such as p(->) or p<<= |
pragma_weakName, Target | #pragma weak Name, Target none, or #pragma weak Name = Target |
A #pragma pack line is consumed by pack_tokens/2, which c_tokens/3 and c_pp apply to a whole token list: each struct or union keyword under #pragma pack(N) is followed by the attribute ⟪__nanofil_pack__N⟫, which c_types' su_layout/5 honors. GCC and clang apply the pragma to the definitions after it.
Tokens are the tok(T, File:Line) terms of the C text Chars, read as file File from line 1. Throws nanofil_error/1 on a lexical error.
Tokens are c_tokens/3's tokens of the preprocessed (.i) File, lexed a line at a time, so that a line's characters are garbage once it is lexed: the characters of a whole file, 24 bytes each, would sit beside its tokens (a 31 MB tree-sitter parser's are 750 MB). A line marker sets the next line's location, as in c_tokens/3; a block comment that spans lines, which cc -E never leaves, is lexed with the lines up to its end.
Ts0-Ts are the tokens of line Cs at Loc (with the lines after it while a comment is open); Loc1 is the location after them.
Cs are the characters of S's next line, with its newline; [] at the end of the file.
Tokens are Tokens0 without their pragma_pack(Op) tokens, each struct or union keyword under a #pragma pack(N) followed by the attribute __nanofil_pack__(N), and each #pragma weak inside braces moved to the end, where it is a declaration of the unit: GCC applies one in a function body to the file-scope name, as glibc's td_ta_map_lwp2thr writes #pragma weak ps_get_thread_area in a case. Tokens0 is not copied when it has no pragma (GLib's resource blobs are millions of tokens).
Ts is Ts0, at brace depth Depth, without its #pragma weak tokens inside braces, ending in Tail; Hoisted are those tokens, in order.
the #pragma pack stack after Op: the current value, then the saved entries s(Label, Value), Label none for an unnamed push. A pop restores the latest entry, or with a label the latest entry of that label, dropping those above it; a pop that finds none changes nothing (clang warns), except that an unnamed pop of an empty stack restores the default, as GCC's.
Op is the pragma_pack operation of #pragma pack(Args), Args its operands as atoms and integers: pack() sets the default, pack(N) sets N, pack(push[, Label][, N]) saves the current value first, pack(pop[, Label]) restores it, and pack(show) changes nothing. N is a power of two up to 16. Fails for any other form, which nanofil refuses.
Lines, blanks and comments
the rest of a comment after its /*; Loc is Loc0 advanced past the comment's newlines.
a directive line after its #; Loc is the location of the line that ends it. A line marker # N "file" flags or #line N "file" says the next line is line N of file. Any other directive is skipped.
pragma weak Name or pragma weak Name = Target after the line's #, up to its newline.
Character classes as ASCII ranges: char_type/2's classes differ between Prolog hosts, and C and IR source only need these.
C is a decimal digit character, '0' to '9'.
Tokens
one token, the T of a tok(T, Loc) term. Wide and u8 prefixes are tried before identifiers, which they would start.
Cs is the longest run of identifier characters (ident_char/1) next in the input.
Numbers
T is the int or float token of the pp-number that starts with the already-read characters Prefix.
T is int(Value, Suffix-Radix), float(Chars) or, for a floating constant with GNU's imaginary suffix i or j before or after its f or l (1.0iF, glibc's _Complex_I), imag(float(Chars)) without it, for the pp-number Chars; anything else is an error.
Value is the number Digits spell in Base.
Suffix names the rest of the pp-number, an integer suffix in either case and order (u, l, ll, ul, ull or none). It fails if the rest is not such a suffix.
Only the characters are checked here; the lowering rejects floats.
Character and string bodies: lists of byte values.
Bytes is the literal body up to the closing Quote, with escapes decoded. The source is read as bytes (main and c_pp read files so), and a narrow literal keeps them as written, as clang does, valid UTF-8 or not (oniguruma's EUC-JP tests); a code point above 255, from text that was not read as bytes, is encoded as UTF-8.
Byte is the value of an escape after its backslash, reduced to 8 bits.
An unknown escape is the character itself, as clang has it with only a warning (C11 6.4.4.4p8 leaves it undefined): btrfs-progs prints "\(".
a universal character name after its backslash.
Wide literals: elements are code points, not UTF-8 bytes.
like lit_chars//3, but Codes are the body's code points: the source's bytes decoded as UTF-8 (a byte that starts no valid sequence stands for itself).
Digits are exactly N hexadecimal digits.
Value is the single element of a wide character constant, as an int; wchar_t is int here.
like wide_char_value/4 for [V|Codes].
Value is the single byte of a character constant, as a (signed) char promoted to int. A multi-character constant is rejected.
the next bytes are the UTF-8 encoding of Code, of two to four bytes.
not overlong
Bytes is the UTF-8 encoding of Code.
Punctuators, longest first.
Spelling is one of C's punctuators of two or three characters (C11 6.4.6), digraphs aside; the one-character ones are punct//1's.
Spelling is a digraph (C11 6.4.6p3), a preprocessing token that stands for [, ], {, } or #.
Keywords, with GNU alternate spellings folded.
Keyword is the kw/1 name of the C or GNU keyword spelled Spelling.
| 772 | auto | auto |
| 773 | break | break |
| 774 | case | case |
| 775 | char | char |
| 776 | const | const |
| 777 | continue | continue |
| 778 | default | default |
| 779 | do | do |
| 780 | double | double |
| 781 | else | else |
| 782 | enum | enum |
| 783 | extern | extern |
| 784 | float | float |
| 785 | for | for |
| 786 | goto | goto |
| 787 | if | if |
| 788 | inline | inline |
| 789 | int | int |
| 790 | long | long |
| 791 | register | register |
| 792 | restrict | restrict |
| 793 | return | return |
| 794 | short | short |
| 795 | signed | signed |
| 796 | sizeof | sizeof |
| 797 | static | static |
| 798 | struct | struct |
| 799 | switch | switch |
| 800 | typedef | typedef |
| 801 | union | union |
| 802 | unsigned | unsigned |
| 803 | void | void |
| 804 | (volatile) | (volatile) |
| 805 | while | while |
| 806 | '_Alignas' | alignas |
| 807 | '_Alignof' | alignof |
| 808 | '_Atomic' | atomic |
| 809 | '_Bool' | bool |
| 810 | '_Complex' | complex |
| 811 | '_Generic' | generic |
| 812 | '_Imaginary' | imaginary |
| 813 | '__real__' | real |
| 814 | '__real' | real |
| 815 | '__imag__' | imag |
| 816 | '__imag' | imag |
| 817 | '_Noreturn' | noreturn |
| 818 | '_Static_assert' | static_assert |
| 819 | '_Thread_local' | (thread_local) |
| 820 | '__restrict' | restrict |
| 821 | '__restrict__' | restrict |
| 822 | '__inline' | inline |
| 823 | '__inline__' | inline |
| 824 | '__const' | const |
| 825 | '__const__' | const |
| 826 | '__volatile' | (volatile) |
| 827 | '__volatile__' | (volatile) |
| 828 | '__signed' | signed |
| 829 | '__signed__' | signed |
| 830 | asm | asm |
| 831 | '__asm' | asm |
| 832 | '__asm__' | asm |
| 833 | '__attribute' | attribute |
| 834 | '__attribute__' | attribute |
| 835 | '__extension__' | extension |
| 836 | '__auto_type' | auto_type |
| 837 | '__typeof' | typeof |
| 838 | '__typeof__' | typeof |
| 839 | typeof | typeof |
| 840 | '__alignof' | alignof |
| 841 | '__alignof__' | alignof |
| 842 | '__int128' | int128 |
| 843 | '_Float16' | float16 |
| 844 | '__bf16' | bf16 |
| 845 | '__thread' | (thread_local) |
Tokens are Tokens0 read as C23, where bool, true, false, nullptr, static_assert, alignas, alignof and thread_local are keywords (C23 6.4.1p1); C17 spells the type-like ones _Bool, _Static_assert, ... and has no true, false or nullptr keywords. main applies it when NANOFIL_STD is c23 (nanofil-cc sets it for -std=c23 or gnu23).
throw a nanofil_error/1 located at Loc.