c_lex.pl
The C lexer (milestone 4a): it turns preprocessed C text into a list of tokens, each with its file and line.
Input
The input is preprocessed text. A line marker # N "file" or #line N "file" sets the location of the next line, and #pragma weak and #pragma pack are tokens; any other directive line, such as another #pragma or #ident, is skipped. Comments are skipped too, so hand-written test input works without a preprocessor.
Tokens
A token is tokT, File:Line, with T one of:
| T | Is |
|---|---|
idAtom | an identifier |
kwAtom | a keyword; GNU spellings are folded, so __restrict__ is kwrestrict and __asm__ is kwasm |
intValue, Suffix−Radix | an integer constant: Suffix is none, u, l, ul, ll or ull; Radix is dec, or other for hex and octal, which type differently |
floatChars | a floating constant, its characters as written |
chrValue | a character constant, as an int |
strBytes | a string literal, its bytes without the terminator (u8"..." too, as UTF-8 bytes) |
wstrType, Codes | L"...", u"..." or U"...": Type is the element type, int (wchar_t), ushort (char16_t) or uint (char32_t), and Codes the elements without the 0 |
pAtom | a punctuator, such as p(->) or p<<= |
pragma_weakName, Target | #pragma weak Name, Target none, or #pragma weak Name = Target |
A #pragma pack line is consumed by pack_tokens/2, which c_tokens/3 and c_pp apply to a whole token list: each struct or union keyword under #pragma pack(N) is followed by the attribute __attribute__((__nanofil_pack__(N))), which c_types' su_layout/5 honors. GCC and clang apply the pragma to the definitions after it.
Tokens are the tok(T, File:Line) terms of the C text Chars, read as file File from line 1. Throws nanofil_error/1 on a lexical error.
Tokens are c_tokens/3's tokens of the preprocessed (.i) File, lexed a line at a time, so that a line's characters are garbage once it is lexed: the characters of a whole file, 24 bytes each, would sit beside its tokens (a 31 MB tree-sitter parser's are 750 MB). A line marker sets the next line's location, as in c_tokens/3; a block comment that spans lines, which cc -E never leaves, is lexed with the lines up to its end.
Ts0-Ts are the tokens of line Cs at Loc (with the lines after it while a comment is open); Loc1 is the location after them.
Cs are the characters of S's next line, with its newline; [] at the end of the file.
Tokens are Tokens0 without their pragma_pack(Op) tokens, each struct or union keyword under a #pragma pack(N) followed by the attribute __nanofil_pack__(N). Tokens0 is not copied when it has no pragma (GLib's resource blobs are millions of tokens).
the #pragma pack stack after Op: the current value, then the saved entries s(Label, Value), Label none for an unnamed push. A pop restores the latest entry, or with a label the latest entry of that label, dropping those above it; a pop that finds none changes nothing (clang warns), except that an unnamed pop of an empty stack restores the default, as GCC's.
Op is the pragma_pack operation of #pragma pack(Args), Args its operands as atoms and integers: pack() sets the default, pack(N) sets N, pack(push[, Label][, N]) saves the current value first, pack(pop[, Label]) restores it, and pack(show) changes nothing. N is a power of two up to 16. Fails for any other form, which nanofil refuses.
Lines, blanks and comments
the rest of a comment after its /*; Loc is Loc0 advanced past the comment's newlines.
a directive line after its #; Loc is the location of the line that ends it. A line marker # N "file" flags or #line N "file" says the next line is line N of file. Any other directive is skipped.
pragma weak Name or pragma weak Name = Target after the line's #, up to its newline.
Character classes as ASCII ranges: char_type/2's classes differ between Prolog hosts, and C and IR source only need these.
C is a decimal digit character, '0' to '9'.
Tokens
one token, the T of a tok(T, Loc) term. Wide and u8 prefixes are tried before identifiers, which they would start.
Cs is the longest run of identifier characters (ident_char/1) next in the input.
Numbers
T is the int or float token of the pp-number that starts with the already-read characters Prefix.
T is int(Value, Suffix-Radix) or float(Chars) for the pp-number Chars; anything else is an error.
Value is the number Digits spell in Base.
Suffix names the rest of the pp-number, an integer suffix in either case and order (u, l, ll, ul, ull or none). It fails if the rest is not such a suffix.
Only the characters are checked here; the lowering rejects floats.
Character and string bodies: lists of byte values.
Bytes is the literal body up to the closing Quote, with escapes decoded. The source is read as bytes (main and c_pp read files so), and a narrow literal keeps them as written, as clang does, valid UTF-8 or not (oniguruma's EUC-JP tests); a code point above 255, from text that was not read as bytes, is encoded as UTF-8.
Byte is the value of an escape after its backslash, reduced to 8 bits.
An unknown escape is the character itself, as clang has it with only a warning (C11 6.4.4.4p8 leaves it undefined): btrfs-progs prints "\(".
a universal character name after its backslash.
Wide literals: elements are code points, not UTF-8 bytes.
like lit_chars//3, but Codes are the body's code points: the source's bytes decoded as UTF-8 (a byte that starts no valid sequence stands for itself).
Digits are exactly N hexadecimal digits.
Value is the single element of a wide character constant, as an int; wchar_t is int here.
like wide_char_value/4 for [V|Codes].
Value is the single byte of a character constant, as a (signed) char promoted to int. A multi-character constant is rejected.
the next bytes are the UTF-8 encoding of Code, of two to four bytes.
not overlong
Bytes is the UTF-8 encoding of Code.
Punctuators, longest first.
Spelling is one of C's punctuators of two or three characters (C11 6.4.6), digraphs aside; the one-character ones are punct//1's.
Spelling is a digraph (C11 6.4.6p3), a preprocessing token that stands for [, ], {, } or #.
Keywords, with GNU alternate spellings folded.
Keyword is the kw/1 name of the C or GNU keyword spelled Spelling.
| 736 | auto | auto |
| 737 | break | break |
| 738 | case | case |
| 739 | char | char |
| 740 | const | const |
| 741 | continue | continue |
| 742 | default | default |
| 743 | do | do |
| 744 | double | double |
| 745 | else | else |
| 746 | enum | enum |
| 747 | extern | extern |
| 748 | float | float |
| 749 | for | for |
| 750 | goto | goto |
| 751 | if | if |
| 752 | inline | inline |
| 753 | int | int |
| 754 | long | long |
| 755 | register | register |
| 756 | restrict | restrict |
| 757 | return | return |
| 758 | short | short |
| 759 | signed | signed |
| 760 | sizeof | sizeof |
| 761 | static | static |
| 762 | struct | struct |
| 763 | switch | switch |
| 764 | typedef | typedef |
| 765 | union | union |
| 766 | unsigned | unsigned |
| 767 | void | void |
| 768 | (volatile) | (volatile) |
| 769 | while | while |
| 770 | '_Alignas' | alignas |
| 771 | '_Alignof' | alignof |
| 772 | '_Atomic' | atomic |
| 773 | '_Bool' | bool |
| 774 | '_Complex' | complex |
| 775 | '_Generic' | generic |
| 776 | '_Imaginary' | imaginary |
| 777 | '_Noreturn' | noreturn |
| 778 | '_Static_assert' | static_assert |
| 779 | '_Thread_local' | (thread_local) |
| 780 | '__restrict' | restrict |
| 781 | '__restrict__' | restrict |
| 782 | '__inline' | inline |
| 783 | '__inline__' | inline |
| 784 | '__const' | const |
| 785 | '__const__' | const |
| 786 | '__volatile' | (volatile) |
| 787 | '__volatile__' | (volatile) |
| 788 | '__signed' | signed |
| 789 | '__signed__' | signed |
| 790 | asm | asm |
| 791 | '__asm' | asm |
| 792 | '__asm__' | asm |
| 793 | '__attribute' | attribute |
| 794 | '__attribute__' | attribute |
| 795 | '__extension__' | extension |
| 796 | '__auto_type' | auto_type |
| 797 | '__typeof' | typeof |
| 798 | '__typeof__' | typeof |
| 799 | typeof | typeof |
| 800 | '__alignof' | alignof |
| 801 | '__alignof__' | alignof |
| 802 | '__int128' | int128 |
| 803 | '_Float16' | float16 |
| 804 | '__bf16' | bf16 |
| 805 | '__thread' | (thread_local) |
Tokens are Tokens0 read as C23, where bool, true, false, nullptr, static_assert, alignas, alignof and thread_local are keywords (C23 6.4.1p1); C17 spells the type-like ones _Bool, _Static_assert, ... and has no true, false or nullptr keywords. main applies it when NANOFIL_STD is c23 (nanofil-cc sets it for -std=c23 or gnu23).
throw a nanofil_error/1 located at Loc.