filc_amd64.pl
The Fil-C backend for x86-64: filc_abi.pl lowers checked functions to frame, call and entry products; this module places their words and emits asm.pl lines whose GNU as text links with code clang builds for filnix staging Fil-C 0.687. Checked pointer-memory emission remains here, as does global-data serialization and getter initialization.
Every rule is chosen to be easy to check, not fast; docs/m3.md has them in full.
The frame
Each temporary has an 8-byte slot below %rbp, except a check's proof, which stays in %rax:%rcx for the access fused with it. A p temporary's slot holds its address; its lower lives in a lowers[] entry of the function's filc_frame, which the prologue pushes on thread->top_frame and every return and exception exit pops. Values are never kept in registers across a call or safepoint, so every live capability is always in a slot the GC scans. The prologue zeroes lowers[], so the GC sees no stale lowers. Under --registers=temps and pointers (amd64_regs.pl; pointers is the -O default), scalar w/l homes, and under pointers a pointer's address word, may live in registers: rbx and r12-r15 across calls, and r10 and r11 too otherwise. A lower never does, so every live capability stays in the frame record the GC scans. The register checker refuses any home that overlaps that record or a local object. setjmp functions and --registers=stack (plain's default) keep every home in the frame.
Temporaries whose closed live intervals are disjoint share a slot (frame_slots.pl). A function that calls setjmp is the exception: each of its temporaries gets a slot of its own.
Instructions
- The IR must be checked (ir.pl, pizlonate.pl): each check comes right before the raw access or call it covers, and the two are emitted as one instruction,
fusedCheck, Access: the check's sequence, then the operation. Function identity checks come fromcheck_function, not from inferring that an indirect call needs one. A later scalar access that reuses a heap proof (docs/m7.md) isreusedAccess: the access at the checked pointer, reloaded from its slot, plus the access's offset (a related pointer's distance), without a check. - Pointer loads and stores go through the object's aux array; a pointer store ensures the aux array and runs the store barrier.
flightsupplies its address calculation and lower source separately. Emission preserves neither by inference from aparithmetic operation; pizlonation has already chosen both.allocis a heap allocation.- An edge to
pollLruns a pollcheck: a call of filc_pollcheck_slow when the thread's state asks for one. - Scalar integer and SSE2 instructions share amd64:
compute_op//5. Local mode selects integer immediates before scalar stack caching. - Float bits have no lower. At ABI boundaries they travel in XMM registers, or in generic payload words with zero aux (docs/abi-findings.md).
Calls and entries
Calls use the fast convention: the thread, the callee's lower, then the argument words (a buffered return passes its buffer first). A function that throws pops its frame and returns has_exception = 1, in %al or in the return buffer; a caller that sees it propagates it the same way.
Origins preserve the existing can_throw=0: exception flags propagate, but the runtime will not unwind a C++ exception through this frame.
Every function gets a getter, a function object and a generic entry, so its address can be taken; an exported one also gets the global alias for its signature and a global getter, so clang code can call it directly or indirectly.
Variable conventions
| Variable | Means |
|---|---|
| Ctx | the function context (function_ctx/8) |
| Fn | the amd64 context inside Ctx |
| K | a word class: w, l, p, s or d |
| V | a value, tmpN or constN |
| D | the number of a destination tmpD |
| T | a load type |
| W | a store width |
| S, F | symbols |
| Off | a byte offset |
| Form | a placed return form, reg or memBytes (target_form/2) |
Lines is the asm.pl line list of IR module Module for the Fil-C ABI: each item's code and data, a weak thunk for each function the module calls but does not define, and the GNU-stack note. Throws nanofil_error(Msg) if an item cannot be lowered.
As filc_module_asm/2, with the emission choices of main.pl's options. Emission is emit(Registers, Branches), or just Registers with reference branches. Registers is stack (the reference), temps (validated whole-function register homes, amd64_regs.pl, with validated register operands and frame cells, on a frame with a home per scalar temporary: docs/perf.md items 3, 5a-loads and 3d), pointers (temps with pointer address words as homes too, 5b-2), or limited(R, N) for R on functions of at most N instructions; Branches is reference or selected (branch selection, below). emit(Registers, Branches, Copies) also gives Copies: call (the reference) or inline, the small-copy filter of docs/perf.md item 7a; emit/2 means call. Reports records each item's register result (amd64_regs:regs_asm/6), without driver-side state.
limited(Registers, Limit): Registers (temps or pointers) for a function of at most Limit instructions, and the reference emission for a larger one (main.pl's --registers-limit).
temp_slots/9), so integer homes stay eligible.temps mode's first lowering, with a home per w/l temporary. First is placed(Lines, Report) when it has at most own_frame_limit/1 such homes, else again(Groups, Fused, Ranges) with each temporary's group for the second lowering, from a function-mode (3a) allocation of the first, and the fused blocks and live ranges the second reuses (the groups change only which slots the ranges share); the first's lines are dropped, so the second lowering does not hold them. Homes is temps or pointers, as for placed_lines/7.
own scalar homes (512 bytes) worth a second lowering to share their slots (docs/perf.md 3d).
Calls Emit(Lines) for each part of the lines filc_module_asm/4 gives, in order, so a large module (SQLite) need not hold all its lines.
The only stronger-than-SysV call rule (docs/abi-findings.md). Route calls before allocation/validation; never retarget a checked stream.
Entry rsp is 8 mod 16. Allocate saves and padding before calling the ordinary C helper. These are not GC roots; lower stores stay in place.
Calls Emit(Plan) for the ABI plan (filc_abi:lower_function/4) of each function of the checked Module, in order, as the reference emission lowers it: without its source positions, its checks fused with their accesses, the frame's pools shared. Then it calls Emit(thunks(Thunks)) with the module's weak thunk plans (filc_abi:lower_thunks/2). The Gallina backend's plans are compared with these (rocq/backend).
Complete the ABI product before any instruction selection. Fusion is still target preparation; its separate liveness view preserves indices.
Registers is stack, temps(Homes) or map(Homes), Homes being temps (scalar homes) or pointers (pointer address words too). Groups pairs each temporary in a scalar home with the register the map gave the home, or frame, for temps mode's second lowering; [] for data and the non-allocating modes.
A setjmp function keeps reference selection, branches included.
Temps mode reserves a save slot for each callee-saved register, so the stream the checker compares with has the same frame. Map is temps mode's first lowering when it only needs the register map.
shadow_storage/8): one home each, withtemps mode allocates with register operands and the frame cells (docs/perf.md, 5a-loads): the 8-byte words of lowers[], the scalar and pointer pools, phi shadows, thread, aux, retbuf, and stack-aux roots, as cells(Ds, Others) with the other frame ranges (local storage, wide slots, save slots) the checker keeps them apart from. The header words (parent, origin) are never cells.
home(Disp, Classes) for each scalar-pool slot, with the classes of the temporaries sharing it.
ABI facts (docs/abi-findings.md), at the pin
Symbol is the name under which code finds Role of the function or global F: its implementation (the fast entry) and FI alias for signature Sig, its getter, its function object, or its data object and aux array.
Mem is the operand of Field of the Struct whose address is in register Base. A function object's fields are at its lower.
| 486 | thread | stack_limit | 0 |
| 487 | thread | state | 8 |
| 488 | thread | top_frame | 16 |
| 489 | thread | cc_inline_buffer | 128 |
| 490 | thread | cc_inline_aux_buffer | 384 |
| 491 | thread | cc_outline_buffer | 640 |
| 492 | thread | cc_outline_aux_buffer | 648 |
| 493 | thread | cc_outline_size | 656 |
| 494 | function_object | fast_entry | 0 |
| 495 | function_object | generic_entry | 8 |
| 496 | function_object | signature | 16 |
Bits are the flags Flags in an aux word, whose low 48 bits are the aux pointer (aux_pointer_mask/1) and whose high 16 the flags. The special type is a field of the flags.
the special type field, bits 7-10
Frame layout
Place the already chosen payloads, roots and aliases below %rbp.
Named object storage is not a temporary slot and is never reused. Payload is 16-aligned; aux is a count header followed by lower words.
A long double temporary's wide slot is 16 bytes, 10 of them used.
Slots maps Key to lowers[I0] of the filc_frame at -FrameBase(%rbp).
The function context
function_ctx(+Name, +Ret, +Slots, +Blocks, +Size, +FrameBase, +Selection, -Ctx): Ctx = ctx(Fn, Size, FrameBase, Labels, Selection) is the context of function Name, where Fn = f(Name, Ret, Slots, Blocks) is the context amd64's load//4 and compute_op//4 take. Labels maps each block label to its phis, so an edge finds them without searching Blocks.
Immediate selection: stack (the reference) or local.
Branch selection: reference, or selected(Uses) (branch selection).
Small constant copies: call (the reference) or inline (item 7a).
Key's slot is at -Off(%rbp).
Local labels: A is the label atom in each relation.
A label shared by the whole function, e.g. exc for exception exits.
The label of the block named Label.
A label inside the code of instruction Index (or a jump) in Block.
A is the parts' text, as format's ~w writes them; atomic_list_concat/2 is several times faster than format/3, which was a twentieth of emission for SQLite.
The frame's origin: c (calls, can_catch) or n (pollchecks and checks).
Functions
The lines of a module item, a function or a data object.
A constructor or destructor table entry (ir.pl's init linkage), as clang's Fil-C emits one: a native forwarder in the .init_array or .fini_array section, which hands F's function object to the runtime (filc_defer_or_run_global_ctor or filc_run_global_dtor) to call.
A function: its implementation (prologue, blocks, the shared exception exit and failure blocks), then its function object and origins.
fused_blocks(+Fn, +Params, +Blocks0, -Blocks, -LiveBlocks): Blocks are the checked Blocks0 with each check and the raw access after it, which reads its proven address, fused into fused(Check, Access) at one index, so labels and frame slots number instructions as they did before checks were explicit. Access reads the checked pointer P itself: a proof is a register value (%rax:%rcx after the check) and needs no slot. LiveBlocks have the bare accesses, for liveness and layout, which so keep P live through the access, as the reload after ensure-aux needs. Function proofs fuse with raw_call in the same way: the original pointer and all arguments stay live through the call, and neither proof gets a frame slot. Throws unless the IR is checked and every cycle polls.
Q-P for each heap check of P defining tmp(Q) where Q has one definition in the function and P at most one, counting parameters and phis. Only these may be reused in another block: with strict SSA, reloading P there gives the value that was checked.
the names that occur twice or more.
Proofs are Q-P for the block's heap checks so far, Single those of the function that may be reused in another block. A later scalar access at one of them is reused(Access) at P, with no check, at its offset from P (nonzero for a related pointer, round 3): validation (ir.pl, docs/m7.md) has established that its evidence still holds, which is not repeated here. Its liveness view reads P, so P keeps its rooted slot.
The block's proofs so far are pr(ByQ, ByP): ByQ maps each proof Q to its checked pointer P, and ByP each name N to the Qs once paired with tmp(N), perhaps since redefined. Q has at most one pair, since its definition drops the old one, so a kill of D is the removal of D's pair and of each pair whose pointer is tmp(D), without scanning the block's other proofs (a struct copy has tens of thousands).
a renew of Q1 and the access after it, at P, the replaced check's pointer, which has Q1's checked pointer's address and capability (the validator's same_pointer/3, docs/m7.md round 4 addendum). P1 is not read, so it need not stay live. Fusion and offsets as for a check.
A guarded renewal (docs/m7.md, round 5) fuses as a check does, and keeps its guard G for emission and liveness.
A half check (docs/half.md) fuses as a check does, at P; its source proof Q1 only justifies it and is not read.
A hoisted check (docs/bounds.md) fuses as a check does, and keeps its limit U for emission and liveness.
access_check//2 checks [P, P+Size) for alignment 1, 2, 4 or 8 (aAccess0 is a raw access at tmp(Q) and Access is the same at P.
Atomic pointer accesses are heap-only: no proven_local_access clause.
A touch (docs/vector-access.md) uses the proof and accesses nothing.
A cas, heap only too (escape.pl), reads Expected and New beside P.
Layout sees exactly the former whole-pointer operation at this index. In particular both original p operands survive every internal helper. A guarded access reads its guard too, so G keeps its slot to here.
zreturn: filc_prepare_to_return_with_data(thread, V, no origin) copies V's words into the thread's call buffer and returns their byte count, which %rdx carries out through pop_frame to the generic entry.
The small-copy filter (docs/perf.md item 7a): a copy of N bytes, 1 =< N =< 16, from S to D, done inline when the runtime's memmove would pass its checks and have nothing to do for aux; otherwise a jump to the unchanged call after it. D is in %rax:%rcx and S in %rsi:%rdi; the checks and the copy use %rdx and %r8. Only the call's scratch registers, which nothing allocates, are named.
A jump to Slow unless the pointer P with lower L may be accessed for N bytes (CHECK_BOUNDS_FAST: lower =< P, N =< upper - P) and the header word at -8(L) has none of Mask's bits: the aux pointer (low 48 bits) and, for the destination, READONLY | FREE | SPECIAL (CHECK_WRITE_FAST).
Every load before any store, so an overlapping memmove copies the source as it was.
The call's result is its destination, address and lower.
The zeroing of the N entries of lowers[], so the GC sees no stale lowers.
The store of %rax into lowers[I] for each I in Is.
The store of the c or n origin (origin_label/3) into the filc_frame.
The pop of the filc_frame: thread->top_frame back to its parent.
has_exception = 1, in %al or in the return buffer.
The failure block of function_check//1, if the function calls through a pointer.
The failure block of jump(unreachable)//3, if a block ends in one: filc_error(Message, Origin) with the frame's n origin.
clang's message, so a nanofil panic reads as clang's does.
Word I of a generic call's arguments or results is at thread offset Addr, word I of the inline buffer, with its lower at Lower, word I of the inline aux buffer.
The release of Bytes of pushed stack words, if any.
Blocks and jumps
A block's code is its first: committing frees the choicepoints that would keep a large function's (sqlite3VdbeExec's) garbage alive. Blocks are the ones after it, for branch selection's fallthrough.
The code of the jump that ends block From.
A reached unreachable is a Fil-C safety panic, as clang's is (FilPizlonator's UnreachableInst case): one shared block per function.
The returned value V of class K, in %rdx, and its lower in %rcx for p.
The words of a buffered return, into the buffer at %rax.
Phi copies for the edge to Target, a label or poll(Label), then the pollcheck if Target asks for one, then the jump.
Line is a move that cannot reach a safepoint: no call, poll, helper, trap or jump. Phis of different blocks share shadow slots and roots (filc_abi.pl's shadow_storage/8) because no safepoint is inside an edge's parallel copy; this keeps it so.
Branch selection (docs/perf.md). A block whose last instruction is an integer comparison read only by the block's branch branches on the comparison's flags; a branch whose edge needs no phi copies or poll jumps straight to its block; an edge to the next block falls through. The reference above stores the comparison's boolean, tests it and jumps on both arms. Setjmp functions keep the reference.
reference, or selected(Uses), where Uses counts each temporary's occurrences in Blocks, its definition included.
a last integer comparison that only the branch reads moves into the jump.
The flags hold condition CC exactly when the branch goes to True.
An edge to a block without phis, and without a poll.
The edge's code without its jump when it goes to the next block.
The pollcheck: a call of filc_pollcheck_slow if the thread's state asks for it.
The parallel copy of an edge's phis: first every incoming value into its phi's shadow slots (phi_reads//3), then every shadow into the phi (phi_writes//2).
The incoming value V, of class K, into the shadow slots of phi tmp(D). A shared phi's lower is already in its entry (shared_phi/3).
The second half of the parallel copy: each phi's shadow into the phi.
For a p phi tmp(D), its shadow lower into its lower, unless it is shared.
The lower of a p value: its lowers[] entry, or 0 for a constant.
A p value's address into %rax and lower into %rcx.
The flight pointer in %rax:%rcx into tmp(D) and its lower.
Instructions
An instruction's code is its first, as a block's is (block_once/5): inst//4's clauses leave choicepoints (shared first-argument functors, reg_name/3), which would keep every earlier instruction of a long block's frames alive (libaom: 30k instructions overflowed 1 GB).
the code of instruction Inst, the Indexth of block Block. Instructions go by their op's kind (ir.pl), stores and calls by their width and callee.
A check with shared parts (check_share.pl): with the split bounds under local selection, the null test is left out when an earlier check of the same lower passed, and the upper reused right after one; otherwise the check as above.
A renew (docs/m7.md, round 4): the check's bounds and alignment still hold, so only its temporal part is tested, the free flag, and for a write the read-only one, in the flags byte write_check//2 tests. A failure goes to the same reporter with the same pointer and origin.
A half check (docs/half.md): the check with only one bound tested, the other being known from an earlier check of the object, and the same flags and alignment tests. Its failure goes to the check's label with %rax:%rcx = P, so the runtime reports what the full check would.
A guarded renewal (docs/m7.md, round 5): the flight is loaded once. When the span test G passed, the bounds hold, and only the flags a renew tests and the alignment are tested here; otherwise the full check runs out of line (guard_slow_blocks//2). Both fail to the check's label with %rax:%rcx = P, so the report is the check's.
A hoisted check (docs/bounds.md): the bounds against the limit U that limit/1 computed before the loop, then the free flag, and for a write the read-only one, as a renewal tests them, and the alignment. Every failure goes to the check's label with %rax:%rcx = P, so the report is the full check's.
A reused pointer load needs P's lower in %rcx too, for its aux read.
Check exact logical bounds without manufacturing a heap lower. Offset and size are the stack reporter's arguments. Address formation happens only after the full range/alignment check, even for constant accesses.
The addresses of the access's aux word and payload byte. The check has passed, so an offset outside 0..Limit or misaligned never gets here; any other offset is a constant displacement, which the register checker sees exactly (docs/perf.md item 3c). Otherwise index by %rax.
as inst//4, for tmp(D) =K Op Args with Op of Kind. Integer instructions go to amd64:compute_op//5; the pointer ones are by K, or (copy at p) by the operand.
A span test (docs/m7.md, round 5): 1 when P0's lower is not null and lower =< P0+Lo < P0+Hi =< upper, unsigned, addresses modulo 2^64; else 0. It reads only what a full check reads and never traps.
A limit (docs/bounds.md): P0's bounds for Size-byte accesses in the encoding the shape's limit_check//3 reads. It never traps.
RDX is the snapshot payload; RCX is scratch. EAX's full native int survives restoration, which contains no calls, polls or checked loads.
the code of the integer instruction tmp(D) =K Op Args.
The helper returns payload and auxiliary *addresses*, not a flight pointer. Consume both without a safepoint, then root the loaded lower.
x87 long double (docs/float.md). An x temporary has a 16-byte wide slot, of which fldt and fstpt use the first 10 bytes. Each instruction pushes its operands on the x87 stack, computes and pops the stack empty again, so no value stays in an x87 register across instructions, calls or safepoints. Scalars and constants go through the red zone below %rsp: no call or signal frame can intervene within one instruction's sequence.
tmp(D) =K Op Args, with x87. Operands go in reverse, so %st is the first and %st(1) the second.
Above 2^63, subtract 2^63, convert, and restore bit 63, as SSE2's unsigned qword conversion does (amd64:float_integer//4).
The integer operand in %rax, extended to 64 bits as Op reads it.
fildll read an unsigned l with bit 63 set as negative: add 2^64, exactly, since the 64-bit significand holds every such sum.
Truncate %st toward zero into tmp(D) of class K and pop it.
push the x value V. A constant's 10 bytes go through the red zone, by %rdx, which no raw access holds.
Push the s or d value V, widened exactly.
scalar add or sub.
a copy at K.
Flight's inputs are all read before its destination is written. The second address uses %rdx, not the scalar ALU's %rcx, which holds the explicitly selected lower. No call or safepoint splits the pair.
the code of the raw Access at P, the Indexth instruction of Block, after its check left P in %rax and its lower in %rcx.
the cas(T) of Expected E and New N at the checked address in %rax (lower in %rcx), its old value into tmp(D) at class K (ir.pl). An integer is lock cmpxchg on the address in %rdx, Expected in %rax and New in %rcx; %rax is then old, extended as load(T) would. A pointer is Fil-C's filc_strong_cas_ptr_with_manual_tracking(thread, filc_ptr dest, offset, filc_ptr expected, filc_ptr new), as clang calls it: dest in %rsi/%rdx, offset 0 in %rcx, Expected's pair in %r8/%r9 and New's on the stack, the old pair back in %rax/%rdx. The helper checks again, ensures aux (it may exit), barriers New and boxes the slot; P and New stay rooted in their slots, and flight_store roots the result at once.
%rax is the checked payload address, %rdx its explicit aux word. Neither register is an ordinary root. Only loaded/stored *contents* have heap lowers, and no helper or safepoint splits these operations.
The effect list determines execution; only its caller publishes the result. No-aux jumps to lower_resolve's null arm. Resolving a box reads only its lower: address_read always reads the primary payload.
Fil-C's filc_load_ptr_atomic_with_manual_tracking_outline(filc_ptr): the checked pair in %rdi/%rsi, the result pair in %rax/%rdx. The pair leaves %rax/%rcx before origin_store, which uses %rax. The helper takes no thread and cannot exit; flight_store roots the result at once, as clang's frame slot does (no tracking call).
a scalar load of type T at K into tmp(D). Pointer loads require their explicit effect list.
a store of the pointer V at A with its check, for an instruction whose store is internal (setjmp's jmp_buf).
A pointer store may exit during ensure-aux. Reload both rooted inputs after the helpers, before writing the lower and then the address.
Fil-C's filc_store_ptr_atomic_outline(thread, filc_ptr dest, filc_ptr value): the checked destination in %rsi/%rdx, the value from its rooted slots in %rcx/%r8. The helper checks again, ensures aux (it may exit), barriers the value, boxes the slot and stores the pair atomically, then the address; V stays rooted throughout.
Scalar stores have no aux or barrier effects.
The code of a store of the integer V of width W at %rax.
A call through an already checked function, whose lower is in the aux slot. The fast entry if the signature matches the call site's, else the generic entry with every argument as an 8-byte word at thread+128+8i and its lower at thread+384+8i (docs/abi-findings.md).
The generic result cells, after the product's required-size check.
Arguments, the entry registers and the call itself, for the fast convention in form Form; Callee is direct(Symbol) or indirect (through the aux slot).
The entry registers and the call of the fast entry of Callee. Pop is the stack words' size, which a buffered return's buffer sits above.
has_exception, then the result slots from %rdx:%rcx or the buffer.
the register of each K-Slot word of a returned tuple, in order: integer words in %rdx then %rcx, float words in %xmm0 then %xmm1, as LLVM returns {i1, ...}.
Integer registers first: a float word goes into its slot through %rax.
The store of each register into the matching slot.
The copy of each buffer word into its slot; Slots start at Off(%rsp).
A generic return's words, from the thread at %rsi, into the slots.
For a p word at thread offset Off, its lower into lower(T).
The check that the generic return size in %rdx is at least Need, with a call of filc_cc_rets_check_failure if it is not.
The store of each argument into the thread's generic argument area. Invariant: Args are the arguments from argument I on.
argument cell I from %rax, its lower from %rcx, with the thread in register Thread: in the inline buffers, or for a cell past their 256 bytes in the outline ones (clobbering %rsi), as storeCC writes them.
for an argument packet of Size bytes past the 256 inline ones, make the thread's outline buffers hold the rest (filc_thread_ensure_cc_outline_buffer), before any cell is set.
With a flight pointer in %rax:%rcx, jump to Fail unless it points at a function object. Clobbers %rdx, %rsi and %rdi.
The aux pointer of the object at lower %rcx into %rdx, setting ZF if it has none.
The fast convention as data. A signature is a list of words wf(Class, Source): one per w, l, s or d value and two (address, then lower, both l) per p. typed_locations/5 places each word: integer words in the form's registers, s and d in %xmm0-7, the rest in stack words stack(I) at 8I(%rsp) at the call. Entries, call sites, the generic entry, the variadic adapter and external thunks all read this one placement, so they agree by construction.
NS stack words take Bytes, keeping %rsp 16-aligned at the call.
An integer register word goes straight to its slot; others via %rax.
Adapter sources are operands (thread/snapshot/saved stack), not IR values. Load all stack words first, then registers; only RAX is scratch.
Physical operands for the already flattened adapter sources.
thunk(+Product)//: the lowered weak thunk resolves its symbol through the getter and calls the fast entry when the supplied signature matches, else transports its generic cells and result channels. Every incoming word, integer or XMM, is saved below rbp before the getter can exit or clobber caller-saved registers; rbx holds the thread (r12 the return buffer) and -24(%rbp) the function object. The saves are not roots: until the callee's entry records them, pointer authority is rooted only by the caller (abi-findings.md).
The callee's function object in %rax:%rcx. A weak declaration's getter may be null, and then so is the object: the function check fails, as clang's thunk does.
A thunk copies an integer stack word whole, as a register one: w and l share a signature digit, so one emitted for a 32-bit call also serves another object's 64-bit calls (the linker keeps one weak thunk).
Inline cells go straight to the thread at %rbx; later ones through cell_stores//2 into the outline buffers.
Access checks: k(Size, Write, Align), with the pointer in %rax and its lower in %rcx, which the failure block passes on.
A jump to the kind's failure block unless the pointer may be accessed: lower not 0, lower =< ptr =< upper - Size, the object's flags permitting a write if Write is 1, and ptr 8-aligned if Align is 8.
bounds_check(+Shape, +Selection, +Size, +Fail)//: a jump to Fail unless lower (%rcx) is not 0 and lower =< ptr (%rax) and ptr + Size =< upper, in the shape --bounds selects (docs/bounds.md). tests/bounds_smt.py proves each shape's lines equivalent to that condition.
or_bounds(+Size)//: %rdx = (ptr - lower) | (upper - Size - ptr), after D. J. Bernstein: both terms are small and non-negative exactly when the access is in bounds, also for a freed object (upper = lower) and one shorter than Size. --bounds=sign then branches on the sign, his proposal; it also passes pointers within Size of 2^63 above lower, whose accesses are non-canonical and fault in hardware instead of trapping in Fil-C. --bounds=or tests bits 56 and up instead, which is exact while objects stay below 2^56 bytes (five-level paging's user space): an out-of-bounds term is at least 2^56 or, negative, at most -(2^56 + Size). %rsi and %rdx are scratch.
merged_bounds(+Size, +Fail)//: lower =< ptr =< upper - Size as one unsigned comparison (--bounds=merged, docs/bounds.md), after D. J. Bernstein's suggestion: with T = ptr - lower and M = (upper - lower)
- (Size - 1), the valid T are exactly 0 =< T < M. A pointer below
lower makes T at least 2^64 - 2^48, above any M. M is negative when the object is shorter than Size - 1 bytes (malloc(1) read as 8): the subtraction borrows, and cmovb then makes T = M so that the test fails, where comparing T with upper - Size - lower alone would pass. lower =< upper always, and free sets upper = lower (filc_runtime.c free_impl), so M =< 0 for a freed object. %rsi and %rdx are scratch.
the shape --bounds selects (NANOFIL_BOUNDS).
limit_value(+Shape, +Size, +Done)//: %rdx = the limit of the lower in %rcx for Size-byte accesses (a hoisted check, docs/bounds.md), which limit_check//3 reads in a loop, Done a fresh local label. With L the lower and U the upper now:
| Shape | L not 0 | L = 0 |
|---|---|---|
| split | U - Size + 1, or 0 if that borrows | 0 |
| merged | U - L - Size + 1, or 0 if that borrows | 0 |
| or, sign | U - Size | -1 |
The header is read only when L is not 0. tests/hoist_smt.py proves each pair of limit_value and limit_check equivalent to Fil-C's condition, given that the object is not freed in between or its free flag is set.
limit_check(+Shape, +Write, +Fail)//: a jump to Fail unless the pointer in %rax, with the lower in %rcx, may be accessed by the limit in %rdx (limit_value//3), the object's free flag is clear and, for a write, its read-only flag. The flags byte is read only once the bounds passed, so never at a null lower. %rsi and %rdx are scratch.
%rdx = (ptr - lower) | (limit - ptr), djb's two terms.
half_bounds(+Bound, +Selection, +Size, +Write, +Fail)//: a half check's tests (docs/half.md), with ptr in %rax and lower in %rcx, neither null test nor the known bound. upper: split's upper test, then the read-only test for a write as access_check//2 makes it (a freed object has upper = lower, so this fails for it). lower: lower =< ptr compared signed, since an access below a low object may wrap below 0, then the free flag, and for a write the read-only one, as a renew tests them: the upper the earlier check saw holds until a free. %rdx is scratch. tests/half_smt.py proves both equivalent to Fil-C's check.
The flags a renew tests (renew_mask/2): free, and read-only for a write.
The borrow test of upper - Size. Local selection drops it for Size =< 16: a nonnull lower has its 16-byte header below it and upper >= lower, so upper >= 16 >= Size (docs/perf.md, item 6a). Checked IR may state larger sizes (tests/filc/check-range.ir), which keep it.
Reg = %rax + Off, modulo 2^64, leaving %rax; an offset beyond 32 bits goes through %rsi, a scratch register no temporary is homed in.
for each guarded renewal, the full check it runs when its span test failed, out of line, then back to its access. %rax:%rcx still hold P from its flight load.
The labels of a check kind's failure block and origin.
The distinct check kinds the function's accesses need.
Inst's access needs a check of Kind; fails if Inst does not access memory.
A failure block for each check kind, calling the runtime's reporter.
Function objects: every function gets a getter, a generic entry and a function object, so its address can be taken; exported ones also get the FI alias and a global getter.
Generic-only public entry, with a private fast body. Promotion is the only exiting call before the body roots the hidden snapshot parameter.
the symbol a function object names its fast entry by. A global Impl is preemptible, so the dynamic linker would bind the object's word to the first definition of that name it loaded: every HDF5 plugin netcdf opens with RTLD_GLOBAL defines H5PLget_plugin_info, and each one's object called the first one's. clang's names a local alias, .L<Impl>.local, bound at link time (docs/abi-findings.md, *Function objects name a local alias*).
an alias's getter returns its target's getter's result, so the alias's function or data object is the target's, as Fil-C's (filc_alias_global): it calls the target's getter, directly when the target is local, else through the PLT, with the stack kept 16-aligned.
The getter of a function object FO: its flight pointer, whose lower is its address, in %rax:%rdx.
The function object FO with the entries Fast and Generic (symbols, or 0 for none) and signature Sig. Its header has upper = lower, so it cannot be read or written, and an aux word that points at itself.
An exported function's FI alias of Impl and its global Getter.
A weak function has a weak getter and no FI alias, as clang's: callers elsewhere go through their thunks to whichever definition the linker keeps.
Data objects, as clang lays out globals (docs/abi-findings.md): DO_d = {upper, aux} then the payload rounded up to 8 bytes, a 16-byte cache GP_d, an aux array DA_d if the payload holds pointers, and a getter whose slow path registers the object and runs the relocations that fill in its pointers. A payload aligned to more than 16 has Align - 16 zero bytes before DO_d, so that DO_d + 16 is aligned.
Thread-local data, as clang lays it out (docs/abi-findings.md, *Thread-local variables*): a .tbss word TP_d holding this thread's payload address, null until the thread first asks, and a getter that returns it or calls the slow path GS_d. GS_d allocates the thread's object with filc_allocate_thread_local (which roots it in the thread), copies the initializer's bytes from a template TI_d and runs its relocations, as a global's slow path does, then sets TP_d.
some byte of data Items is not zero, so the fresh (zeroed) object needs a copy of them.
AuxWord is the aux word of a global data object with relocations Rs and other flags Flags: its flags, and its static aux array DA if it holds pointers.
The aux array DA of Size bytes and the relocation table Relocs, if there are relocations Rs.
The getter's call that runs the relocations Rs, if there are any, of data object DO from the table Relocs.
Line is the payload directive of Item at offset Off0, and Rs adds to Rs0 a relocation r(Offset, Symbol, Addend) if Item is a pointer, whose slot stays 0 until the relocation fills it in. Off is the offset after Item.
as payload_item/4 for item(W, Value).
Each relocation is {u64 offset, u32 kind, pad, target}: kind 0 calls a getter, kind 1 a constexpr node {u32 add-immediate = 0, u32 0, getter, u64 addend}.
The relocation table. Invariant: Rs are the relocations from the Ith.
The constexpr node of each relocation with an addend. Invariant: Rs are the relocations from the Ith.
Node labels the constexpr node of relocation I of data object Name.
the return, in form Form, of the generic entry's result words GWs, of classes RetKs and Need bytes, then a jump to Out.
The result words, if RetKs has any, in %rdx:%rcx.
The load of each generic return word, at thread offset Off, into its register.
The copy of each generic return word, at thread offset Src, into the return buffer at Off(%r12).
The generic entry: {i1, i64 ret_size} (thread, callee_lower, args_size). Argument i is 8 bytes at thread+128+8i, with its lower (for p) at thread+384+8i, and so is result word i. It calls the fast entry through callee_lower, with a buffer on the stack for a buffered return.
generic_entry_call(+Form, +Channel, +Packet, +Pop, +RetSize, +Done)//: call the fast entry in placed Form, then transport Channel into the placed Packet and return RetSize in %edx. Pop is the native stack argument size. Exceptions skip all result reads and size production.
The result words from %rdx:%rcx at their thread offsets; a void result is one zero word.
The store of each result register at its thread offset.
The copy of each return buffer word, from Off(%rsp) on, to thread offset Dest.
Static data: function origins (staging's 48-byte layout), origins, and one access-check origin per check kind.
The name string, the c and n function origins with their origins, and the check origins of function Name.
A function origin and the origin (origin_label/3) that points at it.
An access-check origin for each check kind.