- DFlash._greedy_drafts: post norm, LM head, Markov bias, confidence head and
the argmax of the 5 draft steps as one region graph when every row is greedy
with no top-p/top-k filter.
- DEEPINFER_NEURON_HOST_EMBED defaults off: in torch-neuronx every host<->device
copy drains the stream, and the host lookup (D2H ids, H2D rows) left the cores
idle for 50-100 ms gaps between the draft graphs (runtime trace bench36).
- Eager device embedding lookups are one F.embedding with a validity mask (no
host scalar operand).
- bench34 / bench37 (512 in, 128 out, TP=64, nextn=5): TPOT 268.2 -> 264.0 ->
234.7 ms, step 1025 -> 907 ms, TTFT 5.44 -> 5.01 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- DFlash forward_drafts per layer and update_main_cache (projection, then each
layer's cache write) run as region graphs: eagerly the fp8 cast of the
projection norm does not compile.
- neuronx-cc rejects the sparse decode attention at 6 rows (a DSpark step with
5 drafts; ISA validation 'same_element_count_t2d'), 4 and 8 rows compile:
multi-row decode steps pad to a power of two with empty rows.
- bench_neuron.py --nextn: DSpark drafts, reports tokens per decode step.
- Tiny checkpoint on device, cp=2, free-running greedy with drafts: nextn=3,
5, 7 all 9/9 golden tokens (short), nextn=5 long 9/9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- fused_sample reject mode for greedy rows (accept iff the draft is the
argmax), get_accept_result / finalize_verification (static final rows),
store_mtp_info (static pool writes, slot 0 kept), confidence_head_fused,
dflash_input_store, get_fused_dflash_get_input: the frontend refs with the
kernels' argument lists.
- CP followers run the draft forward (its MoE has collectives); the driver's
accepted offsets reach them through one all-reduce on Neuron (no eager
broadcast), the request first rows through the graph metadata.
- run_deepinfer_tiny.py --nextn: free-running greedy with drafts. Host tiny,
nextn=5, eager: short 9/9 golden tokens; long matches up to its known near
tie at step 5.
- bench17: the two-level MoE all-reduce does not move TPOT (433 ms).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
