whn09 5e36f90015 Neuron table attention: decode only the blocks a decode step's requests reference ... The table path decoded the whole KV pool every layer (160 blocks x 128 slots); the fp8 record decode is integer work on GpSimd and took 6-10 ms a layer (device profile of the 6-row attention graph, bench47). A decode step's requests reference at most ceil(ctx / page) + 1 blocks of a scope each: those blocks are chosen with cumsum + scatter (static shapes), gathered whole (one contiguous run each; a per-slot gather was slower than the full decode) and the indices remapped into them. Sizes are rounded to a power of two (neuronx-cc ISA validation 'same_element_count_t2d' otherwise). Prompt-chunk buckets keep the whole pool. The graph runner sets the bounds (requests, context bucket) per replay. bench50 (512 in, 128 out, TP=64, nextn=5): TPOT 234.1 -> 208.4 ms, step 907 -> 805 ms, per-request tokens-per-step identical to bench41 (3.85, 4.38, 1.98). Tiny cp=2 nextn=5 with a 128-block pool: 9/9 golden tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>		2026-10-25 17:58:11 +08:00
..
__init__.py	initial commit	2026-10-20 12:09:42 +08:00
base.py	Neuron: CP group as a replicated-decode group; MoE all-reduce mode (#28)	2026-10-23 22:46:04 +08:00
cache_engine.py	Neuron: separate state regions, chunk-wide indexer context (issue #26)	2026-10-23 03:46:56 +08:00
cuda_graph_runner.py	initial commit	2026-10-20 12:09:42 +08:00
decode_perf_cache.py	initial commit	2026-10-20 12:09:42 +08:00
dflash_cuda_graph_runner.py	initial commit	2026-10-20 12:09:42 +08:00
model_runner.py	Neuron: sampling flags read once a step; runtime profiling on the driver only; one-lookup embedding	2026-10-25 03:52:47 +08:00
neuron_graph_runner.py	Neuron table attention: decode only the blocks a decode step's requests reference	2026-10-25 17:58:11 +08:00
runtime_config.py	initial commit	2026-10-20 12:09:42 +08:00
worker.py	initial commit	2026-10-20 12:09:42 +08:00
zos_status.py	initial commit	2026-10-20 12:09:42 +08:00