LLMs · via X
thdxr: one inference rack served 6.77T tokens in a day
Dax Raad shared 24-hour stats from a single inference rack: 6.77 trillion tokens, 1.16s median time to first token and 143 tokens/s, with p95 hurt by one flaky tray.
LLMs · via X
Dax Raad shared 24-hour stats from a single inference rack: 6.77 trillion tokens, 1.16s median time to first token and 143 tokens/s, with p95 hurt by one flaky tray.