cs.CL 2605.22821

Tokenisation via Convex Relaxations

ConvexTok uses convex relaxations to optimize tokenisation, achieving near-optimal compression within 1% at 128k vocabulary, improving BpB significantly.

Jan Tempus, Philip Whittington, Craig W. Schmidt et al.

2026-05-22 467