Document Type
Conference Proceeding
Publication Date
9-10-2025
Publisher
Association for Computing Machinery
Abstract
Generative AI workloads and services have started to dominate cloud infrastructure capacities with exploding demand in terms of tokens being processed with each training, finetuning or inference workload beginning and ending with Tokenization. Exponentially increasing tokenization volume presents a unique opportunity to leverage network devices already present on the data path for AI pipelines and offload parts of it such as the tokenizer to programmable dataplanes. The tokenizer traditionally run on general purpose CPUs without any hardware accelerators has lightweight hashing as well as table lookups as majority of its tasks. To eliminate host OS kernel overhead and utilize specialized network flow processors we introduce NICTokenizer, a novel framework to offload the tokenizer to SmartNICs to cut down per token latency, boost throughput and freeing up CPU cycles for more intensive workloads. Initial evaluations show that NICTokenizer outperforms the conventional tokenizer implementations run on the server by lowering the latency by 61.3% and increasing the throughput by 407.62% while maintaining a short tail latency guarantee. In addition, offloading the tokenizer to a SmartNIC frees up CPU clock cycles which could be utilized by other processes.
Recommended Citation
Galvankar, S., Mehta, S., & Choi, S. (2025). NICTokenizer: Tokenizer offload to a SmartNIC. Proceedings of the ACM SIGCOMM 2025 Posters and Demos, ACM SIGCOMM Posters and Demos ’25, 79–81. https://doi.org/10.1145/3744969.3748428
Comments
Open access to this article is funded by Santa Clara University Library.
This work is licensed under Creative Commons Attribution International 4.0.