H-net: End-to-End Hierarchical Sequence Modeling via Dynamic Chunking

About this episode

On this July 15, 2025 collaboration between Carnegie Mellon University and Cartesia AI researchers introduce H-net in the paper "Dynamic Chunking for End-to-End Hierarchical Sequence Modeling". H-Net is a hierarchical, **tokenizer-free** large language model that processes raw data like **bytes** or **DNA sequences** directly. Unlike traditional models that rely on predefined subword chunks, H-Net employs a **dynamic chunking (DC)** mechanism to learn semantically meaningful boundaries end-to-end through a differentiable **smoothing module**. The architecture uses efficient **encoder-decoder** stages, often powered by **Mamba-2**, to compress sequences for a high-capacity main network. This design addresses the inherent flaws of fixed tokenization, such as **multilingual unfairness** and fragility to **textual perturbations**. Experimental results demonstrate that H-Net achieves competitive performance and superior **robustness** compared to standard subword-based Transformers. By enabling **recursive hierarchy**, the model scales effectively across diverse modalities including **text, code, and genomic data**.

H-Net excels at long-context processing through it's **hierarchical architecture** that progressively compresses raw inputs into significantly shorter sequences ($L_S \ll L_0$), allowing the heavy computational work to be performed on compact, high-level abstractions rather than long streams of raw bytes. This efficiency is driven by **Dynamic Chunking** and the integration of **State Space Models (Mamba-2)** in the encoder and decoder layers, which are specifically selected for their ability to handle long, uncompressed sequences with linear computation scaling,. By recursively compressing sequence length, H-Net creates a global structure that mitigates the information retrieval limitations common in long sequences, allowing the model to maintain a logarithmic state size while reasoning over extended contexts.

Sources:

July 15, 2025

Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

https://arxiv.org/pdf/2507.07955

Project tracking general advancements in this space:

https://github.com/zjysteven/Awesome-Byte-LLM