HTR-MB: Combining Mamba and BiLSTM for Handwritten Text Recognition

Handwritten Text Recognition (HTR) aims to automatically transcribe text from manuscript images. The current state-of-the-art methods leverage Transformers for this task. Nevertheless, Transformers have quadratic complexity w.r.t. sequence length and the non-monotonic nature of self-attention conflicts with the inherently left-to-right, monotonic alignment of HTR. On the other side, Mamba was recently proposed as an alternative to Transformers having linear complexity, monotonic nature and an inference cost that is constant for each token. Even with these features, its applicability on HTR is not trivial. In particular, (a) Mamba was designed to tackle long-range dependencies efficiently while HTR task mostly relies on short-range dependencies to achieve optimal results. Additionally, (b) available datasets are limited in size and require custom regularization techniques, which were mainly designed for Transformers, in order to avoid overfitting. To address this, we propose HTR-MB, a Mamba-based architecture that effectively models short-range dependencies by combining Mamba with a BiLSTM head, which refines the features extracted by Mamba and, at the same time, contributes to regularization through recurrent locality bias. We conduct extensive ablation studies on the proposed architecture finally demonstrating that our method surpasses the current state-of-the-art in terms of Character Error Rate and Word Error Rate across all evaluated datasets (IAM, READ2016, and LAM). Code and pretrained models are available at: https://github.com/FilippoBotti/RNN-HTR