In this paper, we propose FrankenMask, a novel framework that allows swapping and rearranging face parts in semantic masks for automatic editing of shape-related facial attributes. This is a novel yet challenging task as substituting face parts in a semantic mask requires to account for possible spatial misalignment and the adaptation of surrounding regions. We obtain such a feature by combining a Transformer encoder to learn the spatial relationships of facial parts, with an encoder–decoder architecture, which reconstructs a complete mask from the composition of local parts. Reconstruction and attribute classification results demonstrate the effective synthesis of facial images, while showing the generation of accurate and plausible facial attributes. Code is available at https://github.com/TFonta/FrankenMask_semantic.

FrankenMask: Manipulating semantic masks with transformers for face parts editing / Fontanini, Tomaso; Ferrari, Claudio; Lisanti, Giuseppe; Galteri, Leonardo; Berretti, Stefano; Bertozzi, Massimo; Prati, Andrea. - In: PATTERN RECOGNITION LETTERS. - ISSN 0167-8655. - (2023). [10.1016/j.patrec.2023.10.010]

FrankenMask: Manipulating semantic masks with transformers for face parts editing

Fontanini, Tomaso
;
Ferrari, Claudio;Bertozzi, Massimo;Prati, Andrea
2023-01-01

Abstract

In this paper, we propose FrankenMask, a novel framework that allows swapping and rearranging face parts in semantic masks for automatic editing of shape-related facial attributes. This is a novel yet challenging task as substituting face parts in a semantic mask requires to account for possible spatial misalignment and the adaptation of surrounding regions. We obtain such a feature by combining a Transformer encoder to learn the spatial relationships of facial parts, with an encoder–decoder architecture, which reconstructs a complete mask from the composition of local parts. Reconstruction and attribute classification results demonstrate the effective synthesis of facial images, while showing the generation of accurate and plausible facial attributes. Code is available at https://github.com/TFonta/FrankenMask_semantic.
2023
FrankenMask: Manipulating semantic masks with transformers for face parts editing / Fontanini, Tomaso; Ferrari, Claudio; Lisanti, Giuseppe; Galteri, Leonardo; Berretti, Stefano; Bertozzi, Massimo; Prati, Andrea. - In: PATTERN RECOGNITION LETTERS. - ISSN 0167-8655. - (2023). [10.1016/j.patrec.2023.10.010]
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11381/2962452
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
social impact