Groups of Congruences and Restriction Matrices