One can still define a map by taking the fractional part of the matrix-vector product, but the resulting map won't be continuous (with respect to the topology of the torus). In addition, if one wants the map be a homeomorphism (continuous with continuous inverse) then the determinant must have absolute value 1.

> It says that the underlying action is on the torus (R/Z)^2. If the entries of the matrix are not integers, do we have a well defined action on the torus? It seems to me that the answer is no because Z^2 would not be invariant by the action.

Ah, good point.