Invertible matrices. Or if you think a bit more generally, an invertible linear map.
Non-invertible are excluded "because" otherwise it would not form a group (every element must have an inverse). This is therefore the largest possible group under matrix multiplication, other matrix multiplication groups being subgroups of it.