Basically a generalization of a matrix, notably to infinite dimensional spaces like Hilbert space.
Calling such functions matrices is stretching it a bit, since we would need to specify infinitely many rows and columns.
For example, the left side of the time-independent Schrodinger equation is a linear map.