A good explanation of how this insane system came up is given at Video 2. "History of Oxford University by Chris Day (2018)".
As if it weren't enough, there are also the 6 Halls: permanent private hall.
The colleges are controlled by its fellows, a small self-electing body of highly successful scholars, usually in the dozens per college number it seems. Each college also usually has different types of fellows, e.g. see he university college page: https://www.univ.ox.ac.uk/about/college-fellowships/ (archive)