The structure of complementary series and special representations

We give a survey of several models of irreducible complementary series representations and their limits, special representations, for the groups SU(n,1) and SO(n,1), including new ones. These groups, whose geometrical meaning is well known, exhaust the list of simple Lie groups for which the identity representation is not isolated in the space of irreducible unitary representations (i.e., which do not have the Kazhdan property) and hence there exist irreducible unitary representations of these groups -- so-called ``special representations'' -- for which the first cohomology of the group with coefficients in these representations is nontrivial. By technical reasons, it is more convenient to consider the groups O(n,1) and U(n,1). Most part of the paper is devoted to the group U(n,1). The main emphasis is on the so-called commutative models of special and complementary series representations: in these models, the maximal unipotent subgroup is represented by multiplicators in the case of O(n,1), and by the canonical model of the Heisenberg representations in the case of U(n,1). Earlier, these models were studied only for the group $SL(2,\RR)$. They are especially important for realization of nonlocal representations of current groups, which will be considered elsewhere. We substantially use the ``density'' of the irreducible representations under study of SO(n,1): their restrictions to the maximal parabolic subgroup $ {P} $ are equivalent irreducible representations. Conversely, in order to extend an irreducible representation of ${P}$ to a representation of SO(n,1), we must additionally define only one involution. For the group U(n,1), the situation is similar but slightly more complicated.