Structure of the complementary series and special representations of the groups O ( n ,1) and U ( n ,1)

This is a survey of several models (including new models) of irreducible complementary series representations and their limits, special representations, for the groups and . These groups, whose geometrical meaning is well known, exhaust the list of simple Lie groups for which the identity representation is not isolated in the space of irreducible unitary representations (that is, which do not have the Kazhdan property) and hence there exist irreducible unitary representations of these groups, so-called `special representations', for which the first cohomology of the group with coefficients in these representations is non-trivial. For technical reasons it is more convenient to consider the groups and , and most of this paper is devoted to the group . The main emphasis is on the so-called commutative models of special and complementary series representations: in these models, the maximal unipotent subgroup is represented by multipliers in the case of , and by the canonical model of the Heisenberg representations in the case of . Earlier, these models were studied only for the group . They are especially important for the realization of non-local representations of current groups, which will be considered elsewhere. Substantial use is made of the `denseness' of the irreducible representations under study for the group : their restrictions to the maximal parabolic subgroup  are equivalent irreducible representations. Conversely, in order to extend an irreducible representation of  to a representation of , it is necessary to determine only one involution. For the group , the situation is similar but slightly more complicated.