Discrete groups of automorphisms of the unit ball
of
C
2
and their limit sets
Anders Södergren
U.U.D.M. Project Report 2005:5
Examensarbete i matematik, 20 poäng Handledare och examinator: Burglind Juhl-Jöricke
Juni 2005
AND THEIR LIMIT SETS
ANDERS S ¨ODERGREN
ABSTRACT. We summarize and compare known results on iteration of rational func-tions on the Riemann sphere leaving a disc invariant and on Fuchsian groups. After summarizing prerequisites on automorphisms of the unit ball in Cnwe study groups of such automorphisms. We give some sufficient conditions for discreteness of such groups and prove that in the considered case the closure of the orbit of the fixed points of the generating elements is a Cantor set. We prepare the study of the limit set. Our conjecture is that in the case mentioned above the limit set itself is a Cantor set.
Date: May 25,2005.
CONTENTS
Introduction 3
1. Background and Motivation 3 1.1. Complex Dynamics and Blaschke Products 3
1.2. Discontinuous Groups 9
2. Automorphisms of Bn 12
2.1. Description of the Automorphisms in Bn(n > 1) 12
2.2. Automorphisms of B2leaving some complex line through zero invariant 15
3. Discrete Subgroups of Aut(B2)and their Limit sets 16
3.1. A useful lemma 16
3.2. A particular example 17
3.3. Concluding remarks 22
Acknowledgements 22
INTRODUCTION
Our goal is to study complex dynamics on the unit ball Bn in Cn, n > 1. By a theorem of H. Alexander there are no rational mappings of the projective space Pn which leave Bn and its boundary ∂Bn invariant except automorphisms of the ball.
The dynamics of the iterates of a single automorphism is not interesting. We will study subgroups of automorphisms. Our aim is to give some sufficient conditions for discreteness of subgroups of automorphisms of the ball and to prepare the study of limit sets of discrete groups. The respective theory is developed in the case of real hyperbolic spaces (with constant negative curvature) of arbitrary dimension. Only few results are known for the complex case (see [2] where necessary conditions for discreteness are given). In the complex case curvature is varying negative and the theory gets more complicated. However, we expect that the complex structure of the ball Bn, respectively the contact structure of its boundary, will give interesting new
effects.
We start to summarize known results for complex dimension one concerning ra-tional iteration (iteration of rara-tional mappings of degree greater than one leaving the unit disc and the unit circle invariant) and Fuchsian groups. These results are ex-plained in the literature. Sometimes we are working out a sketchy explanation in full detail. Some new results concerning the case of the unit ball B2 is presented here. In particular we study the group generated by the mappings
ϕ1(z1, z2) = z1− a 1 − az1 ,(1 − |a| 2)1/2z 2 1 − az1 and ϕ2(z1, z2) = (1 − |b|2)1/2z1 1 − bz2 , z2− b 1 − bz2
where a, b ∈] − 1, 1[ and give sufficient conditions for discreteness of this group. Fur-thermore we show that the closure of the orbit of the fixed points of ϕ1 and ϕ2 is a
Cantor set. We conjecture that the limit set itself is also a Cantor set. 1. BACKGROUND ANDMOTIVATION
1.1. Complex Dynamics and Blaschke Products. Complex dynamics is the study of iteration of analytic functions. Thus we need a nice way of denoting the n:th iterate of a function f . In this paper we will use the notation f1 = f and fn = fn−1◦ f for n ≥ 2. We will be interested in analytic functions on the extended complex plane, b
C. As a set bC = C ∪ {∞} and the complex structure near infinity on bC is induced by f (z) = 1z.
degR := max(degP, degQ), where degP and degQ are the ordinary polynomial degrees of P and Q.
Remark 1.1.2. It can be shown that the rational maps on bC described above are precisely the analytical mappings of bC into itself.
The dynamics of rational maps R of degree one, i.e. M ¨obius transforms, are simple and no advanced structure arises. Nevertheless we will have a look at this situation since we will need it later on. It turns out that there are two cases.
First we consider the case that R has a single fixed point ζ. If ζ = ∞ then R(z) = z + cfor some nonzero c ∈ C and obviously limn→∞Rn(z) = ∞for all z ∈ C. If ζ ∈ C
we conjugate R with the mapping φ(z) = z−ζ1 and get a function ˜Rwith ∞ as its only fixed point. Hence R is conjugate to ˜Rwhich dynamics we have already discussed above and it follows that limn→∞Rn(z) = ζfor all z ∈ bC. We conclude that if R has a unique fixed point ζ ∈ bC then limn→∞Rn(z) = ζfor all z ∈ bC.
Next we consider the case where R has two distinct fixed points ζ1and ζ2. If ζ1 = 0
and ζ2 = ∞then R(z) = kz (k 6= 0), and for z different from ζ1 and ζ2 we have:
limn→∞Rn(z) = 0if |k| < 1, limn→∞Rn(z) = ∞if |k| > 1 and |Rn(z)| = |z|if |k| = 1.
When |k| = 1 either Rnis the identity for some n ≥ 1 or the points Rn(z)are dense on the circle {ζ ∈ C : |ζ| = |z|}.
Now suppose that ζ1 and ζ2 are any pair of distinct points in bC. If we conjugate Rwith the mapping ψ(z) = z−ζ1
z−ζ2 we get a function ˆR with 0 and ∞ as fixed points.
Since R and ˆR are conjugate functions and we can (as above) easily determine the dynamics of ˆR, we come to the conclusion that either Rn(z)converge to one of ζ
1and
ζ2, or they move cyclically through some finite set, or they form a dense subset of
some circle or line.
The dynamics of rational functions of higher degree is more interesting. We start with the following definition.
Definition 1.1.3. Let R be a rational function on bC of degree d ≥ 2. Define the Fatou set of R to be F (R) = {z ∈ bC : ∃Uz, a neighbourhood of z, s.t.Rn|Uzis normal} and the Julia set
of R to be J (R) = bC \ F (R).
From this definition we see that the Fatou set is open and that the Julia set is com-pact. We list a few basic properties of the Julia set in a theorem. Proofs of these results can be found in any book on complex dynamics, see e.g. [4, p. 55-57].
Theorem 1.1.4. Let R be a rational function of degree d ≥ 2.
(1) The Julia set J (R) is nonempty, perfect (i.e. compact and without isolated points) and completely invariant.
(2) J (R) coincides with J (Rn)for all n ≥ 1.
(4) Attracting fixed points and irrationally neutral fixed points around which R is con-jugate to irrational rotation belong to the Fatou set. All other fixed points of R belong to the Julia set.
The Julia set of a rational function is generally a very complicated fractal set with a rich structure. In case the rational function preserves the unit disc D, its boundary T, and also the complement of the closed unit disc bC \ D the dynamics becomes much easier.
Definition 1.1.5. A rational function on bC of the form B(z) = eiθ d Y j=1 z − aj 1 − ¯ajz ,
with d ∈ N, θ ∈ R and aj ∈ D for all j=1...d, is called a Blaschke product of degree d.
Blaschke products of finite degree are exactly the rational mappings with the above mentioned property.
Proposition 1.1.6. The Julia set of a Blaschke product B, of degree d ≥ 2, is either T or a
totally disconnected perfect subset of T.
Totally disconnected perfect sets are called Cantor sets.
In the proof of Proposition 1.1.6 we will discuss total disconnectedness of subsets of the circle T. In preparation for this we state the following remark.
Remark 1.1.7. For closed subsets of lines or circles total disconnectedness is equiv-alent to having empty interior. First we show this property for lines.
If a subset of a line is totally disconnected it does not contain any intervals of pos-itive length and hence the set has empty interior. In particular this holds if the set is closed.
Any subset X of a line can be written as a disjoint union of its connected compo-nents. If X is closed these components are either closed intervals of positive length or points. If furthermore X has empty interior none of the connected components can be intervals, hence all connected components are points, i.e. X is totally disconnected.
Now when we know that this property holds for lines a simple application of stere-ographic projection shows that the property holds also for circles.
Proof of Proposition 1.1.6. Since Bnmaps D onto itself the iterates Bnof B are bounded
on D, hence they are normal there which implies that D ⊂ F(B). In the same way all Bnleave bC\D invariant. Hence B1n maps bC\D onto D and is therefore a normal family
on bC \ D. Hence the Bnare normal on bC \ D and thus bC \ D ⊂ F (B). We conclude that J (B) ⊂ T.
that F(B) ∩ T is a union of open arcs. Let z be an endpoint of one of the arcs, i.e. let z ∈ J (B) ∩ ∂T(F (B) ∩ T), and let z0 ∈ B−n(z). The goal now is to show that also z0
is an endpoint of one of these arcs.
None of the points z0, B(z0)and Bn(z0)equals zero. Thus in some neighbourhood
of z0 a branch of the function log(Bn(ζ)) and a branch of the function log(B(ζ)) are
defined. Hence also continuous branches of arg(Bn)and arg(B) are defined in this
neighbourhood of z0. Let bj(ζ) = 1− ¯ζ−aajjζ. By the argument principle we have that
∆Targ(bj) = 2π. Since bj is a M ¨obius transform, i.e. a diffeomorphism from bC to b
C, we have that arg(bj(ζ)) is strictly increasing on any small arc in T. Hence the
same must hold for the function arg(B(ζ)) since arg(B(ζ)) = θ +Pd
j=1arg(bj(ζ)).
In turn this gives that also arg(Bn)is strictly increasing on small arcs in T, since Bn
is a composition of functions with strictly increasing argument on such small arcs. This implies that Bnis a homeomorphism of a small arc A1 around z0 onto a small
arc A2 around z (both arcs in T). Let ( fBn)−1 be the local branch of the inverse of Bn
that takes A2 to A1. Since A2is an arc of T containing z we know that some open arc
A3 ⊂ A2, on one side of - and adherent to z, is contained in F(B). F(B) is completely invariant so we have ( fBn)−1(A
3) ⊂ F (B). Since ( fBn)−1 is a homeomorphism from
A2 onto A1, ( fBn)−1(A3)is an arc on one side of - and adherent to z0. Thus also z0 is
an endpoint of an arc in F(B) ∩ T.
Finally this last fact implies that J (B) is totally disconnected. Suppose that this is not true and let w1, w2 ∈ J (B) be two distinct points belonging to the same connected
component of J (B). This means that one of the arcs of T that connect w1 and w2 is
contained in J (B). Let w3 be any interior point of this arc. In particular this means
that w3 has a neighbourhood in T which has no point in common with F(B) ∩ T.
But we know that the backward iterates of z are dense in J (B), and from above we have that all these points are endpoints of arcs in F(B) ∩ T. Together this forces w3
to belong also to ∂T(F (B) ∩ T). This is a contraction so we must have that J (B) is
totally disconnected.
It turns out that we can also tell which Blaschke products that have T, respectively some Cantor set on T, as their Julia set. But before we do this we need Theorem 1.1.9 that classify the periodic components of the Fatou set.
Definition 1.1.8. Let R be a rational function of degree d ≥ 2. A periodic component U ,
of period n, of the Fatou set F (R) is called
(1) parabolic if there, on its boundary, is a neutral fixed point for Rn with multiplier 1,
such that all points of U converge to this fixed point under iteration of Rn.
(2) a Siegel disk if U is simply connected and Rn|U is conjugate to irrational rotation on
a disk.
Theorem 1.1.9(Sullivan’s classification theorem). Suppose that U is a periodic compo-nent of the Fatou set F (R). Then precisely one of the following four alternatives holds:
(1) U contains an attracting periodic point of R. (2) U is parabolic.
(3) U is a Siegel disk. (4) U is a Herman ring.
For the proof of this theorem see e.g. [4, p. 74-79].
Now we return to the problem of how to determine what the Julia set of a particular Blaschke product looks like. We begin by observing a few properties of the fixed points of a Blaschke product B. Let B(z) = eiθQd
j=1 z−aj
1− ¯ajz and assume that B has a
fixed point ζ, i.e. that B(ζ) = ζ. Then also 1/¯ζis a fixed point of B, since B(1¯ ζ) = e iθ d Y j=1 1 ¯ ζ − aj 1 − ¯aj1ζ¯ = eiθ d Y j=1 1 − ajζ¯ ¯ ζ − ¯aj = e−iθ d Y j=1 1 − ¯ajζ ζ − aj = 1 B(ζ) = 1 ζ = 1¯ ζ.
We are also interested in finding the type of the fixed point 1/¯ζgiven the type of ζ. By the product-rule the derivative of B equals
B0(z) = B(z) d X j=1 1 − |aj|2 (z − aj)(1 − ¯ajz) . This means that for the fixed point 1/¯ζthe derivative is
B0(1¯ ζ) = 1 ¯ ζ d X j=1 1 − |aj|2 (1ζ¯− aj)(1 − ¯aj1ζ¯) = d X j=1 1 − |aj|2 (1ζ¯− aj)( ¯ζ − ¯aj) = ζ¯ d X j=1 1 − |aj|2 (1 − ajζ)(¯¯ ζ − ¯aj) = B0(ζ).
This implies that 1/¯ζis a fixed point of the same type as ζ is.
For a Blaschke product B the Fatou set F(B) consists of one or possibly two compo-nents. In either case the component/components are fixed, so in particular periodic. Hence we can apply Theorem 1.1.9. First of all we see that cases 3 and 4 in the theorem never occur for Blaschke products of degree greater than two. Both Siegel discs and Hermann rings are Fatou components on which the mapping is one-to-one. Since a Blaschke product B, of degree greater than two, is not one-to-one on any of its Fatou components, no component of F(B) can be a Siegel disc or a Herman ring.
the basins of attraction and hence J (B) = T. If B has an attracting fixed point that belongs to T, we must have that J (B) is totally disconnected. Indeed, attracting fixed points belong to the Fatou set so obviously J (B) 6= T.
If the Fatou set contains an attracting cycle of length greater than one, it must con-tain at least as many components as members of the cycle. This is so since each point in the cycle has its own attracting basin which consists of at least one Fatou compo-nent. Since we saw above that if an attracting periodic point belongs to T there is only one Fatou component, we must have that this cycle belongs to either D or bC \ D. But this means that disjoint attracting basins, i.e. a union of disjoint Fatou components, are supposed to be contained in D or else in bC \ D, which themselves are Fatou com-ponents. This is clearly not possible. Hence no periodic points with period greater than one are attracting, i.e. all attracting periodic points for B must be fixed.
Finally we have a look at case 2. Assume that we have a parabolic fixed point on T with one attracting petal. The fixed point belongs to J (B) and the petal is contained in F(B). The petal contains a neighbourhood of the fixed point minus a cusp around some ray starting at the fixed point. Hence the petal contains an arc of T on one side of the fixed point. Hence J (B) 6= T and so is totally disconnected by Proposition 1.1.6.
Next assume that B has a parabolic fixed point on T with two attracting petals. Then the petals must belong to different Fatou components and J (B) must separate the components so J (B) = T. A parabolic fixed point with more than two petals can not exist since it needs more than two components of the Fatou set. Finally no peri-odic parabolic points exist by the same reason as no periperi-odic points are attracting.
We summarize what we have found above in the following proposition.
Proposition 1.1.10. Let B be a Blaschke product of degree d ≥ 2. If B has an attracting
fixed point on T, or if B has a parabolic fixed point on T with only one attracting petal, then J (B) is a Cantor set on T. If B has an attracting fixed point in D, or if B has a parabolic fixed point on T with two attracting petals, then J (B) = T. These are all possible cases.
Consider now the higher dimensional case. The following result of H. Alexander states in particular that the only rational mappings on Pnthat leave Bn and ∂Bn in-variant (just as the Blaschke products do when n = 1) are the automorphisms of the ball. His result is even much sharper.
Theorem 1.1.11. Let F = (f1, f2, ..., fn) : Ω −→ Cn be holomorphic on a connected
neighbourhood Ω of some point of the boundary of the open unit ball Bn in Cn(n > 1). Suppose that Pn
j=1|fj(z)|2 ≡ 1 for z ∈ Ω ∩ ∂Bn. Then either F is a constant map or F
extends to be an automorphism of Bn.
This result, together with a generalization of the facts on iteration of M ¨obius trans-formations of bC, implies that the only interesting dynamics on Bn arises for discrete subgroups of the automorphism group of Bn.
1.2. Discontinuous Groups. We start this section with some definitions in a general setting before we restrict our attention to Kleinian groups.
Let X be a topological space and G a group of homeomorphisms of X onto itself.
Definition 1.2.1. Let Γ be a subgroup of G. The point x ∈ X is called a limit point of Γ
if there is a point z ∈ X and an infinite sequence {γn}∞n=1⊂ Γ of distinct elements such that
γn(z) → xas n → ∞. The set of limit points of Γ is called the limit set of Γ and is denoted by
Λ(Γ). If x ∈ X does not belong to Λ(Γ), it is called an ordinary point of Γ. The set of ordinary points is called the ordinary set of Γ and is denoted by Ω(Γ).
It is immediate from this definition that the limit set, Λ(Γ), is Γ−invariant and hence the same is true for the ordinary set.
Definition 1.2.2. A group Γ ⊂ G is called discontinuous at the point x ∈ X if x is an
ordinary point of Γ. The group Γ is said to be a discontinuous group if it is discontinuous somewhere.
Note that subgroups of discontinuous groups are discontinuous.
Definition 1.2.3. A group Γ ⊂ G is called properly discontinuous at the point x ∈ X if
there exists an open neighbourhood U of x such that γ(U ) ∩ U = ∅ for every γ ∈ Γ \ {Id}. The group Γ is said to be properly discontinuous if it is properly discontinuous somewhere.
The notion of proper discontinuity is frequently used by different sources. One problem is that different sources use different definitions of this property and several of these definitions are not equivalent. The definition of proper discontinuity stated above is preferred in this text because we will have use of the following lemma in the proof of Corollary 3.2.2.
Lemma 1.2.4. Proper discontinuity of a group Γ ⊂ G implies discontinuity of Γ.
Proof. Let Γ be properly discontinuous at α ∈ X and let U be a neighbourhood of α such that γ(U) ∩ U = ∅ for every γ ∈ Γ \ {Id}. Assume that Γ is not discontinuous at α, i.e. that there exists a point z ∈ X and a sequence of distinct elements {γi}∞i=1⊂ Γ
such that γi(z) → αas i → ∞. Then there exists N ∈ N such that γi(z) ∈ U for i ≥ N .
Hence z ∈ γi−1(U )for i ≥ N . This contradicts that γ(U) ∩ U = ∅ for every γ ∈ Γ \ {Id} and thus no such z ∈ X and {γi}∞i=1⊂ Γ exist. Hence Γ is discontinuous at α and thus
a discontinuous group.
1.2.1. Kleinian groups. Now let X = bC and let G be the set of all analytical homeomor-phisms of bC onto itself, i.e. G = M :=z 7→ az+bcz+d|a, b, c, d ∈ C, ad − bc = 1 .
Definition 1.2.5. A discontinuous subgroup of M is called a Kleinian group.
Definition 1.2.6. A Kleinian group Γ is called discrete if it does not contain any convergent
sequence of distinct elements.
The group M is naturally identified with the matrix group P SL(2, C). It is of-ten more convenient to work with the matrix representation of M ¨obius transforma-tions than with the M ¨obius transformatransforma-tions themselves, i.e. to study subgroups of P SL(2, C) instead of subgroups of M. Since the definition of discontinuity is in terms of groups of transformations acting on bC, we need a corresponding notion for sub-groups of P SL(2, C). It turns out that we need the notion of discreteness. It is im-mediate that discontinuity of the group Γ implies discreteness of Γ. Even though the opposite implication is not true we have the following theorem.
Theorem 1.2.7. Let Γ be a subgroup of M. Γ is discontinuous at the point z ∈ bC if and only if Γ is discrete as a subgroup of P SL(2, C) and Γ is a normal family on some neighbourhood of z.
The proof of this theorem can be found in [6, p. 98-99].
From this theorem it is clear that Ω(Γ) is open and hence that Λ(Γ) is closed for all Kleinian groups Γ. We also mention that an easy argument shows that all Kleinian groups are countable.
We will now focus on the limit sets of Kleinian groups and in order to study them we have great use of the following theorem.
Theorem 1.2.8. Let Γ be a Kleinian group and let x ∈ Λ(Γ). Then for each z ∈ bC with the possible exception of z = x and of one other point the set Γz accumulates at x.
For the proof of the theorem see [6, p. 103-104].
This result implies that Λ(Γ) = ∂Ω(Γ) for all Kleinian groups Γ. Hence it follows that Λ(Γ) equals either bC or a nowhere dense subset of bC. Another rather immediate consequence of Theorem 1.2.8 is the following corollary.
Corollary 1.2.9. Let Γ be a Kleinian group. If S is a Γ−invariant closed set containing at
least two points, then Λ(Γ) ⊂ S.
Kleinian groups for which the limit set consists of not more than two points are called elementary groups. For non-elementary Kleinian groups Γ we have from above that Λ(Γ) is the smallest non-empty Γ−invariant closet subset of bC.
Theorem 1.2.10. The limit set of a non-elementary Kleinian group is perfect.
Any transformation γ in M \ {Id} is conjugate in M to z 7→ z + 1, z 7→ λz (|λ| = 1, λ 6= 1) or z 7→ λz (|λ| > 1). The transformation γ is called parabolic, elliptic and loxodromic respectively. Finally loxodromic transformations with real λ are called hyperbolic transformations. Note that this type of an element in M is preserved under conjugation.
We conclude this subsection with another characterization of the limit set for Kleinian groups with more than one limit point.
Theorem 1.2.11. Let Γ be a Klenian group. If Λ(Γ) contains more than one point, it is the
closure of the set of fixed points of the loxodromic transformations of Γ.
Proof. Let A be the set of all fixed points of loxodromic transformations in Γ. It can be shown that in this situation Γ must contain at least one loxodromic transforma-tion. Hence A contains at least two distinct elements. From the characterization of loxodromic transformations above it is clear that A ⊂ Λ(Γ). Since Λ(Γ) is closed this implies that also A ⊂ Λ(Γ).
Let z ∈ Λ(Γ) \ A. Choose any distinct x1, x2 ∈ A. Theorem 1.2.8 yields that at
least one of the orbits Γx1and Γx2accumulates at the point z. We now prove that all
images of elements of A belong to A. Take w ∈ A, a fixed point for the loxodromic transformation ψ ∈ Γ, and let γ ∈ Γ be arbitrary. γ(w) is a fixed point for the trans-formation γ ◦ ψ ◦ γ−1∈ Γ. Since the type of ψ is preserved under conjugation we see that also γ(w) ∈ A. This implies that Γx1, Γx2⊂ A which concludes the proof.
1.2.2. Fuchsian Groups. We now introduce a special kind of Kleinian groups with an interesting geometric property.
Definition 1.2.12. A Kleinian group Γ is called Fuchsian if it preserves the interior and
exterior of a circle. (Here a line is thought of as a circle through infinity.) The invariant circle is called the principal circle of Γ.
It can be shown that a non-elementary Kleinian group is Fuchsian if and only if it does not contain any non-hyperbolic loxodromic elements.
Since the principal circle is a closed invariant set containing more than two points, we know from Corollary 1.2.9 that the limit set of a Fuchsian group Γ is contained in the principal circle of Γ. When the limit set contains more than two points we can say even more.
Theorem 1.2.13. Let Γ be a Fuchsian group with principle circle Σ. If Γ is non-elementary,
then either
(1) Λ(Γ) = Σ, or
Proof. Above we noticed that Λ(Γ) ⊂ Σ. Assume that Λ(Γ) 6= Σ, i.e. that Ω(Γ) ∩ Σ 6= ∅. Λ(Γ) is prefect according to Theorem 1.2.10 so we must show that it is totally disconnected. Take any point z ∈ Ω(Γ) ∩ Σ. Since Ω(Γ) is an open set there is an open neighbourhood U of z contained in Ω(Γ). By possibly shrinking U we can assume that UΣ= U ∩ Σis an open arc in Ω(Γ).
Choose any point x ∈ Λ(Γ). By Theorem 1.2.8 there is a point y ∈ UΣ and a
se-quence {γn}∞n=1of distinct elements of Γ such that γn(y) → xas n → ∞. Since Σ and
Ω(Γ)are Γ−invariant we get that {γn(y)}∞n=1 ⊂ Σ ∩ Ω(Γ). Hence each open arc in Σ
containing x also contains points belonging to Ω(Γ). Since x ∈ Λ(Γ) was arbitrary this means that no arc of Σ is contained in Λ(Γ), i.e. Λ(Γ) is totally disconnected. The Fuchsian groups for which the limit set is the entire principle circle are called Fuchsian groups of the first kind. All other Fuchsian groups are called Fuchsian groups of the second kind.
2. AUTOMORPHISMS OFBn This section is based on [9, p. 25-28].
2.1. Description of the Automorphisms in Bn (n > 1). In order to study the
auto-morphisms of Bnwe introduce a family of rational mappings, {φ
w}w∈Bn. Let w ∈ Bn
and Pw be the orthogonal projection of Cn on the linear subspace generated by w.
Furthermore let Qw = Id − Pwand sw = (1 − |w|2)1/2. We now define φw as follows,
(1) φw(z) =
w − Pw(z) − swQw(z)
1 − hz, wi .
It is immediate from the definition that φw(w) = 0and φw(0) = w.
Since |w| < 1 we have that φwis holomorphic in Bn. As we shall see below it turns out
that the mappings φwactually are automorphisms of Bn. Before we can prove that we
need the following two lemmas.
Lemma 2.1.1. For every w ∈ Bn, we have that φ0w(0) = −s2wPw− swQw
and φ0w(w) = −s12 wPw−
1 swQw.
Proof. We use the geometric series expansion of 1−hz,wi1 , which is convergent for all z ∈ Bn, to rewrite (1): φw(z) = 1 + hz, wi + hz, wi2+ · · · w − (Pw+ swQw)(z) = w + hz, wiw − (Pw+ swQw)(z) + O(|z|2) = φw(0) + |w|2Pw(z) − (Pw+ swQw)(z) + O(|z|2) = φw(0) + − s2wPw− swQw(z) + O(|z|2).
Hence we have that φ0w(0) = −s2
By insertion of w + h, where h is small enough to ensure that s2w > |hh, wi|, in (1) we get: φw(w + h) = −Pw(h) − swQw(h) s2 w− hh, wi = − 1 s2 w Pw(h) + swQw(h) 1 −hh,wis2 w .
Using the geometric series expansion of the denominator of this last fraction yields φw(w + h) = − 1 s2 w 1 +hh, wi s2 w +hh, wi s2 w 2 + · · · Pw(h) + swQw(h) = φw(w) + − 1 s2 w Pw− 1 sw Qw(h) + O(|h|2).
Hence we have also that φ0w(w) = −s12 wPw−
1
swQw.
Lemma 2.1.2. For every w ∈ Bn, φ
wsatisfies the following identity for all z1, z2 ∈ Bn:
(2) 1 − hφw(z1), φw(z2)i =
(1 − hw, wi)(1 − hz1, z2i)
(1 − hz1, wi)(1 − hw, z2i)
.
Proof. In (1) we see that φw(z)splits into one component in span{w} and one
compo-nent in the orthogonal complement to span{w}. With this in mind we get the follow-ing, hφw(z1), φw(z2)i = Dw − Pw(z1) 1 − hz1, wi − swQw(z1) 1 − hz1, wi ,w − Pw(z2) 1 − hz2, wi − swQw(z2) 1 − hz2, wi E (3) = hw − Pw(z1), w − Pw(z2)i + s 2 whQw(z1), Qw(z2)i (1 − hz1, wi)(1 − hw, z2i) .
Now recall that orthogonal projections, in particular Pw and Qw, are self-adjoint and
idempotent operators. Hence we get
(4) hPw(z1), wi = hz1, Pw(w)i = hz1, wi, (5) hQw(z1), Qw(z2)i = hQw2(z1), (z2)i = hQw(z1), (z2)i and hw − Pw(z1), w − Pw(z2)i = hw − Pw(z1), Pw(w) − Pw(z2)i (6) = hPw(w) − Pw2(z1), w − z2i = hw − Pw(z1), w − z2i.
We also note that
With the help of (3), (4),(5), (6) and (7) we can now rewrite the left side of (2) as follows: 1 − hφw(z1), φw(z2)i = 1 − hw − Pw(z1), w − z2i + s2whQw(z1), z2i (1 − hz1, wi)(1 − hw, z2i) = (1 − hz1, wi)(1 − hw, z2i) − hw − Pw(z1), w − z2i − s 2 whQw(z1), z2i (1 − hz1, wi)(1 − hw, z2i) = 1 + hw, wihPw(z1), z2i − hw, wi − hPw(z1), z2i − (1 − hw, wi)hQw(z1), z2i (1 − hz1, wi)(1 − hw, z2i) = 1 − hw, wi + (hw, wi − 1)hPw(z1), z2i − (1 − hw, wi)hz1, z2i + (1 − hw, wi)hPw(z1), z2i (1 − hz1, wi)(1 − hw, z2i) = (1 − hw, wi)(1 − hz1, z2i) (1 − hz1, wi)(1 − hw, z2i) .
Since this is exactly the right hand side of (2), the proof is complete. If we let z1 = z2= zin (2) we get
(8) 1 − |φw(z)|2 =
(1 − |w|2)(1 − |z|2) |1 − hz, wi|2 ,
which holds for all z ∈ Bn. It follows directly from this identity that |φ
w(z)| < 1 ⇔
|z| < 1, so in particular that φw(Bn) ⊂ Bn. We also have that |φw(z)| = 1 ⇔ |z| = 1, so
φwmaps ∂Bninto ∂Bn.
Theorem 2.1.3. For every w ∈ Bn, φwis an automorphism of Bn.
Proof. Consider the mapping ϕw = φw◦ φw. From what we have discussed above it is
clear that ϕwis holomorphic in Bn, maps Bninto Bnand maps ∂Bninto ∂Bn. It is also
immediate that ϕw(0) = 0. Lemma 2.1.1 and the chain rule gives us that
ϕ0w(0) = φ0w(w) ◦ φ0w(0) = − 1 s2 w Pw− 1 sw Qw ◦ − s2wPw− swQw = Pw+ Qw= Id.
The next to last equality follows from that Pwand Qware idempotent and the obvious
equalities PwQw = QwPw = 0.
Hence ϕw : Bn −→ Bn is a holomorphic function on a bounded region in Cn, with
ϕw(0) = 0and ϕ0w(0) = Id. By a theorem of Cartan (see [9, p. 23]) this tells us that
ϕw(z) = z for every z ∈ Bn. This means that φw is a bijective map of Bn onto Bn
and also that φ−1w = φw. Hence φwis a biholomorphic mapping of Bnonto Bn, i.e. an
automorphism of Bn.
It is also immediate that all the unitary transformations on Cnare automorphisms
Theorem 2.1.4. If ψ is an automorphism of Bnand a = ψ−1(0) then there is a unique unitary transformation U such that
ψ = U φa.
Proof. The map χ = ψ ◦ φa is an automorphism of Bn and such that χ(0) = 0. By
another theorem of Cartan (see [9, p. 24]) we have that χ is a linear transformation. Since χ also preserves Bn it must be equal to a (unique) unitary transformation U .
Hence we have that ψ ◦ φa = U. Applying φ−1a = φafrom the right to both sides of
this equality gives us that ψ = U ◦ φa.
The family of automorphisms of Bn forms a group under composition which is
called Aut(Bn). We will be interested in its discrete subgroups.
2.2. Automorphisms of B2leaving some complex line through zero invariant. We will discuss automorphisms of B2 leaving the first coordinate line fixed. All other
automorphisms fixing some complex line through the origin can be obtained from these by conjugating with some appropriate unitary transformation.
First we determine which of the functions in the family {φw}w∈B2 leave the line
{z2 = 0} invariant. Note that if w = (0, 0) then φw(z) = −z which clearly leaves
{z2 = 0}invariant. In the following we assume that w = (w1, w2) 6= (0, 0). If we write
φwon coordinate form we get the following:
φw(z1, z2) = w1 1 +z1w1+z2w2 |w|2 (1 − |w|2)1/2− 1 − (1 − |w|2)1/2z 1 1 − z1w1− z2w2 , w2 1 +z1w1+z2w2 |w|2 (1 − |w|2)1/2− 1 − (1 − |w|2)1/2z 2 1 − z1w1− z2w2 = ζ1(z1, z2), ζ2(z1, z2).
Since we are interested in mappings leaving {z2 = 0}invariant, we determine which
conditions that have to be fulfilled in order to have ζ2(z1, 0) = 0for all (z1, 0) ∈ B2.
Insertion of z2 = 0gives ζ2(z1, 0) = w2 1 +z1w1 |w|2 (1 − |w|2)1/2− 1 1 − z1w1 . Hence ζ2(z1, 0) = 0 ∀(z1, 0) ∈ B2 ⇔ w2 = 0 since 1 + z1w1
|w|2 (1 − |w|2)1/2− 1 = 0 clearly can not hold for all z1 ∈ D. Thus if φw
fixes {z2 = 0}we have w = (w1, 0). Some straightforward simplifications give that
such φwlooks like
Note that this also cover the case that w = (0, 0) discussed separately above.
It is clear that if we compose any such mapping with a unitary transformation fixing {z2 = 0}we get another automorphism of B2 fixing {z2 = 0}. It is also clear
from Theorem 2.1.4 that there are no other mappings in Aut(B2) fixing {z2 = 0}.
Hence we can write all automorphisms of B2fixing {z2= 0}on the form
ϕa(z1, z2) = eiθ z1− a 1 − az1 , eiµ(1 − |a| 2)1/2z 2 1 − az1 where a ∈ D and θ, µ ∈ [0, 2π[.
Similarly we get that all elements in Aut(B2)fixing {z1 = 0}are on the form
ψb(z1, z2) = eiη(1 − |b| 2)1/2z 1 1 − bz2 , eiρ z2− b 1 − bz2 where b ∈ D and η, ρ ∈ [0, 2π[.
3. DISCRETESUBGROUPS OFAut(B2)AND THEIRLIMIT SETS
3.1. A useful lemma. The limit set of a subgroup Γ of Aut(B2)will not be defined in the same way as we defined limit sets in section 1.2. Since B2 is open as a subset of C2, sequences in B2can converge to points outside of B2. In order to include all limit points of sequences of the type {γn(z)}∞n=1, with z ∈ B2 and {γn}∞n=1⊂ Γ, in the limit
set of Γ we make the following definition.
Definition 3.1.1. Let Γ be a subgroup of Aut(B2). Extend all elements of Γ holomor-phically to the closed unit ball B2 and let eΓ ⊂ Aut(B2) be the group of all such extended
mappings. The limit set of Γ ,denoted Λ(Γ), is now defined to be equal to the limit set (in the sense of Definition 1.2.1) of the group eΓ.
Definition 3.1.2. A subgroup Γ of Aut(B2) is called discrete if it does not contain any
convergent sequence of distinct elements.
We begin our study of limit sets of discrete subgroups of Aut(B2)with the following
lemma.
Lemma 3.1.3. Let Γ be a discrete subgroup of Aut(B2). Suppose that Γ leaves the
inter-section of a complex line L, passing through 0, with B2 invariant. Then the limit set of Γ
coincides with the limit set of Γ|L∩B2 , i.e. Λ(Γ) = Λ(Γ|L∩B2).
Proof. We begin by proving the lemma for the case that Γ fixes the complex line L = L0 = {z = (z1, z2)|z2= 0}.
First we consider Γ|L0. This is a discrete group of automorphisms of D0, the unit disc
in L0. Hence it is a Fuchsian group and we know from the theory of Fuchsian groups
From section 2.2 we know what the elements in Aut(B2) that fix L0, and hence
in particular the elements in Γ, look like. For each γ ∈ Γ we have that γ(z1, z2) =
(w1, w2) where w1 depends only on z1. This means that γ(z1, z2) has the same first
coordinate as γ|L0(z1, 0)has. Hence the projection of the Γ−orbit of z = (z1, z2)onto
L0 will coincide with the Γ|L0−orbit of (z1, 0). Since convergence in C
2 is equivalent
to coordinate-wise convergence we conclude that a point z = (z1, z2) ∈ Λ(Γ)only if
(z1, 0) ∈ Λ(Γ|L0). But from above we know that |z1| = 1 for all (z1, 0) ∈ Λ(Γ|L0). This
means that since Λ(Γ) ⊂ B2 we must have that |z
2| = 0, i.e. that z = (z1, 0). Hence
z ∈ Λ(Γ|L0)so we have that Λ(Γ) ⊂ Λ(Γ|L0). The other inclusion, Λ(Γ|L0) ⊂ Λ(Γ),
holds trivially and hence the lemma is proved for this case.
Now it is left to prove the general case of the lemma. Suppose that Γ ⊂ Aut(B2) is discrete and such that it fixes the complex line C passing through 0. There exists a unitary transformation U mapping L0onto C. We use U to construct the group
Γ0= U−1ΓU = {γ0 ∈ Aut(B2)|γ0= U−1γU , for some γ ∈ Γ}
which is discrete and fixes L0. From above we know that Λ(Γ0)coincides with Λ(Γ0|L0).
This clearly implies that also Λ(Γ) = Λ(Γ|C)and the proof is complete.
An immediate consequence of Lemma 3.1.3 and Theorem 1.2.13 is the following corollary.
Corollary 3.1.4. Let Γ be a discrete subgroup of Aut(B2) that leaves a complex line L,
passing through zero, invariant. Then Λ(Γ) ⊂ L contains either 0, 1, 2 or uncountably many points. When the cardinality of Λ(Γ) is uncountably infinite Λ(Γ) equals either the circle ∂B2∩ L or a Cantor set in ∂B2∩ L.
3.2. A particular example. We continue by considering the group Γ1 ⊂ Aut(B2)that
is generated by the mappings ϕ1(z1, z2) = z1− a 1 − az1 ,(1 − |a| 2)1/2z 2 1 − az1 (10) and ϕ2(z1, z2) = (1 − |b|2)1/2z1 1 − bz2 , z2− b 1 − bz2 (11)
where a, b ∈] − 1, 1[. From section 2.2 we know that ϕ1fixes the first coordinate line
and that ϕ2fixes the second coordinate line.
We are interested in choosing the constants a and b in such a way that Γ1is discrete.
In the literature there is not much written about discreteness conditions for subgroups of Aut(B2). In [2] some necessary conditions for discreteness are developed. For example they find a higher dimensional analog of Jørgensen’s inequality. We did not find sufficient conditions for discreteness of subgroups of automorphisms of the ball in the litterature. Our aim is now to find a sufficient condition for the group Γ1 to be
Lemma 3.2.1. There exists a real number c ∈] − 1, 0[ close to −1 such that if a, b < c then
there exists a small neighbourhood U of 0 whose orbit oscillates between small neighbourhoods of the fixed points of ϕ1and ϕ2. In particular U is such that γ(U )∩U = ∅ for all γ ∈ Γ1\{Id}.
For further reference we give these small neighbourhoods of the fixed points of ϕ1
and ϕ2names. The small neighbourhood around the fixed point z of ϕ1or ϕ2is called
Wz.
We postpone the proof of the lemma for a moment.
Corollary 3.2.2. Let c be as in Lemma 3.2.1. Then a, b < c implies that Γ1is discrete.
Proof. Since a, b < c where c is as in Lemma 3.2.1 we know that there exists a neigh-bourhood U of 0 such that γ(U) ∩ U = ∅ for all γ ∈ Γ1\ {Id}. It follows that Γ1 is
properly discontinuous at the point 0. We now use Lemma 1.2.4 to conclude that Γ1is
also discontinuous at 0 and hence a discontinuous group. Finally since discontinuity of a group trivially implies discreteness of the group we have the result that if a and bare as described above then the group Γ1is discrete.
Proof of Lemma 3.2.1. Let us for a moment study the group ∆1 ⊂ Aut(B2)generated by
ϕ1. Since ϕ1fixes the first coordinate line it follows that ∆1leaves the first coordinate
line invariant and thus we can use Lemma 3.1.3 to find the limit set of ∆1. Hence we
are interested in the orbits, i.e. the dynamics, of the rational function R(z) = 1−azz−a. Solving the equation R(z) = z gives us that R has the fixed points z = ±1. The derivative of R is R0(z) = (1−az)1−a22. Evaluating the derivative at the points z = ±1 and
using that a is chosen to be a negative real number gives that |R0(1)| < 1 and that |R0(−1)| > 1. Hence the point 1 is an attracting fixed point of R and the point −1 a
repelling fixed point of R. Recalling the discussion in section 1.1 on the dynamics of rational mappings on bC of degree one, we see that Rn(z) → 1 as n → ∞ and that Rn(z) → −1 as n → −∞ for all z ∈ bC \ {±1}. This implies that Λ(∆1|{z2=0}) =
{(±1, 0)} and hence by Lemma 3.1.3 we have that Λ(∆1) = {(±1, 0)}.
Since ϕ2 fixes the second coordinate line we have that the group ∆2 ⊂ Aut(B2)
generated by ϕ2 leaves the second coordinate line invariant. Since S(z) = 1−bzz−b has
the same type of dynamics as that of R when b is a negative real number we know that Sn(z) → 1 as n → ∞ and that Sn(z) → −1as n → −∞ for all z ∈ b
C \ {±1}. As in the case with ∆1we can use Lemma 3.1.3 to conclude that ∆2 has the limit set
Λ(∆2) = {(0, ±1)}.
Now let us consider a neighbourhood of 0 in B2of the form B
= {z ∈ B2: |z| < }
for some small (say that < 0.1). Independent of the choice of a we see that ϕ1(0, 0) = (−a, 0) and that ϕ−11 (0, 0) = (a, 0). This means that if a is chosen close
to −1 we have that ϕ1 maps (0, 0) to a point close to (1, 0) and that ϕ−11 maps (0, 0)
to a point close to (−1, 0). Now we want to choose a in such a way that B will be
mapped by ϕ1to a region close to (1, 0) which projection onto the second coordinate
If we consider the circular disc B∩ {z2 = 0}we know from the theory of M ¨obius
transforms that ϕ1 will map this disc to another circular disc inside B2 ∩ {z2 = 0}.
We note that ϕ1 leaves the real subspace of {z2 = 0} invariant. Since M ¨obius
trans-forms also preserves angles we know that the image of ∂{z2=0}B ∩ {z2 = 0} will
intersect the real subspace of {z2 = 0} in right angles at the points ϕ1(−, 0) and
ϕ1(, 0). This information is enough for the image of B∩ {z2 = 0}to be completely
determined. In particular it is clear that all points in ϕ1(B ∩ {z2 = 0}) satisfies
Re(z1) > Re(R(−)) = −1+a+a. Since |z1| < for all z ∈ B and the first
coordi-nate function of ϕ1 is independent of the z2−variable it follows that Re(z1) > −1+a+a
actually holds for all points in ϕ1(B).
We note that if Re(z1) >
√
1 − 2 for all z ∈ ϕ
1(B) we achieve both that ϕ1(B)
is close to (1, 0) and that ϕ1(B) has the projection onto the second coordinate line
within the set {z2 ∈ C : |z2| < }. From the previous paragraph we know that this
inequality will be satisfied if −1+a+a >√1 − 2. We rewrite this inequality as follows:
− + a 1 + a > p 1 − 2⇔ − − a > (1 + a)p1 − 2 ⇔ a < − + √ 1 − 2 1 + √1 − 2.
Hence if we have chosen a small > 0 we can choose a such that −1 < a < −+√1−2
1+√1−2 < 0and this choice will make ϕ1(B)satisfy the condition we are
inter-ested in. Note that with such a choice of a also ϕ−11 (B)satisfy a similar condition, i.e.
ϕ−11 (B) is close to (−1, 0) and ϕ1(B)has the projection onto the second coordinate
line within the set {z2 ∈ C : |z2| < }.
A similar argument shows that if b is chosen such that −1 < b < −+
√ 1−2
1+√1−2 < 0
we will have that ϕ±12 (B)are close to (0, 1) respectively (0, −1) and both sets will be
such that their projections onto the first coordinate line will be contained within the set {z1 ∈ C : |z1| < }.
Now consider the Γ1−orbit of B, i.e. the set Γ1B =
S
γ∈Γ1γ(B). If we first apply
the mappings ϕ±11 and their iterates to Bwe see that for all k ∈ Z\{0}, ϕk1(B)is close
to either (1, 0) or (−1, 0) (here close is in the sense above, i.e. that Re(z1) > 1 − 2
re-spectively Re(z1) < −1+2) and satisfy that the projection onto the second coordinate
axis is within the set {z2 ∈ C : |z2| < }. Since ϕ2’s second coordinate function only
depends on the z2−variable the argument above together with the choice of b implies
that if we apply any iterate of the mapping ϕ2 to any of these sets the image will
end up either close to (0, 1) or close to (0, −1). Furthermore this image will be such that the projection onto the first coordinate axis is within the set {z1 ∈ C : |z1| < }.
Repeated use of the argument presented in this paragraph shows that the images of B under elements of Γ1 \ {Id} oscillate between small neighbourhoods of the fixed
points of ϕ1and ϕ2. It is also clear from the choices made above that all elements γ in
Γ1\ {Id} satisfy γ(B) ∩ B = ∅. Hence c = − + √
1−2
1+√1−2 and U = Bworks for small
It is clear from the proof above that the following corollary holds.
Corollary 3.2.3. Let a and b be as in Lemma 3.2.1. Then the orbit of any fixed point of ϕ1
or ϕ2 oscillates between the small neighbourhoods Wziof the fixed points zi of ϕ1and ϕ2.
We continue by studying the limit set of Γ1 for such values of a and b. In order to
do so we introduce the set A = S
z∈EΓ1z, where E = {(±1, 0), (0, ±1)} is the set of
points which are fixed by either ϕ1 or ϕ2. We now note that the closure of the orbit
of one of the fixed points, say z0 = (1, 0), is already equal to the set A. This is indeed
so since the orbit of z0 accumulate at all the other fixed points. By continuity of ϕ1
and ϕ2 this means that Γ1z0 accumulate at all points of the orbits of the other fixed
points. Hence it is clear thatS
z∈EΓ1z ⊂ Γ1z0 and hence that
S
z∈EΓ1z ⊂ Γ1z0. Since
the opposite inclusion trivially holds we arrive at the equality A = Γ1z0.
The following proposition states the key properties of the set A.
Proposition 3.2.4. Let a and b be as in Lemma 3.2.1. Then A equals a Cantor set in
R2∩ ∂B2.
Proof. All points in E belong to R2 ∩ ∂B2. Since both a and b are chosen to be real
it is clear from (8), (10) and (11) that Γ1 leaves R2 ∩ ∂B2 invariant. Hence the set
B =S
z∈EΓ1zis contained in R2∩ ∂B2which implies that also A ⊂ R2∩ ∂B2.
Since A = Γ1z0it is clear that A is closed. Hence in order to prove that A is perfect
it is left to show that no points of A are isolated. We will show that B has no isolated points which clearly implies that A has no isolated points.
Take any w ∈ B. The point w can be written as w = ˜ϕ(z)for some z ∈ E and ˜ϕ ∈ Γ1.
zis a fixed point for either ϕ1or ϕ2, say that ϕ1(z) = z. Choose v ∈ E such that it is a
fixed point for ϕ2. Then ϕn1(v) → z either when n → ∞ or when n → −∞. Assume
for simplicity that limn→∞ϕn1(v) = z. Note that all points in the sequence {ϕn1(v)}∞n=1
belong to B. Now composing this sequence with the mapping ˜ϕgives us the sequence { ˜ϕ ◦ ϕn1(v)}∞n=1in B which by continuity of ˜ϕsatisfies limn→∞ϕ ◦ ϕ˜ n1(v) = w. Hence
wis not isolated in B. (A similar argument gives the same result if z is a fixed point for ϕ2.) Since w ∈ B was arbitrary this shows that B has no isolated points.
Finally we need to establish that the set A is totally disconnected. We will use the relation A = Γ1z0, where z0 = (1, 0), and prove that Γ1z0is totally disconnected.
Choose any two distinct points w1 and w2in Γ1z0. They are of the form w1 = γ1(z0)
and w2 = γ2(z0)for some γ1, γ2 ∈ Γ1. Our aim is to show that there are open arcs in
R2∩ ∂B2\ Γ1z0which separate w1 and w2 in R2∩ ∂B2. Since w1and w2are arbitrary
points of Γ1z0this will imply that all points in Γ1z0are separated from any other point
in Γ1z0 by arcs in R2∩ ∂B2 \ Γ1z0. This in turn will clearly imply that A = Γ1z0 is
totally disconnected.
Apply the mapping γ1−1 to w1and w2. This results in the points z0 = γ1−1◦ γ1(z0)
with out loss of generality assume that γ1−1◦ γ2= ψ ◦ ϕk2◦ ϕm1 with k 6= 0 and ψ ∈ Γ1.
Note that there are arcs in R2 ∩ ∂B2\ Γ
1z0 separating w1 and w2 in R2∩ ∂B2 if and
only if there are arcs in R2∩ ∂B2\ Γ
1z0separating z0and w3in R2∩ ∂B2.
We know from Corollary 3.2.3 that there is an arc U1in the first quadrant of R2∩∂B2,
between the neighbourhood Wz0 of z0 and the neighbourhood W(0,1) of the point
(0, 1), which contains no points of Γ1z0. Furthermore we know that there is an arc
U2 in the fourth quadrant of R2∩ ∂B2, between the neighbourhood Wz0 of z0 and the
neighbourhood W(0,−1)of the point (0, −1), which contains no points of Γ1z0. We are
interested in the point w3and the first thing we can say is that id◦ϕk2◦ϕm1 (z0) = ϕk2(z0)
is in either W(0,1)or W(0,−1). Hence ϕk2 ◦ ϕm1 (z0)is separated from z0 in R2∩ ∂B2 by
the arcs U1 and U2 in R2 ∩ ∂B2\ Γ1z0. We want to use this information to conclude
that also z0and w3are separated in R2∩ ∂B2by arcs in R2∩ ∂B2\ Γ1z0.
Since w3 ∈ Γ1z0 we know that w3 ∈ Wz˜ for some ˜z ∈ E. If w3 does not belong
to W(1,0)it is clear that z0 and w3 are separated in R2∩ ∂B2 by the arcs U1and U2 in
R2∩ ∂B2\ Γ1z0. Thus it is left to study the situation when w3 ∈ W(1,0). There are two
cases to consider.
First we consider the case when ψ = ϕq1 for some q ≥ 1. Since z0 is fixed by ϕq1
and ϕq1 is a homeomorphism of the circle R2∩ ∂B2onto itself it follows that the arcs
U1 and U2in R2∩ ∂B2\ Γ1z0, separating z0and ϕk2◦ ϕm1 (z0), are mapped onto arcs in
R2∩ ∂B2\ Γ1z0separating z0and w3 = ϕ1q(ϕk2◦ ϕm1 (z0))in R2∩ ∂B2.
The only remaining possibility, with a chance for w3to be contained in W(1,0), is that
ψ = ϕp1◦ϕs
2◦ξ for some p ≥ 1, s 6= 0 and ξ ∈ Γ1. We have that ϕs2◦ξ(ϕk2◦ϕm1 (z0))belongs
to either W(0,1)or W(0,−1)and hence by a similar argument as in the paragraph above
we can conclude that there are arcs in R2∩ ∂B2\ Γ
1z0separating z0and w3in R2∩ ∂B2.
Hence in all cases z0 and w3are separated in R2∩ ∂B2 by arcs in R2∩ ∂B2\ Γ1z0.
This implies that also w1and w2 are separated in R2∩ ∂B2by arcs in R2∩ ∂B2\ Γ1z0.
According to the reasoning above this implies that A = Γ1z0 is totally disconnected.
Hence A is a perfect and totally disconnected subset of R2∩ ∂B2, i.e. a Cantor set in
R2∩ ∂B2.
Since Λ(∆1), Λ(∆2) ⊂ Λ(Γ1)we know that E ⊂ Λ(Γ1). The invariance of the limit
set ensures that also B ⊂ Λ(Γ1). Furthermore since A is a perfect set and Γ1z0 = A,
it is also clear that A ⊂ Λ(Γ1). The following conjecture states that the set A is all of
Λ(Γ1).
Conjecture 3.2.5. Let a and b be as in Lemma 3.2.1. Then Λ(Γ1) = A, i.e. Λ(Γ1)equals
a Cantor set in R2∩ ∂B2.
Since we already have deduced that A ⊂ Λ(Γ1)it is left to show the opposite
in-clusion. For this purpose it is enough to prove that the Γ1−orbit of any point in B2
can be proved using normality arguments but it is not perfectly clear how to proceed to show that also the Γ1−orbit of any point in ∂B2accumulates only at points in A.
3.3. Concluding remarks. In the example considered in section 3.2. the limit set is conjectured to be a Cantor set on a circle. This is one of the situations that can occur also for Fuchsian groups. But even though it is not shown in this text it is our belief that the situation for the ball B2 is much more complicated than for Fuchsian groups. For example it is likely to be the case that limit sets of discrete subgroups of Aut(B2)
can be for example Cantor sets of circles and even more complicated structures of linked circles. But as long as no general theory is developed it is a lot of work to find such examples. First one needs to find a candidate for such a group. Then one has to go through the same kind of machinery as in section 3.2 in order to verify that the group is discrete and that the limit set really is what you want it to be.
ACKNOWLEDGEMENTS
I would like to thank my supervisor professor Burglind Juhl-J ¨oricke for support and advice during the work with this thesis.
REFERENCES
[1] H. Alexander, Holomorphic Mappings from the Ball and the Polydisc, Math. Ann. 209, pp.249-256 (1974).
[2] A. Basmajian, R. Miner, Discrete subgroups of complex hyperbolic motions, Invent. math. 131, 85-136 (1998).
[3] A. F. Beardon, Iteration of Rational Functions, Springer-Verlag, 1991. [4] L. Carleson, T. W. Gamelin, Complex Dynamics, Springer-Verlag, 1993. [5] S. Katok, Fuchsian Groups, The University of Chicago Press, 1992.
[6] J. Lehner, Discontinuous Groups and Automorphic Functions, American Mathematical Soci-ety, 1964.
[7] B. Maskit, Kleinian Groups, Springer-Verlag, 1988.
[8] S. Morosawa, Y. Nishimura, M. Taniguchi, T. Ueda, Holomorphic Dynamics, Cambridge University Press, 2000.
