WEAKLY MIXING GROUP ACTIONS: A BRIEF SURVEY AND AN EXAMPLE

V. BERGELSON^{∗}, A. GORODNIK

Dedicated to Anatole Katok on the occasion of his 60th birthday.

1. Introduction

At its inception in the early 1930’s, ergodic theory concerned itself with continuous one-parameter flows of measure preserving transformations ([Bi], [vN1], [KvN], [Ho1], [Ho2]). Soon it was realized that working with ℤ-actions rather than with ℝ-actions, has certain advantages. On the one hand, while the proofs become simpler, the results for ℝ-actions can often be easily derived from those for ℤ-actions (see, for example, [Ko]). On the other hand, dealing with ℤ-(or even with ℕ-) actions extends the range of applications to measure preserving transformations which are not necessarily embeddable in a flow. Weakly mixing systems were introduced (under the name dynamical systems of continuous spectra) in [KvN]. By the time of publishing in 1937 of Hopf’s book [Ho3], the equivalence of the following conditions (which, for convenience, we formulate for ℤ-actions) was already known. It is perhaps worth noticing that, while in most books either (i) or (ii) below is taken as the “official” definition of weak mixing, the original definition in [KvN] corresponds to the condition (vi). Theorem 1.1. Let T be an invertible measure-preserving transformation of a probability measure space (X,B,μ). Let U_{T} denote the operator defined on the space of measurable functions by (U_{T}f)(x) = f(T x). The following conditions are equivalent:

(i) For any A,B ∈ B,

lim_{N→∞} \frac{1}{N} ∑_{n=0}^{N−1} |μ(A ∩ T^{−n} B) − μ(A)μ(B)| = 0.

(ii) For any A,B ∈ B, there is a set P ⊂ ℕ of density zero such that

lim_{n→∞,n∉P} μ(A ∩ T^{−n} B) = μ(A)μ(B).

(iii) T × T is ergodic on the Cartesian square of (X,B,μ).

(iv) For any ergodic probability measure preserving system (Y,D,ν,S), the transformation T ×S

is ergodic on X × Y.

(v) If f is a measurable function such that for some λ ∈ ℂ, U_{T}f = λf a.e., then f = const a.e. (vi) For f ∈ L^{2}(X,B,μ) with ∫_{X}fdμ = 0, consider the representation of the positive definite

sequence ⟨U_{T}^{n}f,f⟩,n ∈ ℤ, as a Fourier transform of a measure ν on 𝕋 = ℝ/ℤ:

⟨U_{T}^{n}f,f⟩ = ∫_{𝕋} e^{2πinx}dν, n ∈ ℤ

(this representation is guaranteed by Herglotz theorem, see [He]). Then ν has no atoms. Remark 1.2. It is not too hard to show that condition (i) can be replaced by the following more general condition:

∗The first author was supported by the NSF Grants DMS-0070566 and DMS-0345350.

(i′) For any A,B ∈ B and any sequence of intervals I_{N} = [a_{N}+ 1,a_{N}+ 2,...,b_{N}] ⊂ ℤ, N ≥ 1,

with |I_{N}| = b_{N}− a_{N} → ∞, one has

lim_{N→∞} \frac{1}{|I_{N}|} ∑_{n=a_{N}+1}^{b_{N}} |μ(A ∩ T^{−n} B) − μ(A)μ(B)| = 0.

Condition (i′), in its turn, is equivalent to a still more general condition in which the sequence of intervals {I_{N}}_{N≥1} is replaced by an arbitrary Følner sequence, i.e. a sequence of finite sets F_{N} ⊂ ℤ, N ≥ 1, such that for any a ∈ Z,

\frac{|(F_{N}+ a) ∩ F_{N}|}{|F_{N}|} → 1 as N → ∞.

This more general form of condition (i′) makes sense for any (countably infinite) amenable group and, as we shall see below (cf. Theorem 1.6), can be used to define the notion of weak mixing for actions of amenable groups. Remark 1.3. If (X,B,μ) is a separable space (which will be tacitly assumed from now on), the condition (ii) can be replaced by the following condition (see Theorem I in [KvN]):

(ii′) There exists a set P ⊂ ℕ of density zero such that for any A,B ∈ B, one has

lim_{n→∞,n∉P} μ(A ∩ T^{−n} B) = μ(A)μ(B).

Condition (ii) in Theorem 1.1 indicates the subtle but significant difference between weak and strong mixing: while for strong mixing one has μ(A ∩ T^{−n} B) → μ(A)μ(B) as n → ±∞ for any pair of measurable sets, a weakly mixing system which is not strongly mixing is characterized by the absence of mixing for some sets along some rarefied (i.e. having density zero) sequence of times. Although the first examples of weakly but not strongly mixing measure preserving transformations were quite complicated, numerous classes of measure preserving systems that satisfy this property are known by now. For instance, one can show that the so-called interval exchange transformations (IET) are often weakly mixing ([KS], [V]). On the other hand, A. Katok proved in [Ka] that the IET are never strongly mixing. It should be also mentioned here that weakly mixing measure preserving transformations are “typical”, whereas strongly mixing ones are not (see, for example, [H]). Before moving our discussion to weakly mixing actions of general groups, we would like to formulate some more recent results which exhibit new interesting facets of the notion of weak mixing. Theorem 1.4. Let T be an invertible measure-preserving transformation of a probability measure space (X,B,μ). The following conditions are equivalent:

(i) The transformation T is weakly mixing.

(ii) Weakly independent sets are dense in B. (Here a set A ∈ B is weakly independent if there exists a sequence n_{1} < n_{2} < ···such that the sets T^{−n_{i}} A, i ≥ 1, are mutually independent).

(iii) For any A ∈ B and k ∈ ℕ, k ≥ 2, one has

lim_{N→∞} \frac{1}{N} ∑_{n=0}^{N−1} μ(A ∩ T^{−n} A ∩ T^{−2n} A ∩ ··· ∩ T^{−kn}A) = (μ(A))^{k+1}.

(iv) For any k ∈ ℕ, k ≥ 2, any f_{1},f_{2},...,f_{k} ∈ L^{∞} (X,B,μ), and any non-constant polynomials p_{1}(n),p_{2}(n),...,p_{k}(n) ∈ ℤ[n] such that for all i ≠ j, deg(p_{i}− p_{j}) > 0 , one has

lim_{N→∞} \frac{1}{N} ∑_{n=0}^{N−1} f_{1}(T^{p_{1}(n)}x)f_{2}(T^{p_{2}(n)}x)··· f_{k}(T^{p_{k}(n)}x) = ∫ f_{1}dμ∫ f_{2}dμ··· ∫ f_{k}dμ

in L^{2}-norm. Remark 1.5. Condition (ii) is due to U. Krengel (see [Kr] for this and related results). Condition (iii) plays a crucial role in Furstenberg’s ergodic proof of Szemerédi’s theorem on arithmetic progressions (see [F1] and [F2]). Criterion (iv) was obtained in [Be1]. Similarly to the “linear” case (iii), the condition (iv) (or, actually, some variations of it) plays an important role in proofs of polynomial extensions of Szemerédi’s theorem (see [BeL1], [BeM1], [BeM2], [L]). Note that the assumption k ≥ 2 in (iii) and (iv) is essential. Indeed, for k = 1 condition (ii) expresses just the ergodicity of T, whereas for k = 1, condition (iv) is equivalent to the assertion that all non-zero powers of T are ergodic. The following equivalent form of condition (iv) is, however, both true and nontrivial already for k = 1 (cf. condition (ii′) in Remark 1.3):

(iv′) For any k ≥ 1 and any nonconstant polynomials p_{1}(n),...,p_{k}(n) ∈ ℤ[n] such that for all i ≠ j, deg(p_{i} − p_{j}) > 0, there exists a set P ⊂ ℕ having zero density such that for any sets A_{0},...,A_{k} ∈ B, one has

lim_{n→∞,n∉P} μ(A_{0}∩ T^{p_{1}(n)}A_{1}∩ ··· ∩ T^{p_{k}(n)}A_{k}) = μ(A_{0})μ(A_{1})...μ(A_{k}).

Theorems 1.1, 1.4, and numerous appearances and applications of weakly mixing one-parameter actions in ergodic theory hint that the notion of weak mixing could be of interest and of importance for actions of more general groups. One wants, of course, not only to be able to come up with a definition (this is not too hard: for example, condition (iii) in Theorem 1.1 makes sense for any group action), but also to be able to have, similarly to the case of one-parameter actions, many diverse equivalent forms of weak mixing including those which pertain to independence and higher degree mixing properties of the type given in Theorem 1.4.

Let (T_{g})_{g∈G} be a measure preserving action of a locally compact group G on a probability measure space (X,B,μ). If G is amenable, one can replace condition (i) in Theorem 1.1 (or, rather, condition (i′) in remark 1.2) by the assertion that the averages of the expressions |μ(A∩T_{g}B)−μ(A)μ(B)| taken along any Følner sequence in G converge to zero. If G is noncommutative, one also has to replace condition (v) by the assertion that the only finite-dimensional subrepresentation of (U_{g})_{g∈G} (where U_{g} is defined by (U_{g}f)(x) = f(T_{g}^{−1} x), f ∈ L^{2}(X,B,μ)) is the restriction to the subspace of constant functions. H. Dye has shown in [D] that under these modifications the conditions (i), (iii), and (v) in Theorem 1.1 are equivalent. Dye’s results are summarized in the following theorem (cf. [D], Corollary 1, p. 129). Again, for the sake of notational convenience, we state the theorem for the case of a countable group G. Theorem 1.6. Let (T_{g})_{g∈G} be a measure preserving action of a countable amenable group G on a probability measure space (X,B,μ). Then the following conditions are equivalent:

(i) For every Følner sequence (F_{n})_{n=1}^{∞} in G and any A,B ∈ B, one has

lim_{n→∞} \frac{1}{|F_{n}|} ∑_{g∈F_{n}}|μ(A ∩ T_{g}B) − μ(A)μ(B)| = 0.

(ii) The only finite dimensional subrepresentation of (U_{g})_{g∈G} is its restriction to the space of

constant functions. (iii) The diagonal action of (T_{g} × T_{g})_{g∈G} on the product space (X × X,B ⊗ B,μ ⊗ μ) is ergodic

(i.e. has no nontrivial invariant sets). Remark 1.7. As a matter of fact, it is not too hard to show that conditions (ii) and (iii) in Theorem 1.6 are equivalent for any locally compact noncompact second countable group. See, for instance, [Moore], Proposition 1, p. 157.

A measure preserving system (X,B,μ,T) is called a system with discrete spectrum if L^{2}(X,B,μ) is spanned by the eigenfunctions of the induced unitary operator U_{T}. It is not hard to show that the condition (v) in Theorem 1.1 implies that a measure preserving system (X,B,μ,T) is weakly mixing if and only if it does not have a nontrivial factor which is a system with discrete spectrum. Remark 1.7 hints that a natural generalization of this fact to general group actions holds as well. (A measure preserving action of a group G on a probability space (X,B,μ) has discrete spectrum if L^{2}(X,B,μ) is representable as a direct sum of finite-dimensional invariant subspaces.)

In [vN2] and [H] von Neumann and Halmos have shown that an ergodic one-parameter measure preserving action has discrete spectrum if and only if it is conjugate to an action by rotations on a compact abelian group. Again, this result has a natural extension to general group actions. See [Mac] for details and further discussion.

The duality between the notion of weak mixing and discrete spectrum extends to the relative case, namely, to the situation where one studies the properties of a system relatively to its factors. The theory of relative weak mixing is in the core of highly nontrivial structure theory developed by H. Furstenberg in the course of his proof ([F1]) of Szemerédi theorem. See also [FK1] and [F2], Chapter 6.

In [Z1] and [Z2] the duality between weak mixing and discrete spectrum is generalized to extensions of general group actions. In particular, Zimmer established a far reaching “relative” version of Mackey’s results on actions with discrete spectrum.

A useful interpretation of condition (i) in Theorem 1.6 is that if (T_{g})_{g∈G} is a weakly mixing action of an amenable group G, then for every A,B ∈ B and ε > 0, the set

R_{A,B}= {g ∈ G: |μ(A ∩ T_{g}B) − μ(A)μ(B)| < ε}

is large in the sense that it has density 1 with respect to any Følner sequence (F_{n})_{n=1}^{∞}:

lim_{n→∞} \frac{|R ∩ F_{n}|}{|F_{n}|} = 1.

A natural question that one is led to by this fact is whether there is a similar characterization of the sets R_{A,B}in the case when G is not necessarily amenable.

It turns out that for every locally compact group which acts in a weakly mixing fashion on a probability space, the set R_{A,B} is always “conull”, and in more than one sense. One approach, undertaken in [BeRo], is to utilize the classical fact that functions of the form ψ(g) = μ(A∩T_{g}A) are positive definite. This implies that such ψ(g), as well as a slightly more general functions of the form ϕ(g) = μ(A ∩ T_{g}B), are weakly almost periodic (see [Eb]). By a theorem of Ryll-Nardzewski (see [R-N]), there is a unique invariant mean on the space WAP(G) of weakly almost periodic functions. Denoting this mean by M and assuming that for every A,B ∈ B, the function g ↦ μ(A ∩ T_{g}B) is continuous on G, let us call the action (T_{g})_{g∈G} weakly mixing if for all

f_{1},f_{2} ∈ L_{0}^{2}(X,B,μ) =^{def} {f ∈ L^{2}(X,B,μ) : ∫_{X}fdμ = 0},

one has

M ( |∫_{X}f_{1}(x)f_{2}(T_{g}x)dμ(x)|) = 0.

Theorem 1.8. ([BeRo], Theorem 4.1) Let (T_{g})_{g∈G} be a measure preserving action of a locally compact second countable group G on a probability space (X,B,μ). The following are equivalent:

(i) (T_{g})_{g∈G} is weakly mixing. (ii) For every f_{1},f_{2} ∈ L^{2}(X,B,μ),

M ( |∫_{X}f_{1}(x)f_{2}(T_{g}x)dμ(x) − ∫ f_{1}dμ∫ f_{2}dμ| ) = 0.

(iii) For every f_{0},...,f_{n} ∈ L_{0}^{2}(X,B,μ) and ε > 0, there exists g ∈ G with

|∫_{X}f_{0}(x)f_{i}(T_{g}x)dμ(x)|< ε, i = 1,...,n.

(iv) For every g_{1},...,g_{n} ∈ G, f ∈ L_{0}^{2}(X,B,μ), and ε > 0, there exists g ∈ G such that

|∫_{X}f(T_{g}x)f(T_{g_{i}}x)dμ(x)|< ε, i = 1,...,n.

(v) For all F ∈ L^{2}(X,B,μ), where F is not equivalent to a constant, the set {f(T_{g}x) : g ∈ G} is not relatively compact in L^{2}(X,B,μ).

(vi) L_{0}^{2}(X,B,μ) contains no nontrivial finite dimensional invariant subspaces of (U_{g})_{g∈G}.

(vii) (T_{g}× T_{g})_{g∈G} is ergodic.

(viii) (T_{g}× T_{g})_{g∈G} is weakly mixing.

We shall describe now one more approach to weak mixing for general group actions (see [Be3], Section 4, for more details and discussion). Let G be a countably infinite, not necessarily amenable discrete group. For the purposes of the following discussion it will be convenient to view βG, the Stone-Čech compactification of G, as the space of ultrafilters on G, i.e. the space of {0,1}-valued finitely additive probability measures on the power set P (G) of G. Since elements of βG are {0,1}- valued measures, it is natural to identify each p ∈ βG with the set of all subsets having p-measure 1, and so we shall write A ∈ p instead of p(A) = 1. (This explains the terminology: ultrafilters are just maximal filters.) Given p,q ∈ G, one defines the product p · q by

A ∈ p · q ⇔ {x: Ax^{−1} ∈ p} ∈ q.

The operation defined above is nothing but convolution of measures, which, on the other hand, is an extension of the group operation on G. (Note that elements of G are in one-to-one correspondence with point masses, the so-called principal ultrafilters.) It is not hard to check that the operation introduced above is associative and that (βG,·) is a left topological compact semigroup (which, alas, is never a group for infinite G). For a comprehensive treatment of topological algebra in the Stone-Čech compactification, the reader is referred to [HiS]. By a theorem due to R. Ellis [El], any compact semigroup with a left continuous operation has an idempotent. (There are, actually, plenty of them since there are 2^{𝔠} disjoint compact semigroups in βG.) Idempotent ultrafilters find numerous applications in combinatorics (see, for example, [Hi] and [HiS], Part 3) and also are quite useful in ergodic theory and topological dynamics (see, for example, [Be2], [Be3]). Given an ultrafilter p ∈ βG and a sequence (x_{g})_{g∈G} in a compact Hausdorff space, one writes

p-lim_{g∈G}x_{g} = y

if for any neighborhood U of y, one has

{g ∈ G: x_{g} ∈ U} ∈ p.

Note that in compact Hausdorff spaces p-limit always exists and is unique.

The following theorem, which is an ultrafilter analogue of Theorem 1.7 from [FK2], illustrates the natural connection between idempotents in βG and ergodic theory of unitary actions. Theorem 1.9. Let (U_{g})_{g∈G} be a unitary action of a countable group G on a Hilbert space H. For any nonprincipal idempotent p ∈ βG and any f ∈ H one has

p-lim_{g∈G}U_{g}f = Pf (weakly)

where P is the orthogonal projection on the subspace H_{r} of p-rigid elements, that is, the space defined by

H_{r} = {f : p-limU_{g}f = f}.

Theorem 1.9 has a strong resemblance to the classical von Neumann’s ergodic theorem. In both theorems a generalized limit of U_{g}f, g ∈ G, (in case of von Neumann’s theorem this is the Cesáro limit) is equal to an orthogonal projection of f on a subspace of H. But while von Neumann’s theorem extends via Cesáro averages over Følner sets to amenable groups only, Theorem 1.9 holds for nonamenable groups as well.

Given an element p ∈ βG, it is easy to see that R = p·βG is a right ideal in βG (that is, R·βG ⊆ R). By using Zorn’s lemma one can show that any right ideal contains a minimal ideal. It is also not hard to prove that any minimal right ideal in a compact left topological semigroup is closed (see [Be3], Theorem 2.1 and Exercise 6). Now, by Ellis’ theorem, any minimal ideal in βG contains an idempotent. Idempotents belonging to minimal ideals are called minimal. It is minimal idempotents which allow one to introduce a new characterization of weak mixing for general groups. Recall that a set A ⊆ ℤ is called syndetic if it has bounded gaps and piecewise-syndetic if it is an intersection of a syndetic set with a union of arbitrarily long intervals. The following definition extends these notions to general semigroups. Definition 1.10. Let G be a (discrete) semigroup.

(i) A set A ⊆ G is called syndetic if for some finite set F ⊂ G, one has

⋃_{t∈F}At^{−1} = G.

(ii) A set A ⊆ G is piecewise syndetic if for some finite set F ⊂ G, the family

{ ( ⋃_{t∈F}At^{−1} ) a^{−1} : a ∈ G}

has the finite intersection property. The following proposition establishes the connection between minimal idempotents and certain

notions of largeness for subsets of G. It will be used below to give a new sense to the fact that for a weakly mixing action on a probability space (X,B,μ), the set R_{A,B}is large for all ε > 0 and A,B ∈ B. Theorem 1.11. (see [Be3], Exercise 7) Let G be a discrete semigroup and p ∈ (βG,·) a minimal idempotent. Then

(i) For any A ∈ p, the set B = {g: Ag^{−1} ∈ p} is syndetic.

(ii) Any A ∈ p is piecewise syndetic.

(iii) For any A ∈ p, the set

A^{−1} A = {x ∈ G: yx ∈ A for somey ∈ A}

is syndetic. (Note that if G is a group, then A^{−1} A = {g_{1}^{−1} g_{2} : g_{1},g_{2} ∈ A}.) Definition 1.12. A set A ⊆ G is called central if there exists a minimal idempotent p ∈ βG such that A ∈ p. A set A ⊆ G is called a C^{∗}-set (or central^{∗} set) if A is a member of any minimal idempotent in βG. Remark 1.13. The original definition of central sets (in ℤ), which is due to Furstenberg (see [F2], p. 161), was the following: a subset S ⊆ ℕ is a central set if there exists a system (X,T), a point x ∈ X, a uniformly recurrent point y proximal to x, and a neighborhood U_{y} of y such that S = {n : T^{n}x ∈ U_{y}}. The fact that central sets can be equivalently defined as members of minimal idempotents was established in [BeH]. See also Theorem 3.6 in [Be3].

The following theorem gives yet another characterization of the notion of weak mixing. Theorem 1.14. (see [Be3], Section 4) Let (T_{g})_{g∈G} be a measure preserving action of a countable group G on a probability space (X,B,μ). Then the following are equivalent:

(i) (T_{g})_{g∈G} is weakly mixing.

(ii) For every f ∈ L^{2}(X,B,μ) and any minimal idempotent p ∈ βG, one has

p-lim_{g∈G}f(T_{g}x) = ∫_{X}fdμ (weakly).

(iii) There exists a minimal idempotent p ∈ βG such that for any f ∈ L^{2}(X,B,μ), one has

p-lim_{g∈G} f(T_{g}x) = ∫_{X}fdμ (weakly). (iv) For any A,B ∈ B and any ε > 0, the set

{g ∈ G: |μ(A ∩ T_{g}B) − μ(A)μ(B)| < ε}

is a C^{∗}-set.

Given a weakly mixing action of, say, a countable (but not necessarily amenable) group G, one would like to know whether the action has higher order mixing properties along some massive and/or well-organized subsets of G. For example, it is not hard to show that for any weakly mixing ℤ-action and any nonconstant polynomial p(n) ∈ ℤ[n], one can find an IP-set S such that for any A,B ∈ B, one has

lim_{n→∞,n∈S} μ(A ∩ T^{p(n)}B) = μ(A)μ(B).

(An IP-set generated by a sequence {n_{i} : i ≥ 1} is, by definition, any set of the form {n_{i_{1}} +··· +n_{i_{k}} : i_{1} < ···< i_{k}; k ∈ ℕ}.)

Another example of higher degree mixing along structured sets is provided by a theorem proved in [BeRu], according to which any weakly mixing action of a countable infinite direct sum G = ⊕_{n≥1} ℤ_{p}, where ℤ_{p} is the field of residues modulo p, has the property that the restriction of the action of G to an infinite subgroup (which is isomorphic to G) is Bernoulli (see also [BeKM1], [BeKM2], [BeKLM], [JRW], [J], [B1]).

In Section 2 below we give a detailed analysis of higher order mixing properties for a concrete classical example — the standard action of SL(2,ℤ) on the 2-dimensional torus 𝕋^{2}. Since SL(2,ℤ) contains mixing automorphisms (namely, hyperbolic automorphisms), this action is weakly mixing. On the other hand, this action is not strongly mixing because SL(2,ℤ) contains nontrivial unipotent elements.

While many of the results obtained below hold (sometimes, after an appropriate modification) for toral actions of SL(n,ℤ) and even in more general situations, we intentionally deal here with SL(2,ℤ)-actions in order to make the paper more accessible and important issues more transparent.

Here is a sample of what is proved in the next section:

• (cf. Proposition 2.10) Let T_{1},...,T_{k} ∈ SL(2,ℤ). Then the following assertions are equivalent:

(i) For every A_{0},...,A_{k} ∈ B,

μ(A_{0}∩ T_{1}^{n}A_{1}∩ ··· ∩ T_{k}^{n}A_{k}) → μ(A_{0})··· μ(A_{k}) as n → ∞.

(ii) Each T_{i} is hyperbolic, T_{i} ≠ ±T_{j} for i ≠ j, and for every ρ > 1, there are at most two

matrices among T_{i}, i = 1,...,k, having an eigenvalue λ such that |λ| = ρ.

• (cf. Proposition 2.20) Let T_{1},...,T_{k} ∈ SL(2,ℤ) be hyperbolic automorphisms. Denote by λ_{i} the eigenvalue of T_{i} such that |λ_{i}| > 1. Put a_{0,n} = 0, n ≥ 1. Let k ≥ 1 and a_{i,n} ∈ ℤ, i = 1,...,k, be such that

min{|log|λ_{i}| · a_{i,n}− log|λ_{j}| · a_{j,n} |: 0 ≤ i < j ≤ n} → ∞ as n → ∞.

Then for every A_{0},...,A_{k} ∈ B,

μ(A_{0}∩ T_{1}^{a_{1,n}}A_{1}∩ ··· ∩ T_{k}^{a_{k,n}}A_{k}) → μ(A_{0})··· μ(A_{k}) as n → ∞.

This result generalizes Rokhlin’s theorem [R] in the case of 2-dimensional torus. See also Proposition 2.24 for an analogue of this result for unipotent automorphisms.

• While every abelian group of automorphisms G, which acts in a mixing fashion on 𝕋^{2}, is mixing of order k for every k ≥ 1, (that is, for every k ≥ 1 and sequences g_{0,n} = e, g_{1,n},...,g_{k,n}∈ G such that

g_{i,n}^{−1}g_{j,n} → ∞ as n → ∞ for 0 ≤ i < j ≤ k,

one has

μ(A_{0}∩ g_{1,n}A_{1}∩ ··· ∩ g_{k,n}A_{k}) → μ(A_{0})··· μ(A_{k}) as n → ∞)

a nonabelian group of automorphisms of 𝕋^{2} is never mixing of order 2 (see Proposition 2.31). Note that there are nonabelian groups of automorphisms that act in a mixing fashion on 𝕋^{2} (see the discussion after Proposition 2.30).

2. SL(2,ℤ)-action on torus

Definition 2.1. A sequence T_{n} ∈ SL(2,ℤ), n ≥ 1, is called mixing if for every f_{1},f_{2} ∈ L^{∞} (𝕋^{2}),

(1)  ∫_{𝕋^{2}} f_{1}(T_{n}ξ)f_{2}(ξ)dξ → ( ∫_{𝕋^{2}} f_{1}(ξ)dξ) ( ∫_{𝕋^{2}} f_{2}(ξ)dξ) as n → ∞.

A transformation T ∈ SL(2,ℤ) is called mixing if the sequence T^{n}, n ≥ 1, is mixing.

Note that this definition is different from the one given in [BBe].

Recall that a matrix T is called hyperbolic if its eigenvalues have absolute values different from 1, and unipotent if all its eigenvalues are equal to 1. It is well-known that an automorphism T ∈ SL(2,ℤ) is mixing on the torus 𝕋^{2} if and only if it is hyperbolic. This implies that the action of SL(2,ℤ) on 𝕋^{2} is weakly but not strongly mixing and motivates the following problem: give necessary and sufficient conditions for a sequence T_{n} ∈ SL(2,ℤ), n ≥ 1, to be mixing.

We start with a useful and straightforward lemma (cf. Theorem 3.1(1) in [B2]). For a matrix T, denote by^{t}T its transpose. Lemma 2.2. A sequence T_{n} ∈ SL(2,ℤ), n ≥ 1, is mixing if and only if for every (x,y) ∈ (ℤ^{2})^{2} − {(0,0)}, the equality^{t}T_{n}x + y = 0 holds for finitely many n only.

Proof. To prove that T_{n} is mixing, it is sufficient to check (1) for f_{1} and f_{2} in the dense subspace of trigonometric polynomials. It follows that T_{n} is mixing if and only if (1) holds for f_{1} and f_{2} that are characters of the form

(2)  χ_{x}(ξ) = e^{2πi⟨x,ξ⟩}, x ∈ ℤ^{2},ξ ∈ 𝕋^{2}.

For x,y ∈ ℤ^{2}, one has

∫_{𝕋^{2}} χ_{x}(T_{n}ξ)χ_{y}(ξ)dξ = ∫_{𝕋^{2}} χ_{^{t}Tnx+y}(ξ)dξ = \{ \begin{matrix}0 & if ^{t}T_{n}x + y ≠ 0, \\ 1 & if ^{t}T_{n}x + y = 0. \end{matrix}

It follows that for (x,y) ∈ (ℤ^{2})^{2}− {(0,0)},

∫_{𝕋^{2}} χ_{x}(T_{n}ξ)χ_{y}(ξ)dξ → ( ∫_{𝕋^{2}} χ_{x}(ξ)dξ) ( ∫_{𝕋^{2}} χ_{y}(ξ)dξ) = 0 as n → ∞

if and only if the equality^{t}T_{n}x + y = 0 holds for finitely many n only. This proves the lemma. □

Denote by M(2,𝕂) the set of 2×2-matrices over a field 𝕂. Using Lemma 2.2, we can now prove the following proposition. Proposition 2.3. Let T_{n} ∈ SL(2,ℤ), n ≥ 1, ∥ · ∥ be the max-norm on M(2,ℝ), and D ⊂ M(2,ℝ) denote the set of limit points of the sequence \frac{T_{n}}{∥T_{n}∥} as n → ∞. Then the sequence T_{n} is not mixing if and only if there exist A ∈ M(2,ℚ) and B ∈ M(2,ℚ) such that B ∈ D and T_{n} = A + ∥T_{n}∥B for infinitely many n ≥ 1.

Proof. We may assume that ∥T_{n}∥ → ∞. (Indeed, if ∥T_{n}∥ ↛ ∞, then there exists a matrix T_{0} such that T_{n} = T_{0} for infinitely many n, and the statement is obvious.)

“⇐”: Let T_{n} = A + ∥T_{n}∥B. Since

detB = lim_{n→∞} det( \frac{T_{n}}{∥T_{n}∥}) = lim_{n→∞} \frac{1}{∥T_{n}∥^{2}}= 0,

B is degenerate. Thus, there exists x ∈ ℤ^{2} − {0} such that ^{t}Bx = 0. Then for infinitely many n, tT_{n}x =^{t}Ax, and, by Lemma 2.2, T_{n} is not mixing.

“⇒”: By Lemma 2.2, there exists (x,y) ∈ (ℤ^{2})^{2}−{(0,0)} such that^{t}T_{n}x = −y for infinitely many n. By passing, if needed, to a subsequence, we may assume that this equality holds for all n ≥ 1. It is clear that gcd(x_{1},x_{2}) = gcd(y_{1},y_{2}). Thus, we may assume that x and y are primitive (that is, the gcd of their coordinates is 1). Take C,D ∈ SL(2,ℤ) such that Ce_{1} = x and De_{1} = −y where e_{1} = (1,0). Then

tT_{n} = D( \begin{matrix}1 & a_{n} \\ 0 & 1 \end{matrix} ) C^{−1} = DC^{−1} + a_{n}D( \begin{matrix}0 & 1 \\ 0 & 0 \end{matrix} ) C^{−1}

for some a_{n} ∈ ℤ. Put^{t}F_{1} = DC^{−1} and ^{t}F_{2} = D( \begin{matrix}0 & 1 \\ 0 & 0 \end{matrix} ) C^{−1} . We have

(3)  T_{n} = F_{1}+ a_{n}F_{2},

and

(4)  |a_{n}| · ∥F_{2}∥ − ∥F_{1}∥ ≤ ∥T_{n}∥ ≤ |a_{n}| · ∥F_{2}∥ + ∥F_{1}∥.

Hence, ∥T_{n}∥ ∼ |a_{n}| · ∥F_{2}∥ as n → ∞. Replacing, if necessary, F_{2} by −F_{2} and a_{n} by −a_{n} we may

assume that a_{n} > 0 for infinitely many n. Then B =^{def} \frac{F_{2}}{∥F_{2}∥} ∈ D. Passing to a subsequence, we get that a_{n} > 0 for n ≥ 1. By triangle inequality and (4),

∥T_{n}− ∥T_{n}∥\frac{F_{2}}{∥F_{2}∥}∥≤ ∥T_{n}− a_{n}F_{2}∥ + ∥a_{n}F_{2}− ∥T_{n}∥\frac{F_{2}}{∥F_{2}∥}∥= ∥F_{1}∥ + |a_{n}∥F_{2}∥ − ∥T_{n}∥|≤ 2∥F_{1}∥.

Thus, for infinitely many n, T_{n}−∥T_{n}∥B = A for some A ∈ M(2,ℚ). This proves the proposition. □

We illustrate the usefulness of Proposition 2.3 by the following two propositions. Proposition 2.4. Let U,V ∈ SL(2,ℤ) be unipotent matrices. Then the sequence T_{n} = U^{−n} V^{n} is mixing if and only if UV ≠ V U.

Proof. If U and V commute, one can show that they are powers of a single unipotent transformation. Hence, in this case, the sequence T_{n} = U^{−n} V^{n} is not mixing.

Conversely, suppose that UV ≠ V U. There exist A,B ∈ SL(2,ℤ) such that

U = A^{−1} ( \begin{matrix}1 & u \\ 0 & 1 \end{matrix} ) A and V = B^{−1} ( \begin{matrix}1 & v \\ 0 & 1 \end{matrix} ) B

for some u,v ∈ ℤ − {0}. It is sufficient to show that the sequence S_{n} = AT_{n}B^{−1} is mixing. Let

AB^{−1} = ( \begin{matrix}a & b \\ c & d \end{matrix} ) . We have

S_{n} = ( \begin{matrix}1 & −nu \\ 0 & 1 \end{matrix} ) AB^{−1} ( \begin{matrix}1 & nv \\ 0 & 1 \end{matrix} ) = ( \begin{matrix}a − (cu)n & b − (av + du)n − (cv)n^{2} \\ c & d + (cv)n \end{matrix} ) .

When c = 0,

V = B^{−1} ( \begin{matrix}1 & v \\ 0 & 1 \end{matrix} ) B = B^{−1} (AB^{−1} )^{−1} ( \begin{matrix}1 & v \\ 0 & 1 \end{matrix} ) (AB^{−1} )B = A^{−1} ( \begin{matrix}1 & v \\ 0 & 1 \end{matrix} ) A,

and it follows that U and V commute. Thus, c ≠ 0.

We apply now Proposition 2.3. For sufficiently large n, ∥S_{n}∥ = |b − (av + du)n − (cv)n^{2}|. Also

\frac{S_{n}}{∥S_{n}∥} −→ ( \begin{matrix}0 & −sign(cv) \\ 0 & 0 \end{matrix} ) =^{def} C.

Since S_{n}− ∥S_{n}∥C is not constant for infinitely many n, the sequence S_{n} is mixing. □

Remark 2.5. When U,V ∈ SL(2,ℤ) are commuting unipotent transformations, the sequence U^{−n} V^{n} is relatively mixing in the sense of Definition 2.22 below.

Using a similar argument, one proves the following proposition: Proposition 2.6. Let U,V ∈ SL(2,ℤ) such that U is unipotent, and V is hyperbolic. Then the sequence T_{n} = U^{−n} V^{n} is mixing.

Proof. Denote by E_{ij} the 2 × 2 matrix with 1 in position (i,j) and 0’s elsewhere. For some A,B ∈ GL(2,ℝ) and λ with |λ| > 1,

U = A^{−1} ( \begin{matrix}λ & 0 \\ 0 & λ^{−1} \end{matrix} ) A and V = B^{−1} ( \begin{matrix}1 & 1 \\ 0 & 1 \end{matrix} ) B.

We write

T_{n} = A^{−1} ( \begin{matrix}λ^{−n} & 0 \\ 0 & λ^{n} \end{matrix} ) AB^{−1} ( \begin{matrix}1 & n \\ 0 & 1 \end{matrix} ) B = λ^{−n} C + λ^{−n} nD + λ^{n}E + λ^{n}nF

where

C = A^{−1} E_{11}A, D = A^{−1} E_{11}AB^{−1} E_{12}B, E = A^{−1} E_{22}A, F = A^{−1} E_{22}AB^{−1} E_{12}B.

Suppose that F ≠ 0. Then \frac{T_{n}}{∥T_{n}∥} → \frac{F}{∥F ∥}. By Proposition 2.3, we need to show that there is no X ∈ M(2,ℝ) such that T_{n}− ∥T_{n}∥\frac{F}{∥F ∥} = X for infinitely many n. Since F is degenerate, one of the matrices C, D, E is not a scalar multiple of F (say C). Take a basis of M(2,ℝ) which contains C and F. The C-coordinate of T_{n}− ∥T_{n}∥\frac{F}{∥F ∥} with respect to this basis is equal to λ^{−n} + αλ^{−n} n + βλ^{n} for some α,β ∈ ℝ. This shows that the sequence T_{n}− ∥T_{n}∥\frac{F}{∥F ∥} consists of distinct matrices for sufficiently large n. Thus, T_{n} is mixing.

Suppose that F = 0. Then \frac{T_{n}}{∥T_{n}∥}→ \frac{E}{∥E∥}. By the same argument as in the previous paragraph, the sequence T_{n}− ∥T_{n}∥\frac{E}{∥E∥} consists of distinct matrices for sufficiently large n. This implies that T_{n} is mixing. □

Remark 2.7. When U,V ∈ SL(2,ℤ) are hyperbolic and U ≠ V, the sequence U^{−n} V^{n} is mixing. This follows from Proposition 2.10 below.

Next, we study multiple mixing for general sequences. Definition 2.8. Let T_{i,n} ∈ SL(2,ℤ), n ≥ 1, i=1,…,k. The sequences T_{1,n},...,T_{k,n} are jointly mixing if for every f_{i} ∈ L^{∞} (𝕋^{2}), i = 1,...,k + 1,

(5)  ∫_{𝕋^{2}} f_{1}(T_{1,n}ξ)··· f_{k}(T_{k,n}ξ)f_{k+1}(ξ)dξ → ( ∫_{𝕋^{2}} f_{1}(ξ)dξ) ··· ( ∫_{𝕋^{2}} f_{k+1}(ξ)dξ)

as n → ∞. Transformations T_{1},...,T_{k} are called jointly mixing if the sequences T_{1}^{n},...,T_{k}^{n}, n ≥ 1, are jointly mixing.

In [B2], this property was called w-jointly strongly mixing (see Definition 3.6 in [B2]).

In the course of proving Proposition 2.10 below, we shall need the following immediate extension of Lemma 2.2 (cf. Theorem 4.3(1) in [B2]). Lemma 2.9. Let T_{i,n} ∈ SL(2,ℤ), n ≥ 1, i=1,…,k. The sequences T_{1,n},...,T_{k,n} are jointly mixing if and only if for every (x_{1},...,x_{k+1}) ∈ (ℤ^{2})^{k+1} − {(0,...,0)} the equality

tT_{1,n}x_{1}+ ··· +^{t}T_{k,n}x_{k}+ x_{k+1} = 0

holds for finitely many n only. Proposition 2.10. Let T_{i} ∈ SL(2,ℤ), i = 1,...,k. The transformations T_{1},...,T_{k} are jointly mixing if and only if each of T_{i} is hyperbolic, T_{i} ≠ ±T_{j} for i ≠ j, and for every ρ > 1, there are at most two matrices among T_{i}, i = 1,...,k, having an eigenvalue λ such that |λ| = ρ.

Proof. If a matrix T ∈ SL(2,ℤ) has complex eigenvalues, they are units in an imaginary quadratic field. This implies that the eigenvalues of T are roots of unity. Hence, the transformation T is not mixing on 𝕋^{2}. Therefore, we may assume that the eigenvalues of T are real.

Next, we note that one can assume without loss of generality that the eigenvalues of T_{i}, i = 1,...,k, are positive. Indeed, put T̃_{i}= −T_{i} if the eigenvalues of T_{i} are negative and T_{i} otherwise. Clearly, transformations T_{i}, i = 1,...,k, are jointly mixing if and only if transformations T̃_{i}, i = 1,...,k, are jointly mixing.

Let the transformations T_{1},...,T_{k} be jointly mixing. Then each of the sequences T_{i}^{n} and T_{i}^{−n} T_{j}^{n}, i ≠ j, is mixing too. This implies that all T_{i} are hyperbolic and T_{i} ≠ T_{j} for i ≠ j. To show that the conditions of the theorem are necessary, we consider transformations T_{1},T_{2},T_{3} ∈ SL(2,ℤ) that have the same eigenvalue λ > 1. We claim that there exists (x_{1},x_{2},x_{3}) ∈ (ℤ^{2})^{3}− {(0,0,0)} such that

(6)  ^{t}T_{1}^{n}x_{1}+^{t}T_{2}^{n}x_{2}+^{t}T_{3}^{n}x_{3} = 0

for every n ≥ 1, which, in view of Lemma 2.9, implies that the sequences T_{1}^{n},T_{2}^{n},T_{3}^{n} are not jointly mixing.

Since T_{i}, i = 1,2,3, have the same eigenvalues, there exist A,B ∈ GL(2,ℝ) such that

(7)  T_{2} = A^{−1} T_{1}A and T_{3} = B^{−1} T_{1}B.

Note that the matrix A is a solution of the matrix equation

(8)  XT_{2} = T_{1}X,

which can be rewritten as a homogeneous system of linear equations with rational coefficients. The set of rational solutions of (8) is dense in the space of real solutions. It follows that there exists a rational solution (8) such that det(X) ≠ 0. This shows that we may choose A and B in GL(2,ℚ). For every v ∈ ℝ^{2}, v = v_{+} + v_{−} where v_{+} and v_{−} are eigenvectors of ^{t}T_{1} with eigenvalues λ and λ^{−1} respectively (λ > 1). Define linear maps P_{+} : v ↦ v_{+} and P_{−} : v ↦ v_{−} . Then

(9)  ^{t}T_{1} = λP_{+} + λ^{−1} P_{−} , P_{+} P_{−} = P_{−} P_{+} = 0, P_{±}^{2} = P_{±} , P_{+} + P_{−} = id.

Note that P_{+} ,P_{−} ∈ M(2,ℚ(\sqrt{d})) for some d ∈ ℕ determined by λ. When \sqrt{d} ∈ ℚ, λ and λ^{−1} are algebraic integers in ℚ, and it follows that that λ = ±1, which is a contradiction. Thus, \sqrt{d} ∉ ℚ. Denote by σ the nontrivial Galois automorphism of the field extension ℚ(\sqrt{d})/ℚ. Then λ^{σ} = λ^{−1} and (P_{+} )^{σ} = P_{−} . Using (7) and (9), we may rewrite equation (6) as

λ^{n}(P_{+} x_{1}+^{t}AP_{+} ^{t}A^{−1} x_{2}+^{t}BP_{+} ^{t}B^{−1} x_{3})+ λ^{−n} (P_{−} x_{1}+^{t}AP_{−} ^{t}A^{−1} x_{2}+^{t}BP_{−} ^{t}B^{−1} x_{3})= 0.

The columns of the matrices P_{+} , ^{t}AP_{+} ^{t}A^{−1} , and ^{t}BP_{+} ^{t}B^{−1} lie in the vector space ℚ(\sqrt{d})^{2} that has dimension 4 over ℚ. Thus, these columns are linearly dependent over ℚ, and there exists (x_{1},x_{2},x_{3}) ∈ (ℤ^{2})^{3}− {(0,0,0)} such that

P_{+} x_{1}+^{t}AP_{+} ^{t}A^{−1} x_{2}+^{t}BP_{+} ^{t}B^{−1} x_{3} = 0.

Applying σ to this equality, we get

P_{−} x_{1}+^{t}AP_{−} ^{t}A^{−1} x_{2}+^{t}BP_{−} ^{t}B^{−1} x_{3} = 0.

This implies (6) and proves that the conditions in the proposition are necessary for mixing.

To prove sufficiency consider S_{i},T_{i} ∈ SL(2,ℤ), i = 1,...,k, such that S_{i} and T_{i} have the same eigenvalue λ_{i} > 1, and λ_{i} < λ_{j} for i < j. We need to show that the transformations S_{1},T_{1},...,S_{k},T_{k} are jointly mixing. By Lemma 2.9, it is enough to prove that there is no (x_{1},y_{1},...,x_{k},y_{k},z) ∈ (ℤ^{2})^{2k+1} − {(0,...,0)} such that the equality

(10)  ^{t}S_{1}^{n}x_{1}+^{t}T_{1}^{n}y_{1}+ ··· +^{t}S_{k}^{n}x_{k}+^{t}T_{k}^{n}y_{k}+ z = 0.

holds for infinitely many n ≥ 1. Suppose that such a (2k+1)-tuple exists. Without loss of generality, we may assume that y_{k} ≠ 0. As above, we define P_{i,±} ,Q_{i,±} ∈ M(2,ℝ) such that

tS_{i} = λ_{i}P_{i,+} + λ_{i}^{−1} P_{i,−} , P_{i,+} P_{i,−} = P_{i,−} P_{i,+} = 0, P_{i,±}^{2} = P_{i,±} , P_{i,+} + P_{i,−} = id,
tT_{i} = λ_{i}Q_{i,+} + λ_{i}^{−1} Q_{i,−} , Q_{i,+} Q_{i,−} = Q_{i,−} Q_{i,+} = 0, Q_{i,±}^{2} = Q_{i,±} , Q_{i,+} + Q_{i,−} = id.

Then (10) can be rewritten as

∑_{i=1}^{k} λ_{i}^{−n} (P_{i,−} x_{i}+ Q_{i,−} y_{i}) + ∑_{i=1}^{k} λ_{i}^{n}(P_{i,+} x_{i}+ Q_{i,+} y_{i}) + z = 0.

Dividing this equality by λ_{k}^{n} and taking a limit over a subsequence n_{j} → ∞, we deduce that

P_{k,+}x_{k}+ Q_{k,+}y_{k} = 0.

Suppose that Q_{k,+}y_{k} = 0. Then y_{k} ≠ 0 is a rational eigenvector of S_{k} with eigenvalue λ_{k}^{−1} . It follows that λ_{k},λ_{k}^{−1} ∈ ℚ. On the other hand, λ_{k} and λ_{k}^{−1} are algebraic integers. Hence, λ_{k} = ±1, which is a contradiction. This shows that

v =^{def} P_{k,+}x_{k} = −Q_{k,+}y_{k} ≠ 0.

We have

S_{k}v = λ_{k}v = T_{k}v.

As above, we denote by σ_{k} the nontrivial automorphism of the quadratic extension ℚ(λ_{k})/ℚ. Then

P_{k,+}^{σ_{k}} = P_{k,−}, Q_{k,+}^{σ_{k}} = Q_{k,−}, λ_{k}^{σ_{k}} = λ_{k}^{−1} ,

and it follows that

S_{k}v^{σ_{k}} = λ_{k}^{−1} v^{σ_{k}} = T_{k}v^{σ_{k}}.

Since v and v^{σ_{k}} are linearly independent, this implies that S_{k} = T_{k}, which is a contradiction. Thus,

(10) holds for finitely many n only. The proposition is proved. □

Proposition 2.10 shows, in particular, that transformations T_{1},T_{2},T_{3} ∈ SL(2,ℤ) need not be jointly mixing even when every two of them are jointly mixing. Nonetheless, pairwise conditions are sufficient to imply mixing in the commutative situation. Proposition 2.11. Let T_{1},...,T_{k} ∈ SL(2,ℤ) be commuting automorphisms of 𝕋^{2}. Then they are jointly mixing if and only if the transformations T_{i} and T_{i}^{−1} T_{j}, i ≠ j, are mixing.

Proof. It is clear that if T_{1},...,T_{k} are jointly mixing, then T_{i} and T_{i}^{−1} T_{j}, i ≠ j, are mixing.

Conversely, suppose that T_{i} and T_{i}^{−1} T_{j}, i ≠ j, are mixing. Then T_{i} ≠ ±T_{j} for i ≠ j. Since T_{i} and T_{j} commute and are hyperbolic, they can be simultaneously reduced to the diagonal form. Thus, if T_{i} and T_{j} have the same eigenvalues of the same modulus, then T_{i} = ±T_{j}^{±1} . It follows that the conditions of Proposition 2.10 are satisfied and hence, T_{1},...,T_{k} are jointly mixing. □

One can show that the natural analog of Proposition 2.11 holds in every dimension.

In the case of a single measure preserving transformation T, T is mixing if and only if for every k ≥ 1, the transformation T^{k} is mixing. In the following proposition, we investigate what happens for general sequences in our group: Proposition 2.12. Let T_{n} ∈ SL(2,ℤ), n ≥ 1, be hyperbolic automorphisms. Let λ_{n} be the eigenvalue of T_{n} with |λ_{n}| > 1.

(1) For any k ≥ 1, if the sequence λ_{n} is bounded, then T_{n} is mixing if and only if T_{n}^{k} is mixing.

(2) For any k ≥ 2, if λ_{n} → ∞, the sequence T_{n}^{k} is always mixing.

Proof. Let t_{n} = Trace(T_{n}). Then T_{n} is a root of its characteristic polynomial x^{2}− t_{n}x + 1. Using the polynomial identity:

x^{k} = P(x)(x^{2}− t_{n}x + 1) + α_{n,k}x + β_{n,k}

where α_{n,k},β_{n,k}∈ ℤ,

α_{n,k}= \frac{λ_{n}^{k}− λ_{n}^{−k}}{λ_{n}− λ_{n}^{−1}} , β_{n,k}= \frac{λ_{n}^{−k+1} − λ_{n}^{k−1}}{λ_{n}− λ_{n}^{−1}} ,

we deduce that

(11)  T_{n}^{k} = α_{n,k}T_{n}+ β_{n,k}.

Suppose that λ_{n}, n ≥ 1, is bounded. Then the sequences α_{n,k}and β_{n,k}are bounded, hence take on only finitely many values. Hence, the equality

(12)  ^{t}T_{n}^{k}x + y =^{t}T_{n}(α_{n,k}x) + (β_{n,k}x + y) = 0

holds for some (x,y) ∈ (ℤ^{2})^{2}−{(0,0)} and infinitely many n if and only if the equality^{t}T_{n}x′+y′= 0 holds for some (x′,y′) ∈ (ℤ^{2})^{2}− {(0,0)} and infinitely many n. By Lemma 2.2, this proves the first part of the proposition.

We assume now that λ_{n} → ∞. Then

α_{n,k}∼ λ_{n}^{k−1} and β_{n,k}∼ −λ_{n}^{k−2} as n → ∞.

By Lemma 2.2, it is sufficient to show that if (12) holds for infinitely many n, then x = y = 0. Suppose that (12) holds for infinitely many n. Dividing by α_{n,k}and taking a limit over a subsequence n_{j} → ∞, we conclude that T_{n_{j}}x → 0. Since the sequence T_{n_{j}}x is discrete, it follows that x = 0, and y = 0. Thus, T_{n}^{k} is mixing. □

Remark 2.13. Note that the statement in part (2) of Proposition 2.12 fails for k = 1. For example, let

T_{n} = ( \begin{matrix}n & n − 1 \\ 1 & 1 \end{matrix} ) , n ≥ 1,

If λ_{n} denotes the largest eigenvalue of T_{n}, then clearly, λ_{n} → ∞. However, the sequence T_{n}, n ≥ 1, is not mixing. (This follows from Proposition 2.3.)

Recall a theorem of Rokhlin [R]: Theorem 2.14. (Rokhlin) Let T be a mixing automorphism of a compact abelian group. Then the sequences T^{a_{1,n}},...,T^{a_{k,n}} are jointly mixing provided that

min{|a_{i,n}− a_{j,n}|: 0 ≤ i < j ≤ n} → ∞ as n → ∞,

where a_{0,n} = 0.

The following proposition shows that a naive generalization of Rokhlin’s theorem to a general sequence of automorphisms T_{n} is false. Proposition 2.15. Let T_{n} ∈ SL(2,ℤ), n ≥ 1. Denote by λ_{n} the eigenvalue of T_{n} such |λ_{n}| > 1. If the sequence λ_{n}, n ≥ 1, is bounded, then for any choice of a_{i} ∈ ℤ, i = 1,...,k, (k > 1) the sequences T_{n}^{a_{1}},...,T_{n}^{a_{k}} are not jointly mixing.

Proof. Without loss of generality, we may assume that a_{i} > 0, i = 1,...,k.

By Lemma 2.9, it sufficient to show that there exists a tuple (x_{1},...,x_{k+1}) ∈ (ℤ^{2})^{k+1} −{(0,...,0)} such that for infinitely many n,

tT_{n}^{a_{1}}x_{1}+ ··· +^{t}T_{n}^{a_{k}}x_{k}+ x_{k+1} = 0.

By (12), the last equality reduces to

(13)  ^{t}T_{n}(α_{n,a_{1}}x_{1}+ ··· + α_{n,a_{k}}x_{k}) + β_{n,a_{1}}x_{1}+ ··· + β_{n,a_{k}}x_{k}+ x_{k+1} = 0.

Since the sequence λ_{n}, n ≥ 1, is bounded, the sequences α_{n,a_{i}} and β_{n,a_{i}} are bounded too. Thus, they are constant for infinitely many n. Now one can easily choose x_{i} ∈ ℤ^{2}, i = 1,...,k + 1, not all zero, such that (13) holds. For example, one can take all x_{i}’s to be multiples a fixed nonzero integer vector. □

Remark 2.16. Even if a sequence T_{n} ∈ SL(2,ℤ), n ≥ 1, is such that (i) T_{n} is hyperbolic and mixing on 𝕋^{2} for all n, (ii) λ_{n} → ∞, where λ_{n} is the eigenvalue of T_{n} such that λ_{n} > 1,

the sequences T_{n} and T_{n}^{2} need not be jointly mixing. For example, put

T_{n} = ( \begin{matrix}n & n^{2}− 1 \\ 1 & n \end{matrix} ) , n ≥ 1.

Then^{t}T_{n}^{2}x+^{t}T_{n}y+z = 0 for x =^{t}(0,1), y =^{t}(−2,0), z =^{t}(0,−2) which implies that the sequences T_{n} and T_{n}^{2} are not jointly mixing. On the other hand, it follows from Proposition 2.3 that the sequence T_{n} is mixing. This example also demonstrates that pairwise conditions are not sufficient to guarantee joint mixing even when the elements commute for every fixed n.

We give here a generalization of Rokhlin’s theorem in the case of the 2-dimensional torus. (A similar extension of Rokhlin’s theorem holds in any dimension.) Proposition 2.17. Let T_{n} ∈ SL(2,ℤ), n ≥ 1. Denote by λ_{n} the eigenvalue of T_{n} such that |λ_{n}| ≥ 1. Put a_{0,n} = 0, n ≥ 1. Let k ≥ 1 and a_{i,n} ∈ ℤ, i = 1,...,k. Denote

γ_{n} = min{|a_{i,n}− a_{j,n}|: 0 ≤ i < j ≤ n}.

Suppose that one of the following conditions holds:

(1) The sequence T_{n} is mixing, and

\{ \frac{∥T_{n}∥}{λ_{n}^{γ_{n}}}: n ≥ 1\} is bounded.

(2)

\frac{∥T_{n}∥}{λ_{n}^{γ_{n}}}→ 0 as n → ∞.

Then the sequences T_{n}^{a_{1,n}},...,T_{n}^{a_{k,n}} are jointly mixing. Remark 2.18. Part (2) of the theorem with T_{n} = T, n ≥ 1, implies Rokhlin’s theorem for the case of 2-dimensional torus.

Proof. Since T_{n}is measure-preserving, we are allowed to replace a_{i,n}by a_{i,n}−min{a_{i,n}: i = 0,...,k}. It follows that without loss of generality, we may assume that

min{a_{i,n}: i = 0,...,k} = 0.

Also by changing order and passing, if needed, to subsequences, we may assume that

max{a_{i,n}: i = 1,...,k} = a_{k,n}.

Suppose that the sequences T_{n}^{a_{1,n}},...,T_{n}^{a_{k,n}} are not jointly mixing. By Lemma 2.9, there exists a tuple (x_{1},...,x_{k+1}) ∈ (ℤ^{2})^{k+1} − {(0,...,0)} such that the equality

tT_{n}^{a_{1,n}}x_{1}+ ··· +^{t}T_{n}^{a_{k,n}}x_{k}+ x_{k+1} = 0

holds for infinitely many n. By (11), the last equality is equivalent to

∑_{i=1}^{k} α_{n,a_{i,n}}^{t}T_{n}x_{i}+ ∑_{i=1}^{k} β_{n,a_{i,n}}x_{i}+ x_{k+1} = 0.

Note that in both cases, λ_{n}^{γ_{n}} → ∞ as n → ∞. Therefore, it follows that λ_{n}^{a_{n,k}−a_{n,i}} → ∞.

α_{n,a_{i,n}} ∼ \frac{λ_{n}^{a_{i,n}}}{λ_{n}− λ_{n}^{−1}} , β_{n,a_{i,n}} ∼ \frac{−λ_{n}^{a_{i,n}−1}}{λ_{n}− λ_{n}^{−1}} , i = 1,...,k.

Then

tT_{n}x_{k} = − ∑_{i=1}^{k−1} \frac{α_{n,a_{i,n}}}{α_{n,a_{k,n}}} ^{t}T_{n}x_{i}− ∑_{i=1}^{k} \frac{β_{n,a_{i,n}}}{α_{n,a_{k,n}}} x_{i}− \frac{x_{k+1}}{α_{n,a_{k,n}}}
= O( \frac{∥T_{n}∥}{λ_{n}^{γ_{n}}}) + O(λ_{n}^{−γ_{n}}) + λ_{n}^{−1} x_{k}.

Assume that condition (1) holds. Then the sequence^{t}T_{n}x_{k} is bounded by infinitely many n. Thus, it is constant for infinitely many n. It follows from Lemma 2.2 that x_{k} = 0.

Suppose that condition (2) holds. We prove that x_{k} = 0. If λ_{n} → ∞, then^{t}T_{n}x_{k} → 0 as n → ∞, and this implies that x_{k} = 0. Otherwise, the sequences λ_{n} and ^{t}T_{n}x_{k} are bounded for infinitely many n, and consequently, they are constant for infinitely many n. Thus, ^{t}T_{n_{j}}x_{k} = λ_{n_{j}}^{−1} x_{k} for a subsequence n_{j}, and if x_{k} ≠ 0, then λ_{n_{j}},λ_{n_{j}}^{−1} ∈ ℚ. Since λ_{n_{j}} is an algebraic integer, λ_{n_{j}} = ±1, which contradicts condition (2). This shows that x_{k} = 0.

Now the proof can be completed by induction on k. □

Remark 2.19. Condition (1) is not necessary for joint mixing of the sequences T_{n}^{a_{1,n}},...,T_{n}^{a_{k,n}}. For example, put

T_{n} = ( \begin{matrix}n^{2} & n^{3}− 1 \\ 1 & n \end{matrix} ) , n ≥ 1, and a_{i,n} = i, i = 1,2.

Even though \frac{∥T_{n}∥}{λ_{n}}→ ∞ as n → ∞, one can check with the help of Lemma 2.9 that the sequences T_{n} and T_{n}^{2} are jointly mixing. It would be of interest to find a necessary and sufficient condition for joint mixing of sequences of the form T_{n}^{a_{1,n}},...,T_{n}^{a_{k,n}}.

The following proposition is yet another generalization of Rokhlin’s theorem. Proposition 2.20. Let T_{1},...,T_{k} ∈ SL(2,ℤ) be hyperbolic automorphisms. Denote by λ_{i} the eigenvalue of T_{i} such that |λ_{i}| > 1. Put a_{0,n} = 0, n ≥ 1. Let k ≥ 1 and a_{i,n} ∈ ℤ, i = 1,...,k be such that

(14)  min{|log|λ_{i}| · a_{i,n}− log|λ_{j}| · a_{j,n} |: 0 ≤ i < j ≤ n} → ∞ as n → ∞.

Then the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are jointly mixing.

Proof. As in the proof of Proposition 2.17, we reduce the proof to the case when

log|λ_{i+1} | · a_{i+1,n} − log|λ_{i}| · a_{i,n} → ∞ as n → ∞

for i = 0,...,k − 1.

Suppose that the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are not jointly mixing. By Lemma 2.9, there exists (x_{1},...,x_{k},y) ∈ (ℤ^{2})^{k+1} − {(0,...,0)} such that the equality

(15)  ^{t}T_{1}^{a_{1,n}}x_{1}+ ··· +^{t}T_{k}^{a_{k,n}}x_{k}+ y = 0

holds for infinitely many n. Let P_{i,+} ,P_{i,−} ∈ M(2,ℝ), i = 1,...,k, be such that

tT_{i} = λ_{i}P_{i,+} + λ_{i}^{−1} P_{i,−} , P_{i,+} P_{i,−} = P_{i,−} P_{i,+} = 0, P_{i,±}^{2} = P_{i,±} , P_{i,+} + P_{i,−} = id.

By (15),

(16)  ∑_{i=1}^{k} λ_{i}^{a_{i,n}}P_{i,+} x_{i}+ ∑_{i=1}^{k} λ_{i}^{−a_{i,n}} P_{i,−} x_{i}+ y = 0

holds by infinitely many n. Dividing by λ_{k}^{a_{k,n}} and taking limit over a subsequence n_{j} → ∞, we conclude that P_{k,+}x_{k} = 0.

If x_{k} ≠ 0, it is an eigenvector of ^{t}T_{k} with the eigenvalue λ_{k}^{−1} . This implies that λ_{k} ∈ ℚ. On the other hand, λ_{k} is an algebraic integer. Thus, λ_{k} = ±1. This contradiction shows that x_{k} = 0. Using induction on k, we deduce from (16) that x_{i} = 0 for i = 1,...,k. This shows that the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are jointly mixing. □

Remark 2.21. It clear that condition (14) in Proposition 2.20 follows from the following condition:

a_{1,n} → ∞ and \frac{a_{i+1,n}}{a_{i,n}} → ∞, i = 1,...,k, as n → ∞,

which also appears in Proposition 2.24. Definition 2.22. Let T_{i} ∈ SL(2,ℤ), i = 1,...,k. Denote by P_{i} : L^{2}(𝕋^{2}) → L^{2}(𝕋^{2}), i = 1,...,k, the orthogonal projection on the subspace of T_{i}-invariant functions. Let a_{i,n} ∈ ℤ, i = 1,...,k, n ≥ 1. We call the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} relatively jointly mixing if for every f_{i} ∈ L^{∞} (𝕋^{2}), i = 1,...,k + 1,

(17) ∫_{𝕋^{2}} f_{1}(T_{1}^{a_{1,n}}ξ)··· f_{k}(T_{k}^{a_{k,n}}ξ)f_{k+1}(ξ)dξ → ∫_{𝕋^{2}}(P_{1}f_{1})(ξ)··· (P_{k}f_{k})(ξ)f_{k+1}(ξ)dξ as n → ∞.

We have the following criterion for relative joint mixing of tuples of unipotent elements: Proposition 2.23. Let T_{i} ∈ SL(2,ℤ), i = 1,...,k, be unipotent elements. Denote by v_{i}, i = 1,...,k, a nonzero vector such that^{t}T_{i}v_{i} = v_{i}. Let a_{i,n}∈ ℤ, i = 1,...,k, n ≥ 1. Then the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are relatively jointly mixing if and only if for every (α_{1},...,α_{k}) ∈ ℤ^{k}−{(0,...,0)} and z ∈ ℤ^{2}−{0}, the equality

(18)  ∑_{i=1}^{k} α_{i}a_{i,n}v_{i}+ z = 0

holds for finitely many n only.

Proof. For some A_{i} ∈ SL(2,ℤ) and s_{i} ∈ ℤ − {0},

(19)  ^{t}T_{i}^{a_{i,n}} = A_{i}^{−1} ( \begin{matrix}1 & s_{i}a_{i,n} \\ 0 & 1 \end{matrix} ) A_{i} = E + s_{i}a_{i,n}B_{i}

where B_{i} = A_{i}^{−1} ( \begin{matrix}0 & 1 \\ 0 & 0 \end{matrix} ) A_{i} ∈ SL(2,ℤ), and E is the identity matrix. To establish relative mixing,

it is sufficient to check (17) in the case when f_{i}, i = 1,...,k, are characters of the form (2). For x_{1},...,x_{k+1} ∈ ℤ^{2}, one has

∫_{𝕋^{2}} χ_{x_{1}}(T_{1}^{a_{1,n}}ξ)··· χ_{x_{k}}(T_{k}^{a_{k,n}}ξ)χ_{x_{k+1}} (ξ)dξ = ∫_{𝕋^{2}} χ_{(^{t}T{a_{1,n} \atop 1}x_{1}+··· +^{t}T{a_{k,n} \atop k}x_{k}+x_{k+1} )}(ξ)dξ

= \{ \begin{matrix}1 & if ^{t}T_{1}^{a_{1,n}}x_{1}+ ··· +^{t}T_{k}^{a_{k,n}}x_{k}+ x_{k+1} = 0, \\ 0 & if ^{t}T_{1}^{a_{1,n}}x_{1}+ ··· +^{t}T_{k}^{a_{k,n}}x_{k}+ x_{k+1} ≠ 0. \end{matrix}

Note that for every x ∈ ℤ^{2},

P_{i}χ_{x} = \{ \begin{matrix}χ_{x} & if B_{i}x = 0, \\ 0 & if B_{i}x ≠ 0. \end{matrix}

Thus, (17) always holds for f_{i} = χ_{x_{i}} provided that B_{i}x_{i} = 0 for all i = 1,...,k. It follows that the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are relatively jointly mixing if and only if for every (x_{1},...,x_{k}) ∈ (ℤ^{2})^{k} such that for some i = 1,...,k, B_{i}x_{i} ≠ 0 (equivalently, T_{i}x_{i} ≠ x_{i}) the equality

(20)  ^{t}T_{1}^{a_{1,n}}x_{1}+ ··· +^{t}T_{k}^{a_{k,n}}x_{k}+ x_{k+1} = 0

holds for finitely many n only. By (19), the last equality is equivalent to

∑_{i=1}^{k} s_{i}a_{i,n}B_{i}x_{i}+ z = 0

where z = ∑ _{i=1}^{k+1} x_{i}. Note that the columns of the matrix B_{i} are rational multiples of the vector v_{i}. Thus, s_{i}B_{i}x_{i} = α_{i}v_{i} for some α_{i} ∈ ℚ. Since B_{i}x_{i} ≠ 0 for some i, (α_{1},...,α_{k}) ≠ (0,...,0). This shows that (20) holds if and only if

∑_{i=1}^{k} α_{i}a_{i,n}v_{i}+ z = 0

for some (α_{1},...,α_{k}) ∈ ℚ^{k}− {(0,...,0)} and z ∈ ℤ. Multiplying by a fixed integer, we get that α_{i} ∈ ℤ. This proves the proposition. □

We record here a convenient corollary of Proposition 2.23. Proposition 2.24. Let T_{i} ∈ SL(2,ℤ), i = 1,...,k, be unipotent elements, and a_{i,n} ∈ ℤ, i = 1,...,k, n ≥ 1, such that

a_{1,n} → ∞ and \frac{a_{i+1,n}}{a_{i,n}} → ∞, i = 1,...,k, as n → ∞.

Then the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are relatively jointly mixing.

Proof. Suppose that the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are not relatively jointly mixing. Then by Proposition 2.23, (18) holds for infinitely many n. Dividing (18) by a_{k,n} and taking the limit over a subsequence n_{s} → ∞, we deduce that α_{k} = 0. Similarly, it follows that α_{i} = 0 for i = 1,...,k. This shows that the sequences T_{1}^{a_{1,n}},...,T_{k}^{a_{k,n}} are relatively jointly mixing. □

Let T,S ∈ SL(d,ℤ). It was observed by Boshernitzan that it follows from the fact that the set of common periodic points of T and S is dense in 𝕋^{d} that for every nonempty open subset U of 𝕋^{d},

U ∩ T^{n}U ∩ S^{n}U ≠ ∅

for infinitely many n. A measurable analogue of this fact is far less trivial. The following conjecture seems plausible: Conjecture 2.25. Let T,S ∈ SL(d,ℤ), and let D be a Borel subset of 𝕋^{d} of positive measure. Then

limsup_{n→∞}μ(D ∩ T^{n}D ∩ S^{n}D) > 0.

In fact, in all known to us examples,

limsup_{n→∞}μ(D ∩ T^{n}D ∩ S^{n}D) ≥ μ(D)^{3}.

Remark 2.26. Note that when T and S generate a (virtually) nilpotent group, Conjecture 2.25 follows from a general “nilpotent” multiple recurrence theorem proved in [L] (see also Theorem E in [BeL2]). It was, however, proved in [BeL3], that for any finitely generated solvable group of exponential growth G, there exist a measure preserving action (T_{g})_{g∈G} on a probability space (X,B,μ), elements g_{1},g_{2} ∈ G and a set D ∈ B with μ(D) > 0 such that for T = T_{g_{1}} and S = T_{g_{2}}, one has μ(D ∩ T^{n}D ∩ S^{n}D) = 0 for all n ≠ 0. Nevertheless, we believe that for our special action of SL(d,ℤ) on 𝕋^{d}, the Conjecture is true.

We obtain below some partial results on the conjecture in the case of the 2-dimensional torus. Note that when T and S are hyperbolic the conjecture follows from Proposition 2.10. In fact, in this case,

lim_{n→∞} μ(D ∩ T^{n}D ∩ S^{n}D) = \{ \begin{matrix}μ(D)^{2} & if T = S, \\ μ(D)^{3} & if T ≠ ±S \end{matrix}

and when T = −S, the limit set of the sequence μ(D ∩T^{n}D ∩S^{n}D) consists of two numbers: μ(D)^{2}, μ(D ∩ −D)μ(D). In particular, this shows that liminfmight be 0 even when μ(D) > 0.

We can also settle the case when T and S are unipotent and hyperbolic respectively. For this, we need a lemma: Lemma 2.27. Let T ∈ SL(2,ℤ) be unipotent, and S ∈ SL(2,ℤ) hyperbolic. Then the sequences T^{n} and S^{n}, n ≥ 1, are relatively jointly mixing.

Proof. As in the proof of Proposition 2.23, it is sufficient to show that for every x,y,z ∈ ℤ^{2} such that either T x ≠ x or y ≠ 0, the equality

(21)  ^{t}T^{n}x +^{t}S^{n}y + z = 0

holds for finitely many n only. We have

tT^{n} = E + nB

where E is the identity matrix and B ∈ M(2,ℤ). Let λ be the eigenvalue of S such that |λ| > 1. For some P_{+} ,P_{−} ∈ M(2,ℝ),

tS^{n} = λ^{n}P_{+} + λ^{−n} P_{−} , P_{+} P_{−} = P_{−} P_{+} = 0, P_{±}^{2} = P_{±} , P_{+} + P_{−} = id.

Equality (21) is equivalent to

λ^{n}P_{+} y + λ^{−n} P_{−} y + nBx + (x + z) = 0.

Suppose that it holds for infinitely many n. Dividing by λ^{n} and taking the limit as n → ∞, we deduce that P_{+} y = 0. Then y is an eigenvector of S. If y ≠ 0, then y is a rational eigenvector of S, and λ and λ^{−1} are rational numbers that are algebraic integers. Hence, λ = ±1, which is a contradiction. This implies that y = 0. Then it follows that Bx = 0 (equivalently, T x = x). This shows that (21) holds for finitely many n only. Thus, the sequences T^{n} and S^{n}, n ≥ 1, are relatively jointly mixing. □

Lemma 2.27 implies the following special case of Conjecture 2.25. Proposition 2.28. Let T ∈ SL(2,ℤ) be unipotent, and S ∈ SL(2,ℤ) hyperbolic. Then for any measurable D ⊆ 𝕋^{2}, the limit of μ(D ∩ T^{n}D ∩ S^{n}D) as n → ∞ exists, and

lim_{n→∞} μ(D ∩ T^{n}D ∩ S^{n}D) ≥ μ(D)^{3}.

Moreover, the equality holds if and only if μ(D) = 1 or 0.

Proof. Let f be the characteristic function of the set D. Denote by P_{T} and P_{S} the orthogonal projections on the the spaces of T - and S-invariant functions respectively. Since S is ergodic, P_{S}f = μ(D). By Lemma 2.27,

lim_{n→∞} μ(D ∩ T^{n}D ∩ S^{n}D) = ∫_{𝕋^{2}} f(P_{T}f)(P_{S}f)dμ = μ(D)∥P_{T}f∥_{2}^{2} ≥ μ(D)^{3}.

□

In the case when T and S are unipotent, Conjecture 2.25 seems to be open in general. We prove a partial result for sets of special form. For a function f ∈ L^{∞} (𝕋^{d}), its Fourier coefficients are denoted by

f̂(x) = ∫_{𝕋^{d}} f(ξ)χ_{−x} (ξ)dξ, x ∈ ℤ^{d}.

Proposition 2.29. Let T,S ∈ SL(2,ℤ) be unipotent.

(1) For any measurable D ⊆ 𝕋^{2}, the limit of μ(D ∩ T^{n}D ∩ S^{n}D) as n → ∞ exists.

(2) Suppose that T S ≠ ST. Let A ∈ SL(2,ℤ) be such that A^{−1} T A is lower triangular unipotent. Then for every set of the form D = A(D_{1} × D_{2}) where D_{1} and D_{2} are measurable subsets of 𝕋^{1},

lim_{n→∞} μ(D ∩ T^{n}D ∩ S^{n}D) ≥ μ(D)^{3}.

Moreover, the equality holds if and only if μ(D) = 1 or 0.

Proof. We prove (1) in the case when T and S do not commute. (When T and S commute, they are powers of the same transformation, and the proof goes along the same lines as the proof below.)

Let v and w be primitive integer vectors such that^{t}T v = v and^{t}Sw = w. We claim that for f,g,h ∈ C^{∞} (𝕋^{2}),

(22)  ∫_{𝕋^{2}} f(ξ)g(T^{n}ξ)h(S^{n}ξ)dξ → ∑_{i,j∈ℤ} f̂(−iv − jw)ĝ(iv)ĥ(jw) as n → ∞.

It follows from a standard argument that it is sufficient to check (22) when f, g, h are characters of the form (2).

Let f = χ_{x}, g = χ_{y}, and h = χ_{z} for some x,y,z ∈ ℤ^{2}. First, suppose that x = −iv − jw, y = iv, z = jw for some i,j ∈ ℤ^{2}. Then

∫_{𝕋^{2}} f(ξ)g(T^{n}ξ)h(S^{n}ξ)dξ = ∫_{𝕋^{2}} χ_{x+^{t}T^{n}y+^{t}S^{n}z}(ξ)dξ = 1.

This implies (22) in this case.

Now we consider the case when x, y, z are not of the above form. We need to show that the equality x +^{t}T^{n}y +^{t}S^{n}z = 0 holds for finitely many n only. Suppose that it holds for infinitely many n. Write ^{t}T = E + B and ^{t}S = E + C where E is the identity matrix and B,C ∈ SL(2,ℤ) such that B^{2} = C^{2} = 0. Then

x +^{t}T^{n}y +^{t}S^{n}z = (x + y + z) + n(By + Cz) = 0.

holds for infinitely many n. This implies that x + y + z = 0 and By = −Cz. Note that the columns of matrix B are multiples of the vector v, and the columns of C are multiples of w. If By ≠ 0, v is multiple of w, and it follows that in some basis of ℝ^{2} both T and S are unipotent upper triangular. Then T S = ST, and this contradicts the initial assumption. Thus, By = Cz = 0. Equivalently, Ty = y and Sz = z. Hence, x = −iv − jv, y = iv, and z = jw for some i,j ∈ ℤ. This is a contradiction. We have proved (22).

Replacing T by A^{−1} T A and S by A^{−1} SA, we reduce the problem to the case when T is lower triangular and D = D_{1}× D_{2}. Then v =^{t}(1,0). Let w =^{t}(a,b). Let f be the characteristic function of the set D, and f_{1} and f_{2} be characteristic functions of the sets D_{1} and D_{2} respectively. Note that for s,t ∈ ℤ, f̂(s,t) = f̂_{1}(s)f̂_{2}(t). To prove part (2), we need to show that

∑_{i,j∈ℤ} f̂(−i − aj,−bj)f̂(i,0)f̂(aj,bj) ≥ μ(D)^{3}.

Using the Plancherel formula and the fact that f_{1}^{2} = f_{1}, we have

∑_{i,j∈ℤ} f̂(−i − aj,−bj)f̂(i,0)f̂(aj,bj)  = ∑_{j∈ℤ} ( ∑_{i∈ℤ} f̂_{1}(−aj − i)f̂_{1}(i)) f̂_{2}(−bj)f̂_{2}(0)f̂(aj,bj)
= ∑_{j∈ℤ} ˆ (f_{1}^{2})(−aj)f̂_{2}(−bj)μ(D_{2})f̂(aj,bj)
= μ(D_{2}) ∑_{j∈ℤ} |f̂_{1}(aj)|^{2}|f̂_{2}(bj)|^{2}
≥ μ(D_{2})|f̂_{1}(0)|^{2}|f̂_{2}(0)|^{2} ≥ μ(D)^{3}.

We are done. □

Next, we investigate mixing properties of subgroups of SL(2,ℤ). Proposition 2.30. Let H be a subgroup of SL(2,ℤ). The action of H on 𝕋^{2} is mixing if and only if H contains no nontrivial unipotent elements.

Proof. If the action of H is mixing, then the action of every infinite subgroup of H is mixing, and consequently, H does not contain nontrivial unipotent elements.

Conversely, suppose that the action of H is not mixing. By Lemma 2.2, there exists a sequence h_{n} ∈ H, n ≥ 1, and (x,y) ∈ (ℤ^{2})^{2}− {(0,0)} such that ^{t}h_{n}x = −y for every n ≥ 1 and h_{n} → ∞. Then th_{1}^{−1} ^{t}h_{n}x = x for every n ≥ 1. Thus, h_{n}h_{1}^{−1} ∈ H is a nontrivial unipotent element for sufficiently large n. This proves the proposition. □

A subgroup of SL(2,ℤ) is called nonparabolic if it contains no nontrivial unipotent elements. Nonparabolic subgroups are of interest from the point of view of ergodic theory because they are precisely the groups that act in a mixing fashion on the torus 𝕋^{2}. It follows from the pigeonhole principle that every subgroup of SL(2,ℤ) of finite index contains a nontrivial unipotent element. First examples of nonparabolic subgroups were constructed by B. H. Neumann in [N] (see also [Mag]). Any Neumann subgroup has the property that powers of a single unipotent element form a complete system of representatives of the cosets of this group. In particular, Neumann subgroups are maximal nonparabolic subgroups. There are examples of maximal nonparabolic subgroups that are not Neumann (see [T], [BrL1], [BrL2]). If F is a free normal subgroup of finite index in SL(2,ℤ) which is not equal to the commutant of SL(2,ℤ), then the commutant of F is nonparabolic (see [Mas]).

Although there are large subgroups in SL(2,ℤ) (e.g. Neumann subgroup) whose actions on the torus 𝕋^{2} are mixing, the following proposition shows that for nonabelian subgroups the higher order of mixing is impossible. Proposition 2.31. A nonabelian subgroup of SL(2,ℤ) cannot be mixing of order 2.

Proof. Let H be a nonabelian subgroup of SL(2,ℤ). Suppose that H is mixing of order 2. Take g,h ∈ H such that gh ≠ hg. Since H is mixing, g and h are hyperbolic. Note that if g^{2}h = hg^{2}, g^{2} and h can be both reduced to the diagonal form, and this would imply that g and h commute. Thus, g^{2}h ≠ hg^{2}. Put h_{i} = g^{−i} hg^{i}, i = 1,2,3. Comparing eigenvalues, we deduce that h_{i} ≠ −h_{j}. Also, h_{i} ≠ h_{j} for i ≠ j, since otherwise it would follow that g and h commute. Therefore, by Proposition 2.10, the transformations h_{i} and h_{j} are jointly mixing for i ≠ j. In particular, h_{i}^{−n} h_{j}^{n} → ∞ and h_{i}^{n} → ∞ as n → ∞. On the other hand, since h_{1}, h_{2}, h_{3} have the same eigenvalues, it follows from Proposition 2.10 (and its proof) that for some f_{1},f_{2},f_{3} ∈ L^{∞} (𝕋^{2}),

lim_{n→∞} ∫_{𝕋^{2}} f_{1}(h_{1}^{n}ξ)f_{2}(h_{2}^{n}ξ)f_{3}(h_{3}^{n}ξ)dξ ≠ ∫_{𝕋^{2}} f_{1}(ξ)dξ∫_{𝕋^{2}} f_{2}(ξ)dξ∫_{𝕋^{2}} f_{3}(ξ)dξ.

This implies that the sequences h_{1}^{−n} h_{2}^{n} and h_{1}^{−n} h_{3}^{n} are not jointly mixing, which, in its turn, contradicts the assumption that the group H is mixing of order 2. □

More generally, a similar argument allows one to show that the standard action on the torus 𝕋^{d} of a subgroup H of SL(d,ℤ) which is not virtually abelian can not be mixing of order d. Another approach to the proof of this fact can be found in [Bh] where it is utilized for derivation of isomorphism rigidity for the action of H. Remark 2.32. It should be noted that in contrast with the nonabelean situation, any mixing ℤ^{d}- action on 𝕋^{d} is mixing of all orders (see [S, Corollary 27.7]). On the other hand, if an SL(d,ℤ)-action is a restriction of an ergodic measure preserving SL(d,ℝ)-action, then by a theorem of S. Mozes (see [Moz]), it is mixing of all orders.

We conclude by proving a result of Krengel-type [Kr], which can be considered as a generalization of the fact that every ergodic automorphism of the torus has countable Lebesgue spectrum. Proposition 2.33. Let H be a subgroup of SL(2,ℤ) which acts in mixing fashion on 𝕋^{2}. Then for every f ∈ L^{2}(𝕋^{2}) and every ε > 0, there exist f_{0} ∈ L^{2}(𝕋^{2}) and a subgroup H_{0} of finite index in H such that

(23)  ∫_{𝕋^{2}} f_{0}(hξ)f̄_{0}(ξ)dξ = 0

for every h ∈ H_{0}, h ≠ e.

Proof. First, we note that since H is mixing, for every (x,y) ∈ (ℤ^{2})^{2}−{(0,0)} there is at most one h such that^{t}hx = y. Indeed, if^{t}h_{1}x =^{t}h_{2}x for some h_{1},h_{2} ∈ H, then h_{1}h_{2}^{−1} ∈ H is a unipotent element.

One can choose

f_{0} = ∑_{i=1}^{m} a_{i}χ_{x_{i}}

with some a_{i} ∈ ℂ and x_{i} ∈ ℤ^{2} such that ∥f − f_{0}∥_{2} < ε. We have

∫_{𝕋^{2}} f_{0}(hξ)f̄_{0}(ξ)dξ = ∑_{i,j=1}^{m} a_{i}ā_{j} ∫_{𝕋^{2}} χ_{^{t}hx_{i}−x_{j}} (ξ)dξ.

Let h_{ij} ∈ H − {e} be the unique element such that ^{t}h_{ij}x_{i} = x_{j} (if such an element exists). Since SL(2,ℤ) is finitely approximable, the subgroup H is finitely approximable too. There exists a subgroup H_{0} of finite index in H such that h_{ij} ∉ H_{0} for every i,j = 1,...,m. Then (23) holds for every h ∈ H_{0}, h ≠ e. This proves the proposition. □

Acknowledgments

We would like to thank D. Berend, N. Hindman, and R. Pavlov for useful comments on the preliminary version of the paper. We also would like to thank K. Schmidt for bringing Bhattacharya’s paper [Bh] to our attention.

References

[B1] B. Begun, Weakly wandering vectors for abelian group actions. To appear in Ergodic Theory Dynam. Systems.

[B2] D. Berend, Joint ergodicity and mixing. J. d’Analyse Math 45 (1985), 255–284.

[BBe] D. Berend and V. Bergelson, Ergodic and mixing sequences of transformations. Ergodic Theory Dynam. Systems 4 (1984), no. 3, 353–366.

[Be1] V. Bergelson, Weakly mixing PET. Ergodic Theory Dynam. Systems 7 (1987), no. 3, 337–349.

[Be2] V. Bergelson, Ergodic Ramsey theory—an update. Ergodic theory of ℤ^{d}-actions (Warwick, 1993–1994), editors: M. Pollicott and K. Schmidt, 1–61, London Math. Soc. Lecture Note Ser. 228, Cambridge Univ. Press, Cambridge, 1996.

[Be3] V. Bergelson, Minimal idempotents and ergodic Ramsey theory. Topics in dynamics and ergodic theory, editors: S. Bezuglyi and S. Kolyada, 8–39, London Math. Soc. Lecture Note Ser. 310, Cambridge Univ. Press, Cambridge, 2003.

[BeH] V. Bergelson and N. Hindman, Nonmetrizable topological dynamics and Ramsey theory. Trans. Amer. Math. Soc. 320 (1990), 293-320.

[BeKM1] V. Bergelson, I. Kornfeld, and B. Mityagin, Unitary ℤ^{d}-actions with continuous spectrum. Proc. Amer. Math. Soc. 119 (1993), no. 4, 1127–1134.

[BeKM2] V. Bergelson, I. Kornfeld, and B. Mityagin, Weakly wandering vectors for unitary actions of commutative groups. J. Funct. Anal. 126 (1994), no. 2, 274–304.

[BeKLM] V. Bergelson, I. Kornfeld, A. Leibman, and B. Mityagin, A Krengel-type theorem for finitely-generated nilpotent groups. Ergodic Theory Dynam. Systems 21 (2001), no. 5, 1359–1369.

[BeL1] V. Bergelson and A. Leibman, Polynomial extensions of van der Waerden’s and Szemerédi’s theorems. J. Amer. Math. Soc. 9 (1996), no. 3, 725–753.

[BeL2] V. Bergelson and A. Leibman, A nilpotent Roth theorem. Invent. Math. 147 (2002), no. 2, 429–470.

[BeL3] V. Bergelson and A. Leibman, Failure of Roth theorem for solvable groups of exponential growth. To appear in Ergodic Theory Dynam. Systems.

[BeM1] V. Bergelson and R. McCutcheon, Uniformity in the polynomial Szemerédi theorem. Ergodic theory of ℤ^{d}-actions (Warwick, 1993–1994), editors: M. Pollicott and K. Schmidt, 273–296, London Math. Soc. Lecture Note Ser. 228, Cambridge Univ. Press, Cambridge, 1996.

[BeM2] V. Bergelson and R. McCutcheon, An ergodic IP polynomial Szemerédi theorem. Mem. Amer. Math. Soc. 146 (2000), no. 695.

[BeRo] V. Bergelson and J. Rosenblatt, Mixing actions of groups. Illinois J. Math. 32 (1988), no. 1, 65–80.

[BeRu] V. Bergelson and D. Rudolph, Weakly mixing actions of F^{∞} have infinite subgroup actions which are Bernoulli. Dynamical systems (College Park, MD, 1986–87), editor: J. Alexander, 7–22, Lecture Notes in Math. 1342, Springer, Berlin, 1988.

[Bh] S. Bhattacharya, Higher order of mixing and rigidity of algebraic actions on compact abelian groups. To appear in Israel Journal of Mathematics.

[Bi] G. D. Birkhoff, Proof of the ergodic theorem. Proc. Nat. Acad. Sci. USA 17 (1931), 656–660.

[BrL1] J. L. Brenner and R. C. Lyndon, Nonparabolic subgroups of the modular group. J. Algebra 77 (1982), no. 2, 311–322.

[BrL2] J. L. Brenner and R. C. Lyndon, Maximal nonparabolic subgroups of the modular group. Math. Ann. 263 (1983), no. 1, 1–11.

[D] H. Dye, On the ergodic mixing theorem. Trans. Amer. Math. Soc. 118 (1965), 123–130.

[Eb] W. Eberlein, Abstract ergodic theorems and weak almost periodic functions. Trans. Amer. Math. Soc. 67 (1949), 217– 240.

[El] R. Ellis, Distal transformation groups. Pacific J. Math. 8 (1958), 401–405.

[F1] H. Furstenberg, Ergodic behavior of diagonal measures and a theorem of Szemerédi on arithmetic progressions. J. d’Analyse Math. 31 (1977), 204–256.

[F2] H. Furstenberg, Recurrence in ergodic theory and combinatorial number theory. Princeton University Press, 1981.

[FK1] H. Furstenberg and Y. Katznelson, An ergodic Szemerédi theorem for commuting transformations. J. d’Analyse Math. 34 (1978), 275–291.

[FK2] H. Furstenberg and Y. Katznelson, An ergodic Szemerédi theorem for IP-systems and combinatorial theory. J. d’Analyse Math. 45 (1985), 117–168.

[H] P. Halmos, Lectures on ergodic theory. Chelsea Publishing Co., New York, 1960.

[HvN] P. R. Halmos and J. von Neumann, Operator Methods in Classical Mechanics. II, Ann. Math. (2) 43 (1942), 332-350.

[He] H. Helson, Harmonic analysis. Addison-Wesley, Reading, MA, 1983.

[Hi] N. Hindman, Ultrafilters and combinatorial number theory. Number theory, Carbondale 1979 (Proc. Southern Illinois Conf., Southern Illinois Univ., Carbondale, Ill., 1979), pp. 119–184, Lecture Notes in Math. 751, Springer, Berlin, 1979.

[HiS] N. Hindman and D. Strauss, Algebra in the Stone-Čech compactification. Theory and applications. De Gruyter Expositions in Mathematics 27. Walter de Gruyter & Co., Berlin, 1998.

[Ho1] E. Hopf, Complete transitivity and the ergodic principle. Proc. Nat. Acad. Sci. USA 18 (1932), 204–209.

[Ho2] E. Hopf, Proof of Gibbs’ hypothesis on the tendency toward statistical equilibrium. Proc. Nat. Acad. Sci. USA 18 (1932), 333–340.

[Ho3] E. Hopf, Ergodentheorie. Ergebnisse der Mathematik und ihrer Grenzgebiete 5, Springer, Berlin, 1937.

[J] A. del Junco, Vectors with orthogonal iterates along IP-sets in unitary group actions. Ergodic Theory Dynam. Systems 22 (2002), no. 1, 227–243.

[JRW] A. del Junco, K. Reinhold, and B. Weiss, Partitions with independent iterates along IP-sets. Ergodic Theory Dynam. Systems 19 (1999), no. 2, 447–473.

[Ka] A. Katok, Interval exchange transformations and some special flows are not mixing. Israel J. Math. 35 (1980), no. 4, 301–310.

[KS] A. Katok and A. Stepin, Approximations in ergodic theory. Uspehi Mat. Nauk 22 (1967) 81–106; Russian Math. Surveys 22, 77–102.

[Ko] A. N. Kolmogorov, Ein vereinfacher Beweis des Birkhoff-Khintchineschen Ergodensatzes. Recueil Math. (Mat. Sborn.) 2(44) (1937), 367–368.

[Kr] U. Krengel, Weakly wandering vectors and weakly independent partitions. Trans. Amer. Math. Soc. 164 (1972), 199– 226.

[KvN] B.O. Koopman and J. von Neumann, Dynamical systems of continuous spectra. Proc. Nat. Acad. Sci. USA 18 (1932), 255–263.

[L] A. Leibman, Multiple recurrence theorem for measure preserving actions of a nilpotent group. Geom. Funct. Anal. 8 (1998), no. 5, 853–931.

[Mac] G. Mackey, Ergodic transformation groups with a pure point spectrum. Illinois J. Math. 8 (1964), 593–600.

[Mag] W. Magnus, Noneuclidean tesselations and their groups. Pure and Applied Mathematics, Vol. 61. Academic Press, New York-London, 1974.

[Mas] A. W. Mason, Lattice subgroups of free congruence groups. Glasgow Math. J. 10 (1969), 106–115.

[Moore] C. Moore, Ergodicity of flows on homogeneous spaces. Amer. J. Math. 88 (1966), 154–178.

[Moz] S. Mozes, Mixing of all orders of Lie groups actions. Invent. Math. 107 (1992), no. 2, 235–241; Erratum: Invent. Math. 119 (1995), no. 2, 399.

[N] B. H. Neumann, Uber ein gruppentheoretisch-arithmetisches Problem. S.-B. Preus. Akad. Wiss. Phys.-Math. Kl. 7/10 (1933), 429–444.

[R] V. A. Rokhlin, On endomorphisms of compact commutative groups. (Russian) Izvestiya Akad. Nauk SSSR. Ser. Mat. 13 (1949), 329–340.

[R-N] C. Ryll-Nardzewski, On fixed points of semigroups of endomorphisms of linear spaces, Proc. Fifth Berkeley Sympos.

Math. Statist. and Probability (Berkeley, Calif., 1965/66), Vol. II: Contributions to Probability Theory, Part 1, 55–61.

[S] K. Schmidt, Dynamical systems of algebraic origin. Progress in Mathematics 128. Birkhäuser Verlag, Basel, 1995.

[T] C. Tretkoff, Nonparabolic subgroups of the modular group. Glasgow Math. J. 16 (1975), no. 2, 91–102.

[V] W. Veech, The metric theory of interval exchange transformations. I. Generic spectral properties. Amer. J. Math. 106 (1984), no. 6, 1331–1359.

[vN1] J. von Neumann, Proof of the quasiergodic hypothesis. Proc. Nat. Acad. Sci. USA 18 (1932), 70–82.

[vN2] J. von Neumann, Zur Operatorenmethode in der klassischen Mechanik. Ann. of Math. (2) 33 (1932), 587–642.

[Z1] R. Zimmer, Extensions of ergodic group actions. Illinois J. Math. 20 (1976), no. 3, 373–409.

[Z2] R. Zimmer, Ergodic actions with generalized discrete spectrum. Illinois J. Math. 20 (1976), no. 4, 555–588.