OPPENHEIM CONJECTURE FOR PAIRS CONSISTING OF A LINEAR FORM AND A QUADRATIC FORM

ALEXANDER GORODNIK

Abstract. Let Q be a nondegenerate quadratic form, and L is a nonzero linear form of dimension d > 3. As a generalization of the Oppenheim conjecture, we prove that the set {(Q(x), L(x)) : x ∈ ℤ^{d}} is dense in ℝ^{2} provided that Q and L satisfy some natural conditions. The proof uses dynamics on homogeneous spaces of Lie groups.

1. Introduction

It was proved by Margulis [Ma89] (see [Ma97] for an up-to-date survey) that if Q is a real indefinite nondegenerate quadratic form of dimension d ≥ 3 which is not proportional to a rational form, then {Q(x) : x ∈ ℤ^{d}} is dense in ℝ. A similar problem was considered for pairs (Q, L) where Q is a quadratic form as above, and L is a nonzero linear form. The known results are limited to dimension 3. It was proved by Dani and Margulis [DM90] that {(Q(x), L(x)) : x ∈ ℤ^{3}} is dense in ℝ^{2} if no nonzero linear combination of Q and L^{2} is rational, and the plane {L = 0} is tangent to the surface {Q = 0}. Clearly, the first condition is neccesary for the density to hold. The second condition guarantees that the stabilizer in SL(3, ℝ) of the pair (Q, L) is unipotent, so that the results on orbits of unipotent flows can be used. One can hope to remove the second condition. However, Dani proved in [Da00] that if the surface {Q = 0} and the plane {L = 0} intersect transversally, the density can fail for a set pairs of full Hausdorff dimension. On the other hand, it is easy to see using Moore ergodicity criterion that the density holds for a set of pairs of full measure provided that the surfaces {Q = 0} and {L = 0} have nonzero intersection.

Denote by P(ℤ^{d}) the set of primitive integer vectors in ℤ^{d}. The results mentioned above still hold when ℤ^{d} is replaced by P(ℤ^{d}).

In this paper we prove a density result for pairs consisting of a linear form and a quadratic form of dimension d ≥ 4: Theorem 1. Let Q be a nondegenerate quadratic form, and L be a nonzero linear form in dimension d ≥ 4. Suppose that

1. Q|_{L=0} is indefinite.

2. For every (α, β) ≠ (0, 0), with α, β ∈ ℝ, αQ + βL^{2} is not rational.

Then {(Q(x), L(x)) : x ∈ P(ℤ^{d}}) is dense in ℝ^{2}.

For d = 3, condition (1) implies that the surfaces {Q = 0} and {L = 0} intersect transversally. Therefore, it follows from the result of Dani [Da00] mentioned above that the analogue of Theorem 1 does not hold for d = 3.

2000 Mathematics Subject Classification. Primary 11J13, 11H55, 37A17.

This article is a part of author’s PhD thesis at Ohio State University done under supervision of Prof. Bergelson.

It is easy to see that the second condition is neccesary for the conclusion of the theorem to hold. A condition similar to the first condition is required to insure that

{(Q(x), L(x)) : x ∈ ℝ^{d}} = ℝ^{2}.

It is possible that the first condition can be weakened (see Conjecture 15 below).

Acknowledgment: The author is deeply grateful to his advisor V. Bergelson for suggesting the problem as well as for many helpful discussions.

2. Canonical forms

Let Q_{i}, i = 1, 2, be quadratic forms, and L_{i}, i = 1, 2, be linear forms of dimension d. We say that the pairs (Q_{1}, L_{1}) and (Q_{2}, L_{2}) are equivalent if Q_{1}(x) = λQ_{2}(g·x) and L_{1}(x) = μL_{2}(g·x) for some λ, μ ∈ ℝ − {0} and g ∈ SL(n, ℝ). That is, they can be transformed into each other by a linear change of coordinates and scaling. Proposition 2. Every pair (Q, L), where Q is a nondegenerate quadratic form, and L is a nonzero linear form, is equivalent to one and only one of the following pairs:

(I) (x_{1}^{2} + … + x_{s}^{2} − x_{s+1}^{2} − … − x_{d}^{2}, x_{d})

where s = 1, …, d. (II) (x_{1}^{2} + … + x_{s}^{2} − x_{s+1}^{2} − … − x_{d−2}^{2} + x_{d−1}x_{d}, x_{d}) where s = 0, …, [\frac{d−2}{2}].

Proof. Without loss of generality, L(x) = x_{d}. Applying a linear transformation in x_{1}, …, x_{d−1}, we transform Q to the form

∑_{i≤k} ±x_{i}^{2} + (∑_{j} a_{j}x_{j}) x_{d} for some k ≤ d − 1.

Using linear transformations x_{i} → x_{i} ± \frac{a_{i}}{2} x_{d} for i ≤ k, we can make a_{i} = 0 for i ≤ k. If a_{i} = 0 for all i ≠ d, then k = d − 1 because Q is nondegenerate. Thus, we are in the case (I). Let a_{l} ≠ 0 for some l < d. Making a linear change of variables x_{l} ← ∑_{j} a_{j}x_{j}, we get

Q = ∑_{i≤k} ±x_{i}^{2} + x_{l}x_{d}.

Since Q is nondegenerate, k = d − 2 and l = d − 1. This is the case (II). Note that a pair (Q, L) is of type (I) iff Q + αL^{2} is degenerate for some α ∈ ℝ. In particular, this shows that pairs of type (I) and (II) are not equivalent. Signatures of quadratic forms Q and Q|_{L=0} are invariants of equivalence with one possible exception that pairs with signatures (s, d − s) and (d − s, s) could be equivalent. For example, pairs of type (I) with s = 0 and s = d are equivalent. Let (Q, L) be as in (I) with s = i for i ≠ 0, d. Then Q|_{L=0} has signature (i, d − i − 1). On the other hand, when s = d − i, Q|_{L=0} has signature (d − i, i − 1). Clearly, these two cases are not equivalent unless n = 2i, and they coincide. When (Q, L) is of type (II), it is easy to see that the cases s = i and s = d − 2 − i are equivalent for i = 0, …, d − 2. This finishes the proof. □

Using the explicit formulas for canonical forms, it is easy to prove the following lemma: Lemma 3. Let Q be a nondegenerate quadratic form, and L is a nonzero linear form such that Q|_{L=0} is indefinite. Then

{(Q(x), L(x)) : x ∈ ℝ^{d}} = ℝ^{2}.

3. Reduction to lower dimension

In this section, we prove Proposition 4, which reduces the proof of Theorem 1 to the case of dimension 4. Proposition 4. Let (Q, L) satisfy the conditions of Theorem 1. If the dimension of the space is greater than 4, then there exists a rational subspace U of codimension 1 such that (Q|_{U}, L|_{U}) satisfies the same conditions. Moreover, U can be chosen such that (Q|_{U}, L|_{U}) is of type (I).

Consider the set Ω = {(ℓ_{1}, ℓ_{2}, ℓ_{3})} ⊂ ℝ^{3d} of triples of linear forms of dimension d defined by the following conditions:

1. ℓ_{1}, ℓ_{2}, ℓ_{3} are linearly independent.

2. L ≠ 0 on the space {ℓ_{i} = 0 : i = 1, 2, 3}.
3. Q is nondegenerate on the space {ℓ = 0 : ℓ ∈ T } for any subset T ⊆ {ℓ_{1}, ℓ_{2}, ℓ_{3}}.
4. (Q|_{ℓ_{i}=0}, L|_{ℓ_{i}=0}) is of type (I) for i = 1, 2, 3.
5. Q|_{{ℓ_{i}=0}∩{L=0}} is indefinite for i = 1, 2, 3.

Lemma 5. The set Ω is nonempty and open.

Proof. Let Ω_{0} be the set of triples of linear forms that satisfy conditions (1) and (2), and Ω_{j} the set of triples in Ω_{0} that satisfy condition (j), j = 3, 4, 5. Note that the set Ω_{0} is the complement of a proper algebraic set (i.e. a set defined by polynomial equations).

The sets Ω_{j}, j = 3, 4, 5, are not empty. This is easy to see using the canonical forms from Proposition 2. For example, when (Q, L) is the canonical form (II), (ℓ_{1}, ℓ_{2}, ℓ_{3}) ∈ Ω_{4} for ℓ_{i}(x̄) = x_{d−4−i} + x_{d−1} + x_{d}, i = 1, 2, 3. It is also clear that the sets Ω_{j}, j = 3, 4, 5, are open. To show this for Ω_{4}, we use that a pair (Q, L) is of type (I) iff

det(Q) ≠ 0, det(Q|_{L=0}) ≠ 0, L ≠ 0.

It suffices to show that Ω_{3} and Ω_{4} are dense. Indeed, then it follows that Ω_{3} ∩ Ω_{4} is an open dense set, and Ω = Ω_{3} ∩ Ω_{4} ∩ Ω_{5} is nonempty and open.

We claim that for every T ⊆ {ℓ_{1}, ℓ_{2}, ℓ_{3}}, the set of triples (ℓ_{1}, ℓ_{2}, ℓ_{3}) ∈ Ω_{0} such that Q is nondegenerate on the space {ℓ = 0 : ℓ ∈ T } contains the complement of a proper algebraic subset. This will imply that the set Ω_{3} is dense. To simplify notations, we only consider the case of one linear form. Namely, we show that the set of nonzero linear forms ℓ such that Q|_{ℓ=0} is nondegenerate contains the complement of a proper algebraic subset. The cases of pairs and triples are handled similarly. For a nonzero linear form ℓ, there exists a basis {v_{i}^{ℓ} : i = 1, …, d − 1} of the space {ℓ = 0} such that the coordinates of the vectors v_{i}^{ℓ} are rational functions of the coefficients of ℓ. Define a rational function

ϕ(ℓ) =^{def} det (Q(v_{i}^{ℓ}, v_{j}^{ℓ}) : i, j = 1, …, d − 1) .

For a linear form ℓ such that ϕ(ℓ) is defined, Q|_{ℓ=0} is nondegenerate iff ϕ(ℓ) ≠ 0. If ϕ = 0 on its domain, Q|_{ℓ=0} is nondegenerate only for ℓ in a proper algebraic subset where ϕ is undefined. This is a contradiction because Ω_{3} is nonempty and open. Hence, ϕ ≠ 0. The form Q|_{ℓ=0} is nondegenerate for ℓ ∈ {ℓ : ϕ(ℓ) ≠ 0}. This set is the complement of a proper algebraic set.

To prove that Ω_{4} is dense, we show that the set of nonzero linear forms ℓ such that (Q|_{ℓ=0}, L|_{ℓ=0}) is of type (I) contains the complement of a proper algebraic subset. The form (Q|_{ℓ=0}, L|_{ℓ=0}) is of type (I) iff

det(Q|_{ℓ=0}) ≠ 0, det(Q|_{{L=0,ℓ=0}}) ≠ 0, L|_{ℓ=0} ≠ 0.

Note that the first inequality and the third inequality define sets that contain the complements of proper algebraic subsets. Hence, it is enough to prove that the set of linear forms ℓ that satisfy the second inequality contains the complement of a proper algebraic subset. There exists a basis {w_{i}^{ℓ} : i = 1, …, d − 2} of the space {L = 0, ℓ = 0} such that the coordinates of the basis vectors are rational functions of the coefficients of ℓ. Define a rational function

ψ(ℓ) =^{def} det (Q(w_{i}^{ℓ}, w_{j}^{ℓ}) : i, j = 1, …, d − 2) .

For ℓ such that ψ(ℓ) is defined, Q|_{{L=0,ℓ=0}} is nondegenerate iff ψ(ℓ) ≠ 0. Therefore, we deduce as above that the set of ℓ such that Q|_{{L=0,ℓ=0}} is nondegenerate contains the complement of a proper algebraic subset. This implies that Ω_{4} is dense and proves the lemma. □

Denote by π : ℝ^{d} × ℝ^{d} × ℝ^{d} → ℝ^{d} × ℝ^{d} the projection on the first two coordinates, and by p : ℝ^{d} × ℝ^{d} × ℝ^{d} → ℝ^{d} the projection on the first coordinate. Lemma 6. Suppose that (l_{1}, m_{1}), (l_{2}, m_{2}), (m_{1}, m_{2}) ∈ π(Ω). Then there exists a rational linear form n such that

(l_{1}, m_{1}, n), (l_{2}, m_{2}, n), (m_{1}, m_{2}, n) ∈ Ω.

Proof. As in the previous lemma, one shows that if (l, m) ∈ π(Ω), the set of n such that (l, m, n) satisfies conditions (1)–(4) is open and dense. Thus, the set of linear forms n such that the triples (l_{1}, m_{1}, n), (l_{2}, m_{2}, n), and (m_{1}, m_{2}, n) satisfy conditions (1)–(4) is open and dense. The condition that Q|_{{n=0}∩{L=0}} is indefinite holds on an open set of n. Hence, such a linear form n exists. □

For a linear form ℓ, consider a linear map F_{ℓ} from the space of quadratic forms on ℝ^{d} to the space of quadratic forms on {ℓ = 0} defined by

F_{ℓ}(Q) = Q|_{ℓ=0}.

This map is rational if ℓ is rational. The kernel of F_{ℓ} is

⟨x_{i}ℓ : i = 1, …, d⟩ .

Proof of Proposition 4. Suppose that for any rational linear form l from p(Ω), the quadratic form (αQ+βL^{2})|_{l=0} is rational for some α and β with (α, β) ≠ 0. This means that F_{l}(⟨Q, L^{2}⟩) has a rational subspace of codimension 1, or, equivalently, for some α, β ∈ ℝ such that (α, β) ≠ 0, the space ⟨αQ + βL^{2}, x_{i}l : i = 1, …, d⟩ is rational.

Case 1: For any rational linear forms l and m such that (l, m) ∈ π(Ω), the space ⟨Q, L^{2}, x_{i}l, x_{j}m : i, j = 1, …, d⟩ is rational.

We will use the following claim: Claim. Let S be not in ⟨Q, L^{2}⟩. Then there are rational linear forms l and m such that (l, m) ∈ π(Ω), and S is not in

⟨Q, L^{2}, x_{i}l, x_{j}m : i, j = 1, …, d⟩ .

Proof. Suppose not. Then for any rational linear forms l and m such that (l, m) ∈ π(Ω), we have S = α(l, m)Q+β(l, m)L^{2} on the subspace {l = 0}∩{m = 0} for some α(l, m), β(l, m) ∈ ℝ. First, we show that α(l, m) and β(l, m) are independent of (l, m) on a large set of pairs. Let (l, m, n) ∈ Ω for some rational linear forms l, m, n. Because of the condition (3), Q has rank at least 2 on the space

V =^{def} {l = 0} ∩ {m = 0} ∩ {n = 0}.

Therefore, Q|_{V} and L^{2}|_{V} are linearly independent. It follows that

α(l, m) = α(m, n) = α(n, l) and β(l, m) = β(m, n) = β(n, l).  (1)

Fix (l_{0}, m_{0}) ∈ π(Ω) with rational linear forms l_{0} and m_{0}. Consider the set

O = {(l, m) ∈ π(Ω) : (m, m_{0}) ∈ π(Ω)}.

This is a nonempty open set in π(Ω). By Lemma 6, there is a rational linear form n such that (l_{0}, m_{0}, n), (l, m, n), (m, m_{0}, n) ∈ Ω. Then using (1), we obtain

α(l_{0}, m_{0}) = α(m_{0}, n) = α(m, n) = α(l, m),

and similarly, β(l_{0}, m_{0}) = β(l, m). Hence, the coefficients α(l, m) and β(l, m) are constant for rational linear forms l and m such that (l, m) ∈ O. Thus, S − αQ − βL^{2} = 0 on the space {l = 0} ∩ {m = 0} for rational linear forms l and m such that (l, m) ∈ O. It follows that S−αQ−βL^{2} = 0 on an open subset of ℝ^{d}. Hence, S = αQ+βL^{2}. This is a contradiction. □

Using the claim, we conclude that the space ⟨Q, L^{2}⟩ is an intersection of spaces of the form ⟨Q, L^{2}, x_{i}l, x_{j}m : i, j = 1, … n⟩ for some rational linear forms l and m such that (l, m) ∈ O. By the assumption of Case 1, each of these spaces is rational. Hence, the space ⟨Q, L^{2}⟩ is rational too. This gives a contradiction.

Case 2: For some rational linear forms l and m such that (l, m) ∈ π(Ω), the space ⟨Q, L^{2}, x_{i}l, x_{j}m : i, j = 1, …, d⟩ is not rational.

By the assumption, the space ⟨αQ + βL^{2}, x_{i}l : i = 1, …, d⟩ is rational for some α, β ∈ ℝ such that (α, β) ≠ 0. Then the space

⟨αQ + βL^{2}, x_{i}l, x_{j}m : i, j = 1, …, d⟩

is rational too. It follows from the assumption of the Case 2 that the pair (α, β) with this property is uniquely defined up to a scalar multiple. Let Q̃ = αQ + βL^{2}. We show that Q̃ is proportional to a rational form. The pair (α, β) can be chosen such that the form Q̃|_{l=0} is rational.

Because of the uniqueness of (α, β), Q̃|_{k=0} is proportional to a rational form for any rational linear form k ∈ p(Ω) ∩ ⟨l, m⟩. Note that the later set is not empty because l and m is in it. Let {e_{i} : i = 1, …, d − 2} be a rational basis of {l = 0} ∩ {m = 0}. We complete it to a rational basis of ℝ^{d} by vector e_{d−1} and e_{d} such that l(e_{d−1}) = 0. Let

U_{t} = ⟨e_{i}, e_{d−1} + te_{d} : i = 1, …, d − 2⟩ for t ∈ ℚ.

Since U_{0} = {l = 0} and l ∈ p(Ω), a rational form k_{t} that defines U_{t} is in p(Ω) for sufficiently small t. Also k_{t} ∈ ⟨l, m⟩ because

{l = 0} ∩ {m = 0} ⊆ U_{t}.

Therefore, α_{t}Q̃|_{U_{t}} is rational for some α_{t} ≠ 0 when t is rational and sufficiently small. Since (l, m) ∈ π(Ω), Q̃ ≠ 0 on the space {l = 0} ∩ {m = 0}. Take a rational vector x ∈ {l = 0} ∩ {m = 0} such that Q̃(x) ≠ 0. Then Q̃(x) ∈ ℚ, and α_{t}Q̃(x) ∈ ℚ. Therefore, α_{t} ∈ ℚ, and Q̃|_{U_{t}} is rational for sufficiently small t. In particular, when t = 0, Q̃(e_{i}, e_{j}) ∈ ℚ for i, j < d. Also

Q̃(e_{i}, e_{d−1} + te_{d}) = Q̃(e_{i}, e_{d−1}) + tQ̃(e_{i}, e_{d}) ∈ ℚ,

so that Q̃(e_{i}, e_{d}) ∈ ℚ for i < d. Finally,

Q̃(e_{d−1} + te_{d}) = Q̃(e_{d−1}) + 2tQ̃(e_{d−1}, e_{d}) + t^{2}Q̃(e_{d}) ∈ ℚ.

Thus, Q̃(e_{d}) ∈ ℚ. This shows that Q̃ is rational which is a contradiction. □

4. Pairs of type (I)

In this section we consider a pair (Q, L) of type (I) and dimension 4. By Proposition 2, the pair (Q, L) is equivalent to either

(x_{1}^{2} + x_{2}^{2} − x_{3}^{2} − x_{4}^{2}, x_{4}) or (x_{1}^{2} − x_{2}^{2} − x_{3}^{2} − x_{4}^{2}, x_{4}).

We consider the first case. The other case can be done the same way. Let

Q_{0} = x_{1}^{2} + x_{2}^{2} − x_{3}^{2} − x_{4}^{2} and L_{0} = x_{4}.  (2)

Let

H = ( \begin{matrix}SO(2,1) & 0 \\ 0 & 1 \end{matrix} ) ⊆ SL(4, ℂ).  (3)

Note that the group H leaves the pair (Q_{0}, L_{0}) invariant.

First, we collect some simple facts. Proposition 7. Let F be a connected semisimple algebraic subgroup of SL(4, ℂ) which acts irreducibly on ℂ^{4} such that

H ⊊ F ⊊ SL(4, ℂ).

Then G = SO(S, ℂ) for some nondegenerate quadratic form S.

Proof. Let 𝔣 be the Lie algebra of F. For convenience of the reader, let us reproduce in Table 1 the list of complex semisimple Lie algebras of dimension up to 14 and dimensions of their faithful irreducible representations (see [OV]). Note that an irreducible representation of a semisimple algebra is a tensor product of irreducible representations of simple factors. One can see that 𝔣 should be of type A_{1} + A_{1} or C_{2}. The 4-dimensional irreducible representation of the algebra of type C_{2} is simplectic. On the other hand, H ⊂ F acts irreducibly on a 3-dimensional subspace, and leaves invariant a nonzero symmetric form on this subspace. This shows that F cannot leave a nonzero symplectic form invariant. Thus, 𝔣 is of type A_{1} + A_{1}. Since 𝔣 has a unique irreducible 4-dimensional representation, this representation is equivalent to the representation of 𝔰𝔬(4, ℂ) on ℂ^{4}. This means that 𝔣 = g𝔰𝔬(4, ℂ)g^{−1} for some g ∈ SL(4, ℂ). Equivalently, 𝔣 = 𝔰𝔬(S, ℂ) for some nondegenerate quadratic form S. Since F is connected, F = SO(S, ℂ). □

Lemma 8. Let G ⊆ GL(d, ℂ) be an algebraic group that acts irreducibly on ℂ^{d}. Then its unipotent radical is trivial.

\begin{matrix}𝔤 & dim 𝔤 & dim ρ \\ A_{1} & 3 & 2, 3, 4, … \\ A_{1} + A_{1} & 6 & 4, … \\ A_{1} + A_{1} + A_{1} & 9 & 8, … \\ A_{1} + A_{1} + A_{1} + A_{1} & 12 & 16, … \\ A_{2} & 8 & 3, 8, … \\ A_{2} + A_{1} & 11 & 6, … \\ A_{2} + A_{1} + A_{1} & 14 & 12, … \\ B_{2} = C_{2} & 10 & 4, 5, … \\ B_{2} + A_{1} & 13 & 8, … \\ G_{2} & 14 & 7, … \end{matrix}

Table 1. Complex irreducible representations

Proof. Let U be the unipotent radical of G, and V = ℂ^{d}. Consider the space

V^{U} = {x ∈ V : Ux = x}.

Since U is unipotent, V^{U} ≠ 0. Since U is normal in G, V^{U} is G-invariant. Thus, V^{U} = V, and U = 1. □

Lemma 9. Let Q be a quadratic form on ℂ^{4} which is invariant under the action of H (H is defined in (3)). Then Q = αQ_{0} + βL_{0}^{2} for some α, β ∈ ℂ, where Q_{0} and L_{0} are defined in (2).

Proof. Let us write Q = Q(x_{1}, x_{2}, x_{3}, 0) + L(x_{1}, x_{2}, x_{3})x_{4} + βx_{4}^{2} for some linear form L and β ∈ ℂ. Then Q(·, ·, ·, 0) and L is SO(2, 1)-invariant. It follows that Q(·, ·, ·, 0) = αQ_{0}(·, ·, ·, 0) for some α ∈ ℂ and L = 0. □

The following proposition is the main result of this section and a partial case of Theorem 1. Proposition 10. Let Q be a nondegenerate quadratic form, and L be a linear form such that (Q(x), L(x)) = (λQ_{0}(gx), μL_{0}(gx)) for some g ∈ SL(4, ℝ) and λ, μ ∈ ℝ − {0}, where Q_{0} and L_{0} are defined in (2). Suppose that for every (α, β) ≠ (0, 0), with α, β ∈ ℝ, the linear combination αQ + βL^{2} is not rational.

Then {(Q(x), L(x)) : x ∈ P(ℤ^{4})} is dense in ℝ^{2}.

Proof. Without loss of generality, λ = μ = 1. Let G = SL(4, ℂ) and H be as in (3). We study the action of H_{ℝ} on G_{ℝ}/G_{ℤ}. By Ratner’s Theorem [Ra91], the closure \overline{H_{ℝ}gG_{ℤ}} ⊆ G_{ℝ}/G_{ℤ} is a homogeneous space. Moreover by the result of Shah [Sh91, Proposition 3.2], \overline{H_{ℝ}gG_{ℤ}} = gF_{ℝ}^{o}G_{ℤ} where F is the smallest algebraic ℚ-subgroup containing g^{−1}Hg, and the radical of F is unipotent. Here F_{ℝ}^{o} denotes the connected component of F_{ℝ} with respect to the Euclidean topology. Since H is connected as an algebraic group, F is connected too.

First, we consider the case when F acts irreducibly on ℂ^{4}. By Lemma 8, F is semisimple, and by Proposition 7, F is one of the subgroups g^{−1}Hg, G, SO(S, ℂ) for some nondegenerate quadratic form S. Since F acts irreducibly on ℂ^{4}, F ≠ g^{−1}Hg. Suppose that F = SO(S). Since F is defined over ℚ, the quadratic form S is proportional to a rational form (see, for example, [Bo95]). By Lemma 9, S = αQ + βL^{2} for some α, β ∈ ℂ. This is a contradiction. Thus, F = G. We conclude that \overline{H_{ℝ}gG_{ℤ}} = G_{ℝ}/G_{ℤ}. Then

\overline{\{(Q(x), L(x)) : x ∈ P(ℤ^{4})\}}
= \overline{\{(Q_{0}(hgγe_{1}), L_{0}(hgγe_{1})) : h ∈ H_{ℝ}, γ ∈ G_{ℤ}\}}
= {(Q_{0}(ae_{1}), L_{0}(ae_{1}) : a ∈ G_{ℝ}} = {(Q_{0}(x), L_{0}(x) : x ∈ ℝ^{4}} = ℝ^{2}.

Here e_{1} = (1, 0, …, 0), and the last equality holds by Lemma 3.

Now we assume that the action of F on ℂ^{4} is not irreducible. Then the action of F on the dual space L is reducible too. The dual action of H has irreducible components ⟨x_{1}, x_{2}, x_{3}⟩ and ⟨x_{4}⟩. Moreover, for any linear form l ∉ ⟨x_{4}⟩, the smallest H-invariant subspace containing l should contain ⟨x_{1}, x_{2}, x_{3}⟩. Therefore, ⟨x_{1}, x_{2}, x_{3}⟩ and ⟨x_{4}⟩ are the only nontrivial H-invariant subspaces. For i = 1, …, 4, define linear forms ℓ_{i}(x) = (gx)_{i}. Then

Q = ℓ_{1}^{2} + ℓ_{2}^{2} − ℓ_{3}^{2} − ℓ_{4}^{2} and L = ℓ_{4}.

It follows that L_{1} =^{def} ⟨ℓ_{1}, ℓ_{2}, ℓ_{3}⟩ and L_{2} =^{def} ⟨ℓ_{4}⟩ are the only candidates for nontrivial F - invariant subspaces. Let F be semisimple. Then its action is completely reducible. Therefore, L_{1} and L_{2} are F -invariant, and since F has no nontrivial characters, the space of F -fixed vectors L^{F} = L_{2} = ⟨L⟩. Using that F is a ℚ-group, we conclude that ⟨L⟩ is a ℚ-subspace. This means that L is proportional to a rational form, which is a contradiction. Therefore, F is not semisimple, and its unipotent radical U ≠ 1. Note that U is defined over ℚ. The subspace of U-fixed vectors L^{U} is F -invariant, so that either L^{U} = L_{1} or L^{U} = L_{2} = ⟨L⟩. If the later holds, L is proportional to a rational form by the same argument as above. Thus, we may assume that L^{U} = L_{1}. In particular, L_{1} is a ℚ-subspace. Since F is defined over ℚ, it has a Levi subgroup F_{0} which is defined over ℚ. It follows from the description of F -invariant subspaces that F_{0} acts irreducibly on L^{U}. There exists an F_{0}-invariant complement for L^{U} generated by a linear form ℓ_{0}. We have that L^{F_{0}} = ⟨ℓ_{0}⟩. In particular, ⟨ℓ_{0}⟩ is a ℚ-subspace, and the linear form ℓ_{0} can be chosen to rational. By Malcev’s theorem, g^{−1}Hg ⊆ u^{−1}F_{0}u for some u ∈ U. The forms L(u^{−1} ·x) and ℓ_{0}(x) are both fixed by F_{0}, so that L(u^{−1} ·x) = α·ℓ_{0}(x) for some α ∈ ℝ − {0}.

Suppose that g^{−1}Hg = u^{−1}F_{0}u. Then F_{0} stabilizes the pair

(Q(u^{−1} · x), L(u^{−1} · x)) = (ℓ_{1}^{2} + ℓ_{2}^{2} − ℓ_{3}^{2} − α^{2}ℓ_{0}^{2}, αℓ_{0})

Since F_{0} is a ℚ-group, the space of quadratic forms that are fixed by F_{0} is a ℚ-space. By Lemma 9, this space is spanned by ℓ_{1}^{2} + ℓ_{2}^{2} − ℓ_{3}^{2} − α^{2}ℓ_{0}^{2} and ℓ_{0}^{2}. Therefore, for some α, β ∈ ℝ, the form

Q̃ =^{def} α(ℓ_{1}^{2} + ℓ_{2}^{2} − ℓ_{3}^{2}) + βℓ_{0}^{2}

is rational. Since the space L_{1} = ⟨ℓ_{1}, ℓ_{2}, ℓ_{3}⟩ is rational, there exists a rational vector x_{0} ≠ 0 such that ℓ_{1}(x_{0}) = ℓ_{2}(x_{0}) = ℓ_{3}(x_{0}) = 0. Clearly, ℓ_{0}(x_{0}) ≠ 0. Then Q̃(x_{0}) = β · ℓ_{0}(x_{0})^{2} ∈ ℚ. Therefore, β ∈ ℚ, and the form Q + L^{2} = ℓ_{1}^{2} + ℓ_{2}^{2} − ℓ_{3}^{2} is proportional to a rational form, which is a contradiction.

Now we may assume that g^{−1}Hg ⊊ u^{−1}F_{0}u. Then

F_{0} = ug^{−1} ( \begin{matrix}SL(3) & 0 \\ 0 & 1 \end{matrix} ) gu^{−1}.

Also

F_{1} =^{def} g^{−1} ( \begin{matrix}SL(3) & 0 \\ 0 & 1 \end{matrix} ) g ⊆ F.

An element u ∈ U acts on L as follows: u · ℓ = ℓ for ℓ ∈ L_{1}, and u · ℓ_{4} = ℓ_{4} + ℓ_{u} for some linear form ℓ_{u} ∈ L_{1}. Then the adjoint action of F_{1} on U corresponds to the usual action of F_{1} on the space spanned by linear forms ℓ_{u}, u ∈ U. It follows that for every ℓ ∈ L_{1}, there exists u ∈ U such that u · ℓ_{4} = ℓ_{4} + ℓ.

Let {e_{i} : i = 1, …, 4} be the standard basis of ℂ^{4}. Fix (a, b) ∈ ℝ^{2}. Take x ∈ P(ℤ^{4}) − ⟨g^{−1}e_{4}⟩. By the previous remark, there exists u ∈ U_{ℝ} such that

L(ux) = (u · ℓ_{4})(x) = b.

Write ux = x_{1} + x_{2} for x_{1} ∈ g^{−1} ⟨e_{1}, e_{2}, e_{3}⟩ and x_{2} ∈ g^{−1} ⟨e_{4}⟩. Since Q is indefinite on the subspace g^{−1} ⟨e_{1}, e_{2}, e_{3}⟩, there exists f ∈ F_{1} such that Q(fx_{1}) = a − Q(x_{2}). Then

Q(fux) = Q(fx_{1}) + Q(x_{2}) = a.

Since

\overline{g^{−1}H_{ℝ}gG_{ℤ}} = F_{ℝ}^{o}G_{ℤ} = F_{1ℝ}U_{ℝ}G_{ℤ},

there exist h_{n} ∈ g^{−1}H_{ℝ}g and γ_{n} ∈ G_{ℤ} such that h_{n}γ_{n} → fu as n → ∞. Finally,

(Q(γ_{n}x), L(γ_{n}x)) = (Q(h_{n}γ_{n}x), L(h_{n}γ_{n}x))
→ (Q(fux), L(fux)) = (a, b).

The proposition is proved. □

Remark 11. Proposition 10 combined with Proposition 4 implies Theorem 1 for dimension d ≥ 5.

5. Pairs of type (II)

Now we prove Theorem 1 for pairs of type (II) of dimension 4. This will finish the proof of Theorem 1. Let

Q_{0} = x_{1}x_{4} − x_{2}x_{3} and L_{0} = x_{4}.  (4)

Note that we use different (cf. Proposition 2) canonical form to simplify calculations. Let

H = \{( \begin{matrix}1 & a & b & ab \\ 0 & 1 & 0 & b \\ 0 & 0 & 1 & a \\ 0 & 0 & 0 & 1 \end{matrix} ) : a, b ∈ ℂ\} ⊆ SL(4, ℂ).  (5)

It is easy to check that Q_{0}(hx) = Q_{0}(x) and L_{0}(hx) = L_{0}(x) for h ∈ H. Lemma 12. The only nontrivial H-invariant subspaces of the dual action of H (i.e the action on the space L of linear forms) are

• ⟨x_{4}⟩.

• ⟨αx_{2} + βx_{3}, x_{4}⟩ for some α, β ∈ ℂ.
• ⟨x_{2}, x_{3}, x_{4}⟩.

Proof. One can check that x_{4} is the only fixed vector of H (up to a scalar multiple). Since the action of H is unipotent, every nontrivial H-invariant subspace V contains a nonzero vector fixed by H. It follows that ⟨x_{4}⟩ ⊆ V. Consider a factor space L/ ⟨x_{4}⟩. The subspace ⟨x_{2}, x_{3}, x_{4}⟩ / ⟨x_{4}⟩ consists of H-fixed vectors, and for any v ∉ ⟨x_{2}, x_{3}, x_{4}⟩, one has ⟨Hv, x_{4}⟩ = L. The conclusion follows. □

Lemma 13. 1. Let G be a simple group, and Ḡ = G × G. Denote by π_{1} and π_{2} the projection maps. Let H ⪇ Ḡ be such that π_{i}(H) = G for i = 1, 2. Then H = {(α(g), g) : g ∈ G} for some automorphism α of G.

2. Let 𝔤 = 𝔰𝔩(2, ℂ), and 𝔤̄ = 𝔤 ⊕ 𝔤. Denote by π_{1} and π_{2} the projection maps. Let 𝔥 ⪇ 𝔤̄ be such that π_{i}(𝔥) = 𝔤 for i = 1, 2. Then 𝔥 = {(Ad(g)x, x) : x ∈ 𝔤} for some g ∈ SL(2, ℂ).

Proof. Let

S_{1} = {g ∈ G : (g, e) ∈ H} and S_{2} = {g ∈ G : (e, g) ∈ H}.

It is easy to check that S_{1} and S_{2} are normal subgroups of G, that is, g^{−1}S_{i}g ⊆ S_{i} for every g ∈ G, i = 1, 2. Since G is simple, S_{i} is either {e} or G.

Suppose that S_{1} = G. We show that H = Ḡ. Let (g, h) ∈ Ḡ. For some g_{1} ∈ G, (g_{1}, h) ∈ H. Then

(g, h) = (g_{1}, h)(g_{1}^{−1}g, e) ∈ H.

Hence, H = Ḡ. Similarly, H = Ḡ if S_{2} = G.

Suppose that S_{1} = S_{2} = {e}. For any g ∈ G, there exists a unique element α(g) such that (α(g), g) ∈ H. Because of the uniqueness, α is a homomorphism. It is surjective because π_{1}(H) = G and injective because S_{2} = {e}. This proves the first part of the lemma.

It is straightforward to rewrite this argument for simple Lie algebras. It is known that any automorphism of 𝔰𝔩(2, ℂ) is inner. Thus, the second part of the lemma follows. □

The following proposition finishes the proof of Theorem 1. Its proof is similar to the proof of Proposition 10. Proposition 14. Let Q be a nondegenerate quadratic form, and L be a linear form such that (Q(x), L(x)) = (λQ_{0}(gx), μL_{0}(gx)) for some g ∈ SL(4, ℝ) and λ, μ ∈ ℝ − {0}, where Q_{0} and L_{0} are defined in (4). Suppose that for every (α, β) ≠ (0, 0), with α, β ∈ ℝ, the linear combination αQ + βL^{2} is not rational.

Then {(Q(x), L(x)) : x ∈ P(ℤ^{4})} is dense in ℝ^{2}.

Proof. We may assume that λ = μ = 1. Let G = SL(4, ℂ), and H be as in (5). Consider the action of H_{ℝ} on G_{ℝ}/G_{ℤ}. As in the proof of Proposition 10, we know that \overline{H_{ℝ}gG_{ℤ}} = gF_{ℝ}^{o}G_{ℤ} where F is the smallest algebraic ℚ-subgroup containing g^{−1}Hg. The group F is connected as an algebraic group, and its radical is unipotent. There is Levi decomposition F = F_{0}U where F_{0} is a connected (as an algebraic group) semisimple ℚ-subgroup, and U is the unipotent radical of F. Note that U is defined over ℚ.

To prove the proposition, it is enough to show that

\overline{\{(Q(fz), L(fz)) : f ∈ F_{ℝ}^{o}, z ∈ P(ℤ^{4})\}} = ℝ^{2}.  (6)

Consider the action of F on the space of linear forms L. Let V be a nontrivial F -invariant subspace. Since H is unipotent, V contains a nonzero vector fixed by g^{−1}Hg. By Lemma 12, the only vector fixed by g^{−1}Hg is L (up to a scalar multiple). Thus, ⟨L⟩ ⊆ V. It follows that there is a unique F -irreducible subspace V ⊆ L. Namely, it is the intersection of all nontrivial F -invariant subspaces. This subspace is contained in every F -invariant subspace. Also V is defined over ℚ. Indeed, for any σ ∈ Gal(ℂ/ℚ), V^{σ} is F (ℚ)-invariant, and F (ℚ) is Zariski dense in F because F is connected. Therefore, V^{σ} is F -invariant, and V ⊆ V^{σ}. Comparing dimensions, we conclude that V = V^{σ}. This shows that V is defined over ℚ.

Let L^{U} be the subspace of vectors fixed by U. Since U is normal in F, the space L^{U} is F -invariant, so that V ⊆ L^{U}. It follows that F_{0} act irreducibly on V. Since F_{0} is semisimple, the F -action on L^{U} is completely reducible. Suppose that V ⊊ L^{U}. Then L^{U} = V ⊕ W for some F -invariant subspace W. However, this contradicts the description of H-invariant subspaces in Lemma 12. Thus, V = L^{U}.

We can write

L_{ℝ} = V_{ℝ} ⊕ W_{ℝ}

for some real (F_{0})_{ℝ}-invariant subspace W_{ℝ}. Then L = V ⊕ W where W = W_{ℝ} ⊗ ℂ is F_{0}-invariant, because (F_{0})_{ℝ} is Zariski dense in F_{0}. Note that W is defined over ℝ. Let

V = {v ∈ ℂ^{4} : ℓ(v) = 0 for ℓ ∈ V}

and

W = {v ∈ ℂ^{4} : ℓ(v) = 0 for ℓ ∈ W}.

Clearly, ℂ^{4} = V ⊕W, V is F -invariant, and W is F_{0}-invariant. Moreover, W is F_{0}-irreducible because W ≃ ℂ^{4}/V as F_{0}-modules, and any nontrivial F_{0}-invariant subspace of ℂ^{4}/V would give by duality a nontrivial F_{0}-invariant subspace in V. The space (ℂ^{4}/V)^{U} is nonzero and F_{0}-invariant. It follows that (ℂ^{4}/V)^{U} = ℂ^{4}/V, so that U acts trivially on ℂ^{4}/V.

Let ℓ_{i}(x) = (gx)_{i} for i = 1, …, 4. Then Q = ℓ_{1}ℓ_{4} − ℓ_{2}ℓ_{3} and L = ℓ_{4}.

Consider several cases:

\underline{Case 1: dim V = 1.} Then V = ⟨L⟩, and since V is defined over ℚ, L is a multiple of a rational form. This is a contradiction.

\underline{Case 2: dim V = 2.} Then dim V = dim W = 2. Since F_{0} is semisimple, the action of F_{0} on V is either trivial or irreducible.

Denote

A = {g ∈ SL(4, ℂ) : g|_{V} = id, g|_{W} = id + a for a ∈ End(W, V)}.

Suppose that F_{0} acts irreducibly on V. Then U acts trivially on V because otherwise V^{U} ≠ 0 is a nontrivial F_{0}-invariant subspace. Since it was shown above that U acts trivially on ℂ^{4}/V too, this implies that U ⊆ A. The Lie algebra 𝔣_{0} ⊆ 𝔰𝔩(V) × 𝔰𝔩(W) of F_{0} satisfies the conditions of Lemma 13 unless 𝔣_{0} = 𝔰𝔩(V) × 𝔰𝔩(W).

First, we consider the case when the last equality holds. Then A is an irreducible F_{0}- module. Suppose that U = 1. Then V and W are F -invariant subspaces such that V + W = L. Since every two 2-dimensional subspaces in Lemma 12 are contained in the unique 3dimensional subspace, this gives a contradiction. Thus, U ≠ 1, and since it is a submodule of the irreducible F_{0}-module A, we conclude that U = A. In particular, for any x ∉ V_{ℝ} and any v ∈ V_{ℝ}, there exists u ∈ U_{ℝ} such that ux = x + v. We will use this fact latter.

Now we assume that 𝔣_{0} is a proper subalgebra of 𝔰𝔩(V) × 𝔰𝔩(W). Then by Lemma 13,

𝔣_{0} = {(ϕ^{−1}xϕ, x) : x ∈ 𝔰𝔩(W)}

for some isomorphism ϕ : V → W. Since 𝔣_{0} is an ℝ-subalgebra, ϕ can be taken to be an ℝ-map. Let 𝔞 ≃ {a ∈ End(W, V)} be the Lie algebra of A. The adjoint action of 𝔣_{0} on 𝔞 is isomorphic to adjoint action of 𝔰𝔩(W) on 𝔤𝔩(W). The isomorphism is a → ϕ ∘ a. It follows that 𝔞 has the only nontrivial 𝔣_{0}-modules: {λϕ^{−1} : λ ∈ ℂ} and {ϕ^{−1} ∘ b : b ∈ 𝔰𝔩(W)}.

Let U correspond to {λϕ^{−1} : λ ∈ ℂ}. Take a basis {v_{1}, v_{2}} of V. Then {v_{1}, v_{2}, ϕ(v_{1}), ϕ(v_{2})} is a basis of ℂ^{4}, and with respect to this basis,

F = \{( \begin{matrix}g & λg \\ 0 & g \end{matrix} ) : g ∈ SL(2), λ ∈ ℂ\} .

Let S be an F -invariant quadratic form. Then the subspace V and W are totally isotropic

with respect to S. Thus, the matrix of S is of the form ( \begin{matrix}0 & X \\ X & 0 \end{matrix} ) for some matrix X such

that ^{t}gXg = X for all g ∈ SL(2). It follows that X = ( \begin{matrix}0 & −u \\ u & 0 \end{matrix} ) for some u ∈ ℂ, and

the quadratic form S is unique up to a scalar multiple. Since F is a ℚ-group, the space of F -invariant quadratic forms is defined over ℚ. Therefore, S is a multiple of a rational form, and g^{−1}Hg is contained in the ℚ-group SO(S). Now we can argue as in the Case 4 when F is of type A_{1}+A_{1} (see below) to get a contradiction. Hence, we may assume that U corresponds to {ϕ^{−1} ∘ b : b ∈ 𝔰𝔩(W)}. Then for any x ∉ V_{ℝ} and any v ∈ V_{ℝ}, there exists u ∈ U_{ℝ} such that ux = x + v. We will use this fact latter on.

Suppose that F_{0} acts trivially on V. If U acts trivially on V, F has linearly independent F -invariant vectors. They correspond to distinct 3-dimensional F -invariant subspaces in L. This contradicts Lemma 12. Thus, U acts nontrivially on V. Let v_{1} ∈ V be a U-fixed vector in V. Take v_{2} such that V = ⟨v_{1}, v_{2}⟩. Let

B = {g ∈ SL(4, ℂ) : gv_{1} = v_{1}, gv_{2} = v_{2} + tv_{1}, g|_{W} = id for t ∈ ℂ}.

Clearly, U ⊆ AB and U ⊈ A. If U ⊆ B, then the subspaces V and W would be F -invariant, which contradicts Lemma 12. Thus, U ⊈ B. Let u = ab ∈ U for a ∈ A−{1} and b ∈ B −{1}. Note that A is abelian, and F normalizes A. Thus,

u^{−1}(U ∩ A)u = b^{−1}(U ∩ A)b ⊆ U ∩ A.

Since the action of B is algebraic, it follows that B normalizes U. Let

a = ( \begin{matrix}1 & ã \\ 0 & 1 \end{matrix} )

for some ã ∈ End(W, V) − {0} with respect to the decomposition V ⊕ W. Since F_{0} acts irreducibly on W, the restriction map is F_{0} → SL(W) is surjective. For g ∈ SL(W), take f ∈ F_{0} such that f = id ⊕ g with respect to the decomposition V ⊕ W. Then

f^{−1}uf = ( \begin{matrix}1 & ãg \\ 0 & 1 \end{matrix} ) · b ∈ U.

Thus,

( \begin{matrix}1 & ãg − ã \\ 0 & 1 \end{matrix} ) ∈ U

for every g ∈ SL(W). In particular, it follows that U ∩ A ≠ 1. The only nontrivial BF_{0}- submodule of A is

A_{0} =^{def} ( \begin{matrix}1 & End(W, ⟨v_{1}⟩) \\ 0 & 1 \end{matrix} ) .

Suppose that U ⊆ A_{0}B. Then the linear form corresponding to the projection to ⟨v_{2}⟩ is fixed by F. This contradicts Lemma 12. Hence, there exists a_{1}b_{1} ∈ U for a_{1} ∈ A − A_{0} and b_{1} ∈ B. Write

a_{1} = ( \begin{matrix}1 & ã_{1} \\ 0 & 1 \end{matrix} )

for some ã_{1} ∈ End(W, V) − End(W, ⟨v_{1}⟩). As above,

( \begin{matrix}1 & ã_{1}g − ã_{1} \\ 0 & 1 \end{matrix} ) ∈ U

for all g ∈ SL(W). One can choose g such that ã_{1}g − ã_{1} ∉ End(W, ⟨v_{1}⟩). This shows that U ∩ A = A. It follows that for any x ∉ V_{ℝ} and any v ∈ V_{ℝ}, there exists u ∈ U_{ℝ} such that ux = x + v.

Finally, we finish the proof in the Case 2. By Lemma 12,

V = ⟨αℓ_{2} + βℓ_{3}, ℓ_{4}⟩

for some α, β ∈ ℂ. Let v ≠ 0 be a vector such that

ℓ_{2}(v) = ℓ_{3}(v) = ℓ_{4}(v) = 0.

Then v ∈ V ∩ V^{⊥}, and since Q is nondegenerate, it follows that W ⊈ ⟨v⟩^{⊥}. Note that L|_{V} = ℓ_{4}|_{V} = 0. In particular, L|_{W} ≠ 0. Fix (a, b) ∈ ℝ^{2}. Take z ∈ P(ℤ^{4})−V. Let z = z′ +z′′ for z′ ∈ V and z′′ ∈ W − {0}. Since L|_{W} ≠ 0, there exists g ∈ SL(W_{ℝ}) such that L(gz′′) = b. Also we can choose g_{n} ∈ SL(W_{ℝ}) such that g_{n} → g and Q(g_{n}z′′, v) ≠ 0. Take f_{n} ∈ F_{0ℝ}^{o} such that f_{n}|_{W_{ℝ}} = g_{n} and u_{n} ∈ U_{ℝ} such that

u_{n}(f_{n}z) = f_{n}z + t_{n}v

where t_{n} = \frac{a−Q(f_{n}z)}{2Q(g_{n}z′′,v)}. Then

Q(u_{n}f_{n}z) = Q(f_{n}z + t_{n}v) = Q(f_{n}z) + 2t_{n}Q(f_{n}z, v)
= Q(f_{n}z) + 2t_{n}Q(g_{n}z′′, v) = a.

Also

L(u_{n}f_{n}z) = L(f_{n}z′ + f_{n}z′′ + t_{n}v) = L(g_{n}z′′) → b.

This shows (6).

\underline{Case 3: dim V = 3.} Let V = ⟨v⟩. By Lemma 12, V = ⟨ℓ_{2}, ℓ_{3}, ℓ_{4}⟩. It follows that Q(v) = 0. Since F has no nontrivial characters, F v = v. We show that W is a ℚ-subspace. For any σ ∈ Gal(ℂ, ℚ), W^{σ} is F_{0}-invariant (because F_{0} is connected). Since W ∩ W^{σ} ≠ {0} and W is F_{0}-irreducible, it follows that W = W^{σ}. Thus, W is defined over ℚ.

For any z ∈ W_{ℤ} − {0}, define

F_{0}^{z} = {g ∈ F_{0ℝ}^{o} : Q(gz, v) ≠ 0}.

We claim that F_{0}^{z} is not empty. Suppose that Q(gz, v) = 0 for every g ∈ F_{0ℝ}^{o}. Since F_{0ℝ}^{o} is Zariski dense in F_{0}, ⟨F_{0ℝ}^{o}z⟩ = W_{ℝ}. Thus, W_{ℝ} ⊆ ⟨v⟩^{⊥}. Since Q(v) = 0, the vector v lies in the radical of Q. This gives a contradiction. Thus, F_{0}^{z} is a nonempty Zariski open subset of F_{0ℝ}^{o}. It follows that F_{0}^{z} is dense in F_{0ℝ}^{o} in Euclidean topology. Put

F_{0}^{∞} = ⋂ {F_{0}^{z} : z ∈ P(W_{ℤ})} .

This set is dense in F_{0ℝ}^{o} by Baire Category Theorem.

Consider the orbit O = [L]^{F_{0ℝ}^{o}} in the real projective space ℙ(W_{ℝ}^{∗}) where W_{ℝ}^{∗} is the dual space of W_{ℝ}. If O consists of rational points, then [L]^{F_{0ℝ}^{o}} = [L] and [L]^{F_{0}} = [L], which is a contradiction. Thus, there exists g ∈ F_{0ℝ}^{o} such that the linear form L(gx), x ∈ W, is not proportional to a rational form. Then L(gP(W_{ℤ})) is dense in ℝ. Therefore, \overline{L(F_{0ℝ}^{o}P(W_{ℤ}))} = ℝ and \overline{L(F_{0}^{∞}P(W_{ℤ}))} = ℝ.

Every element u ∈ U acts on ℂ^{4} as follows: uv = v and for w ∈ W, uw = w + l_{u}(w)v for some linear form l_{u} on W. Using this notations, the action of F_{0} on U by conjugation corresponds to the usual action on the space of linear forms spanned by l_{u}, u ∈ U. Thus, this action is irreducible. Note that U ≠ 1 because L^{U} = V. It follows that for every linear form l on W, there exists u ∈ U such that uw = w + l(w)v for w ∈ W. In particular, for any w ∈ W and t ∈ ℝ, there exists u ∈ U_{ℝ} such that uw = w + tv.

Now we are ready to finish the proof of Case 3. Fix (a, b) ∈ ℝ^{2}. There exist g_{n} ∈ F_{0}^{∞} and z_{n} ∈ P(W_{ℤ}) such that L(g_{n}z_{n}) → b. Take u_{n} ∈ U_{ℝ} such that u_{n}g_{n}z_{n} = g_{n}z_{n} + t_{n}v where t_{n} = \frac{a−Q(g_{n}z_{n})}{2Q(g_{n}z_{n},v)}. Then

Q(u_{n}g_{n}z_{n}) = Q(g_{n}z_{n} + t_{n}v) = Q(g_{n}z_{n}) + 2t_{n}Q(g_{n}z_{n}, v) = a,

and

L(u_{n}g_{n}z_{n}) = L(g_{n}z_{n}) → b.

This shows (6).

\underline{Case 4: dim V = 4.} By Lemma 8, F is semisimple. From Table 1, F is one of the types A_{1}, A_{1} + A_{1}, C_{2}. The first case is impossible because F contains a 2-dimensional unipotent subgroup. Denote by 𝔣 the Lie algebra of F.

Let F be of type C_{2}. Then 𝔣 = g𝔰𝔭(4, ℂ)g^{−1} for some g ∈ SL(4, ℂ). Equivalently, 𝔣 = 𝔰𝔭(S, ℂ) for a nondegenerate symplectic form S over ℂ. The form S is proportional to a real form because F is defined over ℝ. Fix (a, b) ∈ ℝ^{2}. By Lemma 3, there exists x ∈ ℝ^{4} such that Q(x) = a and L(x) = b. Take x_{n} ∈ ℚ^{4} − {0} such that x_{n} → x, and y_{n} ∈ ℚ^{4} − ⟨x_{n}⟩ such that S|_{⟨x_{n},y_{n}⟩} is nondegenerate. Let V_{n} = ⟨x_{n}, y_{n}⟩. Take z_{n} ∈ P(V_{nℤ}). Every element of the form g ⊕ id for g ∈ SL(V_{nℝ}) with respect to decomposition V_{n} ⊕ V_{n}^{⊥} is in Sp(S, ℝ) for any g ∈ SL(V_{nℝ}). Thus, there exists f_{n} ∈ F_{ℝ}^{o} = Sp(S, ℝ) such that x_{n} = f_{n}z_{n}. Then Q(f_{n}z_{n}) → a and L(f_{n}z_{n}) → b. This shows (6).

Let F be of type A_{1} + A_{1}. Using an argument as in Proposition 7, F = SO(S) for some real nondegenerate quadratic form S.

Let 𝔥 be the Lie algebra of H. It is easy to check that the normalizer of 𝔥 in 𝔰𝔩(4, ℂ) is

\{( \begin{matrix}u & x & y & t \\ 0 & v & 0 & y \\ 0 & 0 & −v & x \\ 0 & 0 & 0 & −u \end{matrix} ) : u, v, x, y, t ∈ ℂ\} ,  (7)

and the centralizer of 𝔥 in 𝔰𝔩(4, ℂ) is

\{( \begin{matrix}0 & x & y & t \\ 0 & 0 & 0 & y \\ 0 & 0 & 0 & x \\ 0 & 0 & 0 & 0 \end{matrix} ) : x, y, t ∈ ℂ\} .  (8)

Since F_{ℝ} contains closed unipotent subgroup g^{−1}H_{ℝ}g, it cannot be compact. Thus, F_{ℝ} is isomorphic to either SO(3, 1)_{ℝ} or SO(2, 2)_{ℝ}. Recall that SO(3, 1)_{ℝ} is isogenous to SL(2, ℂ), and SO(2, 2)_{ℝ} is isogenous to SL(2, ℝ) × SL(2, ℝ). In both cases, the group g^{−1}H_{ℝ}g is a maximal unipotent subgroup of F. Let 𝔥_{ℝ} be the Lie algebra of H_{ℝ}. Consider the map

ϕ : N_{SL(4,ℝ)}(gH_{ℝ}g^{−1})^{o} → GL(Ad(g)𝔥_{ℝ})

defined by ϕ(n)h = Ad(n)h. By (7) and (8), the image of ϕ is isogenous to ℝ_{>0} × ℝ_{>0}. If F_{ℝ} were isogenous to SL(2, ℂ), then ϕ(N_{F_{ℝ}}(gH_{ℝ}g^{−1}))^{o} would be isogenous to ℂ^{×} ≃ SO(2, ℝ) × ℝ_{>0}. To check the last statement, one can note that it is obvious when g^{−1}H_{ℝ}g is the subgroup of upper unipotent matrices in SL(2, ℂ), and g^{−1}H_{ℝ}g is conjugate to this subgroup. This shows that S has signature (2, 2).

Denote E = SO(Q_{0}) and G = SL(4, ℂ). Then F = g^{−1}g_{1}^{−1}Eg_{1}g for some g_{1} ∈ G. First, we show that g_{1} ∈ EC_{G}(H). We have H and g_{1}Hg_{1}^{−1} ⊆ E. Since maximal unipotent subgroups are conjugate, eHe^{−1} = g_{1}Hg_{1}^{−1} for some e ∈ E. Thus, e^{−1}g_{1} ∈ N_{G}(H), and g_{1} ∈ EN_{G}(H). Without loss of generality, g_{1} ∈ N_{G}(H). Consider the map ψ : N_{G}(H) → GL(𝔥) defined by ψ(n)h = Ad(n)h. Since this map is algebraic, its image is an algebraic subgroup in GL(𝔥). By (7) and (8), dim ψ(N_{G}(H)) = 2 and ψ(N_{G}(H))^{∘} is generated by diagonal matrices. Thus, ψ(N_{G}(H))^{∘} is abelian. Let

T = {diag(u, v, v^{−1}, u^{−1}) : u, v ∈ ℂ^{×}}.

This is a maximal torus of E. Since ψ(T) has dimension 2, it is a maximal torus of GL(𝔥), and so is ψ(g_{1}^{−1}T g_{1}). Since they commute,

ψ(g_{1})^{−1}ψ(T)ψ(g_{1}) = ψ(T),

i.e. ψ(g_{1}) normalizes ψ(T). The normalizer of ψ(T) in GL(𝔥) is generated by ψ(T) and the transformation that permutes two elements of the basis of 𝔥. It is easy to see that this transformation is in ψ(E). Thus, ψ(g_{1}) ∈ ψ(E). Since ker ψ = C_{G}(H), it follows that g_{1} ∈ EC_{G}(H).

It follows from Lemma 12 that H has a unique fixed vector v (up to a scalar multiple). Then for c ∈ C_{G}(H), cv = λv for some λ ∈ ℂ. Suppose that μ ∈ ℂ−{λ} be an eigenvalue of c. The complex eigenspace corresponding to μ and λ are H-invariant. Each of these subspaces contains a nonzero vector fixed by H (because H is unipotent). This contradicts Lemma 12. Thus, c has a unique eigenvalue, and c = (λI)c_{0} where I is the identity matrix, and c_{0} ∈ C_{G}(H) is unipotent. Let C_{0} be the set of unipotent elements of C_{G}(H). If c_{1}, c_{2} ∈ C_{0}, c_{1}v = c_{2}v = v, and c_{1}c_{2}^{−1}v = v. Then c_{1}c_{2}^{−1} is unipotent. Thus, C_{0} is a subgroup. We have C_{G}(H) = Z(G)C_{0}. Clearly, C_{0} is unipotent algebraic subgroup, so that it is connected. By (8), C_{0} = Hu_{ℂ} where

u_{t} = ( \begin{matrix}1 & 0 & 0 & t \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix} ) .

Thus, g_{1} ∈ EZ(G)u_{ℂ}. Then

F = g^{−1}g_{1}^{−1}Eg_{1}g = g^{−1}u_{t}^{−1}SO(Q_{0})u_{t}g = g^{−1}SO(Q_{0} + tx_{4}^{2})g
= SO(Q + tL^{2})

for some t ∈ ℂ. Since F is a ℚ-group, the quadratic form Q+tL^{2} is proportional to a rational form. This is a contradiction. The proposition is proved. □

6. Conclusion

Let (Q, L) be a pair such that Q is a nondegenerate quadratic form, and L is a nonzero linear form. It would be interesting to obtain neccesary and sufficient conditions for the property

\overline{\{(Q(x), L(x)) : x ∈ P(ℤ^{d})\}} = ℝ^{2}

to hold. In this context, we formulate the following conjecture:

Conjecture 15. Let Q be a nondegenerate quadratic form, and L a nonzero linear form in dimension d ≥ 4. Suppose that

1. For every β ∈ ℝ, Q + βL^{2} is indefinite.

2. For every (α, β) ≠ (0, 0), with α, β ∈ ℝ, αQ + βL^{2} is not rational.

Then {(Q(x), L(x)) : x ∈ P(ℤ^{d})} is dense in ℝ^{2}.

The first condition in the conjecture is neccesary for the density to hold. Indeed, suppose that Q + βL^{2} is definite (say, positive definite) for some β ∈ ℝ. By Proposition 2,

Q = ℓ_{1}^{2} + … + ℓ_{d−1}^{2} − βℓ_{d}^{2} and L = ℓ_{d}  (9)

for some linearly independent linear forms ℓ_{i}, i = 1, …, d. If ¯0 ∈ ℝ^{2} is an accumulation point of the set {(Q(x), L(x)) : x ∈ ℤ^{d}}, then ¯0 ∈ ℝ^{d} is an accumulation point of the set {(ℓ_{1}(x), …, ℓ_{d}(x)) : x ∈ ℤ^{d}} which is impossible.

Due to Theorem 1, it remains to prove Conjecture 15 in the case when (Q, L) is of type (II), and Q|_{L=0} is positive definite, i.e.

Q = ℓ_{1}^{2} + … + ℓ_{d−2}^{2} + ℓ_{d−1}ℓ_{d} and L = ℓ_{d}  (10)

for some linearly independent linear forms ℓ_{i}, i = 1, …, d. The method of the proof of Proposition 14 with minor modifications allows to prove Conjecture 15 in dimension 4. However, one should keep in mind that the method of reduction to lower dimension (Proposition 4) fails to work in this case. Namely, for every d ≥ 4, there exist pairs (Q, L) of dimension d that satisfy the conditions of Conjecture 15, but for every rational subspace V of codimension 1, the set {(Q(x), L(x)) : x ∈ V_{ℤ}} is not dense in ℝ^{2}. To construct such an example, we take a pair (Q, L) as in (10) such that the space ⟨ℓ_{1}, …, ℓ_{d−2}, ℓ_{d}⟩ contains no nonzero rational forms, and (Q, L) satisfies the second condition of Conjecture 15. Clearly, such pairs exist. In fact, such pairs are generic in suitable sense. Let ℓ be nonzero rational form. It is easy to see from Lemma 4 that the pair (Q|_{ℓ=0}, L|_{ℓ=0}) is of type (II) iff Q|_{W} is degenerate where W is the space {ℓ = 0, L = 0}. The forms ℓ_{i}|_{W}, i = 1, …, d − 2, are linearly independent because ℓ ∉ ⟨ℓ_{1}, …, ℓ_{d−2}, ℓ_{d}⟩. Therefore,

Q|_{W} = (ℓ_{1}^{2} + … + ℓ_{d−2}^{2})|_{W}

is nondegenerate, and (Q|_{ℓ=0}, L|_{ℓ=0}) is of type (I). Since Q|_{L=0} is positive definite, the pair (Q|_{ℓ=0}, L|_{ℓ=0}) is as in (9). As we saw above, density fails in this case.

References

[Bo95] A. Borel, Values of indefinite quadratic forms at integer point and flows on spaces of lattices, Bull. Amer. Math. Soc. 32 (1995), 184–204.

[Da00] S. G. Dani, On Values of linear and quadratic forms at integer points, Number theory, pp. 107–119, Trends Math., Birkhäuser, Basel, 2000.

[DM90] S. G. Dani, G. A. Margulis, Orbit closures of generic unipotent flows on homogeneous spaces of SL(3, R), Math. Ann. 286 (1990), 101–128.

[Ma89] G. Margulis, Discrete subgroups and ergodic theory, Number theory, trace formulas and discrete groups (Oslo, 1987), 377–398, Academic Press, Boston, MA, 1989.

[Ma97] G. A. Margulis, Oppenheim Conjecture, Fields Medalists’ lectures, pp. 272–327, World Sci. Publishing, River Edge, NJ, 1997.

[OV] A. Onishchik, E. Vinberg, Lie groups and Lie algebras, Encyclopedia Math. Sci. Vol. 41, Springer-Verlag, New York — Berlin, 1994.

[Ra91] M. Ratner, Raghunathan’s topological conjecture and distributions of unipotent flows, Duke Math. J. 63 (1991), 235–280.

[Sh91] N. A. Shah, Uniformly distributed orbits of certain flows on homogeneous spaces, Math. Ann. 289, 315–334, 1991.

Department of Mathematics, University of Michigan, Ann Arbor, MI 48109

  E-mail address: gorodnik@umich.edu