Wavelet Compression, Data Fitting and Approximation Based on Adaptive Composition of Lorentz-type Thresholding and Besov-type Non-threshold Shrinkage

(1)

Wavelet Compression, Data Fitting and Approximation Based on Adaptive Composition of Lorentz-type Thresholding and Besov-type Non-threshold Shrinkage

Lubomir T. Dechevsky ¹ , Joakim Gundersen ¹ and Niklas Grip ²

1

Narvik University College, Norway

2

Lule˚ a University of Technology, Sweden

7th International Conference on

Large-Scale Scientific Computations

Sozopol, Bulgaria, June 4–8 2009

(2)

(3)

Outline

1 The central role of wavelet expansions in Besov space theory

2 Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage

Non-thresholding wavelet shrinkage Composite Besov–Lorentz shrinkage

3 Besov–Lorentz shrinkage versus firm thresholding

4 Comparisons on a noisy benchmark image

Overview now, details and further references in the paper and in later

discussions.

(4)

Overview now, details and further references in the paper and in later

discussions.

(5)

Outline

1 The central role of wavelet expansions in Besov space theory

2 Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage

Non-thresholding wavelet shrinkage Composite Besov–Lorentz shrinkage

3 Besov–Lorentz shrinkage versus firm thresholding

4 Comparisons on a noisy benchmark image

Overview now, details and further references in the paper and in later

discussions.

(6)

Overview now, details and further references in the paper and in later

discussions.

(7)

Outline

1 The central role of wavelet expansions in Besov space theory

2 Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage

Non-thresholding wavelet shrinkage Composite Besov–Lorentz shrinkage

3 Besov–Lorentz shrinkage versus firm thresholding

4 Comparisons on a noisy benchmark image

Overview now, details and further references in the paper and in later

discussions.

(8)

+

j≥0 k∈Z

ⁿ

2 l=1

β _j,k  ,

with local smoothness index s _j,k = s(x ₀ + 2 ^−j k ).

(9)

The central role of wavelet expansions in Besov space theory

Wavelet expansions:

Unconditional frame/basis for a large range of Besov spaces B

p,q^s(·)

(R

ⁿ

)

B _p,q ^s(·) (R ⁿ ) defined by wavelet expansion

f = X

k∈Z

ⁿ

α _0,k ϕ ^[0] _0,k (x) + X

j≥0

X

k∈Z

ⁿ

2

ⁿ

−1

X

l=1

β _j,k ^[l] ψ ^[l] _j,k (x), a.e. x ∈ R ⁿ

and quasi-norm topology computed on the wavelet coefficients

kf k B

p,q^s(·)

=



 X

k∈Z

ⁿ

|α _0,k | ^p

!

^q_p

+ X

j≥0

X

k∈Z

ⁿ

2 ^j

h s

_j,k

n

1 2

−

¹_p

i p 2

ⁿ

−1

X

l=1

β _j,k ^[l]

p !

^q_p





q

,

with local smoothness index s _j,k = s(x ₀ + 2 ^−j k ).

(10)

+

j≥0 k∈Z

ⁿ

2 l=1

β _j,k  ,

with local smoothness index s _j,k = s(x ₀ + 2 ^−j k ).

(11)

The central role of wavelet expansions in Besov space theory

Wavelet expansions:

Unconditional frame/basis for a large range of Besov spaces B

p,q^s(·)

(R

ⁿ

)

B _p,q ^s(·) (R ⁿ ) defined by wavelet expansion

f = X

k∈Z

ⁿ

α _0,k ϕ ^[0] _0,k (x) + X

j≥0

X

k∈Z

ⁿ

2

ⁿ

−1

X

l=1

β _j,k ^[l] ψ ^[l] _j,k (x), a.e. x ∈ R ⁿ

and quasi-norm topology computed on the wavelet coefficients

kf k B

p,q^s(·)

=



 X

k∈Z

ⁿ

|α _0,k | ^p

!

^q_p

+ X

j≥0

X

k∈Z

ⁿ

2 ^j

h s

_j,k

n

1 2

−

¹_p

i p 2

ⁿ

−1

X

l=1

β _j,k ^[l]

p !

^q_p





q

,

with local smoothness index s _j,k = s(x ₀ + 2 ^−j k ).

(12)

(13)

Classical wavelet threshold shrinkage

“Classical” wavelet threshold shrinkage:

Appropriate for relatively regular (smooth) signals.

(14)

Hard and soft thresholding are limiting cases of firm thresholding:

(15)

Classical wavelet threshold shrinkage

“Classical” wavelet threshold shrinkage:

Appropriate for relatively regular (smooth) signals.

Hard and soft thresholding are limiting cases of firm thresholding:

(iii) → (i) when λ 2 → λ ₁

(16)

Hard and soft thresholding are limiting cases of firm thresholding:

(iii) → (i) when λ 2 → λ ₁ and (iii) → (ii) when λ 2 → ∞

(17)

Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage

(iv) Lorentz-curve thresholding: Vidakovic 1999 [V], based on the Lorentz curve for the energy in the wavelet decomposition.

(v) General Lorentz thresholding [DRP]: Generalization based on combined use of two function-analytical and operator-theoretical facts:

(A) Computability of the Peetre K -functional between Lebesgue spaces in terms of non-increasing rearrangements of a measurable function.

(B) Isomorphism of Besov Spaces to vector-valued spaces of Lebesgue type.

When applicable (=for smooth signals), these threshold techniques also gives compression. However, not all signals/functions are smooth!

[V] Vidakovic, B., Statistical Modeling by Wavelets. Wiley, New York, (1999).

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(18)

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(19)

(iv) Lorentz-curve thresholding: Vidakovic 1999 [V], based on the Lorentz curve for the energy in the wavelet decomposition.

(v) General Lorentz thresholding [DRP]: Generalization based on combined use of two function-analytical and operator-theoretical facts:

(A) Computability of the Peetre K -functional between Lebesgue spaces in terms of non-increasing rearrangements of a measurable function.

(B) Isomorphism of Besov Spaces to vector-valued spaces of Lebesgue type.

When applicable (=for smooth signals), these threshold techniques also gives compression. However, not all signals/functions are smooth!

[V] Vidakovic, B., Statistical Modeling by Wavelets. Wiley, New York, (1999).

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(20)

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(21)

(iv) Lorentz-curve thresholding: Vidakovic 1999 [V], based on the Lorentz curve for the energy in the wavelet decomposition.

(v) General Lorentz thresholding [DRP]: Generalization based on combined use of two function-analytical and operator-theoretical facts:

(A) Computability of the Peetre K -functional between Lebesgue spaces in terms of non-increasing rearrangements of a measurable function.

(B) Isomorphism of Besov Spaces to vector-valued spaces of Lebesgue type.

When applicable (=for smooth signals), these threshold techniques also gives compression. However, not all signals/functions are smooth!

[V] Vidakovic, B., Statistical Modeling by Wavelets. Wiley, New York, (1999).

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(22)

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(23)

(iv) Lorentz-curve thresholding: Vidakovic 1999 [V], based on the Lorentz curve for the energy in the wavelet decomposition.

(v) General Lorentz thresholding [DRP]: Generalization based on combined use of two function-analytical and operator-theoretical facts:

(A) Computability of the Peetre K -functional between Lebesgue spaces in terms of non-increasing rearrangements of a measurable function.

(B) Isomorphism of Besov Spaces to vector-valued spaces of Lebesgue type.

When applicable (=for smooth signals), these threshold techniques also gives compression. However, not all signals/functions are smooth!

[V] Vidakovic, B., Statistical Modeling by Wavelets. Wiley, New York, (1999).

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(24)

shrinkage using GM Waves. Mathematical Methods for Curves and Surfaces, ed. M. Dæhlen and K. Mørken and L.

Schumaker, 263–274, Nashboro Press, Brentwood, North Carolina, (2005).

[DGG] Dechevsky, L. T., Grip, N., Gundersen, J., A new generation of wavelet shrinkage: adaptive strategies based on compostion of Lorentz-type thresholding and Besov-type non-thresholding shrinkage. In: Proceedings of SPIE: Wavelet Applications in Industrial Processing V, Boston, MA, USA 6763(2007), article 676308, pp. 1–14.

[BL] Bergh, J., L¨ofstr¨om, J., Interpolation spaces. An introduction. Grundlehren der Mathematischen Wissenshaften, 223,

(25)

Three different families of wavelet shrinkage methods Non-thresholding wavelet shrinkage

Non-thresholding wavelet shrinkage [DRP,MGD,DGG]:

Appropriate for relatively non-regular (fractal) signals.

A fairly new family of nonthreshold (=no compression!) wavelet shrinkage estimators. Appropriate for fractal signals, e.g., continuous but everywhere nonsmoooth signals, such as the Weierstrass function.

Then typically a full, non-sparse sequence of wavelet coefficients.

Parallels Wahba’s spline smoothing technique based on Tikhonov regularization of ill-posed inverse problems, but now for wavelets.

Based on 2 function-analytical and operator-theoretical facts [BL]:

(C) Metrizability of quasinormed abelian groups via the Method of Powers.

(D) The Theorem of Powers for the real interpolation method of

Peetre–Lions, leading to explicit computation of the K -functional (in this case also called quasilinearization).

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov regularity constraints.

Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

[MGD] Moguchaya, T., Grip, N., Dechevsky, L. T., Bang, B., Laks˚a, A., Tong, B., Curve and surface fitting by wavelet shrinkage using GM Waves. Mathematical Methods for Curves and Surfaces, ed. M. Dæhlen and K. Mørken and L.

(26)

(27)

Non-thresholding wavelet shrinkage [DRP,MGD,DGG]:

Appropriate for relatively non-regular (fractal) signals.

A fairly new family of nonthreshold (=no compression!) wavelet shrinkage estimators. Appropriate for fractal signals, e.g., continuous but everywhere nonsmoooth signals, such as the Weierstrass function.

Then typically a full, non-sparse sequence of wavelet coefficients.

Parallels Wahba’s spline smoothing technique based on Tikhonov regularization of ill-posed inverse problems, but now for wavelets.

Based on 2 function-analytical and operator-theoretical facts [BL]:

(C) Metrizability of quasinormed abelian groups via the Method of Powers.

(D) The Theorem of Powers for the real interpolation method of

Peetre–Lions, leading to explicit computation of the K -functional (in this case also called quasilinearization).

(28)

(29)

Non-thresholding wavelet shrinkage [DRP,MGD,DGG]:

Appropriate for relatively non-regular (fractal) signals.

A fairly new family of nonthreshold (=no compression!) wavelet shrinkage estimators. Appropriate for fractal signals, e.g., continuous but everywhere nonsmoooth signals, such as the Weierstrass function.

Then typically a full, non-sparse sequence of wavelet coefficients.

Parallels Wahba’s spline smoothing technique based on Tikhonov regularization of ill-posed inverse problems, but now for wavelets.

Based on 2 function-analytical and operator-theoretical facts [BL]:

(C) Metrizability of quasinormed abelian groups via the Method of Powers.

(D) The Theorem of Powers for the real interpolation method of

Peetre–Lions, leading to explicit computation of the K -functional (in this case also called quasilinearization).

(30)

(31)

Three different families of wavelet shrinkage methods Composite Besov–Lorentz shrinkage

Composite Besov–Lorentz shrinkage:

Finds points of singularity and uses Besov non-thresholding there

First announced in [DGG]. Relying on certain real interpolation spaces (B _qq ^σ , B _pp ^s ) _θ,p(θ) = B _p(θ)p(θ) ^s(θ) , 0 ≤ θ ≤ 1 and 1

p(θ)

def = 1 − θ q + θ

p . Parameter θ: Determines the degree to which the composite operator is of Lorentz type or a Besov shrinkage-type estimator.

Blending of Lorentz threshold and non-threshold method ⇒ θ is also control parameter for regulating the compression rate.

Based on six function-analytical facts:

(A)-(D) as in the previous slides

(E) The reiteration theorem for the real interpolation method of Peetre-Lions.

(F) The generalization, via the Holmstedt formula, of the formula for computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.

(32)

computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.

(33)

Composite Besov–Lorentz shrinkage:

Finds points of singularity and uses Besov non-thresholding there

First announced in [DGG]. Relying on certain real interpolation spaces (B _qq ^σ , B _pp ^s ) _θ,p(θ) = B _p(θ)p(θ) ^s(θ) , 0 ≤ θ ≤ 1 and 1

p(θ)

def = 1 − θ q + θ

p . Parameter θ: Determines the degree to which the composite operator is of Lorentz type or a Besov shrinkage-type estimator.

Blending of Lorentz threshold and non-threshold method ⇒ θ is also control parameter for regulating the compression rate.

Based on six function-analytical facts:

(A)-(D) as in the previous slides

(E) The reiteration theorem for the real interpolation method of Peetre-Lions.

(F) The generalization, via the Holmstedt formula, of the formula for computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.

(34)

computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.

(35)

Composite Besov–Lorentz shrinkage:

Finds points of singularity and uses Besov non-thresholding there

First announced in [DGG]. Relying on certain real interpolation spaces (B _qq ^σ , B _pp ^s ) _θ,p(θ) = B _p(θ)p(θ) ^s(θ) , 0 ≤ θ ≤ 1 and 1

p(θ)

def = 1 − θ q + θ

p . Parameter θ: Determines the degree to which the composite operator is of Lorentz type or a Besov shrinkage-type estimator.

Blending of Lorentz threshold and non-threshold method ⇒ θ is also control parameter for regulating the compression rate.

Based on six function-analytical facts:

(A)-(D) as in the previous slides

(E) The reiteration theorem for the real interpolation method of Peetre-Lions.

(F) The generalization, via the Holmstedt formula, of the formula for computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.

(36)

computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.

(37)

Composite Besov–Lorentz shrinkage:

Finds points of singularity and uses Besov non-thresholding there

First announced in [DGG]. Relying on certain real interpolation spaces (B _qq ^σ , B _pp ^s ) _θ,p(θ) = B _p(θ)p(θ) ^s(θ) , 0 ≤ θ ≤ 1 and 1

p(θ)

def = 1 − θ q + θ

p . Parameter θ: Determines the degree to which the composite operator is of Lorentz type or a Besov shrinkage-type estimator.

Blending of Lorentz threshold and non-threshold method ⇒ θ is also control parameter for regulating the compression rate.

Based on six function-analytical facts:

(A)-(D) as in the previous slides

(E) The reiteration theorem for the real interpolation method of Peetre-Lions.

(F) The generalization, via the Holmstedt formula, of the formula for computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.

(38)

thresholding is of entropy type and general, but inflexible with respect

to introduction of meaningful bias information

(39)

Besov–Lorentz shrinkage versus firm thresholding

Optimization with respect to all parameters of the Besov-Lorentz model would be a considerably more challenging computational problem than optimization related to firm thresholding.

However, several advantages:

1

Derived from the important function-analytic properties (A)-(F) stated in the previous slides. (Rather than only unification of hard and soft thresholding.)

2

Convenient framework for fine control of the optimization.

Can be performed under a rich variety of meaningful constraints.

Allows introduction of bias in the estimation process, whenever information about such bias is available, with drastic improvement in the quality of estimation.

3

The optimization proposed by Gao and Bruce 1997 for firm

thresholding is of entropy type and general, but inflexible with respect

to introduction of meaningful bias information

(40)

thresholding is of entropy type and general, but inflexible with respect

to introduction of meaningful bias information

(41)

Optimization with respect to all parameters of the Besov-Lorentz model would be a considerably more challenging computational problem than optimization related to firm thresholding.

However, several advantages:

1

Derived from the important function-analytic properties (A)-(F) stated in the previous slides. (Rather than only unification of hard and soft thresholding.)

2

Convenient framework for fine control of the optimization.

Can be performed under a rich variety of meaningful constraints.

Allows introduction of bias in the estimation process, whenever information about such bias is available, with drastic improvement in the quality of estimation.

3

The optimization proposed by Gao and Bruce 1997 for firm

thresholding is of entropy type and general, but inflexible with respect

to introduction of meaningful bias information

(42)

thresholding is of entropy type and general, but inflexible with respect

to introduction of meaningful bias information

(43)

4

The limiting cases soft and hard thresholding can both be

implemented within the wavelet pennalization strategy proposed in [DRP]. Therefore, firm thresholding itself can be implemented with the wavelet pennalization strategy.

5

Trade-off between error of approximation and rate of compression . . . . . . efficiently controllable with Besov–Lorentz shrinkage.

No such control available with firm thresholding.

6

Besov–Lorentz shrinkage outperforms firm thresholding in fitting singularities.

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(44)

(45)

4

The limiting cases soft and hard thresholding can both be

implemented within the wavelet pennalization strategy proposed in [DRP]. Therefore, firm thresholding itself can be implemented with the wavelet pennalization strategy.

5

Trade-off between error of approximation and rate of compression . . . . . . efficiently controllable with Besov–Lorentz shrinkage.

No such control available with firm thresholding.

6

Besov–Lorentz shrinkage outperforms firm thresholding in fitting singularities.

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(46)

(47)

4

The limiting cases soft and hard thresholding can both be

implemented within the wavelet pennalization strategy proposed in [DRP]. Therefore, firm thresholding itself can be implemented with the wavelet pennalization strategy.

5

Trade-off between error of approximation and rate of compression . . . . . . efficiently controllable with Besov–Lorentz shrinkage.

No such control available with firm thresholding.

6

Besov–Lorentz shrinkage outperforms firm thresholding in fitting singularities.

[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov

regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).

(48)

−0.5 0 0.5 1 1.5

−0.2 0

x

(49)

Comparisons on a noisy benchmark image

The same example but now with noise variance 0.1.

−0.5 0 0.5 1 1.5

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4

x

y

Noisy Original Firm Lorentz−Besov

(50)

Wavelet Compression, Data Fitting and Approximation Based on Adaptive Composition of Lorentz-type Thresholding and Besov-type Non-threshold Shrinkage