Wavelet Compression, Data Fitting and Approximation Based on Adaptive Composition of Lorentz-type Thresholding and Besov-type Non-threshold Shrinkage
Lubomir T. Dechevsky 1 , Joakim Gundersen 1 and Niklas Grip 2
1 Narvik University College, Norway
2 Lule˚ a University of Technology, Sweden
7th International Conference on
Large-Scale Scientific Computations
Sozopol, Bulgaria, June 4–8 2009
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 1 / 13
Wavelet Compression, Data Fitting and Approximation Based on Adaptive Composition of Lorentz-type Thresholding and Besov-type Non-threshold Shrinkage
Lubomir T. Dechevsky
1, Joakim Gundersen
1and Niklas Grip
2 1 Narvik University College, Norway 2 Lule˚a University of Technology, Sweden7th International Conference on
Large-Scale Scientific Computations
Sozopol, Bulgaria, June 4–8 2009
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
My talk will be about different data fitting and approximation techniques for signals belonging to a wide range of Besov spaces. This work is done together with my colleagues Lubomir T. Dechevsky and Joakim
Gundersen.
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 2 / 13
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
I’m from Lule˚ a University of Technology in Sweden and my co-authors
are from Narvik University College in Norway.
Outline
1 The central role of wavelet expansions in Besov space theory
2 Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage
Non-thresholding wavelet shrinkage Composite Besov–Lorentz shrinkage
3 Besov–Lorentz shrinkage versus firm thresholding
4 Comparisons on a noisy benchmark image
Overview now, details and further references in the paper and in later discussions.
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 3 / 13
1The central role of wavelet expansions in Besov space theory
2Three different families of wavelet shrinkage methods
Classical wavelet threshold shrinkage Non-thresholding wavelet shrinkage Composite Besov–Lorentz shrinkage
3Besov–Lorentz shrinkage versus firm thresholding4Comparisons on a noisy benchmark image
Overview now, details and further references in the paper and in later discussions.
2009- 06- 07
Wavelet Compression, Data Fitting and . . . Outline
1. I will begin with a very brief explanation of the central role that wavelet expansions play in giving a characterazion of Besov spaces.
2. Then I will describe three different families of wavelet coefficient shrinking methods for data fitting and approximation.
3. Then I will describe in more detail some main differences between two of these approaches.
4. Finally, I will show a comparison of these methods on a noisy benchmark image.
My aim in this talk will be to give an overview of the different methods. I will therefore have to mention some of the deeper mathematical parts only by name, and leave the details and further references for our paper and for later discussions.
The central role of wavelet expansions in Besov space theory
Wavelet expansions:
Unconditional frame/basis for a large range of Besov spaces B p s( ,q ·) ( R n )
B p s( ,q ·) ( R n ) defined by wavelet expansion
f =
k ∈Z n
α 0 ,k ϕ [0] 0 ,k (x) +
j ≥0
k ∈Z n 2 n −1
l =1
β j [l ] ,k ψ [l ] j ,k (x) , a.e. x ∈ R n
and quasi-norm topology computed on the wavelet coefficients
f B s( ·) p ,q =
⎡
⎣
k ∈Z n
|α 0 ,k | p
q
p
+
j ≥0
k ∈Z n
2 j
s j ,k n
1 2 − 1 p
p 2 n −1
l =1
β j [l ] ,k p
q p ⎤
⎦
q
,
with local smoothness index s j ,k = s(x 0 + 2 −j k).
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 4 / 13
Wavelet expansions:
Unconditional frame/basis for a large range of Besov spaces Bp,qs(·)(Rn)
B
s(·)p,q(R
n) defined by wavelet expansion
f =
k∈Znα
0,kϕ
[0]0,k(x) +
j≥0 k∈Zn 2n−1 l =1β
[l ]j,kψ
j,k[l ](x), a.e. x ∈ R
nand quasi-norm topology computed on the wavelet coefficients
f
Bp,qs(·)=
⎡
⎣
k∈Zn
|α
0,k|
p qp+
j≥0
k∈Zn
2
jsj,k n1
2 −1 p
p2n−1
l =1β
[l ]j,kp q p
⎤
⎦
q, with local smoothness index s
j,k= s(x
0+ 2
−jk).
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
The central role of wavelet expansions in Besov space theory
Wavelet expansions:
• A large range of Besov spaces, can be characterized by unconditional wavelet frame or basis series expansions of its elements f as a sum of a coarse scale approximation. . .
• . . . and a telescope sum that adds finer and finer details . . .
• . . . with Besov space quasi-norm topology computed on the sequence space of wavelet coefficients.
• This talk is about different shrinkage techniques for noise reduction, data
fitting and approximation.
Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage
“Classical” wavelet threshold shrinkage:
Appropriate for relatively regular (smooth) signals.
Hard and soft thresholding are limiting cases of firm thresholding:
( iii) → (i) when λ 2 → λ 1 and ( iii) → (ii) when λ 2 → ∞
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 5 / 13
“Classical” wavelet threshold shrinkage:
Appropriate for relatively regular (smooth) signals.
Hard and soft thresholding are limiting cases of firm thresholding:
(iii) → (i) when λ
2→ λ
1and (iii) → (ii) when λ
2→ ∞
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage
“Classical” wavelet threshold shrinkage:
1. The most well-known such technique is tresholding shrinkage, which is aimed for relatively regular signals, that is, signals whose shape is mainly determined by a few relatively large wavelet coefficients.
2. A threshold shrinkage rule sets to zero all coefficients with absolute value smaller than some threshold λ. If this is the only thing done, then we get . . 3. Hard thresholding.
4. Soft thresholding also shrinks the remaining coefficients to avoid the discontinuity in (i).
5. Hard and soft thresholding are limiting cases of firm thresholding.
6. These limiting cases occur when λ 2 → λ 1 . . . 7. . . . and when λ 2 → ∞.
Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage
(iv) Lorentz-curve thresholding: Vidakovic 1999 [V], based on the Lorentz curve for the energy in the wavelet decomposition.
(v) General Lorentz thresholding [DRP]: Generalization based on combined use of two function-analytical and operator-theoretical facts:
(A) Computability of the Peetre K -functional between Lebesgue spaces in terms of non-increasing rearrangements of a measurable function.
(B) Isomorphism of Besov Spaces to vector-valued spaces of Lebesgue type.
When applicable (=for smooth signals), these threshold techniques also gives compression. However, not all signals/functions are smooth!
[V] Vidakovic, B., Statistical Modeling by Wavelets. Wiley, New York, (1999).
[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 6 / 13
(iv)
Lorentz-curve thresholding: Vidakovic 1999 [V], based on the Lorentz curve for the energy in the wavelet decomposition.
(v)
General Lorentz thresholding [DRP]: Generalization based on combined use of two function-analytical and operator-theoretical facts:
(A)Computability of the Peetre K -functional between Lebesgue spaces in terms of non-increasing rearrangements of a measurable function.
(B)Isomorphism of Besov Spaces to vector-valued spaces of Lebesgue type.
When applicable (=for smooth signals), these threshold techniques also gives compression. However, not all signals/functions are smooth!
[V]Vidakovic, B., Statistical Modeling by Wavelets. Wiley, New York, (1999).
[DRP]Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
Three different families of wavelet shrinkage methods Classical wavelet threshold shrinkage
1. One further refinement of these techniques is Lorentz-curve thresholding, proposed by Vidakovic. It chooses the threshold by incorporating a certain use of the Lorentz curve for the energy in the wavelet decomposition.
2. A far-going generalization of Lorentz thresholding was proposed in a paper from the same year.
3. This generalization is based on combined use of two function-analytical and operator-theoretical facts:
(A) . . . and (B) . . .
4. To recapitulate, this was different thresholding techniques that, when
applicable, that is, for smooth signals, provides not only noise reduction,
data fitting or approximation, but also comppression of the signal, since
there is only a relatively small number of nonzero wavelet coefficients that
need to be stored. However, not all signals in this world are smooth!
Three different families of wavelet shrinkage methods Non-thresholding wavelet shrinkage
Non-thresholding wavelet shrinkage [DRP,MGD,DGG]:
Appropriate for relatively non-regular (fractal) signals.
A fairly new family of nonthreshold (=no compression!) wavelet shrinkage estimators. Appropriate for fractal signals, e.g., continuous but everywhere nonsmoooth signals, such as the Weierstrass function.
Then typically a full, non-sparse sequence of wavelet coefficients.
Parallels Wahba’s spline smoothing technique based on Tikhonov regularization of ill-posed inverse problems, but now for wavelets.
Based on 2 function-analytical and operator-theoretical facts [BL]:
(C) Metrizability of quasinormed abelian groups via the Method of Powers.
(D) The Theorem of Powers for the real interpolation method of
Peetre–Lions, leading to explicit computation of the K -functional (in this case also called quasilinearization).
[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov regularity constraints.
Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).
[MGD] Moguchaya, T., Grip, N., Dechevsky, L. T., Bang, B., Laks˚ a, A., Tong, B., Curve and surface fitting by wavelet shrinkage using GM Waves. Mathematical Methods for Curves and Surfaces, ed. M. Dæhlen and K. Mørken and L.
Schumaker, 263–274, Nashboro Press, Brentwood, North Carolina, (2005).
[DGG] Dechevsky, L. T., Grip, N., Gundersen, J., A new generation of wavelet shrinkage: adaptive strategies based on compostion of Lorentz-type thresholding and Besov-type non-thresholding shrinkage. In: Proceedings of SPIE: Wavelet Applications in Industrial Processing V, Boston, MA, USA 6763(2007), article 676308, pp. 1–14.
[BL] Bergh, J., L¨ ofstr¨ om, J., Interpolation spaces. An introduction. Grundlehren der Mathematischen Wissenshaften, 223, Springer-Verlag, Berlin-New York, (1976). Ch. 3 & 5
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 7 / 13
Non-thresholding wavelet shrinkage [DRP,MGD,DGG]:
Appropriate for relatively non-regular (fractal) signals.
A fairly new family of nonthreshold (=no compression!) wavelet shrinkage estimators. Appropriate for fractal signals, e.g., continuous but everywhere nonsmoooth signals, such as the Weierstrass function.
Then typically a full, non-sparse sequence of wavelet coefficients.
Parallels Wahba’s spline smoothing technique based on Tikhonov regularization of ill-posed inverse problems, but now for wavelets.
Based on 2 function-analytical and operator-theoretical facts [BL]:
(C)Metrizability of quasinormed abelian groups via the Method of Powers.
(D) The Theorem of Powers for the real interpolation method of Peetre–Lions, leading to explicit computation of the K -functional (in this case also called quasilinearization).
[DRP]Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov regularity constraints.
Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).
[MGD]Moguchaya, T., Grip, N., Dechevsky, L. T., Bang, B., Laks˚a, A., Tong, B., Curve and surface fitting by wavelet shrinkage using GM Waves. Mathematical Methods for Curves and Surfaces, ed. M. Dæhlen and K. Mørken and L.
Schumaker, 263–274, Nashboro Press, Brentwood, North Carolina, (2005).
[DGG]Dechevsky, L. T., Grip, N., Gundersen, J., A new generation of wavelet shrinkage: adaptive strategies based on compostion of Lorentz-type thresholding and Besov-type non-thresholding shrinkage. In: Proceedings of SPIE: Wavelet Applications in Industrial Processing V, Boston, MA, USA 6763(2007), article 676308, pp. 1–14.
[BL]Bergh, J., L¨ofstr¨om, J., Interpolation spaces. An introduction. Grundlehren der Mathematischen Wissenshaften, 223, Springer-Verlag, Berlin-New York, (1976). Ch. 3 & 5
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
Three different families of wavelet shrinkage methods Non-thresholding wavelet shrinkage
Non-thresholding wavelet shrinkage [DRP,MGD,DGG]:
• (Basically just state each of the above points.)
• . . . to make a long story short: (C) and (D)
If somebody asks: This is also a theory of best approximation but with stochastic metric. Estimator of type ˆ f =
j ,k β j ,k ψ j ,k with stochastic coefficients β j ,k . . .
E f − ˆf 2 = E (f − E ˆf) + (E ˆf − ˆf) 2
= E
(f − E ˆf) + (E ˆf − ˆf), (f − E ˆf) + (E ˆf − ˆf)
= E
f − Eˆf 2 + 2
f − E ˆf, E ˆf − ˆf
+ E ˆf − ˆf 2
= E f − E ˆf 2
=bias
+2
f − E ˆf, E ˆf − E ˆf
=0
+ E E ˆf − ˆf 2
=variance
Thus we have the problem of joint minimization of both the bias and the variance.
Often, small bias instead gives big variance.
Three different families of wavelet shrinkage methods Composite Besov–Lorentz shrinkage
Composite Besov–Lorentz shrinkage:
Finds points of singularity and uses Besov non-thresholding there
First announced in [DGG]. Relying on certain real interpolation spaces (B qq σ , B pp s ) θ,p(θ) = B p( s( θ) θ)p(θ) , 0 ≤ θ ≤ 1 and 1
p( θ)
def = 1 − θ q + θ
p . Parameter θ: Determines the degree to which the composite operator is of Lorentz type or a Besov shrinkage-type estimator.
Blending of Lorentz threshold and non-threshold method ⇒ θ is also control parameter for regulating the compression rate.
Based on six function-analytical facts:
(A)-(D) as in the previous slides
(E) The reiteration theorem for the real interpolation method of Peetre-Lions.
(F) The generalization, via the Holmstedt formula, of the formula for computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.
[DGG] Dechevsky, L. T., Grip, N., Gundersen, J., A new generation of wavelet shrinkage: adaptive strategies based on compostion of Lorentz-type thresholding and Besov-type non-thresholding shrinkage. In: Proceedings of SPIE: Wavelet Applications in Industrial Processing V, Boston, MA, USA 6763(2007), article 676308, pp. 1–14.
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 8 / 13
Composite Besov–Lorentz shrinkage:
Finds points of singularity and uses Besov non-thresholding there
First announced in [DGG]. Relying on certain real interpolation spaces (B
σqq, B
spp)
θ,p(θ)= B
s(θ)p(θ)p(θ), 0 ≤ θ ≤ 1 and 1
p(θ)
def= 1− θ q + θ
p . Parameterθ: Determines the degree to which the composite operator is of Lorentz type or a Besov shrinkage-type estimator.
Blending of Lorentz threshold and non-threshold method⇒ θ is also control parameter for regulating the compression rate.
Based on six function-analytical facts:
(A)-(D)as in the previous slides
(E)The reiteration theorem for the real interpolation method of Peetre-Lions.
(F) The generalization, via the Holmstedt formula, of the formula for computation of the Peetre K -functional between Lebesgue spaces in terms of the non-increasing rearrangement of a measurable function.
[DGG]Dechevsky, L. T., Grip, N., Gundersen, J., A new generation of wavelet shrinkage: adaptive strategies based on compostion of Lorentz-type thresholding and Besov-type non-thresholding shrinkage. In: Proceedings of SPIE: Wavelet Applications in Industrial Processing V, Boston, MA, USA 6763(2007), article 676308, pp. 1–14.
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
Three different families of wavelet shrinkage methods Composite Besov–Lorentz shrinkage
Composite Besov–Lorentz shrinkage:
1. OK, now I have described one shrinkage approach for smooth signals and one for nonsmooth or fractal signals, but what if we need to analyze signals that are a little of both? Smooth at some places and nonsmooth or even fractal at other places? Can we then somehow “detect” the local level of smoothness and then locally choose shrinkage method accordingly?
2. Yes we can! and this is what composite Besov-Lorentz shrinkage is about.
3. (Basically just state each of the above points.)
4. Again, to make a long story short, . . . (A)–(D), (E), (F).
Besov–Lorentz shrinkage versus firm thresholding
Optimization with respect to all parameters of the Besov-Lorentz model would be a considerably more challenging computational problem than optimization related to firm thresholding.
However, several advantages:
1 Derived from the important function-analytic properties (A)-(F) stated in the previous slides. (Rather than only unification of hard and soft thresholding.)
2 Convenient framework for fine control of the optimization.
Can be performed under a rich variety of meaningful constraints.
Allows introduction of bias in the estimation process, whenever information about such bias is available, with drastic improvement in the quality of estimation.
3 The optimization proposed by Gao and Bruce 1997 for firm
thresholding is of entropy type and general, but inflexible with respect to introduction of meaningful bias information
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 9 / 13
Optimization with respect to all parameters of the Besov-Lorentz model would be a considerably more challenging computational problem than optimization related to firm thresholding.
However, several advantages:
1
Derived from the important function-analytic properties (A)-(F) stated in the previous slides. (Rather than only unification of hard and soft thresholding.)
2
Convenient framework for fine control of the optimization.
Can be performed under a rich variety of meaningful constraints.
Allows introduction of bias in the estimation process, whenever information about such bias is available, with drastic improvement in the quality of estimation.
3
The optimization proposed by Gao and Bruce 1997 for firm thresholding is of entropy type and general, but inflexible with respect to introduction of meaningful bias information
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
Besov–Lorentz shrinkage versus firm thresholding
(Basically just state each of the above points.)
Besov–Lorentz shrinkage versus firm thresholding
4 The limiting cases soft and hard thresholding can both be
implemented within the wavelet pennalization strategy proposed in [DRP]. Therefore, firm thresholding itself can be implemented with the wavelet pennalization strategy.
5 Trade-off between error of approximation and rate of compression . . . . . . efficiently controllable with Besov–Lorentz shrinkage.
No such control available with firm thresholding.
6 Besov–Lorentz shrinkage outperforms firm thresholding in fitting singularities.
[DRP] Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 10 / 13
4
The limiting cases soft and hard thresholding can both be implemented within the wavelet pennalization strategy proposed in [DRP]. Therefore, firm thresholding itself can be implemented with the wavelet pennalization strategy.
5
Trade-off between error of approximation and rate of compression . . .
. . . efficiently controllable with Besov–Lorentz shrinkage.No such control available with firm thresholding.
6
Besov–Lorentz shrinkage outperforms firm thresholding in fitting singularities.
[DRP]Dechevsky, L. T., Ramsay, J. O., Penev, S. I., Penalized wavelet estimation with Besov regularity constraints. Mathematica Balkanika (N.S.), NY, 13(3–4):257–356, (1999).
2009- 06- 07
Wavelet Compression, Data Fitting and . . .
Besov–Lorentz shrinkage versus firm thresholding
(Basically just state each of the above points.)
Comparisons on a noisy benchmark image
White unbiased noise, variance 0.01, Daubechies 6 wavelets.
−0.5 0 0.5 1 1.5
−0.2 0 0.2 0.4 0.6 0.8 1 1.2
x
y
Noisy Original Firm Lorentz−Besov
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 11 / 13
White unbiased noise, variance 0.01, Daubechies 6 wavelets.
−0.5 0 0.5 1 1.5
−0.2 0 0.2 0.4 0.6 0.8 1 1.2
x
y
Noisy Original Firm Lorentz−Besov
2009- 06- 07
Wavelet Compression, Data Fitting and . . . Comparisons on a noisy benchmark image
• In the presence of small to medium noise, the performance of Besov–Lorent is slightly better at the singularity at 0 than that of firm thresholding, whic oversmooths at 0.
• In “the smooth part of the signal”, the two are comparable (and very close to each other).
Comparisons on a noisy benchmark image
The same example but now with noise variance 0.1.
−0.5 0 0.5 1 1.5
−0.6
−0.4
−0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4
x
y
Noisy Original Firm Lorentz−Besov
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 12 / 13
The same example but now with noise variance 0.1.
−0.5 0 0.5 1 1.5
−0.6
−0.4
−0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4
x
y
Noisy Original Firm Lorentz−Besov
2009- 06- 07
Wavelet Compression, Data Fitting and . . . Comparisons on a noisy benchmark image
• In the presence of large or very large noise amplitude (comparable to or eve exceeding the size of the signal), Besov–Lorentz considerably outperforms firm thresholding at the singularity.
• Firm thresholding drastically oversmooths at the singularity, while Besov–Lorentz clearly detects the singularity.
• In the remaining part of the graph, thetwo yield comparable (even seemingl identcal!) results.
• In “the smooth part of the signal”, the two are comparable (and very close
to each other).
Comparisons on a noisy benchmark image
Remark
If conventional orthonormal Daubechies wavelets are used, the good fit of isolated singularities comes at the price of overfitting smooth parts of the signal neighbouring the respective isolated singularity. However, this overfit can be removed by using multiwavelets or wavelet packets which, unlike Daubechies orthogonal wavelets, simultaneously combine sufficient smoothness with narrow support.
Dechevsky, Gundersen and Grip () Wavelet Compression, Data Fitting and . . . LSSC’09 Sozopol 13 / 13
Remark