What are the Types of Sparse Dictionary Learning Algorithms?
Types of Sparse dictionary learning algorithms is a kind of representation learning method where we express the information as a sparse linear combination of an overcomplete basis ML set. This is actually formulated as an optimization issue which is called NP-Hard.
A general solution of sparse dictionary machine learning algorithms uses a two-step iterative method which involves either a clustering based solution or convex relaxation. One main issue with the typical method is that it may be harsh to interpret the meaning of dictionary learning atoms.
Sparse representations have found varieties of applications in signals and image processing, programming, compression, classification, and other such fields. Their success actually relies on parsimony principle: few members of an overcomplete basis can provide an immense variety of models. The overcomplete basis, or dictionary learning, can be fixed or adapted to the types of dictionary learning algorithms application.
Learned dictionaries outperform fixed dictionaries in many applications. There are already well-established algorithms for the standard problem, but the subject remains open for variants of the learning problem, as well as applications.
7 Types of Sparse Dictionary Learning Algorithms
Most of the types of dictionary learning algorithms are based on the idea of iteratively updating one & then the other.
The issue of finding an optimal sparse coding R with a provided dictionary D is called as sparse approximation (or just sparse coding issue). There are number of dictionary learning algorithms to solve the problems (like matching pursuit and LASSO) which are incorporated into the dictionary learning algorithms algorithms described below.
1. Method of optimal directions (MOD)
The method of optimal directions (or you can say MOD) was one of the first methods used and introduced to tackle the sparse dictionary learning issue. The main idea behind it is to solve the minimization issue subject to the limited number of non-zero methods or components of the representation vector:
Here, in the equation F denotes the Frobenius norm. MOD in types of dictionary learning algorithms alternates between getting the sparse coding using a process such as matching pursuit and updating the types of dictionary learning algorithms by computing the analytical solution of this numerical problem given by D = XR+ where R+ is a Moore-Penrose pseudoinverse.
After this update D is renormalized or reestablished to fit the constraints and the new sparse programming is obtained again. The method is repeated until convergence (or until a sufficiently tiny residue). MOD in types of dictionary learning algorithms has proved to be a very reliable and efficient method for the low-dimensional input data x requiring just a limited iterations to converge.
2. K-SVD
K-SVD is a types of dictionary learning algorithm that performs SVD at its main part to update the atoms of the dictionary learning one by one and basically it is a general aization of K-means. It enforces that each key element of the input data xi is then encoded by a linear combination of not more than the element T0 in a way identical to the MOD method:
This type of dictionary learning algorithm’s essence is to first fix the dictionary part, and then find the best possible R under the above constraint (with Orthogonal Matching Pursuit) & then iteratively update the elements of dictionary D in the following manner:
The next steps of this ML algorithm include rank-1 approximation of residual matrix Ek , updating dk and enforcing the sparsity algorithm of xk after the update. This dictionary learning algorithm is considered to be standard for learning and is used in numerous applications.
But it shares weaknesses with MOD algorithm being efficient only for signals with extremely low dimensionality and having the great possibility for being stuck at local minima.
3. Stochastic gradient descent
You can also apply a widespread stochastic gradient descent process with iterative projection to solve this issue or problem. The idea of this type of dictionary learning algorithm is to update the dictionary with the first order stochastic gradient and then project it on the constraint set C. The main step that occurs at i-th iteration is described by this expression below:
where S is a random subset of the {1….K} and x2
where is a random subset of the
} and
is a gradient step.
4. Lagrange dual method
A dictionary learning algorithm based on solving a dual Lagrangian problem and provides an efficient and reliable way to solve for the dictionary problems by having no complications induced by the sparsity learning function. Consider the following Lagrangian equation:
where is a constraint on the norm of the atoms/elements and
are the so-called dual variables – forming the diagonal matrix
.
We can then offer an analytical expression for the Lagrange dual after the minimization over :
After applying one of the optimization techniques to the value of the dual (like Newton’s process or conjugate gradient) we get the value of :
Solving this major problem is less computational hard because of the amount of dual variables is a lot of times much more less than the amount of variables in this primal issue.
5.   LASSO
In this approach of dictionary learning algorithm, the optimization issue is formulated as:
here is the permitted error in the reconstruction LASSO.
It finds an estimate of about by minimizing the least square error subject to a L1-norm constraint in solution vector, formulated as:
here controls the trade-off among sparsity & the reconstruction error. This provides the global optimal solution.
6.   Parametric training methods in Sparse Dictionary Learning Algorithms
Parametric training methods or techniques are aimed to focus on incorporate and contribute the best of the both worlds – the analytically constructed dictionaries & the learned ones. This helps to construct more powerful generalized kerning dictionaries that can possibly be applied to the cases of arbitrary-sized learning signals. Notable approaches of this learning algorithm include:
- Multiscale Dictionaries: This technique focuses on constructing a dictionary learning method that is composed of differently scaled dictionaries algorithms to improve sparsity.
- Translation-invariant Method in Dictionaries Algorithm: These type of dictionary learning technique is made by the translations of the atoms coming from the dictionary which is constructed for a finite-size signal patch. This helps the resulting dictionary to provide a representation for the actual arbitrary-sized signal.
- Sparse Dictionaries: This method mainly focuses on not only giving a sparse representation but also constructing dictionary learning algorithms which is enforced by the equation
where
is some predefined analytical dictionary with desirable properties like fast computation and
is a sparse matrix. These formulation helps to directly combine the fast implementation of the analytical dictionaries with the flexibility of sparse techniques.
7.   Online dictionary learning (LASSO approach)
Many common methods to sparse dictionary learning algorithm rely on this fact that the whole of the input data (or at least a large part of training dataset) is available for this algorithm.
But this might not be the actual case in real-world scenario as size of the input data might be too big to handle and fit it into memory. The other scenario where this assumption can not be created is when the input data comes in a shape of a stream. These cases lie in the field of study of online learning algorithms which importantly suggests iteratively updating this ML model upon the new data points becoming more available. A dictionary algorithm can be learned in an online manner the following way:
- For
- Draw a new sample
- Find a sparse code using LARS:
Update the dictionary with block-coordinate approach:
This technique allows us to slowly update the dictionary learning algorithm as new data becomes available for the sparse representation learning & helps drastically reduce the space of memory needed to store the dataset (which often has a large size).
Applications of Sparse Dictionary Learning Algorithms
This Machine Learning method can be applied to classification issues in such a way that if we have made specific dictionary algorithms for each type of class, the input signal can be then classified by finding and searching the dictionary method similar to the sparsest representation.
The dictionary learning algorithms – namely the linear decomposition of any sort of input signal with extremely low basis elements – learned from the information itself, has led to the state-of-art results in various image & video processing industry.
One of the essential applications of the sparse learning algorithm is in the field of signal recovery or compress sensing. In the compress sensing technique, high dimensional signal can be easily recovered with only a few linear measurements.
Types of dictionary learning algorithms are used to analyse medical signals in complete detail. Such medical signals include signals from electroencephalography (EEG), electrocardiography (ECG), functional MRI (fMRI), & ultrasound computer tomography (USCT), where varieties of assumptions are used to analyze each computer signal.
Conclusion
Dictionary learning algorithm is the method or procedure of designing dictionaries based on samples from the procedure to be modeled. In many different types of dictionary learning algorithm applications, learned dictionaries offer much better learning performance than fixed ones.
Types of dictionary learning algorithms has been successfully applied to many fields like image, video & audio processing related tasks as well as to texture synthesis & unsupervised clustering in the evaluations with the Bag-of-Words model, sparse learning algorithm was found empirically to outperform other programming approaches on the object category recognition works.
About Post Author
