Over the past a few decades, much attention has been drawn to large-scale incremental
data analysis, where researchers are faced with huge amount of high-dimensional data
acquired incrementally. In such a case, conventional algorithms that compute the result from
scratch whenever a new sample comes are highly inefficient. To conquer this problem, we
propose a new incremental algorithm IRLS that incrementally computes the solution to the
regularized least squares (RLS) problem with multiple columns on the right-hand side.
More specifically, for a RLS problem with c (c > 1) columns on the right-hand side, we update
its unique solution by solving a RLS problem with single column on the right-hand side
whenever a new sample arrives, instead of solving a RLS problem with c columns on the
right-hand side from scratch. As a direct application of IRLS, we consider the supervised
dimensionality reduction of large-scale data and focus on linear discriminant analysis (LDA).
We first propose a new batch LDA model that is closely related to RLS problem, and then
apply IRLS to develop a new incremental LDA algorithm. Experimental results on real-world
datasets demonstrate the effectiveness and efficiency of our algorithms.