Bart Vandereycken

Explicit stabilised gradient descent for faster strongly convex optimisation

by , , ,

Abstract:

We introduce the explicit stabilised gradient descent method (ESGD) for strongly convex optimisation problems. This new algorithm is based on explicit stabilised integrators for stiff differential equations, a powerful class of numerical schemes to avoid the severe step size restriction faced by standard explicit integrators. For optimising quadratic and strongly convex functions, we prove that ESGD nearly achieves the optimal convergence rate of the conjugate gradient algorithm, and the suboptimality of ESGD diminishes as the condition number of the quadratic function worsens. We show that this optimal rate is obtained also for a partitioned variant of ESGD applied to perturbations of quadratic functions. In addition, numerical experiments on general strongly convex problems show that ESGD outperforms Nesterov's accelerated gradient descent.

Download links:

[pdf] [doi]

Reference:

A. Eftekhari, B. Vandereycken, G. Vilmart, K. Zygalakis, "Explicit stabilised gradient descent for faster strongly convex optimisation", In BIT Numer. Math., 2020.

Bibtex Entry:

@article{Eftekhari_VVZ:2020,
    Abstract = {We introduce the explicit stabilised gradient descent method (ESGD) for strongly convex optimisation problems. This new algorithm is based on explicit stabilised integrators for stiff differential equations, a powerful class of numerical schemes to avoid the severe step size restriction faced by standard explicit integrators. For optimising quadratic and strongly convex functions, we prove that ESGD nearly achieves the optimal convergence rate of the conjugate gradient algorithm, and the suboptimality of ESGD diminishes as the condition number of the quadratic function worsens. We show that this optimal rate is obtained also for a partitioned variant of ESGD applied to perturbations of quadratic functions. In addition, numerical experiments on general strongly convex problems show that ESGD outperforms Nesterov's accelerated gradient descent.},
    Author = {Eftekhari, A. and Vandereycken, B. and Vilmart, G. and Zygalakis, K.},
    Journal = {BIT Numer. Math.},    
    Title = {Explicit stabilised gradient descent for faster strongly convex optimisation},
    Year = {2020},
    Pdf = {http://www.unige.ch/math/vandereycken/papers/published_Eftekhari_VVZ_2020.pdf},
    Doi = {10.1007/s10543-020-00819-y}
    }