TY - JOUR

T1 - Convergence of Non-Convex Non-Concave GANs Using Sinkhorn Divergence

AU - Adnan, Risman

AU - Saputra, Muchlisin Adi

AU - Fadlil, Junaidillah

AU - Ezerman, Martianus Frederic

AU - Iqbal, Muhamad

AU - Basaruddin, Tjan

N1 - Publisher Copyright:
© 2013 IEEE.

PY - 2021

Y1 - 2021

N2 - Sinkhorn divergence is a symmetric normalization of entropic regularized optimal transport. It is a smooth and continuous metrized weak-convergence with excellent geometric properties. We use it as an alternative for the minimax objective function in formulating generative adversarial networks. The optimization is defined with Sinkhorn divergence as the objective, under the non-convex and non-concave condition. This work focuses on the optimization's convergence and stability. We propose a first order sequential stochastic gradient descent ascent (SeqSGDA) algorithm. Under some mild approximations, the learning converges to local minimax points. Using the structural similarity index measure (SSIM), we supply a non-asymptotic analysis of the algorithm's convergence rate. Empirical evidences show a convergence rate, which is inversely proportional to the number of iterations, when tested on tiny colour datasets Cats and CelebA on the deep convolutional generative adversarial networks and ResNet neural architectures. The entropy regularization parameter $\varepsilon $ is approximated to the SSIM tolerance $\epsilon $. We determine that the iteration complexity to return to an $\epsilon $ -stationary point to be $\mathcal {O}\left ({\kappa \, \log (\epsilon ^{-1})}\right)$ , where $\kappa $ is a value that depends on the Sinkhorn divergence's convexity and the minimax step ratio in the SeqSGDA algorithm.

AB - Sinkhorn divergence is a symmetric normalization of entropic regularized optimal transport. It is a smooth and continuous metrized weak-convergence with excellent geometric properties. We use it as an alternative for the minimax objective function in formulating generative adversarial networks. The optimization is defined with Sinkhorn divergence as the objective, under the non-convex and non-concave condition. This work focuses on the optimization's convergence and stability. We propose a first order sequential stochastic gradient descent ascent (SeqSGDA) algorithm. Under some mild approximations, the learning converges to local minimax points. Using the structural similarity index measure (SSIM), we supply a non-asymptotic analysis of the algorithm's convergence rate. Empirical evidences show a convergence rate, which is inversely proportional to the number of iterations, when tested on tiny colour datasets Cats and CelebA on the deep convolutional generative adversarial networks and ResNet neural architectures. The entropy regularization parameter $\varepsilon $ is approximated to the SSIM tolerance $\epsilon $. We determine that the iteration complexity to return to an $\epsilon $ -stationary point to be $\mathcal {O}\left ({\kappa \, \log (\epsilon ^{-1})}\right)$ , where $\kappa $ is a value that depends on the Sinkhorn divergence's convexity and the minimax step ratio in the SeqSGDA algorithm.

KW - Convergence

KW - generative adversarial networks

KW - optimal transport

KW - Sinkhorn divergence

UR - http://www.scopus.com/inward/record.url?scp=85104665079&partnerID=8YFLogxK

U2 - 10.1109/ACCESS.2021.3074943

DO - 10.1109/ACCESS.2021.3074943

M3 - Article

AN - SCOPUS:85104665079

VL - 9

SP - 67595

EP - 67609

JO - IEEE Access

JF - IEEE Access

SN - 2169-3536

M1 - 9410544

ER -