Combining normal sparse into discriminative deep belief networks

Faisal Khalid, Mohamad Ivan Fanany

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Training deep models are time consuming and face many local minima. For dealing with this problem, we can use DBN (Deep Belief Network) with Contrastive Divergence (CD). Sparse representations are more efficient. This paper aims to find the best structure of incorporating sparsity into discriminative DBN. We use a DBN architecture with 784 units as input, two layers each with 500 hidden units, one layer with 2000 hidden units, and 10 units as the final output. We argue that combining sparsity and discriminative DBN may increase the accuracy, but no previous studies suggest the best structure or configuration of that combination that can give the best accuracy. We took three stages of experiments to find the best configuration, namely preliminary, intermediate, and final stages. Each analysis of each stage serves as a background for consideration of the next experiment We use normal sparse for generative DBN and discriminative DBN. Experimental studies on MNIST dataset show that the best structure or scenario to combine normal sparse into deep belief networks is as follows: input - generative (CD) - generative (CD) - normal sparse discriminative (CD).

Original languageEnglish
Title of host publication2016 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages373-378
Number of pages6
ISBN (Electronic)9781509046294
DOIs
Publication statusPublished - 6 Mar 2017
Event8th International Conference on Advanced Computer Science and Information Systems, ICACSIS 2016 - Malang, Indonesia
Duration: 15 Oct 201616 Oct 2016

Publication series

Name2016 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2016

Conference

Conference8th International Conference on Advanced Computer Science and Information Systems, ICACSIS 2016
Country/TerritoryIndonesia
CityMalang
Period15/10/1616/10/16

Keywords

  • Deep Belief Networks
  • Discriminative Deep Belief Network
  • Normal Sparse
  • Restricted Boltzmann Machines
  • Sparsity

Fingerprint

Dive into the research topics of 'Combining normal sparse into discriminative deep belief networks'. Together they form a unique fingerprint.

Cite this