Deep and Wide Neural Networks Covariance Estimation

Argimiro Arratia; Alejandra Cabaña; José Rafael León

doi:10.1007/978-3-030-61609-0_16

Deep and Wide Neural Networks Covariance Estimation

Argimiro Arratia^*, Alejandra Cabaña, José Rafael León

^*Corresponding author for this work

Department of Mathematics

Research output: Chapter in Book › Chapter › Research › peer-review

Abstract

It has been recently shown that a deep neural network with i.i.d. random parameters is equivalent to a Gaussian process in the limit of infinite network width. The Gaussian process associated to the neural network is fully described by a recursive covariance kernel determined by the architecture of the network, and which is expressed in terms of expectation. We give a numerically workable analytic expression of the neural network recursive covariance based on Hermite polynomials. We give explicit forms of this recursive covariance for the cases of neural networks with activation function the Heaviside, ReLU and sigmoid.

Original language	English
Title of host publication	Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings
Editors	Igor Farkaš, Paolo Masulli, Stefan Wermter
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	195-206
Number of pages	12
ISBN (Print)	9783030616083
DOIs	https://doi.org/10.1007/978-3-030-61609-0_16
Publication status	Published - 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12396 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Keywords

Deep neural networks
Gaussian process
Hermite polynomials
Kernels

Access to Document

10.1007/978-3-030-61609-0_16

Cite this

Arratia, A., Cabaña, A., & León, J. R. (2020). Deep and Wide Neural Networks Covariance Estimation. In I. Farkaš, P. Masulli, & S. Wermter (Eds.), Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings (pp. 195-206). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12396 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-61609-0_16

Arratia, Argimiro ; Cabaña, Alejandra ; León, José Rafael. / Deep and Wide Neural Networks Covariance Estimation. Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings. editor / Igor Farkaš ; Paolo Masulli ; Stefan Wermter. Springer Science and Business Media Deutschland GmbH, 2020. pp. 195-206 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inbook{8ea007fbcab44752889e65abe7616277,

title = "Deep and Wide Neural Networks Covariance Estimation",

abstract = "It has been recently shown that a deep neural network with i.i.d. random parameters is equivalent to a Gaussian process in the limit of infinite network width. The Gaussian process associated to the neural network is fully described by a recursive covariance kernel determined by the architecture of the network, and which is expressed in terms of expectation. We give a numerically workable analytic expression of the neural network recursive covariance based on Hermite polynomials. We give explicit forms of this recursive covariance for the cases of neural networks with activation function the Heaviside, ReLU and sigmoid.",

keywords = "Deep neural networks, Gaussian process, Hermite polynomials, Kernels",

author = "Argimiro Arratia and Alejandra Caba{\~n}a and Le{\'o}n, {Jos{\'e} Rafael}",

year = "2020",

doi = "10.1007/978-3-030-61609-0_16",

language = "English",

isbn = "9783030616083",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "195--206",

editor = "Igor Farka{\v s} and Paolo Masulli and Stefan Wermter",

booktitle = "Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings",

}

Arratia, A, Cabaña, A & León, JR 2020, Deep and Wide Neural Networks Covariance Estimation. in I Farkaš, P Masulli & S Wermter (eds), Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12396 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 195-206. https://doi.org/10.1007/978-3-030-61609-0_16

Deep and Wide Neural Networks Covariance Estimation. / Arratia, Argimiro; Cabaña, Alejandra; León, José Rafael.
Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings. ed. / Igor Farkaš; Paolo Masulli; Stefan Wermter. Springer Science and Business Media Deutschland GmbH, 2020. p. 195-206 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12396 LNCS).

Research output: Chapter in Book › Chapter › Research › peer-review

TY - CHAP

T1 - Deep and Wide Neural Networks Covariance Estimation

AU - Arratia, Argimiro

AU - Cabaña, Alejandra

AU - León, José Rafael

PY - 2020

Y1 - 2020

N2 - It has been recently shown that a deep neural network with i.i.d. random parameters is equivalent to a Gaussian process in the limit of infinite network width. The Gaussian process associated to the neural network is fully described by a recursive covariance kernel determined by the architecture of the network, and which is expressed in terms of expectation. We give a numerically workable analytic expression of the neural network recursive covariance based on Hermite polynomials. We give explicit forms of this recursive covariance for the cases of neural networks with activation function the Heaviside, ReLU and sigmoid.

AB - It has been recently shown that a deep neural network with i.i.d. random parameters is equivalent to a Gaussian process in the limit of infinite network width. The Gaussian process associated to the neural network is fully described by a recursive covariance kernel determined by the architecture of the network, and which is expressed in terms of expectation. We give a numerically workable analytic expression of the neural network recursive covariance based on Hermite polynomials. We give explicit forms of this recursive covariance for the cases of neural networks with activation function the Heaviside, ReLU and sigmoid.

KW - Deep neural networks

KW - Gaussian process

KW - Hermite polynomials

KW - Kernels

UR - http://www.scopus.com/inward/record.url?scp=85096593821&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-61609-0_16

DO - 10.1007/978-3-030-61609-0_16

M3 - Chapter

AN - SCOPUS:85096593821

SN - 9783030616083

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 195

EP - 206

BT - Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings

A2 - Farkaš, Igor

A2 - Masulli, Paolo

A2 - Wermter, Stefan

PB - Springer Science and Business Media Deutschland GmbH

ER -

Arratia A, Cabaña A, León JR. Deep and Wide Neural Networks Covariance Estimation. In Farkaš I, Masulli P, Wermter S, editors, Artificial Neural Networks and Machine Learning – ICANN 2020 - 29th International Conference on Artificial Neural Networks, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 195-206. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-61609-0_16

Deep and Wide Neural Networks Covariance Estimation

Abstract

Publication series

Keywords

Access to Document

Other files and links

Fingerprint

Cite this