Date of Award
6-2025
Document Type
Dissertation
Publisher
Santa Clara : Santa Clara University, 2025
Degree Name
Doctor of Philosophy (PhD)
Department
Electrical and Computer Engineering
First Advisor
Tokunbo Ogunfunmi
Abstract
We are at the dawn of an interesting era in the history of humankind where artificial intelligence (AI) is starting to play a significant role in our daily lives. Large Language Models (LLMs) like ChatGPT, Bard, Llama and others have shown a remarkable ability to converse with humans in many different languages across myriad topics. Machines which “learn” do so by creating a network of artificial neurons which loosely mimic the neurons in the human brain. Even though humans are now increasingly interacting with these machines, a clear understanding of how these machines exactly learn, remains elusive. In this study we look under the hood to explore how information flows through the layers of the machine’s artificial neural network, some of which are several levels deep. This dissertation looks at deep feed-forward neural networks, Convolutional Neural Networks (CNNs) and Generative AI models through the lens of Information Theory. It uses information theoretical concepts such as Rényi’s generalized entropy (of which Shannon’s entropy is a special case), mutual information, information channel capacity, various f-divergence measures and information geometry on Riemannian manifolds to understand the inner workings of these deep neural networks. Information Theoretic Learning (ITL) techniques based on information particles and information potential are used in this thesis to illustrate how information theory can be used to improve the learning process in neural networks. This thesis also combines principles from physics and statistical mechanics with information theory to describe how a generative AI model can learn the latent parameters of the input data to generate new samples.
Recommended Citation
Deb, Manas, "Information Theoretical Analysis of Deep Neural Networks" (2025). Engineering Ph.D. Theses. 62.
https://scholarcommons.scu.edu/eng_phd_theses/62
