Transfer Talk - Sean Quinn - 3rd June 2020

Video Category: 
Transfer Talk

Title: Towards Architecture-Agnostic Knowledge Transfer in Deep Neural Networks

Modern artificial neural networks can leverage large amounts of data to be trained to
perform hard tasks such as recognising objects in an image or translating languages.
The neural learning process exposes the underlying compositional and hierarchical
structure of concepts contained within high dimensional data; however, this pro-
cess does not typically provide high-level access to such structure or easily facilitate
its re-use in related tasks. Neural knowledge transfer (NKT) methods aim to con-
strain the hidden representation of one neural network to be similar, or have similar
properties, to another by applying specially designed loss functions between the two
networks hidden layers. In this way the intangible knowledge encoded by the net-
works weights is transferred without having to replicate exact weight structures or
alter the knowledge representation from its natural highly distributed form. This
research hypotheses that we can increase the applicability of knowledge transfer
between deep neural networks by focusing on the identification and removal of archi-
tectural constraints inherent to contemporary neural knowledge transfer methods.
We view the removal of these architectural constraints as a key enabler in facilitating
this type of knowledge transfer in a much wider set of domains and use cases. In this
research we discuss the context of neural knowledge transfer within the body of deep
learning literature, analyse the current approaches, identify trends in NKT method
design, identify and discuss architectural constraints and conduct an experimental
comparison. To the best of our knowledge this is the first research conducted on
neural knowledge transfer from the perspective of architectural constraints and our
experimental comparison is more expensive than those currently available in the
literature. On final completion of this research project we aim to have progressed
the state of the art in neural knowledge transfer and made a signi cant contribution
to the body of knowledge, laying the foundations for future advances.