Active Dendrites avoid catastrophic forgetting - Interview with the Authors

#multitasklearning #biology #neuralnetworks This is an interview with the paper's authors: Abhiram Iyer, Karan Grewal, and Akash Velu! Paper Review Video: https://youtu.be/O_dJ31T01i8 Check out Zak's course on Graph Neural Networks (discount with this link): https://www.graphneuralnets.com/p/int... Catastrophic forgetting is a big problem in mutli-task and continual learning. Gradients of different objectives tend to conflict, and new tasks tend to override past knowledge. In biological neural networks, each neuron carries a complex network of dendrites that mitigate such forgetting by recognizing the context of an input signal. This paper introduces Active Dendrites, which carries over the principle of context-sensitive gating by dendrites into the deep learning world. Various experiments show the benefit in combatting catastrophic forgetting, while preserving sparsity and limited parameter counts. OUTLINE: 0:00 - Intro 0:55 - Sponsor: GNN Course 2:30 - How did the idea come to be? 7:05 - What roles do the different parts of the method play? 8:50 - What was missing in the paper review? 10:35 - Are biological concepts viable if we still have backprop? 11:50 - How many dendrites are necessary? 14:10 - Why is there a plateau in the sparsity plot? 20:50 - How does task difficulty play into the algorithm? 24:10 - Why are there different setups in the experiments? 30:00 - Is there a place for unsupervised pre-training? 32:50 - How can we apply the online prototyping to more difficult tasks? 37:00 - What did not work out during the project? 41:30 - How do you debug a project like this? 47:10 - How is this related to other architectures? 51:10 - What other things from neuroscience are to be included? 55:50 - Don't miss the awesome ending :) Paper: https://arxiv.org/abs/2201.00042 Blog: https://numenta.com/blog/2021/11/08/c... Link to the GNN course (with discount): https://www.graphneuralnets.com/p/int... Authors: Abhiram Iyer, Karan Grewal, Akash Velu, Lucas Oliveira Souza, Jeremy Forest, Subutai Ahmad Links: TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick YouTube: https://www.youtube.com/c/yannickilcher Twitter: https://twitter.com/ykilcher Discord: https://discord.gg/4H8xxDF BitChute: https://www.bitchute.com/channel/yann... LinkedIn: https://www.linkedin.com/in/ykilcher