Upcoming Events
Past Events
20181007 Tutorial on "Interpretable Deep Learning: Towards Understanding & Explaining Deep Neural Networks" at
ICIP 2018 in Athens, Greece
20180916 Tutorial on "Interpretable Machine Learning" at
MICCAI 2018 in Granada, Spain
20180618 Tutorial on "Interpreting and Explaining Deep Models in Computer Vision" at
CVPR 2018 in Salt Lake City, USA
20170305 Tutorial at
ICASSP 2017 on "Methods for Interpreting and Understanding Deep Neural Networks" in New Orleans, USA
This webpage aims to regroup publications and software produced as part of a joint project at Fraunhofer HHI, TU Berlin and SUTD Singapore on developing new method to understand nonlinear predictions of stateoftheart machine learning models.
Machine learning models, in particular deep neural networks (DNNs), are characterized by very high predictive power, but in many case, are not easily interpretable by a human. Interpreting a nonlinear classifier is important to gain trust into the prediction, and to identify potential data selection biases or artefacts.
The project studies in particular techniques to decompose the prediction in terms of contributions of individual input variables such that the produced decomposition (i.e. explanation) can be visualized in the same way as the input data.
Interactive LRP Demos
Draw a handwritten digit and see the heatmap being formed in realtime. Create your own heatmap for natural images or text. These demos are based on the Layerwise Relevance Propagation (LRP) technique by Bach et al. (2015).
MNIST: A simple LRP demo based on a neural network that predicts handwritten digits and was trained using the MNIST data set.
Caffe: A more complex LRP demo based on a neural network implemented using Caffe. The neural network predicts the contents of the picture.
Text: A LRP demo that explains classification on natural language. The neural network predicts the type of document.
How and Why LRP ?
Layerwise Relevance Propagation (LRP) is a method that identifies important pixels by running a backward pass in the neural network. The backward pass is a conservative relevance redistribution procedure, where neurons that contribute the most to the higherlayer receive most relevance from it. The LRP procedure is shown graphically in the figure below.
The method can be easily implemented in most programming languages and integrated to existing neural network frameworks. When applied to deep ReLU networks, LRP can be understood as a Deep Taylor Decomposition of the prediction.
LRP Software
Resources
CVPR Tutorial 2018
Tutorial website
Videos:
Download slides:
More Talks & Tutorials

2018 Int. Explainable AI Symposium Keynote(Website  Slides)
Explainable ML, Applications
 ICIP 2018 Tutorial (Website  Slides:
1Intro,
2Methods,
3Evaluation,
4Applications)
Explainable ML, Applications
 MICCAI 2018 Tutorial (Website  Slides)
Explainable ML, Medical Applications

Talk at Int. Workshop ML & AI 2018 (Slides)
Deep Taylor Decomposition, Validating Explanations

WCCI 2018 Keynote (Slides)
Explainable ML, LRP, Applications

GCPR 2017 Tutorial (Slides)

ICASSP 2017 Tutorial (Slides 1Intro, 2Methods, 3Applications)
Publications
Tutorial / Overview Papers
 G Montavon, W Samek, KR Müller. Methods for Interpreting and Understanding Deep Neural Networks
Digital Signal Processing, 73:115, 2018 [preprint  bibtex]
 W Samek, T Wiegand, KR Müller. Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
ITU Journal: ICT Discoveries  Special Issue 1  The Impact of AI on Communication Networks and Services, 1(1):3948, 2018 [preprint, bibtex]
Methods Papers
 S Bach, A Binder, G Montavon, F Klauschen, KR Müller, W Samek. On Pixelwise Explanations for NonLinear Classifier Decisions by Layerwise Relevance Propagation
PLOS ONE, 10(7):e0130140, 2015 [preprint, bibtex]
 G Montavon, S Lapuschkin, A Binder, W Samek, KR Müller. Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
Pattern Recognition, 65:211–222, 2017 [preprint, bibtex]
 L Arras, G Montavon, KR Müller, W Samek. Explaining Recurrent Neural Network Predictions in Sentiment Analysis
EMNLP Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, 159168, 2017 [preprint, bibtex]
 A Binder, G Montavon, S Lapuschkin, KR Müller, W Samek. Layerwise Relevance Propagation for Neural Networks with Local Renormalization Layers
Artificial Neural Networks and Machine Learning – ICANN 2016, Part II, Lecture Notes in Computer Science, SpringerVerlag, 9887:6371, 2016 [preprint, bibtex]
 PJ Kindermans, KT Schütt, M Alber, KR Müller, D Erhan, B Kim, S Dähne. Learning how to explain neural networks: PatternNet and PatternAttribution
International Conference on Learning Representations (ICLR), 2018
 J Kauffmann, KR Müller, G Montavon. Towards Explaining Anomalies: A Deep Taylor Decomposition of OneClass Models
arXiv:1805.06230, 2018
Evaluation Explanations
Software Papers
 M Alber, S Lapuschkin, P Seegerer, M Hägele, KT Schütt, G Montavon, W Samek, KR Müller, S Dähne, PJ Kindermans iNNvestigate neural networks!.
arXiv:1808.04260, 2018
 S Lapuschkin, A Binder, G Montavon, KR Müller, W Samek The Layerwise Relevance Propagation Toolbox for Artificial Neural Networks
Journal of Machine Learning Research (JMLR), 17(114):1−5, 2016 [preprint, bibtex]
Application to Sciences
 I Sturm, S Bach, W Samek, KR Müller. Interpretable Deep Neural Networks for SingleTrial EEG Classification
Journal of Neuroscience Methods, 274:141–145, 2016 [preprint,
bibtex]
 A Binder, M Bockmayr, M Hägele, S Wienert, D Heim, K Hellweg, A Stenzinger, L Parlow, J Budczies, B Goeppert, D Treue, M Kotani, M Ishii, M Dietel, A Hocke, C Denkert, KR Müller, F Klauschen. Towards computational fluorescence microscopy: Machine learningbased integrated prediction of morphological and molecular tumor profiles
arXiv:1805.11178, 2018
 F Horst, S Lapuschkin, W Samek, KR Müller, WI Schöllhorn. What is Unique in Individual Gait Patterns?
arXiv:1808.04308, 2018
Application to Text
 L Arras, F Horn, G Montavon, KR Müller, W Samek. "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
PLOS ONE, 12(8):e0181142, 2017 [preprint, bibtex]
 L Arras, G Montavon, KR Müller, W Samek. Explaining Recurrent Neural Network Predictions in Sentiment Analysis
EMNLP Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, 159168, 2017 [preprint, bibtex]
 L Arras, F Horn, G Montavon, KR Müller, W Samek. Explaining Predictions of NonLinear Classifiers in NLP
ACL Workshop on Representation Learning for NLP, 17, 2016 [preprint, bibtex]
 F Horn, L Arras, G Montavon, KR Müller, W Samek. Exploring text datasets by visualizing relevant words
arXiv:1707.05261, 2017
Application to Images & Faces
 S Lapuschkin, A Binder, G Montavon, KR Müller, W Samek. Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 29122920, 2016 [preprint, bibtex]
 S Lapuschkin, A Binder, KR Müller, W Samek. Understanding and Comparing Deep Neural Networks for Age and Gender Classification
IEEE International Conference on Computer Vision Workshops (ICCVW), 16291638, 2017 [preprint, bibtex]
 C Seibold, W Samek, A Hilsmann, P Eisert. Accurate and Robust Neural Networks for Security Related Applications Exampled by Face Morphing Attacks
arXiv:1806.04265, 2018
 S Bach, A Binder, KR Müller, W Samek. Controlling Explanatory Heatmap Resolution and Semantics via Decomposition Depth
Proceedings of the IEEE International Conference on Image Processing (ICIP), 22712275, 2016 [preprint, bibtex]
 A Binder, S Bach, G Montavon, KR Müller, W Samek. Layerwise Relevance Propagation for Deep Neural Network Architectures
Proceedings of the 7th International Conference on Information Science and Applications (ICISA), 6679:913922, Springer Singapore, 2016 [preprint, bibtex]
 F Arbabzadah, G Montavon, KR Müller, W Samek. Identifying Individual Facial Expressions by Deconstructing a Neural Network
Pattern Recognition  38th German Conference, GCPR 2016, Lecture Notes in Computer Science, 9796:344354, 2016 [preprint, bibtex]
Application to Video
 C Anders, G Montavon, W Samek, KR Müller. Understanding PatchBased Learning by Explaining Predictions
arXiv:1806.06926, 2018
 V Srinivasan, S Lapuschkin, C Hellge, KR Müller, W Samek. Interpretable human action recognition in compressed domain
Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 16921696, 2017 [preprint, bibtex]
Application to Speech
Short Papers
 W Samek, G Montavon, A Binder, S Lapuschkin, and KR Müller. Interpreting the Predictions of Complex ML Models by Layerwise Relevance Propagation
NIPS Workshop on Interpretable ML for Complex Systems, 15, 2016 [preprint, bibtex]
 G Montavon, S Bach, A Binder, W Samek, KR Müller. Deep Taylor Decomposition of Neural Networks
ICML Workshop on Visualization for Deep Learning, 13, 2016 [preprint, bibtex]
 A Binder, W Samek, G Montavon, S Bach, KR Müller. Analyzing and Validating Neural Networks Predictions
ICML Workshop on Visualization for Deep Learning, 14, 2016 [preprint, bibtex]
Downloads
BVLC Model Zoo Contributions