Hadi Daneshmand

Postdoctoral Researcher at the Foundation of Data Science Institute Contact MIT D32-588 Cambridge, USA Boston University, 665 Commonwealth ave. dhadi at mit dot edu

I do research on the foundations of machine learning at FODSI, as a postdoctoral researcher hosted by MIT and Boston University. Before, I was a SNSF postdoctoral researcher at Princeton University and INRIA Paris. I accomplished my PhD in computer science at ETH Zurich.

Research (Google Scholar)

• Resolving Computational Challenges in Deep Learning


• [19] Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion with Alexandru Meterez, Amir Joudaki, Francesco Orabona, Alexander Immer, Gunnar Ratsch. ICLR 24
• Theory of Large Language Models


• [18] Transformers learn to implement preconditioned gradient descent for in-context learning with Kwangjun Ahn, Xiang Cheng and Suvrit Sra. NeurIPs 23
• Bridging the gap between theory and practice in mean-field theory for neural networks (slides)


• [17] On the impact of activation and normalization in obtaining isometric embeddings at initialization with Amir Joudaki and Francis Bach. NeurIPs 23
• [16] On bridging the gap between mean field and finite width in deep random neural networks with batch normalization with Amir Joudaki and Francis Bach, ICML 23 (code, poster)
• [15] Efficient displacement convex optimization with particle gradient descent with Jason D. Lee, and Chi Jin. ICML 23 (code, poster)
• Data representations in deep random neural networks (slides)


• [14] Batch normalization orthogonalizes representations in deep random networks with Amir Joudaki, and Francis Bach. (spotlight) NeurIPS 21 (code, poster)
• [13] Batch normalization provably avoids ranks collapse for randomly initialised deep networks with Jonas Kohler, Francis Bach, Thomas Hofmann, Aurelien Lucchi. NeurIPS 20 (code, poster)
• Learning dynamics in deep neural networks
  
  
  • [12] Exponential convergence rates for batch normalization: The power of length-direction decoupling in non-convex optimization with Jonas Kohler, Aurelien Lucchi, Thomas Hofmann, Ming Zhou and Klaus Neymeyr. AISTATS 19
  • [11] Local saddle point optimization: A curvature exploitation approach with Leonard Adolphs, Aurelien Lucchi and Thomas Hofmann. AISTATS 19 (poster)
  • [10] Polynomial-time Sparse Measure Recovery: From Mean Field Theory to Algorithm Design with Francis Bach, arXiv 2022
• Modeling accelerated optimization methods by ordinary differential equations
  
  
  • [9] Rethinking the Variational Interpretation of Accelerated Optimization Methods with Peiyuan Zhang, and Antonio Orvieto. NeurIPS 21 (poster)
  • [8] Revisiting the role of euler numerical integration on acceleration and stability in convex optimization with Peiyuan Zhang, Antonio Orvieto, Thomas Hofmann and Roy S Smith. AISTATS 18 (poster)
• Stochastic optimization for machine learning


• [7] Escaping saddles with stochastic gradients with Jonas Kohler, Aurelien Lucchi and Thomas Hofmann. (long presentation) ICML 18(slides)
• [6] Adaptive newton method for empirical risk minimization to statistical accuracy with Aryan Mokhtari, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann and Alejandro Ribeiro. NeurIPS 16
• [5] Starting small-learning with adaptive sample sizes with Aurelien Lucchi and Thomas Hofmann. ICML 16
• Inference of graphs from diffusion processes


• [4] Estimating diffusion network structures: Recovery conditions, sample complexity & soft-thresholding algorithm with Manuel Gomez-Rodriguez, Le Song and Bernhard Schoelkopf. ICML 14 (Recomended to JMLR)
• [3] Estimating diffusion networks: Recovery conditions, sample complexity & soft-thresholding algorithm with Manuel Gomez-Rodriguez, Le Song and Bernhard Schoelkopf. JMLR 16
• [2] A Time-Aware Recommender System Based on Dependency Network of Items with Amin Javari, Seyed Ebrahim Abtahi and Mahdi Jalili. The Computer Journal 14
• [1] Inferring causal molecular networks: empirical assessment through a community-based effort with Steven M Hill, Laura M Heise et al. Nature Methods

• Early Postdoc Mobility grant from SNSF for Bridging the gap between local and global optimization in machine learning, 20. Outputs: [10], [15] and [16]
• Reviewer awards for my service to ICML 22, NeurIPS 20 and ICML 19.
• Best poster award in deep learning workshop of Max Planck ETH center for learning systems 2016.

• Beyond Theoretical Mean-field Neural Networks at ISL Colloquium, Stanford University, July 23 (slides)
• Data representation in deep random neural networks at ML tea talks, MIT, March 23 (slides)
• The power of depth in random neural networks at Princeton University, April 22
• Representations in Random Deep Neural Networks at Winter Seminar Series, Sharif University 22(slides)
• Batch normalization orthogonalizes representations in deep random neural networks, spotlight at NeurIPs 21 (slides)
• Representations in Random Deep Neural Networks at INRIA 21 (slides)
• Escaping Saddles with Stochastic Gradients at ICML 18 (slides)

• Area chair for NeurIPS 23
• An organizer of TILOS & OPTML++ seminars at MIT
• Conference reviewer for ICML, NeurIPs, ICLR and AISTATS
• Journal reviewer for Data Mining and Knowledge Discovery, Neurocomputing, TPAMI, and TSIPN

Mentorship

• Amir Joudaki, Ph.D. student at ETH Zurich (since 20)
• Jonas Kohler, former Ph.D. student at ETH (18-20), joined Meta
• Antonio Orvieto, Ph.D. student at ETH Zurich (20-21)
• Kwangjun Ahn, Ph.D. student at MIT (since 23)
• Ashkan Soleymani, Ph.D. student at MIT (since 23)
• Peiyuan Zhang, Master's student at ETH (19-21), joined Yale University for Ph.D.
• Leonard Adolphs, Master's student at ETH (18-19), joined ETH Zurich for Ph.D.
• Alexandru Meterez, Master's student at ETH (since 22)
• Flowers Alec Massimo, Master's student at ETH (since 23)
• Alexandre Bense, Master's student at ETH (22)
• Alireza Amani, Intern at ETH (18), joined London Business School for Ph.D.