Методы градиентной оптимизации

Уровень 0 · материалов: 3

В кластер входят документы, описывающие алгоритмы и теорию оптимизации на основе градиентов, и не входят документы об ансамблевых методах обучения.

Общие признаки: градиентные методы оптимизации, оптимизаторы нейронных сетей, математические основы сходимости, функции потерь

Группа выше: Методы обучения и оптимизации моделей

Материалы

• #7612 — Overview of Gradient Methods in Mathematical Optimization — Представитель этого кластера

  Обзор градиентных методов в задачах математической оптимизации — malkovsky

  Смысл: The main idea is to synthesize the core concepts and various modifications of gradient-based optimization methods, moving from simple theoretical foundations to advanced practical applications in machine learning and mathematical modeling.

  A comprehensive guide to gradient descent and its variations, covering accelerated, stochastic, subgradient, and proximal methods for mathematical optimization.

• #20782 — Neural Network Optimization Methods Analysis

  Методы оптимизации нейронных сетей — Siarshai

  Смысл: The main idea is to demystify the mathematical intuition behind popular neural network optimizers (SGD, Nesterov, Adagrad, RMSProp, Adadelta, Adam, and Adamax), explaining how they address the inherent challenges of gradient-based optimization to achieve faster and more stable convergence.

  A technical deep dive into neural network optimizers, explaining the mechanics of momentum and adaptive learning rates through mathematical formulas, visual experiments, and practical code.

• #24157 — Training Neural Networks Using 19th Century Physics Methods

  Я удалил backpropagation из нейросети. Она обучилась методами XIX века — inkedsymon

  Смысл: The main idea is that modern neural network optimization is essentially a digital implementation of classical mechanics from the 17th-19th centuries, where the loss landscape acts as a potential field and optimizers act as equations of motion for a physical particle.

  The author demonstrates that neural networks can be trained without backpropagation using classical physics analogies, revealing that modern optimizers like Adam and SGD are essentially simulations of a ball moving with inertia and friction through a potential field.