392219 Applied Optimization (V) (WiSe 2026/2027)

The course teaches optimization heuristics and algorithms, from their mathematical foundations to their application. The topics are:

• Foundational concepts: Definition of optimization problems and optimality, modeling, neighborhoods
• Gradient-Free optimization methods: Nelder-Mead, Tabu Search, Simulated Annealing, Covariance Matrix Adaptation Evolutionary Strategy (CMA-ES), Bayesian Optimization, Ant Colony Optimization (APO)
• Continuous optimization theory: Taylor expansions, Gradient and Hessian, analytical optimization, convexity
• Gradient-Based optimization methods: Gradient Descent, Stochastic gradient descent, Adam, Automatic Differentiation, Steepest Descent, Conjugate Gradient, Newton's method, L-BFGS, Trust region methods
• Constrained optimization theory: Lagrange dual, weak and strong duality, Slater's condition, KKT conditions, Wolfe Dual
• Constrained optimization methods: Analytics solutions, barrier method (interior point), penalty method, projection method
• Linear and quadratic programming: Linear programs, Quadratic programs, Sequential quadratic programming, mixed integer linear programs, relaxation, branch & cut
• Probabilistic optimization: negative log likelihood, prior and posterior, logistic regression, expectation maximization, max-product algorithm
• Alternative Optimization Frameworks: Multi-Criteria Optimization and Pareto Fronts, robust optimization

The core skills taught in the course are:

• modeling: translating a natural language description of an optimization problem into a mathematical form
• mathematical reasoning: manipulating the formalized version of an optimization problem using mathematical tools, including solving problems with pure math; understanding the mathematical theory behind optimization methods and performing simple proofs
• computational solving: correctly selecting the right optimization algorithm for a certain problem and applying it correctly in Python code
• visualization: accurately visualizing problems, models, and solutions to support formalization, mathematical reasoning and interpretation
• interpretation & critical thinking: critically reflecting and discussing optimization applications and solutions, correctly interpreting models and solutions, and debugging formalizations and algorithm applications accordingly

The skills are taught in the lecture first, then deepenend in exercises (to be done in small groups), and finally checked in a written exam at the end of the term.

Requirements for participation, required level

Linear Algebra and Analysis at university level are strictly required. Some prior experience with mathematical modeling and analytical optimization is expected. Probability Theory is helpful.

Basic Python skills are expected for later exercises (but can be refreshed at the start of the course).

Bibliography

Only the script for the lecture is required reading. Complementary information can be found in the following books:

• Boyd, Stephen and Lieven Vandenberghe (2004). Convex Optimization. Cambridge, UK: Cambridge University Press. http://web.stanford.edu/~boyd/cvxbook/.
• Hillier, Frederick S. and Gerald J. Liebermann (2010). Introduction to Operations Research. 9th ed. New York City, NY, USA: McGraw-Hill
• Nocedal, Jorge and Stephen J. Wright (1999). Numerical Optimization. Springer Series in Operations Research and Financial Engineering. New York: Springer. doi: 10.1007/b98874.
• Petersen, Kaare Brandt and Pedersen, Michael Syskind (2012). The Matrix Cookbook. https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf
• Simon, Dan (2013). Evolutionary Optimization Algorithms. Hoboken, NJ, USA: Wiley.