
Tales of Statisticians: George B. Dantzig - KhalilK
https://www.umass.edu/wsp/resources/tales/dantzig.html
======
idunning
Simplex method was a game changer.

Its a funny algorithm: worst-case time complexity is exponential in the input
size, and is pretty easy to demonstrate (see "Klee-Minty cube" on Wikipedia).
It has a couple of "rules" you can change out that will fix the exponential
problem for some cases, only to introduce new problematic cases elsewhere.
However the reality is that it demonstrates polynomial-like performance on
almost all problems of interest, and that is why it is so widely used.

Later, interior point methods arose that can be faster sometimes (and are
polynomial time complexity), but they didn't kill the simplex method. This is
partly due to one key property of the simplex method: at optimality, you can
change the linear program in many different small ways and start the algorithm
again from where you left off (sometimes you need the dual simplex method, a
sibling method). You'll return to feasible optimality in usually only a few
iterations. This is what powers the branch-and-bound approach to integer
programming, which is the really useful application of LP these days. Interior
point methods don't really have good warm starts to this day, certainly not
good enough for branch-and-bound.

I've met several professors who kinda don't like the simplex method because
(they say) it is not a beautiful algorithm from a theory perspective, but I
think its wonderful.

Oh, and PSA: very difficult to implement correctly! The textbook algorithm
will fail terribly on real problems due to floating point issues - please use
an existing implementation if you need to solve LPs!

~~~
graycat
> I've met several professors who kinda don't like the simplex method because
> (they say) it is not a beautiful algorithm from a theory perspective, but I
> think its wonderful.

Ah, while we try to keep taste out of math, people differ in their _taste_ of
what is really good in applied math! I'll describe some aspects of the simplex
algorithm I believe are really good:

Algebraically, the simplex algorithm is just elementary row operations on a
system of linear equations but the operations selected to improve the value of
the _objective function_ , that is, the quantity to be _optimized_ , that is,
made as large (or small) as possible, e.g., cost.

Geometrically the simplex algorithm works on a finite intersection of closed
half spaces which, then, is a convex set with flat sides and sharp extreme
points (not a convex combination of two other points of the convex set) with
the extreme points joined by edges. Well, the simplex algorithm works by
moving from one extreme point along an edge to an _adjacent_ extreme point,
that is, an extreme point just one edge away.

Or take a solid piece of cheese and whack at it to create a lot of flat sides.
Put the cheese on a table and ask an ant to move along edges from one extreme
point to another until get to the highest point on the cheese, which of course
will be an extreme point.

When the simplex algorithm _cycles,_ that is, goes through a sequence of
feasible solutions and returns to an earlier one, geometrically the algorithm
is staying at some one extreme point. R, Bland has a nice anti-cycling rule
which, then, shows that the simplex algorithm is finite, that is, will end, as
we wish, in finitely many iterations.

For min cost flows on networks, also a linear programming problem, the simplex
algorithms is super-cute: One of the extreme points and a _basic feasible
solution_ correspond to a spanning tree of arcs in the network. Then, a
simplex iteration adds an arc to the tree and thus creates a circuit. Then run
flow around the circuit in the direction that improves the objective function
value until the flow on some arc goes to zero, remove that arc from the basis,
and get a new spanning tree. Darned cute. W. Cunningham has a cute
modification that works with what he calls _strongly feasible bases (trees)_
that provides an anti-cycling rule.

Cute result: For such a network, if the arc capacities are all integer and if
have an initial feasible solution that is all integer, then the simplex
algorithm will find only integer solutions and terminate with an optimal
integer solution (if an optimal solution exists). So, get an important case of
integer linear programming for free. There's money to be made here. We also
just showed that there is an optimal solution that is integer.

Yes, a special case of least cost flows on networks is the _transportation
problem_ (how to ship widgets at least cost from several factories to several
warehouses honoring supply and demand constraints) that won the Nobel prize in
economics for Kantorovich.

Yes, for shortest paths in a network, say, from San Francisco to New York
City, take the roads, call them a network, give each arc a maximum flow
(capacity) of 1 and a cost per unit of flow the length of the arc, and ask for
the min cost flow from San Francisco to New York. Start with any initial
integer solution and then apply the network simplex algorithm. Okay? Sorry
Dantzig!

Knowing that the simplex algorithm is finite yields a nice list of other
results such as various theorems of the alternative important in, e.g., the
Kuhn-Tucker conditions of nonlinear programming and the saddle point theorem
of two person game theory.

And linear programming and the simplex algorithm are also important in some
cases of nonlinear programming.

More generally linear programming and some of the properties of the simplex
algorithm are nice tools for theorem proving elsewhere in pure and applied
math. E.g., in Fleming, _Functions of Several Variables,_ basically a text in
_advanced calculus,_ there is an exercise: For the real numbers R and a
positive integer n, a linear function on R^n bounded above on a finite
intersection of closed half spaces achieves its least upper bound. This result
sounds like it would yield to, say, some compactness or completeness argument,
but (AFAIK) it doesn't. Instead, deceptively, the result is combinatorial --
e.g., holds over the rationals! For the combinatorial part, use the simplex
algorithm with Bland's rule. Bland discovered his rule working in matroid
theory.

Work of K. Borgward showed that with meager probabilistic assumptions the
simplex algorithm is polynomial or some such (I don't have the details in
front of me). So, this result helps explain the good performance in practice,
e.g., commonly number of iterations just 3 times the number of rows in the
system of linear equations. That is, darned fast.

As I recall, the simplex algorithm was named one of the best innovations in
engineering in the 20th century.

------
paperwork
"The Lady Tasting Tea"[1] is a fantastic book about statisticians. Someone
recommended that book to me as an intro to stats, which it most certainly is
not. However, if you have had even a single course in statistics, you will
recognize many of the names and learn how they influenced the modern era.

Frankly, I wish the author had devoted a couple of pages at the end of each
chapter to delve deeper into the technical concepts. Nonetheless, the book
makes an interesting read, even for non-statisticians.

[1][http://www.amazon.com/The-Lady-Tasting-Tea-
Revolutionized/dp...](http://www.amazon.com/The-Lady-Tasting-Tea-
Revolutionized/dp/0805071342)

------
jccalhoun
I want to read Glenn Danzig's tales of statisticians.

