The book (and film) is specifically talking about absent fathers. The next line is "Our fathers were our models for God. If our fathers bailed, what does that tell you about God?"
It's pretty much the theme of the film: Masculinity as a desperate absence and conversely a desperate danger, in all it's facets from the comedic ("In death a member of Project Mayhem has a name") to the bathetic ("remaining men together") to the deeply sinister and violent. In a time when it is culturally deracinated and absent from so many men's childhoods.
My father was raised by his dad and mom. I was raised by my dad and mom. My grandfather's were raised by their fathers and mothers. So that's three generations of men not raised only by their mothers.
Civilization is built upon the inclusion of men in childrearing, so ultimately all civilizations include men raised by both men and women (at least the successful ones).
