1) The libraries are guaranteed to be decoupled.
2) The libraries are better maintained individually.
3) It's much easier (sometimes painless) to replace them if they don't suit your needs.
4) ... probably many more.
The hardest part of bootstrapping a new application is the boilerplate. Let someone else take care of managing session state, generating SQL queries, URL routing, DI containers, buffered logging, templating, etc. If you haven't found something that suits your needs, sure, write it yourself.
I wouldn't say what you're doing is stupid. Do what makes you successful and happy. Just don't be stubborn ;)
One of the directions LedgerSMB (http://www.ledgersmb.org) is going is from a framework which was originally written by one guy to something which outsources as much as possible to CPAN modules. This poses some issues, but in general it works pretty well. We probably have as much framework code as we used to but the code does a lot more. We decided not to go with other frameworks because of the dependency issues that sometimes arise.
Also I will personally vouch for #3. We started off using Std::Config for processing ini files. But this isn't well supported in distros or well maintained so we switched to Config::IniFiles and it was a pretty painless effort.
the bigger thing though is you don't have to do it all yourself. :-)