That's likely a false impression, and has to do with things like number of database calls, design, and other non-language-defining features. Don't let perceived speed be the deciding factor in your language choice: If speed was that important, you'd be writing in C.
When I first started using Django, I wasn't sure about it either, for a few different reasons - but performance wasn't one of them. Long story short, I'm glad I invested the time in taking it up...
Django was actually my first use of Python - so you can imagine how pleased I was to replace a custom PHP app with something I just learned - and in record time. (if you're interested, the web app was: http://www.snapact.com/)
I have noticed no issues with performance as compared to PHP.
Perhaps it is possible that Django's highly flexible nature is giving you enough rope to hang yourself here. Performance is not an issue for me, though (as always) I have to be careful not to do stupid things to the database.
