I've never really been able to get my head around things like when and how a certificate also contains a private key (and when that private key is "exportable"), what all the "key usages" are and where they get their strange identifiers from (and how those things are actually enforced and used), etc.
EDIT: I should have kept reading, the linked Wikipedia articles on X.509, ASN.1, DER and PEM in the next section appear to be about what I'm looking for. I suppose I've just always been confused by all the formality and the number of specifications about different things.
There is a file format called PKCS #12 which lets you bundle things together. It is very common, particularly on Windows systems, to see this format used with files (often names ending .PFX) to bundle together a _certificate_ and a _private key_. This is presumably "convenient" in some sense, but since the security requirements for these two things are utterly different (a certificate is a public document that can be shown to anyone, a private key must not be seen by anybody else) it's a hazard in practice.
It's especially problematic that the resulting bundle (which you mustn't show to anybody) is often labelled a "certificate" when by far the more important thing inside it is a private key...
The same bundling idea can be done in PEM (the big blobs of Base 64 encoded stuff more often seen in Unix systems) by just concatenating a private key PEM to a certificate PEM. But at least if you look at it in a text editor you can see what somebody did, whereas in PFX it's a bit opaque without specialist tools.
Bundling these two things together most often happens in the context where somebody _else_ makes your key (and issues you a certificate). This is bad practice, not only is it patronising ("Poor dear don't understand crypto, we'll just give them the keys ready made") but it means the end user is vulnerable to copies kept nefariously or by accident. Avoid.
Certificate Authorities are prohibited from doing this for certificates in the Web PKI ("SSL certificates") but their resellers often still offer it as a "service". Say no. In fact try "Hell no".