The two biggest dangers are panic and shock when you first enter the water, hypothermia after leaving the water, and even sudden death as the body re-warms after leaving it. For that last, I read (but can't find) a story of several survivors of a shipwreck who died after drinking a hot beverage which warmed their bodies too quickly -- all the survivors died, one after the other, on the rescue ship.
My first summer up here it was a beautiful sunny day and my friends were telling me about how people die in the lake every year. Being a very strong swimmer and from Australia, I didn't believe them.
They bet me $1000 I couldn't stay in the water 5 minutes, which I thought I would have a good go at. I've swum in water below 40F before. I honestly didn't last 30 seconds before I was scared I would not be able to get out again, because I could not feel my extremities at all. It was surreal.
I know it sounds crazy, though it's worth noting more than one professional canoer / kayaker has died in the Yukon from capsizing their craft only a few hundred meters from shore.
We had a guy die not 5 minutes from town last summer, and some local Alaskans did the same thing just over the border.
I was trying to ford a river once, to retrieve a canoe I'd put into a logjam. Had two friends with me, holding one end of a long rope from shore while I had the other end looped under my shoulders.
Got maybe halfway across before the current was just to strong (and I was too exhausted) to keep my footing anymore - it's just unbelievable how quickly being submerged in cold water and fighting against the current like that sucks the energy out of you.
It was all I could do to keep my head above water and kick myself back to shore, I doubt I'd have been able to drag myself out of the river if I'd been by myself.
And that river (north fork of the Chena) wasn't even glacier fed, like a lot of rivers in Alaska are...
That said: if you are acclimated to the shock (and possibly have a genetic propensity to survive colder water), it's definitely survivable.
Capsizing a kayak is something you need to know how to deal with to execute properly anyhow, and yes, cold water makes appropriate response and motor control much more challenging.
I've done extended swims (30+ minutes) in ~54F water, and shorter dips in colder (40s, high 30s). That last was more a dash, plunge, and run back to the hot tub, truth be told. Even in the 50s, the cold shock and numbing are quite noticeable, though once you get your head in the right space, it's very tolerable.
The story you read was likely from Bill Bryson's book "A Walk in the Woods", which talks about 12 Danish fishermen, but that story doesn't seem to be correct in its details.
In fact, many hypothermia victims die each year in the process of being rescued. In "rewarming shock," the constricted capillaries reopen almost all at once, causing a sudden drop in blood pressure. The slightest movement can send a victim's heart muscle into wild spasms of ventricular fibrillation. In 1980, 16 shipwrecked Danish fishermen were hauled to safety after an hour and a half in the frigid North Sea. They then walked across the deck of the rescue ship, stepped below for a hot drink, and dropped dead, all 16 of them.
Reddit has some discussion of this, largely dismisses the 16 Danish fishermen story as a myth (though points to a couple of published sources as well).
http://www.hse.gov.uk/research/othpdf/500-599/oth519.pdf (UK compendium of sea immersion rescue issues, no mention of Danish sailors).
Basic Essentials: Hypothermia, by William W. Forgey, 1999. Does include a mention, but includes no citations, references, footnotes, or endnotes.
Having read several of Bryson's entertaining works, I know that he is not a rigorous researcher of the details.
The link I gave was to "Hypothermia and Cold Stress" by Evan L. Lloyd, 1986. It shows a citation to Lee and Lee 1971. I am unable to track down that reference based just on free internet searches. According to Lloyd's recounting, the rescuees did drink some alcohol. I can see how upon retelling that might be turned into a "hot drink."
I am quite willing to say that the previous poster heard about the Bryson story and the false details from that. However, I would rather state that the details of the story - that being of Danish fishermen - were wrong rather than the point of the story, which is the death of people after they have been rescued. In any case, the parent poster said nothing about Danish fishermen.
Even if "Lee and Lee 1971" citation I pointed to is too vague of a confirmation for you, the HSE report gives many examples of "post-rescue collapse and death" from both Allied and Axis sources, including from ship-wreck survivors (Critchley 1943).
I think it's unsafe describe the situation in a way that could be described as implying an urban legend, which is why I've specifically said that the details are wrong, and pointed to places where the details are correct. (To be sure, you are not doing that. I write this to explain why I wrote it as I did.)