I visited Reykjavik during the winter solstice, and stayed there for a bit over a week. I had the same misapprehension as you. At noon, the sun was just barely over the horizon and it felt like days never got much brighter than twilight. It was very strange.