> All models were trained on 70% of the data and tested on the remaining 30% of the data. Note that each month record for each participant was considered an independent data point.
I'm almost certain that this is a mining leak.
Data from one patient will end up both in the training and test set and result in fantastic accuracy. Of course it will be from different months. The correct way to do this is to cross-validate/split across the population. It seems unlikely, from this description, that the authors have done so.
It's a very common error and it should be easy to catch but....I've even seen a study that treated individual slices of an MRI as independent, which is laughably wrong.
I think part of the problem is that the "analysts" are increasingly uninvolved in the data collection, and just treat it as a tuple of (X, y). If you thought about what they mean, even for a second, ("Oh, Mr. Smith is always an awful driver"), the problem is obvious.
One way to tell if a Machine Learning model is any good is to see how it does on unseen/new patients.
Of course, we don't wait to try it on real patients, so typically you'd partition the data you have already into (a) what you show to the machine learner (training data), and (b) what you hide from the learner (test data). The latter is only used to evaluate, i.e. you get the answer from the ML model and compare it to the real answer you have already. If information about the test data some how makes it to the training data, its referred to as a mining leak [1].
In this paper, they treat each month of a patient as an independent observation. However, GPS driver behaviour will be very similar from one month to the next for the same person. Genetic information is exactly the same. So for every month that the model is tested (test data), the learner has already seen very similar data in the training set - for some of the other months (for the same person) that happen to be in the training set. The split is typically done randomly. So it will do well.
The test results are therefore optimistic and do not support the conclusions.
Suppose that you have 3 data points, on June 14, 15, and 16, that due to personal driving quirks all appear to belong to the same person. If the 14th and 16th are in your dataset, and both correspond to Alzheimer-free Bob, that may be a strong hint that the data from the 15th is also Alzheimer-free.
But this doesn't help you in the real world where you won't necessarily have near neighbors corresponding to the same person, with a known diagnosis.
Specifically, those with preclinical Alzheimer's tended to drive more slowly, make abrupt changes, travel less at night, and logged fewer miles overall, for example. They also visited a smaller variety of destinations when driving, sticking to slightly more confined routes.
Isn’t this pretty much everyone as they get older? Who hasn’t seem these traits in older relatives?
Everyone I know who gets older tends to drive less at night because as you get old your eyesight gets worse —- particularly if you’ve had cataract surgery, which are people who are in the study / risk group.
Unless I missed a more nuanced point this article reads like “old age is a good predictor of Alzheimer’s.”
Personally, my processing speed and reaction time fell off a cliff in my early 40s. I used to be a more aggressive driver but had to back off because I can't handle the flood of information now. It's easy to overlook things and miss objects and cars so I have to check twice.
Interesting, I’m certainly older than that but i feel i can still handle trafic pretty well but i drive slower because i stopped giving a crap if i get late somewhere or i get out earlier and i’m less aggressive because I realize it’s pointless. If that’s getting old i’m happy with it.
Age was another input into the model. However, it's no surprise that the signs of aging correlate strongly with Alzheimer's. While one 70 year old in very good mental/physical health may be very active including driving, another may seem and "act" a lot older.
In particular, Driving from point A to point B depends on a certain kind of memory that degrades quickly in Alzheimer's. Forgetting names is one thing but forgetting whole places is another.
* Making abrupt changes: perhaps they are suddenly recalling something they need to do like turn onto a certain street? Not quite sure.
* Driving at night removes a lot of visual cues putting an even larger emphasis on memory.
* Longer travels & having more destinations both depend on memory so that one makes sense.
I think the quote ties in with what else we know about Alzheimers: it tends to develop in those with smaller social circles, in those who experience less novelty and less healthy lifestyles. If the person is out in society learning a new language, meeting new people, learning a new instrument, exercising their minds' regenerative capacities, that's a great "counter" to aging.
If they're sitting at home, not driving anywhere, watching cable news all day, that's a recipe for rutted-grooves of thought and deterioration of the mind.
I tend to drive relatively little, visit fairly few distinct places, and generally take the same route each time I visit them. I'm also not a huge fan of night driving. I don't think I drive particularly slowly or make too many abrupt changes (I've never had an accident involving another car, am never the slowest one on the freeway, and have never had a complaint from any of my passengers).
The interesting part for me here is that these behaviors do, in fact, reflect a neurological issue I have with visual/spatial processing. In a nutshell, I get lost easily, because it takes me much more time to associate visual landmarks with a travel route than the typical human. I suspect my dog has a far, far better sense of direction than do I. :P
I'm not even close to old enough to have preclinical signs of Alzheimer's, yet, but I shall have to remember to remind my doctors of my visual-spatial issues in a few decades when I am.
Got a bit of a victim-blamey vibe here. Intuitively it makes sense that if you're more active and social, you don't get Alzheimer's - but it's also pretty tough to separate correlation and causation. Does being shut-in cause Alzheimer's, or do people end up being shut-ins because of Alzheimer's? Intuition is often wrong.
I know this is merely an anecdote, but this fits what I saw in my mother. I was learning to drive, visited the parents, we went on a practice drive… and she found speed sufficiently frightening that she asked me to limit myself to 40 MPH, while we were on a 70 MPH dual carriageway A-road. My driving lessons came with a booklet saying doing under 50 on such a road (in normal conditions) would be considered dangerous.
> My driving lessons came with a booklet saying doing under 50 on such a road (in normal conditions) would be considered dangerous.
Yes! A slow driver (under 50) can cause very high closing speeds for drivers who aren't paying attention. A likely accident scenario could be a driver who tailgates a white van that is itself closing fast on the slow vehicle, blocking the view of the tailgater (all are in the inside lane). If the van makes a late overtaking manoeuvre (to avoid the slow vehicle) rather than smoothly braking, the tailgater has very little time to react, and depending on traffic density, might not themselves be able to change lanes.
I saw this exact situation happen a few days ago. I was in lane 2 about 100m further back down the road. The van changed lanes at the last possible moment. The tailgater slammed on his brakes and came very close to shunting the slow vehicle. Luckily there was no-one behind him so a braking wave didn't propagate back down the lane.
If the slow driver is fully capable of driving a normal speed and makes the choice to drive well below the speed limit, they are making a choice that directly increases the probability of an accident (regardless of if they are involved in the actual accident or not).
If the slow car has an issue that makes it so that they cannot go any faster they should be putting their hazards on and slowing down, in which case it would be the drivers who are not paying attention's fault.
Really in both cases the drivers not paying attention share some of the blame, but in the first case the slow driver is also being irresponsible.
>What if the "slow driver" was a normal driver who just got a puncture and was slowing down?
Then they'd have a good excuse.
Same with the shitbox running on 3cyl.
Same with the concrete truck that pretty much has to drive under the prevailing speed uphill because it's so slow.
To voluntarily create a hazard when you can "just not" by behaving like everyone else (which is not exactly an activity that requires a lot of cognitive thought) is simply asinine.
Just to be clear, I didn't assign blame when describing my little scenario. The slow driver is creating an avoidable problem that increases the impact of the poor driving behaviours exhibited by the other drivers (going too fast, too close). All three are doing at least one thing wrong.
The difference is that the impaired driver is a preventable hazard; the puncture is sudden and unexpected. The actual issue is the lack of accessible public transportation that perpetuates the need for a person with some impairment to continue driving a vehicle.
I wish. Unfortunately the moron causing the hazard almost never gets caught up in the resulting accident like you describe.
What happens is you wind up with a merge (and all the shenanigans that causes) behind the blockage and given a long enough duration you eventually run into the case someone doesn't check their mirrors/blind spot and hits someone who's not on their A-game because there's no exit there and they're not expecting the person to merge there.
> Specifically, those with preclinical Alzheimer's tended to drive more slowly, make abrupt changes, travel less at night, and logged fewer miles overall, for example. They also visited a smaller variety of destinations when driving, sticking to slightly more confined routes.
Aside from the abrupt changes this could describe my driving habits. I hate driving, driving at night most of all.
I was thinking that these changes weren't described in specific enough detail too. I mean, these could also be used to guess age in general, if I'm reading it right.
Presumably there is some presumption that everyone in the study has more or less the same driving needs. Possibly for a small area in MO, that might be true, or at least close enough. I mean, I don't drive much either, but I live in a place where I don't have to. I don't need the car to get to work or do most of my ordinary errands. Now, if they want to track my bike, they might get more data.
My take from the article is that the deviation happens over time from your preexisting baseline. So it's not your current habits, its that tracking your habits over time will show a difference.
> Specifically, those with preclinical Alzheimer's tended to drive more slowly, make abrupt changes, travel less at night, and logged fewer miles overall, for example. They also visited a smaller variety of destinations when driving, sticking to slightly more confined routes.
I wonder what the direction of the causality is. Maybe there is something else that triggers the slow driving. Visiting the same destinations regularly instead of varying ones might also just point to a very routine lifestyle. Maybe that contributes to Alzheimer’s a lot? It feels a stretch to use these attributes as casuals for attribution of Alzheimer’s.
There have been some interesting findings in aging studies (e.g. the 90+ study [0]) that suggest some people who have all the physiological signs of Alzheimers, but show no symptoms of dementia.
I suspect the arrow goes:
X -> brain degradation
brain degradation -> cognitive decline
cognitive decline -> driving bad
brain degradation -> post-mortem Alzheimers diagnosis
The loose link there is the path from brain degradation to cognitive decline. The 90+ findings (plus other findings on TBI and plasticity) suggest that by maintaining high plasticity through brain assaults (old age, trauma), it seems the brain can "route around damage" and remain comparatively functional.
This plasticity idea is really neat.
Anecdotal and I don’t know the correlation holds up, but my one grandmother was very set in her habits and particularly mindset and died early of Alzheimer’s and Parkinson’s. My other grandmother is flexible of mind and is still kicking it at 90+.
I’m wondering which way the causality goes. Whether there is something physical which makes people more open minded and wandering whereas others are more closed which eventually leads to more brain diseases. So in the end the open mindedness / flexibility is just another indirect indicator.
Without the measured changes being caused by Alzheimer's, it is unlikely they will have any specificity. On the other hand, the results reported here give at least circumstantial evidence for this causality.
Although this potential scenario is a long way off for the insurance market, it's something that might worry existing owners of black boxes who have already experienced issues with their devices' accuracy in the past.
Autopilot-style driver aids can mess with these data points to some extent. Still very interesting that diversity in destination is considered an indicator.
Insurance companies will be all over personal data (search, location, and profiles) so that they can statistically discriminate against their clients beyond what they have been thus far able. HIPAA is fairly toothless in many ways, and doesn't even cover cases like this. It's time to start implementing criminal penalties for illegally exfiltrating private health data before it is too late.
Isn't statistical discrimination the basis of fair cohort definition in insurance underwriting?
What do you think insurance, as a financial product, is? An entity looking to hedge against financial uncertainty relies on being placed in a cohort that represents equivalent risk profile to their own.
I think insurance is nothing but a serendipitous, temporary alignment of finance, levels of information availability/tracking, and the fundamental right of all humans to life and health.
Right now, despite their best efforts, insurance companies are still fairly limited in their ability to discriminate and exclude certain swathes of people from access to affordable healthcare. Once they manage it, they become useless to us as a society.
Ideally, we as a society want all of our members to be able to see a doctor when they are sick or injured.
Ideally, insurance companies want to exclusively insure people to whom they will never need to pay out. They optimize their profit margin to be essentially 100%. Which then makes them a leech, and they're probably a leech well below 100% as well.
(To be fair, since the system is completely inhuman, individuals will also play the same game and try to leave the risk pool if they are sufficiently low-risk.)
So, there will come a time when we have to either give up any facade that we value human health intrinsically, or we must stop using private insurance as the primary method of ensuring it (no pun intended).
In the UK it's commonplace for insurers to offer "black box" policies, where a small device including GPS and accelerometers with an mobile internet connection must be fitted to the vehicle in exchange for a discount.
> All models were trained on 70% of the data and tested on the remaining 30% of the data. Note that each month record for each participant was considered an independent data point.
I'm almost certain that this is a mining leak.
Data from one patient will end up both in the training and test set and result in fantastic accuracy. Of course it will be from different months. The correct way to do this is to cross-validate/split across the population. It seems unlikely, from this description, that the authors have done so.
[1] https://alzres.biomedcentral.com/articles/10.1186/s13195-021...