For the record, I was being facetious on purpose to show how ridiculous this is.
But more to your point, the adapter would belong with the phone not with the headphones. There are potentially N number of headphones, and one phone for which the adapter will be used. So the adapter being carried with the phone instead of N adapters for N headsets makes more sense.
Or a simple analog jack would work too.