Wow. How many photons that bounce of the planet, travel a few lightyears and get caught in this 40m mirror are we talking about here? 1 million per night? Are we close to the limits of detection or what else is possible?
A million photons would be overkill for a solid detection. A standard CCD saturates at about 65,000 counts (or about that many photons, assuming the efficiency is high). A source that is detected well enough to do interesting measurements usually has a ~1000 - 20,000 photons per pixel. Spread out over ~15 pixels, a good detection could require as few as ~15,000 photons.
In this particular application it would be more difficult since you not only have to detect the source on its own, you have to block out the glare from the parent star. Usually the main problem in this sort of work is making sure that the source you are detecting is actually a real thing and is not just an artifact from the way you block out the light from the parent star.
The proposed adaptive optics imager for ELT has a (1 sigma/1hour) limit in the visible of a 32mag object - with better than HST resolution.
A 0mag star gives you about 1000photons/s/cm^2/Angstrom so the 40m telescope in visible light would capture about 2E13 photons/s for the brightest star in the sky.
But 32mag is 2.51^32 = 6E12 times fainter so only a few 1000 photons in an hour long exposure.
