Hi, one of the authors here. First, though I worked on the paper, I am not employed by CGD, so these comments are my own.
The main reason for using the non-headless Firefox WebDriver was that we wanted the script to access the site just like a human user. This made it easy to explain to non-technical people exactly how we had gotten the data. We didn't want to do anything that could be seen as circumventing the interface that the World Bank had created for that purpose.
Up to a point, performance was not a concern. In fact, as slow as Selenium is, we still artificially limited the speed of the script by waiting three seconds between each set of queries. However, when it came to selecting options, it could take Selenium tens of seconds, so that was done with js.
The main reason for using the non-headless Firefox WebDriver was that we wanted the script to access the site just like a human user. This made it easy to explain to non-technical people exactly how we had gotten the data. We didn't want to do anything that could be seen as circumventing the interface that the World Bank had created for that purpose.
Up to a point, performance was not a concern. In fact, as slow as Selenium is, we still artificially limited the speed of the script by waiting three seconds between each set of queries. However, when it came to selecting options, it could take Selenium tens of seconds, so that was done with js.