The test checked literacy, numeracy, and IT skills. US did fine on literacy, so-so on numeracy, and last on IT skills. Which is interesting given where most tech comes from.
> Level 3: Meeting rooms (Item ID: U02) Difficulty score: 346
This task involves managing requests to reserve a meeting room on a particular date using a reservation system. Upon discovering that one of the reservation requests cannot be accommodated, the test-taker has to send an e-mail message declining the request. Successfully completing the task involves taking into account multiple constraints (e.g., the number of rooms available and existing reservations). Impasses exist, as the initial constraints generate a conflict (one of the demands for a room reservation cannot be satisfied). The impasse has to be resolved by initiating a new sub-goal, i.e., issuing a standard message to decline one of the requests. Two applications are present in the environment: an e-mail interface with a number of e-mails stored in an inbox containing the room reservation requests, and a web-based reservation tool that allows the user to assign rooms to meetings at certain times. The item requires the test-taker to “[u]se information from a novel web application and several e-mail messages, establish and apply criteria to solve a scheduling problem where an impasse must be resolved, and communicate the outcome.” The task involves multiple applications, a large number of steps, a built-in impasse, and the discovery and use of ad hoc commands in a novel environment. The test-taker has to establish a plan and monitor its implementation in order to minimize the number of conflicts. In addition, the test-taker has to transfer information from one application (e-mail) to another (the room-reservation tool).
Basically, users are given an intentionally badly designed user interface in which they receive no training, and a task that is impossible to accomplish within the obvious constraints of the interface, and asked to accomplish a goal. It simulates the experience of being a low paid customer service rep in the third world using crappy software and seeing if you can handle it or not. If one has common sense, intelligence, and a sense of valuing their own time, they will recognize this tasks as useless BS and refuse to cooperate further in the test.
> Basically, users are given an intentionally badly designed user interface in which they receive no training, and a task that is impossible to accomplish within the obvious constraints of the interface, and asked to accomplish a goal.
This is a gross misrepresentation of the test. The task is to manage incoming reservation requests. Having to decline such a request does not mean the "task is impossible to accomplish". It just means that the correct answer is to deny the incoming request within the framework of the test. Also, the test simulates an entirely realistic simulation. In everyday live, I often have to decline meeting requests and propose alternate dates. Also, having to use badly designed user interfaces without training seems very realistic to me. Being good at handling that is a relevant skill in modern life.
Here's an example of the sort of task you had to complete (http://nces.ed.gov/pubs2016/2016039.pdf):
> Level 3: Meeting rooms (Item ID: U02) Difficulty score: 346 This task involves managing requests to reserve a meeting room on a particular date using a reservation system. Upon discovering that one of the reservation requests cannot be accommodated, the test-taker has to send an e-mail message declining the request. Successfully completing the task involves taking into account multiple constraints (e.g., the number of rooms available and existing reservations). Impasses exist, as the initial constraints generate a conflict (one of the demands for a room reservation cannot be satisfied). The impasse has to be resolved by initiating a new sub-goal, i.e., issuing a standard message to decline one of the requests. Two applications are present in the environment: an e-mail interface with a number of e-mails stored in an inbox containing the room reservation requests, and a web-based reservation tool that allows the user to assign rooms to meetings at certain times. The item requires the test-taker to “[u]se information from a novel web application and several e-mail messages, establish and apply criteria to solve a scheduling problem where an impasse must be resolved, and communicate the outcome.” The task involves multiple applications, a large number of steps, a built-in impasse, and the discovery and use of ad hoc commands in a novel environment. The test-taker has to establish a plan and monitor its implementation in order to minimize the number of conflicts. In addition, the test-taker has to transfer information from one application (e-mail) to another (the room-reservation tool).
Basically, users are given an intentionally badly designed user interface in which they receive no training, and a task that is impossible to accomplish within the obvious constraints of the interface, and asked to accomplish a goal. It simulates the experience of being a low paid customer service rep in the third world using crappy software and seeing if you can handle it or not. If one has common sense, intelligence, and a sense of valuing their own time, they will recognize this tasks as useless BS and refuse to cooperate further in the test.