The following information will be collected during the duration of the exam:
Your microphone
Your webcam
Your physical location
Your head movements
Your eye movements
Your mouth movements
Creepy as all get-out. By all means, lets have more leaks like this.
It's worth noting that these are Proctorio's (https://proctorio.com/) restrictions; they're just a third-party company that Amazon is using to administer the test. I've used a couple of similar systems (e.g. ProctorU - https://www.proctoru.com/, where a human will actually watch you via webcam(!), or ProctorTrack - http://www.proctortrack.com/), and "monitoring" of this sort is pretty much par for the course (no pun intended).
These companies are typically used in settings where some level of rigor is expected (e.g. administering an exam for an accredited university). In that context, I think these restrictions make sense in order to ensure some degree of validity in the test results (especially since you're only expected to run the software for the duration of the test).
> It's worth noting that these are Proctorio's (https://proctorio.com/) restrictions; they're just a third-party company that Amazon is using to administer the test.
If Amazon has chosen to use use Proctorio, with an understanding of how it operates, then it's very much Amazon's restrictions.
It just sends the wrong message. As a company, you are advocating tech that spies on people or in this case the test taker. Overall, it leaves a nasty taste in your mouth that a company would use technology such as this. What else would they be okay with doing once you start working there?
Why not do an onsite if you want to make sure of the validity?
>Why not do an onsite if you want to make sure of the validity?
Convenience and logistics. Is it worth yours (or Amazon's) time to pay for travel expenses, when applicants could literally be located anywhere? Evidently not, at least not in the early stages of the process.
If it's not worth Amazon's time to interview me in person then it's not worth my time to consider them as a potential employer. This is just lazy and they will end up with lower quality candidates because of it.
This seems to be a way for them to screen for the 1000s of entry level applicants, who don't have large supporting work histories. They need to screen them somehow. One way is "Only from this list of schools." Another is "Based on what some person thinks of your resume writing ability as a new engineering grad." Another is "Based on what our algorithm thinks of your ability to place keywords in your resume." I don't think this is that much worse an alternative for those who may not have other means to get into a prestigious company.
When I was last in school I struggled with "networking your way into a company" and wound up landing my internship (and ultimately full employment) on the one company who offered a test before the "Fly you out to meet us" step.
They will probably fly you out after this, for another stage of the interview. But at the "FizzBuzz stage", they do really need to make sure that the person answering the question is the same person they're later flying out. There are a surprisingly large number of college students who've gotten used to "paying for grades", who attempt things like that.
Because the point of FizzBuzz is not to get a working FizzBuzz class that can be integrated into an application, it's to see if the candidate can actually produce a working version themselves.
Imagine you're hiring someone to do your taxes. Sure, you'd expect them to use a calculator to do all the hard addition/subtraction, or more likely spreadsheets. But if you asked them to add 5 and 3, and they said "I would use a calculator, this is a stupid question, I can't just tell you off the top of my head" then you're probably not going to hire them.
It's a super basic test of 'Can this person code themselves out a paper bag."
which can be determined entirely without the aforementioned methodology. if you read the guys article, you'd see that he'd already completed the basic tests demonstrating basic competency.
being able to solve problems is more important than how you solve problems. (generally speaking, yes i know there are exceptions)
And that's why Amazon's phone-interview coding exercise is a rather complicated coding question delivered under proctoring, rather than simply fizzbuzz itself: to actually filter those people out early and avoid the cost of flying them to Seattle, rather than just "going through the motions" of doing so.
I interviewed with them a decade or so back. First interview was a phone screen. Ok makes sense. Second interview was another phone screen. Never mind the fact I lived about ten miles away.
On there other hand two years ago I interviewed with a company on the other side of the country. After a half dozen phone interviews they decided to fly me out... For more interviews
I agree with your reasoning, i think the alternative perspective is that traveling somewhere to interview is a significant time commitment on my part, and I might welcome alternative testing approaches where I don't have to fly somewhere until I'm reasonably confident I'm going to get an offer.
Is it worth any applicant's time to participate in a silly dehumanizing screening process that boils you down to a number before ever speaking to someone?
> dehumanizing screening process that boils you down to a number
That's a very long-winded way to describe a prescreening questionnaire. Are you saying that you think no one should ever take a job if there's a prescreening process?
Isn't a normal class room in which you take tests while required to be silent and with an instructor/TA/proctor keeping en eye on you very much akin to spying on people? (using people rather than tech, but why is that distinction important?)
I'd be more concerned about the the level of access the monitoring software has to my computer and any services with vulnerabilities it might leave behind.
I think I'd only consider doing this if I used a computer I could wipe completely before and after. Even then the amount of distrust is off-putting.
Not necessarily, because that classroom proctor is not asking to install spyware on my personal laptop.
I can understand why ProctorU does what it does, but that process puts a significant burden on the test taker, one which is entirely different than taking a proctored exam in a classroom.
Depending on which definition of "spying" you want to apply, it is spying. Moreover, it is invasive and unsettling.
Perhaps most critically, it is a more subtle test for the prospective employee-- are you the kind of person who would comply with a ridiculous request that violates your privacy in the hopes that it will land you a job?
I dunno. If I were an employer and an applicant actually installed this without question, it would make me seriously reluctant to hire them-- this is the sort of unquestioning, docile employee who could be more vulnerable to social engineering,who would not be likely to consider consequences of their actions, who does not take security, privacy, and safety as a priority for even themselves (and is therefore less likely to do so for customers and clients).
These may not be qualities that some companies value, but from my perspective, I think the "nope" letter to Amazon shows strength in independent thinking, creativity, and moral character.
It's like those companies that required you to hand over your social media passwords so HR could dig through your history.
That is horrifying. If Amazon doesn't hire you because of this, it says quite a bit about their company, fuck 'em. Their loss. There are companies out there that actually respect their employees.
> Depending on which definition of "spying" you want to apply, it is spying.
Without the secrecy aspect, I don't think you can say it's spying by the definition people typically use. If you went to an examination hall and took a real proctored exam [1], would you consider that the proctor was "spying" on you?
> Moreover, it is invasive and unsettling.
This, I can't argue with. It's hard to pin a definition on what makes things creepy or unsettling. Some people find clowns creepy. Why? I have no idea. But there's no sense in arguing about it.
Personally, I think it's a clever use of technology to streamline and scale a traditional process (exam proctoring) that avoids making you travel to an exam hall or interview.
> Without the secrecy aspect, I don't think you can say it's spying by the definition people typically use.
I mean it in the sense of "I went on a blind date, but my parents sat at the adjacent booth to spy on me". I knew they were there, but it was inappropriate invasion of privacy.
> If you went to an examination hall and took a real proctored exam [1], would you consider that the proctor was "spying" on you?
If I was taking the exam on a personal laptop and was handed mysterious software at the examination hall to install on my machine... (along with images of the inside of my home), then yeah...
I think you're reading too much into this. It's to make sure that the interviewee isn't trying to game the system. Pretty simple. You can opt-out anytime you want. You don't even have to start in the first place. It's not spying by any definition of the word. Unless you make one up for it.
I understand that the goal is to stop cheaters. That's an admirable goal, and a perfectly legit one. However it's the completely wrong way to do it and sends a very strong message to anyone applying that "our company, for any reason that serves our interest, expect you to be compliant, to give up your personal privacy, and to submit to our invasive internal processes whenever we feel it's necessary. Expect more."
Asking this of you, before you're even working for them is presumptive and not a good sign that this is a place you're going to want to be. It's basically a huge flashing "We Fuck Our Employees!" right there on your job application.
It's probably the cheapest and most efficient way to screen candidates, and ensure accuracy. Not disagreeing with you -- just saying that their primary goal is probably efficiency (no surprise).
I don't know what is so wrong with this. It's only for the test duration when they want to ensure that you don't cheat. You can remove the extension once the test is over.
I took a test with ProctorU a few months ago for my university class. It didn't seem creepy at all. They installed LogMeIn and watched my webcam, but as soon as the exam was over they disabled both. It's just a way of making sure there isn't any cheating.
I can't believe what I'm reading. A university installed effectively spyware on your device, to watch you as you were sitting an exam?
I sat about a dozen university exams in the last couple of years and in no instance did I have someone staring at my face, or peeking over my shoulder, during them. There were always supervisors in the theater, but they mostly walked along the aisles of desks and made sure to be unobtrusive and quiet as a mouse.
I really don't see any justification for what you describe.
I really don't see any justification for what you describe.
In the case you describe you have to be physically present at a fixed location at a fixed time. In many cases that might be inconvenient or impossible, and I can certainly see situations where it'd be nice to have an alternative. Not having to have my physical location coincide with the physical location of my university of choice could be quite handy.
Had the same experience. I passed my PRINCE2 (similar to PMP certification in the US) with ProctorU from my home.
I connected with a Lady via cam and she asked me to show via cam in a 360 style if I had anything else in the room I was in (books, someone helping,etc).
Took the exam after that and passed it. It is a bit unusual but very very convinient. If I had to take the exam physically somewhere I would have needed to take a day off.
Yep, and a pretty convenient one at that. When using these services, the effortlessness of taking a test in the comfort of my own home (at a time that worked for my schedule) outweighed privacy concerns, although I certainly empathize with those that don't feel the same way.
Seems like we're ok with using technology for telepresence, but would prefer standard apps like Skype or google hangouts. The question is why standard apps aren't sufficient.
This immediately brings to mind the history of scientific racism, specifically with craniometry being used to 'prove' that certain races are inferior. I realize this probably isn't what they're doing, but the fact that they're giving themselves permission to store facial data for later analysis seems questionable given the context.
> This immediately brings to mind the history of scientific racism, specifically with craniometry being used to 'prove' that certain races are inferior.
I can't imagine why...? That would be like seeing a freeway and instantly thinking of the Nazis.
If some corporation did want to be racist (which has not been established), they'd hardly need fancy face-analysis technology to do it. They could, you know, just look at the applicants.
How is a computer looking at the applicant data, including their faces, more insidious than humans looking at it? In either case the company could claim they were totally not paying attention to skin color.
You should check out the history of IQ testing particularly in the US. On paper, it seemed to be a perfectly objective way to determine an individual's intelligence. But correlating the data, it was later shown to disadvantage blacks and that is why IQ testing for jobs is now illegal.
US corporations have a multi-hundred year history of using thinly veiled proxies for race as part of their hiring practices. This continues to this day, including in the tech industry.
It's not exactly that far fetched to think that someone might someday think it's a good idea to start applying some of those new AI APIs to the candidate interview videos just to see what happens. In fact given enough time and enough departments/companies, it frankly seems more likely than not.
Especially given that we already have AI scouring GitHub looking for people to recruit. Assuming it's not already happening, it seems unlikely that there won't be startups applying AI to hiring interview videos within the next couple years.
> US corporations have a multi-hundred year history of using thinly veiled proxies for race as part of their hiring practices. This continues to this day, including in the tech industry.
That's a pretty serious accusation, one that could spark a lot of successful lawsuits against those companies -- especially in the current political climate in the states they tend to be located in. Do you have any specific evidence to back it up?
Google won't you apply to work there without giving your college GPA, despite the fact that all the academic research, and also their own internal research, shows that GPA doesn't correlate at all with job performance.
I don't think Google is asking for GPA anymore. Not for experienced hires, anyway. I can imagine they might still ask it for interns and college hires, where it's relevant. But they didn't ask me two years ago (I was seven years out of university at the time).
Using GPA as a screening measure when members of a racial demographic routinely preform more poorly due to socioeconomic factors, it starts to look more like a proxy for race.
Keep in mind, this is during the test. What are you going to do, take a break and load up some porn?
These conditions only apply for the 24 minutes in which you take the test. How is this different than if a person were watching you during an interview?
Paid homework projects are far superior. Plagiarism checks can be run on the provided solutions, and a follow-up conversation concerning the project at the interview should easily suss out anyone who had a third party to do it for them.
That approach also has the advantage of actually being a relevant to the position—unlike some generalized intelligence test.
Frankly if people are putting up with the test they have now, Amazon could probably get away with issuing unpaid homework projects, however classless that would be.
I've found that the answer to all this is to give a 30-60m remote test (no proctoring), where they get to write the thing as they want, and a 15m discussion where you analyze their solution and assign one new feature then let them walk you through the changes. No code, no whiteboard, just discussion.
A lot of people can't whiteboard code from scratch but can take a piece of code they've just written and discuss next steps. And it's also an actual work skill that many good coders are bad at, so it gives you a good insight into high and low-level people.
THIS! This is the real issue. Such unfettered access presents the risk that they won't just collect data on what happens during the 20-something minute window. The risk is that they gather files, collect your browser history, and read your communications.
Keep in mind that if you have extensions installed, many of those extensions will make callbacks to other services. This will track the use of those, too.
>> How is this different than if a person were watching you during an interview?
It's different because they're recording you, not to mention they're doing it by installing spyware on your computer. That's not what a person does when they watch you, unless they're watching you through a camera.
I think it's a bit silly to get so worked up about this. The test is simply to gauge programming skill in an online test, and they want to check you are not cheating. Though I think they could have worded it without freaking people out: "You are going to be video monitored during the test".
Presumably you're aware that job interviews and the actual job are different things? At a real job no one cares if you use Stack Overflow because there's ample other data to assess your productivity (like...your actual productivity over the months and years that you're employed). The challenge with interviewing is trying to extrapolate performance from a much, MUCH shorter time horizon[1]. You're not testing the exact same environment, but you're using a different environment (the test/interview) to hopefully assess skills that are relevant in the environment you're optimizing for.
This is hardly a new concept: exams work pretty much the same way. There are other ways to assess candidates (like looking at their portfolios), but they have their own weaknesses, and testing skills in a constrained environment can still reveal things.
[1] And this is for the benefit of both parties: I've lost count of how many people I've heard complain about companies that have gauntlets of technical interviews.
Presumably you're aware that job interviews and the actual job are different things?
This is the core problem with tech hiring circa 2016. The sooner this idea dies, the better. It's surprising that more companies haven't noticed the most effective interviews are ones that resemble the company's day-to-day work as closely as possible.
Doing the test with a friend? I wouldn't want someone who can't complete coding exercises on their own. Also I'd expect people to answer simple questions quickly.
I dunno man. I'd much rather see a well understood vetted email validation(or whatever )copied from stack overflow than some thrown together hack in production code.
being able to look things up is a different skill than being able to understand and properly apply what you looked up. Or being able to understand what you are looking up and why.
When I was taking exams at university I am certain the professor could monitor my eye and mouth movements. I could also of course be seen at all times as well as any sounds I made were heard .
How is this different?
Having just reread 1984, it seemed familiar to me. Sure, people cheat all the time to try to get a job (I have no idea why, even a cursory phone call will eliminate the ditch digger). Maybe Amazon should focus on making the jobs more enjoyable so people stick around and they wouldn't need such excess to find live bodies.
The reason Amazon (and other large tech giants) are hiring so many people isn't because they are loosing good talent -- it's because the company is growing at such an aggressive rate that there are always new positions opening.
"Secure Exam Proctor Exam Environment: You can generally utilize the Secure Exam Proctor for taking a proctored examination without revealing any Personally Identifiable Information about yourself. The types of information collected depend on the exam settings and can include video, audio, desktop recording, and websites visited. The aforementioned data is encrypted and will not be used by Proctorio in anyway. This information may be used by "Authorized School Officials" to review the actions of Students during exam administration. Proctorio may collect additional information set forth below in the "Aggregate Information" and anonymous information. Aggregate and anonymous information will be used to improve the quality of the Secure Exam Service.
Note: Proctorio utilizes zero-knowledge encryption to keep your information safe."
If it is using proctorio.com as mentioned in other comment, I think it is normal.
On Juniper or Cisco certification test (using Pearson vue) they warn the candidate something like this (not fully the same especially for eye and mouth movements, but the candidate is video recorded).
Basically it is to make sure the candidate is not cheating.
The different is it is using Desktop provided by testing center partner.
So, what if I run the browser in a VM, and cheat in a second VM? With a secondary input device to switch between which VM is displayed on my screen? Or even an overlay?
They’d be completely fooled.
It’s completely useless, and so needlessly invasive.
They have a human watch you all the time through the camera, so they’d notice if you look anywhere else. And they require you to show them all around your room first, to prevent that.
That’s why you install the system below the desk, and use it that way.
No, an in-person interviewer will not know the tabs on my browser or the programs running on my computer.
Also at an in-person interview, the interviewee has a symmetrical power relationship to the interviewer in that they can both observe on another. This practice slides the power relationship heavily toward the interviewer.
Furthermore, at an in-person interview, the information is not analyzed in an unspecified manner which further skews the power balance toward the large corporation.
Finally, at an in-person interview, the information is not digitally stored at an unspecified location for an unspecified time and transmitted to unspecified parties. Nor is it subject to unauthorized access by unspecified parties in unspecified countries due to unspecified security arrangements.
The interviewer is most certainly looking at your computer screen if you are using it during the interview.
I will probably never be in a situation that comes close to the staggering power asymmetry of someone deciding whether or not I will have a job. Even if I'm charged with a crime the court has less power over me than an interviewer, in that its decisions are subject to juries and appeals and I get to watch the proceedings.
The information is analyzed unscientifically according to the personal biases and tastes of the interviewer to form an emotional impression he may not even be conscious of. "Not enough eye contact, I don't like this guy." I'd much rather it be judged by an algorithm trained on actual job performance years in.
Ever heard of Greenhouse? Your interview performance (as reported by the interviewer) is absolutely stored indefinitely in some 3rd party's database subject to unspecified controls.
Amazon has piloted a program to make offers solely on the basis of the test, not just to put a gate in front of the onsite. They should be making it at least as resilient to fraud as an onsite, which is why I think these measures are appropriate.
Doesn't really fade me. I mean, it's just for the duration of the exam. Install on a throwaway laptop, do your interview, be done with it.
It's not so much the recording of these metrics that I'd be worried about, but them storing it to build profiles. Different story.
These particular metrics are now quite common for online exam systems.
Mind you, these are valid concerns, and anyone should feel free to reject them. But on the other hand it's nice to have a remote option (for certifications, exams, interviews...) and at the same time a cheat-prevention system.
You'd be creeped out by the stories I have of freelancers or even consulting companies having some "star" employees doing interviews and then other employees showing up at work instead (remotely or even physically). Once you've been burned by stuff like that, you understand they take precautions.
In OPs position, I'd ask for the storage and retention terms and conditions though.
That they have the webcam and microphone is not 'creepy'.
They just want you to be able to take a test remotely without cheaters.
If this were a regular website, sure, but it's for a specific reason, during a specific event, that ultimately you are participating in.
If you were to go and 'take an exam' on Amazon campus - they'd be doing the same thing - but a human proctor would be looking for the same things :)
I think the contextualization of it (i.e. it's 'your home') definitely seems 'creepy' - but I don't think it's a cause for concern - there's a reason they are doing it, it's not nefarious, they are open about it.
Again - it may feel a little creep-show because we're not used to this, or rather, in any other situation it would be truly creepy ... but I don't think there is cause for concern here.
I also think that the logical questions are not unreasonable.
Is it really creepy? When I interview you in-person I am able to note all of those things if I want to and I can include "Your odor" on the list. They probably just have a legal team somewhere that feels that it is worth disclosing these items that really are common sense in a face-to-face.
I don't see how this is creepy at all. If you're building a product (especially in the entertainment field) and you want to get feedback from users a really good way of doing it is by having access to their facial expressions as they go through your product. This should be done with their consent (such as in this case), so I don't see the issue.
they weren't building a product. it was a job interview, he was at home, and this was software they required to be installed on his own computer (at home)
The same idea applies. The company wants to ensure people aren't cheating and also wants to get some insight into how people respond to their interview process. This only improves the process for future interviewees.
if you're interviewing someone and your interviews questions can be answered by a quick google, maybe you should think that during the course of a work day, the same applies and you should think about a test that actually tests skills involved in the work the candidate will be doing
i have no problem with a candidate doing something totally by themself in part 2 of 5 before in in person interview; theres still another 3 different interviews (many of which will be in person) to catch cheating and such
