Metrics to track might include the number of comments made, number of hits on the site, number of upvotes / downvotes, and probably many more.
Some people obviously feel strongly about this issue, and it might be necessary to take steps to prevent gaming of the outcome. Keeping the metrics secret until after the test would help with this. So would publicly announcing that only a small (say 10%) but undisclosed subset of users will be used to determine the outcomes of the testing. So any individual user wouldn't know if their behaviour would affect the results, and so would have little incentive to waste their time trying to affect the outcome.
It'd be nice to take a data-driven approach to resolving this question.