1. "Hey... this bug didn't use to happen. Where'd that get introduced?"
2. OK, this unit test will tell me when the bug is fixed.
3. Now lets automate bisect to tell me where this
test first failed even though I just wrote it.
Also, if you do commit your unit test (a "bad" habit of mine) I have no idea how I'm supposed to work with this. I end up copying the unit test by hand to each commit it tries to test.
In summary, it seems like bisect was built with manual testing in mind. I know you can automate running a shell script, but I don't write my automated tests in shell script.
EDIT: Keep in mind, not all projects are interpreted. Some are compiled.
Obviously in practice you'd probably just use whatever the most recent version of the test is.
Really I think you're problem is that you're overspecifying rule 2. It's not about identifying a "unit test" per se, it's about identifying any specific process to detect the bug. If the tests in your tree work, great. If not, you may have to do a little development to make one that is automatable.
Easy way to do this is put your test external to the tree. Then, bisect as normal for git, and run your test.
This is probably not uncommon for kernel tests, as you often have a test which could be "does this random laptop resume from sleep correctly?" Not really easy to automate that. And doesn't live in your tree.
that's what rtcwake is for
So, yes, you may be able to automate it later, once you fully understand the specifics that cause it to not rewake. Possibly specific to how it went to sleep with the lid close. Until then, it is nice to be able to do ad hoc tests when necessary.
I wish there were a guide or some official "this is how to manage your git repository along with your tests so bisect will work wonders" - IIRC, bisect gets tripped up by commits that won't build and also some merges... would be nice to know what exactly to do about those once they're in your history.
E.g. for a Rails app, I might end up using a command like `rspec ~/Desktop/my_new_test_spec.rb`.
git checkout -b testing
git rebase -i HEAD~50 (or whatever)
move test commit to bottom, then save.
The best part is I should be able to resolve any conflicts before I start bisecting.
Where's the "accept as the right answer" check mark? :)
As far as having other uncommitted changes; just stash those. "git stash" is a tool you should use very often for any time you have uncommitted work that you need to clear out before doing some other git operation.
Your tests aren't a shell script, but certainly you can compile and run your tests from a shell script.
git bisect run my_script arguments
* How much time does it take to do a clean build?
* How much time does it take to do a deploy?
* How much time does it take to reproduce the issue manually?
See also "make help" for more available targets.
With something as complex as the kernel, I also wonder about automatic bisect being led astray by unrelated changes. The bisect script might fail for a different reason and zone in on the wrong commit.
If the bisect script fails because of another problem, it is often possible to refine the script, or run bisect again on a fixed up branch.
1) it's a kernel bug; there is a regression in the kernel's documented behaviour
2) it's a bug in your tool because it took advantage of undefined or undocumented behaviour
The bisect found exactly the right patch series, but bisect can only answer the question 'when did it stop working', not 'why'.
You can read about many instances of 2) on Raymond Chen's blog The Old New Thing.
Does the author not understand binary search? It would take about 14 tries to find the failed commit.
A comment as to why you're wrong by the downvoter(s) would be infinitely more helpful.
> Binary search over 15,000 commits would take about 14 tries to find the failed one.
although I don't think that's an important point to be making. (As erikb said, that's still a lot of commits, and that was only the first run.)