It may be dated by now but there is also a tutorial on writing checkers (https://clang-analyzer.llvm.org/checker_dev_manual.html), or follow one of the many examples in the codebase.
My first checker is still there — it just turned 10 years old, and is under 100 lines without comments. https://github.com/llvm/llvm-project/blob/main/clang/lib/Sta...
As said, there are few false positives. For embedded software the fact that Z3 understand bitfields and bit operations is nice. But in general we found that it makes better use of CTU information than a proprietary tool we're using (and now consider phasing out).
When used with cross compiled software the CC+ClangSA combination can be made to work, but is sometimes a bit rough when there are GCC/Clang subtle incompatibilities. But nothing too bad, and CC provided the hooks to work around any issue we found.
Even in "verification" mode, Z3 also helps the analysis as I understand from the docs. Some cheap Z3 analysis are used to prune some paths, which also helps the classical range analysis. Those tools must make some approximation to deal with the combinatorial explosion, so pruning useless paths early can help exploring others leading to bugs. So Z3, even in the default mode, is not only to filter out false positive but can also help finding issues.
I would highly recommend it for people using C and C++. With those languages the tooling around is very important to (partially) mitigate their known "dangerosity".
Very briefly, the analysis tries to approximate the "states space" for the program, and if it intersect problematic value then it detects a bug. For example, if it can approximate the range of values a variable "x" can take, and this range includes 0 and x is used for division, then a division by zero bug can happen.
When an issue is found, those tools can give you the detailed path leading to the bug, from input down to a call chain to the bug.
The challenge here is to find a good approximation of the possible states. If the tool over-approximate, there will be false alarms. If it under approximate, it will miss bugs. Most tools do a bit of both ;)