I think it means that he wont be starting fights and will be avoiding them. It does not mean he is naive about why no one attacked him (weight and tough look plays role according to him). It does not mean that if someone attacks him when guards dont look, he will let himself being beaten without attempting self defense.
