What is it that you find challenging about math? If it's the actual computation/calculation -- the part where you are finding a numeric answer -- don't worry, that's nowhere to be found in abstract algebra (or any higher level math). Pure math (of which abstract algebra is a part) is about the study of patterns more than anything that has to do with numbers.
In fact, the name "abstract" refers to the fact that it's concerned with abstract collections of things -- for instance, groups. You'll study sets of operations on groups -- if you are able to identify Collection X as a group, you immediately know you can apply theorems a, b, c, etc. to it. For these reasons, the sort of things you are likely learning in pre-alg on Khan Academy don't have much direct applicability.
I think it's an enormously beneficial subject for programmers to study, maybe the only math course beyond the standard discrete math that I think should be required. (I want to add category theory, but I don't feel I can as I only have the barest grasp of the fundamentals myself...) As with all pure math courses, it will quickly move beyond the depth/level you can actively use in programming, but the mind-expanding it does is really great at encouraging the sort of abstract thinking the OP's post is about. It has strong relations to generics, interfaces, polymorphism, etc.
As for how to get into it, I used this book: https://www.amazon.com/Book-Abstract-Algebra-Second-Mathemat....