If I saw a lot of effort in an answer (e.g., assume that there are a lot of derivations, code, and plots), but I'm not sure that the math is correct, should I upvote to encourage the person answering it?
Some users on this site feel compelled to provide input on things they don't understand and this sounds like a more nuanced version of that...Obviously you can vote however you want but, since you asked...
If you are sincerely unsure about whether or not the answer is correct, I don't know why you'd upvote it: that adds noise to the system and could mislead future readers if the answer turns out to be wrong. I don't see the purpose in giving an "A for effort". Let the people who do understand the answer be the ones who vote. Just because an answer looks complicated or sophisticated doesn't mean it's right. I'd venture to say that a view like the one I'm putting forward is the reason that complicated questions (and the resulting answers) don't get as much credit on this site as the repetitive questions and answers that a greater proportion of users can more easily relate to and understand.
Caveat: I'm talking about the scenario where the correctness of the math is a crucial part of the answer. If it is secondary, and the general approach to the problem has value regardless of whether the math is right, I could see a purpose in upvoting even if you're unable to evaluate the math.