1. To what extent is this a common experience of new answerers? Do we lose noticeable number of potentially active participants this way?
I think I felt similarly when I started participating, and it was years before I became a regular user. As with most communities, this one has some unusual norms that take a while to learn - especially since there's no central archive of all of them (the help pages cover things partially at best). The biggest challenge for me was finding questions simple enough to answer that were not going to be closed as duplicates (or for any other reason). Having spent a lot of time here, I now have a decent idea about whether a question has been addressed before or is going to meet the closure criteria. But earlier, finding a question closed half-way through writing an answer could be discouraging.
2. Is there some way to avoid this?
I agree that upvoting (and downvoting!) more would be helpful, but I'd be surprised if it had more than a small effect. The number of upvotes I get for my answers is mainly a function of how much attention the question gets, and this seems quite random. My most-upvoted answers are nothing special: they just happened to be to questions that ended up being popular. So a few additional intentional upvotes (while appreciated) are probably not going to change this general dynamic. Edited to add a partial retraction of the above point: on reflection, I realised that those few additional intentional upvotes are not trivial. They would have mattered a lot more when I was new to the site and had a low rep score.
But to return to your question about avoiding this engagement problem for new users. I think the problems I faced are structural and hard to address. I think the biggest problem is that the community has to spend a lot of effort to just remove bad questions. This work is largely done by a relatively small pool of people, but new users suffer the consequences too - the new questions page is generally filled with a high proportion of low quality questions. Participating would be easier if this 'background noise' level was lower, but I don't see a good way to make that happen. I'd be fine with mandating some moderately time-consuming steps for new participants to ensure they learn what a good question is before posting, but that's not going to happen. And I don't think anything else would work very well.
[Edited to add] A second problem is that because simple questions are very likely to have been asked before (and hence would be closed as duplicates), the remaining on-topic questions are more specialised and complex. Answering these is a valuable contribution, but the problem is that there's fewer people who have the expertise to understand these answers. And if I can't understand an answer, I cannot justify upvoting it. This also penalises new users, but I can't see a good solution there either.
3. Regarding your upvoting criterion "This answer is more likely to improve the statistical practice of the OP than to make it worse"
I don't know. Pitching your answer at a level that the OP can understand is valuable, of course, and sometimes I'm happy to upvote answers for the reason you cite.
But I don't follow this as a general rule. There are frequently important lessons to be taught that supersede the specific question being asked. Questions can be so misguided that a technically correct answer could still be a bad way to proceed, and conceptual frameworks can serve as straightjackets that hinder the search for good solutions. For example, instead of telling someone how to do the exact test/p-value calculation they have asked about (which might improve their statistical practice), it can sometimes be more important to point out that there may be better ways to think about their problem than through the framework of null hypothesis statistical testing.
