Should new users be allowed to provide questions similar to others?
Depends on what you mean with "allowed" and "similar". A new user can post similar questions, but if it is similar to the point some users consider it a duplicate (aka, a borderline duplicate) it should be closed. Duplicate questions are ok to ask (as long it is not recurrent from the same user which is not doing any previous search within the site before posting; remembering if a user post too many questions that get closed he/she will receive a temporary question ban, etc; that is why it is not ok asking too many duplicates in a row), because there are many ways asking the same question, but it is important that answers don't be split across the site. So pointing to the canonical post (which not always is the older one) will help users finding the best answer.
But what if "similar" means it is not a duplicate, and it is closed anyway? No problem, the "On hold" notice/message will tell the OP to edit his/her question and explain why it is not a duplicate. Edits to questions with "On hold" status will be sent right away to the Reopen queue, where it can be voted to reopen. Moreover, a user with more than 3k reputation can also vote to reopen a question without any need of an edit being made, and the question will also be sent to the Reopen queue.
Recently on CrossValidated I have found that new members may not have studied all the rules prior to asking their first question.
Unfortunately, nowadays this is a big problem, but not exclusive to SE. Are you helping moderating the community leaving comments to such OP's to visit the tour page and the help center, for example?
Additionally, I think the actual problem is that many new users don't study any of the rules at all (not even the tour).
I find that moderators and other users will quickly vote to close either because they think it is a duplicate or statement of the question is not clear or the question is off topic.
Really? I find it is still a slow process here in CrossValidated, with experienced users refraining to rapidly closing questions to the point they will collect poor answers and OP will not be satisfied. You probably know that a question with a positive score answer is considered answered, so the community bot will not bump this question to the active page; it will be buried with that poor answer. On the other hand, if a question is quickly and correctly put "On hold" the OP will have a chance to edit the question to make it more accurate and chances of receiving a better answer will increase. At the same time, we keep our site the most clean as possible.
What we could try keeping improving, is to give a chance for a new user to improve his/her question before doing any downvote; and if possible, leaving comments with specific feedback, beyond what the closing messages already say.
Such decisions are a matter of judgment.
Yes, that is why closing, reopening, approving edits are mostly processes which are submitted to peer review (I say mostly, because mods and users with gold badges have the hammer vote; but I am confident to say in most cases this is not a problem. In this respect, I think we have a minor problem with obsolete comments, but this is another discussion).
I have seen cases where these new members become very frustrated.
No comments to this sentence, because you did not bring any example. It is difficult to say something.
...I think in such cases a comment to the new member suggesting what is wrong with the question should be done first.
I think it could be done at the same time. That will help. In fact, we have carried some important efforts to incentivize users leaving comments. See the work of Silverfish and gung here in meta: How best to use the review queue?.
They should wait a reasonable amount of time for a an edit to the question or a comment from the new member before taking action.
Well, I think they shouldn't if they don't want to. We are all volunteers here and time can be limited. We are already curating the site and answering questions. OPs need to make their part, i.e., helping users to help them. Reading the rules in the help center and taking the tour is one thing they can do. Another action they can take is to post the question and be ready to quickly provide replies in comments and edit their question. After all, it is in the OP's best interest to improve their own question.
On two occasions I did things that I thought were reasonable but got moderators upset with me.
Again, difficult to comment without real examples. I will make some general comments though.
Situation 1: A new user asked his first question that I told him was not complete and looked like a self study question. He commented that he couldn't edit the question because he thought he was limited in the question length. So I agreed to edit it. The question was put on hold. I don't remember whether it was before or after I made the comment. So I edited the question adding the information he gave me in comments. I also added the self help tag. I then found a second question with similar problems so I edited that one too. Two moderators criticized me for this for different reasons. On the second question I was told that I should have given the OP the chance to edit the question himself (not knowing the discussion that went on with the first question). The other moderator didn't think I should have edited a question that was already put on hold.
There is a very specific agreed internal policy for the self-study tag. Did you follow it? I don't agree having this tag on our site (despite I agree the specific closing reason and its respective guidelines to OPs is very helpful; but tag and closing reason are different things), but the majority of users agree with such policy, therefore it should be respected.
I suggest you to narrow down your question (with examples) to emphasize better what points you don't agree with the feedback received and how they are different from what the policy says.
Situation 2: A new user asked a simple question. I had a satisfactory answer but I also knew that there were many situations (possible for different modeling methods) where this problem came up.
It could be considered a "Too broad" question; hence, putting it "On hold" and asking the OP to narrow it down to which "situation". But if you felt it was not the best option, then, keep your opinion (like you said, it can be a subjective decision).
The user was worried that she did something wrong. I thought that it would be best for me to give the answer and point the OP to a search method that using the keywords "negative R square" would yield many hits. I thought it would be a difficult task to see if I could consider any of them an exact duplicate. I thought that the OP should look through a few of them to see how commonly the question comes up with a variety of models. The OP and others liked my question and I got 2 up votes for it. A moderator suggested that I should not have answered the question but rather should have searched for a duplicate and recommend it for close/hold. He assumed that there would be an exact duplicate and that I was cluttering the site with repeat questions and an answer that was short (not canonical) and not as informative as a longer supposed duplicate.
We should always give a deep thought to what moderators say. They are experienced users when it comes to moderation, they curate the site, they know better than most users the overall content we have within the site, because they need to read a lot.
With that being said, users have the right to disagree. If you think the best option was to answer that question, go ahead. Perhaps, you did some previous search and did not find a duplicate question. Always try to use your best judgement.