Should we encourage people with unanswered question to reformulate
their question or to step down a level in the difficulty? Or to expand
furthermore the problem?
Yes we should encourage people to make their questions easier to follow. In many cases more explanation of the background, or of the aims of their analysis, or the addition of an example† to illustrate the issue, would help. On the other hand, extraneous material & excessive detail is off-putting. But we can't really tell them to ask an easier question just because the one they want to ask is too difficult.
Is there best practices in that cases, where you are willing to help
but the questions are one step ahead of your competencies?
Well you can do some research‡, or perhaps provide a partial answer if appropriate. But if you can't answer, you can't answer—keep an eye open for questions you can.
If you think you know the answer, but aren't completely sure, give it anyway. Of course don't pretend to a certainty you don't have, explain your reasoning, & provide any relevant references. Even if you happen to be quite wrong you might provoke someone into writing a better answer or suggesting corrections to yours—the person who asked the question, you, & future readers all win.
† Minimal working examples can be as useful in Statistics as in programming.
‡ A good place to start researching is, of course, Cross Validated; I suspect there are many duplicates & near-duplicates among unanswered questions. Sometimes a question just needs to be flagged as a duplicate; sometimes a short answer explaining just how answers to a linked post or posts relate to it is needed.