1
$\begingroup$

My very first question here has been closed as "unclear what you're asking".

The "How to Ask" section says

  • Is your question about statistics?
    As far as predictions belong to statistics.
  • We prefer questions that can be answered, not just discussed.
    And so do I.
  • Provide details. Share your research.
    I posted everything I could. I could do more, but with little effect:
    • I could invent some ad doc data-clustering algorithm, which might fit the bill, but I'd prefer some more scientific approach - where I have nothing to say.
    • I could implement some machine learning, but that seems to be an overkill for this task.

The task at hand is really specified as I wrote: Do some prediction, it's clear that it can't be perfect.

I surely agree that my question could use some more details and I'll gladly add them. However, I disagree that it can't be answered.

Reply to comments by Glen_b

I think your question still seems pretty vague about what precisely is required

This is no research task with some error function to be minimized. There's a customer's wish to "make some predictions", so I'm trying. I'm sure, I'm not the only one and actually, I'd expect this to be well-known problem. I'm well aware that given the tiny and irregular data sets, not much can be done. OTOH something can be done and the quality of the result can be evaluated on the current and future data.

it talks about how something "seems"

I was deliberately using such phrases as I'm only describing how I think the output for a given data set should look like. A different output is acceptable.

guess how that might generalize for your feelings about how things seem

You don't need to generalize my feelings, just provide any solution. Or multiple solutions. Imagine, I was asking about how the sequence 1, 2, 3 continues. You could tell me, that it's impossible to tell, and it'd be correct. Correct but useless. Or you could point me to the arithmetic progression and Fibonacci series and whatever and I could try it and verify the predictions using future data.

an answer would rely too heavily on opinion about how to interpret it

So don't be scared and give me some answer. It may fit and then we're done. Or it may not, and then I'll see what I've missed.

If there wasn't the problem with events around midnight, I'd simply determine the most common weekday (using some decay) and compute the average time and then look how good it works. I guess, this alone would allow to predict the next event with some 80% certainty and two hours tolerance in half the data sets (some data sets contain events happening bi-weekly, etc.) and even this could be an acceptable solution for now.

Reply to the answer by Scortchi

You seem to be part-way through developing a heuristic method for prediction

Actually, I still haven't really started besides extracting the input data and analyzing maybe six data sets manually.

it's concerning that my knee-jerk prediction for the first example is quite different from yours

Yours is equally valid. In fact, it's something I haven't though about. And the data continues: 180320-Tue-2222. So you were right.

the importance of providing context

IIRC you're the first telling me about it. I simply left the context out, as I don't think it's important for this question (though it surely is for many others). I may be right as you lacking context could make a better prediction than me.

Can you share the data?

I guess so, I'll ask.

(tons of question)

I'll answer them in the question.

leading to a mare's nest of conflicting answers

I really don't think, it'd be a problem. I may be wrong...

Summary

I'll reword the answer, provide the context and add lots of data. We'll see if it helps.

$\endgroup$
17
  • 4
    $\begingroup$ Frankly, I cannot understand your question. "Do some prediction" is uselessly vague. You refer to "events" but illustrate them using only times. Your uses of the words "extract," "sort of prediction," and "false positives and negatives" are obscure. The example is so small that we are unlikely to be able to guess what you want just from examining it. I can tell that you might be concerned with aggregating times, but that's it. $\endgroup$
    – whuber Mod
    Mar 18, 2018 at 19:26
  • $\begingroup$ @whuber "uselessly vague" - I'll tell my boss about your opinion. :D +++ The events have no attributes, it's just that something happens. +++ Obscure use of words... "extract" = "produce output", "sort of prediction" = "prediction which is expected to be wrong sometimes", "false positives and negatives" - what's wrong here? +++ I've added three real data sets with expected outputs. $\endgroup$
    – maaartinus
    Mar 18, 2018 at 20:35
  • 8
    $\begingroup$ I'm just trying to indicate where you might focus to make your question understandable. Complaining won't get you anywhere, but an honest effort to communicate is always received warmly. Although of course your question can be answered, because it is vague it admits distinct interpretations. In such cases answers can be worse than useless because they might mislead readers who don't adopt the same interpretation. That's why we have a mechanism to close questions. The method for reopening is to edit the original question and keep doing so until people vote to reopen it. $\endgroup$
    – whuber Mod
    Mar 18, 2018 at 21:00
  • 2
    $\begingroup$ @whuber I'm really grateful for your comments. While I'm new here, I'm pretty seasoned on SO. What I really hate is closing a question without any feedback, unless the question is a hopeless mess. You're the only one having given me any feedback and it's really appreciated. $\endgroup$
    – maaartinus
    Mar 18, 2018 at 21:48
  • $\begingroup$ @whuber But concerning "edit the original question and keep doing so until people vote to reopen it", I must disagree. This is a small site, the question has got 16 views in 47 hours and I may keep editing it for months until someone notices. This sounds like a bad joke. Especially given your "Complaining won't get you anywhere" as without having complained I'd get no feedback at all. Adding unsolicited information makes only limited sense as soon the question gets too long for anyone to read. You guys should really try to be a bit user-friendly. $\endgroup$
    – maaartinus
    Mar 18, 2018 at 21:59
  • 5
    $\begingroup$ I don't think whuber was objecting to the question as complaining, but to your response to his first comment, which does seem to be argumentative. I think your question still seems pretty vague about what precisely is required (besides some of the things whuber mentioned still being at issue, it talks about how something "seems" and we have no good basis to guess how that might generalize for your feelings about how things seem) - I'd have voted to close as unclear as well, and might well have done so even as it stands now - an answer would rely too heavily on opinion about how to interpret it $\endgroup$
    – Glen_b
    Mar 18, 2018 at 22:53
  • $\begingroup$ @Glen_b I've answered in my question. I don't mind closing my question, I mind doing so it without dropping a note. Anyway, isn't my question crystal clear compared to e.g., this one? $\endgroup$
    – maaartinus
    Mar 19, 2018 at 2:39
  • 6
    $\begingroup$ I certainly agree that putting a question on hold should normally be accompanied by a clear explanation of the problem (whenever the pink-boxed on-hold text doesn't already make it explicit). On the other hand, the fact that you might find a question that you regard as less clear than yours doesn't make yours any easier to answer; it simply indicates that there may be another question that should have been put on hold. (It's possible the lack of answer there may be as a result of our failure to require additional clarification) $\endgroup$
    – Glen_b
    Mar 19, 2018 at 2:48
  • $\begingroup$ @Glen_b Before I even think about closing a question, I'm asking myself, if that does any good. Duplicate or pure non-sense -> yes, unclear question -> probably not. Building Q-A archives is one goal, helping directly is another. Commenting is cheap and there's no deadline for closing. I don't think the question I linked should be closed. IMHO it should be commented and clarified and answered (or maybe closed). The comments alone may be sometimes good enough for the OP. $\endgroup$
    – maaartinus
    Mar 19, 2018 at 4:54
  • 8
    $\begingroup$ Placing on hold is a temporary thing to prevent premature answers; frequently unclear questions get answers that the OP (and certainly many other people seeing the front page) is not aware is not necessarily a suitable answer to what they actually need to know, and before anyone comments that there's a problem, the OP has posted "thanks!" and left again (perhaps never to be seen again). Placing a question on hold while it is clarified rather than leaving it open is sometimes important; it can be a matter of judgement, but I don't think poor judgement was applied in this case. $\endgroup$
    – Glen_b
    Mar 19, 2018 at 7:35
  • 4
    $\begingroup$ At first blush, it doesn't look like it could be answerable. Certainly not by us, & probably not by you. My recommendation would be to go back to your client &/or boss & ask for greater clarity about what should be done. I often tell clients that there is nothing I can do until they have a clearer sense of what they have & what they want to do with it. $\endgroup$
    – gung - Reinstate Monica
    Mar 19, 2018 at 13:39
  • 3
    $\begingroup$ It's unfortunate that this first encounter with CV gave you a bad impression -- especially because your experience and reputation elsewhere on SE shows that you understand the overall system well. But equally neither moderators nor experienced users can usually discern what really lies behind a question that looks unclear or too broad. In reverse, I often see questions on Stack Overflow that boil down to please write the code I don't have or read the documentation for me. So, sad but essential fact: whatever looks unclear has to be bounced back to the OP. $\endgroup$
    – Nick Cox
    Mar 19, 2018 at 17:59
  • $\begingroup$ @Glen_b In theory, I agree with the prevent-premature-answers theory, but IMHO it practically means that everything just gets delayed. Anyway, closing a question without any comment means zero progress - unless the OP gets miraculously enlightened. A comment or a misplaced answer would lead somewhere, nothing leads nowhere. Anyway, now I've got quite some feedback and will improve my question soon. $\endgroup$
    – maaartinus
    Mar 23, 2018 at 3:10
  • $\begingroup$ @gung I'm claiming, it is answerable as the task is "to generate predictions as good as you can (within reasonable effort)". I know the context and I'll add it (while I strongly believe, it's not helpful) and improve the question otherwise, too. $\endgroup$
    – maaartinus
    Mar 23, 2018 at 3:20
  • $\begingroup$ I am seeing the question shifting from (very) unclear to broad (thank you Maaartinus that is a very big change, even when you might not consider this as such by yourself). This poses a much bigger problem. While unclear question can be (often) "blamed" on the person that poses the question, a question that is too broad, is often too difficult to be fixed by the (novice) person that created the question. ------- I believe that we should fix this as a community either by creating good answers or by turning the question into a community wiki and edit it to make it suitable. $\endgroup$
    – Sextus Empiricus
    Mar 25, 2018 at 20:51

1 Answer 1

Reset to default
6
$\begingroup$

You seem to be part-way through developing a heuristic method for prediction using implicit assumptions based on your knowledge of what the data represent & how they've been collected, & on patterns observed over the hundreds of data sets; & to be asking either how the method can be refined, or what other method might be appropriate. It sounds very interesting, but it's concerning that my knee-jerk prediction for the first example is quite different from yours—I'd've said the first event of the week happened earlier than usual on Tuesday & we shouldn't expect another till Wednesday now, whereas you're still pretty confident that another event will occur on Tuesday evening.

Because you're very close to the problem it might appear that people are being wilfully obtuse when they say they don't understand your question. Try to step back from it a bit, & think about how to frame it for a new-comer. See How can we encourage askers to provide context? for the importance of providing context. What are the events? What do you know about them that could aid prediction? What have you learnt from looking at the data-sets? Can you share the data? What assumptions have you made & why? When are you making predictions? What are you going to do with them? How are you going to test different approaches? What do you expect from an answer?

If the question had been left open in its current state there'd be a real risk that fools rush in—& any of us could be among those fools on different occasions—with a different set of implicit assumptions, leading to a mare's nest of conflicting answers.

I said you seem to be part-way through developing a heuristic method for prediction because that's what I took the part of your question where you write "I need to extract some information like [...]" & then go on to describe aggregating events by day to be about. (It comes out of the blue: again, step back, spell things out for your readers.)

I wouldn't make too much of my having got one thing right in one case. In fact I was making up a context when I thought about it—withdrawing money from a cash machine. Think how much the context you have included helps: just the word "users" suggests that day of the week & time of the day might be relevant (& that the pattern of events might be different at weekends & holidays); but not phase of the moon (you didn't write "anglers" or "werewolves").

Several answers using different approaches to answer the same question would be a good thing; several answers using different approaches to answer different questions (i.e. different interpretations of yours) would be a bad thing. Trust us on this: it's happened often enough.

$\endgroup$
3
  • $\begingroup$ Thank you for your answer. I replied in my question here. TL;DR I'll edit my question soon and we'll see. $\endgroup$
    – maaartinus
    Mar 23, 2018 at 3:03
  • $\begingroup$ Concerning holidays and phases of the moon: That's surely a valid point. However, there must be some general procedure allowing some (possibly low quality) predictions even in case you know no context at all. When I see a series of events occurring every Monday, I'd predict the next occurrence on Monday. Sure, holidays may prove me wrong, but they're not that common so most my predictions stay valid. I guess, I can safely ignore holidays in my data: ... $\endgroup$
    – maaartinus
    Apr 2, 2018 at 0:26
  • $\begingroup$ There's definitely no direct rule like "holidays shift the event to the next day", there isn't even something like "holidays prevent about 30% of events" there. Such a rule may possibly be derived from a much bigger data collection, collected over longer periods when multiple holidays happend. It'd probably require analysis of local holidays, which is possible, but in the moment surely not worth the effort. My questions has been reopened one week ago, but there's still no answer. Is there anything else I can do? I'd really prefer anything more scientific to just coding some algorithm $\endgroup$
    – maaartinus
    Apr 2, 2018 at 0:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .