5
$\begingroup$

I’m posting this thread reopening request here in keeping with the procedures outlined in the Meta post "How do you reopen a closed question?" I only learned fairly recently that my question “What are the known existing practical applications of chaos theory in data mining?” had been closed on the grounds of being too broad. There are several reasons why this no longer applies. As per the Meta reopening checklist, I did an extensive rewrite of the post before Christmas in order to sharply limit the question solely to known, existing implementations of chaos theory, specifically in data mining and related fields, not ordinary scientific inquiry. Secondly, I’ve put a significant amount of effort since I posted it two years ago to acquiring the knowledge to answer it myself; as a result, I now know the reasons why the list of implementations in data mining etc. are so short in comparison to other scientific applications (with the Three-Body Problem being one of the classic examples of the latter). I retained the numbered list of potential applications in the post precisely because the striking contrast with the paucity of existing implementations is the story; if I can get this reopened, I can provide an insightful, detailed answer to my own question that will explain precisely why this contrast exists. There some inherent, overarching limitations on the scope of possible applications to data mining and related fields like neural nets, pattern recognition etc., which arise by logical necessity from certain broad characteristics of chaos theory. These characteristics neatly separate the implementations in ordinary scientific studies from the kind of use cases we face in data mining and the like.

If I can get the green light to get this question reopened, my intention is to answer it myself within a week or two (give me a little leeway to craft my post) and accept my own answer, in order to bring this intriguing question (which generated a lot of interest) to a more constructive closure. The answer will reference case studies from sources like A.B. Cambel's Applied Chaos Theory: A Paradigm for Complexity and Alligood, et al.'s Chaos: An Introduction to Dynamical Systems (the latter is incredibly useful as a sourcebook for this topic).

The major issues with overly broad Stack Exchange questions is that they may encourage rambling, far-ranging, sometimes opinionated discussions without a clear direction or focus, often without approaching a conclusion. Now that I’ve researched this thoroughly and narrowed the focus to existing implementations, I can explain why such implementations of chaos theory in these particular fields are almost certain to remain few and far between in comparison to general scientific applications. There is no longer a danger of clutter from a wide-ranging discussion, since I intend to answer it and close it myself, by delving into the rather clear reasons for this strange dichotomy. I hate to see the question go to waste and remain forever closed and unanswered, especially since a clear answer is at hand, one I went to great trouble to find. I made an effort to contact the closers first, as mentioned in the Meta checklist on question reopening, but couldn’t find any direct contact info; I even started a chatroom the other day, but have been unable to invite them there to talk to them directly.

Let me know if there’s anything further I can do to get the thread reopened just briefly enough for me to close it myself. If necessary, I can go into more detail here about how certain aspects of chaos theory constrict its applicability, but I’d rather save the bulk of the detail for my answer. If necessary, I suppose I could even submit my answer to moderators first, if that sort of thing’s allowed. I’d also like to retain the numbered list of potential applications in the original post precisely because they’re germane to the answer, which won’t be overtly obvious till I’ve posted it. This thread may help other users like myself who initially had a “the sky’s the limit!” mentality about the applications of chaos theory to these fields understand more quickly just why they’re likely to remain relatively rare, in contrast to other scientific uses. Thanks.

$\endgroup$
  • 1
    $\begingroup$ If I understand you correctly would it not be possible to post your new question with a link back to the old one and then post your answer there? $\endgroup$ – mdewey Jan 12 at 17:46
  • $\begingroup$ @mdewey - So you would like me to start a new post with the same question, answer it, then link back to the original post? Can a moderator comment on whether or not that would be allowed (it seems like it would circumvent the whole StackExchange closing functionality by creating a duplicate)? The easiest thing to do would be for a moderator or voters to simply reopen the original question, so I can answer it and close it myself by accepting the answer. That's what the reopen-request tag on this Meta post is designed to set in motion. $\endgroup$ – SQLServerSteve Jan 13 at 15:43
  • $\begingroup$ I meant to say the reopen-closed tag - then "reopen-request" tag is used at other StackExchange sites for the same purposes, apparently. Thanks for the suggestion. $\endgroup$ – SQLServerSteve Jan 13 at 15:49
  • 1
    $\begingroup$ If I understood correctly your researches mean that your new question would be different from the original one, although related obviously. By linking back to the original closed one you would be able to state why it is not a duplicate. $\endgroup$ – mdewey Jan 13 at 16:23
  • $\begingroup$ @mdewey Actually I've already edited the original question at the link above to sharply reduce the scope, so there's no need to post a different question. It's the same question, just not as broad in the range of potential answers. All I need is to get the original one reopened, either with enough reopen votes by users or by moderator intervention, just long enough for me to post my detailed answer, accept it and close it myself. Thanks $\endgroup$ – SQLServerSteve Jan 13 at 16:37
  • $\begingroup$ I've got 2 reopening votes on the original thread + 4 upvotes on this Meta post. I think it takes 5 votes to reopen a closed question, so if I can get 3 of the upvoters to cast a reopening vote, the issue will be resolved. I'll then write up my answer as quickly as possible, accept the answer and close the question. Thanks. $\endgroup$ – SQLServerSteve Jan 14 at 23:41
  • $\begingroup$ I can only speak for myself. Nevertheless I would be surprised if my position was unique. I won't vote for re-opening because this isn't my field and the proposition seems to be that you want approval for posting an answer when we can't see what it is. That doesn't make your motives, still less your answer, suspect; it just means that I won't vote for a candidate I can't assess. Further, this thread seems pointless: as @mdewey has clearly hinted, you can just post a new question and a new answer on the main site and then people can vote, add further answers or comment exactly as they please. $\endgroup$ – Nick Cox Jan 15 at 14:30
  • $\begingroup$ A counter-proposal is just to delete the original. I can't see that there could be any objection to re-using some of the text in a new thread so long as all the other criteria of CV are met. $\endgroup$ – Nick Cox Jan 15 at 15:05
  • $\begingroup$ @Nick Cox - I can post my answer ahead of time if you'd like for review, or run it by a moderator if you'd like (if that's permissible). The only reason I haven't so far is that it's likely to take significant effort write & I want to make sure I'm not wasting my time. I'd rather reopen the original thread, in part because it got a lot of upvotes & favorites; the people who responded to it will be looking for an answer there. Besides, deleting & reposting circumvents the whole StackExchange reopening process & isn't the preferred mechanism. Thanks for the suggestion though. $\endgroup$ – SQLServerSteve Jan 15 at 15:27
  • $\begingroup$ There is no pre-publication review here by moderators or anyone else. I can't help on whether your answer will be well received for simple but fundamental reasons already mentioned: it's not in my field and I can't begin to imagine what it would be like. I usually care a lot about following documented procedures, but they don't seem to be helping you here. The 4 upvotes here aren't matched by the votes to re-open on the main site. The only comments I can see at all are from @mdewey and I agree with everything he has said. $\endgroup$ – Nick Cox Jan 15 at 15:34
  • $\begingroup$ The preferred procedure in the Meta post "How do you reopen a closed question?" I cited above Includes steps like a) rewriting the question to make it less broad and b) creating a reopening request post like this one in the appropriate Meta forum. Deleting the original and reposting it are not among the listed options; I'm just following the script the powers that be at StackExchange have established. $\endgroup$ – SQLServerSteve Jan 15 at 15:35
  • $\begingroup$ @Nick Cox Thanks for the suggestions. I'll see if more time can garner more reopen votes, or if a moderator reopens it; I only posted my suggestion for the 3 more reopening votes last night. It may be awhile before some of the original upvoters see this post, or the rewrite, or the comments. I'll stick to the StackExchange procedure for now if possible. Thanks $\endgroup$ – SQLServerSteve Jan 15 at 15:40
  • 1
    $\begingroup$ Indeed; that's not encouraged for rather mundane and sordid reasons. Newbies with reputations of order 1 or 10 sometimes just post another version of the same question (or the same question yet again) if they get a disappointing response. That doesn't match this case. The Catch-22 here is that if you wait for 5 votes to re-open you are likely to wait a long time. Most active people act promptly or not at all. $\endgroup$ – Nick Cox Jan 15 at 15:40
  • 1
    $\begingroup$ About as much effort has gone into this Meta post and comments as would go into a typical answer on the main site. You really don't have to be so diffident. Worst scenario: you lose 80 reputation on the main site if the original question is deleted. If your new post is any good, you will claw most or all of that back. (I lost about 1000 points once on SO when someone who likes my stuff got fed up with the site and withdrew, and their upvotes disappeared....) $\endgroup$ – Nick Cox Jan 15 at 16:11
  • 2
    $\begingroup$ I appreciate all that, but I don't think you'd lose many privileges if your reputation dropped by 80. Don't get me wrong: I much prefer my reputation to any smaller number, but once you can comment, vote and suggest edits you've made the biggest steps. My own case (I'm certainly not a statistician) suggests that going for low-hanging fruit can be a way to build up reputation. $\endgroup$ – Nick Cox Jan 15 at 16:49
7
$\begingroup$

Minimal changes to the question

In this meta post you suggest re-opening the question because now you have an answer to it. But to re-open questions you should change the question itself; closing a question is about the question (and re-opening should also be about the question, not about the fact that you now have an answer).

Now you changed your question a lot, but mostly by providing an explanation about your proposed answer. You have only minimally changed your question (although it is on an essential point).

"What are the known, existing practical applications of chaos theory in data mining?"

(The emphasis—by me—shows the change in your question.)

You are right that in this way the question has become less vague/broad (because 'unknown applications' is very open-ended). But it is still of a type that asks for a list, which is not so common here. It is not that it cannot be done, but it does make the question more difficult. For example, what do you mean by "applications": how is that defined? Do you look for a list of books, a list of articles, a list of techniques, etc.? What kind of list are you looking for? Is it something like an example 'what are some known...', or something that is exhaustive 'what are the known' (cf., How can requests for references be "too broad"?)?

Unclear question

With the above in mind, it does not help much that your question is a bit hidden in a large amount of text. Your question is written in the title but does not occur anywhere else in the body text. It is verbose, including lots of additional (but superfluous) text such as:

"In the two years since my original post..."

or

"While casually reading some mass market works..."

That makes the intention of your question confusing. What kind of practical applications are you looking for? Why do you look for such a list?

It should be possible to write that up in just a few words (say like an article abstract) and then many more people will be able to vote on your question.

Proposal

How about reframing the question as "Why are there so few existing practical applications of chaos theory in data mining?" (possibly posting as a new question, but if that means you delete the old then I would say you should better edit the old question). Then you can make a simple and brief question (the current question is very long-winded) which can contain your list of "very rare existing practical applications". You should be able to write this question in a few minutes with the work that you have already done.

Then you can spend a week on the answer while people open your question (and who knows possibly also answer it).


What I would remove:

While casually reading some mass market works on chaos theory over the last few years I began to wonder how various aspects of it could be applied to data mining and related fields, like neural nets, pattern recognition, uncertainty management, etc.

To date, I've run into so few examples of such applications in the published research that I wonder if a) they've actually been put into practice in known, published experiments and projects and b) if not, why are they used so little in these interrelated fields?

Most of the discussions of chaos theory I've seen to date revolve around scientific applications that are entirely useful, but have little to do with data mining and related fields like pattern recognition; one of the archetypical examples is the Three-Body Problem from physics. I want to forego discussion of ordinary scientific applications of this kind and restrict the question solely to those applications which are obviously relevant to data mining and related fields, which seem to be few and far between in the literature. The list of potential applications below can be used as a starting point of a search for published research, but I'm only interested in those applications that have actually been put into practice, if any. What I'm looking for are known implementations of chaos theory to data mining, in contradistinction to the list of potential applications, which is much broader. Here's a small sampling of off-the-cuff ideas for data mining applications that occurred to me while reading; perhaps none of them are pragmatic, perhaps some are being put to practical use as we speak, but go by terms that I'm not yet familiar with:

  • Identifying self-similar structures in pattern recognition, as Mandelbrot did in a practical way in the case of error bursts in analog telephone lines a few decades ago.
  • Encountering Feigenbaum's Constant in mining results (perhaps in a manner similar to how string theorists were startled to see Maxwell's Equations pop up in unexpected places in the course of their research).
  • Identifying the optimal bit depth for neural net weights and various mining tests. I wondered about this one because of the vanishingly small numerical scales at which sensitivity to initial conditions comes into play, which are partially responsible for the unpredictability of chaos-related functions.
  • Using the notion of fractional dimensions in other ways not necessarily related to fascinating fractal curiosities, like Menger Sponges, Koch Curves or Sierpinski Carpets are. Perhaps the concept can be applied to the dimensions of mining models in some beneficial way, by treating them as fractional?
  • Deriving power laws like the ones that come into play in fractals.
  • Since the functions encountered in fractals are nonlinear, I wonder if there's some practical application to nonlinear regression.
  • Chaos theory has some tangential (and sometimes overstated) relationships to entropy, so I wonder if there's some way to calculate Shannon's Entropy (or limits upon it and its relatives) from the functions used in chaos theory, or vice versa.
  • Identifying period-doubling behavior in data.
  • Identifying the optimal structure for a neural net by intelligently selecting ones that are most likely to "self-organize" in a useful way.
  • Chaos and fractals etc. are also tangentially related to computational complexity, so I wonder if complexity could be used to identify chaotic structures, or vice-versa.
  • I first heard of the Lyapunov exponent in terms of chaos theory and have noticed it a few times since then in recipes for specific neural nets and discussions of entropy.

There are probably dozens of other relationships I haven't listed here; all of this came off the top of my head. I'm not narrowly interested in specific answers to these particular speculations, but am just throwing them out there as examples of the type of applications that might exist in the wild. I'd like to see replies that have examples of current research and existing implementations of ideas like this, as long the applications are specifically applicable to data mining.

In the two years since my original post, I've done more research into the issue and at this point, I plan on answering this question myself, if I can get it reopened; I will discuss some case studies and explain why there is such a marked tension between the potential applications vs. actual implementations of chaos theory to data mining. There are probably other extant implementations I’m not aware of, even in areas I'm more familiar with (like information theory, fuzzy sets and neural nets) and others I those I have even less competence in, like regression, so more input is welcome. If no other answers are forthcoming though, my current plan is to draw the question to a productive conclusion by accepting my own answer, which will touch on why it is necessary to cast a wide net across this range of potential applications (precisely because the ones that have been implemented are so few and far between). My practical purpose here is to determine whether or not to invest more in learning about particular aspects of chaos theory, which I'll put on the back burner if I can't find some obvious utility. I’ve recently found that there are some instructive, deeply structural reasons for this unexpected dichotomy between the seemingly wide scope of potential uses for chaos theory in data mining vs. the narrow list of practical implementations to date. This disparity in scope is inseparable from the question itself and is a logical byproduct of the nature of chaos theory.

I did a search of CrossValidated but didn't see any topics that directly address the utilitarian applications of chaos theory to data mining etc. The closest I could come was the thread Chaos theory, equation-free modeling and non-parametric statistics, which deals with a specific subset.

Into:

I began to wonder how various aspects of chaos theory could be applied to data mining and related fields, like neural nets, pattern recognition, uncertainty management, etc.

To date, I've run into so few examples of such applications in the published research that I wonder if

  • a) they've actually been put into practice in known, published experiments and projects?
  • b) if not, why are they used so little in these interrelated fields?

Most of the discussions of chaos theory I've seen to date revolve around scientific applications that are entirely useful, but have little to do with data mining and related fields like pattern recognition. I want to forego discussion of ordinary scientific applications of this kind and restrict the question to those applications which are relevant to data mining and related fields.

The list of potential applications below can be used as a starting point of a search for published research, but I'm only interested in those applications that have actually been put into practice, if any. Perhaps none of them are pragmatic, perhaps some are being put to practical use as we speak, but go by terms that I'm not yet familiar with:

  • Identifying self-similar structures in pattern recognition, as Mandelbrot did in a practical way in the case of error bursts in analog telephone lines a few decades ago.
  • Encountering Feigenbaum's Constant in mining results (perhaps in a manner similar to how string theorists were startled to see Maxwell's Equations pop up in unexpected places in the course of their research).
  • Identifying the optimal bit depth for neural net weights and various mining tests. I wondered about this one because of the vanishingly small numerical scales at which sensitivity to initial conditions comes into play, which are partially responsible for the unpredictability of chaos-related functions.
  • Using the notion of fractional dimensions in other ways not necessarily related to fascinating fractal curiosities, like Menger Sponges, Koch Curves or Sierpinski Carpets are. Perhaps the concept can be applied to the dimensions of mining models in some beneficial way, by treating them as fractional?
  • Deriving power laws like the ones that come into play in fractals.
  • Since the functions encountered in fractals are nonlinear, I wonder if there's some practical application to nonlinear regression.
  • Chaos theory has some tangential (and sometimes overstated) relationships to entropy, so I wonder if there's some way to calculate Shannon's Entropy (or limits upon it and its relatives) from the functions used in chaos theory, or vice versa.
  • Identifying period-doubling behavior in data.
  • Identifying the optimal structure for a neural net by intelligently selecting ones that are most likely to "self-organize" in a useful way.
  • Chaos and fractals etc. are also tangentially related to computational complexity, so I wonder if complexity could be used to identify chaotic structures, or vice-versa.
  • I first heard of the Lyapunov exponent in terms of chaos theory and have noticed it a few times since then in recipes for specific neural nets and discussions of entropy.

I'd like to see replies that have examples of current research and existing implementations of ideas like this, as long the applications are specifically applicable to data mining.

$\endgroup$
  • $\begingroup$ Thanks for the response, I will think this through. The reason I added the seemingly superfluous text was to illustrate what had changed in the interim between the original post and my edit. I could eliminate some of that, but the post will still be relatively long in terms of text, because it's a complicated issue that requires a lot of context. $\endgroup$ – SQLServerSteve Jan 15 at 17:06
  • $\begingroup$ My initial impression is that this would result in a thread that is even more top-heavy in the question, but with a shorter answer. I'd still have to enumerate the existing implementations in these fields, demonstrate that the list is a lot shorter than that of ordinary scientific applications, talk about why this is surprising in light of the widespread perception of a list of potential applications, etc. The difference is that all of this would have to be moved from the answer to the question & be more awkward. At present, the answer and the question would be about the same length. $\endgroup$ – SQLServerSteve Jan 15 at 17:10
  • $\begingroup$ Questions and answers with explicit edits are often difficult to read. A question/answer is eventually the best when it stands on it's own without additional history how you came to the answer/question. If you like then you might refer with a small text to the edit-history and/or this meta-question (it is Q&A, not a forum or chatbox, see for another example of separating discussion/history also Wikipedia which refers to discussion seperately). $\endgroup$ – Martijn Weterings Jan 15 at 17:15
  • $\begingroup$ How about the question in a single phrase and then a list (which I imagine, based on you texts, is small) as an example. So two sentences and a list, and you are done. $\endgroup$ – Martijn Weterings Jan 15 at 17:21
  • $\begingroup$ @martjin Weterings I think you're right - this was my first attempt at a significant question edit on StackExchange and it probably led to unnecessary verbosity. Give me a few days to mull over yours and Nick's suggestions and see if I can think of a means of reorganizing the thread, without going to the other extreme of a top-heavy question as I mused about above. $\endgroup$ – SQLServerSteve Jan 15 at 17:22
  • $\begingroup$ I'd have to go longer than two sentences and a list, because I still have to explain why there's a perception of wide-ranging applicability to data mining, etc. - which still entails some kind of list of potential applications like the one in the original post. Let me chew on this for a few days. Thanks $\endgroup$ – SQLServerSteve Jan 15 at 17:25
  • $\begingroup$ There's no way of getting out of providing a lot of context, which means several parargraphs of text at minimum; I've already cut out equations, function definitions, etc. But let me see if I can think of a more optimal way of going about it, in light of your suggestions. $\endgroup$ – SQLServerSteve Jan 15 at 17:28
  • $\begingroup$ Additional note to my first comment, at first I did not read your full question and did not come to the point where you say "In the two years since my original post, I've done..." $\endgroup$ – Martijn Weterings Jan 15 at 17:29
  • $\begingroup$ You are possibly making the question(ing) more complicated than it should be. You can always ask a simple question with a brief explanation/motivation, and make it more extended later. $\endgroup$ – Martijn Weterings Jan 15 at 17:37
  • $\begingroup$ One thing I could do is to follow your advice and create a new question, but move the list of "potential applications" in my original post to the answer and explain why this set of uses cases is only tangentially related to chaos theory. if I don't mention them at all, others will bring them up - but they're really a separate class from the main use, which is ruling out chaotic effects in function approximation. Let me see if I can think of more workarounds like this. I'll get back to you in a few days. Thanks. $\endgroup$ – SQLServerSteve Jan 15 at 17:47
  • $\begingroup$ List of potential applications, set of use cases, those types of lists/sets have fuzzy borders which makes the question too broad here. But is it about the list? I guess you have something else with it in mind. Make the question about that and then care less about list which, if you really like, can add to your question or answer (by list I imagine just a list of 10 links with keywords or short sentences, or are you planning to fully explain each of them?). $\endgroup$ – Martijn Weterings Jan 16 at 7:51
  • $\begingroup$ @Martjin Weterings I am awaiting feedback in chat from a moderator on whether or not mdewey & Nick's option of deleting/reposting is permissible. It's not my preferred option anyways. In the meantime I'm crafting another rewrite in lines with the Q/A format you suggested here & should have it ready by early next week. I'll get back to you then. If the feedback on the format is positive, I'll accept your answer here and substitute the new rewrite. Thanks for your constructive suggestions. $\endgroup$ – SQLServerSteve Jan 17 at 14:42
  • $\begingroup$ " It's not my preferred option anyways" then why waiting for some moderator's approval for that option? (btw, I am not sure that the moderators here are waiting to make such decisions and like being placed in the position of some sort of higher court judge. You have already posted this meta-question here to let the jury decide. I do not think that people will be upset whatever you do regarding 'deleting-reposting versus editing' since you have tried your best to get a clear answer, and it turns out that it is not so clearcut.)..... $\endgroup$ – Martijn Weterings Jan 17 at 15:12
  • $\begingroup$ @Martjin That's a good point. But if the moderators say that the deletion/reposting thing isn't an option, the people who suggested it might instead back your option, the one I prefer. I've seen a few Meta threads about how to contact mods about rule clarification so I thought it was the way to go. I'm moving forward with the rewrite in the meantime while I clear up that loose end, if possible. If the mods don't reply, I'll just press forwards on your option. $\endgroup$ – SQLServerSteve Jan 17 at 15:46
  • 3
    $\begingroup$ @SQLServerSteve Brevity is the soul of wit.—William Shakespear $\endgroup$ – Alexis Jan 22 at 2:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .