"Help I have some data and need to analyse it what do I dooo?"

Question

I have been on this stackexchange long enough now to see there is quite a lot of questions along the lines of:

Help, I'm a data science intern/amateur statistician and I've been given a load of data from my boss/professor and told to look for insights/trends, and I've spent half a day browsing wikipedia/stackexchange and now i'm really confused. What do I do?"

General features of these questions:

• The asker is a first time user with 1 rep point.
• The asker has been tasked with this by someone high up who has noticed the buzz surrounding Machine Learning and thinks this is definitely something their company ought to be doing, but doesn't know how to go about it.
• The asker is not a professional statistician and previously had no idea how many different techniques there were.

As "Machine learning" as a corporate buzzword continues to gain widespread use and as neural networks and other AI techniques continue to produce miraculous and highly visible results for big tech companies, we will continue to see an ever-increasing influx of these questions, no matter how punitively we downvote or close them.

As such, I think there should be at least one example of a well-worded question of this nature that has a good answer that we can point to, involving some general rules of thumb about how to do data analysis - stuff about where to start, checking the data is clean, graphing it to get a sense of what's going on, a couple of basic pitfalls, etc.

After that, a series of links on specific topics, or where to go to get help. The trickiest part to answer is the dreaded "what projects should I try with this data?". This part can probably be answered only anecdotally, with specific examples/links of worthwhile DS undertakings at other companies, along with general platitudes about trying to understand the company's needs and difficulties.

Downvoting and closing as "too broad" without giving any useful feedback to the asker will not curtail the influx of these sorts of questions. It merely discourages the new user from asking questions here in future.

There is a specific question I have in mind that I want to have a go at answering, but I can't because it is put on hold. I am happy to have a look for a good "What do I do?" question and develop a generic answer that will hopefully be useful for future searchers.

My general point is, these questions will be asked whether we want them to or not, so we ought to at least make some minimally helpful generic answer that we can use to point them in the right direction.

Answer 1

I'm sorry, but I don't think this is a useful approach.

The questions you are referring to are essentially "please teach me data science". There are just far too many possible flavors. If you set up a general question and answer that future questions of this type can be directed to, this "generic" Q&A will be either a full book-length treatment, or so vague as to be meaningless:

• "Plot your data" - yes, but what should I plot if I have a credit scoring task? Or am trying to forecast a million time series? Or want to cluster stuff in 37 dimensions?
• "Clean your data" - how do I clean a time series? Categorical data with two levels? Categorical data with 42 levels? Should I remove? Impute?
• "Distinguish between statistical significance, importance and predictive power" - what is this "statistical significance" thing I keep hearing about? Should I do stepwise variable selection? What quality KPI should I be using for my classification task? For my forecasts, should I use mape or mae or rmse?

There is a reason why Chemistry.SE does not have a "generic" Q&A on "How do I do a chemical analysis?"

And everything in-between "a full 1000-page tome" and "so vague to be meaningless" has been covered about 1e9 times in MOOCs, tutorials, blog post series and, yes, textbooks that are somewhat shorter than 1000 pages.

So, sorry, I will continue to VTC such questions as "too broad".

Answer 2

The endless variations on "How do I analyze these data?" are not answerable questions. Full stop.

When I first started my career, I worked as a consultant specializing in data science. I didn't know much about math and machine learning, but I learned. (I was a "unicorn" that had some statistics background and some subject matter knowledge which made me qualified for a specialized role.) Even though I was picking up programming and math skills which were critical to my job, the absolutely most important lesson that I ever learned on that job was to ask this question, early and often:

What problem are we trying to solve?

The answer to that question will provide immediate and obvious direction. The answer to that question is also why it's impossible to produce a catch-all thread to answer "How do I analyze these data?"-type questions. "How do I analyze these data?" is only a question in the strictly grammatical sense. It's not a statistical question, it's not a data science question, it's not a machine learning question. It's a category error masquerading as a technical issue.

But note also that "What problem are you trying to solve?" also isn't a statistics question. It's a business question, for a broad definition of business. We can't answer these questions, either. But consultants can!

Answer 3

Here is another possible approach.

I still stand by my opinion that any answer to "I have a dataset, please help me analyze it" will either be so vague as to be meaningless, or require a book-length treatment.

Happily, this does not need to be the end of the story. Because such book-length treatments exist! For instance, our How best to use the review queue? thread contains boilerplate that I added as something that should be helpful for many forecasting questions:

This question is very broad, and I believe you would profit
from reading an introductory level textbook, e.g., the free online
[*Forecasting: Principles and Practice* by
Hyndman & Athanasopoulos](https://otexts.org/fpp2/).
If after reading this you still have more specific questions, then
please do ask them here. If you already *have* read such a textbook,
please edit your question to make it more specific. Thank you!

So: if anyone knows of a good textbook that would be helpful for this kind of question, we could direct people there. Actually, we could use the existing Free statistical textbooks thread to collect (more) books.

And then close questions like this one not necessarily as "too broad", but as duplicates of Free statistical textbooks. Or close them as "too broad" and insert a link to that thread, e.g.:

This question is very broad, and I believe you would profit from 
reading an introductory level textbook. [We have a helpful list of
free statistical
textbooks.](https://stats.stackexchange.com/q/170/1352) If
afterwards you still have more specific questions, then please do ask
them here. If you already *have* read such a textbook, please edit 
your question to make it more specific. Thank you!

When the question is about neural networks, we can supply a more specific set of references.

This question is very broad, and I believe you would profit from 
reading an introductory level textbook. [We have a helpful list of 
textbooks and courses about neural networks.] 
(https://stats.stackexchange.com/questions/226911/neural-network- 
references-textbooks-online-courses-for-beginners) If
afterwards you still have more specific questions, then please do ask
them here. If you already *have* read such a textbook, please edit 
your question to make it more specific. Thank you!