21
$\begingroup$

I have this hypothesis. Ceteris paribus the answers that have formulas (equations) in them get more votes than plain English answers. Do you agree?

$\endgroup$
  • 1
    $\begingroup$ Not sure - checking they're right takes longer so you might just not vote. $\endgroup$ – Scortchi - Reinstate Monica Dec 21 '17 at 14:53
  • 3
    $\begingroup$ In my opinion, $\Pr(\text{you are right}) > 0.5$, but I may be wrong. Please upvote if you agree ;) Seriously: you probably can query data.stackexchange.com in a smart way that recognizes $\TeX$ (e.g. looks for $...$) and verify such hypothesis using the data. $\endgroup$ – Tim Dec 21 '17 at 15:06
  • 1
    $\begingroup$ @Tim, I was hoping some did it already :) The trick is to find the correct answers to the same question but with and without formula. That is difficult because how do we know the answers are equivalent in other regards? $\endgroup$ – Aksakal Dec 21 '17 at 15:07
  • 8
    $\begingroup$ Well, that is actually the problem, since many answers that use formatting (a) may simply be higher quality (e.g. you know $\TeX$ since you use it in academia), (b) may show greater effort in answering, so the effort was +1'd no matter that the answer was not that "outstanding". $\endgroup$ – Tim Dec 21 '17 at 15:17
  • 8
    $\begingroup$ It looks to me like you want to answer a causal question (about the effect of adding a formula on the resulting number of votes) based on observational data (what seems to have happened organically on the site to date). There are a lot of issues here, which are discussed heavily in various threads on the main site. You could do a small experiment for yourself by composing answers, then flipping a coin and adding a formula or not. $\endgroup$ – gung - Reinstate Monica Dec 21 '17 at 16:39
  • $\begingroup$ @gung, I thought about an experiment. It's not that easy to conduct either, because we'd need to separate the identity of answer's impact, so an accomplice is necessary. $\endgroup$ – Aksakal Dec 21 '17 at 16:42
  • 4
    $\begingroup$ The identity of the answerer would be controlled by virtue of the fact that you are the only answerer in the study. One might wonder how well it would generalize, though. $\endgroup$ – gung - Reinstate Monica Dec 21 '17 at 17:27
  • 2
    $\begingroup$ An additional idea, aside your proposed "self administered A/B test", is to look for duplicate questions that have not been closed. That should allow for some pseudo-replication in the one thread contains formulas and the other does not. (Afterwards though you would have to flag the threads for merging as a good CV citizen). (I think Tim's hits the nail in the head; the presence of $\LaTeX$ is strongly associated with other features that also affect a post's overall votes.) $\endgroup$ – usεr11852 says Reinstate Monic Dec 21 '17 at 23:51
  • 3
    $\begingroup$ The idea put forward by @gung is the only practicable one. There are too many confounding factors otherwise. They include the topic of the question, how well the question was formulated, the mathematical level at which it was formulated, how often the thread has been viewed, the number of locations of links to it, how much attention it received initially and over time, the reputation of the person who answers, how effectively they use mathematical language, when the answer was posted, who else was posting answers around the same time, and whether there are also illustrations in the answer. $\endgroup$ – whuber Dec 28 '17 at 20:01
  • $\begingroup$ Related: stats.meta.stackexchange.com/questions/1611/…. (/cc @MartijnWeterings). $\endgroup$ – Andre Silva Dec 29 '17 at 15:19
25
$\begingroup$

In short: I think that the bottom-line is that in terms of correlations, there is not a clear effect. It differs per user, and if we would scale the number of equations by the size of the post then actually the post scores become lower for more equations.

To see if there are causal effects one might still do some alternative experiments, but the experimental correlations below suggest that the patterns are small, with lot's of noise and confounding variables.


It is easy to gather the data:

-- Enter Query Title
-- Enter Query Description

DECLARE @UserId int = ##UserId##

SELECT
   Id as [Post Link],
   Score,
   Len(Body) As CharacterNum,
   round((Len(Body)-len(replace(Body, '$$', '')))/4,1) As center_FormulaNum,
   round((len(replace(Body, '$$', ''))-len(replace(Body, '$', '')))/2,1) As total_FormulaNum
FROM
   posts
WHERE 
   OwnerUserId = @UserId
   and posttypeid = 2 
   group by Id, Score,  Len(Body),
   round((Len(Body)-len(replace(Body, '$$', '')))/4,1),
   round((len(replace(Body, '$$', ''))-len(replace(Body, '$', '')))/2,1)
ORDER BY Score desc;

But it is difficult to make something out of it.

If the number of equations in a post is high (counted by $$), then there is somewhat an increase of the scores of the posts. See the density function of the scores below. With four different selections, 0 equations, 1 equations, 2 equations, >2 equations. (Figure 1)

But more equations also correlate with more text, and if we scale the number of equations per post by the number of characters per post, then there is much less difference or sometimes we see lower scores for posts with (relatively) more equations. (Figure 2)

These correlations are a lot different from user to user. Maybe this may be a lead to think about causal relationships (although I suspect it is widely varying).

Fig1 : score density functions categorized by number of equations per post score density functions

Fig2 : score density functions categorized by number of equations per post divided by characters per post score density functions

------------------------------------------------------------------------------------------------------------

Because the above picture are small and noisy, I have created another, different view.

This time no split into 4 categories (making the numbers per category small, and instead a split into either functions with and without equations made by double $$). Also, the densities are not each by themselves scaled to 1 but instead are together scaled to 1. In this way also the relative difference can be seen how often a post has equations and how often not.

Behold, the Whuber-Amoeba-Polarization-Effect:

It is interesting to see that the post with high scores are fifty-fifty with and without equations. But at lower scores Whuber makes relatively more posts with equations, and Amoeba makes relatively more posts without equations. Of course it is just guessing what could be the cause of this effect, but the correlation is certainly interesting.

the whuber amoeba polarization

------------------------------------------------------------------------------------------------------------

Another plot inspired by EngrStudent's comment:

I think the difference between with/without after a score of 20 for Amoeba is interesting.

The above histogram/distribution for Whuber and Amoeba is now plotted for the nine members, and as a cumulative score on a log scale on the y-axis and an inverted square scale on the x-axis.

The differences above 20 were not so well visible on the previous plot and it looked much the same. This new plot suggest some type of exponential distribution or some hyper-exponential distribution.

The differences between 'with' and 'without' equations seem to be in

  1. the rate parameter(s) of the components (exponential terms) in these distributions
  2. the relative ratio that a member writes answers with and without equations

While using these correlations still leaves us guessing for interpretations in terms of causal effects, I believe that it is very likely that the majority of the differences between answers with and without equations is due to the correlation with confounding variables and that it is more like the topic of the answer/question is influencing the scores, instead of the 'fact that there is an equation in the post'. I believe that the widely varying distributions and answering styles of different users supports this, and we may rightfully wonder about the common adagio that 'adding an equation to a book/presentation is going to reduce the attention'.

cumulative distribution

small note: given the variations in the slopes of these cumulative distributions, I guess that the age of the posts may be important, and improvements of these plots can be made when we also take the age of the post into account (maybe I get back to that some time).

$\endgroup$
  • 1
    $\begingroup$ I think the difference between with/without after a score of 20 for Amoeba is interesting. $\endgroup$ – EngrStudent - Reinstate Monica Dec 29 '17 at 0:10
  • 1
    $\begingroup$ @EngrStudent i have added a new plot which allows to observe this difference a bit better. I think it is a though problem since the distributions seem to be a mixture of multiple exponential distributions. Maybe some other type of, flexible, distribution could be fitted to it and aggregates the sum of multiple effects... But I believe that, given the nature of many different types of posts and topics, a model mixing multiple curves would be the more accurate underlying model. $\endgroup$ – Sextus Empiricus Dec 29 '17 at 11:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .