tag:blogger.com,1999:blog-6894866515532737257.comments2016-02-02T05:54:36.639-08:00Probably Overthinking ItAllen Downeyhttps://plus.google.com/111942648516576371054noreply@blogger.comBlogger584125tag:blogger.com,1999:blog-6894866515532737257.post-38629862845048001002016-01-19T00:14:10.670-08:002016-01-19T00:14:10.670-08:00Makes me wonder how the clearly coincidental corre...Makes me wonder how the clearly coincidental correlations mentioned were found. If I gather all of the data I can find and am able to pull some spurious relationships out, what hypothesis am I testing? Did I do many, many tests to get a hit? How relevant is the demonstrable existence of pure coincidence to the interpretation of a well designed experiment?<br /><br />Sometimes I feel like people know just enough statistics to be a little afraid of it, so they take the hard line textbook interpretation.<br /><br />Thanks for the post, and the response. I was not aware of the opium/Everest correlation. Definitely going to use that in my science literacy class. This stuff boggles my mind.EhADH2http://www.blogger.com/profile/07401717907931197796noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-73295091982788572782016-01-02T16:03:10.159-08:002016-01-02T16:03:10.159-08:00It would be beneficial for those of us trying to i...It would be beneficial for those of us trying to intuit this if, instead of "Plug in Bayes's theorem" we got a little clearer elaboration. Thanks though.Unknownhttp://www.blogger.com/profile/14067952676942833778noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-25165398853614104232015-12-23T03:45:19.520-08:002015-12-23T03:45:19.520-08:00Good post you shared here i hope you will write mo...Good post you shared here i hope you will write more.Mark Dawkinshttp://www.blogger.com/profile/03049412878821578827noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-13777433760043308842015-12-22T04:05:53.930-08:002015-12-22T04:05:53.930-08:00With regard to question 4, although the question d...With regard to question 4, although the question does not ask for posterior probability, I am attempting<br />to solve it using the posterior probability approach to see which proposition has higher odd of <br />finding the criminal.<br /><br />I define 3 propositions which I think are exclusive and exhaustive with regards to this problem,<br />A: Criminal1 is Oliver. Criminal2 is other.<br />B: Criminal1 is other. Criminal2 is Oliver.<br />C: Criminal1 is other. Criminal2 is other.<br /><br />I then attempt to solve it using thinkbayes.Suite approach.<br /><br />def bloodTraceProblem():<br /> '''<br /> Two people have left traces of their own blood at the scene of a crime. <br /> <br /> A suspect, Oliver, is tested and found to have type O blood. <br /> <br /> The blood groups of the two traces are found to be of type O <br /> (a common type in the local population, having frequency 60%) <br /> and of type AB (a rare type, with frequency 1%). <br /> <br /> Do these data (the blood types found at the scene) give evidence <br /> in favour of the proposition that Oliver was one of <br /> the two people whose blood was found at the scene?<br /> <br /> We define 3 propositions,<br /> A: Criminal1 is Oliver. Criminal2 is other.<br /> B: Criminal1 is other. Criminal2 is Oliver.<br /> C: Criminal1 is other. Criminal2 is other.<br /> <br /> Data is the blood trace found at the scene<br /><br /> ''' <br /><br /> class BloodTrace(thinkbayes.Suite):<br /> <br /> def Likelihood(self, data, hypo):<br /> blood_trace = data<br /> <br /> if (hypo == 'A'):<br /> if (blood_trace == "O"):<br /> return 1 <br /> elif (blood_trace == 'AB'):<br /> return 0.01<br /> else:<br /> return (1 - 0.6 - 0.01)<br /> elif (hypo == 'B'):<br /> if (blood_trace == 'O'):<br /> return 1<br /> elif (blood_trace == 'AB'):<br /> return 0.01<br /> else:<br /> return (1 - 0.6 - 0.01)<br /> else: # hypothesis C<br /> if (blood_trace == "O"):<br /> return 0.6<br /> elif (blood_trace == 'AB'):<br /> return 0.01<br /> else:<br /> return (1 - 0.6 - 0.01)<br /> <br /> blood_trace = BloodTrace("ABC")<br /> blood_trace.Update('O')<br /> blood_trace.Update('AB')<br /><br /> blood_trace.Print()<br /> <br />The output is <br />A 0.384615384615<br />B 0.384615384615<br />C 0.230769230769<br /><br />The output is suggesting that we have better odd of solving the crime by assuming that Oliver is one of the criminal and to focus<br />all effort to find the 'AB' guy. This approach narrows the search to 1% of population rather than 60% of the population. The decision may be <br />different if we believe that there is dependency between criminal 1 and criminal 2 i.e. finding either one will increase the <br />odd of finding the other because they probably know each other.<br /><br />I will be glad to hear from you if there is any areas that I may have overlooked in my solution.AIK KIANG HENGhttp://www.blogger.com/profile/16662487046759383884noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-63561843926024689272015-12-21T01:55:23.520-08:002015-12-21T01:55:23.520-08:00I must say that's impressive post.I must say that's impressive post. Mark Dawkinshttp://www.blogger.com/profile/03049412878821578827noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-44360892513327499552015-12-14T16:15:35.122-08:002015-12-14T16:15:35.122-08:00Regarding p-values, I'd recommend reading Cosm...Regarding p-values, I'd recommend reading Cosma Shalizi's blog post here: http://bactra.org/weblog/1111.html<br /><br />If I'm interpreting him correctly, he would disagree with you that you can say much about H given an adequately small p-value. However, he would also say that this implies that the p-value is a very limited measure since, among other reasons, p-values tend to shrink exponentially fast as the sample size grows. Ted Fujimotohttp://www.blogger.com/profile/04377265481902807970noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-74520205678160723412015-12-11T08:31:00.226-08:002015-12-11T08:31:00.226-08:00Hello Allen,
Long-time reader, first-time poster. ...Hello Allen,<br />Long-time reader, first-time poster. <br /><br />A couple of comments:<br />1) Correlation / causation. Even though this may sound pedantic, I think semantics make a difference here. The word "imply" is often used in the sense of "logically implying". When used in this sense, it is in fact true that the the belief that correlation implies causation is the logical fallacy of affirming the consequent. That being said, if you talk about "evidence" (and not "implication") of correlation in favor of causation you are correct. I don't know how most people typically interpret "imply" - in the logical sense, or in the sense of shifting evidence. Maybe you like this quote from XKCD: Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing "look over there."<br /><br />2) Regression, matching, (and weighting) all have the same underlying causal assumptions, namely "ignorability" (sometimes also called "selection on observables" or "unconfoundedness"). In fact, Angrist and Pischke in their book "Mostly harmless econometrics" formally prove that all three estimators are in the same class. You can re-express regression as a particular weighting scheme, and you can do the same with matching. I am not sure how widespread the belief is that you cite - if you were to go to a conference like Atlantic Causal Inference Conference, I would think that all participants would know that there is nothing magical about matching, and that all these methods share the same underlying assumption. There are practical advantages and disadvantages to all these methods though. <br /><br />3) Regarding reversing regressions - this is not a theoretically sound way to determine causal direction (or provide evidence for one or the other regression direction). Judea Pearl proved I think in the 80s that the models that you are suggesting are all in the same Markov equivalence class, and that parameters yielded by those models cannot be used to distinguish which one might be the true causal model. Apologies for the self-promotion but my paper on reversing arrows in mediation models also shows this point. That being said, if you are willing to make certain untestable assumptions about distributions of disturbance terms, you can use methods that you suggest (reversing regressions) to determine causal direction. The work of Bernhard Schoelkopf is important in this domain. Unfortunately, the assumptions needed to make these methods work will by definition always be untestable, and thus be subject to debate. <br /><br />All the best,<br />FelixFelix Thoemmeshttp://www.blogger.com/profile/02139944750643002756noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-13091621369497358542015-12-10T07:49:49.272-08:002015-12-10T07:49:49.272-08:00Alex, I think we are agreeing. If all you know is...Alex, I think we are agreeing. If all you know is the p-value, the conclusions you can reach about H are pretty weak, and qualitative, even with my additional assumptions. That's why I say that traditional NHST is mostly useless.<br /><br />But if the p-value is small, you can usually conclude that the observed effect is probably not due to random sampling, and you can turn your attention to other possible sources of error.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-73284483541353960172015-12-10T07:38:14.378-08:002015-12-10T07:38:14.378-08:00I don't object to making additional assumption...I don't object to making additional assumptions so that you can say something about H. If you want to compare hypotheses, then yes go ahead and compare them (with Bayes Factor or likelihood ratios, for god's sakes, something!). <br /><br />You can 'say' or decide what you want, but what is the number that backs up such a statement? I would argue not the p-value. If there are tools that do exactly what you want (to compare H0 to H, for example), why not use them?<br /><br />Using the p-value does not stand up to even basic additional scrutiny. For example, you get a p < .05 and you say H is 'more likely'. Then someone (a reviewer, a skeptic, an interested friend) asks, well how much more likely? 1.2 times, 20 times? How do you respond to that? Seems vital to me. Alex Perronehttp://www.blogger.com/profile/07994443036194382882noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-74082226208484844402015-12-10T07:32:56.684-08:002015-12-10T07:32:56.684-08:00For rlgblg I used logistic regression, so the inte...For rlgblg I used logistic regression, so the intercept is in log odds. For someone born in 1960, with 12 years of education, income 5 (on a 10 point scale), with minimal media use and no access to the Internet, the value is close to 1, which corresponds to a probability near 50%.<br /><br />For rlgdgr, the same hypothetical person would be expected to report 5.7 on a 10 point scale.<br /><br />But don't take that too seriously, because it's an unweighted average across different countries, and it's at the extreme end of some independent variables.<br /><br />At this point I'm just warming up the models.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-25418074848110844852015-12-10T07:23:00.811-08:002015-12-10T07:23:00.811-08:00It sounds like you are agreeing with the rule that...It sounds like you are agreeing with the rule that a small p-value does not allow you to say anything at all about H, and therefore that hypothesis testing is completely useless.<br /><br />I don't love NHST, but I am a slightly bigger fan than you. In this previous article, I explain why:<br /><br />http://allendowney.blogspot.com/2015/05/hypothesis-testing-is-only-mostly.html<br /><br />It's true that you have to make some additional assumptions in order to say anything about H, and it sounds like you object to that.<br /><br />But the assumptions are very weak, and nearly always true in practice. So if you are trying to do something practical, like guide decision-making under uncertainty, why would you not accept reasonable assumptions? Especially when the alternative is to provide no guidance whatsover?Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-85168274114043668182015-12-10T07:10:29.227-08:002015-12-10T07:10:29.227-08:00Intercept is the value of the dependent variable t...Intercept is the value of the dependent variable that the model gives, when the independent variable are zero.Freddyhttp://www.blogger.com/profile/11634775055143782437noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-79742919794453484662015-12-10T07:07:21.873-08:002015-12-10T07:07:21.873-08:00"Assuming that they are more likely under H (..."Assuming that they are more likely under H (which is almost always the case), you can conclude that the data are evidence in favor of H and against H0." <br /><br />So you don't accept a rule because you choose to assume something else? Not much of an argument. And also unsubstantiated, as there are plenty of examples to the contrary. I'd say generally, merely a significant p-value in social science research where data are noisy, analysis is likely p-hacked or a garden of forking paths, etc. provides fairly weak evidence against H0; I strongly disagree that you can comment directly on H from the p-value without lifting a finger on modeling H. Please see Wagenmakers 2007 p. 792-793, where p = 0.05 can even indicate that H0 is likely to be true, or for another example Nickerson 2000, p. 249-251. There are many criticisms out there. I'd also highly recommend Schmidt and Hunter paper below as general overview. <br /><br />The main thing is that you are wanting a p-value to be some kind of likelihood ratio or Bayes Factor, which it is not. A p-value is completely one-sided and only concerns the probability of the data under H0, not even the probability of H0. Overall, you are disagreeing with the interpretation of p-values by mis-interpreting them even more than the mess that brings about the current reproducibility crisis, Ionnadis' "most published research is false", etc. <br /><br />Ioannidis, J. P. (2005). Why most published research findings are false. Chance, 18(4), 40-47. Accessed at: <br />http://robotics.cs.tamu.edu/RSS2015NegativeResults/pmed.0020124.pdf<br /><br />Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychological methods, 5(2), 241.<br />Accessed at: http://psych.colorado.edu/~willcutt/pdfs/Nickerson_2000.pdf<br /><br />Schmidt, F. L., & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of signiﬁcance testing in the analysis of research data. What if there were no signiﬁcance tests, 37-64.<br />Accessed at: http://www.phil.vt.edu/dmayo/personal_website/Schmidt_Hunter_Eight_Common_But_False_Objections.pdf<br /><br />Wagenmakers, E. J. (2007). A practical solution to the pervasive problems ofp values. Psychonomic bulletin & review, 14(5), 779-804. Accessed at:<br />http://www.ejwagenmakers.com/2007/pValueProblems.pdf<br />Alex Perronehttp://www.blogger.com/profile/07994443036194382882noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-45594059976587916162015-12-10T04:13:45.320-08:002015-12-10T04:13:45.320-08:00"Quoting rules is not an argument." Exce..."Quoting rules is not an argument." Excellent point (especially re. the correlation/causation maxim you mention earlier. This speaks to a larger trend in citing uncertainty as a means of rejecting any and all evidence out there. Great post.Me (Mara)http://www.blogger.com/profile/06750044263111951360noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-62775660818006783322015-12-09T05:50:58.299-08:002015-12-09T05:50:58.299-08:00Yes, that would probably be a good idea. I think ...Yes, that would probably be a good idea. I think I am getting the same effect by perturbing one variable at a time and standardizing the effect size. But it might have been simpler to transform all the variables at the beginning.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-78389332385354490712015-12-09T05:46:58.850-08:002015-12-09T05:46:58.850-08:00Good question. I did that partly to deal with dat...Good question. I did that partly to deal with data issues: there are some suspiciously large values in eduyrs, partly to standardize the variables (which makes it easier to interpret the estimated parameters) and partly in order to make these variables relative to other respondents from the same country, to avoid comparisons across countries with different levels of income and education.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-49149893649517815272015-12-09T04:23:45.276-08:002015-12-09T04:23:45.276-08:00Very nice analysis indeed Allen! I would only sugg...Very nice analysis indeed Allen! I would only suggest you to scale and normalise all predictors and the two outcomes you're studying. In particular I would use a technique I've learned from Gelman & Hill (2006) of subtracting the mean from each observation and then dividing by two times the standard deviation, so that a 1-unit change in the rescaled predictor corresponds to a change from 1 standard deviation below the mean, to 1 standard deviation above. This is to maintain coherence when considering binary input variables.iamgianlucahttp://simplyanalyticsblog.com/noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-63013818912451212372015-12-09T04:13:02.601-08:002015-12-09T04:13:02.601-08:00Nice blog Allen! Why did you replace each value of...Nice blog Allen! Why did you replace each value of `eduyrs` and `hinctnta` with its rank (from 0-1)? Is this a particular technique used when controlling variables in a regression model?iamgianlucahttp://simplyanalyticsblog.com/noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-9139990187088675242015-12-08T11:52:58.578-08:002015-12-08T11:52:58.578-08:00You are right, I should have included "just p...You are right, I should have included "just plain coincidence" on the list of explanations. Thanks for the comment and the link.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-27305986890188792882015-12-08T11:27:23.875-08:002015-12-08T11:27:23.875-08:00I bet the rules are quoted and interpreted in such...I bet the rules are quoted and interpreted in such an extreme way because people who have learned a little statistics are feeling smug about it, because there really is a naive tendency to make mistakes that the rules are designed to point out.<br /><br />There is a funny site that you have probably seen where Tyler Vigen shows strong correlation that is coincidence. You could argue that the correlation is evidence of causation, but that would require a definition of evidence a bit more weak than I think most people would assume.<br /><br />My favorite is the correlation between the production of opium in Afghanistan and a picture of Mount Everest.<br /><br />https://twitter.com/tylervigen/status/603204482856591360<br /><br />In this case, none of the relationships you mentioned are likely to obtain. "A might cause B, B might cause A, or any number of other factors, C, might cause both A and B." Instead, in this case C caused A and D caused B, but they still look similar on a plot.Ed Cashinhttp://www.blogger.com/profile/04078366179657821736noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-15658791872492715502015-12-06T17:26:34.087-08:002015-12-06T17:26:34.087-08:00I'm still learning too (studied mathematical l...I'm still learning too (studied mathematical logic in a past academic life). The King and Nielson paper is pretty new as well. <br /><br />Also, any future posts on causal inference would be appreciated! Don't worry too much about the redditers. People have different opinions about causality. It's a difficult subject but still worth investigating. In particular, it would be great to see the scientific python community do more work in this area. Lots of stuff out there (Pearl, Rubin, van der Laan, etc.), we just need to pythonize it! Ted Fujimotohttp://www.blogger.com/profile/04377265481902807970noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-87619410390163272982015-12-04T07:36:54.546-08:002015-12-04T07:36:54.546-08:00Thanks for the links. Interesting to hear about t...Thanks for the links. Interesting to hear about the problems with propensity score matching. I still have lots to learn.<br /><br />But I probably won't implement much more. I need to wrap up this project and move on!Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-33126599100661495472015-12-03T14:38:25.730-08:002015-12-03T14:38:25.730-08:00Great article!
Incidentally I just found this to...Great article! <br /><br />Incidentally I just found this today: http://gking.harvard.edu/files/gking/files/psnot.pdf?m=1439838506<br /><br />https://www.youtube.com/watch?v=rBv39pK1iEs<br /><br />Maybe another matching method in the next post?<br /><br />:/Ted Fujimotohttp://www.blogger.com/profile/04377265481902807970noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-26757839815763295132015-12-01T10:55:08.708-08:002015-12-01T10:55:08.708-08:00Very interesting. Thanks for that link. The samp...Very interesting. Thanks for that link. The sampling bias described in that article is an example of the inspection paradox I wrote about here: <br /><br />http://allendowney.blogspot.com/2015/08/the-inspection-paradox-is-everywhere.html<br /><br />In particular, someone who serves x different prison sentences is x times more likely to be selected using the standard methodology, so the results tend to overestimate the rate of recidivism. This paper explains why that's misleading:<br /><br />http://cad.sagepub.com/content/early/2014/09/26/0011128714549655.abstract<br /><br />and the authors present a methodology that corrects the bias.<br /><br />As for how that applies to the data I worked with: I'm afraid I don't know. It depends on how the sample was selected, and I don't have that info. It is likely that it used the standard methodology, which is to select a cohort of prisoners released during a time interval, and follow them for a number of year.<br /><br />But for the intended use case (making parole decisions), that methodology is actually correct, because the sample corresponds to the relevant population. The sample overrepresents people with multiple convictions, but so does the population of parole candidates.<br />Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-81575575530829694432015-11-30T17:22:39.307-08:002015-11-30T17:22:39.307-08:00In the context of your analysis above, I was wonde...In the context of your analysis above, I was wondering what you thought of the paper referenced in this article:<br />http://www.slate.com/articles/news_and_politics/crime/2015/10/why_do_so_many_prisoners_end_up_back_in_prison_a_new_study_says_maybe_they.html<br /><br />It's fascinating that such a major cultural belief (sky-high recidivism rates) could be based on a relatively simple statistical error. This also reminds me of the Unseen Species analysis you did in Think Bayes, and I wonder how/if that analysis takes into account sampling John Frieshttp://www.blogger.com/profile/08506666431936417165noreply@blogger.com