tag:blogger.com,1999:blog-6894866515532737257.comments2017-07-27T07:46:01.067-07:00Probably Overthinking ItAllen Downeyhttps://plus.google.com/111942648516576371054noreply@blogger.comBlogger716125tag:blogger.com,1999:blog-6894866515532737257.post-4334673450263988762017-07-24T06:30:49.135-07:002017-07-24T06:30:49.135-07:00So, I am wondering about the likelihood that what ...So, I am wondering about the likelihood that what you're seeing in those data amounts to, "people without a religion (in America) used to all say they were Protestant, but now some of them say they have no religion." Trying to think of a statistical way of testing this theory, and I'm not coming up with much...rosshttps://www.blogger.com/profile/02587634589065610863noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-11204383529430514502017-07-20T14:24:23.839-07:002017-07-20T14:24:23.839-07:00Thanks for the comment. I hear this sentiment fro...Thanks for the comment. I hear this sentiment from students and readers quite often. Although I understand why people feel this way, and I sometimes feel the same discomfort when I use other people's libraries, I have given it a lot of thought, and tested alternatives, and I have come to the conclusion that using my libraries is better for educational purposes, even though it makes people uncomfortable. The primary reason is cognitive load: we can only handle so much new stuff at a time. Providing my own library allows me to hide details when they are not needed, so readers can focus on what's relevant, and then I can choose the best time to open the hood and reveal more details. <br /><br />If someone wants more details before I provide them, they always have the option of reading the source. Most of my libraries are thin wrappers around functions from NumPy, SciPy, etc. So they are very readable (and generally well documented and commented, I think).<br /><br />This might be a longer reply than you wanted, but I wanted to take the chance to explain my thinking.<br /><br />Thanks for the comment!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-25420341463085369702017-07-20T12:51:04.874-07:002017-07-20T12:51:04.874-07:00Hey Allen :) Just reading your book, it's real...Hey Allen :) Just reading your book, it's really valuable, thank you so much! I only wish you use and explain raw pandas, matplotlib, etc code instead of your wrappers. It would be more verbose, but I think, more beneficial for education purpose.imbolchttps://www.blogger.com/profile/17354029544468760955noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-86806129231215500222017-06-27T04:35:14.315-07:002017-06-27T04:35:14.315-07:00P(heads|awake) has several possible values. It is ...P(heads|awake) has several possible values. It is not unique unless assumptions are made that are not explicitly stated in the problem. <br /><br />A rational Sleeping Beauty can compute the fair price X for the bet "Bet owner gets $1 if the coin landed heads and must pay the purchase price twice if the coin landed tails". (Sleeping beauty must set one price X for betting on heads that applies on both Monday and Tuesday because she cannot distinguish one day from the other) The answer for the "fair price" is X = 1/3 and this is calculated independently of the controversial P(heads| awake). It only requires using the unconditional probability P(heads) = 1/2.<br /><br />The answer from the betting strategy computation does not have the interpretation of being Sleeping Beauty's estimate of P(heads|awake). She doesn't use any estimate of P(heads|awake) to calculate X = 1/3. And she is not making the "pure" bet "Bet owner gets $1 if the coin landed heads". The actual bet she is offered has more consequences.<br /><br />There are various plausible assumptions that lead to the "thirder" answer. They involve somehow connecting the expected frequencies for the events (heads, Monday, awake), (tails, Monday, awake), (tails, Tails, Tuesday, awake) as generated by the experiment to the probabilities that those situations are the single situation that happens "when Sleeping Beauty awakes". However, there is no information in the problem that explicitly makes such a connection.<br /><br />The "halfer" answer for P(Heads | awake) is not unique, but it satisfies the given information. It does not contradict that the best betting strategy is X = 1/3 because a rational "halfer" would calculate X in the same manner as indicated above.<br /><br />A specific model for the probability distribution of the situation "when Sleeping Beauty is awakened" that is consistent with the "halfer" answer is:<br />1) Toss the coin and run the experiment. 2) From the sitation(s) that arise the experiment, pick a situation, giving each situation the same probability of being selected if we must pick from among two.<br /><br />That model produces conditional probabilities that offend somes people's intuition, but it does not mathematically contradict any information given in the problem. The "halfer" answer in not the unique answer for P(heads | awake) because there is also a "thirder" probability model that is consistent with the information given in the problem.<br /><br />It's tempting to think that the Sleeping Beauty problem is equivalent to a typical balls-in-urns problem. For example, Urn H contains 1 amber-colored ball and one sienna-colored ball. Urn T contains 2 amber colored balls. A fair coin is flipped. Urn H is chosen if the coin lands heads, otherwise urn T is chosen. A ball is drawn at random from the chosen urn. <br /><br />Question 1) Given the ball that is drawn is amber-colored, what is the probability that Urn H was chosen? <br /><br />However that does not exemplify the scenario in the Sleeping Beauty problem. By analogy to the Sleeping Beauty problem , all the balls are drawn from the selected urn. Then the ill-posed question is asked:<br /><br />Question 2: Upon observing a draw where the ball is amber colored, what is the probability that Urn H was selected.<br /><br />Question 2 does not say that each of the two balls that are drawn have the same probability of being the one observed. It doesn't specify that the draw observed is the first draw or the second draw. It doesn't rule out that the observer has some bias -like always choosing to observe only the first draw from urn T. <br /><br />Since P(heads | awake) has no objectively calculable value, one may resort to making assumptions by invoking The Principle of indifference. If we believe that the Principle of Indifference cannot be paradoxical then all solutions computed that way should be the same.Bobo Cheverovskyhttps://www.blogger.com/profile/17774595689219106458noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-76186844286320870002017-06-19T07:26:34.113-07:002017-06-19T07:26:34.113-07:00Thank you for your kind words!Thank you for your kind words!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-55261796629716828572017-06-18T17:38:07.453-07:002017-06-18T17:38:07.453-07:00Professor Downey,
I just want to write a message...Professor Downey, <br /><br />I just want to write a message of appreciation for you and your work. By studying your books on my own, I went from working a pretty crappy job chopping carrots in a kitchen to working a great job as a data engineer at a startup. I now feel like my opportunities are endless.<br /><br />You're books are amazing. They are accessible, yet not-watered-down. They are fun and pertinent. By doing the example problems, and by building my own projects using your libraries, I not only learned a ton about object oriented programming in python and statistics, I was inspired by things like "the relationship between fractals and pink noise" (which I read about in ThinkComplexity and which led me to read ThinkDSP) and Bayesian decision science, which I never would have been exposed otherwise. <br /><br />These books are an incredible gift to the world. Thank you.Unknownhttps://www.blogger.com/profile/00970185295593391556noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-23813965685977427182017-06-04T12:38:05.487-07:002017-06-04T12:38:05.487-07:00It might be a good idea to consider other types of...It might be a good idea to consider other types of statistical analysis used in the field. For example, survival analysis, to model the time until a delinquent returns to prison.Julio Vegahttps://www.blogger.com/profile/16878262091789831073noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-83810021807900091262017-06-04T10:26:45.327-07:002017-06-04T10:26:45.327-07:00You'll want to add the Journal of Experimental...You'll want to add the Journal of Experimental Marine Biology and Ecology, which very explicitly says, "All papers should all be written in third person, passive voice." Having perused their journal, though, I noticed that not all of their articles even do that!Mecholskyhttps://www.blogger.com/profile/05183078822188895919noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-76794178087946521662017-06-02T22:27:17.513-07:002017-06-02T22:27:17.513-07:00Hi Allen,
Thank you for posting this problem. It ...Hi Allen,<br /><br />Thank you for posting this problem. It seems very interesting. I have a quick question: why do we compute the probability of YYY? This information is already given in the problem ("All 3 friends tell you that "Yes" it is raining."), so P{YYY} should be 1?<br /><br />Thank you!Dina Jankovichttps://www.blogger.com/profile/10621469573786218341noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-17921425988010242252017-05-27T12:08:04.398-07:002017-05-27T12:08:04.398-07:00Can we flip the question? Instead of calculating ...Can we flip the question? Instead of calculating the probability of how likely Smith as a member of the reference group is to re-offend, I'm interested in the probability of characteristics A, B, and C producing an individual who re-offends. For example, low income, low education, and low employment as the characteristics.<br /><br />Philosophically, this is a useful question for policy makers trying to undo mass incarceration. What do we need to change in the socio-economic conditions of a person on probation or parole in order to lower his/her risk of recidivism?<br /><br />It doesn't appear that the dataset you have can answer this question. But I note that the objections you note above do not necessarily apply to it--and excluding race from the analysis is easily justified (i.e. it's an immutable trait, so why bother?)Joshua K.https://www.blogger.com/profile/06068902191362983004noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-48934009109016427692017-05-01T08:24:18.395-07:002017-05-01T08:24:18.395-07:00Interesting article, but there are two things I no...Interesting article, but there are two things I noted which might want to be looked into - a) the growth of cohabiting (common law marriage) couples who are effectively married in all but name and b) growing numbers of people opting-out of finding relationships, particularly with women I found when looking at UK Office of National Statistics data on living arrangements and filtering out population change.Garethhttps://www.blogger.com/profile/00112674655504156048noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-33939553486119102182017-04-04T10:58:50.167-07:002017-04-04T10:58:50.167-07:00Hi Riya,
Not sure if you are still checking this b...Hi Riya,<br />Not sure if you are still checking this blog, but the computational error in the answer of the MIT course is just confined to the labels of two branches (test1, disease absent). They switched the labels with the probs of test1 detecting or not detecting the disease. However, the final probs at the leafs of their table are correct (they assume proper labels in their computations). And once those are correct, then the answer is in fact identical to 5/13 (as they report). <br /><br />Long story short, the results of the MIT course and the results presented here are identical, they are both correct, and the only mistake is the wrong label on two branches. <br /><br />And finally, I punched these numbers in the Netica program (not that it's really needed beyond the worksheet), and also get the correct 5/13 answer. Unfortunately I don't know how to share Netica worksheets in blog post, otherwise I would post it...Felix Thoemmeshttps://www.blogger.com/profile/02139944750643002756noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-6617413490974662692017-02-18T17:26:27.722-08:002017-02-18T17:26:27.722-08:00I have a question though. I mentioned that this qu...I have a question though. I mentioned that this question was taken from the MIT 2006 OCW final exam for computer science. The solution was also available on the internet. Even though they have mentioned the answer to be 5/13 I.e. 10/26 but there's some calculation error in their method. The correct answer according to their method is 25/63. But I really don't understand why they did the question that way. I'm adding the links to both the question and the answer here. It's the fourth question. <br /><a href="https://www.google.co.in/url?sa=t&source=web&rct=j&url=https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-042j-mathematics-for-computer-science-fall-2010/exams/MIT6_042JF10_fnl_2006_sol.pdf&ved=0ahUKEwiWmOT3-5rSAhVCpo8KHbR6DbMQFgglMAA&usg=AFQjCNGh_rIRn7AyI4WO9Paj9m-iq_riUw&sig2=ZbyrZqKgd6YJaSjmAEWhIg" rel="nofollow">answer</a><br /><a href="https://www.google.co.in/url?sa=t&source=web&rct=j&url=https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-042j-mathematics-for-computer-science-fall-2010/exams/MIT6_042JF10_final_2006.pdf&ved=0ahUKEwiWmOT3-5rSAhVCpo8KHbR6DbMQFggoMAE&usg=AFQjCNGfuqKhxH7H6Wpbt4nB9DjAWN4JvA&sig2=GmFaDfgVkq0T3cb3O3BeuA" rel="nofollow">question</a><br />Riyahttps://www.blogger.com/profile/05095295993427070799noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-12877653335172783042017-02-18T10:34:12.984-08:002017-02-18T10:34:12.984-08:00This comment has been removed by the author.Riyahttps://www.blogger.com/profile/05095295993427070799noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-78346079788040356192017-02-17T12:24:31.688-08:002017-02-17T12:24:31.688-08:00I have corrected that error. Thanks!I have corrected that error. Thanks!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-14487104130235535242017-02-17T10:13:13.953-08:002017-02-17T10:13:13.953-08:00I enjoyed the problem, but I got a different answe...I enjoyed the problem, but I got a different answer. I thought the prior for TEST1 was 2/3.David Bodyhttps://www.blogger.com/profile/09987602296796651688noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-20559111740998241692017-02-17T07:26:52.253-08:002017-02-17T07:26:52.253-08:00Thanks for letting me know about the source of the...Thanks for letting me know about the source of the problem. If you have a link, I'll include it in the article.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-69936903023640887172017-02-17T07:16:54.521-08:002017-02-17T07:16:54.521-08:00Hey thanks Mr. Allen for taking up the question.I ...Hey thanks Mr. Allen for taking up the question.I would just like to mention that I came across this question from the MIT OCW final exam for computer science.<br />I have thought of two approaches and of course they're giving different answers, although I'm a little biased towards one of the approaches. I would be really glad if I could finally understand what's happening in this question. Riyahttps://www.blogger.com/profile/05095295993427070799noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-23560589514582152542017-02-16T07:39:09.813-08:002017-02-16T07:39:09.813-08:00Excellent question! I just turned it into a blog p...Excellent question! I just turned it into a blog post. I'll give readers a few days before I post a solution: http://allendowney.blogspot.com/2017/02/a-nice-bayes-theorem-problem-medical.htmlAllen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-46894475359124415992017-02-16T04:53:17.056-08:002017-02-16T04:53:17.056-08:00I have a question. Exactly 1/5th of the people in ...I have a question. Exactly 1/5th of the people in a town have Beaver Fever . There are two tests for Beaver Fever, TEST1 and TEST2. When a person goes to a doctor to test for Beaver Fever, with probability 2/3 the doctor conducts TEST1 on him and with probability 1/3 the doctor conducts TEST2 on him. When TEST1 is done on a person, the outcome is as follows: If the person has the disease, the result is positive with probability 3/4. If the person does not have the disease, the result is positive with probability 1/4. When TEST2 is done on a person, the outcome is as follows: If the person has the disease, the result is positive with probability 1. If the person does not have the disease, the result is positive with probability 1/2. A person is picked uniformly at random from the town and is sent to a doctor to test for Beaver Fever. The result comes out positive. What is the probability that the person has the disease?Riyahttps://www.blogger.com/profile/05095295993427070799noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-32237289467857960462017-02-12T16:05:30.273-08:002017-02-12T16:05:30.273-08:00Interesting! For a recent working paper on the eff...Interesting! For a recent working paper on the effect of Photo ID laws on turnout, see here [1]. They find an average suppression of 7.7% among Democrats and 4.6% among Republicans. <br /><br />[1] Voter Identification Laws and the Suppression of Minority Votes<br />http://pages.ucsd.edu/~zhajnal/page5/documents/voterIDhajnaletal.pdf<br /><br />Abstract:<br /><br />The proliferation of increasingly strict voter identification laws around the country has raised concerns about voter suppression. Although there are many reasons to suspect that these laws could harm groups like racial minorities and the poor, existing studies have been limited, with most occurring before states enacted strict identification requirements, and they have uncovered few effects. By using validated voting data from the Cooperative Congressional Election Study for several recent elections, we are able to offer a more definitive test. The analysis shows that strict identification laws have a differentially negative impact on the turnout of racial and ethnic minorities in primaries and general elections. We also find that voter ID laws skew democracy toward those on the political right.Anonhttps://www.blogger.com/profile/15513912296075560498noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-56785656768293897892017-01-18T21:13:56.190-08:002017-01-18T21:13:56.190-08:00Why is it that when calculating the variance in Ka...Why is it that when calculating the variance in Kaplan Meier survival curves, the underlying distribution of the population is not taken into account--only the number at risk is? This is counter-intuitive. The only references are to a report by Greenwood in 1926 which doesn't really seem to answer the question.Mark Phillipshttps://www.blogger.com/profile/03120283972507328308noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-47616136231365182042017-01-18T10:46:50.416-08:002017-01-18T10:46:50.416-08:00There was disagreement among statisticians about h...There was disagreement among statisticians about how to do hypothesis testing; what we ended up with is actually a weird hybrid that makes less sense than the alternatives. See https://en.wikipedia.org/wiki/Statistical_hypothesis_testing#Origins_and_early_controversy<br /><br />Regarding precision and recall, those are the terms most often used in the context of information retrieval. In the context of stats, it's usually false positives and false negatives (or type I and type II errors). And in the context of machine learning, it's usually a confusion matrix. But they are all representations of the same information.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-45725377773039094592017-01-17T15:02:17.176-08:002017-01-17T15:02:17.176-08:00I think there was some other place where I found a...I think there was some other place where I found an explanation that there are two approaches to hypothesis validation. One used p-values. The other one was something like precision and recall (false positives and negatives) or something similar. I think it mentioned some disagreement between statisticians before p-values became popular. May this be possible?trylks yeahhttps://www.blogger.com/profile/14821108955806430210noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-84018391878133304622017-01-05T09:34:34.881-08:002017-01-05T09:34:34.881-08:00Great examples. Congratulations! Great examples. Congratulations! Hugo Pireshttps://www.blogger.com/profile/00400200472463525909noreply@blogger.com