tag:blogger.com,1999:blog-6894866515532737257.comments2017-11-23T05:23:19.678-08:00Probably Overthinking ItAllen Downeyhttps://plus.google.com/111942648516576371054noreply@blogger.comBlogger729125tag:blogger.com,1999:blog-6894866515532737257.post-91141384713586942022017-11-20T07:13:02.308-08:002017-11-20T07:13:02.308-08:00I wonder if you have seen stuff by Peter Turchin o...I wonder if you have seen stuff by Peter Turchin of U Conn on 'cliodynamics' (some posted on evonomics blog). I wonder how these social trends are related (Turchin is predicting an uptick in political violence for awhile). mediahttps://www.blogger.com/profile/16362083339472635217noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-22010083889966409672017-11-17T09:28:23.626-08:002017-11-17T09:28:23.626-08:00I wonder if you are still following and interested...I wonder if you are still following and interested in this thread. <br /><br />I was trying to map the example into other examples---eg you have a city of people and take a sample of 1, 2, and then 3 and test them for drugs . the drug test is only 2/3rds accurate ( if they have drugs in their system then 2/3rds of the time the test says Yes they do, and 1/3rd No they don't . Suimialrily if they dont have drugs in them, then 2/3rds of the time they come out 'negative', and 1/3rd positive. <br /><br />You can consider cases like 'maybe i dont know the size of the city--is it 1, 2, 3 or a million people'? To apply Bayes theorem you might need a 'prior' like what % of people in that city are on drugs. <br /><br />I'm really interested in a 'recursion' so you can go from 1 sample response (in your example, one friend), to 2, to 3, etc. <br /><br />I actually think of this in terms of the ising model and spin glasses of statistical physics. <br /><br />Also in term s of Venn diagrams--in you case the big circle is Seattle, and its cut in 2 by a line saying on one side of the circle (CITY) its raining and other side its not. Then you sample somone in the city but you dont know if they are on the rainy or sunny side, and also you only know they are telling the truth 2/3rds of the time when you ask them if its raining. (or you could ask them if they are on drugs, or did the crime.) <br /><br /> m impression is bayesianism reduced to frequentism if P(Rain) is unknown (just 50/50). <br /> also if you ask 3 people do you use (1/3rd +1/3rd +1/3rd)/3 or (1/3)^3 (times probabilities) . i think those are 2 different slightly different assumptions---one independent random variables (multiply) versus additive --which reduces to same thing sometimes. <br /> <br />i just view bayes theorem or rule as a version of the definition of conditional probability. <br /><br /> mediahttps://www.blogger.com/profile/16362083339472635217noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-12103913207608888652017-10-24T18:46:12.993-07:002017-10-24T18:46:12.993-07:00Except the data does not support the thesis that r...Except the data does not support the thesis that religion, in general, is in decline. If I were to show the first two figures with the lines relabeled as Group A, B, C, and D then most people would come to the conclusion that only group A (protestants) are declining while shifting into group C (nones). This would indicate that Protestantism, and not religion in general, is driving the rise of the nones.<br /><br />The thesis that religion in general is in decline is only weakly supported by the data. The only thing the data clearly tells us is that Protestantism is in decline. The same trend can be seen in population data from Scotland, England, and Australia. The percentage of Catholics has remained fairly constant, while the members of the Anglican church or the Church of Scotland have decreased significantly.<br /><br />The story here, as indicated by the data, is that people are leaving Protestantism to join no religion. The question is why are they doing that? If Catholics were experiencing the same decline they should make up 17-18% of the population, and not the 22-23% they do now.Quantumleap42https://www.blogger.com/profile/16711817313734546305noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-47976042402176662162017-10-24T13:21:07.776-07:002017-10-24T13:21:07.776-07:00No Islam?No Islam?Unknownhttps://www.blogger.com/profile/04633643066221822195noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-76012410724399430992017-10-22T19:44:54.747-07:002017-10-22T19:44:54.747-07:00As science progresses, there is less need for supe...As science progresses, there is less need for superstition. Religion, and god in general, are the result of lack of knowledge of how the universe works. We used to worship the sun. Now, that would seem silly. Eric Dorrhttps://www.blogger.com/profile/05263390017117518099noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-50100922639450944412017-10-21T21:49:01.115-07:002017-10-21T21:49:01.115-07:00Fabulous continuation of what we all see in our ev...Fabulous continuation of what we all see in our everyday lives - less and less reliance on mythology. I urge all without religion to speak up! The more we come out of the last big closet, the more influence we can have on protecting our deliberately secular government. Our founding fathers knew the dangers of theocracy, having lived with one in England. Too bad many evangelical Christians today are ignorant of the lesson.Janethttps://www.blogger.com/profile/05638619419040138313noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-15096854461125196742017-10-21T09:04:58.527-07:002017-10-21T09:04:58.527-07:00Although trends are informative, they are not tota...Although trends are informative, they are not totally predictive. If one believes in the Holy Spirit, and the possibility of revival, these trends can be reversed. I personally prefer not to lose heart over statistical trends.William L.https://www.blogger.com/profile/00007956179969941372noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-29829481005736076392017-10-20T20:52:16.346-07:002017-10-20T20:52:16.346-07:00Really appreciate all the number crunching and the...Really appreciate all the number crunching and the graphs. Nicely done.Doris Fromagehttps://www.blogger.com/profile/07804899422584862557noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-55017125869480919352017-10-20T18:18:34.653-07:002017-10-20T18:18:34.653-07:00No surprise. BUT, Nones are not the same as secula...No surprise. BUT, Nones are not the same as secularists, let alone full-blown atheists. Can include New Agers, "spiritual but not religious," and more.Gadflyhttps://www.blogger.com/profile/13075757287807731373noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-61681819996692578552017-10-20T08:39:33.861-07:002017-10-20T08:39:33.861-07:00Very informative. Thank you.Very informative. Thank you.gatorlandkalihttps://www.blogger.com/profile/02583597856687748532noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-65765091100581765582017-09-11T19:59:25.551-07:002017-09-11T19:59:25.551-07:00Iirl. recall when Martin Gardner cited a math expe...Iirl. recall when Martin Gardner cited a math expert who noted that when mathematicians went wrong, it was almost always in probability. <br /><br />Your old roommate planed 2 kids (only) and chose names for every probability: Ben, Joe, Sue, and Floridamissimus (Flo for short). You meet him years later and ask about the family...His response is that he had the 2 kids but he regretted naming one Floridamissimus.<br /><br />Sensing despair, you change the subject to the fact the other child is 2:1 to be a boy. How do you get that he asks? Easy you say, there are 4 ways to have Floridmissimus and a boy but only 2 to have a second girl. Show me he says.<br /><br />Sure: Flo then Sue<br /> Sue then Flo<br /> Flo then Ben<br /> Ben then Flo<br /> Flo then Joe (JDBudd)<br /> Joe then Flo<br />So there are 4 ways to have boy-girl and only 2 to have girl-g Next problem please.Unknownhttps://www.blogger.com/profile/17516065349693480312noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-21899418623358391332017-08-25T06:28:56.329-07:002017-08-25T06:28:56.329-07:00Fixed. Thank you!Fixed. Thank you!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-77575119445249033662017-08-22T23:04:55.491-07:002017-08-22T23:04:55.491-07:00We see P(E|A) = 1, instead it should be P(E|C) = 1...We see P(E|A) = 1, instead it should be P(E|C) = 1Ripunjay Tripathihttps://www.blogger.com/profile/08986459991083183084noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-4334673450263988762017-07-24T06:30:49.135-07:002017-07-24T06:30:49.135-07:00So, I am wondering about the likelihood that what ...So, I am wondering about the likelihood that what you're seeing in those data amounts to, "people without a religion (in America) used to all say they were Protestant, but now some of them say they have no religion." Trying to think of a statistical way of testing this theory, and I'm not coming up with much...rosshttps://www.blogger.com/profile/02587634589065610863noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-11204383529430514502017-07-20T14:24:23.839-07:002017-07-20T14:24:23.839-07:00Thanks for the comment. I hear this sentiment fro...Thanks for the comment. I hear this sentiment from students and readers quite often. Although I understand why people feel this way, and I sometimes feel the same discomfort when I use other people's libraries, I have given it a lot of thought, and tested alternatives, and I have come to the conclusion that using my libraries is better for educational purposes, even though it makes people uncomfortable. The primary reason is cognitive load: we can only handle so much new stuff at a time. Providing my own library allows me to hide details when they are not needed, so readers can focus on what's relevant, and then I can choose the best time to open the hood and reveal more details. <br /><br />If someone wants more details before I provide them, they always have the option of reading the source. Most of my libraries are thin wrappers around functions from NumPy, SciPy, etc. So they are very readable (and generally well documented and commented, I think).<br /><br />This might be a longer reply than you wanted, but I wanted to take the chance to explain my thinking.<br /><br />Thanks for the comment!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-25420341463085369702017-07-20T12:51:04.874-07:002017-07-20T12:51:04.874-07:00Hey Allen :) Just reading your book, it's real...Hey Allen :) Just reading your book, it's really valuable, thank you so much! I only wish you use and explain raw pandas, matplotlib, etc code instead of your wrappers. It would be more verbose, but I think, more beneficial for education purpose.imbolchttps://www.blogger.com/profile/17354029544468760955noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-86806129231215500222017-06-27T04:35:14.315-07:002017-06-27T04:35:14.315-07:00P(heads|awake) has several possible values. It is ...P(heads|awake) has several possible values. It is not unique unless assumptions are made that are not explicitly stated in the problem. <br /><br />A rational Sleeping Beauty can compute the fair price X for the bet "Bet owner gets $1 if the coin landed heads and must pay the purchase price twice if the coin landed tails". (Sleeping beauty must set one price X for betting on heads that applies on both Monday and Tuesday because she cannot distinguish one day from the other) The answer for the "fair price" is X = 1/3 and this is calculated independently of the controversial P(heads| awake). It only requires using the unconditional probability P(heads) = 1/2.<br /><br />The answer from the betting strategy computation does not have the interpretation of being Sleeping Beauty's estimate of P(heads|awake). She doesn't use any estimate of P(heads|awake) to calculate X = 1/3. And she is not making the "pure" bet "Bet owner gets $1 if the coin landed heads". The actual bet she is offered has more consequences.<br /><br />There are various plausible assumptions that lead to the "thirder" answer. They involve somehow connecting the expected frequencies for the events (heads, Monday, awake), (tails, Monday, awake), (tails, Tails, Tuesday, awake) as generated by the experiment to the probabilities that those situations are the single situation that happens "when Sleeping Beauty awakes". However, there is no information in the problem that explicitly makes such a connection.<br /><br />The "halfer" answer for P(Heads | awake) is not unique, but it satisfies the given information. It does not contradict that the best betting strategy is X = 1/3 because a rational "halfer" would calculate X in the same manner as indicated above.<br /><br />A specific model for the probability distribution of the situation "when Sleeping Beauty is awakened" that is consistent with the "halfer" answer is:<br />1) Toss the coin and run the experiment. 2) From the sitation(s) that arise the experiment, pick a situation, giving each situation the same probability of being selected if we must pick from among two.<br /><br />That model produces conditional probabilities that offend somes people's intuition, but it does not mathematically contradict any information given in the problem. The "halfer" answer in not the unique answer for P(heads | awake) because there is also a "thirder" probability model that is consistent with the information given in the problem.<br /><br />It's tempting to think that the Sleeping Beauty problem is equivalent to a typical balls-in-urns problem. For example, Urn H contains 1 amber-colored ball and one sienna-colored ball. Urn T contains 2 amber colored balls. A fair coin is flipped. Urn H is chosen if the coin lands heads, otherwise urn T is chosen. A ball is drawn at random from the chosen urn. <br /><br />Question 1) Given the ball that is drawn is amber-colored, what is the probability that Urn H was chosen? <br /><br />However that does not exemplify the scenario in the Sleeping Beauty problem. By analogy to the Sleeping Beauty problem , all the balls are drawn from the selected urn. Then the ill-posed question is asked:<br /><br />Question 2: Upon observing a draw where the ball is amber colored, what is the probability that Urn H was selected.<br /><br />Question 2 does not say that each of the two balls that are drawn have the same probability of being the one observed. It doesn't specify that the draw observed is the first draw or the second draw. It doesn't rule out that the observer has some bias -like always choosing to observe only the first draw from urn T. <br /><br />Since P(heads | awake) has no objectively calculable value, one may resort to making assumptions by invoking The Principle of indifference. If we believe that the Principle of Indifference cannot be paradoxical then all solutions computed that way should be the same.Bobo Cheverovskyhttps://www.blogger.com/profile/17774595689219106458noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-76186844286320870002017-06-19T07:26:34.113-07:002017-06-19T07:26:34.113-07:00Thank you for your kind words!Thank you for your kind words!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-55261796629716828572017-06-18T17:38:07.453-07:002017-06-18T17:38:07.453-07:00Professor Downey,
I just want to write a message...Professor Downey, <br /><br />I just want to write a message of appreciation for you and your work. By studying your books on my own, I went from working a pretty crappy job chopping carrots in a kitchen to working a great job as a data engineer at a startup. I now feel like my opportunities are endless.<br /><br />You're books are amazing. They are accessible, yet not-watered-down. They are fun and pertinent. By doing the example problems, and by building my own projects using your libraries, I not only learned a ton about object oriented programming in python and statistics, I was inspired by things like "the relationship between fractals and pink noise" (which I read about in ThinkComplexity and which led me to read ThinkDSP) and Bayesian decision science, which I never would have been exposed otherwise. <br /><br />These books are an incredible gift to the world. Thank you.Unknownhttps://www.blogger.com/profile/00970185295593391556noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-23813965685977427182017-06-04T12:38:05.487-07:002017-06-04T12:38:05.487-07:00It might be a good idea to consider other types of...It might be a good idea to consider other types of statistical analysis used in the field. For example, survival analysis, to model the time until a delinquent returns to prison.Julio Vegahttps://www.blogger.com/profile/16878262091789831073noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-83810021807900091262017-06-04T10:26:45.327-07:002017-06-04T10:26:45.327-07:00You'll want to add the Journal of Experimental...You'll want to add the Journal of Experimental Marine Biology and Ecology, which very explicitly says, "All papers should all be written in third person, passive voice." Having perused their journal, though, I noticed that not all of their articles even do that!Mecholskyhttps://www.blogger.com/profile/05183078822188895919noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-76794178087946521662017-06-02T22:27:17.513-07:002017-06-02T22:27:17.513-07:00Hi Allen,
Thank you for posting this problem. It ...Hi Allen,<br /><br />Thank you for posting this problem. It seems very interesting. I have a quick question: why do we compute the probability of YYY? This information is already given in the problem ("All 3 friends tell you that "Yes" it is raining."), so P{YYY} should be 1?<br /><br />Thank you!Dina Jankovichttps://www.blogger.com/profile/10621469573786218341noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-17921425988010242252017-05-27T12:08:04.398-07:002017-05-27T12:08:04.398-07:00Can we flip the question? Instead of calculating ...Can we flip the question? Instead of calculating the probability of how likely Smith as a member of the reference group is to re-offend, I'm interested in the probability of characteristics A, B, and C producing an individual who re-offends. For example, low income, low education, and low employment as the characteristics.<br /><br />Philosophically, this is a useful question for policy makers trying to undo mass incarceration. What do we need to change in the socio-economic conditions of a person on probation or parole in order to lower his/her risk of recidivism?<br /><br />It doesn't appear that the dataset you have can answer this question. But I note that the objections you note above do not necessarily apply to it--and excluding race from the analysis is easily justified (i.e. it's an immutable trait, so why bother?)Joshua K.https://www.blogger.com/profile/06068902191362983004noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-48934009109016427692017-05-01T08:24:18.395-07:002017-05-01T08:24:18.395-07:00Interesting article, but there are two things I no...Interesting article, but there are two things I noted which might want to be looked into - a) the growth of cohabiting (common law marriage) couples who are effectively married in all but name and b) growing numbers of people opting-out of finding relationships, particularly with women I found when looking at UK Office of National Statistics data on living arrangements and filtering out population change.Garethhttps://www.blogger.com/profile/00112674655504156048noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-33939553486119102182017-04-04T10:58:50.167-07:002017-04-04T10:58:50.167-07:00Hi Riya,
Not sure if you are still checking this b...Hi Riya,<br />Not sure if you are still checking this blog, but the computational error in the answer of the MIT course is just confined to the labels of two branches (test1, disease absent). They switched the labels with the probs of test1 detecting or not detecting the disease. However, the final probs at the leafs of their table are correct (they assume proper labels in their computations). And once those are correct, then the answer is in fact identical to 5/13 (as they report). <br /><br />Long story short, the results of the MIT course and the results presented here are identical, they are both correct, and the only mistake is the wrong label on two branches. <br /><br />And finally, I punched these numbers in the Netica program (not that it's really needed beyond the worksheet), and also get the correct 5/13 answer. Unfortunately I don't know how to share Netica worksheets in blog post, otherwise I would post it...Felix Thoemmeshttps://www.blogger.com/profile/02139944750643002756noreply@blogger.com