tag:blogger.com,1999:blog-6894866515532737257.post4433429831348089290..comments2024-05-14T00:10:15.219-07:00Comments on Probably Overthinking It: The sun will probably come out tomorrowAllen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-6894866515532737257.post-15229842004358242552012-03-24T15:45:47.491-07:002012-03-24T15:45:47.491-07:00I will run the numbers with a flat prior over the ...I will run the numbers with a flat prior over the odds. I am also curious to see what kind of difference it makes.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-61182736212227918182012-03-24T15:44:53.963-07:002012-03-24T15:44:53.963-07:00Thank you both for the great comments!Thank you both for the great comments!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-41414726575913182802012-03-23T01:37:01.338-07:002012-03-23T01:37:01.338-07:00You are right about cases of truly complete ignora...You are right about cases of truly complete ignorance - application of the law of succession in such cases would need to be done with great care (we are still free to examine the consequences of such a model, but we need to consider other models also), but the rule was not derived for such cases.<br /><br />You might be right that I misrepresent Boole, I can't claim to have made a detailed historical study, but from my limited reading, it seems as though he had strong objections to the principle under any application (certainly many others did). Thanks for the additional sources.<br /><br />(Of course, Laplace was also opposed to arbitrary application of his principles, and made this very clear in his writing.)<br /><br />Its not that I particular wanted to blame Boole either - others attacked the law of succession on really illogical grounds, claiming instances where it demonstrably provided absurd results. They never stopped to wonder how they knew the results were absurd - if it is clear that the outcome is ridiculous, then it is also clear that your reasoning process is making use or prior information much stronger than the uniform prior for which the method was designed.<br /><br />Thanks again for the sources and the different point of view.Tom Campbell-Rickettshttps://www.blogger.com/profile/07387943617652130729noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-23745224558136442502012-03-23T00:39:07.851-07:002012-03-23T00:39:07.851-07:00"Boole was one of many who argued against Lap..."Boole was one of many who argued against Laplace's reasoning, without understanding it. Boole's distribution is quite appropriate for the case where we have no reason to suppose a constant mechanism governing the outcomes of our experiments."<br /><br />That seems unfair to Boole. The case imagined -- where a precocious newborn observes his first sunrise (or in Hume's language, we imagine Adam with fully formed rational capacity but no previous experience) -- is exactly a case where we have no reason to suppose a constant mechanism governs the outcomes of our experiments.<br /><br />"Sadly, the objections of Boole, Venn, and several others prevailed for a while, and frequentist statistics took hold."<br /><br />That is certainly the way Fisher tells the history. But I think there are good reasons to be skeptical of that history. For more detail on this, I highly recommend <a href="http://www.jstor.org/discover/10.2307/2245634?uid=3739256&uid=2&uid=4&sid=47698794241827" rel="nofollow">this paper</a> by Sandy Zabell. Or read some of Boole's <a href="http://www.jstor.org/stable/10.2307/108830" rel="nofollow">later work</a>, in which he explains that he is not opposed to Laplace's principle but to its arbitrary application.<br /><br />Anyway, I agree with Zabell that it wasn't until the 1930s that Bayesian-Laplacean methods fell out of favor.Jonathan Livengoodhttps://www.blogger.com/profile/08264815112941067048noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-83236492965835764552012-03-22T16:18:06.661-07:002012-03-22T16:18:06.661-07:00Jonathan, thanks, you raise a historically importa...Jonathan, thanks, you raise a historically important issue.<br /><br />Boole was one of many who argued against Laplace's reasoning, without understanding it. Boole's distribution is quite appropriate for the case where we have no reason to suppose a constant mechanism governing the outcomes of our experiments - and surprise, surprise, induction is impossible under these cirumstances. <br /><br />But the law of succession (along with Jaynes' pre-prior that I mentioned above) was derived for cases where we decide that it is reasonable to assume such a constant mechanism, specifically, sequences of Bernoulli trials. <br /><br />Of course, we don't have to assume a constant frequency, in order to perform inductive reasoning. It is enough to recognize that our data result from repetitions of similar experiments - the frequencies we fit can vary according to any model that is indicated by the observations. <br /><br />Sadly, the objections of Boole, Venn, and several others prevailed for a while, and frequentist statistics took hold.Tom Campbell-Rickettshttps://www.blogger.com/profile/07387943617652130729noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-63515067128668718172012-03-22T13:27:07.422-07:002012-03-22T13:27:07.422-07:00Thanks for a great post!
Following up (I think) s...Thanks for a great post!<br /><br />Following up (I think) somewhat on Tom's first comment, Boole argued that the flat prior on frequency of success (or probability p of success) could not be rationally preferred to assigning equal probability to all possible states or constitutions of the universe. Assigning equal probabilities to the frequencies is like assigning equal probability to values of a variable that counts the number of successes in a given sequence.<br /><br />To take a simple example, say I've observed three risings out of three trials. Laplace assigns equal probability to {000}, {001, 010, 100}, {011, 101, 110}, and {111}. So, P(111) = 1/4 and P(011) = 1/12 before any evidence is collected. Boole's alternative makes all of the possible sequences equally likely. So, P(111) = P(011) = 1/8 before any evidence is collected.<br /><br />The problem with that assignment (as Boole noticed) is that it makes learning from experience impossible. If one assigns equal probability to the constitutions of the universe, then the probability that the m+1st observation will be a success given that the last m have been successes is 1/2, regardless of the size of m.<br /><br />I think one can get the same effect by letting alpha and beta go to infinity in the beta distribution. But I'm not very confident about this.<br /><br />I don't know if you have time to overthink this any further, but I would really appreciate seeing some simple examples and how some alternatives to the flat prior over p work out. For example, I would like to see how a flat prior over the odds compares.Jonathan Livengoodhttps://www.blogger.com/profile/08264815112941067048noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-76045332878582442682012-03-22T06:38:22.681-07:002012-03-22T06:38:22.681-07:00PPS - Your discussion of the form of the prior dis...PPS - Your discussion of the form of the prior distribution has jogged my memory:<br /><br />In fact the prior that best represents 'complete ignorance' does indeed place more weight near 0 and 1. Jaynes, in 'Prior Probabilities' (http://bayes.wustl.edu/etj/articles/prior.pdf) showed using transformation groups that the correct prior for P(f) in the case of 'complete ignorance' is proportional to 1/(f(1-f)), which converges on the law of succession for large numbers of samples.<br /><br />Oddly, the uniform prior (and resulting law of succession) seems to amount to an additional piece of information to the effect that the frequency is neither 0 nor 1.<br /><br />I see also that in the same article Jaynes discusses the conceptual difficulty of a 'probability of a probability', with an interesting general solution in terms of a population of people, each performing their own probability assignments. I think this can still be confusing, though, and can (probably) be avoided here (the problem, though, is what exactly do I mean by 'a fixed frequency'?).<br /><br />Probably overthinking?Tom Campbell-Rickettshttps://www.blogger.com/profile/07387943617652130729noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-22642247761206837232012-03-22T04:09:37.248-07:002012-03-22T04:09:37.248-07:00It seems to me that the hypotheses presented in th...It seems to me that the hypotheses presented in the Economist article: either the sun will rise or it will not, make no connection between events on successive days, and so it is impossible for the updated probabilities to differ from the originals.<br /><br />Your analysis is evidently a huge improvement, but I wonder if your terminology is a bit confusing when you talk about calculating the probability for a probability. (I noticed a similar terminology recently in your excellent Blinky Monty Hall problem.) I think such phrasing might be hard for some to stomach, is avoidable, and furthermore comes dangerously close to supposing that a probability is a physical property of the system under study, which gets one into all sorts of difficulties.<br /><br />Might it be conceptually easier to suppose that the sun rises with some fixed relative frequency, and that the purpose of the analysis is to ascertain a probability distribution over the possible frequencies?<br /><br />PS Bayes came first, but Laplace was the first to write down the theorem in its general form.Tom Campbell-Rickettshttps://www.blogger.com/profile/07387943617652130729noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-12723310925331905492012-03-21T09:50:20.892-07:002012-03-21T09:50:20.892-07:00Thanks -- I knew that Laplace had already discover...Thanks -- I knew that Laplace had already discovered Bayes's Theorem (and probably should get credit for it) but I didn't know he had posed and solved the Sunrise problem (http://en.wikipedia.org/wiki/Sunrise_problem).Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-49891557551886201032012-03-21T09:43:13.181-07:002012-03-21T09:43:13.181-07:00This is just Laplace's Rule of Succession http...This is just Laplace's Rule of Succession http://en.wikipedia.org/wiki/Rule_of_successionAnonymousnoreply@blogger.com