tag:blogger.com,1999:blog-6894866515532737257.comments2016-07-23T03:07:19.519-07:00Probably Overthinking ItAllen Downeyhttps://plus.google.com/111942648516576371054noreply@blogger.comBlogger629125tag:blogger.com,1999:blog-6894866515532737257.post-79677701644913361342016-07-14T10:23:03.802-07:002016-07-14T10:23:03.802-07:00In a somewhat similar vein, I'm quite fond of ...In a somewhat similar vein, I'm quite fond of McElreath's recent Statistical Rethinking book and lecture videos:<br />http://xcelab.net/rm/statistical-rethinking/Unknownhttp://www.blogger.com/profile/08220616256423969075noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-80019813634769409612016-07-11T10:27:10.797-07:002016-07-11T10:27:10.797-07:00Well, I think you need to bring this essay up to d...Well, I think you need to bring this essay up to date! OOP is so 90's -- to get people's attention I think you need to rework this as an exercise in functional programming. That's what all the cool kids are doing these days. It's a toss up between Haskell (superior FP street cred) and Scala (messy, kitchen-sink approach, but undeniably the next big thing).<br /><br />I'm only half joking -- probability distributions are, like most or all mathematical objects, essentially declarative in nature instead of procedural. Ideally one would just declare the properties of a distribution in one representation and then properties of other representations would just fall out of the machinery. Failing that kind of mathematical programming language (to which existing computer algebra systems are an approximation) I think functional programming is a better fit than procedural. <br /><br />I don't know how one would best define probability distributions in a FP language. I'll call on those with more expertise to give us some ideas at this point.Robert Dodierhttp://www.blogger.com/profile/09299746495471300195noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-6442329817638134332016-07-01T06:27:28.194-07:002016-07-01T06:27:28.194-07:00That's a good example of what I'm talking ...That's a good example of what I'm talking about: if you have enough data, Anderson-Darling and related tests will eventually indicate that your data are not normally distributed, but that doesn't mean a normal model would not be reasonable and useful.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-34350946630670571972016-07-01T06:24:32.435-07:002016-07-01T06:24:32.435-07:00That's correct, and a corollary to Ted's o...That's correct, and a corollary to Ted's objection to my assumption that the mass is spread uniformly.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-6036315821902320882016-07-01T06:19:34.468-07:002016-07-01T06:19:34.468-07:00At the beginningļ¼the center of the spring mass is ...At the beginningļ¼the center of the spring mass is not at its center. It is somewhere below its center. jshttp://www.blogger.com/profile/00529359045734652255noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-50738444932359212602016-07-01T05:42:09.572-07:002016-07-01T05:42:09.572-07:00Hi sir
I have data of all blood pressure readings ...Hi sir<br />I have data of all blood pressure readings max 150 min 70. All data are recorded from normal pattern and patients looks very normal. Totally 872 reading s of 30 patients.However AD test non normal. It fits only nearly 3 parameter logo logistical distribution AD of 4,.. And p of 0.0005<. Now can I use process capability evaluation on these data just considering it as normal data? If I do so I will get excellent results Vickhttp://www.blogger.com/profile/14153431507175713969noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-11331479637563599052016-06-18T08:46:21.927-07:002016-06-18T08:46:21.927-07:00"Bayesian methods don't do the same thing..."Bayesian methods don't do the same things better; they do different things, which are better."<br />...<br />They do different things, indeed. So why insist that one is universally better?<br /><br />One thing I've been mulling over lately: Classical methods were invented for, and by, people in the business of designing and running experiments. "Hypothesis tests" aren't really testing hypotheses about the sample you collected, or the population you sampled, but about *the design of the experiment* you ran. In the simplest case, the question isn't "Is theta 0 or not?" but rather "Is this sample big enough to tell if theta is positive or negative?"<br />This is not always "the wrong question," as you call it. Sometimes yes, but other times it's crucial.<br /><br />Perhaps the main benefit of inventing hypothesis tests was so you could *imagine* doing them as you make power calculations to choose the sample size (and other design details).<br />Engineers find it useful to know their instrument's operating characteristics before they choose which lathe or radio or oscilloscope to buy/use. Same with doing classical power calculations to design a study for science, or A/B testing, or medical trials, or what have you.<br /><br />I don't mean to say the Bayes framework can't do this too (I know nothing about Bayesian experimental design) but just that traditional classical methods have legitimate uses different from, not worse than, traditional Bayesian methods.Unknownhttp://www.blogger.com/profile/03611849167717252118noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-48887959715171039092016-06-16T10:24:22.186-07:002016-06-16T10:24:22.186-07:00Thanks, James. If each die roll yields a random v...Thanks, James. If each die roll yields a random variable, then the sequence of RVs makes a discrete time random process. But since they are independent and identically distributed, I treat them as a single RV. Admittedly I am being imprecise, but I didn't want this article to turn into "What is a random process?"Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-14195594606542183252016-06-16T10:16:57.581-07:002016-06-16T10:16:57.581-07:00Just an observation: your meaning of "random ...Just an observation: your meaning of "random process" makes sense, but it happens not to be how the phrase is generally used. Generally "random process" is used to mean a collection of random variables indexed by time (or some other similar index set). See for example https://en.wikipedia.org/wiki/Stochastic_process . Maybe yours could alternatively be called a "random procedure"?Jameshttp://www.blogger.com/profile/10075856924139777288noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-86795590804703442212016-06-15T05:32:36.924-07:002016-06-15T05:32:36.924-07:00Thank you, Allen (and Sanjoy)! Your workshop gave...Thank you, Allen (and Sanjoy)! Your workshop gave me a lot to think about. Jessicahttp://www.blogger.com/profile/12370010794505516795noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-2056969920698747272016-06-14T09:02:49.510-07:002016-06-14T09:02:49.510-07:00Yes! We take a similar approach here in a class c...Yes! We take a similar approach here in a class called Modeling and Simulation, where we use MATLAB's ode45 function to solve differential equations. That gives students the ability to develop models of physical systems that include realistic factors, like friction and air resistance, that are out of scope in a class that is limited to analytic methods.<br /><br />And I am very interested in the other idea you raise, how to do engineering design on systems that don't lend themselves to mathematical analysis. Do we have to search enormous design spaces? Or can we grow/evolve solutions?Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-42015949885315840192016-06-08T03:53:09.351-07:002016-06-08T03:53:09.351-07:00The paper says that the subjects were diagnosed wi...The paper says that the subjects were diagnosed with NCGS based on clinical diagnostic criteria, not physical measurements. And the point of the study was to test "whether symptoms in non-coeliac patients... are specifically triggered by gluten".Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-38210142737511938522016-06-08T01:35:46.808-07:002016-06-08T01:35:46.808-07:00It appears to me that all 35 patients were known t...It appears to me that all 35 patients were known to have Non-Celiac Gluten Sensitivity (NCGS) by physical measurements. I believe the point of the study was to show only 12 of 35 of patients with NCGS would correctly relay the "diagnosis" to their doctor via a currently-accepted questionnaire.Scott Robertshttp://www.blogger.com/profile/03727573717462028189noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-20879997571517255982016-06-08T00:52:11.800-07:002016-06-08T00:52:11.800-07:00This is amazing stuff. Similar conclusions have be...This is amazing stuff. Similar conclusions have been made about different equations: you just simulate them instead of trying to solve them. And in mechanical engineering, you learn all kinds of methods, but then in the real world, you just learn how to use numerical methods (fundamentally based on D.E.s). Even in the idealized world of electrical engineering, there is finally a push in recent years to go "full spice" (having numerical methods more readily available). Although in mechanical and electrical, generalized concepts in the brain is required to zero in on designs that are most likely to be the most efficient, then switching over to the numerical methods to work out the details. I wonder if there is a similar process needed here.<br /><br />It makes me wonder if physics and math are just compression methods suited for the brain. I mean they actually a priori *are*, but I wonder if they may not have any deeper significance than simply being something like patterns that are "accidentally" more common as a result of deeper, simpler logic rules with less K-complexity like cellular automata (Wolfram). And that the most efficient designs would be discoverable with quantum computing, in some way related to how it can break cryptography.Scott Robertshttp://www.blogger.com/profile/03727573717462028189noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-36353210782310211282016-06-02T06:15:32.847-07:002016-06-02T06:15:32.847-07:00Yes, that's right. I did the computation part...Yes, that's right. I did the computation partly to get the other three probabilities (two well, and one well one sick), and partly to demonstrate the computation I need for some of the other scenarios.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-82424844016103549812016-06-02T00:50:25.533-07:002016-06-02T00:50:25.533-07:00Hi, I have a doubt about Scenario A question 2. I ...Hi, I have a doubt about Scenario A question 2. I computed the solution just considering people statistically independent with respect to the test so I get (1/4)^2 = 1/16. <br />Looking at the solution I saw a lot of computation to get the same answer. Am I missing something or my answer is correct? <br />By the way, thank you for these series of post Allen, they are very useful.<br />Giovannijimi75http://www.blogger.com/profile/00036358528199180534noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-3845811131377724252016-05-26T11:18:04.755-07:002016-05-26T11:18:04.755-07:00Is "condition" as defined by trivers (bi...Is "condition" as defined by trivers (biology) related to any definition of socioeconomic condition? By any definition of "condition", people in the United States are at the peak economic condition. Trivers did not define "condition" as anything close to a value that can be picked off a database. Vijayhttp://www.blogger.com/profile/04598434756892672717noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-6600171632907093122016-05-22T13:14:37.288-07:002016-05-22T13:14:37.288-07:00> For the 18 SOBs we have actually observed
Fa...> For the 18 SOBs we have actually observed<br /><br />Fantastic variable choice!Alex Riinahttp://www.blogger.com/profile/16798593482677040333noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-68623836014249615002016-05-20T06:31:23.268-07:002016-05-20T06:31:23.268-07:00This is great! We have been using Bayesian methods...This is great! We have been using Bayesian methods in economic science for quite some time. I am glad to see that engineering is, finally, starting to pick up on it. Richard Sessahttp://www.blogger.com/profile/02172477842431895627noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-72007134263122731472016-05-18T03:04:06.249-07:002016-05-18T03:04:06.249-07:00Andrew Gelman wrote an article on a similar topic ...Andrew Gelman wrote an article on a similar topic a few years ago, if you've not seen it it's an interesting take on the problem: http://andrewgelman.com/2009/06/21/of_beauty_sex_a/Sam Masonhttp://www.blogger.com/profile/17261968597468673085noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-1408361391641392202016-05-16T07:46:12.411-07:002016-05-16T07:46:12.411-07:00Good questions. What I have done so far is one at...Good questions. What I have done so far is one attempt to find a generational effect, which failed. So there could still be an effect, but my experiment missed it.<br /><br />I am planning a follow-up that will try harder to show the effect, by searching specifically for questions that show a generational pattern and aggregating them.<br /><br />Watch this space.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-17535750982559315332016-05-16T07:38:11.344-07:002016-05-16T07:38:11.344-07:00If it did, how would you reconcile it with the res...If it did, how would you reconcile it with the results of this analysis?Arthur Grabovskyhttp://www.blogger.com/profile/06197260440427038143noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-45096780905262567342016-05-13T14:24:06.848-07:002016-05-13T14:24:06.848-07:00Interesting approach! My first instinct would have...Interesting approach! My first instinct would have been to plot age on the x-axis and an index of social and political attitudes on the y-axis, fit a spline, and examine jumps at the lower and upper bounds of each generational interval. I wonder if this would produce different results. Y.R. Velezhttp://www.blogger.com/profile/16807475997597988399noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-17561859733299112132016-05-10T07:18:47.260-07:002016-05-10T07:18:47.260-07:00Your correction on Scenario C is correct, and your...Your correction on Scenario C is correct, and your answer on Scenario D was correct all along.<br /><br />Interestingly, I made the same mistake on Scenario C (but it took me longer to catch it).<br /><br />Nice job!Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-68274079664777367192016-05-09T14:00:20.197-07:002016-05-09T14:00:20.197-07:00Let me see if I can get it right this time. Feel f...Let me see if I can get it right this time. Feel free to leave my earlier wrong answer up. I deserve to be at least mildly shamed.<br /><br />Scenario C: <br /><br />Let t1=0.2 and t2=0.4. Here are the ways to get a positive test result, with their associated probabilities:<br /><br />Sick: p s<br />Not sick, t=t1: (1-p) t1 / 2<br />Not sick, t=t2: (1-p) t2 / 2<br /><br />The probability that one person is sick, given a positive test result, is <br /><br />ps / (sum of all three terms above),<br /><br />which is 1/4. <br /><br />That's the probability that the first positive-tester is sick, in scenario C (as it was in A and B).<br /><br />In scenario C, each trial is independent, so the answer to question 2 is the square of the answer to question 1, i.e., 1/16.<br /><br />Scenario D: <br /><br />If we hypothesize any given value of t, then we can calculate the probability that any given positive-tester is in fact sick. That turns out to be<br /><br />psick = 1/3 if t= 0.2<br />psick = 1/5 if t = 0.4.<br /><br />In scenario D, we never find out any information that tells us which value of t is correct, so they remain equiprobable. There's a 50% chance that psick=1/3 and a 50% chance that psick = 1/5, so when you meet that first positive-tester, the probability that he's sick is the average of the two.<br /><br />psick =0.5 (1/3+1/5) = 4/15. [Scenario D, question 1]<br /><br />Under each hypothesis for t, the probability that any given positive-tester is sick is independent of the others, so the probability that the first two testers are both sick is the square of the probability that one is sick. We still have no information about which value of t is correct, so the probability that both folks are sick is the average of the probabilities for each of the two t's:<br /><br />0.5 ( 1/9 + 1/25 ) = 17/225 [Scenario D, question 2].Ted Bunnhttp://www.blogger.com/profile/12230509214302717664noreply@blogger.com