tag:blogger.com,1999:blog-6894866515532737257.post3583157396772344171..comments2024-03-27T01:01:09.785-07:00Comments on Probably Overthinking It: Are your data normal? Hint: no.Allen Downeyhttp://www.blogger.com/profile/01633071333405221858noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-6894866515532737257.post-6442329817638134332016-07-01T06:27:28.194-07:002016-07-01T06:27:28.194-07:00That's a good example of what I'm talking ...That's a good example of what I'm talking about: if you have enough data, Anderson-Darling and related tests will eventually indicate that your data are not normally distributed, but that doesn't mean a normal model would not be reasonable and useful.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-50738444932359212602016-07-01T05:42:09.572-07:002016-07-01T05:42:09.572-07:00Hi sir
I have data of all blood pressure readings ...Hi sir<br />I have data of all blood pressure readings max 150 min 70. All data are recorded from normal pattern and patients looks very normal. Totally 872 reading s of 30 patients.However AD test non normal. It fits only nearly 3 parameter logo logistical distribution AD of 4,.. And p of 0.0005<. Now can I use process capability evaluation on these data just considering it as normal data? If I do so I will get excellent results Vickhttps://www.blogger.com/profile/14153431507175713969noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-52876069657499026412014-01-05T11:11:27.720-08:002014-01-05T11:11:27.720-08:00I usually go the other way: I use Bayesian methods...I usually go the other way: I use Bayesian methods to generate a posterior distribution, and then if I need an MLE, I can get it from the posterior. But if all you have is an MLE, that's not enough to determine the distribution (in general).Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-29708610385408308162014-01-05T11:09:10.200-08:002014-01-05T11:09:10.200-08:00And here it is: http://allendowney.blogspot.com/20...And here it is: http://allendowney.blogspot.com/2013/09/how-to-consume-statistical-analysis.htmlAllen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-65625798480991137962014-01-05T10:19:38.719-08:002014-01-05T10:19:38.719-08:00Do you ever use a maximum likelihood estimation to...Do you ever use a maximum likelihood estimation to determine distributions? Bubbahttps://www.blogger.com/profile/09079021607928042787noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-53392123096131589122013-08-16T12:23:54.610-07:002013-08-16T12:23:54.610-07:00Good suggestion. Thanks!
Or there's one more...Good suggestion. Thanks!<br /><br />Or there's one more alternative -- you can generate several rows with size N, sort the rows, then take the average of each column. As the number of rows increases, the averages converge on the rankits.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-48219721729618598112013-08-16T12:18:46.448-07:002013-08-16T12:18:46.448-07:00For the normality plot, I wanted to mention a litt...For the normality plot, I wanted to mention a little trick that helps out with small sample sizes (which will have varying results because of your small random draw from the normal distribution): you can draw a much larger set of normally distributed points, sort them, and then take only a subset of those sorted points. Honestly, the code is simpler than my prose, so here is a little sample:<br /><br />n = 5000 # this makes for very solid normpts<br />data = [820,770,710,910,840,670,830]<br />sdata = sorted(data)<br />normpts = sorted(np.random.normal(0,1,len(data)*n))[n/2::n]Boris Dieseldorffhttps://www.blogger.com/profile/11818300387729526308noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-38874724812999647192013-08-08T05:16:21.072-07:002013-08-08T05:16:21.072-07:00This comment has been removed by the author.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-13334319352321644562013-08-08T05:13:00.434-07:002013-08-08T05:13:00.434-07:00I am working on a talk that addresses exactly this...I am working on a talk that addresses exactly this question. I will post it soon.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-43147258664301701752013-08-08T02:30:51.633-07:002013-08-08T02:30:51.633-07:00Can you say something about why CDF is preferable ...Can you say something about why CDF is preferable to PDF for comparing distributions?Unknownhttps://www.blogger.com/profile/04903969847912039998noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-25840900406074654572013-08-07T21:32:51.732-07:002013-08-07T21:32:51.732-07:00This comment has been removed by a blog administrator.Unknownhttps://www.blogger.com/profile/16800640123933103403noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-78022050276766837672013-08-07T13:17:54.267-07:002013-08-07T13:17:54.267-07:00Yes, most of my objections about this kind of dist...Yes, most of my objections about this kind of distribution testing also apply to hypothesis testing in general.<br /><br />I am picking on this particular application because when people ask about distribution testing, it is almost always the wrong question. What they really want, most of the time, is help making modeling decisions.<br /><br />Thanks for this comment!Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-24538881348191161572013-08-07T13:05:49.082-07:002013-08-07T13:05:49.082-07:00What you're describing is actually a problem w...What you're describing is actually a problem with hypothesis testing in general, rejecting the null does not tell you anything about effect size, so we can fool ourselves with over-powered tests. Goodness of fit tests like Kolmogorov-Smirnov and Anderson-Darling model the empirical process as a Brownian Bridge and look for irregularities in the sup-norm or weighted L2 norm respectively. It's certainly possible to interpret an effect size here, assuming one bothers to report it, and it tells the same kind of stories that your pictures do.<br /><br />For example suppose I want to know if 10^6 data points are well-modeled by a uniform distribution on [0,1] -- I can form the empirical process, which should be approximately the Brownian Bridge. The sup of the empirical process, which is after all a one-sided KS statistic is nothing more than 1000 * (max difference between empirical and desired cdf). If I reject above some level T, I can say what I actually mean, I am rejecting for errors of probability on order T/1000. If this effect size was reported, you could make a decision about the meaning of the test, and would probably conclude for many purposes that it was ridiculously over-powered.<br /><br />What I'm saying is just basic hygiene when we talk about tests for a single parameter, but for some reason as soon as things get non-parametric, common sense goes out the window.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-39338716237350264602013-08-07T10:09:39.808-07:002013-08-07T10:09:39.808-07:00Yes! The QQ plot requires you to compare your dat...Yes! The QQ plot requires you to compare your data to a specific distribution, so it is only as good as your estimated parameters.<br /><br />The normal probability plot does not require you to estimate parameters. In the example I used the standard normal distribution, but I could have used any normal distribution. This works because of the property I mentioned: normal distributions are closed under linear transformation.Allen Downeyhttps://www.blogger.com/profile/01633071333405221858noreply@blogger.comtag:blogger.com,1999:blog-6894866515532737257.post-11176681004055590292013-08-07T09:42:49.026-07:002013-08-07T09:42:49.026-07:00Could you explain why you are using a "normal...Could you explain why you are using a "normal probability plot" instead of a QQ-plot?Jeradhttps://www.blogger.com/profile/12129234441279989358noreply@blogger.com