I received the following enquiry by email. How should I have responded? Is it possible to make assertions about the whole population (the constituency) on the basis of the information given?
" Dear mavin of math, I have a question regarding statistics which I am hoping someone in your department could help me answer. It has been some time since I had statistics in college and to say that I am rusty is an understatement. Before I ask the question you will need some background information. I work for an organization that issues standards for use by public entities. We try to obtain input and feedback from our constituents to ensure the standards meet their needs. We recently put together a task force of 30 people selected from among our constituents to help us evaluate a proposed standard. The 30 members of the task force were judgmentally selected by my organization. The emphasis in selecting these task force members was on obtaining views from the many organizations and groups that constitute our constituency. Thus a representative was selected from each organization and group. We selected multiple representatives from the larger of the organizations and groups. One member of our staff asked what use the task force would be in helping us choose from among the many directions we could go with the proposed standard. I responded as follows, "If 28 out of the 30 members agree on one option, that would be indicative to me that there may be a similar trend in the population." If there is a similar trend in the population we could use the task force input to choose an option that would best meet the needs of our constituency. In making the above statement I am not saying that we could extrapolate the results to the population to say that 28 out of 30 members of the population would also choose the same option. The sample was not randomly selected. Not all organizations and groups that constitute our constituency are equally represented on the task force based on membership. Further, although we know the size of the membership of some of the organizations and groups that constitute our constituency, we don't know the size of all of them. However, together they compose a sufficiently large number (over 5000) that the law of large numbers applies to the population (our constituency). What I am saying is that, if 28 out of 30 members choose the same option, this must be significant and useful to our process. If I remember my statistics correctly, in order to say that something is significant and useful one must also be able to say that confounding factors do not account for the unanimity. We are attempting to control confounding factors at the task force meeting. Thus in selecting the task force members we attempted to find individuals who will represent the views of the groups from which they are selected. Further, all options will be presented to the task force without bias. The task force is split into four small groups so that no one personality will dominate and influence the choices of all others. The moderators of each sub-group will give each member equal time in the discussions. Therefore, if 28 out of 30 task force members arrive at the same conclusion, I believe it would be improbable that such unanimity would be attributable to chance, some confounding factor, or anything other than the merits of the options. Thus, I believe such an outcome would be significant no matter what applicable test for significance one used. But what about useful? I intuit that if 28 out of 30 task force members reach the same conclusion it is not just significant, but it also has predictive value. Perhaps the question becomes, "just how representative of the population is the sample?" Since the sample was not randomly selected I can't quantify what percentage of the population will reach the same conclusions (plus or minus some range, with a given degree of error). Yet, for each member of the task force I believe there is better than a 50/50 chance that they are representative of the group they purport to represent. If not, we have done a lousy job of selecting the task force members. We have, however, invested significant time and effort in selecting the task force members to the end that they would be representative of their various groups. So, with that background information, here is my question: Given a high degree of unanimity among the task force members (28 out of 30 in agreement) and further given that they are more likely than not to represent (be representative of) their various groups, and assuming (or not) that the various groups from which they were chosen are equal in size, could I say with better than, say, 65% confidence that most members of the population (our constituency) would reach the same conclusion as the 28 reached? That would be the ideal outcome for the task force; that they help us select an option that most of our constituents would agree with. If you or your colleagues don't have the time to answer this question(s) would you please recommend some book(s) on the topic that I could slog through without sinking over my head. Your help is greatly appreciated. [Name edited out] "