THE RESEARCH PROCESS

David S. Walonick, Ph.D.

We understand the world by asking questions and searching for answers. Our construction of reality depends on the nature of our inquiry.

Until the sixteenth century, human inquiry was primarily based on introspection. The way to know things was to turn inward and use logic to seek the truth. This paradigm had endured for a millennium and was a well-established conceptual framework for understanding the world. The seeker of knowledge was an integral part of the inquiry process.

A profound change occurred during the sixteenth and seventeenth centuries. Copernicus, Kepler, Galileo, Descartes, Bacon, Newton, and Locke presented new ways of examining nature. Our method of understanding the world came to rely on measurement and quantification. Mathematics replaced introspection as the key to supreme truths. The Scientific Revolution was born.

Objectivity became a critical component of the new scientific method. The investigator was an observer, rather than a participant in the inquiry process. A mechanistic view of the universe evolved. We believed that we could understand the whole by performing an examination of the individual parts. Experimentation and deduction became the tools of the scholar. For two hundred years, the new paradigm slowly evolved to become part of the reality framework of society. The Age of Enlightenment had arrived.

Scientific research methodology was very successful at explaining natural phenomena. It provided a systematic way of knowing. Western philosophers embraced this new structure of inquiry. Eastern philosophy continued to stress the importance of the one seeking knowledge. By the beginning of the twentieth century, a complete schism had occurred. Western and Eastern philosophies were mutually exclusive and incompatible.

Then something remarkable happened. Einstein's proposed that the observer was not separate from the phenomena being studied. Indeed, his theory of relativity actually stressed the role of the observer. Quantum mechanics carried this a step further and stated that the act of observation could change the thing being observed. The researcher was not simply an observer, but in fact, was an integral part of the process. In physics, Western and Eastern philosophies have met. This idea has not been incorporated into the standard social science research model, and today's social science community see themselves as objective observers of the phenomena being studied. However, "it is an established principle of measurement that instruments react with the things they measure." (Spector, 1981, p. 25) The concept of instrument reactivity states that an instrument itself can disturb the thing being measured.

Problem Recognition & Definition

All research begins with a question. Intellectual curiosity is often the foundation for scholarly inquiry. Some questions are not testable. The classic philosophical example is to ask, "How many angels can dance on the head of a pin?" While the question might elicit profound and thoughtful revelations, it clearly cannot be tested with an empirical experiment. Prior to Descartes, this is precisely the kind of question that would engage the minds of learned men. Their answers came from within. The modern scientific method precludes asking questions that cannot be empirically tested. If the angels cannot be observed or detected, the question is considered inappropriate for scholarly research.

A paradigm is maintained as much by the process of formulating questions as it is by the answers to those questions. By excluding certain types of questions, we limit the scope of our thinking. It is interesting to note, however, that modern physicists have began to ask the same kinds of questions posed by the Eastern philosophers. "Does a tree falling in the forest make a sound if nobody is there to hear it?" This seemingly trivial question is at the heart of the observer/observed dichotomy. In fact, quantum mechanics predicts that this kind of question cannot be answered with complete certainty. It is the beginning of a new paradigm.

Defining the goals and objectives of a research project is one of the most important steps in the research process. Clearly stated goals keep a research project focused. The process of goal definition usually begins by writing down the broad and general goals of the study. As the process continues, the goals become more clearly defined and the research issues are narrowed.

Exploratory research (e.g., literature reviews, talking to people, and focus groups) goes hand-in-hand with the goal clarification process. The literature review is especially important because it obviates the need to reinvent the wheel for every new research question. More importantly, it gives researchers the opportunity to build on each others work.

The research question itself can be stated as a hypothesis. A hypothesis is simply the investigator's belief about a problem. Typically, a researcher formulates an opinion during the literature review process. The process of reviewing other scholar's work often clarifies the theoretical issues associated with the research question. It also can help to elucidate the significance of the issues to the research community.

The hypothesis is converted into a null hypothesis in order to make it testable. "The only way to test a hypothesis is to eliminate alternatives of the hypothesis." (Anderson, 1966, p.9) Statistical techniques will enable us to reject a null hypothesis, but they do not provide us with a way to accept a hypothesis. Therefore, all hypothesis testing is indirect.

Creating the Research Design

Defining a research problem provides a format for further investigation. A well-defined problem points to a method of investigation. There is no one best method of research for all situations. Rather, there are a wide variety of techniques for the researcher to choose from. Often, the selection of a technique involves a series of trade-offs. For example, there is often a trade-off between cost and the quality of information obtained. Time constraints sometimes force a trade-off with the overall research design. Budget and time constraints must always be considered as part of the design process (Walonick, 1993).

Many authors have categorized research design as either descriptive or causal. Descriptive studies are meant to answer the questions of who, what, where, when and how. Causal studies are undertaken to determine how one variable affects another. McDaniel and Gates (1991) state that the two characteristics that define causality are temporal sequence and concomitant variation.

The word causal may be a misnomer. The mere existence of a temporal relationship between two variables does not prove or even imply that A causes B. It is never possible to prove causality. At best, we can theorize about causality based on the relationship between two or more variables, however, this is prone to misinterpretation. Personal bias can lead to totally erroneous statements. For example, Blacks often score lower on I.Q. scores than their White counterparts. It would be irresponsible to conclude that ethnicity causes high or low I.Q. scores. In social science research, making false assumptions about causality can delude the researcher into ignoring other (more important) variables.

There are three basic methods of research: 1) survey, 2) observation, and 3) experiment (McDaniel and Gates, 1991). Each method has its advantages and disadvantages.

The survey is the most common method of gathering information in the social sciences. It can be a face-to-face interview, telephone, or mail survey. A personal interview is one of the best methods obtaining personal, detailed, or in-depth information. It usually involves a lengthy questionnaire that the interviewer fills out while asking questions. It allows for extensive probing by the interviewer and gives respondents the ability to elaborate their answers. Telephone interviews are similar to face-to-face interviews. They are more efficient in terms of time and cost, however, they are limited in the amount of in-depth probing that can be accomplished, and the amount of time that can be allocated to the interview. A mail survey is generally the most cost effective interview method. The researcher can obtain opinions, but trying to meaningfully probe opinions is very difficult.

Observation research monitors respondents' actions without directly interacting with them. It has been used for many years by A.C. Nielsen to monitor television viewing habits. Psychologists often use one-way mirrors to study behavior. Social scientists often study societal and group behaviors by simply observing them. The fastest growing form of observation research has been made possible by the bar code scanners at cash registers, where purchasing habits of consumers can now be automatically monitored and summarized.

In an experiment, the investigator changes one or more variables over the course of the research. When all other variables are held constant (except the one being manipulated), changes in the dependent variable can be explained by the change in the independent variable. It is usually very difficult to control all the variables in the environment. Therefore, experiments are generally restricted to laboratory models where the investigator has more control over all the variables.

Sampling

It is incumbent on the researcher to clearly define the target population. There are no strict rules to follow, and the researcher must rely on logic and judgment. The population is defined in keeping with the objectives of the study.

Sometimes, the entire population will be sufficiently small, and the researcher can include the entire population in the study. This type of research is called a census study because data is gathered on every member of the population.

Usually, the population is too large for the researcher to attempt to survey all of its members. A small, but carefully chosen sample can be used to represent the population. The sample reflects the characteristics of the population from which it is drawn.

Sampling methods are classified as either probability or nonprobability. In probability samples, each member of the population has a known probability of being selected. Probability methods include random sampling, systematic sampling, and stratified sampling. In nonprobability sampling, members are selected from the population in some nonrandom manner. These include convenience sampling, judgment sampling, quota sampling, and snowball sampling. The other common form of nonprobability sampling occurs by accident when the researcher inadvertently introduces nonrandomness into the sample selection process. The advantage of probability sampling is that sampling error can be calculated. Sampling error is the degree to which a sample might differ from the population. When inferring to the population, results are reported plus or minus the sampling error. In nonprobability sampling, the degree to which the sample differs from the population remains unknown. (McDaniel and Gates, 1991)

Random sampling is the purest form of probability sampling. Each member of the population has an equal chance of being selected. When there are very large populations, it is often difficult or impossible to identify every member of the population, so the pool of available subjects becomes biased. Random sampling is frequently used to select a specified number of records from a computer file.

Systematic sampling is often used instead of random sampling. It is also called an Nth name selection technique. After the required sample size has been calculated, every Nth record is selected from a list of population members. As long as the list does not contain any hidden order, this sampling method is as good as the random sampling method. Its only advantage over the random sampling technique is simplicity.

Stratified sampling is commonly used probability method that is superior to random sampling because it reduces sampling error. A stratum is a subset of the population that share at least one common characteristic. The researcher first identifies the relevant stratums and their actual representation in the population. Random sampling is then used to select subjects for each stratum until the number of subjects in that stratum is proportional to its frequency in the population.

Convenience sampling is used in exploratory research where the researcher is interested in getting an inexpensive approximation of the truth. As the name implies, the sample is selected because they are convenient. This nonprobability method is often used during preliminary research efforts to get a gross estimate of the results, without incurring the cost or time required to select a random sample.

Judgment sampling is a common nonprobability method. The researcher selects the sample based on judgment. This is usually and extension of convenience sampling. For example, a researcher may decide to draw the entire sample from one "representative" city, even though the population includes all cities. When using this method, the researcher must be confident that the chosen sample is truly representative of the entire population.

Quota sampling is the nonprobability equivalent of stratified sampling. Like stratified sampling, the researcher first identifies the stratums and their proportions as they are represented in the population. Then convenience or judgment sampling is used to select the required number of subjects from each stratum. This differs from stratified sampling, where the stratums are filled by random sampling.

Snowball sampling is a special nonprobability method used when the desired sample characteristic is rare. It may be extremely difficult or cost prohibitive to locate respondents in these situations. Snowball sampling relies on referrals from initial subjects to generate additional subjects. While this technique can dramatically lower search costs, it comes at the expense of introducing bias because the technique itself reduces the likelihood that the sample will represent a good cross section from the population.

Data Collection

There are very few hard and fast rules to define the task of data collection. Each research project uses a data collection technique appropriate to the particular research methodology. The two primary goals for both quantitative and qualitative studies are to maximize response and maximize accuracy.

When using an outside data collection service, researchers often validate the data collection process by contacting a percentage of the respondents to verify that they were actually interviewed. Data editing and cleaning involves the process of checking for inadvertent errors in the data. This usually entails using a computer to check for out-of-bounds data.

Quantitative studies employ deductive logic, where the researcher starts with a hypothesis, and then collects data to confirm or refute the hypothesis. Qualitative studies use inductive logic, where the researcher first designs a study and then develops a hypothesis or theory to explain the results of the analysis.

Quantitative analysis is generally fast and inexpensive. A wide assortment of statistical techniques are available to the researcher. Computer software is readily available to provide both basic and advanced multivariate analysis. The researcher simply follows the preplanned analysis process, without making subjective decisions about the data. For this reason, quantitative studies are usually easier to execute than qualitative studies.

Qualitative studies nearly always involve in-person interviews, and are therefore very labor intensive and costly. They rely heavily on a researcher's ability to exclude personal biases. The interpretation of qualitative data is often highly subjective, and different researchers can reach different conclusions from the same data. However, the goal of qualitative research is to develop a hypothesis--not to test one. Qualitative studies have merit in that they provide broad, general theories that can be examined in future research.

Data Analysis

Modern computer software has made the analysis of quantitative data a very easy task. It is no longer incumbent on the researcher to know the formulas needed to calculate the desired statistics. However, this does not obviate the need for the researcher to understand the theoretical and conceptual foundations of the statistical techniques. Each statistical technique has its own assumptions and limitations. Considering the ease in which computers can calculate complex statistical problems, the danger is that the researcher might be unaware of the assumptions and limitations in the use and interpretation of a statistic.

Reporting the Results

The most important consideration in preparing any research report is the nature of the audience. The purpose is to communicate information, and therefore, the report should be prepared specifically for the readers of the report. Sometimes the format for the report will be defined for the researcher (e.g., a dissertation), while other times, the researcher will have complete latitude regarding the structure of the report. At a minimum, the report should contain an abstract, problem statement, methods section, results section, discussion of the results, and a list of references (Anderson, 1966).

Validity and Reliability

Validity refers to the accuracy or truthfulness of a measurement. Are we measuring what we think we are? "Validity itself is a simple concept, but the determination of the validity of a measure is elusive" (Spector, 1981, p. 14).

Face validity is based solely on the judgment of the researcher. Each question is scrutinized and modified until the researcher is satisfied that it is an accurate measure of the desired construct. The determination of face validity is based on the subjective opinion of the researcher.

Content validity is similar to face validity in that it relies on the judgment of the researcher. However, where face validity only evaluates the individual items on an instrument, content validity goes further in that it attempts to determine if an instrument provides adequate coverage of a topic. Expert opinions, literature searches, and pretest open-ended questions help to establish content validity.

Criterion-related validity can be either predictive or concurrent. When a dependent/independent relationship has been established between two or more variables, criterion-related validity can be assessed. A mathematical model is developed to be able to predict the dependent variable from the independent variable(s). Predictive validity refers to the ability of an independent variable (or group of variables) to predict a future value of the dependent variable. Concurrent validity is concerned with the relationship between two or more variables at the same point in time.

Construct validity refers to the theoretical foundations underlying a particular scale or measurement. It looks at the underlying theories or constructs that explain a phenomena. This is also quite subjective and depends heavily on the understanding, opinions, and biases of the researcher.

Reliability is synonymous with repeatability. A measurement that yields consistent results over time is said to be reliable. When a measurement is prone to random error, it lacks reliability. The reliability of an instrument places an upper limit on its validity (Spector, 1981). A measurement that lacks reliability will necessarily be invalid. There are three basic methods to test reliability : test-retest, equivalent form, and internal consistency.

A test-retest measure of reliability can be obtained by administering the same instrument to the same group of people at two different points in time. The degree to which both administrations are in agreement is a measure of the reliability of the instrument. This technique for assessing reliability suffers two possible drawbacks. First, a person may have changed between the first and second measurement. Second, the initial administration of an instrument might in itself induce a person to answer differently on the second administration.

The second method of determining reliability is called the equivalent-form technique. The researcher creates two different instruments designed to measure identical constructs. The degree of correlation between the instruments is a measure of equivalent-form reliability. The difficulty in using this method is that it may be very difficult (and/or prohibitively expensive) to create a totally equivalent instrument.

The most popular methods of estimating reliability use measures of internal consistency. When an instrument includes a series of questions designed to examine the same construct, the questions can be arbitrarily split into two groups. The correlation between the two subsets of questions is called the split-half reliability. The problem is that this measure of reliability changes depending on how the questions are split. A better statistic, known as Chronbach's alpha (1951), is based on the mean (absolute value) interitem correlation for all possible variable pairs. It provides a conservative estimate of reliability, and generally represents "the lower bound to the reliability of an unweighted scale of items" (Carmines and Zeller, p. 45). For dichotomous nominal data, the KR-20 (Kuder-Richardson, 1937) is used instead of Chronbach's alpha (McDaniel and Gates, 1991).

Variability and Error

Most research is an attempt to understand and explain variability. When a measurement lacks variability, no statistical tests can be (or need be) performed. Variability refers to the dispersion of scores.

Ideally, when a researcher finds differences between respondents, they are due to true difference on the variable being measured. However, the combination of systematic and random errors can dilute the accuracy of a measurement. Systematic error is introduced through a constant bias in a measurement. It can usually be traced to a fault in the sampling procedure or in the design of a questionnaire. Random error does not occur in any consistent pattern, and it is not controllable by the researcher.

Summary

Scientific research involves the formulation and testing of one or more hypotheses. A hypothesis cannot be proved directly, so a null hypothesis is established to give the researcher an indirect method of testing a theory. Sampling is necessary when the population is too large, or when the researcher is unable to investigate all members of the target group. Random and systematic sampling are the best methods because they guarantee that each member of the population will have an known non-zero chance of being selected. The mathematical reliability (repeatability) of a measurement, or group of measurements, can be calculated, however, validity can only be implied by the data, and it is not directly verifiable. Social science research is generally an attempt to explain or understand the variability in a group of people.

References

Anderson, B. (1966) The Psychology Experiment: An Introduction to the Scientific Method. Belmont, CA: Wadsworth.

McDaniel, C. and R. Gates (1991) Contemporary Marketing Research. St. Paul, MN: West.

Carmines, E., and R. Zeller, (1979) Reliability and Validity Assessment. Beverly Hills: Sage.

Spector, P. (1981) Research Design. Beverly Hills: Sage.

Walonick, D. (1993) StatPac Gold IV: Marketing Research and Survey Edition. Minneapolis, MN: StatPac, Inc.