Research Tips: Classroom Observation Data Collection, Part II

Ordinary observation, although a natural process used to gather information, is not valid for research. Observers see events as they happen through their individual lenses, and research must cover a specific area of interest (Hitchcock & Hughes, 1995) in a systematic way. For that reason, a validation plan must be considered, implemented, and reported when using observation in a research context. Following are three observation techniques for data collection. Each technique is described, positive and negative aspects are noted, and a validation plan is provided.Structured ObservationStructured observation is observation using previously defined categories. The observer uses a form with instructions to note when and how often classroom activities exemplifying the category of interest occur. This requires rater training, a theoretical basis for category selection, and instrument validation similar to that used by questionnaires and tests.When categories are low inference, relatively objective and reliable data can be collected and analyzed. However, categories employed are often ambiguous and subject to multiple interpretations. Although one can document high use patterns, it does not follow that what is infrequent is insignificant. Furthermore, quantification of data cannot explain what patterns mean. Observers must be trained and retrained occasionally, and observer bias is possible. Since the data gathered is typically a count of certain features, some sort of statistical analysis is necessary. Ethnographically oriented researchers point out that using predetermined categories narrows attention, causing the observer to miss important details. Still, focusing attention is the goal of structured observation. Researchers who are uncertain of their exact focus should not use this type of instrument.Essentially, there are three options for validation: create a new instrument, use an existing instrument, or modify an existing instrument (see Weir & Roberts, 1994, who discuss six validation strategies). Also, create a log and record in detail every step of the process. Inter-rater reliability is the most common form of reliability reported, so rater training should be documented in detail. Be prepared to deal with the charge of observer bias since the presence of an observer will have some impact. The major issue is validation of the categories. Validation involves establishing a link between the category, a theory or body of knowledge that describes and defines the category, and the purpose of evaluation.Teacher DiaryA teacher diary is a log or journal document written at the end of a class session. It provides a record of what happened. When evaluating another teacher's class, a teacher diary serves primarily as a chronology of events and a repository for emotional reflection. When evaluating one's own class, it serves not only these two functions but also is used as field notes or as a source for descriptive data.Diary data are commonly used to collect ethnographic field notes and come from a long and accepted history. Such a document is important because, without it, a researcher's recollection of events will blur and merge until details are lost, and details provide credibility. Without a diary, researchers run the risk of using only quantitative data, such as test scores, which can be reliable and valid but are difficult to interpret alone.Possible drawbacks include that teacher diary data is remembered rather than directly observed data. The sooner an entry is made after the class-with no delays-the better. It is also hard to separate descriptive from evaluative comments. Comments such as, Good class today are evaluative rather than descriptive and do not specify what happened to make the class good. Observers have to be trained to record details of what actually happened separate from how they feel about the events. Feelings are important, but the events triggering those feelings must also be detailed. …