Policy Perils of International Comparisons: The TIMSS Case.

Just as there is no magic bullet for improving an education system, so there is no clear path to be found by trying to imitate successful neighbors, Mr. Atkin and Mr. Black conclude. In late 1996, the Third International Mathematics and Science Study (TIMSS) began to release the results of its 1995 examination of the status of science and mathematics education in 45 countries. TIMSS measured student achievement. It also analyzed curricula, instructional practices, and certain teacher characteristics. The study found that U.S. student performance on the TIMSS eighth-grade tests was about average - comparable to Germany and other countries of northern Europe but markedly below Korea, Singapore, and Japan. At the time the initial data were released, the U.S. National Research Center for TIMSS launched a major effort to provide analysis of the results and to suggest policy recommendations. Coincident with the TIMSS research, the Organisation for Economic Cooperation and Development (OECD) initiated a somewhat different study of science, mathematics, and technology education.(1) Thirteen countries, all of which were part of the TIMSS analysis as well, participated. While TIMSS centered exclusively on the current status of education in the subjects in question, the OECD study focused entirely on innovations - changes that the countries are trying to make in their existing programs. The importance of the distinction for policy purposes is this: among the 13 countries that participated in the OECD study, there is universal unhappiness with the present state of science and mathematics education. Much of the dissatisfaction runs deep. Furthermore, its intensity is independent of the rank order of these countries in the TIMSS scores. All 13 countries (including Japan, the only Asian country in the OECD study) are trying to make changes in their existing programs. Some of these changes are profound. The purpose of this article is to compare and contrast the policy messages that can be inferred from these two studies. They differ. Neither study can confirm or disconfirm the other. Moreover, as we shall point out, the strong policy recommendations advanced by the TIMSS group find little basis in the TIMSS data themselves (at least those reported so far) and raise far more questions than the research can answer. The OECD Study The 13 countries that participated in the OECD study were Australia (Tasmania), Austria, Canada (British Columbia and Ontario), France, Germany, Ireland, Japan, the Netherlands, Norway, the United Kingdom (Scotland), Spain, Switzerland, and the United States. They generated 23 case studies of innovations, with Canada, Japan, and Norway producing two each and the United States, eight. The OECD is an intergovernmental agency of 26 countries (24 when the examination of innovations in science, mathematics, and technology began in the early 1990s) that conducts studies of interest to its members. No large investigation is initiated without approval of the relevant OECD governing body. These governing bodies are constituted of officially designated representatives of the participating countries. Thus OECD activities are linked closely to government policy concerns. Each participating country selected the specific innovation or innovations it wanted to examine. Since the context for reform differs from country to country, research approaches were chosen to illuminate those differences and their impact on innovations in content selection, strategies for change, teaching strategies, and student assessment. Because countries, not individuals, belong to the OECD, innovations with the strongest relevance for government policy makers tended to be chosen. In most instances, the education ministries, as the appropriate governmental bodies, made the choice. In some federal systems, such as Australia and Canada, the responsibility for selection was delegated to states and provinces. …