Rising Above a Gathering Storm of Data: Triangulating Findings Through Formative and Summative Evaluation Methodologies

Author: Sonya Martin

Internal evaluators used the Science Lesson Plan Analysis Instrument (SLPAI) to analyze unit lesson plans across four subscales: alignment with endorsed practices, cognitive and metagcognitive issues, sociocultural and affective issues, and portrayal and use of the practices of science. Each subscale contains two to eight items (a total of 21 items), which are individually scored and weighted in computing the subscale score and an overall score. The items contained in each subscale include:

• Alignment with endorsed practices: alignment with standards and awareness of science;
• Cognitive and metacognitive issues: education research, goal orientation, content accuracy, content presentation, pre-assessment, meaningful application, student reflection, and assessment;
• Sociocultural and affective issues: equity, student attitudes about science, student engagement, student participation, classroom discourse/fostering community of learners, appropriate use of technology, adaptability, and variety of instructional techniques;
• Portrayal and use of the practices of science: hands-on exploration, nature of science, student practitioners of scientific inquiry, and analytical skills development.

Baseline and Thesis/Capstone data are provided in Table 1 for all teachers completing both baseline and final lesson plans. When more than one reviewer rated lesson plans the scores were averaged. The following data is excerpted from an external evaluation report on this program (Scantlebury, Kahle, & Yue,, 2008).

Table 1. MISEP and MCEP Cohort A Teachers' Baseline 2005 and Final 2007 Scores on Unit Lesson Plans

Score Category	Baseline # of MISEP teachers	Capstone # of MISEP teachers	Baseline # of MCEP teachers	Thesis # of MCEP teachers
Below 35%	1	1	2	3
35-49%	2	0	3	2
50-59%	5	4	2	1
60-69%	3	6	0	1
70% and higher	6	8	1	5

Table 2 shows the average percent for each subscale and overall average percent for MISEP baseline and Capstone unit lesson plans. There were significant differences between the baseline and Capstone lesson plans for the overall SLPAI score, as well as for each subscale score. All findings indicate improvement in the development of lesson plans.

Table 2.SLPAI Overall and Subscale Means for the MISEP Teachers' Unit Plan Analysis: Cohort A, Baseline 2005 and Capstone 2007

SLPAI Subscales	Base/ Capstone	n	Mean	SD	t-value	p-value
Alignment with endorsed practices	Base	15	68	18	3.6	0.0037
	Capstone	15	87	10
Cognitive and metacognitive issues	Base	15	65	20	7.2	0.0001
	Capstone	15	78	9
Sociocultural & affective issues	Base	15	65	18	2.3	0.0159
	Capstone	15	78	20
Portrayal and uses of practices of science	Base	15	50	20	2.3	0.035
	Capstone	15	65	15
Overall average percentage	Base	15	58	14	4.2	0.0003
	Capstone	15	76	9

Table 3 compares average percent for each subscale and overall average percent for MCEP baseline and Thesis unit lesson plans. There were no significant differences for teachers' overall mean score for SLPAI and for the subscale, cognitive and metacognitive issues. There were significant differences between MCEP teachers' baseline and Thesis mean scores for the subscales alignment with endorsed practices, sociocultural and affective issues, and portrayal and uses of practices of science. These findings suggest that MCEP, compared to MISEP teachers were more reluctant to change the style and/or substance of their lesson planning.

Table 3. SLPAI Overall and Subscale Means for the MCEP Teachers' Unit Plan Analysis: Cohort A, Baseline 2005 and Thesis 2007

SLPAI Subscales	Base/ Thesis	n	Mean	SD	t-value
Alignment with endorsed practices	Base	7	46	28	2.3*
	Thesis	7	75	19
Cognitive and metacognitive issues	Base	7	52	16	1.3
	Thesis	7	68	31
Sociocultural & affective issues	Base	7	51	20	2.1**
	Thesis	7	78	28
Portrayal and uses of practices of science	Base	7	35	19	3.2***
	Thesis	7	69	21
Overall average percentage	Base	15	47	17	2.1
	Thesis	15	70	24

* p < .05; ** p<.01; ***p<0.001

Six SLPAI items were identified by the internal evaluators as ones that were a concern for one or both MCEP and MISEP teachers. Pre/post analysis of these six items was completed for MISEP and MCEP teachers using Baseline Portfolio (pre) and Thesis/Capstone (post) data. MISEP teachers showed improvement on four of the six items, while MCEP teacher means improved for all six items, as shown in Table 4.

Table 4. Comparison of SLPAI Subscale Means by Project: Cohort A, Baseline and Capstone/Thesis 2007

Item	Weight	Baseline MISEP Average	Capstone MISEP Average	Baseline MCEP Average	Thesis MCEP Average
Meaningful Application	2	1.3	1.9	0.6	1.4
Student Reflection	2	1.2	1.8	0.9	1.4
Assessment	3	1.3	1.6	1.0	1.1
Student Engagement	2	1.5	1.5	0.9	1.6
Nature of Science	3	0.7	0.7	0.0	0.9
Students as Practitioners of Scientific Inquiry	3	1.2	1.6	0.6	1.3

In summary, the higher scores in MISEP Capstone lesson plans may be attributable to the greater specificity of guidelines, scaffolding around curricular design models, longer duration of their lesson plan unit (MISEP, two weeks; MCEP, two to five days), and/or more extensive support by project staff. It is difficult to ascertain if teachers' low scores on the "Nature of science" subscale is due to a lack of implementation of this topic or to teachers failing to document their understanding and teaching of those issues in their lesson plans. Results from the "Views about science" subscale on the Teacher Questionnaire suggest that teachers have a solid understanding of the "Nature of science."