Relationship of Professional Development Participation to Observed Changes in Teacher Practice and Student Outcomes

Authors: Harriet Lamm, Margaret Hobson, Lee Sloan

Evaluation design: longitudinal quasi-experimental

Data collection:  Attendance data were collected for each professional development offering, including type of professional development content and cumulative hours by individual teacher.  Teacher practice data were collected from the teacher observations conducted by the AIMS Math Specialists using the Mathematics and Science Classroom Observation Protocol Survey (MASCOPS) developed by Dr. Carol Stuessy (2002).  The AIMS Mathematics Specialists were trained in the use of MASCOPS by Dr. Stuessy.  The MASCOPS instrument measures pedagogy as well as subject matter or content knowledge and is intended to help teachers recognize how their time is spent on course content and the pedagogy used to present content by providing a visual representation of classroom interactions.  The instrument examines (1) the involvement level of the teacher (i.e., moving from mostly teacher-directed learning to the students' responsibility for learning) and (2) the students' response and initiative level to what is being presented by the teacher as it affects their acceptance of responsibility for learning (i.e., moving from listening to minimal supervision of independent or group work).  Due to the instrument's design and the data which it provides, trends in the ways teachers were changing their teaching practices could be determined. 

The MASCOPS instrument was extremely helpful to teachers in showing them how they were using class time and the way in which they were providing instruction.  The instrument provides insight into the changes in the teachers' approaches in the classroom which yielded trends in the changes in teaching practices. The Mathematics Specialists observed teachers and, on a subsequent visit, discussed the results with each teacher using a color coded system showing the time usage and the focus of the learning process (teacher-directed or student-centered).  The MASCOPS data were examined to determine whether the teachers were changing the way they were teaching as their content knowledge and pedagogical knowledge were increasing.  

Since between 600 and 700 teachers teach mathematics in the partner ISDs, most of whom participated in AIMS professional development, observing all participants was not feasible.  The observation process included a minimum of two teachers per level (elementary, middle school, high school) per ISD.  Data on how many hours of AIMS professional development they attended was available and allowed determination of increases in pedagogical and content knowledge when second observations were made. 

Changes in teacher practice were indicated by student engagement in the classroom.  Scores of student engagement were weighted for length of observations.  A total of 245 classroom observations of 158 different teachers were conducted by the four Math Specialists.  Of these teachers, one was observed on five different occasions, four were observed four times each, 12 were observed three times, and 35 were observed twice. 

The region where AIMS schools are located (Region 2) covers 10,738 square miles in 11 Texas counties.  Of the 106,654 students in this region, 70% are Hispanic and 61.3% are economically disadvantaged.  They are served by 41 school districts.  In addition to the AIMS MSP, some of the school districts in this region have had the opportunity to participate in the NSF-funded South Texas Rural Systemic Initiative (STRSI).   Teachers in each of 41 school districts have equal access to professional development provided by Region 2 Education Service Center.  Only Corpus Christi ISD (student population 38,576) was large enough to offer additional professional development to teachers.  AIMS MSP offered professional development to nine of the school districts.  Eight other school districts participated in STRSI professional development, leaving 23 districts with no additional professional development opportunities.  AIMS school districts' student population ranged from 342 students to 4,760 students.  Question two of this study compared the 23 districts with only the basic Region 2 professional development opportunities to the nine AIMS districts which had the basic Region 2 professional development plus the AIMS professional development.

Student achievement was measured by the mathematics portion of the Texas Assessment of Knowledge and Skills (TAKS) given annually to all students in grades 3-11.  Scaled scores by campus were collected for 2004-2007.  Campuses were divided into elementary, middle school, and high school as designated by the Texas Education Agency. 

Data analysis Question 1:  The observation data were collected and the categories of teacher-directed and student-centered learning along with the time spent in each category were examined.  Since the observation lengths varied, the number of minutes at each category was divided by the total minutes observed.  The number of hours of professional development prior to the observation, the observer, the grade level, and the grant year were considered in the analyses.  A one-way analysis of covariance (ANCOVA) was used to test for differences between subjects on the number of professional development hours and the weighted observation averages.  The weighted observation was the dependent variable and the number of professional development hours prior to the observation was the covariate.  Controls were used for the observer, the grant year, and the grade level.  There were four observers, five grant years, and the grade levels were elementary, middle school, and high school.  The control for observers was done because inter-rater reliability was not controlled for during data collection.  Grant year was controlled for since the number of observations and the observers varied by year.  The three grade levels were controlled for because of the differences in classroom structure. This ANCOVA was performed for single observations and for multiple observations.    The observed participants may or may not have been presenting a lesson on the same mathematical content.

Data Analysis Question 2: Since the AIMS and the comparison schools had slightly different student populations, correlations between student characteristics and student achievement were examined.  Percent White, Hispanic, and economically disadvantaged were highly correlated to school student achievement scores.  The correlation between school size and achievement was not statistically significant.  Texas Education Agency policy does not allow reporting of scores for student groups with fewer than 30 students.  Since none of the schools had more than 30 Native American or Asian/Pacific Islander students, these groups were not included in data analysis.  Only three out of 45 elementary schools, four out of 32 middle schools, and four out of 26 high schools enrolled more than 30 African American students, so this subgroup was not considered.  Because the number and percent of other minority group students were so small, the percent of White students and Hispanic students were strongly correlated, ranging from r = -.985 to r = -.991.  More than 70% of the campuses were majority Hispanic, so percent Hispanic students was used in analysis of the data. Scaled scores were highly correlated with student characteristics of percent Hispanic, percent economically disadvantaged, and percent English language learners.  An ANCOVA was used to test the differences in scaled scores between AIMS schools and Region II schools by year controlling for percent Hispanic, economically disadvantaged, and English language learners.