REL Southeast Products
The characteristics and education outcomes of American Indian students in grades 6-12 in North Carolina
The purpose of this study was to compare American Indian students in grades 6-12 in North Carolina to all other students in the same grades both within the same schools and statewide on student demographics, school characteristics, and education outcomes. The North Carolina State Advisory Council on Indian Education (SACIE) requested this research based on a prior report identifying achievement gaps between American Indian students and White students. The primary source of quantitative data for this study is longitudinal administrative data provided to the Education Policy Initiative at Carolina by the North Carolina Department of Public Instruction (NCDPI). These data include student-level outcomes for all students in grades 6-12 in North Carolina public schools for the school years 2010/11 through 2013/14. Outcomes considered include state test scores, attendance, retention in grade, advanced course taking, graduation rates, and disciplinary referrals. Quantitative analyses include all American Indian students in grades 6-12 in North Carolina public schools for school years 2010/11 through 2013/14. Students of other ethnicities in the same grades and years both within the same schools and statewide serve as comparison groups. Descriptive analyses compare averages for all student characteristics, school characteristics, and education outcomes for American Indian students compared to their within school and statewide peers. Regression analyses using multilevel modeling were used examine the extent to which controlling for student, school, and teacher characteristics accounts for differences in outcomes between American Indian students and their peers. The analyses found that American Indian students are demographically different from non-American Indian students statewide, but similar to other students attending the same schools. Schools attended by American Indian students are more likely to be rural and in the Coastal plain. American Indians also tend to attend schools that serve more economically disadvantaged students and more disadvantaged minority students. Across all middle school and high school standardized tests, American Indian students have lower average scores than other students statewide and within the same schools. American Indian students are absent more often on average than their peers both statewide and within the same school, are less likely to take advanced courses, and graduate at lower rates, but are equally likely to be retained in grade as their peers. When school and student demographics are held constant, the size of the gaps on most outcomes between American Indian students and their peers both within the same schools and statewide are substantially reduced.
The purpose of this descriptive study was to identify characteristics of charter school leaders in South Carolina, determine how they spend their work hours, understand the time they spend on challenges to their work, and learn who influences their schools' policies. REL Southeast researchers collaborated with the South Carolina Department of Education (SCDE) and other charter school policymakers and practitioners to develop a survey based on items from the school and principal questionnaires of the U.S. Department of Education, National Center for Education Statistics Schools and Staffing Survey. SCDE administered the survey to the 66 leaders in charter schools across the state operating during the 2014/15 school year. Forty leaders provided responses. Results indicate that the leaders have many similar demographic, educational, and employment characteristics and reasons for becoming charter school leaders. They worked an average almost 60 hours per week, spending more hours on activities related to communication with families and on school regulations and policies than on other tasks. Many of them spent time daily on school safety. A majority of the leaders were frequently challenged by state education agency requirements and services and sponsor intervention, but leaders were rarely or never challenged by staffing issues or board intervention. In addition, the leaders reported having more influence than any other entity over most of their schools' policies, except policies related to classroom instruction, academic guidance, athletics, and student assessment, which their staff influenced more and board membership policies that their board influenced more. This study was a first step toward understanding what characteristics and activities of charter school leaders in South Carolina may lead to improved school performance. Further research is needed to link school leadership characteristics and time management practices to school and student performance and other outcomes.
Professional Learning Communities Facilitator's Guide for the What Works Clearinghouse Practice Guide Foundational Skills to Support Reading for Understanding in Kindergarten Through 3rd Grade
The Regional Educational Laboratory (REL) Southeast developed a Professional Learning Community (PLC) Facilitators Guide to support educators in the implementation of recommendations from the What Works Clearinghouse. The practice guide focuses on the foundational reading skills that enable students to read words, relate those words to their oral language, and read connected text with sufficient accuracy and fluency to understand what they read. The practice guide, developed by a panel of experts comprised of researchers and practitioners, presents four recommendations that educators can use to improve literacy skills in the early grades.
Professional Learning Communities (PLCs) are a form of professional development in which small groups of educators with shared interests work together with the goals of expanding their knowledge and improving their craft. REL Southeast developed PLC materials focused on the practice guide that were designed to assist a literacy leader in guiding a professional learning community in applying the recommendations from the practice guide. The materials include a facilitator's guide, participant activities, and videos. The facilitator's guide includes a framework for facilitators to conduct each of the ten PLC sessions. It also includes participant activities, discussion questions, small- and whole-group activities, and implementation and reflection activities. The participant's activities include reflection questions, lesson plan examples and templates, video-viewing guides, and sharing opportunities. The videos illustrate practices presented in the foundational reading skills practice guide.
This report describes the results of a REL Southeast study comparing short- and longer-term student successes after completion of online credit recovery courses compared to student successes after completion of other credit recovery options, such as traditional face-to-face courses and summer school courses. Credit recovery refers to when a student fails a course and then retakes the same course to earn high school credit. This research question was motivated by the growing importance of online learning in traditional public school settings and a desire on the part of many stakeholders to understand better how students are adjusting to that transition. The data for this study covered eleven core high school courses (courses required for graduation) taken between 2008/09 and 2011/12 in North Carolina. The study compares the likelihood of a student: (a) succeeding on the state end-of-course test for the recovered course; (b) succeeding in the next course in a recovered course sequence (for instance, in English II after English I); (c) remaining in school after credit recovery; and (d) graduating and graduating on time. Results suggest that there was little difference between the short-term success rates of students who completed state-supported online credit recovery and students who completed other credit recovery options. However, on measures of longer-term success, students who completed state-provided online credit recovery courses and did not subsequently drop out were more likely than other credit recovery students to graduate on time. Among credit recovery participants in state-provided online courses, Black students were less likely to reach proficiency in their recovered courses but more likely than their peers to succeed in later coursework after their online experience. Because of limitations in the analyses possible with available data, it is not possible to directly attribute these outcomes to participation in online credit recovery, but the results do point toward intriguing and potentially beneficial areas for future, more rigorous study.
The Grades 3-8 Self-Study Guide for Implementing Literacy Interventions was developed to help district- and school-based practitioners conduct self-studies for planning and implementing literacy interventions. It is intended to promote reflection about current strengths and challenges in planning for implementation of literacy interventions, spark conversations among staff, and identify areas for improvement. This guide provides a template for data collection and guiding questions for discussion that may improve the implementation of literacy interventions.
District and state education leaders and teachers frequently use assessments to identify students who are at risk of performing poorly on end-of-year reading achievement tests. This study explores the use of a universal screening assessment of reading skills for the identification of students who are at risk for low achievement in mathematics and provides support for the interpretation of screening scores to inform instruction. The study results demonstrate that a reading screening assessment predicted poor performance on a mathematics outcome (the Stanford Achievement Test) with similar levels of accuracy as screening assessments that specifically measure mathematics skills. These findings indicate that a school district could use an assessment of reading skills to screen for risk in both reading and mathematics, potentially reducing costs and testing time. In addition, this document provides a decision tree framework to support implementation of screening practices and interpretation by teachers.
This guide describes core ideas for colleges to consider when developing a screening tool for estimating college readiness. A key focal point within the guide is a discussion of ways to improve how well a screening tool can identify individuals needing remedial or developmental education along with key considerations that a user or developer of such a tool must address. Specifically, the following steps are discussed:
- Creating an operational definition of success and college readiness
- Selecting potential predictors of college readiness
- Prioritizing types of classification error
- Collecting and organizing the necessary data
- Developing predictive models
- Evaluating the screening results and selecting the final model
This literature review searched the peer-reviewed studies of reading comprehension instructional practices conducted and published between 1994 and 2014 and summarizes the instructional practices that have demonstrated positive or potentially positive effects in scientifically rigorous studies employing experimental designs. Each study was rated by the review team utilizing the What Works Clearinghouse standards. The review of the literature resulted in the identification of 7,144 studies. Of these studies, only 111 met eligibility for review. Thirty-three of these studies were determined by the study team to have met What Works Clearinghouse standards. The 33 studies represented 29 difference interventions or classroom practices. Twelve of these studies demonstrated positive or potentially positive effects. These 12 studies are described and the commonalities among the studies are summarized.
This project explored the type of extended instructional time referred to as double-dosing. The first part of the study examined what the literature says about double-dosing. The second part of the study provided descriptive results about the prevalence and forms of double-dosing in North Carolina.
The purpose of this study was to examine (a) how teachers and school staff individually administer computer-adaptive assessments of literacy to English learner students in grades 3-5, and (b) how they use the assessments to monitor students' growth. Because adaptive assessments maximize precision of information while minimizing time spent gaining it, they are particularly valuable for students whose performance is outside typical grade-level norms such as English learner students. Three elementary schools with high proportions of English learner students participated in the study. Participating students were at the two lowest levels on the state oral language proficiency measure. At the beginning of the year there were 117 participating students and by the end of the year 102 remained at the same school. To address the first question, Regional Educational Laboratory (REL) Southeast staff observed the September training and the fall, winter, and spring administration of the Florida Center for Reading Research Reading Assessment (FRA). To address the second question, teachers and school staff individually administered the FRA to participating students in the fall, winter, and spring. They discussed their observations of students' performance during test administration and students' score reports with REL staff after each assessment period. Findings indicated that teachers in grades 3-5 can be trained to individually administer computer-adaptive assessments of literacy to their English learner students three times a year and to participate in data chats after each assessment period to discuss translation of scores to instruction. The report provides recommendations that may aid districts in implementing such adaptive assessments of literacy to monitor the progress of English learner students.
This Self-study Guide for Implementing High School Academic Interventions was developed to help district- and school-based practitioners plan and implement high school academic interventions. It is intended to promote reflection about current district and school strengths and challenges in planning for implementation of high school academic interventions, spark conversations among staff, and identify areas for improvement. The guide provides a template for data collection and guiding questions for discussion that may improve the implementation of high school academic interventions and decrease the number of students failing to graduate from high school on time.
Beginning with the 2012/13 school year, Florida law required that the 100 lowest-performing elementary schools in reading extend the school day. This study examined how the lowest performing schools implemented the extended school day policy and the trends in school reading performance among the lowest performing schools and other elementary schools. The lowest-performing schools were located throughout Florida and on average, were smaller but served higher proportions of minorities and higher proportions of students receiving free or reduced-price lunch compared to other elementary schools. The lowest-performing schools reported increasing the number of minutes of reading instruction provided to students, increasing staff, and providing different instruction in the extra hour than during other reading instructional blocks. An increase in reading performance was observed for the lowest-performing schools the year the extended school day was implemented. However, this increase did not exceed what would have been expected in the absence of the required increase in reading instruction.
Can scores on an interim high school reading assessment accurately predict low performance on college readiness exams?
The purpose of this study was to examine the relationship between measures of reading comprehension, decoding, and language with college-ready performance. This research was motivated by leaders in two Florida school districts interested in the extent to which performance on Floridaâ€™s interim reading assessment could be used to identify students who may not perform well on the Preliminary SAT/National Merit Scholarship Qualifying Test (PSAT/NMSQT) and ACT Plan. One of the districts primarily administers the PSAT/NMSQT and the other primarily administers the ACT Plan. Data included the 2013/14 PSAT/NMSQT or ACT Plan results for students in grade 10 from these districts, as well as their grade 9 results on the Florida Assessments for Instruction in Reading - Florida Standards (FAIR-FS). Classification and regression tree (CART) analyses formed the framework for an early warning system of risk for each PSAT/NMSQT and ACT Plan subject-area assessment. PSAT/NMSQT Critical Reading performance is best predicted in the study sample by a studentâ€™s reading comprehension skills, while PSAT/NMSQT Mathematics and Writing performance is best predicted by a studentâ€™s syntactic knowledge. Syntactic knowledge is the most important predictor of ACT Plan English, Reading, and Science in the study sample, whereas reading comprehension skills were found to best predict ACT Plan Mathematics results. Sensitivity rates (the percentage of students correctly identified as at risk) ranged from 81 percent to 89 percent correct across all of the CART models. These results provide preliminary evidence that FAIR-FS scores could be used to create an early warning system for performance on both the PSAT/NMSQT and ACT Plan. The potential success of using FAIR-FS scores as an early warning system could enable districts to identify at-risk students without adding additional testing burden, time away from instruction, or additional cost. The analyses should be replicated statewide to verify the stability of the models and the generalizability of the results to the larger Florida student population.
Can scores on an interim high school reading assessment accurately predict low performance on college readiness exams?
This "Stated Briefly" report is a companion piece that summarizes the results of another report of the same name. The purpose of this study was to examine the extent to which performance on Florida's interim reading assessment could be used to identify students who may not perform well on the Preliminary SAT/National Merit Scholarship Qualifying Test (PSAT/NMSQT) and ACT Plan. Data included the 2013/14 PSAT/NMSQT or ACT Plan results for students in grade 10 from two districts, as well as their grade 9 results on the Florida Assessments for Instruction in Readingâ€”Florida Standards (FAIR-FS). PSAT/NMSQT Critical Reading performance is best predicted in the study sample by a student's reading comprehension skills, while PSAT/NMSQT Mathematics and Writing performance is best predicted by a student's syntactic knowledge. Syntactic knowledge is the most important predictor of ACT Plan English, Reading, and Science in the study sample, whereas reading comprehension skills were found to best predict ACT Plan Mathematics results. Sensitivity rates ranged from 81 percent to 89 percent correct across all of the models. These results provide preliminary evidence that FAIR-FS scores could be used to create an early warning system for performance on both the PSAT/NMSQT and ACT Plan.
The Self-study Guide for Implementing Early Literacy Interventions is a tool to help district and school-based practitioners conduct self-studies for planning and implementing early literacy interventions for kindergarten, grade1 and grade 2 students. This guide is designed to promote reflection about current strengths and challenges in planning for implementation of early literacy interventions, spark conversations among staff, and identify areas for improvement. This self-study guide provides a template for data collection and guiding questions for discussion.
A Systematic Review of the Relationships Between Principal Characteristics and Student Achievement
This systematic review of the relationships between principal characteristics and student achievement was created for educators, administrators, policy-makers, and other individuals interested in a comprehensive catalogue of research on relations between principal characteristics and student achievement. It synthesizes what is known about associations between principal characteristics and student achievement; specifically it summarizes the studies, highlights the effects found by the studies, and describes the steps of the systematic review process used. Of the 52 studies included in the comprehensive review, only one used an experimental design, and a positive effect was found. An additional 38 quantitative and two mixed method studies provided evidence that some principal characteristics are positively correlated with student achievement. However, causal relationships could not be established. The remaining eleven qualitative studies mirrored the quantitative findings.
Comparing Success Rates for General and Credit Recovery Courses Online and Face to Face: Results for Florida High School Courses
This report describes the results of a REL Southeast study comparing student success in online credit recovery and general courses taken online compared to traditional face-to-face courses. Credit recovery occurs when a student fails a course and then retakes the same course to earn high school credit. This research question was motivated by the high use of online learning in the Southeast, particularly as a method to help students engage in credit recovery. The data for this study covered all high school courses taken between 2007/08 and 2010/11 in Florida (excluding Driver's and Physical Education). The study compares the likelihood of a student earning a C or better in an online course as compared to a face-to-face course. Comparisons for both general and online courses include those courses taken for the first time and credit recovery courses. The results show that the likelihood of a student earning a grade of C or better was higher when a course was taken online than when taken face-to-face, both for general courses and credit recovery courses. Most subgroups of students also had higher likelihood of success in online courses compared to face-to-face courses, except that English language learners showed no difference in outcomes when taking credit recovery courses online. However, it is not possible to determine whether these consistent differences in course outcomes are attributable to greater student learning, other factors such as differences in student characteristics, or differences in grading standards.
The objective of this project was to create a tool, in the form of a user-friendly checklist, that will support school leaders as they determine which components of literacy and which aspects of instruction they should observe as they conduct walkthroughs during literacy instruction in K-3 classrooms. The tool is intended for school leaders' use in order to:
- enhance literacy instruction knowledge for both school leaders and K-3 teachers;
- provide an effective way to communicate with K-3 teachers expectations about literacy instruction;
- specify consistent language regarding literacy components and instructional strategies; and
- be the basis of communication and collaboration among school leaders and teachers, which, according to research, can improve student achievement.
Development and Examination of an Alternative School Performance Index in South Carolina
The purpose of this study was to examine the extent to which the measures that make up each of the three separate accountability indices of school performance in South Carolina could be used to create an overall, reliable index of school performance. Data from public elementary, middle, and high schools in 2012/13 were used in confirmatory factor analysis models designed to estimate the relations between the measures under different specifications. Four different factor models were compared at each school level, beginning with a one-factor model and ending with a bi-factor model. Results from the study suggest that the measures which currently are combined into three separate indices of school performance can instead be combined into a single index of school performance using a bi-factor model. The reliability of the school performance general factor estimated by the bi-factor model ranged from .89 to .95. Using this alternative school performance rating, the study found that approximately 3 percent of elementary schools, 2 percent of middle schools, and 3 percent of high schools were observed to statistically outperform their predicted performance when accounting for the school's demographic characteristics. These schools, referred to as schools beating the odds, were found in most of the demographic profiles which represent South Carolina schools. The results of this study can inform decisions related to the development of new accountability indices in South Carolina and other states with similar models.
A Descriptive Analysis of the Principal Workforce in Florida Schools
This What's Happening report describes the results of a REL Southeast study using the Florida Department of Education staffing database to describe the demographic make-up, Florida Educator Certificate coverages, career paths from 2001/02 to 2011/12, and evaluation ratings of Florida's educational leaders (i.e., assistant principals and principals) in the 2011/12 school year. The sample consisted of 4273 assistant principals and 2979 principals. The study consists of a cross-sectional component and a longitudinal component. The cross-sectional component analyzed the demographic characteristics of the 2011/12 school year educational leaders. The longitudinal component utilized a retrospective cohort analysis to track the certification coverages and career paths of the educational leaders from 2001/02 to 2011/12. Analyses of frequencies, measures of central tendency and distribution, and cross tabulations were examined to provide insight into the demographic characteristics, training, professional experiences, performance, and career paths of Florida's educational leaders in the 2011/12 school year.
Using Administrative Data for Research: A Companion Guide to "A Descriptive Analysis of the Principal Workforce in Florida Schools"
The Applied Research Methods report outlines the processes and procedures used to analyze data from the Florida education staffing database. The report also provides directions and examples for conducting similar work using administrative databases.
Comparing methodologies for developing an early warning system
The purpose of this report was to explicate the use of logistic regression and classification and regression tree (CART) analysis in the development of early warning systems. It was motivated by state education leaders' interest in maintaining high classification accuracy while simultaneously improving practitioner understanding of the rules by which students are identified as at-risk or not at-risk readers. Logistic regression and CART were compared using data on a sample of grades 1 and 2 Florida public school students who participated in both interim assessments and an end-of-the year summative assessment during the 2012/13 academic year. Grade-level analyses were conducted and comparisons between methods were based on traditional measures of diagnostic accuracy, including sensitivity (i.e., proportion of true positives), specificity (proportion of true negatives), positive and negative predictive power, and overall correct classification. Results indicate that CART is comparable to logistic regression, with the results of both methods yielding negative predictive power greater than the recommended standard of .90. Details of each method are provided to assist analysts interested in developing early warning systems using one of the methods.
Summer Reading Camp Self-study Guide
This guide is designed to facilitate self-studies of planning and implementation of state-required summer reading camp programs for grade 3 students who scored at the lowest level on the state reading assessment. It provides a template for data collection and guiding questions for discussion that may improve instruction and increase the number of students meeting the grade-level standard by the end of the summer reading camp.
The guide is composed of a scoring guide and consensus rating form. The scoring guide includes guiding questions and potential sources of evidence. After completion of the scoring guide, a facilitator will guide the self-study team through a consensus rating process to come to agreement on the current status of implementation in the district or school and planning for next steps. The content of the scoring guide is based on eight areas that research indicates are important for summer reading camp implementation: teacher effectiveness and qualifications; student selection and enrollment; instructional time; content and instruction; assessment selection and data use; mentoring and/or paraprofessional use; learning environment; and communication with administration, staff, and parents.
Using Evidence-Based Decision Trees Instead of Formulas to Identify At-Risk Readers
The purpose of this study was to examine whether the early identification of students who are at-risk for reading comprehension difficulties is improved using logistic regression or classification and regression tree (CART). This research question was motivated by state education leaders' interest in maintaining high classification accuracy while simultaneously improving practitioner understanding of the rules by which students are identified as at-risk or not at-risk readers. Logistic regression and CART were compared using data on a sample of grades 1 and 2 Florida public school students who participated in both interim assessments and an end-of-the year summative assessment during the 2012/13 academic year. Grade-level analyses were conducted and comparisons between methods were based on traditional measures of diagnostic accuracy, including sensitivity (i.e., proportion of true positives), specificity (proportion of true negatives), positive and negative predictive power, overall correct classification, and the receiver operating characteristic area under the curve. Results indicate that CART is comparable to logistic regression, with the results of both methods yielding negative predictive power greater than the recommended standard of .90. The comparability of results suggests that CART should be used due to its ease in interpretation by practitioners. In addition, CART holds several technical advantages over logistic regression.
Beating the Odds: Finding Schools Exceeding Achievement Expectations with High-Risk Students
State education leaders are often interested in identifying schools that have demonstrated success with improving the literacy of students who are at the highest level of risk for reading difficulties. The identification of these schools that are â€œbeating the oddsâ€ is typically accomplished by comparing a schoolâ€™s observed performance on a particular exam, such as a state achievement exam, with how the school would be expected to perform when taking into account its demographic characteristics including the percentage of students classified as economically disadvantaged, minority, or as an English language learner. This study used longitudinal data from the Florida Department of Education on grade 3 public school students for the academic years 2010/11-2012/13 to determine which schools are exceeding student achievement expectations, and what demographic similarities exist between schools that are exceeding expectations and other schools.
Testing the Importance of Individual Growth Curves in Predicting Performance on a High-Stakes Reading Comprehension Test in Florida
REL Southeast at Florida State University evaluated student growth in reading comprehension over the school year and compared the growth to the end-of-year Florida Comprehensive Assessment Test (FCAT). Using archival data for 2009/10, the study analyzes a stratified random sample of 800,000 Florida students in grades 3-10: their fall, winter, and spring reading comprehension scores on the Florida Assessments for Instruction in Reading (FAIR) and their reading comprehension scores on the FCAT. This study examines the relationship among descriptive measures of growth and inferential measures for students in grades 3-10 and considers how well such measures statistically explain differences in end-of-year reading comprehension after controlling for student performance on a mid-year status assessment. Student differences in reading comprehension performance were explained by the four growth estimates (measured by the coefficient of determination, R2) and differed by status variable used (performance on the fall, winter, or spring FAIR reading comprehension screen).
Summary of Research on the Effectiveness of Math Professional Development Approaches
The study identified and screened 910 research studies in a comprehensive literature search for effectiveness studies of math professional development approaches. Of these 910 studies, 643 examined professional development approaches related to math in grades K-12 and were conducted in the United States. Of the 643 studies, 32 focused primarily on math professional development provided to teachers and used a research design for examining effectiveness. Five of those were determined to have met WWC evidence standards (version 2.1) either with or without reservations. And of those five, only two found positive effects on students' math proficiency. Thus, there is very limited causal evidence to guide districts and schools in selecting a math professional development approach or to support developers' claims about their approaches.
Evaluating the Screening Accuracy of the Florida Assessments for Instruction in Reading (FAIR)
This report analyzed student performance on the FAIR reading comprehension screen across grades 4-10 and the Florida Comprehensive Assessment Test (FCAT) 2.0 to determine how well the FAIR and the 2011 FCAT 2.0 scores predicted 2012 FCAT 2.0 performance. The first key finding was that the reading comprehension screen of the Florida Assessments for Instruction in Reading (FAIR) was more accurate than the 2011 Florida Comprehensive Assessment Test (FCAT) 2.0 scores in correctly identifying students as not at risk for failing to meet grade-level standards on the 2012 FCAT 2.0. The second key finding was that using both the FAIR screen and the 2011 FCAT 2.0 lowered the underidentification rate of at-risk students by 12-20 percentage points compared with the results using the 2011 FCAT 2.0 score alone.