The Education Technology Industry Network (ETIN) and Empirical Education Inc. recently released the Guidelines for Conducting and Reporting EdTech Impact research in U.S. K-12 Schools. These guidelines help clarify how research is conducted and how information is presented to users of edtech products based on the changes brought by the Every Student Succeeds Act (ESSA). In “Measuring Edtech Impact in the ESSA Era,” experts delved into the details of the guidelines.
The updated guidelines take into account nearly all edtech products today in the cloud, providing more access to teacher and student usage data. They also account for the timeline for compressed development of edtech products, and standards of evidence having changed to a more developmental scale with ESSA replacing No Child Left Behind (NCLB).
Districts are more frequently asking for their own student and teacher product usage data to perform their own studies. The structure and definitions provided by these new guidelines are useful in helping them obtain what they need and figure out how to do their evaluations.
The 4 Main Sections
The guidelines are divided into four main sections: Getting Started, Designing the Research, Implementing the Design, and Reporting the Results.
When getting started, using a logic model from the product provider is helpful in developing a model for how the product works. Logic models can show the factors that correlate with different outcomes.
When moving on to designing the research, look at the four levels of evidence defined by ESSA. The first step is to use a logic model. Then, look at a correlational study, run a comparison study, and run a randomized experiment. It’s common to run comparison studies, since randomized experiments can often be time consuming and expensive.
It is important to use caution when handling confidential information while implementing the design. With more personally identifiable student information available from both the school district and edtech products, privacy of edtech usage data has become a higher concern.
Lastly, when reporting the results, keep in mind all findings from edtech product evaluations should be made available. For example, publishing the best evaluation out of five that were conducted will not help the market learn. A report should also have enough detail to know if results apply to a particular context. These details tell schools if a product will work for them, and not just the general average.
For more details, download the full guidelines here.
About the Presenters
Denis Newman, lead author of ETIN’s guidelines for research on edtech impact, is the CEO of Empirical Education Inc., a research organization that conducts dozens of RCTs and other evaluations of school programs. He has 35 years of experience improving student-teacher learning processes and instructional technologies and is a pioneer in applying internet to student learning, professional development, and school administration. His Ph.D. in developmental psychology is from The City University of New York.
Andrew Coulson, Chief Data Science Officer, oversees the development of expansion strategies, product-to-market operations and leads a team of data analysts to conduct evaluation of MIND’s activities. Prior to this position, he led MIND’s Education Division for 12 years, helping to devise and execute strategies and programs to scale the organization’s reach to now support student learning in 45 states across the country. Before joining MIND, Coulson was a program officer for a major Orange County foundation, specializing in education. He also worked for 17 years in upper management as a STEM professional in high-tech manufacturing engineering, acquiring experience in operations, process engineering, reliability and technology transfer.
Bridget Foster has worked in all areas of the education market—from classroom teacher, to state level and industry leadership. As EVP & Managing Director of ETIN, she helps companies better understand the education market, so that they can grow their brands worldwide. She holds credentials in English, science, mathematics and school administration.
Join the Community
EdFocus: The EdMarketing Community is a free professional learning community that will help you connect with colleagues in the education industry to share information and resources, raise questions, and get advice.
This broadcast was co-hosted by edWeb.net and MCH Strategic Data.
The recording of the edWebinar can be viewed by anyone here.
[Editor’s note: This piece is original content produced by edWeb.net. View more edWeb.net events here.]
The former Editorial Director for both eSchool News and eCampus News, and was formerly the Managing Editor of eCampus News. Before working at eSchool Media, Meris worked as an assistant editor for The World and I, an online curriculum publication. She graduated from Kenyon College in 2006 with a BA in English, and enjoys spending way too much time either reading or cooking.