Quantitative Research

Student state test data prediction analysis. Increasing accuracy of predictions by examining more factors.

Project Overview

Duration: May 2022

The problem: The current method of predicting student success on state tests is falling flat and not indicating key factors for the failures.

The goal: Discover a more encompassing metric with better accuracy and that provides actionable insights

My role: UX Researcher. I conducted primary and secondary research and utilized sheets to gain understanding of the research.

User Research

Current Methods

The standard metric currently used to predict success on the state exam is a MAP NWEA test score of 219 or higher to pass. Below 219 is a prediction of failure.

There are many limiting factors to this current prediction method

  • Students with frequent absences, truancy issues, suspension, or time at the alternative campus do not get tested for MAP as there are windows for testing and students do not have adequate class time to complete the MAP

  • Students with test anxiety perform differently on exams that they perceive "high risk" verses "low risk"

  • Students with Limited English Proficiency (LEP) have different accommodations available to them through MAP testing and state testing making the predictions far less accurate for LEP students.

MAP data predictions:

N=91,

students with no MAP data= 11*

predicted to pass= 59 students

predicted to fail= 21 students

actual pass= 71 students

actual fail= 19 students

(one student was absent during state testing to add up to the total of 91 students)


Percentage predicted to pass that did: 94.8%

Percentage predicted to pass that failed: 5.2%**

Percentage predicted to fail that passed: 14.3%

Percentage predicted to fail that did: 85.7%

Percentage of missing data students that passed: 36.4%

Percentage of missing data students that failed: 63.6% **

*students without data includes new move-ins, students in alternative campus during testing periods, truant students, and students whose data was not available from MAP website even after testing (most commonly due to student information not being programmed correctly into the computer system either by admin or the teacher)

**these are the most costly and dangerous statistics for our purpose: predicted passings that fail and unexpected failures. Failing the Texas state tests (STAAR) results in mandatory summer school of 30 hours under Texas HB 4545 which costs schools, tax payers, and of course, interrupts students' summer breaks. It also indicates class schedules that must be augmented the next year for remedial courses. For the 91 students in this data set, these missed calculations account for 11 students, a whopping 12% of the group set!

Research painted a picture of students who fell through the cracks and were not predicted through the current method. MAP definitely has strong points and is able to well identify passing students but is missing a significant number of failing students.

Pain Point 1

Newer students to the district/school do not have time to evaluated via MAP to predict state assessment success.

"I had 5 new students added to my roster within the last month of school."

Pain Point 2

Students with many absences, for any reasons (suspension, truancy, sickness, COVID) do not have time to be tested.


Pain Point 3

3 students with predictions to pass did not. These are significant as they represent surprises that change many things for each of these students.

"So I have to change my entire summer plans!?"

Creating a Better Factor

*All identifying information has been removed.


  1. Examine data. A quick color code for those that "Did Not Meet"/fail allowed me to quickly visualize results

  2. Patterns begin to emerge. I noticed that students who were now red had more "yes" columns.

  3. Question: What significance do these "yes" columns have and which ones matter?

One of the columns ended up not showing significant value or new information. The purple cells are the column representing students already designated as special education.

The column with blue cells and the three with yellow cells proved the most significant. These columns referred to (from left to right)

  • Students already enrolled in remedial math classes. To be in these classes students must qualify as either having failed the state exam in the past year, been within one question of failing, or be new to the country in the newcomer program

  • Limited English Proficient (LEP) meaning that English is not the student's first language and it still an emerging skill

  • "At-Risk" this is a factor that school districts determine that tend to include things like low socioeconomic status, parental incarceration, or time in the foster system

  • The final factor has to do with excessive absences. These could be from truancy, sickness, suspension, or time at the alternative campus. This also includes students this year who were bedside with sick parents and caring for siblings during parent illnesses.

Bringing the Data Together

I added a column in which I counted the factors. I cleverly named it "Count of factors"

My goal is to minimize the students that fall through the cracks and identify as many students as possible that are predicted to not pass the state exam to maximize available interventions and tutoring.

After reviewing the data I chose the count factor of 1 or 0 to predict passing and 2 or higher for failing.

Percentage predicted to pass that did: 98.1%

Percentage predicted to pass that failed: 1.9%**

Percentage predicted to fail that passed: 24.3%

Percentage predicted to fail that did: 75.7%

This new method of using Count of Factors to predict state test results reduced the number of students without a failure prediction that did fail from 11 students down to just 1! This represents 300 summer hours of student and teacher time.

More students were predicted to fail who actually passed but for teachers, that is a positive! That means that the interventions used with these students worked well enough to break through these Count of Factors!

Insights

What are the implications of the Count of Factors and how can we lower the Count of Factors for students?

Each factor is going to have its own set of recommendations

Factor 1: Remedial Course Enrollment

Of the 9 students predicted to fail using Count of Factors but passed, 6 of them were enrolled in the remedial course. That tells us that for students predicted to fail that early, specialized intervention and eager engagement in the remedial class have positive outcomes. Schools want to identify students quickly who need help and get it to them as efficiently as possible.

Factor 2: LEP

Students with Limited English Proficiency need to be in language rich classrooms that encourage use of English through low risk activities to promote language growth and decoding skills. There are many resources available to teachers and districts to expand their language output and input in LEP classrooms. Of the 9 predicted to fail who passed 4 of them were labeled LEP and all 4 were also in remedial math. I believe the extra math-language support provided these students the ability to break through

Factor 3: At-Risk

Schools need to be aware of the backgrounds that students bring with them. Students who are distracted in the classroom by factors at home will struggle to engage. Strong mental health outreach and social-emotional competent classrooms will encourage feelings of safety. Teachers need to be trauma informed. Engage to Connect and Empowered to Connect from the works of Dr. Karyn Purvis are great starting points for teachers to increase their trauma competencies to be a safe learning space for all students. All of the break through students were labeled At-Risk. Each of these students had strong relationships with at least one teacher along with positive peer relationships.

Factor 4: Absences

Absences are caused by many different factors but three fall into the same bucket. Truancy, suspension, alternative campus: students have to have "buy-in" in their classrooms to want to be at school and to want to treat their classmates with respect and their bodies with respect. (many suspensions and alternative campus placements come from fights with peers or drug usage/smoking).

Sicknesses: resources for students to stay connected as they are away from the classroom including a well laid out online classroom for each class.

Only three of the breakthrough students had a Count of Factors count for absences.

Points for iteration

  • I want to deep dive into the test reports and Count of Factors for newcomer students

  • Better teacher training for trauma backgrounds and classroom management

  • Relational focus through the first few days of school

What I learned and where to next?

I learned more about parsing through data and focusing on the pieces of significant impact. Through trial and error I used basic linear regression thinking to identify which factors were significant and which factors were superfluous to my goal knowledge.

  • Access to language classes for the whole family to strengthen language

  • Support for family to remove "at-risk" factor

  • Buy-in for students to want to be in class

  • Better academic support for suspended students and students out of the classroom

  • Low-risk opportunities for language practice for all students and especially LEP students