A post-occupancy evaluation for a new engineering facility at the University of Kansas (KU) illuminates the ways physical space influences STEM students’ experiences, and sets new standards for effectively studying project outcomes. According to Tim Reynolds, a principal with the Science & Technology studio of national architecture firm TreanorHL, “Too often, post-occupancy evaluations tend to be rather shallow—focused on things like lighting and furniture rather than actual program experience. We needed better research in order to determine if the work we were doing was making a difference.” [Tim Reynolds will be co-leading “The Fundamentals of Planning and Design of University Science and Engineering Facilities” pre-conference course on Nov. 15, in conjunction with the College and University Science and Engineering Facilities 2017 fall conference in St. Petersburg, Fla., on November 16-17.]
Recent expansions at KU, designed by TreanorHL, were driven by evidence supporting hands-on, collaborative learning. Research, conducted in 2014 by Scott Freeman with the National Academy of Sciences, indicates that students in traditional lecture-based environments are 1.5 times more likely to fail than those in classes with a more hands-on, student-centered pedagogy. “In other words, the intervention of active learning is so powerful that it would actually be doing harm to continue with the traditional method,” says Dr. Caroline Bennett, lead for the School of Engineering’s Course Redesign Initiative. In expanding KU’s engineering program, “We wanted to improve retention, progression, and graduation rates, and to see our students go on to productive careers.”
M2SEC and LEEP 2
The first phase of the expansion was the Measurement, Materials, and Sustainable Environment Center (M2SEC), a new, 46,000-sf building focused on theme-based research that includes a major focus on undergraduate students. “No matter what you’re researching—alternative fuels, composite materials, or some other aspect of sustainable design—you have space to do it,” says Reynolds.
The success of M2SEC led to the fast-tracking of facilities meant to connect M2SEC to the rest of the engineering complex. The 170,000-sf Learned Engineering Expansion Phase 2 (LEEP 2) building earmarks 41 percent of its assignable spaces to collaborative work areas and student success spaces, with the remaining space dedicated to research laboratories, core facilities, classrooms, and instructional laboratories. LEEP 2 features six active learning classrooms, set up for students to work collaboratively. In addition to multiple whiteboards around the room, each has a dedicated screen and device ports, as well as technology allowing work to be shown at every table and at the front of the room.
It was the use of the non-traditional learning environments that particularly intrigued TreanorHL.
Observing from the Inside Out
“As design professionals, we see a lot of things from the outside in. We wanted to see them from the inside out,” says Reynolds. The usual method of conducting such an assessment would be through a series of surveys. While TreanorHL planned to do that eventually, they believed that actually “living” in the building as designed would be a crucial first step.
A TreanorHL science and technology researcher who happened to be a recent KU alumna was the perfect person to seamlessly observe student interactions in real time. This was Tim Reynolds’s daughter, Brittany Reynolds. In attempting to formulate survey questions, “It started to occur to me that there was no tool out there that helps you quantify student activities in the kind of informal spaces that this facility had been designed to feature,” she says. “I thought about what I would want to be asked if I were still a student. I’d want the questions to speak to the uniqueness of the facility and the student body.”
Reynolds eventually made three separate, six-to-eight-hour trips to the campus on different days of the week, specifically avoiding late nights or days leading into exams. “I wanted to see what normal, baseline activity looked like.” Her ability to blend in with the students shaped her observational protocol. “I didn’t want to make them uncomfortable. I didn’t want them to modify their behavior, or worse, get up and leave.” She didn’t initiate conversations, but simply found a table or chair near students in the common areas and used her mobile phone to time specific activities.
She quickly determined that it wasn’t enough simply to observe and take notes. Those observations needed to be translated into usable data. Her previous experience with the Center for Teaching Excellence, and particularly with the work of Dr. Doug Ward, provided a framework: COPUS (Classroom Observation Protocol for Undergraduate STEM), used to quantify students’ in-class experiences. “I realized it could be used not just to track student activities during class, but also in informal learning spaces.”
OPSAILS
Out of COPUS, Brittany Reynolds developed a new protocol: OPSAILS (Observation Protocol for Student Activities in Informal Learning Spaces). The behaviors Ms. Reynolds tracked included the mundane (eating, sleeping, looking at mobile phones) but also the significant: interactions with classmates; group work (in person and online); individual study and homework; and brainstorming or working out problems on the many whiteboards provided by LEEP 2.
“I started by dividing activities into schoolwork- and non-schoolwork-related categories, with the understanding that socializing is an important part of student life and contributes to the experience of the space,” says Ms. Reynolds. “This was a great way to track how they moved through the process of working out problems, posing solutions, and then testing them. We were able to track not only what they were doing, but also how long they were spending on each activity.”
Doing group work consumed 48 percent of their time, a heartening statistic for administrators anxious for just this sort of collaboration. Additionally, 19 percent of students’ time focused on discussing topics related to schoolwork. “Groups tended to spend a lot more time studying; I think they were motivating each other,” says Ms. Reynolds. “Individual students had a tendency to become distracted more easily.” Only about 16 percent of the activity she observed could be categorized as purely social. “In general, it seemed that students in a group setting tend to do a collaborative, practice-oriented method of study.”
More Targeted Survey Questions
Ms. Reynolds’s observations informed the questions included in TreanorHL’s post-occupancy survey of 256 students (representing around 12 percent of the overall engineering student body). Her research made clear that the questions needed to be specific. “We had been lumping the informal learning environments together, but they’re designed to better serve different kinds of activities: individual or group study, or socializing.” TreanorHL could then target questions to students’ preferences with regards to specific spaces, and track time spent on activities there.
The QuestionPro-based survey comprised 50-80 questions (students were exempted from answering questions about which they had no experience) and took approximately 15 minutes to complete. “It came from the dean’s office, which was important,” says Tim Reynolds. “You get more candor and seriousness with that level of authority. It also precluded the email invitation from going into any junk or spam folders.” The survey was open for six weeks, giving participants ample time in which to complete it. In addition, the team offered an incentive in the form of the opportunity to win a $50 gift card.
TreanorHL’s survey included a combination of ratings scales, open-ended questions, comment solicitations, and imagery to keep participants engaged. Having separate sections was important, Brittany Reynolds says, to help orient participants and set expectations. “Each section had an introduction, and also told the student how many questions they would have to answer.” One particularly useful combination was to direct students to an open-ended comment box if they had indicated a negative level of satisfaction on a scaled question. “Those open-ended questions allowed us to better understand why something wasn’t working for a student,” she says.
But ultimately, it was TreanorHL’s unique and (entirely pro-bono) observation of how occupants actually use the facilities that marks its post-occupancy evaluation as a new standard for such studies. “There’s great value in what we learned, not just for KU or for our practice, but for other institutions struggling with space efficiencies and helping students succeed,” says Tim Reynolds.
Better User Experience Data Support Programmatic, Budgetary, and Design Decisions
In the case of KU’s M2SEC and LEEP 2 projects, TreanorHL’s method provided a more in-depth and revealing look at student life than a generic stand-alone survey could have done. While other practitioners might not be fortunate enough to have a similarly inconspicuous observer handy, Brittany Reynolds’s OPSAILS method has clear promise as a means of framing more meaningful post-occupancy studies.
Also in this case, the results indicated an encouraging alignment with the college’s pedagogical goals: 74 percent of students said that they interact often with students in the LEEP 2 classrooms, as opposed to 18 percent in traditional classrooms and a mere 6 percent in lecture halls, and 68 percent reported active participation in class and feeling engaged by the material.
“One of my favorite tidbits that I learned during this process was the extent to which the new facility is functioning as a ‘home’ for students, because that was very much one of the university’s goals,” says Tim Reynolds. “We had one student say that while he’d rather grab a beer after class, if his friends were in the non-traditional spaces, he’d stay.”
“It really has lifted the spirits of the whole school of engineering,” says Bennett. “Half of the students who responded to the survey believed that their study habits had improved because of the building, and 74 percent believe that it has positively impacted their academic success.” This experiential data, she says, is incredibly valuable in establishing the success of the project. “If students report that a structure is positively affecting their academic success, I think it is.”
By Liz Batchelder
