Introduction

Capturing instructional practices in Science, Technology, Mathematics, and Engineering (STEM) classrooms is critical to advance our understanding of effective teaching, identify paths to support faculty in improving their instructional practices, and to measure the impact of professional development.

In this website, we provide a new analytical tool – the COPUS Analyzer - aimed at characterizing instructional practices. This tool is based on an established observation protocol, the Classroom Observation Protocol for Undergraduate STEM (COPUS) . This observation protocol captures instructor’s and students’ behaviors that take place during 2-minute time intervals throughout a lecture. The COPUS Analyzer permits the visualization of occurrence and co-occurrence of these behaviors throughout a single or multiple classroom observations for one or several instructors. More importantly, the COPUS Analyzer automatically classifies classroom observations into specific instructional styles, called COPUS Profiles. The profiles represent a range of instructional styles, from teacher-centered to student-centered. They were developed from a data set that comprised 2,008 individual class periods collected from over 500 STEM instructors across at least 25 institutions in the United States.


Get Started!

Follow the following steps to start taking advantage of this tool:

    1. Read the Instructions tab – it is critical to ensure that data are entered correctly
    2. Fill out the Demographics tab – this information is used to enable various analytical tools that are pertinent to your specific needs
    3. Upload your data in the COPUS Analyzer tab.

Disclaimer: Any data submitted within this app will be saved by the research team on a secure server and used for research purposes. As such, it is critical that data uploaded be de-identified. The Institutional Review Board at Auburn University and University of Nebraska - Lincoln have approved this project.


If you have any questions regarding this tool or this project, please email either Marilyne Stains or Jordan Harshman

This material is based upon work supported by the National Science Foundation under Grant No. CAREER 1552448. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Instructions

Please watch on the video tutorial:

Data Format



This tool will accept data in three formats (you can download each template at the top of this page). All are .xlsx files.

    1. The Minute-by-Minute Template (xlsx) (download here) is for instructors/researchers who have COPUS data for every 2-minute intervals of each class. This can be one class for one instructor or multiple classes for multiple instructors. This template is also used if researchers have multiple coders COPUSing the same video(s) in a training context or for inter-rater reliability.
    2. The Summary Template (xlsx) (download here) is for instructors/researchers who have COPUS data for each classroom observation in percent of two-minute intervals. Each of the template files have a pre-filled in entry equivalent to one instructor's observation to exemplify what the data should look like.
    3. The Inter-rater Reliability Template (xlsx) (download here) is for researchers who have COPUS data for each classroom observation in every 2-minute intervals of each class. This template should only be used if you have multiple coders observing the same class(es) and you want to calculate the precent agreement and kappa values of those coders.

The format of the required variables are as follows:

    time: On a minute-by-minute file, you will need to enter 1, 2, 3... for each time block, starting at 1 and going until the end of class (25 time blocks for a 50 minute class). Do no enter actual times (i.e. 8:00AM) or timestamps (i.e. 0:01:00 - 0:03:00).
    instructor: Can be entered as strings (e.g. 'Instructor 1' or 'Frank') or numbers (e.g. 1, 2, 3...). Please ensure that data entered in this column de-identify the instructor (i.e, if John Smith taught the course do not enter John Smith in this column, rather, use numbers or an acronym).
    course_subject_code: Discipline or department course is in (e.g. 'Chemistry', 'CHEM', etc.).
    course_number: Actual course number (e.g. 101), can also be entered as course name (e.g. 'Intro Chemistry')
    class_date: Date the observation occurred in numeric format of MM/DD/YYYY (i.e. 12/31/2017). If you do not know the exact date, you can also enter '1' for first observation, '2', for second observation, and so on.
    semester_recorded: Semester in which course was taught (Fall, Spring, Winter, or Summer).
    year_recorded: Year in which course was taught.
    class_size: Approximate size of class, rounded to nearest 5 students.
    class_layout: Was the arrangement of the classroom 'Fixed' (chairs and/or tables secured to floor, immovable) or 'Flexible' (chairs and/or tables on wheels and movable).
    the 25 COPUS Behaviors: These columns should only contain numbers. For minute-by-minute data, only a 1 or a 0 should be entered for if the behavior was observed or not. For summary data, only numbers should be entered that represent the percent of a class that a behavior was observed and should not contain any symbols or text (i.e. 'NA', '*').

It is very important to note is that a given instructor can have multiple classes observed on the same day or the same class observed on multiple days or semesters. In order for the COPUS Analyzer to properly account for this, the instructor names needs to be exactly the same as it appears throughout the data.


Data Upload


Step 1

Bear in mind that once you upload your data, we will save a copy on our secure server for use in our research project. For this reason, make sure that the data you upload is de-identified.

Step 2

On the Demographics page, fill out all the required demographics and whether you are uploading data for single or multiple observations and which format (minute-by-minute template or summary template) you have. Make sure that you are using the correct template for each format.

Step 3

Once the demographics survey is completed, you will be able to click the 'Submit' button, which will reveal a new tab. By clicking the submit button, you also agree to have us use your submitted data for our studies.

Step 4

After clicking the 'Submit' button, click on the 'COPUS Analyzer' tab (or the 'IRR & Training' tab if you are doing inter-rater reliability). Once on the COPUS Analyzer page, click the 'Browse' button and navigate to the file that you want to upload (excel files only).


Data Analysis


Below is a list of the analyses that you can do with this tool. Please note that depending the format of the data you upload, not all of these options will be available (i.e. if you don’t have minute-by-minute data, we can't produce a minute-by-minute heat map).

Options for minute-by-minute files
    Generate a summary file: You can convert the minute-by-minute into a summary file that shows the percent of 2-minute intervals spent on each behavior for each instructor for each observation. Additionally, each observation will contain a cluster number that corresponds to the instructional profile that best matches that observation (for more information, see the link included in the profile assignment on the COPUS Analyzer tab). Simply click the Download Summary button to download the summary file.
    Heatmap of COPUS codes You can view and download customizable heat maps that are displayed for each of the 25 COPUS behaviors for each minute in an observation. If you want less/more COPUS behaviors, simply check the boxes to include/remove them. Click the Download HM Behaviors button to download a .png image file of the heatmap currently displayed.
    Heatmap of COPUS categories Heatmap of COPUS categories: You can view and download customizable heat maps that display 8 broad categories (4 student, 4 instructor) for each minute in an observation – this categorization is based on the work from Smith et al. (2014). If you want less/more COPUS collapsed codes, simply check the boxes to include/remove them. Click the Download HM Categories button to download a .png image file of the heatmap currently displayed.
    Generate a report for an instructor: If you have uploaded multiple observations per instructor, you can generate a report for each instructor that will include several visualizations for all observations of an instructor teaching in the selected semester and class, including:
      a) one graph that shows the percent of each COPUS behavior for all observations (for both minute-by-minute and summary files),
      b) a heat map that shows each COPUS behavior for each minute of class (for minute-by-minute files), and
      c) a heat map that shows each COPUS category for each minute of class (for minute-by-minute files).
    To generate this report in a pdf format, simply click the Download Report button.
Options for both summary and minute-by-minute files
    View percentage of each COPUS behavior for each class of a particular instructor: You can view the aggregate percentages for each class of a selected instructor. Simply click the Download Aggregate button to download a .png image file of the heatmap currently displayed.

Calculate inter-rater reliability


If you have a data set with multiple coders for the same instructor's observation(s), you can access inter-rater reliability information. Be sure to answer the question on the demographics that you are using the tool for inter-rater reliability/training. Once you upload your data, you will see two tables that look similar:

    - The top table shows all of the coders for the same observation (remember that the same coder and the same observation needs to be spelled exactly the same) . For example, if you have 4 coders for the same observation, the graph will show the total number of coders (up to 4) who coded each of the 25 behaviors throughout each minute of the class. If all 4 coded the behavior or none of the 4 coded the behavior, that is perfect agreement. If only 1, 2, or 3 marked a code, that represents disagreement. White areas represent perfect agreement and deeper reds represents greater disagreement.
    - The bottom table shows only the individual coders' coding, which you can select in the drop down menu for quick reference when differences arise.

Both of these tables can be exported in a .csv format (opens in Excel).

Finally, this program will calculate percent agreement and Fleiss' kappa for the selected observation. The values will show at the bottom of the second table. These measures are good for determining how similar all the coders rated the same video. If the graph isn't big enough to fit the numbers (this can happen for classes that are longer than 50 minutes), use the slider to increase the height of the plot.


Troubleshooting


In order for this tool work effectively, please ensure the following:

    1. You are using the correct template - see Data Format section at the top of this page.
    2. Variables within the template are in the exact same order as they appear in the template, have entered all required information, and have deleted rows 2 & 3 from the spreadsheet.
    3. You have selected the appropriate radio buttons on the Demographics Tab (combination of minute-by-minute vs. summary data for single vs. multiple observations).
    4. If you have multiple instructors, classes, and dates that should be grouped, they need to be spelled exactly the same (i.e. if your instructor is called 'Jordan' in one row and 'Jordan H' in another, they will be treated as two different instructors or if the class is 'CHEM 101' in one row and 'CHM 101' in another, they will be treated as two separate classes).
    5. Make sure that you do not have any duplicate rows (same instructor, day, class, semester, and minute, if applicable) as this will cause an error. If you only have instructor name but know that it's the same class on different dates, simply make up a class number/name and dates (but those columns need to be filled). If you're doing inter-rater reliability, you will only need to fill in the observer and instructor columns.
    6. This program might not generate an error message if it does not work. You can assume that the tool has encountered an error if it remains inactive for longer than 30 seconds. If this happens, check that your data is correct and try re-uploading your data.

If after taking these steps you find errors or wish to provide feedback regarding its functions, please contact Marilyne Stains or Jordan Harshman . If you cannot get the tool to work, please include the data you are trying to upload in your email.

Demographics

Please fill out the following fields. All fields are required before you are allowed to use the COPUS analysis tool. Click 'Submit' below when you are finished.

Thanks, your response was submitted successfully!

You may now click on 'COPUS Analyzer' tab (or the 'IRR & Training' tab if you are doing inter-rater reliability) to anlayze your COPUS data!

Abbreviations

The following codes are from the Classroom Observation Protocol for Undergraduate STEM (COPUS) paper previously published


Instructor Behaviors

Lec = Lecturing (presenting content, deriving mathematical results, presenting a problem solution, etc.

RtW = Real-time writing on board, doc. projector, etc. (often checked off along with Lec)

FUp = Follow-up/feedback on clicker question or activity to entire class.

PQ = Posing a non-clicker question to students (non-rhetorical).

CQ = Asking a clicker question (mark the entire time the instructor is using a clicker question, not just when first asked).

TAnQ = Listening to and answering student questions with entire class listening.

MG = Moving through class guiding ongoing student work during active learning task.

OoO = One-on-one extended discussion with one or a few individuals, not paying attention to the rest of the class (can be along with MG or TAnQ).

DV = Showing or conducting a demo, experiment, simulation, video, or animation.

Adm = Administration (assign homework, return tests, etc.).

TW = Waiting when there is an opportunity for an instructor to be interacting with or observing/listening to student or group activities and the instructor is not doing so.

TO = Other - explain in comments.


Student Behaviors

L = Listening to instructor/taking notes, etc.

Ind = Individual thinking/problem solving. Only mark when an instructor explicitly asks students to think about a clicker question or another question/problem on their own.

CG = Discuss clicker question in groups of 2 or more students.

WG = Working in groups on worksheet activity.

OG = Other assigned group activity, such as responding to indstructor question.

SAnQ = Student answering a question posed by the instructor with the rest of class listening.

SQ = Student asks question.

WC Engaged in whole class discussion by offering explanations, opinion, judgment, et. to whole class, often facilitated by instructor.

Prd = Making a predication about the outcome of a demo or experiment.

SP = Presentation by student(s)

TQ = Test or quiz

SW = Waiting (instructor late, working on fixing AV problems, instructor otherwise occupied.

SO = Other - explain in comments.


Instructor Categories (Collapsed Codes)

Presenting = Lec, RtW, or DV

Guiding = FUp, PQ, CQ, TAnQ, MG, OoO

Administrative = Adm

Instructor other = TW, TO


Student Categories (Collapsed Codes)

Receiving = L

Talking to class = SAnQ, SQ, WC, SP

Students working = Ind, CG, WG, OG, Prd, TQ

Student other = SW, SO

COPUS Analyzer

IMPORT ------------------

FILTER ------------------

BY CATEGORY ------------------
BY BEHAVIOR ------------------
DOWNLOAD ------------------
Download Summary Download HM Categories Download HM Behaviors Download Aggregate Download Report
BY CATEGORY ------------------

BY BEHAVIOR ------------------

ALL OBSERVATIONS FOR INSTRUCTOR ------------------

INSTRUCTIONAL PROFILE ------------------




COPUS Analyzer

IMPORT ------------------

FILTER ------------------

BY BEHAVIOR ------------------
DOWNLOAD ------------------
Download Summary Download Aggregate
ALL OBSERVATIONS FOR INSTRUCTOR ------------------

INSTRUCTIONAL PROFILE ------------------




COPUS Analyzer

IMPORT ------------------

BY CATEGORY ------------------
BY BEHAVIOR ------------------
DOWNLOAD ------------------
Download Summary Download HM Categories Download HM Behaviors
BY CATEGORY ------------------

BY BEHAVIOR ------------------

INSTRUCTIONAL PROFILE ------------------




COPUS Analyzer

IMPORT ------------------
INSTRUCTIONAL PROFILE ------------------

COPUS IRR

IMPORT ------------------

FILTER ------------------
RESIZE ------------------

DOWNLOAD ------------------
Download top graph Download bottom graph