Teaching & Learning

Why evaluate?

• As a professional, you have a desire (indeed an obligation) to critically reflect on your activities, to judge the quality of those endeavours, and to seek quality improvements. 
• The university is accountable to its various stakeholders – particularly its students – for the quality of learning outcomes, and unit evaluation is one important component of the University’s quality assurance system.

 Quality assurance and quality improvement in terms of student participation and achievements are thus the reasons for evaluation. 

What to evaluate?

This will depend on the purpose of the evaluation – whether it be for course and/or teaching improvement (now, or for the next time the unit is offered), benchmarking against certain standards, or some other defined purpose.

Defining the purpose will determine the focus of your evaluation, which might include: 

• The impact of ones’ teaching
• The effectiveness of course design, including its content and assessment methods
• Adequacy of student support (academic, technical, administrative)
• Course delivery matters (e.g. with respect to web delivery).

When to evaluate?

In one sense, this question can be answered ‘all the time’. As teachers, we are continually gathering feedback informally and responding to, and adjusting, our teaching accordingly. However, such data tends to be gathered in an ad-hoc fashion, anecdotal and fleeting.

We therefore need to complement informal evaluation with more formal (i.e. structured and planned) evaluation.

Decisions on when to formally evaluate a unit depends on numerous factors, such as:

• At what stage the unit is in with respect to its development cycle. For example, it’s often prudent to evaluate a unit that has been offered for the first time
• The last time the unit was reviewed
• Faculty/School requirements, and review schedule
• The purpose of the evaluation: There are two general types of evaluation based on their timing.

Formative evaluation – carried out during the course of teaching, and largely for diagnostic purposes – i.e. checking the unit’s progress, and making adjustments accordingly

Summative evaluation – carried out at the completion of the unit, and largely for judgemental purposes – i.e. the extent to which unit aims and objectives have been achieved, and certain standards reached.

Findings can also inform quality improvements for the next teaching period.

Summative (end-of-semester) evaluations are also required by faculty for quality assurance reporting purposes (see University of Tasmania Teaching and Learning Quality Assurance Manual). As part of the reporting process, summative Student Evaluation of Teaching and Learning (SETL) is required.

Sources of feedback and information

Evaluation should involve gathering data from a range of sources, over a period of time. This includes:

• the students themselves (see below)
• other teaching and support staff involved in the unit
• your own records (e.g. self evaluation, log books)
• colleagues
• student work
• exam results (overall, and by assessment item)
• completion rate.

The University's SETL scheme should be seen as but one contributor to this process.

Data gathering techniques

A variety of methods are available, including:

• Print-based/online questionnaires (issued to individuals)
• Small, focus group meetings
• Individual interviews
• Observations (e.g. by a colleague)
• Analysis of records and other documentation.

For a range of evaluation methods check out the Evaluation Cookbook [PDF] produced by the Learning Technology Dissemination Initiative (LTDI) in the UK. The Cookbook not only tells you "how to", but also indicates the resources and indicative time needed to do use each method. The website is http://www.icbl.hw.ac.uk/ltdi/cookbook/

The method(s) chosen will depend on the sorts of data you wish to collect, which will in turn hinge upon the purposes of the evaluation. Generally you will wish to collect both qualitative and quantitative information.

The Student Evaluation of Teaching and Learning (SETL) questionnaires gather largely quantitative data, with added scope for student comments. In addition there are small focus group qualitative evaluations for units/courses with specific needs.

Data analysis and reporting

See Analysing the data from the UTAS Project Evaluation Toolkit (see 'Evaluating special projects' below).

Reporting options are covered under Reporting the evaluation findings from the Project Evaluation Toolkit.

Evaluating online teaching and learning

For information and advice see Evaluating an online course.

Online evaluation opportunities

The Web can be used to obtain timely data from students. Possibilities include:

• Online questionnaires/surveys
• Discussion on bulletin boards
• Customised feedback forms

There are a number of advantages to online evaluation:

• Easy and quick distribution and collection of data
• Because of its ease, generally higher response rates than equivalent print-based returns. Reminder notices can also be delivered promptly
• Quick data analysis, as quantitative data can be exported to a statistics package for analysis.

Contacts and further information

Contacts

For general advice and assistance to address teaching issues arising from SETL evaluation contact one of the educational staff at CALT via email enquiries@calt.utas.edu.au or phone: on 3504

SETL Administration: Michele Groombridge by email Michele.Groombridge@utas.edu.au

Further information and resources

Flinders University Teaching for Learning website - http://www.flinders.edu.au/teaching/quality/evaluation/evaluation.cfm

This site provides a framework for the evaluation of teaching, areas of teaching that can be evaluated, as well as suggested procedures for self-, peer-, and student evaluation.