Evaluation is a critical stage in course development. From the perspective of instructional design (Davidson-Shivers & Rasmussen, 2006; Dick et al, 2005; Khan & Vega, 1997; Smith & Ragan, 2005), three key categories of criteria are often used to evaluate course development: effectiveness, efficiency, and appeal. 

   Brief descriptions of the three categories of evaluation criteria are presented in the following table:

Evaluation Criteria

Explanation

Effectiveness: mastery of goals and success of learning.

·         Determine whether essential learning has occurred and learners have achieved the instructional goals.

Efficiency: delivering in a timely or cost-saving manner.

·         Determine whether time and other resources are used expediently.

Appeal: gaining and maintaining learner attention and interest; usability.

·         Determine whether attention-getting and interest elements are appropriately used. 

·         Determine ease of use and navigation.


   Formative evaluation is a phase in which course design can be reviewed for its strengths and weaknesses so that the course can be appropriately revised before the course development is completed. Four categories of review can be used for formative evaluation: instructional goals, instructional content, technology, and message design. Course developers can apply these categories of review to the three evaluation criteria (i.e., effectiveness, efficiency, and appeal) to evaluate the course design during the process of course development.


   Generally, the instructional goals are evaluated for accuracy, clarity, completeness, and congruence with instructional content, instructional activities, and learning assessment. The instructional content is evaluated for accuracy, clarity, completeness, and appropriate sequence. Technology is evaluated for functional errors such as spelling, grammar, punctuation, word usage, links, and access. The message design is evaluated for direction clarity, instructional organization and sequence, minimization of distraction, and use of elements such as humor, language, and tone (Davidson-Shivers & Rasmussen, 2006; Khan & Vega, 1997; Lohr, 2003).

Based on the three evaluation criteria and the four categories of review, some sample questions are provided for the formative evaluation in Table 1 (Davidson-Shivers & Rasmussen, 2006; Khan & Vega, 1997; Lohr, 2003). The course developer need not answer all the questions. Instead, select questions that can be directly related to particular online instructional situations in course development. 

Table 1. Sample Formative Evaluation Questions for Course Development

Evaluation Criteria
and Categories

Sample Questions

Effectiveness

 

Goals

·         Are the goals accurate?

·         Are the goals and objectives clear?

·         Are goals and objectives achievable?

·         Are the goals and content appropriate for the delivery method?

Content

·         Is the information complete and covering the content properly?

·         Is there a match among instructional objectives, instructional content, instructional activities, and assessment tools?

·         Are reference citations provided?

·         Do the instructional activities promote different categories of learning (i.e., verbal information, intellectual skills, cognitive strategies, etc.)?

Technology

·         Do the technology applications function properly?

·         Can the instructional materials be easily accessed by students?

·         Are copyright and intellectual property not violated?

Message Design

·         Are directions clear?

·         Do supporting media components enhance learning without distracting?

·         Is humor used appropriately?

·         Is the time frame of the course appropriate?

Efficiency

 

Goals

·         Are the goals stated clearly and concisely?

·         Are the objectives stated clearly and concisely?

·         Is there alignment between instructional goals, instructional content, instructional activities, and learning assessment?

Content

·         Is the content information clearly and concisely presented?

·         Is it appropriate to the discipline?

·         Is it timely, up-to-date?

Technology

·         Does the course provide access to the instructor or other learners?

·         Does the course have a good organization and structure?

·         Do the technology applications function easily and efficiently?

Message Design

·         Is the organization and structure of the message coherent?

·         Are titles and subtitles used to organize the content?

·         Are synchronous and asynchronous activities planned for students?

Appeal

 

Goals

·         Are goals relevant to learners?

Content

·         Is the content interesting?

·         Is the content challenging?

Technology

·         Are spelling, grammar, and punctuation errors distracting?

·         Is the navigation easy?

Message Design

·         Are the message and media components pleasing?

·         Are the screens uncluttered with plenty of white space?

·         Is the vocabulary level and tone appropriate for the content and audience?

·         Are color, typeface, and emphasis used appropriately to enhance learning?

·         Do supporting media components enhance learning without distracting?

·         Do the media components function properly?

·         Does the course have good navigational design?

·         Are the icons easy to use and clear to students?