Week 5&6 Evaluation Methods

Ok - here goes!
1. What type of evaluation i think will best suit my project?
A few years ago i did a job for a TEI (tertiary education institute) creating a marine studies programme for school (year 13) students that would then feed into a science degree. At the time we discussed putting the programme on line. I am interested in doing a front end needs analysis to evaluate the need for this and how it might best be designed. The front end or needs analysis is the most appropriate type evaluation as the elearning programme has not been developed yet!

2. Articles providing theory and models to assist with my front end needs analysis
I went hunting for articles that would be helpful in this project. I have to say i was amazed at how few articles I found (after A LOT of searching) that actually address evaluations at the front end. Most seem to look at evaluating existing programmes. It was interesting that many of relevant articles that i did find were focussed or written by/for business organisations. Which left me pondering the paradigms of the business and education worlds - where businesses are more likely to ascertain needs prior to development than the tertiary sector!?

Anyway - here is a brief review of two articles that i found that were useful in the context of my proposed project:

1. Spiros Ap. Borotis, and Angeliki Poulymenakou. (2004) E-Learning readiness components: key issues to consider before adopting e-Learning interventions. In eLearn 2004 Conference Proceedings, pp. 1622-1629, November 2004.

Borotis and Poulymenakou (2004) reviewed other studies and then defined seven components of e-learning readiness. These are:
1. Business Readiness: referring to the link of organisational business priorities and characteristics, to e-Learning efforts.
2. Technology Readiness: primarily the technical infrastructure.
3. Content Readiness: concerning e-Learning content material.
4. Training Process Readiness: or the ability of organisations to organise, analyse, design, develop, implement and evaluate a concrete training program
5. Culture Readiness: or an organisations perceptions and cultural parameters concerning e-Learning adoption and use.
6. Human Resources Readiness: the availability and set-up of the human-support system including the receptivity and the prerequisites of humans to learn successfully in the new environment, and facilitate itsoperation.
7. Financial Readiness: or in other words the budget allocation. Although e-Learning helps to decrease costs in training, it requires a significant investment to initialize and maintained.

Each of these components is then explored in more depth. In terms of paradigms... ummm.... I guess i can see strands of the Analytic Empirical Positive Quantitative Paradigm - as the models is pretty straight forward and looks at individual elements in isolation from their environment and in this respect reflects the rational scientific approach.

Overall I found the article helpful in developing a front end evaluation model – even though it has been written from the perspective of business rather than education institution e-learning development. Increasingly TEIs are taking a business case / business model approach to new developments which makes much of the findings of the paper relevant to the tertiary sector. I can see a place for at least an element of all seven steps in the evaluation that i am designing for my project.

2. Cook, D.A and D M Dupras (2004) A Practical Guide to Developing Effective Web based Learning, JGIM Volume 19, June 2004.

This article aims to assist in the development of effective educational websites. The authors offer a ten step model for the e effective development of web based learning.

The Ten Steps to Effective Web-based Learning are:
1. Perform a needs analysis and specify goals and objectives
2. Determine your technical resources and needs
3. Evaluate preexisting software and use it if it fully meets
your needs
4. Secure commitment from all participants and identify and
address potential barriers to implementation
5. Develop content in close coordination with website design
6. Encourage active learning—self-assessment, reflection,
self-directed learning, problem-based learning,
learner interaction, and feedback
7. Facilitate and plan to encourage use by the learner
8. Evaluate—both learners and course
9. Pilot the website before full implementation
10. Plan to monitor online communication and maintain the
site by resolving technical problems, periodically

Whilst there are elements of the other steps that i can adapt to use in my evaluation plan: the preparatory steps (pre-development) are most relevant to what I am looking at doing and include:

Step 1. Perform a needs analysis and specify goals and objectives
This includes: a needs analysis, including problem identification, assessment of learners’ needs, and assessment of the teaching environment. The authors discuss the importance of determining learners’ perceived educational needs and preferences as well as evaluating resources and barriers in the teaching environment. The paper suggests the use of the needs analysis to “develop goals and objectives that address the gap between current and ideal performance, taking into account available resources and learner
perceptions” (Cook and Dupras, 2004: 699).

Step 2. Determine your technical resources and needs
Cook and Dupras (2004) suggest that e effective course design requires an understanding of both the subject and the instructional medium and as such suggest a multidisciplinary approach with at least one team member having in-depth understanding of Internet operations is desirable. They also point out the importance of determining the technical resources and needs of learners.

Step 3. Evaluate commercial software and use it if it fully meets your needs
This step involves a critical evaluation of software choice, purchase or development. Cook and Dupras (2004) suggest that most importantly, we need to evaluate how well it promotes active learning and provides for evaluation

Step 4. Secure commitment from all participants and identify and address potential barriers to implementation
The authors state that whilst “technical needs are important in an online course… the human element is critical. Secure acceptance and commitment
from all involved—administrators and faculty in addition to learners. In our experience, potential barriers include resistance to online learning, inadequate computer skills, insufficient time, or perception that the curriculum is a low priority. Identifying barriers early allows you to address them in a timely manner—before implementation begins” (Cook and Dupras, 2004:700)

In terms of paradigm - once again i guess the rational scientific approach is evident in the sense that individual elements are looked at in isolation from the whole. The emphasis of the authors on determining what learners see as their needs provides an element of Constructivist Hermeneutic Interpretivisit Qualitative Paradigm - where the evaluator is expected to immerse themselves in the world of the learner.

Overall – I found this article refreshing in its simplicity and easy to follow practical suggestions. It provides a good overview of one approach to development that includes evaluation in at least two stages of the process.