Models and theories in relation to working with people and ICT: Evaluation of Courses

[divide margin_top=”10″ margin_bottom=”10″ color=”#a0a0a0″]

Abstract

Working with adults in varying contexts and at different stages of their career means that they learn differently and respond to training differently. This is complicated further by the topics that are involved – for example in a relatively short period I may consult or train on; curriculum, accreditation, infrastructure, TUPE, Self Review Framework (SRF), visioning, teaching and learning (T&L), management structure, e-safety, assessment, change management, Learning Platforms, continuing professional development (CPD), development planning, emerging technologies, legislation, data, data analysis, servers, Shibboleth and so on. These consultancies may be with Headteachers, Senior Management Teams, teachers, Teaching Assistants (Learning Support Assistants), advisers, technicians, suppliers, government/local government, other industry and these may be individuals, groups or conferences.

Author: Allison Allen

Publication Date: 2011

[s2If !is_user_logged_in()]More… To see the complete Case Study, please Login or Join.[/s2If]

[divide margin_top=”10″ margin_bottom=”10″ color=”#a0a0a0″]

[s2If is_user_logged_in()]

[two_third_last]

Study

It is a concern to ensure that such sessions are useful and well-received; by this I don’t mean ticking “happy sheets” but undertaking more meaningful evaluation with later follow-up.

Our company is to be the subject of a report by TDA (Teacher Development agency) who are seeking evidence of good practice. Kirkpatrick’s training evaluation model – the four levels of learning evaluation is a useful method of reviewing training and consultancy.

Donald Kirkpatrick first published his ideas in 1959, in a series of articles in the Journal of American Society of Training Directors – he was president of the American Society for Training and Development (ASTD) in 1975. The articles were subsequently included in Kirkpatrick’s book Evaluating Training Programs. Kirkpatrick’s four-level model is now considered an industry standard across the HR and training communities. www.kirkpatrickpartners.com

Kirkpatrick’s four levels of evaluation model

The four levels of Kirkpatrick’s evaluation model essentially measure:

Level 1: Reaction: To what degree participants react favorably to the training
Level 2: Learning: To what degree participants acquire the intended knowledge, skills, attitudes, confidence, and commitment based on their participation in a training event
Level 3: Behavior: To what degree participants apply what they learned during training when they are back on the job
Level 4: Results: To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement

All these measures are recommended for full and meaningful evaluation of learning in organizations, although their application broadly increases in complexity, and usually cost, through the levels from level 1-4.

Planning and managing training in schools can be as difficult as such activities in business; often managers do not see the need to be involved in the learning outcomes, follow-up activities and return on investment. A problem can also occur when CPD is structure to match the school’s vision, but that is not always communicated to the teacher, nor is his/her role in achieving the vision apparent.

Kirkpatrick describes an example of this:

“Let’s put some real faces on what we have talked about. I saw a man (Jim) in front of a hotel in Asia. While I was waiting for a taxi, I went up to him and asked him,

“What is your job here at the hotel?”

Without looking at me, he answered,

“I wash windows.”

Since we had little else to talk about, I went back to my taxi-watching post.

The next day, I was in the country of Brunei, conducting another workshop. During a break, I wandered over to this young man and asked him,

“What is your job here at the resort?”

Chai (who later introduced himself to me by name), stopped what he was doing, walked over to me, looked me in the eye and said;

“I am part of the team that creates exceptional experiences for our guests.”

…..we carried on quite a conversation, which included Chai asking me about my stay, if I had everything I needed, what I thought of his grounds, and what I thought of his country.

On my way back to the U.S., I thought about the Window Washer and Chai. They were both about the same age, so why were their answers to the same question so different? Of course, my thoughts went to training and reinforcement. I never found out what kind of training Window Washer #1 received (he probably took a job down the street when he was offered a modest raise). Chai, on the other hand, told me about his orientation, training, and the coaching and encouragement he received from his supervisor. He was taught that he was in training in order to learn, perform, enhance his career possibilities, and ultimately serve as an ambassador for his resort and his country.

In short, Chai received training and reinforcement that modelled (what we have presented here). While his windows were no cleaner than the Window Washer’s, he knew that the purpose of training and development and his purpose were tied to the bigger picture. So he did more than just clean windows. He made me feel like a welcome guest so I would want to return.”

The more staff understand their role in achieving the school vision and perhaps have contributed to it, the better their understanding of expectations and learning outcomes. If school SMT are involved, then follow-up and further activities can be planned into the term, leading to greater success.

According to IoE/MirandaNet CPD landscape research (Pachler, Norbert and Preston, Christina and Cuthell, John and Allen, Allison and Pinheiro-Torres, Catrin: Institute of Education, University of London | BECTA, corp creators. 2010), there may well be a need to consider some drastic revision of ICT CPD models because each of the surveys cited earlier – by NFER, that on behalf of the TDA and that for the GTC – have identified a significant number of teachers who feel no direct involvement in their own professional development. 72% feel that their school requires their participation in CPD – and, for the majority, 67%, they listened to a lecture or presentation. In this context it is hardly surprising that a significant group of teachers in the NFER survey (between a third and a half of those surveyed) stated that they did not feel empowered to take charge of ICT and use it in their teaching for the benefit of their pupils’ learning. The design and structure of the ICT experience should be to empower teachers in their use of it, and to apply it in the classroom. Unless teachers can use ICT, see its benefits and understand its implementation in the teaching and learning process then their attitudes are unlikely to change.

It is against this background that many teachers have a passive approach to the use of ICT, whilst a small minority is essentially resistant to its use. The reasons both groups of teachers give are essentially external: lack of leadership; lack of, or unreliable, resources; lack of support (NFER, 2007).

The majority of ICT CPD opportunities identified in the TDA database and the e-skills survey include ICT as only one among other learning outcomes for the course, rather than the main focus itself – and the nature of the courses – one-day, at an external venue – mean that those most in need of hands-on involvement with ICT that can be embedded in their own practice are unlikely to receive the stimulus to transform their practice, the cycle of experiential learning (Kolb, 1975) or, more pertinently, “reflection and action upon the world in order to transform it.” (Freire, 1970).

Two possible reasons can be advanced to explain some of the conflicts identified in the studies.

The first is, as noted earlier, that the teachers themselves lack an internal locus of control (Rotter, 1966) or, to use a more modern term, agency (Pachler, Bachmair and Cook, 2010): responsibility for actions and control is located outside the individual. With an external locus of control the individual believes that his or her professional practice is guided by such external forces and agencies as school management, the local authority, Ofsted, the National Curriculum, the catchment area, poor ICT infrastructure and so on. Teachers’ predisposition towards an external locus of control is likely to determine their approach to learning, and the type of pedagogy they are likely to adopt.

A second possible reason is grounded in Maslow’s concept of Self Actualisation – the quest to reach one’s full potential as a person. If a teacher sees CPD as something that is externally imposed then it is unlikely that they will embrace the concept of self-actualisation. Fullan (1995) makes a similar point when he says:

Teachers who want to improve their practice were characterised by four attitudes: they accepted it was possible to improve, were ready to be self-critical, and to recognise better practice than their own within the school or elsewhere, and they were willing to learn what had to be learnt in order to be able to do what needed or had to be done. (P.73)

A few ICT CPD providers are beginning to notice that teachers often use social networking sites as a means of communicating about ICT and ICT CPD. For example, teachers often share CPD experiences on such sites – the outcomes, the learning experience, the benefits and drawbacks of the course even to the detail of food quality. Some teachers also recommend providers. The sites where teachers talk freely about their experiences are detailed in the leaders and practitioners sections under the headings of communities of practice.

There is, as a result, no one group through which providers can influence schools. Only one provider mentioned CPD coordinators as a route to market. Several mentioned the importance of professional networks. In general LAs recognised the importance of user group influence and focussed on leadership groups. Some of these leadership groups are by LAs but peer-selected ICT groups were also in evidence. One example of this is Cambridge. In interview one provider “All of these are created to provide a vehicle to disseminate good practice and to keep decision makers informed.” Larger commercial providers also had powerful groups with recognised value and one government funded provider had an extraordinary range of influence groups; some of these providers would merit a case study.

Evaluation of courses

Delegates were generally asked to complete evaluation forms following courses although some providers do not undertake course evaluation (8%) and the type of evaluations ranged from “happy sheets” to how the course would affect teaching practices. Few companies used online versions of evaluation although where they were used, providers noted improved reflection on the part of the learners.

Most providers who evaluated their own courses reported that they evaluated these using the delegates’ evaluation forms. A few providers mentioned that they had the facilitators complete evaluation forms of the day and then compared these to the delegates. Some providers evaluate those in the light of changing local or national policies. In other cases a correlation exercise is undertaken to map delegate responses with that of every provider. One independent Provider included observation of impact some time after the course and one LA commented: “Measuring impact is a challenge because of other influences. It is difficult to separate out the impact of our courses from other interventions going on in the school at the same time.” These influences might be: other training/policies/strategies, investment (or not) in technology, leadership priorities in school and so that are documented in detail in this study in the leader and practitioner sections.

Most providers have a Mission statement or Vision that links to evaluation. One HE provider gave useful elaboration of what this means in practice: “The mission statement includes the phrase ‘to pursue excellence in education … and professional practice’. ICT CPD relates to this professional practice. It also fits into our mission to engage in ‘consultancy and other services to support and develop the quality of educational systems and related fields of policy and practice.”

ICT CPD drivers were defined as the technical advances and issues that impact on the use of ICT, as well as national, local and school policies and attitudes towards funding technology.

The school development plan or senior management agenda was the most frequently mentioned driver of ICT CPD programmes followed by government policy and Local Authorities(LA). A key source of tension is now that providers said that schools decide what they need not the LA as was the case in the past. What the school demand may be different to what the provider felt they needed or may not follow government direction. Nevertheless, the providers now have to adapt to what a school wants whether they consider this demand to be appropriate or not. Most providers referred to the necessity of accommodating school needs even if the provider has identified other strategies.

On the other hand a different source of concern is the fact that 23% of provider training focuses on software without apparent reference to learning outcomes or teacher confidence. This study did not have the capacity to investigate these tensions more thoroughly but they are of major concern to a government who would wish to see CPD making a difference throughout the whole school system.

A majority of providers (52%) felt that better funding was a key solution improving the effects of CPD and this included personal CPD budgets, funding for release time and funding for change management. These providers felt that embedding ICT/higher thinking was not addressed by many current courses but should be. 36% felt that tailored programmes or tailored delivery was their suggestion of a way to do this. They also commented on the need to acknowledge that teachers are more ICT literate now than in the past. 28% of respondents reflected on the need for changed leadership priorities at school and government level also mentioning the loss of Local Authority ICT advisers as a critical factor.

A leader who is sympathetic to real needs of the staff and the challenges they face in learning about digital technologies is likely to be at the heart of successful of in-house ICT CPD as well as an effective programme of external provision. It is not surprising where leaders are dismissive about the particular challenges for professionals in learning about digital technologies CPD programmes, if they exist, are not successful because they are not tailored to teachers’ real needs.

ICT programmes are often part of the general provision for continuing professional development (CPD) in schools. A very wide range of different methods for needs analysis are being used in schools where there is an organised CPD programme aimed at meeting the individual needs of staff. However, a few schools do not have an agreed plan for developing ICT CPD programmes for staff because of a wide range of challenges including the lack of provision for the time teachers need to learn. Insufficient funds and resources can often be the problem that is often the result of a lack of SMT prioritization by the senior management team. The location of external courses is also a factor in deterring staff from attending. In detail the leaders said that the basis on which the school choose from the ICT CPD provision available were: