Towards a theory of the impact of ICT in education Roger Broadie, BroadieAssociates.co.uk 6 May 2011

Abstract

The use of ICT in schools puts significant costs upon the education system in the UK and other countries. These include the costs of the hardware and software in schools, training for school leaders and teachers, and in some schools costs to the individual pupil and their parents. There are also possible additional costs that introducing ICT might cause, such as distraction from learning and safety considerations. It is necessary to demonstrate that the spend on ICT is at least as effective in producing learning as the spending of the rest of the school budget. It is therefore necessary to see the link between the use of the ICT and learning and for this to be quantified relative to the spend on the school buildings and physical environment, on teachers and other staff, and on resources such as books.

Why is a theory needed?

The use of ICT in schools puts significant costs upon the education system in the UK and other countries. These include the costs of the hardware and software in schools¹, training for school leaders and teachers, and in some schools costs to the individual pupil and their parents. There are also possible additional costs that introducing ICT might cause, such as distraction from learning and safety considerations. It is necessary to demonstrate that the spend on ICT is at least as effective in producing learning as the spending of the rest of the school budget. It is therefore necessary to see the link between the use of the ICT and learning and for this to be quantified relative to the spend on the school buildings and physical environment, on teachers and other staff, and on resources such as books.

Assuming the cost-benefit case can be made for use of ICT in schools, teachers need to understand in what circumstances and why using ICT will improve learning relative to ways they might teach and get pupils to learn that do not employ ICT. The concern here is to ensure use of the best tools and environments to maximise learning. And as there are significant costs in using ICT there needs to be significant improvement of the teaching/learning processes from the use of ICT. This also calls for ways to quantify the improvements relative to previous approaches.

The third reason for a theory of impact is to enable teachers to develop their pedagogy and understanding of how learning happens. Though there may be situations where use of ICT is the only way to enable learning, in the majority of cases ICT will be used in conjunction with the other social and physical resources of the school and its wider community. Understanding the contribution of ICT to the whole learning environment and teaching approach is required in order to balance its use appropriately and to take best advantage of what it offers.

The problems in forming a theory of impact.

ICT is itself very diverse in form and function. The ways that it can be used are extremely wide. This makes the choice of how and when to use ICT very difficult for teachers, in the sense that there is over-choice; ICT might be used in ways that are not very effective when it could be used to much greater effect. When limited to the physical environment of a classroom, the resources it is possible to assemble within it, and the time period allotted for a class or session, there is limited number of ways to organise how pupils will interact with each other and the teacher, and with learning resources. The introduction of ICT radically extends the possibilities; the range of resources it is possible to access is hugely increased, as are the ways pupils can produce work and collaborate. Most importantly, a computer is an interactive entity (dependent obviously on the way it is programmed) that can take on roles that in a non-ICT classroom have to be taken by either pupil or teacher. The computer can present and can provide feedback and assessment, as a teacher would, or can learn or respond collaboratively with suggestions, as pupils might do. The range of possible interactions is therefore also hugely increased, the pupil-teacher relationship becoming triangular, pupil-teacher-computer. Should the school also have an online platform, interactions in the class can be complemented by interactions that happen online before and after the class. Time and space can be used differently, requiring that teachers need to consider what the unique features of pupils being in a classroom are relative to the learning that can happen out of the class, so as to maximise the learning benefits of both. The ways in which teachers teach can be transformed.

Without a theory to guide development of pedagogy new approaches will be adopted because a teacher has chanced upon particular examples of use of ICT that provide an approach they feel will be effective with their class. This is indeed what is currently happening; no teacher can have full breadth of knowledge of how ICT is being used to enhance learning and their development of their pedagogy depends upon what is happening in their school or locality or upon the particular interest-groups of other teachers and educators they have time to engage with.

This is unsatisfactory, particularly as advances in neuroscience are giving increasing insight into how learning happens and advances in online social networking and websites are providing insights into the psychology of engagement and interaction between people, and between people and information.

Both of these points highlight the main problem in forming a theory – the diversity of ICT and its uses.

The nature and use of a theory of impact.

Good theories provide a simplification of what is observed, by identifying underlying structures and mechanisms. They also should enable predictions to be made.

In the current stage of development of the use of ICT in schools, teachers are getting surprising results from their use of ICT. They are trying ways to use ICT without a clear understanding of what will result. When teachers notice a bigger than expected improvement in learning, this is often attributed to higher engagement in learning by the pupils, with only an anecdotal appreciation of why the engagement of pupils is raised and without understanding clearly the mechanisms that have acted to generate more and/or better learning.

Much of the information that is being shared between teachers about these good uses of ICT is being shared through domain-specific groups. Teachers of English are discussing how to use ICT to improve understanding and use of language and literature while mathematics and science teachers are doing similarly in their domains. A theory of the impact of ICT must act across the different subject domains of education and across the different ages and abilities of pupils. With a theory to guide them teachers will be better able to work out how use of ICT can best impact on and improve the learning activities of their pupils and their psychological approach to those activities.

The problems of assessing the impact of ICT on achievement.

There have been many attempts to study the impact of ICT on achievement. It has proved possible to causally link certain uses of ICT to raised achievement when these are limited in time and purpose – for example us of revision software in the weeks leading up to exams. However broader studies such as Becta’s Impact2², analysis of the ICT Test-bed schools ³ and investigations by Ofsted⁴ have largely failed to prove that use of ICT has caused raised achievement while at the same time being able to show correlation between schools with good pupil achievement and those making good use of ICT. Internationally there is a paucity of studies showing that ICT has raised achievement in schools⁵.

It is possible that this failure to link use of ICT with raised achievement to any great extent is that it is fundamentally impossible to do so. As observed above, ICT is almost never used in isolation from other components of the teaching and learning environment. As a result it is impossible to separate out the impact of the ICT from the impact of the ways the other components have changed as a result of, or alongside, introducing the ICT. The diversity of ways in which ICT can impact compounds this.

In order to establish a theory of impact there must be some link to raised achievement in learning, as that is a main purpose of schools and a main rationale for the funding of schools. If it is fundamentally impossible to directly link use of ICT to raised achievement, in the majority of cases, new approach is required.

It is possible to link the use of ICT to the ways in which learning activity changes. Certain parts of the learning activity may have previously been impossible or very difficult without ICT, while it becomes possible for other parts to be changed because of the ways in which ICT is being used. It is therefore possible to create a theory of how ICT impacts on learning activity.

It will then be necessary to show that the changed learning activity, as a whole, leads to raised achievement. This is extremely complex in that it relates to theories of learning about which there is little academic agreement, but it can also be dealt with pragmatically in that teachers can do, through their formative and summative assessments, note pupils improvements in learning. For the purposes of this theory, the link between changed learning activity and raised achievement will be left to others. The purpose of this theory is to make clear the mechanisms of impact, so as to enable professional discussions about the ways learning activity can be changed in ICT-rich environments and which changes are most productive in raising achievement.

Categorising changed learning activities to create a theory.

It has been stated above that the ways in which ICT can be used in learning are very diverse, creating a huge range of learning activities that are possible using ICT. In order to have a manageable and workable theory it is necessary to categorise the ways in which ICT enhances learning, so that different mechanisms of impact can be studied separately to some extent. This will make it easier to study the impact of individual changes in learning activity due to ICT though there will still be considerable overlap; for example the impact of the ability to access learning resources via the Internet will be difficult to separate from the impact of the nature of those resources which are likely to differ from paper-based or physical resources.

An analysis of the value-add impacts of ICT⁶  was conducted in 2001 by the European Education Partnership (E.E.P.). This can act as a starting point to be refined and validated as the number of examples studied increases. The areas of impact identified by the E.E.P. are listed below, but before considering the nature of these one overall factor that is relevant to most of them needs mentioning. This is that in many of these areas ICT can act to produce positive impact on learning both by opening up possible opportunities for learning and by constraining or focussing them. For instance a website can be constructed to link the user to a huge diversity of information and opportunities for communication and collaboration, or it can be constructed to severely focus and guide the user into using pre-determined and limited opportunities. Both can be useful in aiding learning as at times learners need to be able to follow their own learning paths and at other times to be strongly guided and have work scaffolded in small steps.

This ability of ICT to be used to support and enhance very different pedagogies inevitably leads to consideration of which pedagogies it is most appropriate to adopt to promote learning in ICT-rich environments. It is not the purpose of this study to consider this question but it is sensible for readers to be aware of the educational philosophy of the author and of the views of some others. A presentation by Diana Laurillard⁷ considered this question with a tentative conclusion that generally constructivist approaches work best. The Apple Classroom of Tomorrow (ACOT) ⁸ studies were also strongly based in constructivist methodologies. The philosophy I shall bring to analysis of examples of impact is a belief that in general learner-centred and learner-led pedagogies are most effective for learning provided that the learner and their teacher can dynamically shift the pedagogy back to didactic and strongly teacher-led approaches at any moment that this becomes necessary. This is a view very similar to the theory of dynamic leadership that sees managers leading subordinates to take increasingly independent control of delegated work while remaining watchful and ready should any individual task cause the worker to need to revert to structuring of the task by the manager and greater supervision. It should also be borne in mind that ICT itself increasingly has the capacity for this embedded in systems, with online help to provide step-by-step guides to doing things being available in software, and from YouTube (http://www.youtube.com) and other sources.

The E.E.P. categorisation of the impact of ICT on learning.

This analysis found the following categories of positive impact:

Motivation.  Teachers often report increased motivation to work and engagement with work by pupils. It is not clear whether this increased motivation and engagement is a cause of better work or an effect of more satisfying and better structured work that use of ICT creates. Consideration of computer games suggests that the design of the software is responsible for generating engagement, with the motivations to play being largely human; challenging oneself or social interaction with others playing similar games. However many of the examples of ICT impact in learning currently available suggest that the original motivation to engage with a task comes from the teacher but that the use of ICT makes engaging with the task more enjoyable for a wide variety of reasons, which is responsible for increasing pupils’ engagement and their learning.

Access to resources. As mentioned above, this can be a matter of giving easier and faster access to a much wider variety of resources, or it can be a matter of scaffolding and guiding the users’ access. Also in this category we need to consider the impact of search engines and ways of organising information and resources. For example the creator of a CD ROM for students learning immunology decided to structure the whole body of knowledge, that is very inter-related, into a dodecahedron structure, the rationale being that each node of knowledge is only one or two clicks away from all the other nodes (with just one being 3 clicks away). He also used the technique of hot-spotting diagramatic films.

Increasing communication and collaboration. Computer systems now provide many different ways for people to interact online. As well as the fact of being able to communicate at a distance this area of impact must also consider the nature of the communication, for example the asynchronous nature of email and forums that provides time for reflection before answering. Communication forms such as Twitter (http://www.twitter.com) can be used to constrain and focus communication in order to promote learning, forcing users to state what they wish to say in only 148 characters.

Extending learning time. The ways in which learning is extended will be largely covered in other categories, such as ability to access school resources from home and to continue learning dialogues out of class. The purpose of separating extended learning into a separate category is to study how time and space can be used differently for learning in ICT-rich environments. If teachers know that pupils are able and willing to do certain things online out of class, they can transform what they and pupils do in class.

Using more channels. This refers to communication channels into the human brain. Computers have radically increased the opportunities for visual and aural communication. There are also opportunities for the channels to be used to prompt each other, for example highlighting text that is being read by the computer to draw a child’s eye to the written words that they are hearing.

Access for minorities. The minorities referred to are learners who are in a minority in their class or school whose needs are not as well catered for as those of the majority of the pupils in their learning cohort. This is not just students with handicaps but also those who have different needs because of language or previous experience or preferred ways of receiving information or communicating.

Enabling publishing and audience. This covers impacts such as the necessity to produce creative output in certain forms and to standards that suit the publishing approach and intended audience. It also covers impacts deriving from the comment and feedback that audiences can provide. The fact that audiences can be controlled is also an area of impact, enabling learners to control how widely they publish and to whom.

Enabling brain-centred learning. This is a category that the E.E.P. analysis states is far from clear. It resulted from a number of examples of powerful impact that could not be satisfactorily included in the other categories. It includes impacts such as the ‘flow’ of high engagement and concentration generated by games, how the very short latency of response of computers to input can better stimulate thoughts in developing ideas, how typing can make ideas flow better than handwriting because it is possible to commit words to paper at a speed more in tune with thinking and how and why multimedia appears to be so much richer for communication than text or speech. This area could also cover the impacts referred to in discussions of learning styles, such as pupils learning better in mobile activities or by using visual representations such as mind-maps.

Re-balancing teaching and learning. This obviously happens if learning time is extended but it also happens in more subtle ways. A computer is able to act in many of the roles that teachers and pupils undertake in learning. It can for example present as a teacher would, can mark and provide feedback, and can be taught by pupils as they attempt to demonstrate their learning. It therefore opens up the possibility of triangular teaching/learning relationships, allowing both pupil and teacher to adopt different roles, one example being the phrase “guide on the side not sage on the stage’ to refer to how teachers should change their roles. There is an important education management issue here, as staffing costs are the major component of education system funding. Small changes in how teaching and other staff are employed and deployed may be able to release more than sufficient funds to fund the ICT necessary for changed approaches, that produce significantly better learning.

Automate management and recording. There is an obvious impact here in schools saving time and money but this also refers to how learners can be helped to record their learning and make it more visible, and manage their learning.

Increase scalability and replicability. This is an impact of most concern to policy makers and school leaders but benefits learners who gain the better learning opportunities that have been scaled and replicated. A critical issue in this category is the role the teacher in learning. Teachers are different and some are not as good as others in some ways. Some are better at drawing resources together to support their teaching and pupils’ learning, some are better at presenting ideas than others, and some are better at counselling and learning discussions. There are various ways in which a school, a local authority or a country can use ICT to ensure that the learning experience for all pupils has a higher guaranteed baseline no matter what the differences are between teachers, schools and localities. ICT makes it possible to replicate and scale these approaches at relatively low cost (relative to costs of education systems overall).

Examples of analysis of impact to build the theory.

The proposed approach to create a theoretical basis for studying the impact of ICT in education is to analyse different examples where teachers perceive positive impact on learning from the use of ICT, to identify mechanisms of impact that have the potential to be used by educators in different areas of the curriculum and teaching pupils of different ages.

Once mechanisms have been identified, these can be more deeply investigated to find ways whereby the impact on learning can be quantified.

This is best illustrated by a few examples:

Example 1:⁹ Data logging. Leaving the computer to get on with the datalogging while the children are engaged with other things, rather than having the pupils clock watching to take a temperature every few minutes is always more productive. The logging can be left to proceed for as long as desired and then the pupils can engage in analysis of the graphical representation of the physical change that was set in motion minutes or hours earlier. It is in the analysis and discussion that the real transferable learning is taking place.

It is clearly important to be able to read a stopwatch and a thermometer, but once pupils have mastered those skills the constant replication of them over a long period has little if any value and merely provides an opportunity for distraction.

Applying the principles learned in a single activity such as this can then be built into something exciting and challenging such as solving a murder mystery using ‘forensic’ techniques, where pupils can appreciate the learning that has taken place and higher order thinking skills are brought to the fore.

Analysis. This is firstly about using ICT to reduce the time spent on a low-level activity, replacing it with higher order activities that are more likely to generate learning. Secondly with data-logging, the ICT is being used to create visual representations of the data being analysed, more quickly and accurately than can be done by hand, which will aid interpretation and extraction of information from the data. Thirdly data-logging can produce data on very fast phenomena, such as what happens millisecond by millisecond when a match is struck. This is creating a learning resource that would not be available without ICT.

Example 2:¹⁰  Teaching primary school children to make video podcasts and using these to record and publish what they are doing in class is a useful learning activity. However, if you give the same pupils a flip video camera and tell them they have to choose to be a beetle or a butterfly (with a camera) making the podcast, you get a whole new dimension.  The beetles will all be lying flat on the ground, crawling through the grass and pushing their camera through the jungle of stalks, bumping into other bugs, looking up and seeing huge birds etc etc. The butterflies will hold their cameras facing downwards and fly around swooping down towards brightly coloured flowers.  (Or in the case of one 9 yr old group, zooming in on a caterpillar shouting ‘that’s my baby’). This makes for a whole different sort of podcast.

Analysis. This is first about removing an activity that inhibits learning – the act of having to write getting between the creative ideas and their expression – hence speeding up the creative story-telling. Secondly it is about ICT creating a visual representation, which is very similar to the visual possibilities of data-logging in that the visuals stimulate thoughts about things the learner has not previously seen.

Example 3: ¹¹ Twitter , at a ‘literal’ level, is useful to remind pupils about homework deadlines, assignments etc but at a more creative level it can be used for mass role play. For example, working with secondary history teachers doing work with pupils about the WW2 on the home front. It happened to be the Year of the Evacuee. Every pupil had to choose a character (evacuee, mother of evacuee, father, member of host family etc) and they had to pretend it was 1940 and communicate with each other about what was happening to them and how they felt. Except that though they had to think of themselves in 1940, they had mobile phones and twitter and 140 characters to send messages. Within about 3 days there were several hundred pupils following the hashtag and twittering – interestingly, a great many had set up accounts with 1940’s names!

Similar mass role play can be done with English Lit (Macbeth on Twitter is very interesting) or in RE – can you understand the content of eg the Lord’s Prayer well enough and get the key points in 140 characters?

Analysis. Here the use of ICT is slowing down the learning activity and making it asynchronous. It is also about constraining how thoughts are expressed in a single channel of communication (text) rather than adding a channel as in the video podcasts. Presumably the slowing down and the constraining of communication are both promoting more thought, the challenge being how we measure this.

Example 4:¹² A science teacher set a forum question for homework. Just in the last couple of minutes of the lesson with no discussion he said “OK – homework; all of you provide an answer to this question in the online forum, “Which has more energy, an iceberg or a kettleful of hot water?”

This resulted in:

– all the pupils put an answer in the forum; none wished to be very visibly seen to have not done the homework.

– the more timid pupils were able to contribute as much as they wished, without the fear of criticism to their face that they felt in class.

– the less able pupils read what the brighter pupils wrote, before devising their own version of the answer they liked best.

– the brighter pupils engaged in some debate about each others’ answers, exploring several threads.

– The teacher was able to see all the pupils’ answers in a single place, so was more easily able to identify the common misconceptions that would have to be corrected next lesson.

– In the next lesson he was able to ‘ground’ the pupils more quickly, as they could each see their own contribution and could remember some of the other contributions they had looked at and thought about.

– And he had accessible some pupils’ answers to lead the debate, that looked at the problem from their viewpoint. Such as the pupil who said “I’d pour the kettleful of water over the iceberg, and if all of the iceberg melted then the kettle has more energy!”.

Analysis. The learning time was extended because the pupils were more motivated to provide their answer in the forum than they would have been in just preparing an answer to give next lesson, with reduction in fear about speaking publicly aiding some in this. The answers from the more able pupils became a resource for pupils whose understanding was not as good. Communication was extended through some of the pupils debating each other’s answers. The teacher being able to see all the pupils’ answers in one place is an automation of management and recording. Time was also saved in the next lesson. This example therefore shows positive impact in seven different ways.

By analysing a large number of examples of the use of ICT in this way, the categorisation of mechanisms of impact can be extended. This will allow us to see which mechanisms for impact are being used most often and which are most easily used cross-domain. Assessment can also be made of their comparative importance in raising achievement.

Conclusion.

Though the use of ICT with school age young people to promote learning, raise achievement and make education more effective and efficient is highly domain-specific and tailored to fit the individual circumstances of learners, teachers and schools, there are mechanisms through which this impact is gained that are cross-domain. This opens the possibility of studying the ways in which ICT impacts on learning to produce insights that may be generalisable quite widely.

These insights could inform the development of pedagogy in ICT-rich environments to generate a better consensus on the most effective changes that teachers and schools might make.

As the potentially generalisable impacts can be described as mechanisms of impact they are also potentially quantifiable through the ways in which learning activity changes. This would allow comparison with other methods that might be used to generate similarly productive changes in learning activity, which would allow assessment of the return on investment in ICT in comparison to other approaches.

References

1. Regular studies of the costs of the hardware and software in schools are done by the British Educational Suppliers Association (BESA), http://www.besa.org.uk.

2. ImpaCT2, The Impact of Information and Communication Technologies on Pupil Learning and Attainment. ICT in Schools Research and Evaluation series – No.7. A report to the DfES by Colin Harrison, Chris Comber, Tony Fisher, Kaye Haw, Cathy Lewin, Eric Lunzer, published by Becta.

Angela McFarlane, Di Mavers, Peter Scrimshaw, Bridget Somekh and Rob Watling

3. BECTA (2006), Evaluation of the ICT Test Bed Project.

4. The importance of ICT; Information and communication technology in primary and secondary schools, 2005/2008. Ofsted.

5. “The impact of ICT use on learning outcomes is unclear, and open to much debate. Widely accepted, standard methodologies and indicators to assess impact of ICTs in education do not exist.” Trucano, Michael. 2005. Knowledge Maps: ICT in Education. Washington, DC: infoDev / World Bank. Available at: http://www.infodev.org/en/Publication.8.html

6. An analysis of the value-add impacts of ICT conducted in 2001 by the European Education Partnership (E.E.P.), available at http://www.eep-edu.org/InnService/Start/what_addval_start.htm

7. “Realising the vision for e-learning”, a presentation by Diana Laurillard, e-Learning Strategy Unit, at the BETT Show 2004. She identified that the common thread in our understanding of learning from educational thinkers such as John Dewey, Jean Piaget and Vygotsky up to John Biggs and Jean Lave, is the learner as active agent in the learning process, considered what counts as active learning, and how e-learning helps.

8. The Apple Classroom of Tomorrow (ACOT) project, http://ali.apple.com/acot2/, identifies six design principles for the 21^st century high school.

9. Example provided in an email from Jane Finch on the MirandaNet list, 20 April 2011, on the thread ‘Critical Learning Activities?’

10. Example provided in an email from Jenny Hughes on the MirandaNet list, 20 April 2011, on the thread ‘Critical Learning Activities?’

11. A second example provided in the email from Jenny Hughes on the MirandaNet list, 20 April 2011, on the thread ‘Critical Learning Activities?’

12. The example is taken from the report ‘Learning Platforms, a study of use in secondary schools’, available at http://www.broadieassociates.co.uk/page3/page7/files/CapitaLPwhitepaper.pdf