Do CAO points predict third level performance?

In 2010, the HEA produced a report in which they concluded that ‘prior educational achievement’ was strongly correlated with progression rates at third level. This effect was found to be strongest at Level 7 but was still significant at Level 8. Very few academics would be surprised by this.

Yet, a number of very reliable people with whom I have spoken in recent years have said to me (based on sound data analyses) that CAO points are not an especially  good predictor of third level performance. The number typically quoted is that CAO points account for about 20% of the variation in academic performance at third level.

At first glance these two findings seem contradictory; but that is not necessarily the case.

Suppose you take a particular programme with a particular median CAO points value and do a scatter plot of, say, First Year precision mark of individual students versus their CAO points. When you do a linear regression, you’ll probably find a positive slope but a pretty low correlation coefficient.

Now suppose you repeat the exercise a number of years later where the median points value is substantially different. Now when you do the scatter plot and regression you’ll find a similarly low correlation coefficient but it is quite possible, likely even, that the new precision mark data will not overlap with the original set of data. Most academics would suggest that if the median points value in the second study were substantially lower, the scatter plot would be shifted left and down as shown schematically below (ignore the numbers!).

Of course a big factor in all of this is the fact that it is by no means certain that third level standards are constant. I tend to think that standards are constantly being adjusted slightly (and teaching methods being adapted) to meet the capabilities of the incoming students. Thus, any correlations you might expect to see tend to be ‘dampened out’ somewhat.

But it would be a nice to see a real analysis, covering a number of years and a broad range of median points, in the manner of the graph below.

CAo blog

Posted in Uncategorized | 1 Comment

Fighting the spin – some interesting data on college entry

Points in IoT and university sectors – are universities really gaming the system?

CAO all

Points distribution for so-called niche courses (intake <= 10), the devil incarnate in the eyes of many!

niche courses

Points versus intake all courses – intake and points are uncorrelated

Points v Intake

Data as above but with intake <= 100

up to 100

Trends in student preferences


Use of Random Selection – new points system will fix this (apparently!) but is it really a problem now ?

random selection

Points inflation – students are performing better – this is the major source of heat in system along with the herding of students into STEM

points inflation

Posted in Uncategorized | 1 Comment

CAO Points and Generic Entry

Ok, a one off…probably

Over the last four or five years we have heard many interested parties encourage school-leavers to study STEM subjects. Not surprisingly, the entry points for STEM courses have risen steadily. Unsurprisingly, the points for Arts and Humanities subjects have dropped but nobody seems concerned about this despite the rich arts heritage in this country.

Now, in response to the rise in points for STEM subjects, commentators are, somewhat oddly, becoming fixated on courses that have a very low student intake and blaming them for all the points inflation that is going on. These courses account for about 12% of the total number of CAO courses and given their small intake, they probably impact on a relatively small number of students. However, we can’t be sure as we don’t have access to 1-2-3 preference data. But I would bet a large sum that the first preference numbers for these courses are tiny, except perhaps for some of TCD’s programmes where they seem to use random selection an awful lot.

In addition, Philip Nolan and Tom Boland have been out in force pushing the ‘generic courses’ agenda. The belief is that more generic entry will take the heat out of the points systems, although the rise in points for generic entry courses (like Common Entry to Science in DCU) would make one question this argument. Regardless, the media is jumping on this particular bandwagon and the third level institutions are being portrayed as ‘dragging their heels’.

In addition, there is an increased tendency to invent all sorts of pedagogical reasons for generic entry; encouraging better second-level study habits and avoiding over-specialisation at an early age are two of the usual arguments. The first is guesswork and the second ignores the fact that most third level courses are pretty ‘generic’ in the early years anyway and only become very specialised in third and fourth year. Thus, doing a BSc in Neuroscience, for example, doesn’t preclude you from a science career outside neuroscience. And anyway, what ever happened to all the talk about lifelong learning and how in this 21st century everyone has to be flexible and adaptable.

What is really going on here is that policy makers are being political. They want to be seen to be doing something about the points race but in doing so want to pass on the problems of an under-funded, over-crowded education system onto the third-level sector. The CAO system will look nice and tidy but the pressures on students will be postponed until third level where they will be less visible. For example, if, in DCU, we admitted all our science students into a generic, common entry first year, there would be a right old dog-fight to get into two or three high-demand programmes whose numbers must be capped for logistical reasons. Are first year students equipped to deal with the pressures? That is a question that nobody is asking. Indeed, nobody is asking how the student experience, failure rates and drop-out rates might be affected by a more generic entry system. Maybe there will be no effect at all but shouldn’t we be thinking ahead rather than slavishly following the agenda of a small number of parties?

Posted in Uncategorized | Leave a comment

The Last Post

I was going to title this post “On being research active and being a good teacher”, or some such, but instead I’ve decided to make it the ‘Last post’, i.e, the last post.

Reading the Sunday Business Post yesterday I came across some quotes by the President of NUIG and to be honest it made me despair.

Jim Browne made the statement

The best teachers are research active – they are people who have a sense of enquiry and a sense of enthusiasm about their subject”.

I rehearsed my arguments; about how teaching involved a lot more than ‘enthusiasm’, about how there were so many counter examples to the President’s claims, about how there is no absolutely no evidence that research activity and teaching excellence are linked, about how my own teaching has suffered on occasion at the hands of my research, and then I thought: why bother?

Why bother, even if the NUIG President made this loaded and judgmental follow up statement:

“If I’m a lecturer just droning on and on about stuff I read in a book ten years ago, who’s really going to care”.

I was going to work myself into a frenzy about this one, but what’s the point? It’s a statement that falls into the ‘not even wrong’ category.

As an engineering scientist, my instinct is to believe that everything we do should be driven by evidence, by the data. But doing things in an evidence-based way does not come naturally to many, even to academics who, ironically, have spent their lives using evidence-based methods. Thus, we have the recent high-profile pleas by academics for increased funding for basic research, without going to the trouble of assembling the real evidence for how basic, academic research makes the economy and our society better. We’re just supposed to believe them because they know best. An anecdote or two about MRIs or General Relativity is good enough.

Education in Ireland, and elsewhere, is facing huge challenges, and I have some sympathy for politicians. Their ‘life expectancy’ is limited and they have a huge and natural desire to make a quick impact. They fall under the spell of buzzwords like ‘critical mass’, ‘innovation’ and knowledge economy’. But in a way, they are lambs to the slaughter. Many within the education establishment capitalise on these weaknesses – for their own benefit. Hence, for example, the transformation of efficient IoTs into high-cost TUs is driven, partly, by ambitious senior managers rather than the academics who rightly see absolutely no benefit for students in this process. (If anyone can tell me how students in Tralee will benefit from that institution being merged with CIT, I will buy you a pint or two!)

I’m not sure what the future holds. As a country we take pride in our third level participation rates but we don’t seem to know how to pay for it; we have no ideas about how to make the whole sector sustainable. But it’s a hard problem and it is not helped by those of a left-wing persuasion who make ludicrous statements about universities desiring to be ‘profit-making’ institutions. In fact, the problem is that, currently, there isn’t a ‘bean’ in the system and institutions are scrambling around, often sacrificing principle, to try to makes ends meet. They might be going about it in an excessively bureaucratic and managerialist way  but the intention is to make the system pay its way, not to make a profit.

Recent developments in this area have been an exercise in ‘kicking the can down the road’, at least until after the next general election. At a recent meeting in the RIA I was struck by our inability, as academics, to get to the heart of the problem, i.e., that the third level sector is expected, and expecting, to do too much, with too few resources.

Yet, we persist in expanding our portfolio of activities, buying into the prestige race, all the time neglecting, I am sorry to say, our very raison d’etre, i.e., undergraduate education of our population. If anyone doubts this, check out the average undergraduate laboratory in the sciences and engineering.

And so, as the ‘teacher season’ approaches, everyone will have their say about how the Leaving Cert is dysfunctional, about how entry to third level needs to undergo a radical transformation, about rote learning, problem solving, critical thinking, 21st century skills, learning outcomes, graduate attributes, creativity, emotional intelligence, flexibility, adaptability, innovation, apprenticeships, entrepreneurship, coding, philosophy, continuous assessment, and all the countless number of buzzwords that have infected the world of education. And nearly everyone will be shooting from the hip.

And I don’t really have the energy for it any more, and I’m not sure I have anything new to say at this point.

So after about four years and more than 40,000 hits, this blog comes to an end while I focus on what I’m paid to do: teach and do research.

Tks for reading!

Posted in Uncategorized | 5 Comments

Incentivising Thinking

It is well known in the medical world that patients will be compliant, i.e., will take their prescribed medication, if they perceive a direct benefit from taking that medication. Thus, someone with asthma might have no problem taking his/her ‘reliever’ but may be a bit hit-and-miss with their ‘preventer’.

There is an important message in there, one that applies to education as much as medicine; people will buy into something if they can see a real benefit to themselves from it.

Recently I’ve started to use this approach to get students to actually think. I don’t believe critical thinking can be taught but it can be encouraged. And the best way to  get students to think is to reward them for doing so.

So, in two of my modules I’ve posed a challenging problem (essentially computational) and offered students the reward of  2% added to their final module mark if they can solve it. While many students have not engaged, a significant group have. Interestingly, while they are obviously pulled in by the carrot of the 2% bonus, they tend to become like the proverbial dog with a bone and I get a sense that the challenge takes over and the prize becomes less of an issue than the satisfaction of solving the problem. Indeed I have had to tell some of them to let it go because they were spending way too much time on one particular problem.

In doing this I have been hugely encouraged by the ability of some of our students, especially their ability to draw on what they have learned in other  modules – maths modules in this particular case.

There is no doubt that we can do more to get the best out of our students.

Posted in thinking | Tagged , | 1 Comment

Basic Research and ‘That’ Letter

Governments always want some sort of tangible return from academic research. But since the financial crisis began in 2008, the prevailing view in Ireland has been that the primary or even the only role of academic research should be to generate economic growth through product innovation and the creation of start-ups. Personally I think this approach is misconceived because I don’t think academics are natural entrepreneurs and I suspect that the direct impact of academic research on the economy is tiny in comparison with the effect of FDI or tourism or agriculture. Furthermore, the focus on strategic research creates additional costs for institutions because it requires an enhanced administrative sub-structure, and it often involves the secondment of academic staff from teaching to strategic roles.

So how does one make the case for providing funding for academic research? (I refuse to use the word ‘basic’ because it’s such a loaded and subjective word and I don’t buy into the notion that so-called ‘applied research’ is derivative, depending on ‘basic research’ for new ideas. I actually think that applied research often inspires basic research!)

The first thing we need to do is ask ourselves some tough questions, questions that I would ask if I were a decision-maker. Here are some:

  1. What is the evidence for a causative link as opposed to a correlation between academic research and economic prosperity? The answer is not obvious and if you do even a minimal amount of reading on this topic you will find that this is very much a subject of debate amongst economists. Does academic research contribute significantly to the economy or is it a case that economic prosperity creates the conditions whereby academic research can be afforded? Furthermore, a key question here is not whether academic research can have an economic (or social) impact, but whether providing across-the-board funding is an efficient and cost-effective way of doing things. (Money spent on academic research is money not spent elsewhere.) That’s how the policy maker will think at least. And, if you think about it, most of the third level institutions themselves have rejected the ‘broad’ approach, favouring a more strategic one in which an emphasis is placed on a small number of themes. Finally, in the Irish context, what evidence, if any, is there that academic research done in Irish institutions has had a measurable impact on economic prosperity, and has it been worth the investment?
  2. What precise role does academic research play in third level education? It is well known that much undergraduate teaching can only be done effectively with the aid of graduate students – in all disciplines – and it would appear that many policy makers are not quite aware of this fact. But does research activity improve undergraduate education? There is no convincing evidence, for example, that being research-active makes you a better lecturer. It is good to have enthusiasm and up-to-date knowledge, but a lot more is required to make you a good third level teacher. Do undergraduate students benefit from the presence of cutting-edge research laboratories, either within academic units or within research centres? Do they get sufficient access to these laboratories? Perhaps, but would students not benefit more if schools and departments were equipped appropriately rather than existing on a shoestring as they do now. We need to articulate better the link between academic research and the quality of undergraduate education – if indeed there is one – and we need to back up what we say with evidence. Plausibility is not enough.
  3. How significant has the ‘brain drain’ been as a result of state funding for academic research becoming focused on strategic areas? To answer this, we need data on the numbers of high quality BSc and PhD graduates, and experienced researchers, that are being lost abroad and, crucially, data on the contribution that those who stay behind make to the economy. Indeed, how important are PhD graduates – in all disciplines – to the Irish economy? Can we quantify this in any way? Where are these graduates? Are they driving change and innovation in our companies and organisations, and can we link their ability to do these things to their PhD education? These are hard questions and we like to convince ourselves of answers that, frankly, serve our interests, but we need to be scientific and get the data.
  4. How important is our international prestige as a nation of scientists (and engineers) to our economy? Do we have any data, for example, to show the link between FDI and our university rankings or our publication outputs? Again, it would seem plausible that certain types of companies would like to locate in a highly science-literate society but can we quantify this effect?

These are hard questions (and questions for economists and social scientists, not scientists) and maybe it is not even possible to provide answers, at least not yet. Perhaps many academics presume that the answers to these questions are so self-evident that they are not even worth asking. I think that would be a mistake though because I suspect that many observers, including policy-makers, will view the Irish Times letter as academics doing their usual thing and merely looking for funding for ‘pet projects’.

Posted in education, Research, Uncategorized | 1 Comment

Jobs that don’t exist and all that xxxxx

Foolishly (I should be doing other things), I’ve just been watching some of the talks from the Global Education and Skills Forum  being held in some exclusive resort in Dubai where everyone was sitting in nice white armchairs. There was much talk about the 21st Century Learner. Personally, I think this whole concept is, to coin a phrase, bullshit, but I like to think I have an open mind so I did a bit of 21st learning myself, i.e., I Googled around the topic.

I quickly came across this article by Andreas Schleicher, the top education guru in the OECD, and a keynote speaker at the Forum. Here are some quotes from his article.

For most of the last century, the widespread belief among policymakers was that you had to get the basics right in education before you could turn to broader skills. It’s as though schools needed to be boring and dominated by rote learning before deeper, more invigorating learning could flourish. Those that hold on to this view should not be surprised if students lose interest or drop out of schools because they cannot relate what is going on in school to their real lives.”

I’ve no problem with the first sentence but the second is a non-sequitur of the highest order. Where on earth has it ever been implied that acquiring the basics of a discipline has to be boring and based on rote learning? Anyone who actually teaches or has been a student knows that acquisition of basics can, in fact, be ‘invigorating’. In engineering, for example, basics are best acquired by lots of active practice – ‘doing the problems’ as we say – and solving problems is one of the most rewarding things a student can do in education. But the last sentence is one of the more worrying in the article and it touches on something that has been a big issue in the UK in particular. This is the presumption that students – mainly young people – will only engage with education if it relates in some ways to their ‘real lives’. This is a desperately bankrupt, utterly utilitarian view of education that shows a complete lack of faith in students, especially their capacity to engage with all sorts of knowledge. In fact, students will engage with everything from history to art to literature to science, if it is a presented in an engaging manner. But presenting material in an engaging manner does not mean contorting it to fit our daily lives, and engaging doesn’t have to mean ‘innovative’.

A generation ago, teachers could expect that what they taught would last their students a lifetime. Today, because of rapid economic and social change, schools have to prepare students for jobs that have not yet been created, technologies that have not yet been invented and problems that we don’t yet know will arise.”

Let’s say a generation is about 40 years or so and let’s think about the world in 1975. Television (coloured!) was becoming a feature of daily living, the first calculators were being used, digital watches were appearing on the scene (not for long) and inter-continental air travel was becoming commonplace. The Voyager spacecraft were being built in anticipation of their launch into deep space in 1977, the term ‘fractal’ was coined, monoclonal antibodies were produced and on April 4th, Microsoft was born. This was no static world and where exactly is the evidence that the education system presumed that the world was standing still? Indeed, over the next 10 to 15 years, we would go from a situation where we had a handful of mainframe computers in this country to one where everyone had or at least aspired to having their own microcomputer on their desk. Space Invaders, Asteroids and that hypnotic and irritating tennis game appeared and the gaming industry was spawned. Work was made easier as printers, plotters (remember them!) and faxes (remember them) became commonplace. Email arrived. Those of us who were educated in that period were ready, willing and even excited to adapt. No one ever suggested that we needed to be educated to be adaptable, flexible graduates. That was our mind-set, forged from a combination of common sense, enthusiasm and a general awareness of the world around us. Although by no means special, those of us educated in the 1970s and 1980s adapted to the internet age of the 1990s quite easily and there is no evidence that the events of 2015 are so different that the education of today needs to be radically transformed in some way. There is no doubt, however, that education in 2015 poses many new challenges but many of those are not because of the ‘rapidly-changing’ workplace, or ‘connectedness’, or any of those things; it’s due to the fact that education, especially higher education, is no longer the preserve of the few but the right of the many. The education ‘audience’ is just more complicated now.

However, educational success is no longer about reproducing content knowledge, but about extrapolating from what we know and applying that knowledge to novel situations.

Sorry, but when was education ever about reproducing ‘content knowledge’ only. The very essence of my own discipline, engineering, is the application of knowledge and methods, and I’m sure others can make the same argument for their discipline.

If we spend our whole lives in the silo of a single discipline, we cannot develop the imaginative skills to connect the dots or to anticipate where the next invention, and probable source of economic value, will come from.

We hear this kind of thing a lot and, yes, some of the big problems of science and society are on the borders between disciplines, but many are deeply embedded in the disciplines. A huge amount of important, imaginative work is done that is discipline-based and nowhere near the borders. Furthermore, even if the big problems are on the borders, that does not mean that they cannot be tackled by teams of experts with complementary knowledge and skills, and it does not mean that individuals need to learn in a multidisciplinary way. The Apollo missions were the multidisciplinary challenge par excellence and involved the collaboration between experts, experts who were educated to have deep knowledge of their discipline. The Eagle would not have landed, for example, without the thousands of pages of computer code written by Margaret Hamilton, a woman who was single-minded in the pursuit of excellence deep in the heart of her own discipline. The world needs its Margaret Hamiltons.

Traditionally, you could tell students to look into an encyclopaedia when they needed information, and you could tell them that they could generally rely on what they found to be true. But today, literacy is about managing non-linear information structures. Consider the Internet. The more content knowledge we can search and access on the web, the more important the capacity to make sense out of this content becomes. This involves interpreting the frequently conflicting pieces of information that pop up on the web and assessing their value, a skill rendered essential by the appearance of the Internet.”

I’m not sure what point the writer is trying to make here but, in my view, this is a really good argument as to why students need to be taught content (as opposed to vague 21st century skills), because without having been guided by an expert  through the knowledge of any given field, the learner will have not have the framework of their own personal and reliable knowledge to be able to interpret the vast amount of ‘information’ that can be accessed on the web. Unfortunately, there is a large tendency these days for proponents of 21st century learning to figuratively throw their hands in the air and suggest that it’s a waste of time acquiring any knowledge at all – all that is need are learning and even ‘unlearning’ skills, and thinking skills, they say.

Rather than just learning to read, 21st century literacy is about reading to learn and developing the capacity and motivation to identify, understand, interpret, create and communicate knowledge.”

So, what is being said here? In previous generations, children were taught to read just for the sake of it, so that they could make sounds??? I think the writer was just trying to say something ‘clever’ by flipping ‘learning to read’ into ‘reading to learn’. Nice sound bite but devoid of meaning.

Here’s what I think: Education has been hijacked by ‘educationalists’ who have a birds-eye, hands-off view of education and who seem to exist in their own parallel universe where they talk amongst themselves in vague, unsubstantiated, yet plausible-sounding generalities.  The have invented an educational past that never existed and are now obsessed with jobs that don’t exist. Sounds like they live in an imaginary world.

Posted in 21st century, education | Tagged , | 3 Comments