AI After the Bust

I was a postgraduate student when the dot com mania hit. Everyone was trying to start an online business, and investors were throwing fistfuls of cash at anyone with a shonky business plan and a page of HTML. Of course, in 2000 it all went bad, the bubble burst in the (in)famous dot-bomb crash. And a lot of dot com companies disappeared. Many years ago I read the book "Dot Bomb" by J. David Kuo that described how this happened with the internet company Value America. Part of that book's subtitle was along the lines of "from lunatic optimism, to panic and crash"

There is a similar bubble now with AI. Everyone is shovelling AI into their products, or they're starting an AI company that pushes their own twist on large language models, or they're trying to find a way to insert AI into their operations just to be able to say they use AI.

The bubble is going to burst. And when it does, a lot of AI companies are going to get taken out.

So does this mean the end of AI?

Well, no. When the dot com bubble burst, a lot of internet companies were taken out, but other companies survived. These are the companies that actually provided useful services, like Google, or sold products people actually wanted, like Amazon, or facilitated small commercial operations like eBay and TradeMe.  Since then, they have become part of the fabric of our existence, part of the background hum of our lives. And other internet companies have also arisen that build on the idea that if you do something useful, in a way that makes it easy for money to get to you, money will get to you. 

In my opinion, the same principle applies to AI. Those AI companies that provide a useful service, in such a way that money can easily get to them, will continue to make money and will thrive. Those companies that do not do these things will not survive. The lunatic optimism will fade away, just as it did with the dot coms. And the panic that accompanies a crash will fade away. And those that survive the crash, will keep going, and AI will be just become part of the background hum of our lives.

Down with middlemen!

If there is one thing that the rise of e-commerce sites like eBay and Amazon.com has shown it is that the middleman is doomed. Entire bookshop chains like Borders have vanished from the face of the Earth, largely because they were unable to compete with a model that has no physical presence. Bookshops are middlemen: they connect one group of people (publishers) with another group of people (book purchasers, consumers). And the one thing that the Internet is really good at is getting rid of middlemen. Even publishers are middlemen, they don't produce the product, the authors do. The rise of self-publishing is the strongest indicator yet that the publishers are, like bookshops, an endangered species.

Online sales sites like e-bay (and Trademe, Gumtree) have also had an impact on retailers, and second hand dealers in particular: when we moved from Australia back to New Zealand, we had to sell our car and a few others bits and pieces. We didn't take the car to a second hand car dealer, or call a second hand furniture shop about our excess furniture: we just put some adverts up on Gumtree. While it is harder for online retailers to compete on items that require large volumes such as groceries, for smaller-volume or speciality items online sites are slowly but surely eliminating the traditional merchants. In the last ten years the only time I've booked air travel through a travel agent was for business travel, and then only because my employers had a policy of booking through certain agents.

This all raises a question: what is a middleman? If we define middlemen to be someone who does not produce, add value or provide a service that cannot be automated, then a huge number of current professions come under that heading: real estate agents, immigration agents, literary agent, property management agent... Basically, anyone with the word "agent" in their job title is a middleman and is doomed.

How does this relate to computational intelligence or academia? Well, what if journals and universities are really middlemen?

In the past I have blogged about how open-access journals are the future of academic publishing. But how much value do journals of any kind really add? A journal will arange peer-review, format the accepted articles and assign volume/page/DOI numbers. Apart from peer-review, each of these steps can be automated. In an age when every article published is available online, and are indexed by sites like Google Scholar and Citeseer, journals don't add much to the publicity of an article - in fact, the most effective way of publicising an article seems to be to blog or tweet about it. This is still the major advantage of open-access journals, as anyone with an interest can download and read the article (and hopefully cite it).

The measure of the quality of an article is the number of citations it receives, much more so than the supposed quality of the journal it is published in. Metrics like impact factor are so bogus as to be meaningless, despite the arrogant attitude of editors who deem submissions unworthy of publication in their august journal, without bothering to send them to peer-review. A good article will be cited more, no matter where it is published. Articles that aren't useful won't be cited. In other words, articles now can stand on their own, they don't need the support of journals to be useful. The journals, therefore, are middlemen, standing between the producers (the people who do the research and write it up) and consumers (the people who are reading and citing the research). Do we really need journals to arrange peer review? Or is there scope for a journal-agnostic, peer-review service for individual articles?

If individual articles can now stand on their own, how about individual academics? The Khan academy has been described as a revolution in teaching numerous times, and open courses like those offered by MIT have had thousands of students. In many ways universities are middlemen, providing access to resources (academic staff) to consumers (students). Universities provide tuition, consultation (students can ask their instructors for clarification), assessment (tests, assignments and exams), and accreditation (a degree / diploma from an institution has a certain credibility). Tuition can be supplied directly by the lecturer via sites like YouTube. Consultation can be done via discussion boards and live chat. Accreditation remains as an open problem. There are a huge number of accreditations available in a vast range of technical subjects: the IT industry in many ways leads the way in this, with certifications from Microsoft, Cisco, CompTIA and others. Professional organisations like the IEEE publish bodies of knowledge that graduates in certain disciplines are expected to know, and it's only a matter of time before this is expanded to include computational intelligence. Practical work is harder to deal with, but even then the large amount of open source software available means that anyone with a cobbled-together Linux box and a basic internet connection can not only do the lecture and practical work associated with undergrad study but also access the accreditation offered by numerous organisations.

The only problem for which I cannot see an obvious solution is, how would the lecturers get paid? Locking material behind paywalls won't work, people just won't use it. Also, a fixed fee won't work either: $500 might not seem like much for someone in the western world, but for someone in parts of Africa, it's more than they see in a year. The pay-what-you can model might work: this is where someone pays as much as they think something is worth, or as much as they can afford. A few people might take advantage and pay nothing when they could afford to pay, but most people are pretty honest and will pay a fair price. The accreditation agencies could also pay a referral fee to lecturers who direct students to their services, much like the Amazon affiliates program.

Universities would still survive, there still needs to be places where research is carried out, and training of the next generation of researchers (postgraduate students) takes place. The survival of journals is a bit less certain, as self-published peer-reviewed articles are much easier to do. Whatever happens, though, middlemen are on the way out.

The Problem with Academic Journals 7

The following quote was in an email I received from the editor of a certain prestigious general science journal:

"Your manuscript is now undergoing an initial screening to determine whether it will be sent for in-depth review. We will notify the corresponding author of our decision as soon as possible."

That really annoyed me. It annoyed me because it is not the job of the editor to screen submissions. Sure, it is appropriate for them to check that the paper is formatted correctly, that there aren't big sections of it missing, and that it fits the theme of the journal (which is not the case with general science journals like the journal this paper was submitted to). The kind of screening this editor is talking about it a kind of pre-peer review, where the editor is determining whether the paper is worthy of being considered by their august publication. It is, in fact, a rather extreme form of academic arrogance.

Having a paper rejected by peer review is one thing, but being rejected because one person doesn't think it's worthy enough? So many of my colleagues have had so many perfectly good papers rejected by editors without going to peer review. The purpose of peer review is to find errors in the science (and have no doubt about it, computational intelligence is a science). If there are no errors in the science - that is, there are no discernible errors in methodology or interpretation of results - then the paper should be published. Even a rejection is useful, as it allows the authors to improve their research. But editorial rejections eliminate even that, they make the entire process of submitting to that journal a waste of time.

As I've said many times before, the solution is to go to open access journals. Peer review will help catch the errors, and the people reading the papers (and there will be a lot more of them reading open access papers than subscription-only papers) will find the errors the peer reviewers missed. But arrogant editors from expensive subscription-only journals will soon find themselves presiding over a shrinking author base.

On Being a Post-doc

After completing a PhD, most people who wish to stay in academia end up doing one or more post-doctoral positions. Experience as a post-doc is a prerequisite for a career in scientific research, as it is during your post-doc career that you get exposure to ideas and techniques outside of your PhD, and work with a wider range of people than you did during your PhD. The chances of going into a permanent academic position, without doing at least one post-doc, are very slim (most people who manage to do this tend to wind up in the same department they did their PhD in). I've done three post-docs, at Lincoln University in New Zealand, at the University of Sydney, and the University of Adelaide, both of which are in Australia.

So, what's it like being a post-doctoral fellow?

Basically, it sucks. Most post-docs are for two or three years. This term is fixed as the position is usually tied to a particular research grant, which is itself of fixed duration. This means that even if you do extremely well in your research, there is no guarantee of further employment after the contract ends. This means that as a post-doc, you will probably be changing jobs and cities every two years. If you're young and single, that's not entirely a bad thing: travelling and living in different places broadens your mind, can build a wide network of friendships and helps you appreciate different ways of life. Things get harder if you are a couple, as your partner also needs to find work in your new home. If you have even one child, it's a nightmare: you need to find a new school, your child faces the awful wrench of leaving their friends behind, if they're in after-school activities they need to be organised all over again, and if they have even minor health issues, finding adequate care for them can be very challenging. The stress that this can place on your relationship is enormous. In short, being a post-doc is a young (single) person's game.

If your post-doc is tied to a grant, then you will be working on someone else's project. In other words, you'll be working on something that is interesting to someone else (the grant holder). This also means that the outputs you produce (that is, papers) will be of benefit primarily to the grant holder rather than you.

While you should concentrate on doing the work you are paid to do, if you want to move up the academic ladder, then you also need to demonstrate the ability to do independent research. So, in addition to working a full-time job, you're also working part-time on your own research programme.

On top of the above are the dangers of any workplace: while most post-doc supervisors are good and kind people, they get their positions by being good researchers (or occasionally good politicians), not good managers. In the worst case, you might end up working for a narcissistic sociopath. Doing a post-doc with the wrong supervisor (or supervisors) can make your life a living hell. Sociopaths can be pretty hard to spot, too.

My experience is that it can take six months or more to find a new position, which means that shortly after starting a post-doc, you need to start looking for another. If your career is a chess game, then you need to start getting your pieces into place sooner rather than later.

To sum up, being a post-doctoral fellow means a semi-itinerant life of uncertainty and upheaval, serving the research needs of others, while also planning a future career that might not happen.

Was it all worth it for me? While there are many things I would do differently if I had the chance to do it all again, I don't want to live my life in regret: the things in my life, the good and the bad, the joy and the hurt, have all made me the person I am. But I do regret the hurt it has caused my family. Being a post-doc is hard on everyone if you have a family. It's not all bad news, though, and in a future post I'll be discussing ways in which you can make your post-doc career successful.

Publishing and perishing under gameable metrics

My alma mater is in the New Zealand news again, and again it is to do with gaming the metrics by which the research performance of New Zealand tertiary institutions are measured. This time, the article describes how many staff with poor publishing records have been made redundant from the university (that is, they have lost their jobs) prior to the assessment later this year. While I have little sympathy for those in permanent lecturing positions who do not publish (see my previous comments here and here) in this case it seems like the staff who have lost their jobs are predominantly teaching staff, or staff who are still developing their research record (see this post from one who lost her job for the same reason some time ago). If that is the case, then I have to say that the university administration is making a mistake.

Teaching takes a lot of time and energy (my last semester teaching at Otago, I was in the office at least six days a week, and often worked from 7:30 in the morning to 9 or 10 at night). The purpose of having teaching-only staff is to take some of that load off of the lecturers so that they can do their research. Indeed, the major thrust of the article is that the redundancies are putting more stress on the remaining staff, as they are having to pick up extra teaching in addition to lifting their own research outputs. While the teaching load could in theory be reduced by hiring contract lecturers (who would not, as I understand it, be assessed) I have already posted on why this is a bad idea.

From my research with evolutionary algorithms, I know that optimising to one criteria or metric seldom results in optimal or robust systems. By optimising their staff to one (flawed and gameable) metric, the University of Otago is reducing the robustness of their institution. The long-term outcome of these redundancies is yet to be seen, but I do not think that it will be good for anyone concerned. Non-performers need to be removed, for sure, but early-career researchers need coaching and leadership to develop. They don't need the great big stick stick of the threat of redundancy waved at them (such threats are more often than not a sign of dysfunctional management, rather than a sign of competent leadership).

Ultimately, only those who set the metrics can resolve this situation. As long as a metric can be gamed, then institutions will game them. In the meantime, people will have their lives upended and their careers destroyed by narrow-minded administrators and cynical political operators who are trying to wring a few more points out of the system to make themselves look good.

There's no un-gameable metric

I've been a bit quiet on the blog front lately, mostly because I've been working like a dog on several projects, including writing tools for ecological modelling, re-working some websites, and fulfilling my duties both as guest editor of my special issue of Evolving Systems on Applications of Evolving Connectionist Systems, and as vice-chair of the IEEE CIS Social Media Subcommittee. It was also school holidays the last two weeks here in South Australia, and I was able to spend some quality time with my little girl.

Never fear, I'm working on several new blog posts on a variety of topics, including: the relationship between computational intelligence and data mining; further thoughts on doing a PhD (a follow-up to this post); my thoughts on the value of a computational intelligence degree; and my thoughts on collaborating with other researchers. The topic of today's post, though, is assessing academics and universities.

My alma mater has been in the New Zealand news lately (see here and here) after the release of a report by accounting firm KPMG that suggests that Otago had gamed the New Zealand government research assessment process to give themselves a higher score than they were entitled to.

The Performance-Based Research Funding (PBRF) framework rates the research outputs of eligible staff and uses those ratings, along with metrics of institutional performance such as number of research degrees completed, to assign an overall score to the institution. Staff can be rated as R (research inactive - bad for this exercise), C (research active / good), B (very good) or A (world-class). The fewer R's and C's an institution has, and the more B's and A's, the better the institution's score. Something like 25-30% of an institution's income will be determined by this score. There is also the huge marketing advantage of an institution scoring highly in relation to the other universities: in the first PBRF round in 2004, Auckland University made a lot of the fact that their staff were, on average, ranked highest in the country, while Otago made a lot of the fact that they were ranked highest as an institution. This is despite the government of the day clearly saying that PBRF wasn't supposed to be used for such comparisons, or as a management tool.

Eagle-eyed readers may have noticed the term "eligible staff" earlier in the previous paragraph: it is this facet of the process that Otago is accused of gaming.

The accusation is that Otago inappropriately classified staff it knew would get low scores as ineligible for assessment, and thus artificially boosted its ranking compared to other New Zealand institutions. Otago is also accused of firing, or pushing into retirement, staff based on their anticipated PBRF score. The vice-chancellor denies these accusations, and the whole thing is turning into a "he said / she said" situation.

Did Otago really do this? I honestly don't know. I do know that when I was working at Otago in 2004 (the first PBRF assessment round), I was assessed fully, and fairly, even though it would have been pretty easy to classify me as ineligible for assessment. I don't think my score in PBRF at the time was particularly helpful to their overall ranking, but maybe it wasn't too harmful, either. My point is, this entire drama shows that there is no metric of academic performance, of an individual, an institution, or a publication, that can't be gamed. That is, there is no metric that can't be manipulated so that an individual, institution or publication gets a higher score than they otherwise would. Journals can boost their impact factor by asking authors to cite articles from within that publication (and I have had editors ask me to do this). Individuals can boost their h-index by auto-citations, or by organising a special issue and asking every author to cite a review article they have written. Institutions can raise their assessment by head-hunting the top-performers in their fields, or by hiding staff from assessment. Some might argue that it is only prudent to game metrics whenever possible: after all, the future employment prospects of an academic, or the future financial security (and, therefore, job security of staff of) an institution depends on getting a good score on whatever metric is being used. As long as no rules are being broken, and the questions are being answered honestly, what's the harm? If there is wiggle-room, or room for gaming of the metric, isn't it the assessor's fault for designing an inexact metric? Others might argue that adherence to the spirit of the assessment is more important, more fair, and that gaming should be avoided.

This all means that there is no one metric you can use to assess an academic. You have to look at the entire picture: you have to look at their publication count; where they have published; what fields they have published in; how much teaching they have done; their teaching assessments; the quality of their institution; and their service to their institution(s), to professional societies, and to the community. I hope that one day I will rate highly in all of those areas, but for now, don't judge me just by my h-index alone.

Publishing or perishing 2

Or, big trouble in little USyd.

In a previous post, I discussed the principle of "Publish or perish". This phrase is a succinct way of saying that the most important metric by which an academic is judged, is their publication record: those who do not publish enough will not be valued as highly as those who publish more. Recent developments in Australian academia have made this literally true.

The University of Sydney is dismissing 100 academics who have not published at least four times in the last three years. Early career researchers are apparently exempt from this threshold, so it is only senior academics (presumably permanent staff members) who are at risk. In other words, they are in senior positions, yet they have not published, so they are perishing.

To put that number (four papers is three years) in perspective, I'm not a permanent staff member, yet in the last three years I've published fifteen times, and have more than a dozen publications in the pipeline (I expect to publish them this year). While I am currently full-time research, I have taught full-time in the past, yet I still managed to publish research. During my final year teaching, I published papers, was finishing my PhD, and taught / administered two undergrad courses. I got married that year, too.

The only thing that garners any sympathy from me is that claims are being made that late last year academic staff were told that an average publication rate of 0.8 per year (four papers in five years) would be satisfactory. If that is true, then management have moved the goal-posts, which strikes me as rather unfair. However, even when I was working as a post-doc at the University of Sydney, I was expected to produce at least two papers per year. In my current position, my target output is at least six per year (I got ten last year, and I'm on track to get twelve this year - the best way to meet a target, is to aim to exceed it). Did senior lecturers really think they could get away with less? "Publish or perish" is a very old saying.

Now, the University of Sydney is taking this action to make up for a massive short-fall in income (the exact reasons for this sudden short-fall are rather murky) but what would the effect be if this hard form of publish or perish were enforced more often at universities? It might have the effect of sweeping out those academics who have become complacent in their positions, or who are approaching the end of their academic careers (that is, they are no longer capable of performing research at the level required). This would in turn free up positions for younger staff, who are capable of producing publications, yet can't find permanent positions because they're being held by unproductive senior staff. Are there any other professions where those who do not perform, get to keep their jobs? Of course not. A major part of being an academic is publishing: if you're not publishing, you're not doing your job. If you're not doing your job, do you deserve to keep it?

On a humane level, I'm sorry for the people who are losing their jobs. But honestly, I wish this standard were applied at more universities.

Common grammatical errors

English is the result of Norman soldiers attempting to pick up Anglo-Saxon barmaids, and is no more legitimate than any of the other results. - H. Beam Piper

Despite the murky origins of the English language, it is the language of international commerce and science: if you attend an international conference in, for example, China, the presentations and proceedings will be in English. The same applies to the international scholarly journals. Being able to communicate effectively in English is therefore essential for anyone who wants a good career in academia.

Now, if you ever heard me speaking, you'd have probably have trouble understanding me: my New Zealand accent means that I flatten my vowels, my voice is nasal, and I speak too quickly. But, I do write fairly well, and I do avoid the grammatical errors below that really annoy me. Don't feel too badly if you aren't a native English speaker and make these mistakes: I noticed most of these mistakes while teaching New Zealand undergrads. That is, these are the kinds of mistakes that native speakers make in some parts of the world. They're still mistakes, though, and should be avoided.

1) Reversing imply and infer. For example "What are you inferring by saying that?". An inference is made from an implication. That is, the implication is made, then the inference is drawn from it. For example,  someone once said to me: "Are you inferring I'd lie?", to which I replied "No, I'm implying you'd lie - you are inferring it". I know, I'm a bad, bad man.

2) Following from that is the distinction between implicit and explicit. If something is implicit, you know about it through inference. If something is explicit, it is stated. Saying that something was "explicitly implied" (and I have seen that written, but a university student no less) is not just bad grammar, it is complete nonsense. If something is explicit, it cannot be implied.

3) Literally means what you said is as it actually is. If you were to say "it is literally raining cats and dogs", then cats and dogs should be falling from the sky. Jamie Oliver is particularly bad for abusing this word: "and you literally put the mint in the mixture..."!

4) Your and you're. "Your" means it belongs to you. "You're" means you are. Can you see what is wrong with this picture?

Dear Tick Tax: this is your sign, and you're using incorrect grammar.

5) Using the term "begs the question". The original meaning of this is to assume as true something that cannot be taken for granted. The more correct way to saying this is "raises the question". This is probably a lost cause now, as the incorrect meaning of begging the question is widely established.

6) Its and it's. It's is short for it is. Its means something belongs to it. For example: "I saw a dog and said "it's wagging its tail"".

7) There, their and they're. There is a place. Their means something belongs to them. They're means they are. Example: "They're at their house, which is over there".

These are the errors that annoy me the most. For two excellent articles on common errors in academic writing, see "Don't torture your reader" parts I and II, by Professor Corey Bradshaw at the University of Adelaide.