Three helpful stories

Having spent a big chunk of my working life teaching, I've learned how valuable metaphors can be when communicating. Some of the most effective metaphors are in the form of funny or amusing stories. This applies to doing research as well: with the Internet originating in universities, a lot of stories or parables related to research have been circulating on the Internet for a very long time.

None of the stories below are mine, they have all been circulating for a long time. They are, however, relevant to doing research.

The Pot Roast

One day a young woman was cooking a roast. Her husband asked her why she cut the ends off of the roast before putting it in the over. She replied "I don't know, that's the way my mother always did it". Later that night, she phoned her mother and asked her why she always cut the ends off of the roast before putting it in the oven. Her mother replied "I don't know, that's the way your grandmother always did it". So the young woman phoned her grandmother, and asked her why. Her grandmother replied "Because the roast was too big for the pot".

The story of the Russian sentry in the field is in a similar vein.

There might have once been a good reason for doing something a certain way, but that doesn't mean that that reason is still valid. Methods and procedures need to be re-evaluated regularly. Fixating on one way of doing things means that you can miss out on improving your methods and getting better results.

The Three Statisticians

Several versions of this story exist.

Three statisticians were out hunting when they sighted a deer. The first statistician fired and missed one metre to the left. The second statistician fired and missed one metre to the right. The third statistician cried out "We got it!".

Just because you can find an average of values, doesn't mean that that average is meaningful. This applies to other statistical measures as well. It might sound impressive if you use a complicated or obscure statistical method when processing your results, but if it's the wrong method the outcomes won't tell you anything. A good working knowledge of statistics and measurement theory is essential when analysing, and interpreting, data and results.

The Rabbit and the Fox

Another story were there are several different versions online, but all have the same punchline.

One day a fox was walking through the forest when it came across a rabbit sitting outside its burrow, reading a pile of papers. The fox asked the rabbit "What are you doing?" to which the rabbit replied "I'm doing the literature review for my thesis". "What's your thesis on?" the fox asked. "It's on the superiority of rabbits to foxes. Would you like to come inside and discus it?". The fox licked its lips hungrily, followed the rabbit into the rabbit burrow, and was never seen again.

Some time later, a wolf was walking past the burrow when it saw the rabbit sitting outside, poring over a thick volume. "What are you doing?" the wolf asked the rabbit. "I'm proof-reading my thesis" said the rabbit "It's on the superiority of rabbits over wolves. Would you like to come inside and discuss it?". The wolf licked its lips hungrily, followed the rabbit into the rabbit burrow, and was never seen again.

A few months later, a hare was hopping through the forest past the rabbit burrow, and saw the rabbit sitting in the sun, relaxing. "What are you doing?" the hare asked the rabbit. "I'm having a break - I've just defended my thesis on the superiority of rabbits to foxes and wolves. Would you like to come inside and discuss it?". The hare followed the rabbit into the rabbit burrow. In one corner was a pile of fox bones. In another corner was a pile of wolf bones. Sitting between them was a lion.

So, it doesn't really matter what your thesis is about, as long as your supervisor is a lion.

Getting good advice when you are starting out is essential. The right mentor can set you up for life. The wrong mentor can ruin you. I've seen students lose their careers before they began simply because they chose the wrong PhD supervisor. These people have lost years of their lives and gone thousands of dollars into debt, for no gain. Some of them were pushed out before they submitted their thesis, while others failed their thesis examination because their supervisors failed to prepare them for the examination process. Be very careful when choosing a supervisor or a mentor. Better to choose a lion than a fox.

Bias in AI

When my daughter was a toddler, we bought her a batch of Looney Tunes DVDs. She loved watching Daffy Duck, Porky Pig, and Bugs Bunny. On her third birthday, we gave her a Road Runner DVD. She had her birthday at her grandparents' house, and so that's where she first watched Road Runner, after lunch.

The next time we visited, a couple of weeks later, straight after lunch she got up and walked towards the living room, saying she was going to watch "meep meep". I had to explain to her that that DVD was at home, it didn't stay at Grandma and Grandad's house.

Why did my daughter do that? It's because she had only ever seen Road Runner at Grandma and Grandad's house. In other words, she only had one example of where she could see Road Runner, so that was where she thought she would see it.

A couple of years later I was attending the viva of a PhD student, whose thesis I had examined. The thesis topic was predicting a medical condition from gene expression data. Gene expression was assigned certain values according to how much the gene was active. During the viva, the student made the comment that "expression was measured for those patients who had the condition, and was set to zero for those who did not". I was gobsmacked, as this meant that there were really no gene expression values at all for patients who did not have the condition.

A few years after that I was examining a PhD thesis that used neural networks for earthquake prediction. The central idea was that data from seismographs could be used to predict if an earthquake greater than a certain threshold would occur in the next few days. The data was taken from Canterbury in New Zealand, and started from September 2010, and went for just over a year.

The problem with this of course is that on the 4th of September 2010, the Darfield Earthquake struck Canterbury. The aftershocks continued for more than two years, that is, the length of the data set.

These are all examples of making predictions based on biased data. My daughter had biased data because she had only ever seen Road Runner at her grandparents' house. 

The first student had biased data because there was really only data for one class of patient. They got good results, but that was because they trained a model on biased data then tested it on biased data. Their model would have failed utterly if it had been tested on a different data set.

The second student had biased data because the entire data set was constructed over a time when it was known that there were going to be earthquakes. So again, their model worked well when trained and tested on the data set they had, but it would have failed utterly if tested on a different data set. Hilariously, when I challenged the student on this in their viva, they replied "It's the right kind of bias"! A much better data set would have been one that extended over several years before and after the Darfield Earthquake, and had been taken from different regions. 

While biased models in academic settings might not cause a lot of harm - other than to examiners' calm - such models are being used in production systems. This seems to be quite a widespread problem as well. It can even have deadly consequences.

Biased data leads to biased models. This is such a simple concept, yet so many people who build AI models don't seem to grasp it. Identifying biased data can be tricky, as it requires a solid understanding of what the data represents, how it was gathered, and what it is to be used for. 

More insidiously, models built with biased data can show very good results. They will train well, and they will test well, as long as the test data is from the original, biased, data set. That is, as long as the test data is as biased as the training data, the model will show good test results.

This makes the sourcing and use of an independent testing data set essential. And that is probably the number one thing anyone to do to do avoid bias in AI. It's not foolproof - there might still be systemic biases in the process that generates the data - but it is an essential first step.

Cargo Cult Computer Science

I recently attended a presentation by a post-graduate student that I thought was a little bit funny. The presentation was about the experiments they had done on classifying classical music. At the end of the presentation, they proudly declared that algorithm X could identify the composer of a piece (one of Vivaldi, Bach or Mozart), from half a second of music.

The first query I raised was, how many notes are you going to get in half a second? Classical music tends to have a relatively relaxed pace (at least, compared to the music I enjoy) so I doubt there would be more than one or two notes in each sample. The response was, algorithm X is really good at classification so half a second is enough.

The second query I raised was as follows: there was only one piece from each composer in each sample, and the Vivaldi was entirely strings, the Mozart was entirely piano, and the Bach was a mixture of instruments. How do they know that the algorithm didn't just learn to classify instruments?

This is similar to the famous example from the early days of neural networks, when perceptrons were being trained to distinguish photographs that contained images of tanks and those that did not. After some very good results at the start of the project, a second batch of images utterly failed. The reason for that failure was traced to the fact that the photographs with tanks had been developed using a slightly different process to that used to develop the photographs without tanks. That resulted in a slight difference in the overall brightness of the photographs. The neural network had simply learned to distinguish between lighter and darker photographs.

Now, the people who were looking for tanks did one thing right: they tested their algorithm with more data. The post-graduate student at the start of this story didn't do that. They just looked at the results they got, which fit their expectations, and stopped there. That meant that the conclusions they were drawing were not supported by the evidence.

The American physicist Richard Feynman famously spoke of "Cargo Cult Science". This is research that has the superficial form of science, but does not follow the rigor expected of the scientific method.

The scientific method is a process that has developed over many centuries, and requires a certain rigor and self-criticism that is intended to prevent erroneous conclusions being made. It requires scientists to be completely honest with themselves, to consider every objection to their research method and possible factors that could be influencing their results. The scientific method is supposed to prevent researchers from just seeing what they want to see and instead see the reality. The post-graduate student did not do this, and so their conclusions are not necessarily valid.

I've seen this in a lot of papers in computer science, and in more than a few post-graduate theses. Experiments are performed, results are gathered, and conclusions are confidently espoused about the value of their approach. Yet they never consider what else could explain those results. They never consider whether their data is biased in some way, or if their method is flawed so that certain results are favoured over others.

I think there are several reasons this occurs. Students in computer science are not necessarily trained in the scientific method, so they can hardly be blamed for not following it. It is human nature that researchers want their approach, their new algorithm, to work, so they develop a kind of wilful blindness to the flaws in their experimental approach. Finally, and more insidiously, researchers are under immense pressure to publish: "publish or perish" applies in computer science just as much as any other field of academia. It is only through publishing papers that researchers gain employment, get promotion, and secure research funding. There is, then, a system set up to favour rapid and uncritical publication of supportive results and to suppress unfavourable results. We have created a system that favours Cargo Cult Computer Science.

Computer science, if it is to remain worthy of the appellation "science" must fully embrace the scientific method. This means being rigorous, and being self-critical. The consequences of not doing so, could be severe for everyone in the field.

Examining Postgraduate Theses

I've examined a number of postgraduate theses by this point in my career. These are Doctoral and Master's theses from New Zealand and overseas institutions.While most of those theses have been a real pleasure to review, some have been real horrors. Even the ones I enjoyed examining often had errors in them. The errors that appear, though, tend to be the same kind of errors. That is, candidates for higher degrees are making the same errors even though they are from different institutions. So, I've written this post to discuss these errors, and how to avoid them. This post, then, describes what I look for in a thesis, and what I don't want to see in a thesis. Since I've mostly examined Doctoral and Master's theses, these are the focus of this post.

The Examination Process

This varies a bit according to the institution, but the general structure is the same. The process usually goes something like this:

• I am asked if I am interested in examining the thesis. I usually get sent the candidate's name, the title of the thesis and the thesis abstract.

• I say "Yes, I am interested" although I occasionally say "No" if the thesis is outside of my field of expertise.

• Some time later, I receive an examination pack. This usually contains things like:
• The thesis to be examined
• Institutional guidelines for examiners
• A marking sheet, where I make my recommendation and comments
• Other forms for payment of the honorarium and tax 

• I read the thesis several times, making comments on the pages each time.

• Using my comments, I write a report on the thesis and make a recommendation.

• There is sometimes an oral exam / viva held later, although more and more institutions seem to be moving away from those now.

Examiners usually get paid an honorarium, and as a New Zealand resident the honorarium I receive from a New Zealand institution has always had tax deducted from it. The size of the honorarium varies between institutions, but if you consider the time it takes to examine a thesis, it comes out at substantially less than minimum wage.

I prefer to receive the thesis as a PDF, as I find the examination process much easier when done electronically. I usually load the PDF onto my tablet, so I can get some examination work done during my daily commute on the train. When I submit my report and recommendation, I send the marked-up PDF with it.

Clarity

A thesis should be clear. Don't leave the hard mental work to the reader of the thesis! Lay everything out for them, especially why are you doing this? There is presumably a reason for doing the work you did, apart from "your supervisor told you to do it". The motivation for pursuing the research in the thesis should be laid out clearly and as early as possible.

The literature review should relevant to the topic of the thesis. I don't want to have to wade through pages of literature review that don't have anything to do with the thesis. Or, to put it another, way, don't "stuff" your literature review, it just annoys the examiner. A good question to ask about any part of the literature review is "why is this in the thesis?".

The literature review should be to-the-point. Anyone examining the thesis will have been carefully chosen and they will be experts in the field. Spending pages reviewing or describing material that a third-year student in the field should know is a waste of time and space. Better to just cite the key relevant papers and move on.

The literature review should also be critical. What are the holes in the literature? What is wrong with what has been previously published? What could have been done better? The work in a thesis should build upon what has gone before, it is incredibly rare that a thesis introduces an entirely new field.

If a thesis isn't clear, then it won't pass the examination. If you're lucky, then the examiner will give you enough detail to fix it and another examination. If not, then you will fail.

Citations

Any statement that is made should be backed up by data, or logical argument, or citation. In a thesis, most statements will be backed up by citation. This is especially true of the literature review.

Citations should be formatted correctly. Citations that are in-text are usually done something like (Smith, 1999). When the citation is referred to directly, it is something like "as in Smith (1999)". This is also the form used when the citation leads the sentence, for example "Smith (1999) said that...". While most authors nowadays will be using reference-management software, you should know how to use the software and not rely on the defaults.

This might seem like a small thing, but every time a reader comes across an incorrectly-formatted citation, it can break the flow of their reading. Break the flow of reading enough and the reader gets frustrated. That's not what you want when the reader is an examiner with the power to make the last three (or more) years of your life irrelevant.

Typos

Typos are a fact of life. Everyone makes mistakes while writing, but there are some things you can do to reduce the number of mistakes that make it through to the examiner.

Firstly, use a spell-checker. These are so straightforward to use now that there is no excuse for any incorrectly spelt words to appear in a thesis that is going for examination. However, relying on a spell-checker is also dangerous. Spell-checkers only tell you if a word is spelt incorrectly, they won't tell you if they are the wrong word to use. So, proof-reading is still essential.

Grammatical errors should also be checked for. While English has its quirks, these quirks must be known and dealt with. Small errors in grammar can completely change the meaning of a sentence. A common error is using incorrect tenses. For example, experiments reported in the thesis have been done, they are in the past, so use the past tense to refer to them.

Tables and Figures

Tables and figures are one of the most effective ways of presenting data, provided they are used appropriately and carefully. There are some common mistakes that you must avoid in tables and figures.

Firstly, do not use unnecessary precision in a table. If the table is presenting the area of city blocks, then presenting areas to the square millimetre is excessively precise.

Secondly, every column at least should be labelled. There are exceptions, of course, but it is important to consider whether the table could be understood without the labels. The rows and tables should also be in a logical structure, with related values grouped together.

The caption of a table or figure should be stand-alone, and should explain what the table or figure is showing. For tables, that means that column labels need to be described or defined. That is, the reader should be able to interpret the table or figure without having to refer to the main text of the work. This is because a table or figure often ends up being displayed on a different page to the explanation of the table, and having to flip back-and-forth between pages while trying to understand presented data is annoying. This can lead to some long captions.

For plots of data, be careful with legends and labels. These should be informative, not just some default like "x" on the x-axis. Again, the goal is clarity, as the purpose of a plot is to communicate to the reader.

There is, in my opinion, almost no situation under which a 3D plot makes sense. The 3D bar-charts in MS Excel are particularly bad and should not be used under any circumstances. 3D plots serve no purpose other than to show that the author knows how to make them. They do not make data clearer, but they are harder to accurately interpret.

Do not use line plots for discrete data. For example, a school has three terms per year, and students may commence their studies at the start of any of the terms. If we were to plot the number of students who commenced in each term across a period of several years, we would use a scatter plot, because the quantity being plotted (number of students) is discrete. A line plot would imply that the number of students who commenced in a particular term is different halfway through the term than it is at the start of the term. Since we've already established that students commence at the start of the term, this is plainly incorrect.

If presenting several different series of values on the same plot, then distinguish between them by making the point markers clearly different shapes. Do not rely on colour for this! There are two reasons why: Firstly, a non-trivial portion of the population are sufficiently colour-deficient that they will not be able to perceive the difference, especially between red and green; Secondly, a thesis will likely be printed in greyscale, which completely hides the colours.

Check the cross-referencing to tables and figures. I once examined a thesis where all of the cross-referencing was incorrect - the cross-references in the text referred to figures and tables that did not exist - which made the results all but impossible to interpret. If you use a package like LaTeX to write your thesis, and carefully check the error messages when compiling your document, this is not an issue. For other writing software, like MS Word, you need to be a bit more careful.

Finally, do not use the word "plot" in a caption for a plot, or "table" in the caption for a table. I know what a table is, and I know what a plot is. I don't need to be told.

Equations

Used properly, equations are an effective way of communicating complex concepts. It is very easy for equations to become opaque and uninformative. To avoid this, equations must be laid out carefully and consistently. Again LaTeX is good for this kind of thing, its equation tools are very powerful.

Every variable in an equation should be defined somewhere, ideally following the first equation in which it is used. Similarly, variables should not be re-used. A table of symbols can be helpful.

Experiments

You must understand your data. What process created it? What are the variables? What do the variables mean? What are the ranges of the variables? What are the scales of the variables? Are they nominal, ordinal, interval? Remember, just because something is expressed as a number, doesn't mean you can do arithmetic with it. Some statistics are invalid for some kinds of data, so a working knowledge of measurement theory and statistics is essential.

Some data sets will have hidden biases. These biases will influence any model that is built using the data and must therefore be accounted for. Remember, if you are using biased data to build a model, you will end up with a biased model.

The data must be represented in a logical way. Some models like neural networks can only handle discrete values like class labels if they are represented orthogonally. 

When evaluating the accuracy of a classification model, you must give some thought to the distribution of classes in the data set. If 90 % of the data in the data set are from one class, then it is quite simple to create a model that is 90 % accurate: it just classifies every example as the most common class. A simple percentage accuracy is not, therefore, very useful for evaluating the performance of your model.

A single partitioning of data is not going to give an accurate estimate of performance of any model. The standard approach, therefore, is to cross-validate over the data set, with a separate, independent, validation set held out (note that some sources call this the test set - the name given doesn't matter, as long as you use such an independent data set). If the data set is too small to use cross-validation, then jackknife over the data set instead. Or, you can bootstrap the data. The point is, there are several different approaches that can be used to produce statistically reliable results. These approaches are so simple, and well-known, that I consider not using them to be sufficient reason to reject a thesis: the candidate plainly does not have sufficient skill in the field to qualify for a higher degree.

The set-up of experiments must be described in detail, including the parameters of any algorithms used. The goal is for all experiments to be reproducible. The description should also include reasons for selecting any particular algorithm. There is always a reason, and if a candidate can't justify their choice of algorithm, then I do wonder whether they understand the state of the art enough to qualify as a professional researcher. There is always a reason for selecting an algorithm, even if it is really "Because my supervisor told me to use it".

If the thesis is presenting a new or improved algorithm, then is must be compared to existing algorithms. The choice of algorithms compared to should be justified. It is very easy to find an algorithm that performs so badly that it makes a new algorithm look good by comparison. Be clear about why an algorithm was selected.

All results should be subjected to an appropriate statistical analysis. Statistics show us what the numbers are trying to say. Statistics allows us to separate reality from our own prejudices. A good working knowledge of statistics is, therefore, extremely important.

The thesis should interpret the results for the reader. In other words, the thesis should explicitly answer the question "What do the results mean?". This interpretation, of course, must be done within the context of the statistical analysis. The results should also be compared to the literature where possible. Don't leave this interpretation up to the examiner! The examiner might not interpret things the way that you intended.

Response

When the examiners' reports are received by the institution, they will be collated and made available to the candidate. The candidate always has a right of response. Don't be afraid to disagree with an examiner! Examiners are human, they make mistakes, or they might have missed something in the literature. If a candidate does disagree, however, then they should have a solid justification for disagreeing. The candidate will have to convince the examiner that they were mistaken, that means using facts or logical argument. Personally, I am quite prepared to be proven wrong on anything I write in an examiner's report. But I will only be swayed by a convincing, well-reasoned argument based on either logic or data. If a candidate tries to bullsh*t me, then I will not react well.

If a viva is to be held, then the examiners' reports will be made available to the candidate well before. A viva is a way of demonstrating that the candidate really does know what they are talking about in the thesis, and that they are able to handle questions on their own. It is also an opportunity for the examiners to clarify any lingering issues from the examination. I never try to make a candidate feel uncomfortable or upset in the viva, and I don't understand examiners who do that. It is not an opportunity for an examiner to show off how clever they are, or to exercise their limited power over another person. It is the last step of the examination, and it should be carried out in a professional and collegial manner.

Summary

A postgraduate degree represents a substantial investment of time and effort on the part of the candidate and their supervisor. It behooves all involved in that process to minimise the chances that the effort will be wasted. Putting in the effort to avoid the common issues I have identified above will help to achieve this.

Measurement Theory

Just because something is expressed as a number, doesn't mean you can do arithmetic with it. Let's say I give you three numbers: 1,2,3. What's the mean of these numbers? (1+2+3)/3=2, right? Well, what if I told you that 1=apple, 2=pear and 3=banana? What's the mean of an apple, a pear and a banana? Plainly, the question is ridiculous. Yet there is still a substantial number of people in science, and in computational intelligence, who fall into this trap, especially when presenting data to models like neural networks.

Neural networks aren't magic: they can't tell what a number submitted to their input layers mean, they just multiply them by their connection weights, sum the products and apply a transformation function to them. So if the numbers you are submitting to them represent classes, rather than measurements, what are they really modelling?

Measurement is the most fundamental part of data collection, as all natural data originates as measurements of properties of events. Measurements should represent reality and relationships between measurements should reflect the relationships between attributes. This is most important is consideration is given to the principle that data represents reality: as the source of data, measurements must yield and adequate representation of reality.

Measurement theory, as originated by Stevens, helps us achieve this. By specifying and formalising what exactly measurement is, we can better use measurement to gather data. By understanding exactly what the numbers mean, we can better analyse and transform the data into information and knowledge, while avoiding such traps as making meaningless statements about the numbers or performing a meaningless transformation on the data. A crucial point to bear to in mind is that measurements represent reality but are not the same as reality.

Measurement Scales

At the heart of Steven’s measurement theory is the concept of measurement scales. Four such scales are defined (although other have been added since) where each scale is distinguished according to four characteristics:

• Distinctiveness: individuals are assigned different values if the property being measure is different.
• Ordering in magnitude: larger numbers represent greater quantities of the property being measured;
• Equal intervals: a difference in measurement represents the same difference in the property.
• Absolute zero: a measurement of zero represents an absence of the property being measured.

These four characteristics define the “strength” of the measurement scale. The scales, from “weakest” to “strongest” are:

• Nominal
• Ordinal
• Interval
• Ratio
• Absolute

Also associated with each of the measurement scales are specific, permissible statistics and transformations. The term permissible is slightly misleading: if a statistic is not permissible for a certain scale, it is not forbidden. Rather, the results of that statistic or transformation are not reliable, with the unreliability of the result determined by the way in which the measurements were made. Permissible statistics and transformations are simply those statistics and transformations that yield reliable results. A permissible statistic tells us something meaningful about the data, while a permissible transformation maintains the properties of the data as appropriate for the particular scale. A statistic may still be applied to data from a scale, for which that statistic is impermissible, and it may yield useful results, but these results need to be treated with caution, and interpreted within the context of the original measurements. Note also that permissible statistics and transformations are cumulative across scales, that is, all statistics and transformation permissible for a lower scale are permissible for a higher scale.

Nominal Scale

The nominal scale is the weakest of the measurement scales. It possesses only the characteristic of distinctiveness. In other words, if the same attribute of two individuals are assigned the same number, then the attributes are identical. No other conclusions may be drawn from those numbers, as they are simply arbitrary numeric labels. For example, the colours Red, Green, and Blue can be placed on the nominal scale with the measurements Red=1, Green=2, Blue=3. However, two reds do not make a green. They could just as easily be labelled Green=1, Blue=2, Red=3, or any other permutation, without altering their distinctiveness. The only permissible statistics for nominal scale measurements are the number of cases and the mode. Permissible transformations are permutations and one-to-one substitutions.

Ordinal Scale

Measurements on the ordinal scale have the properties of distinctiveness and ordering in magnitude. In other words, objects are ordered in the scale according to some pair-wise comparison. That is, measurements on this scale can be compared to one another with the equality, greater than or less than operators. However, while we can say that one measurement is greater than or less than another, we cannot say how different they are. Numbers in this scale are categories; they do not have the arithmetic properties of numbers. An example of an ordinal scale measurement is teaching evaluations: a teacher’s performance is evaluated by students over several variables, with the performance being rated from one to five, with one being “Poor” and five being “Excellent”. While it is meaningful to draw the conclusion that a score of four is better than a score of two, it is not meaningful to draw the conclusion that a score of four is twice as good as a score of two, nor is it meaningful to say that a score of five is the same “distance” from a score of three, as a score of three is from one. Permissible statistics introduced at the ordinal scale are medians and percentiles. Permissible transformations introduced are monotonic increasing functions, that is, any transformation that will maintain the order of the individuals.

Interval Scale

Measurements on the interval scale have the characteristics of distinctiveness, ordering in magnitude and equal intervals. In this scale, objects are placed in order on a number line with an arbitrary zero point and an arbitrary interval between objects. While the numerical values have no significance other than as labels, differences between the values do have meaning. An example of an interval scale is the date in years. The common era (CE) scale has an arbitrary zero point (set at the putative time of the birth of Christ) and equally sized intervals (the length of a year does not vary, excepting leap years, which actually make up for errors caused by the slight mismatch between the arbitrary length of the year set at 365 days and the actual length of the Earth’s orbit).  It is meaningful to say that 1973 is later than 1928, and that the difference between 1999 and 1973 is twice the difference between 1986 and 1973. It is not meaningful, however, to say that 2004 is twice the year that 1002 was. Permissible statistics introduced at the interval scale are the mean, standard deviation, rank-order correlation and product-moment correlation. Permissible transformations introduced are linear transformations of the format y=ax+b, where x is the measurement, and the constant a cannot be zero. In other words, permissible transformations are those transformations that preserve the order of the objects, and the relative intervals between them.

Ratio Scale

Measurements on the ratio scale have the characteristics of distinctiveness, ordering in magnitude, equal intervals and absolute zero. In this scale, objects are placed in order on a number line with equally sized intervals and a true zero point. A measurement of zero on the ratio scale indicates the absence of the property being measured. A ratio scale can also be defined as the differences between two interval measures: a difference of zero between two interval measurements indicates an absence of difference. In the ratio scale, the values themselves have significance, as do the differences and ratios of those values. Many properties in physics are ratio scale measurements. An example of this is speed. An object with a speed of zero isn’t moving, that is, it has no speed, while an object moving at fifty metres per second is twice as fast as an object moving at twenty-five metres per second.

Permissible statistics introduced at the ratio scale are the coefficient of variation, and permissible transformations are affine transformations, that is, y=ax.

Absolute Scale

Whereas measurements on the ratio scale have an absolute zero point, measurements on the absolute scale have and absolute zero and an absolute upper bound. The classical example of this is probabilities: the probability of an event can range from zero (the event will never happen) to one (the event will always happen). A probability of less than zero or greater than one is meaningless.

Only affine transformations are permissible for measurements on the absolute scale.

Transforming Between Scales

It is possible to transform a measurement made on a particular measurement scale to a weaker scale only. This transformation will involve a loss of information, and cannot be reversed. In other words, it is not possible to transform to a higher measurement scale. For example, consider the heights, in metres, of a group of three people. One person is 1.4 metres tall, the second is 1.8 metres tall, and the third is 2 metres tall. If we say that a person’s height is 1 if they are short, 2 if they are average and 3 if they are tall, then it is possible to transform these ratio scale measurements into the ordinal scale, by assigning the first person’s height a value of 1, the second a value of 2 and the third a value of 3. However, if we know only that a persons height is 2 on this scale, we cannot determine exactly what their true height is.

Summary

The major implication of this is that data must be collected with great care. Once a measurement is made on a particular measurement scale, it cannot be transformed into a higher scale. Once the measurement is made, no further information can be associated with it.

You must know which scale the measurements belong to, as they will determine what you can meaningfully do with the data. They will also inform as to how you represent the data for presentation to your models. A working knowledge of measurement theory, therefore, is essential for any serious practitioner in computational intelligence.

Video: How to publish your research in IEEE CIS publications

A panel discussion on how to publish in the IEEE Computational Intelligence Society's publications. This discussion wraps up the individual talks by Professors Xin Yao, Kay Chen Tan, Derong Liu, Garrison Greenwood, Chin-Teng Lin and Simon M. Lucas . This panel session was part of the CEC 2013 conference.

How to publish your research: Video of Professor Simon M. Lucas

Professor Simon M. Lucas talks about how to publish research in IEEE Transactions on Computational Intelligence and Artificial Intelligence in Games. This talk was part of a panel session at CEC 2013 and mostly talks about what kind of papers are published in TCIAIG.

How to publish your research: Video of Professor Chin-Teng Lin

Professor Chin-Teng Lin, who is the editor-in-chief of IEEE Transactions on Fuzzy Systems, speaks about publishing in that journal. This talk was part of a panel discussion at the CEC 2013 conference. Some of the points he makes in this talk are applicable to publishing in most journals:

• Read existing papers, know the field
• Present an issue of significance
• Choose a journal that fits well with your research
• Use the correct format for that journal
• Have focus & vision, don't be too ambitious with your paper
• Write clearly
• Get pre-review, ask  your colleagues to check your paper before submission
• Proofread!
• Be patient, reviews take time

How to publish your research: Video of Professor Garrison Greenwood

Professor Garrison Greenwood talks about how to publish your research in the IEEE Transactions on Evolutionary Computation. This talk was part of a panel session at the CEC 2013 conference.

The main points of this talk are:

• Establish the context of your research in the Related Research section following the introduction (as a side-note, in other fields this is the introduction: why does computer science insist of separating them?

• Use enough citations, and no more

• Write enough detail for a competent researcher to replicate your work, don't try to write a tutorial

• Read the instructions for authors

• Use good grammar and spelling

How to publish your research: Video of Professor Derong Liu

Professor Derong Liu talks about how to publish your research in IEEE Transactions on Neural Networks and Learning Systems at a panel session at the CEC 2013 conference. This talk gives a lot of information about the editorial and review process, as well as how to increase your chances of having a paper accepted and even how to get on the editorial board of the journal.

How to publish your research: Video of Professor Kay Chen Tan

A video of a talk by Professor Kay Chen Tan as part of a panel session at the CEC 2013 conference. In this video he describes the IEEE CIS Computational Intelligence Magazine.

How to publish your research: Video of Professor Xin Yao

This is a video of a talk given by Professor Xin Yao as part of a panel session at the CEC 2013 conference. A couple of the most salient points that I noticed:

1) If you want to publish your research, you must first do good research
2) Contact the editors of the journals you want to publish in before submitting

Down with middlemen!

If there is one thing that the rise of e-commerce sites like eBay and Amazon.com has shown it is that the middleman is doomed. Entire bookshop chains like Borders have vanished from the face of the Earth, largely because they were unable to compete with a model that has no physical presence. Bookshops are middlemen: they connect one group of people (publishers) with another group of people (book purchasers, consumers). And the one thing that the Internet is really good at is getting rid of middlemen. Even publishers are middlemen, they don't produce the product, the authors do. The rise of self-publishing is the strongest indicator yet that the publishers are, like bookshops, an endangered species.

Online sales sites like e-bay (and Trademe, Gumtree) have also had an impact on retailers, and second hand dealers in particular: when we moved from Australia back to New Zealand, we had to sell our car and a few others bits and pieces. We didn't take the car to a second hand car dealer, or call a second hand furniture shop about our excess furniture: we just put some adverts up on Gumtree. While it is harder for online retailers to compete on items that require large volumes such as groceries, for smaller-volume or speciality items online sites are slowly but surely eliminating the traditional merchants. In the last ten years the only time I've booked air travel through a travel agent was for business travel, and then only because my employers had a policy of booking through certain agents.

This all raises a question: what is a middleman? If we define middlemen to be someone who does not produce, add value or provide a service that cannot be automated, then a huge number of current professions come under that heading: real estate agents, immigration agents, literary agent, property management agent... Basically, anyone with the word "agent" in their job title is a middleman and is doomed.

How does this relate to computational intelligence or academia? Well, what if journals and universities are really middlemen?

In the past I have blogged about how open-access journals are the future of academic publishing. But how much value do journals of any kind really add? A journal will arange peer-review, format the accepted articles and assign volume/page/DOI numbers. Apart from peer-review, each of these steps can be automated. In an age when every article published is available online, and are indexed by sites like Google Scholar and Citeseer, journals don't add much to the publicity of an article - in fact, the most effective way of publicising an article seems to be to blog or tweet about it. This is still the major advantage of open-access journals, as anyone with an interest can download and read the article (and hopefully cite it).

The measure of the quality of an article is the number of citations it receives, much more so than the supposed quality of the journal it is published in. Metrics like impact factor are so bogus as to be meaningless, despite the arrogant attitude of editors who deem submissions unworthy of publication in their august journal, without bothering to send them to peer-review. A good article will be cited more, no matter where it is published. Articles that aren't useful won't be cited. In other words, articles now can stand on their own, they don't need the support of journals to be useful. The journals, therefore, are middlemen, standing between the producers (the people who do the research and write it up) and consumers (the people who are reading and citing the research). Do we really need journals to arrange peer review? Or is there scope for a journal-agnostic, peer-review service for individual articles?

If individual articles can now stand on their own, how about individual academics? The Khan academy has been described as a revolution in teaching numerous times, and open courses like those offered by MIT have had thousands of students. In many ways universities are middlemen, providing access to resources (academic staff) to consumers (students). Universities provide tuition, consultation (students can ask their instructors for clarification), assessment (tests, assignments and exams), and accreditation (a degree / diploma from an institution has a certain credibility). Tuition can be supplied directly by the lecturer via sites like YouTube. Consultation can be done via discussion boards and live chat. Accreditation remains as an open problem. There are a huge number of accreditations available in a vast range of technical subjects: the IT industry in many ways leads the way in this, with certifications from Microsoft, Cisco, CompTIA and others. Professional organisations like the IEEE publish bodies of knowledge that graduates in certain disciplines are expected to know, and it's only a matter of time before this is expanded to include computational intelligence. Practical work is harder to deal with, but even then the large amount of open source software available means that anyone with a cobbled-together Linux box and a basic internet connection can not only do the lecture and practical work associated with undergrad study but also access the accreditation offered by numerous organisations.

The only problem for which I cannot see an obvious solution is, how would the lecturers get paid? Locking material behind paywalls won't work, people just won't use it. Also, a fixed fee won't work either: $500 might not seem like much for someone in the western world, but for someone in parts of Africa, it's more than they see in a year. The pay-what-you can model might work: this is where someone pays as much as they think something is worth, or as much as they can afford. A few people might take advantage and pay nothing when they could afford to pay, but most people are pretty honest and will pay a fair price. The accreditation agencies could also pay a referral fee to lecturers who direct students to their services, much like the Amazon affiliates program.

Universities would still survive, there still needs to be places where research is carried out, and training of the next generation of researchers (postgraduate students) takes place. The survival of journals is a bit less certain, as self-published peer-reviewed articles are much easier to do. Whatever happens, though, middlemen are on the way out.

On Being a Post-doc

After completing a PhD, most people who wish to stay in academia end up doing one or more post-doctoral positions. Experience as a post-doc is a prerequisite for a career in scientific research, as it is during your post-doc career that you get exposure to ideas and techniques outside of your PhD, and work with a wider range of people than you did during your PhD. The chances of going into a permanent academic position, without doing at least one post-doc, are very slim (most people who manage to do this tend to wind up in the same department they did their PhD in). I've done three post-docs, at Lincoln University in New Zealand, at the University of Sydney, and the University of Adelaide, both of which are in Australia.

So, what's it like being a post-doctoral fellow?

Basically, it sucks. Most post-docs are for two or three years. This term is fixed as the position is usually tied to a particular research grant, which is itself of fixed duration. This means that even if you do extremely well in your research, there is no guarantee of further employment after the contract ends. This means that as a post-doc, you will probably be changing jobs and cities every two years. If you're young and single, that's not entirely a bad thing: travelling and living in different places broadens your mind, can build a wide network of friendships and helps you appreciate different ways of life. Things get harder if you are a couple, as your partner also needs to find work in your new home. If you have even one child, it's a nightmare: you need to find a new school, your child faces the awful wrench of leaving their friends behind, if they're in after-school activities they need to be organised all over again, and if they have even minor health issues, finding adequate care for them can be very challenging. The stress that this can place on your relationship is enormous. In short, being a post-doc is a young (single) person's game.

If your post-doc is tied to a grant, then you will be working on someone else's project. In other words, you'll be working on something that is interesting to someone else (the grant holder). This also means that the outputs you produce (that is, papers) will be of benefit primarily to the grant holder rather than you.

While you should concentrate on doing the work you are paid to do, if you want to move up the academic ladder, then you also need to demonstrate the ability to do independent research. So, in addition to working a full-time job, you're also working part-time on your own research programme.

On top of the above are the dangers of any workplace: while most post-doc supervisors are good and kind people, they get their positions by being good researchers (or occasionally good politicians), not good managers. In the worst case, you might end up working for a narcissistic sociopath. Doing a post-doc with the wrong supervisor (or supervisors) can make your life a living hell. Sociopaths can be pretty hard to spot, too.

My experience is that it can take six months or more to find a new position, which means that shortly after starting a post-doc, you need to start looking for another. If your career is a chess game, then you need to start getting your pieces into place sooner rather than later.

To sum up, being a post-doctoral fellow means a semi-itinerant life of uncertainty and upheaval, serving the research needs of others, while also planning a future career that might not happen.

Was it all worth it for me? While there are many things I would do differently if I had the chance to do it all again, I don't want to live my life in regret: the things in my life, the good and the bad, the joy and the hurt, have all made me the person I am. But I do regret the hurt it has caused my family. Being a post-doc is hard on everyone if you have a family. It's not all bad news, though, and in a future post I'll be discussing ways in which you can make your post-doc career successful.

Guest post: Write Right First Time with Brown's Eight Questions

This is a guest post by Stephen G. Matthews. Stephen is a PhD student in the Centre for Computational Intelligence at De Montfort University, UK.

---

Write Right First Time with Brown's Eight Questions

I will share a method that I have found to be really useful. It's short, simple and incredibly effective: Brown's Eight Questions.

Robert Brown introduced Brown's Eight Questions (Brown, 1994/95) as part of an action learning set for improving writing. An action learning set is a group of people (ideally 5) who meet up to discuss common problems and solutions. Brown suggests applying this to writing for publication. An action learning set meets up and each member reviews each other's manuscripts face to face. I will focus on Brown's Eight Questions, but an action learning set for writing is well worth reading about in Brown's article.

So what is Brown's Eight Questions? Well, it is a set of eight questions designed to make an author think about writing before actually writing a first draft. Brown's idea, which was motivated by his experiences as a writer, reviewer and editor, comes from his observation that writers often focus on correcting a manuscript once it is written, rather than planning the manuscript before writing.

Brown’s Eight Questions

1. Who are the intended readers? - list 3 to 5 of them by name;
2. What did you do? (limit - 50 words)
3. Why did you do it? (limit - 50 words)
4. What happened? (limit - 50 words)
5. What do the results mean in theory? (limit - 50 words)
6. What do the results mean in practice? (limit - 50 words)
7. What is the key benefit for your readers? (limit - 25 words)
8. What remains unresolved? (no word limit)

As you can see these are simple questions and the word limit ensures answers are succinct. Brown's Eight Questions can help authors to gain clarity about their manuscript and also support the reader's understanding.

Brown's Eight Questions helps me to structure my thoughts, arguments and the message of a manuscript. It really is a useful method that can be applied to any form of writing such as journal articles, theses and reports. If you have not used it then give it a go!

Brown, Robert (1994/95) “Write Right First Time”, Literati Newsline Special Issue: 1-8. (Available from http://web.archive.org/web/19971014014626/http://www.mcb.co.uk/literati/write.htm)

The problem with academic journals 6

In my previous posts on academic journals (see here, here, here, here, here, and here) I've discussed the major problem with academic journals in the context of the huge cost of accessing the content that the journals receive for free, as well as the importance of open-access journals. This post is concerned with another problem that is becoming apparent with journals: the declining acceptance rate for papers submitted to journals, in attempts to foster an image of exclusivity and quality.

A recent editorial by David Wardle describes a quantitative analysis he performed that compared the acceptance rates of four top-ranked ecological journals with the large open-access journal PLoS One, along with the citation rate of papers published in each. What he found was that the four traditional journals accepted less than 20% of the paper submitted to them, while PLoS One accepted around 69%. However, papers that are published in PLoS One are cited more than papers published in one of the traditional journals. His argument was that the traditional journals rejected papers that were of good scientific quality (that is, they described good work) but were not "worthy" of publication in such "august" journals, with the editors using the excuse that limited page space meant that there wasn't room to print the papers, even though they were quite good. He then goes on to explain that this exclusivity was motivated by a desire to increase the perception of quality of the journals. That is, the editors are trying to foster the impression that the journals must be really good, because they're really picky about which papers they publish.

But, the ultimate measure of the quality of a paper is how often it is cited, as that reflects how useful it is to other scientists, and papers published in the less-exclusive open-access journals are cited more. Thus, the concept that journals with low acceptance rates publish better papers is fatally flawed: these journals are rejecting papers that are scientifically sound and are useful to other scientists.

This leads me to think that the only reason the top journals are the top journals are because people think they are. If someone wants an authoritative citation to back up a statement they make in a paper, they will cite a paper in Nature or Science if they can, because these are the top journals (this doesn't happen much in computational intelligence, because very few papers in this field are published in Nature or Science). But the conclusion of Wardle's study is that acceptance rate is not a reliable metric of the quality of a journal. If anything, it is a measure of the snobbery of a journal.

The purpose of peer review (and of reviewers) is as a crap-filter for papers, to keep work that is incorrectly done or poorly presented from entering the literature. But with exclusive journals, the peer reviewers seem to be spending more time deciding which papers are significant enough to be published in the journal, rather than trying to identify flaws in the work. The whole thing reminds me of the reason the great physicist Richard Feynman quit the US National Academy of Science: because they spent most of their time deciding who was "worthy" of joining the Academy.

Not so long ago, we had to consider the quality of journals because it wasn't feasible to track the impact of a single paper. Now, with tools like Google Scholar, we can track the citation histories of individual papers. In short, the journal in which a paper is published is no longer that important: the usefulness, the contribution of the paper is what is important. By the same token, the quality of an academic is not measured by which institution they work for, but by their contributions. Unfortunately, the bean-counters who make the hiring and promotion decisions, and who make decisions on who gets competitive research funding, haven't grasped this concept yet.

Exclusive journals do not make a good contribution to science, as they keep too much useful material out of the public eye for too long: peer-reviewed open-access journals, with their more liberal acceptance rates, are more important then ever in this situation.

Fraud in science

Ars Technica has a slightly tongue-in-cheek article on how to commit scientific fraud and get away with it. The article discusses eight points:

1. Fake data nobody ever expects to see
2. Work with many collaborators
3. Tell people what they already know
4. Don't do research anyone cares about
5. Don't publish in journals focused on your field
6. Distribute responsibility
7. Don't plagiarize
8. Don't duplicate images

It seems to me that, unfortunately, computational intelligence (CI) is more susceptible to many of these methods than many other fields. I sometimes joke that I am fortunate to work in a field where I can perform solid research by making stuff up as I go along, by which I mean that developing algorithms or techniques is often a more creative process than, for example, research in biology or physics. But think about how many CI papers you've seen that don't make the data available (point 1), or even describe its statistical parameters?

A long list of co-authors is not as common in CI (point 2) as it is in other fields, but I have seen many, many papers that are going over the same topic as has been covered many times before (point 3). Also, many, many papers cover minimal, slightly incremental "improvements" to existing algorithms that are of little true interest to most other researchers (point 4).

While one of the great joys of working in computational intelligence lies in the broad range of applications the field can be applied to, it does provide more opportunity to publish in journals that specialise is other fields (point 5).


The remaining three points (6-8) are more concerned with how not to get caught, or rather, how not to draw attention to yourself while committing fraud.

Fraud is always a problem, and I don't think that it is any less common in CI than in any other field. A greater emphasis on the use of statistics in CI papers would help guard against fraud (see my posts here and here about increasing the statistical basis of CI papers). But apart from that, we still depend on the honesty and integrity of the authors.

Publishing and perishing under gameable metrics 2

This article about the Australian Excellence in Research for Australia (ERA) initiative discusses how the process by which Australian universities and academic are assessed is flawed. It also discusses how Australian institutions have been gaming the metrics, like certain New Zealand institutions have been accused of doing.

In this previous post I described how any metric by which an institution or academic is assessed can be gamed. That is, any way in which an academic or institution is assessed can be manipulated by that institution to gain a higher score. In this post, I discussed how this has a negative effect on the teaching performance of an institution. By removing staff who do not perform well in research assessments due to a heavy teaching load, the institution can lift their research scores, but at the cost of lowering their teaching performance. As the article mentions, teaching is not assessed, so the process optimises towards a single metric at the expense of all others. This is not helpful for the long-term viability of an institution, as undergraduates will not want to attend an institution with a poor reputation for teaching.

This situation is almost certain to increase the use of contract lecturers, as contract lecturers are, as I understand it, exempt from assessment. I've already described why increasing contract lecturers is a bad idea, mostly because of a lack of job security and satisfaction for the contract lecturers, as well as a lack of continuity in teaching from the point of view of the students.

It is becoming increasingly apparent to me that assessing institutions is not as useful as assessing individuals, and that, in today's highly-mobile world, the reputation of an institution is no longer as important as the reputation of an individual researcher. This raises an interesting question:

What would happen if research performance based funding were given directly to the researchers based on their own individual performance, rather than their institutions being given extra funding based on the collective research performance of their staff?

The article linked at the start of this post does an excellent job of describing the problems with collective assessments (like what does it mean if you have one researcher ranked 1 and one ranked 5 - do they have a collective performance of 3? What does that even mean?).

Individual funding would remove a lot of the financial motivation for institutions to game the system, although it wouldn't eliminate it (institutions would still make money by charging the individual researchers over-heads, but these could be capped). Under the current Australian and New Zealand systems, individuals are assessed anyway, so it doesn't require any great changes to the current assessment process. One downside (and it could be a stonking big downside) is that early-career researchers would probably do poorly under this model. Early-career are already disadvantaged by management practices designed to game the system, and a simple weighting mechanism accounting for the length of time an individual has been doing research would go a long way to help. This would encourage researchers to start publishing early (which is essential to master the art of scientific publishing) and to publish consistently (which is essential to maintain your publishing skills). Another downside would be senior researchers taking credit for the work of junior researchers. But, again, this happens anyway, even though it is profoundly unethical. Under this system, though, it would no longer be just unethical, it would be criminal fraud.

Such a scheme could only be successful if it were paired with a scheme for assessing and rewarding teaching. While I have stated several times that an academic in a permanent position who is not publishing is not doing their job, an academic with a low (but not non-existent) research output and a strong teaching performance is an asset to an institution. Therefore, it is, in my opinion, imperative that an objective metric for teaching performance be implemented as soon as possible. That way, quality teachers, as well as quality researchers, would be recognised and rewarded. Those who do both (and this is the ideal for an academic, to teach and do research) would score even higher.

Publishing and perishing under gameable metrics

My alma mater is in the New Zealand news again, and again it is to do with gaming the metrics by which the research performance of New Zealand tertiary institutions are measured. This time, the article describes how many staff with poor publishing records have been made redundant from the university (that is, they have lost their jobs) prior to the assessment later this year. While I have little sympathy for those in permanent lecturing positions who do not publish (see my previous comments here and here) in this case it seems like the staff who have lost their jobs are predominantly teaching staff, or staff who are still developing their research record (see this post from one who lost her job for the same reason some time ago). If that is the case, then I have to say that the university administration is making a mistake.

Teaching takes a lot of time and energy (my last semester teaching at Otago, I was in the office at least six days a week, and often worked from 7:30 in the morning to 9 or 10 at night). The purpose of having teaching-only staff is to take some of that load off of the lecturers so that they can do their research. Indeed, the major thrust of the article is that the redundancies are putting more stress on the remaining staff, as they are having to pick up extra teaching in addition to lifting their own research outputs. While the teaching load could in theory be reduced by hiring contract lecturers (who would not, as I understand it, be assessed) I have already posted on why this is a bad idea.

From my research with evolutionary algorithms, I know that optimising to one criteria or metric seldom results in optimal or robust systems. By optimising their staff to one (flawed and gameable) metric, the University of Otago is reducing the robustness of their institution. The long-term outcome of these redundancies is yet to be seen, but I do not think that it will be good for anyone concerned. Non-performers need to be removed, for sure, but early-career researchers need coaching and leadership to develop. They don't need the great big stick stick of the threat of redundancy waved at them (such threats are more often than not a sign of dysfunctional management, rather than a sign of competent leadership).

Ultimately, only those who set the metrics can resolve this situation. As long as a metric can be gamed, then institutions will game them. In the meantime, people will have their lives upended and their careers destroyed by narrow-minded administrators and cynical political operators who are trying to wring a few more points out of the system to make themselves look good.

The problem with academic journals: An update

 A brief update on the status of the Elsevier boycott (described here): to date, more than 11 000 academics have pledged to not review, submit or do editorial work for any Elsevier journals. My previous post has already described why I oppose such a boycott of a single publisher, and I expect that this boycott is going to cause some unanticipated consequences.

I suspect that this boycott explains why the papers I have under review in Ecological Modelling and Ecological Informatics are taking so long to go through the review process: it's hard enough finding reviewers as it is, and with people refusing to review for Elsevier, it's going to get even harder. That's not punishing Elsevier, that's punishing the researchers who are trying to get their work published and advance their careers.

As I said before, the way real change will come about is by the top researchers supporting open-access journals. At least one of the people who could do this has just done so: Winston Hide, an associate editor at the highly-ranked Elsevier journal Genomics has just resigned from the editorial board, with the avowed intention of focusing his energies on open-access alternatives. I can only hope that some of the top researchers in computational intelligence will do the same.