Karen Robertson1, Robert D. McIntosh2, Cerys Bradley-Scott2, Sarah MacFarlane2 and Jonathan L. Rees1

1Department of Dermatology, and 2Human Cognitive Neuroscience, Psychology, University of Edinburgh, Edinburgh, UK

Robust experimental evidence supporting many attempts to facilitate early melanoma diagnosis is lacking. In an experimental study using a browser interface we have examined diagnostic accuracy, sensitivity and specificity of novices in distinguishing between melanomas and mimics of melanoma. We show that rule-based ABC methods and image training, based on random images of melanoma, improve specificity to similar degrees, with­out effects on sensitivity, leading to small improvements in overall accuracy. There was a significant effect of age with older subjects performing better. Although both the ABC method and image training groups showed improved performance over the control group, overall performance was poor. For instance, for a task in which 1 in 4 test images was a melanoma, and 3 out of 4 benign, both interventions (ABC or image training) increased accuracy from the control value of 53% to around 61%. For reference, dermatology trainees performed at a much higher level of accuracy. Our study provides little support for the use of such methods in public education, but suggests ways in which performance might be improved. Key words: melanoma; ABCD; skin cancer; diagnosis; image training.

Accepted Aug 13, 2012; Epub ahead of print Nov 8, 2013

Acta Derm Venereol

Jonathan Rees, Department of Dermatology, Rm 4.018. Lauriston Building, Lauriston Place, Edinburgh, EH3 9HA, United kingdom. E-mail: reestheskin@me.com

If melanomas are diagnosed early the vast majority can be cured with simple excision. If the tumour is more advanced, metastatic disease is likely to develop, and cure is far less likely (1, 2). Although a significant fraction of melanomas are picked up incidentally in patients seeing medical practitioners for other reasons (3), the majority are brought to attention by the patients themselves (4, 5). A long-standing research question is to what extent this process of early self-detection of melanoma can be improved. On the one hand, we want patients to include any lesions that might be melanoma (sensitivity), but we also require them to exclude the majority of benign lesions (specificity), to avoid overburdening health services with spurious presentations.

The ABCD acronym (A for asymmetry, B for border irregularity, C for colour variation and D for diameter) was devised in the mid 1980s to enhance detection of early melanomas (6). Originally intended as an aid to physicians, the ABCD system was subsequently publicised to the lay public in the form of a simple mnemonic, ‘as easy as ABC’ (7). Today it remains at the heart of most general public education strategies and is included on the websites of the British Association of Dermatologists (8), The American Academy of Dermatology (9) and The Australasian College of Dermatologists (10).

Concern that the ABCD criteria may have discriminated against detection of some melanomas led many groups to suggest modifications to the ABCD acronym. These include such things as the removal of ‘D’ (diameter > 6 mm) (11–13) or changing the meaning to ‘D’ for darkness (a feature noted as seeming suspicious by many patients (11)), the addition of ‘E’ for Evolving (14–17) or Enlargement (14, 15), and even the addition of ‘F’ (family history) and ‘G’ (great numbers) (17).

There are hidden assumptions in the use of all such rule-based strategies particularly when, as with ABCD, there is an attempt to extend their use from experts to novices. As Rigel et al. (2) originally noted when they asked experienced physicians why they thought a lesion was a melanoma, they invariably answered “Because it looked like one.” The evidence suggests that it is likely that experts make their diagnosis holistically, using a process of pattern recognition built up primarily from exposure to prior examples rather than through extraction of key features which can be taught in written or oral form (18–22). Expertise in clinical diagnosis, under this view, results from acquiring a vast repertoire of prior cases against which new cases may be compared (20–23). This process, referred to as non-analytic reasoning (NAR), has been widely studied in relation to disease diagnosis (18–22).

Experts may be able to demonstrate their expertise, but this does not necessarily mean that they can encapsulate this expertise in an explicit format that is suitable for transfer to novices. This is simply because novices do not possess the same richness of experience, which experts are often unaware they themselves possess, when they attempt to formulate rules. Nor, if an authenticated expert reports that they use a rule-based approach, such as the ABCD, does it mean that their expertise actually results from using this rule-based method. It is equally possible that their belief in the ABCD method is confounded with some other aspect of their expertise (such as experience). Virtually all studies of the use of ABCD for physicians have failed to control for this bias, assuming that self-reporting or introspection is a valid method of elucidating psychological mechanisms. As Norman (24) has argued: “strong diagnosticians can tell a credible story about how they might have been thinking, but no one, themselves included, can really be sure that it is an accurate depiction”. Ironically, trying to study the effects of ABCD on diagnostic performance may be easier with novices than experts. This is simply because experimental interventions are not confounded by significant prior opinion or experience.

There are however good reasons to be skeptical of the success of the ABCD methods when used by novices. Experimental studies show that differences in the individual rules that make up the ABCD – when judged by novices – differ little between melanomas and mimics of melanoma (25). This may reflect the fact that expert judgements about these individual factors may not be shared by, or interpretable by, novices; or that the individual factors themselves are not useful for a significant number of melanomas. Furthermore, most studies of ABCD have been confounded by inadvertent image training – that is, while demonstrating ABCD, melanoma images are also shown (discussed in more detail by Aldridge et al. (25)). The result is that it is unclear whether it is the ABCD algorithm itself, or the image training that accompanies it, that accounts for changes in performance.

An alternative approach to novice training is to focus directly on providing examples of melanoma, and benign lesions, and see if people’s innate ability to characterise morphological form is clinically useful. We can find only one randomised study in which ABCD methods were compared with attempts to impart some ‘virtual clinical experience’ to novices via exposure to images of skin lesions (26). In this study Girardi et al. (26) reported that a very short cognitive education with photographs was more effective in improving the ability of novices (n = 255) to recognise a melanoma among benign pigmented lesions than written ABCD information. Image training increased the accuracy of discrimination of melanoma (due to a significant increase in specificity), whereas the ABCD algorithm decreased specificity with­out any substantial increase of sensitivity. In another study (27), the use of images was also shown to improve the appropriate assessment of concern for seborrhoeic keratoses, and to heighten sensitivity to melanoma while reducing sensitivity to benign lesions.

The study by Giraldi et al. was based on prior education and challenge after a one-week interval. One limitation of such designs is that we would expect learning effects to drop off over time, and it is debatable how practical it is to have repeated training sessions. An alternative approach is to consider whether presentation of referent images at the time a patient is concerned about a lesion might be of more use. For instance, Aldridge et al (28) showed that diagnostic performance for non-melanoma skin cancer could be improved by providing referent images to guide novice behaviour, without further explanation. It is important to note however that these two approaches are not identical: prospective education may raise awareness, whereas providing ‘just in time’ material, when a patient is already concerned about a lesion, would not be expected to change awareness to the same extent.

In the present study we set out to compare image training using referent images for melanomas and mimics of melanomas, with the ABC criteria of the ABCD system (D for diameter was omitted), and a control group. Because the world wide web (www) is increasingly the way most people search for health advice, and can be used to deliver image examples at low cost, our experiment took place within a web browser framework.

An obvious issue is how to select individual referent images. We chose to make minimal assumptions about typicality, and therefore chose referent images randomly from a bespoke library. We discuss this particular aspect of our work later in the discussion section.

METHODS

Subjects

Seventy-two laypeople (43 female) were recruited from friends and family of University of Edinburgh staff, relatives of patients attending the Dermatology clinic, and Department of Psychology undergraduate students at the University of Edinburgh. Ages ranged from 18–74 years (mean 34.76, SD 15.86). Individuals with a history of skin cancer were excluded. To study the effect of experience with melanoma recognition, 6 dermatology registrars (aged 26–31 years) with a mean of 2 years experience of dermatology also performed a version of the task.

Ethics

NHS Lothian Research Ethics Committee granted permission for the use of the images and recruitment of persons for the purpose of this study. All participants gave written informed consent, and data was anonymised.

Materials

Five hundred and twenty-five digital images of skin lesions (75 biopsy-proven melanomas, 225 naevi and 225 seborrhoeic keratoses (‘seb K’)) were selected randomly from the University of Edinburgh Department of Dermatology’s image database. This image database comprises over 5,000 images, collected prospectively using the same photographic set-up: Canon EOS 350D 8. IMP cameras, Sigma 70 mm f2.8 macro lens and Sigma EM-140 DG Ring Flash at a fixed distance of 50 cm. Each lesion was cropped from the original digital image to an image of 300 × 300 pixels with the lesion positioned centrally. The library is a research resource (rather than for clinical care) and, as far as possible, is based on images being collected independently of typicality. For instance, the melanomas are based on consecutive patients who agree to this research photograph being taken, rather than images being solely captured for routine care, clinical ‘interest’ or ‘oddness’. Lesions within each class (melanoma, naevi and seb K) were given a unique identification number.

Six ‘experimental sets’ of lesions were created. The reasons for the creation of 6 sets were 2-fold. First, we wanted a large proportion of the available lesions to be used. This may increase real-world validity, and minimises any assumptions about the typicality of particular lesions (because we randomly selected images). Second, we needed to ensure that each set received an adequate number of subjects, and that groups of images could be crossed with other factors in the design in a statistically efficient way (that is, that any particular unique set of images was tested across all the experimental factors such as training method). For each set, 90 images were randomly selected from the total image pool in a stratified manner with respect to class. Each of the 6 sets therefore comprised 42 ‘training’ lesions (21 melanomas and 21 benign), and 48 ‘test’ lesions (12 melanomas and 36 benign). To enable us to study the effect of type of benign lesion class, two versions were made of each experimental set: one with the benign lesions drawn wholly from the naevi class, and one with benign lesions drawn wholly from the seb K class (melanomas remained the same in both cases).

The study materials were presented as a pseudo-website, created using Adobe Dreamweaver, and browsed on an Apple Macintosh computer with 27” monitor, with the intention to mimic a user accessing a www site.

The number of images in each image training set (21) was chosen arbitrarily, since the exact number of prior examples that are required to elicit pattern recognition is unknown.

Procedure

Subjects were assigned to one of 36 different conditions, crossing the between-subjects factors of: experimental sets of images (1–6); benign class (naevi, seb K); and training me­thod (Control, ABC, or Image). The 6 dermatology registrars additionally tested were all administered the control condition with the naevus comparison class only, with each registrar completing a different one of the 6 experimental sets (see Fig. S11).

Subjects first watched a 3-min video which consisted of a brief overview of skin cancer followed by instructions on how to complete the study. The study proceeded with each test lesion (48 in total comprising 12 melanomas and 36 benign lesions) presented centrally on the screen at a size of 8 × 8 cm, and the subject required to click one of two radio buttons to indicate if they thought the test lesion was or was not a melanoma (see Fig. S22). A “Next” button took them to the next test lesion; progression was prevented until a selection had been made. The video instructions and test page layout differed between the 3 training conditions: Control, ABC, and Image (Fig. S21).

Control condition: The video explained that subjects must draw on their own knowledge of melanoma. The test pages showed the test lesion alone, along with the response options.

ABC condition: The video provided an explanation of the ABC criteria. The test pages included a left panel containing a description of the ABC criteria.

Image condition: The video provided an explanation of image training. The test pages included a scrollable left panel containing the training set of 21 melanomas, and a scrollable right panel containing the training set of 21 benign lesions (either naevi or seb K). Each image in the training sets measured 4 × 4 cm on screen.

1http://www.medicaljournals.se/acta/content/?doi=10.2340/00015555-1733

RESULTS

Lay experiment

The mean ages of the 6 groups ranged from 32.8–36.3, and the proportion of females per group ranged from 0.33–0.75. Potentially confounding influences of these demographic characteristics were controlled for statistically in the main analyses reported below. In our design, participants were given no explicit guidance regarding the true prevalence of melanoma images amongst the test lesions (one-in-four). Analysis of overall positive responding rates showed that all 3 lay training groups (Control, ABC, Image) made a ‘melanoma’ response on around half of the trials (0.51, 0.49 and 0.49, respectively). This suggests that participants inferred, implicitly or otherwise, a one-in-two prevalence, distributing their responses evenly between the two options (“melanoma”, “not melanoma”). This discrepancy between positive responding rates and true prevalence suggests that the task was a difficult one, and that accuracy is unlikely to be very high in any lay group. This inference is borne out by the accuracy data.

Diagnostic accuracy was measured as the total proportion of correct diagnoses across all 48 test lesions. For example, if an individual categorised 10 of the 12 melanomas as a melanoma, and 26 of the 36 benign lesions as benign, then accuracy would be (10 + 26)/48 = 0.75. Note that this measure of accuracy relates specifically to these study conditions, in which the prevalence of melanoma was one-in-four.

Fig. 1 shows mean accuracy per group. A between-groups ANOVA with training condition (Control, ABC, Image) and benign comparison class (naevi, seb K) as fixed factors, experimental set (1–6) as a random factor, and age and sex as covariates, found a significant main effect of training condition [F (2,10.5) = 6.15, p < 0.05, ηp2 = 0.54]. Post-hoc t-tests with Bonferroni correction confirmed that the Image training group performed better than the Control group (p < 0.05), whilst the advantage for ABC training over Control narrowly missed the corrected threshold for significance (p = 0.07). Nonetheless, the clear lack of difference between ABC and Image training (p = 1.0) implies that either form of training improves upon naive Control performance. One-way t-tests performed within each condition (collapsed across benign comparison class) confirmed that both the ABC and Image training groups performed significantly better than the chance level of 0.5 [respectively, t (23) = 5.36, p < 0.0005; t (23) = 4.80, p < 0.0005], whilst the Control group did not [t (23) = 1.56, p = 0.13]. Finally, the main ANOVA showed a significant influence of the covariate age [F (1,34) = 6.52, p < 0.05, ηp2 = 0.16], reflecting a tendency for older participants to perform more accurately.

Fig. 1. Mean accuracy for each lay participant group. Grey bars represent benign comparison class naevi, and white bars seborrhoeic keratoses.

Diagnostic performance was further broken down in terms of sensitivity (proportion of melanomas correctly identified) and specificity (proportion of benign lesions correctly classed as not melanoma), with condition means and standard deviations shown in Table I (see also Fig. 2).

A between-groups ANOVA of sensitivity, with training condition (Control, ABC, Image) and benign comparison class (naevi, seb K) as fixed factors, experimental set (1–6) as a random factor, and age and sex as covariates, found a significant main effect of benign comparison class, reflecting overall lower sensitivity for melanomas when intermingled with seb K lesions [F (1,6.49) = 13.63, p < 0.01, ηp2 = 0.68], but the effect of training condition was not significant [F (2,10.35) = 2.02, p = 0.18, ηp2 = 0.28], and nor was the interaction of these factors [F (2,9.75) = 2.26, p = 0.16, ηp2 = 0.32]. A trend for better performance in older participants was present, though the influence of the covariate age did not reach formal significance [F (1,34) = 3.15, p = 0.08, ηp2 = 0.09].

For specificity, ANOVA found only a significant main effect of training condition [F (2,10.53) = 4.51, p < 0.05, ηp2 = 0.46]. Post-hoc comparisons confirmed higher specificity in the Image training group than the Control group (p = 0.05), but no significant difference between ABC and Control or Image and ABC (p = 0.26, p = 1.0, respectively). However, the general numerical pattern tracks the result for overall accuracy, seen in Fig. 1. Finally, age again showed a significant influence as a covariate [F (1,34) = 4.37, p < 0.05, ηp2 = 0.11].

Dermatology registrars

In addition to the lay experiment, 6 dermatology registrars were tested in the Control condition with naevi as the benign comparison class. Their diagnostic performance (R1–6) is illustrated in Fig. 2, alongside the mean specificity and sensitivity of each of the lay groups. Four of the 6 registrars achieved sensitivities higher than any of the lay means, and 4 achieved higher specificities, with one registrar (R2) approaching ideal performance (sensitivity 1.0, specificity 0.89), the only ‘errors’ being diagnostically conservative false alarms. (Note, R3 performed below the mean accuracy of naive controls.)

Case-control statistics were used to test formally whether individual registrars performed significantly beyond the trained layperson range (29). Accuracy was used as the dependent measure, and 6 separate tests were run to compare each dermatologist trainee’s individual performance against the distribution of accuracy scores for the 48 lay participants who received ABC or Image training, with age, sex, and benign comparison class entered as covariates. The hypothesis tests were one-sided, because we expect dermatologist trainees to outperform laypeople. No alpha adjustments were applied for multiple comparisons, because case-control comparisons are inherently low in power (30). According to these criteria, registrars R1, R2 and R5 (upper-right corner of Fig. 2) performed significantly beyond the trained layperson range (respectively: one-tailed p < 0.05, one-tailed p < 0.005, one-tailed p < 0.05). Z-scores (i.e. number of standard deviations from the control mean) were Z = 2.16, Z = 3.88, and Z = 2.14, respectively.

Fig. 2. Sensitivity and specificity for the 6 Registrars individually, with means of each lay participant group plotted for reference.

DISCUSSION

Naive untrained people performed no better than chance in discriminating melanomas from benign lesions in this browser-based task. Short-term diagnostic training by exposure to ABC rules or randomly selected reference lesions boosted performance to above chance levels, but not dramatically so (accuracy 0.61 and 0.62, respectively). These slight increases in accuracy reflected a modestly increased specificity, without any significant change in sensitivity. The poor performance of novices, even with instruction, is not entirely unexpected. Most dermatologists will point out that diagnosing pigmented lesions is a very hard perceptual task, one that even experts perform imperfectly. Nonetheless, the fact that 3 of 6 trainee dermatologists with a mean of two years training significantly outperformed trained controls, with one trainee approaching ideal performance, at least shows that the problem is a tractable one, and validates the presentation of images within a browser-based task as having some meaningful relation to accepted standards of expertise.

Our finding that there is an effect of age with older persons performing better is perhaps not too surprising. The prevalence of abnormal skin lesions rises with age, and older persons may use their own self-observations to guide their performance on the experimental task. It is also unsurprising that the sensitivity for melanomas drops when seen alongside seb K. The ABC system was specifically designed to aid diagnosis between melanocytic naevi and melanoma (and not between melanoma and seb K). There is little reason however to believe that novices are able to decide on the correct class of lesion – the result is that patients misapply the criteria, because like many non-expert physicians, they cannot reliably distinguish melanocytic naevi from seborrhoeic keratoses.

Our findings need to be interpreted in the light of some of the limitations of the approach we have pursued. Our approach was an experimental one, carried out on volunteers. Our study did not use individuals who themselves subsequently presented to a clinician with a lesion that they were worried about. The difficulties in such a ‘real world’ design are obvious: it would probably require different geographical populations to be randomised to different sorts of health advice. Any such large-scale design would likely be unable to exert tight experimental control over exactly what advice individuals received. It would be impractical, for instance, to remove or ban access to currently available strategies including www access and conventional media health campaigns. By contrast, our approach allows tighter experimental control, and the use of a modality of accessing health information that is likely to be the dominant one.

Despite the ABC criteria being principally rule-based, it should be noted that our ABC rules did include 3 images of melanomas. We felt this important to enable understanding of the rule and to mimic how ABC is typically presented in the real world, but it may have acted to diminish any difference between the ABC group and the pure image training group (that is, it may have resulted in an underestimation of the benefit of the image training). Nor did we include D for diameter in our study. In part this relates to not presenting images at life size in our study; rather the lesions were scaled so that they each occupied a similar proportion of the image as a whole. Although this came at the expense of portraying the relative size difference between lesions, we felt that it was more important that each lesion was seen with a similar level of detail (which is difficult to discern in very small lesions).

It is important to understand the limitations of individual summary measures of performance. We have used widely accepted measures of accuracy, sensitivity and specificity. In our experiments we used a ratio of 1:3 of melanoma:benign lesions. Any summary statistics used are influenced by the ratio chosen. For instance, if we had (say) only used a ratio of 1 melanoma to 30 benign lesions then the figures for accuracy, sensitivity and specificity may have been different. This is because participants’ expectations of the (hidden) ratio may influence their behaviour. Of course in the real world the chance of a lesion being a melanoma is very low, but modelling this directly in an experiment would be practically impossible because such a design would require a ratio of considerably less than 1:10,000 (if for example we use the number of lesions that are melanomas in a defined population as the numerator and the number of benign lesions present in the same population as the denominator). By contrast our experimental design may allow comparisons between different educational strategies, as the ratio is the same for all arms of the experiments, though it is not impossible that the base rate of melanomas might interact with a particular training method.

Despite these limitations, we believe the current study, alongside the prior work of others (26, 27) and ourselves (25, 28) is important. Whereas the ABC method has been shown to alter diagnostic behaviour (‘D’ was not tested in our work), its effects are modest and it is not superior to image training. The changes in performance of the ABCD we doubt are clinically meaningful given the low base rates of real melanomas in the general population. Our work and that of others suggests the use of ABCD as a specific education tool is probably not warranted.

Is it perhaps possible to be more sanguine about image training, since this also performed at the level of the ABCD system, despite being based on randomly-selected lesions (rather than a designed or evolved training set). A crucial question here relates to exactly which images are chosen for training. Melanomas are remarkably heterogeneous, and experts acquire experience of lots of different subtypes: red melanomas, nodular melanomas, lentigo maligna melanomas etc. If we are going to guide novices, we need to either include training examples of all the subtypes or hope that there are certain exemplars that cover most lesions that present. Our approach in the present study was to make minimal assumptions, and to include as training examples, melanomas that were randomly selected from a library that had been collected based on consecutive patients who had presented to us and consented to having their image collected. We think there are two implications of this for future work. First, images used in studies need to be made available to other researchers, as results may be influenced by exactly which images are used. Second, it seems possible that some images may confer more useful information than others. If this latter belief is correct, it is possible that diagnostic performance based on image training may be improved. The experimental system we describe lends itself to scaling (via the web), and offers the possibility of testing different sets of melanomas on performance.

ACKNOWLEDGEMENTS

Funded by Cancer Research UK via a Project Grant C1375/A12060 to JLR and RDM.

The authors declare no conflict of interest.

REFERENCES