Sjors A. Koppes1,3, Frank Charles1, Laureen A. Lammers2, Monique Frings-Dresen1, Sanja Kezic1,#, and Thomas Rustemeyer3,#
1Coronel Institute of Occupational Health, Academic Medical Center, 2Hospital Pharmacy, Academic Medical Center, University of Amsterdam, 3Department of Dermatology-Allergology , VU University Medical Center, Amsterdam, The Netherlands
#These authors contributed equally to this work.
The aim of this randomized controlled trial was to assess the efficacy of a cream containing ceramides and magnesium (Cer-Mg) in the treatment of mild to moderate atopic dermatitis and to compare it with hydrocortisone and a commonly used emollient (unguentum leniens; cold cream). A total of 100 patients, randomized into 2 groups, were treated for 6 weeks simultaneously (left vs. right side of the body) with either Cer-Mg and hydrocortisone (group I) or Cer-Mg and emollient (group II). The primary outcome was a reduction in severity of lesions as assessed by (local) SCORAD (SCORing Atopic Dermatitis). Levels of trans-epidermal water loss (TEWL), skin hydration, and natural moisturizing factors (NMF) were then measured. After 6 weeks, group I showed comparable significant improvement in SCORAD and TEWL, while in group II, the decrease in SCORAD and TEWL was significantly greater after Cer-Mg compared with emollient. Finally, Cer-Mg cream was more effective in improving skin hydration and maintenance of levels of NMF than hydrocortisone and emollient. Key words: atopic dermatitis; skin barrier; ceramides; magnesium; RCT; Dermalex.
Accepted Mar 3, 2016; Epub ahead of print Mar 4, 2016
Acta Derm Venereol 2016; 96: XX–XX.
Sanja Kezic, Coronel Institute of Occupational Health, Academic Medical Center, University of Amsterdam, Meibergdreef 9, NL-1105AZ Amsterdam, The Netherlands. E-mail: S.kezic@amc.uva.nl
Atopic dermatitis (AD), a chronic, inflammatory skin disease characterized by dry, pruritic and erythematous skin, affects up to 10% of adults and up to 20% of children in the Western world (1–3). Patients with mild to moderate AD are constrained for long periods to over-the-counter (OTC) emollients or, in some countries, such as the UK and the USA, to low-potency corticosteroids. However, long-term use of corticosteroids is associated with adverse side-effects, such as skin atrophy (4). Such side-effects are well known among the general public and (not always justifiable) anxiety about corticosteroids is a major factor in poor adherence to therapy (5–8). Therefore, emollient therapy is often preferred by patients and is shown to reduce corticosteroid use significantly (9). In general, emollients aim to prevent water loss from the skin, e.g. by occlusion (petrolatum) or by addition of hygroscopic compounds (e.g. glycerol and urea) and lipids (e.g. ceramides). Identification of an inherited deficiency of the epidermal protein filaggrin as a major risk factor for AD, points to the importance of the skin barrier in the aetiology of AD (10–12). The barrier is located mainly in the stratum corneum (SC), which is composed of corneocytes surrounded by lipid lamellae composed of ceramides, cholesterol and free fatty acids (13–15). Although emollients are regarded as basic therapy by the European Task Force on Atopic Dermatitis/European Academy of Dermatology and Venereology (EADV) Eczema Task Force, their efficacy in randomized controlled trials (RCT) has been insufficiently investigated (16–20). Therefore, the aim of the present double-blinded RCT was to assess the efficacy of an emollient containing ceramides and magnesium (Cer-Mg), compounds involved in the maintenance of the skin barrier (21). SC ceramide composition is altered in AD, and reduced levels of ceramides and changes in their relative composition have been shown to correlate with trans-epidermal water loss (TEWL) (12). The role of magnesium in AD is relatively unknown; however, bathing in magnesium-rich water has been shown to have a beneficial effect on the skin barrier in dry atopic skin (22). Furthermore, magnesium is known to be involved in synthesis of ceramides, regulation of epidermal proliferation and differentiation. In addition, children with AD showed a reduced level of serum magnesium (23, 24). Although there is some evidence that both ceramides and magnesium might improve barrier function in AD, their efficacy remains to be elucidated, preferably in RCTs. In the present study the efficacy of the Cer-Mg cream was compared side-by-side with 2 other creams, which are frequently used in treatment of mild and moderate AD: a low-potency topical corticosteroid (hydrocortisone acetate 1% in petrolatum-cetomacrogol) and a commonly used OTC emollient, unguentum leniens; cold cream).
MATERIALS AND METHODS
Trial population
A total of 100 patients were recruited from the outpatient clinic at VU University Medical Center Amsterdam (VUmc). Inclusion criteria were: (i) clinically diagnosed AD conforming to the Hanifin & Raijka criteria (25), (ii) mild to moderate AD, (iii) age 18–70 years, (iv) at least 2 symmetrical (i.e. left and right side of the body) skin sites with comparable AD severity. The exclusion criteria were: (i) extensive ultraviolet (UV) exposure in the last 14 days and/or expected exposure during the study, (ii) skin disease other than AD, (iii) use of antibiotics prior (at least 4 weeks) to the study and/or expected use during the study, (iv) use of systemic immuno-suppressing drugs prior (at least 4 weeks) to the study and/or expected use during the study, (v) severe disorders within the last 6 months, (vi) investigator’s uncertainty about the willingness or ability of the patient to comply with the protocol requirements (e.g. mental disability). In the case of adverse health effects, such as allergic reaction or severe deterioration of the symptoms, patients were prevented from further participation. Patients could not use any AD medication for at least 2 weeks prior to participation (wash-out period). The study was approved by the medical ethics committee of the Academic Medical Centre and VUmc. All patients gave their written informed consent prior to participation.
Patients’ experience
After participation patients were asked, in a short questionnaire, what their personal preferred treatment was.
Registration and medical ethics approval
The trial was registered under the number NTR 4541. Medical ethics approval was obtained on the basis of the study protocol (AMC registration number: METC 2014_090).
Randomization and blinding
The randomization list was produced prior to treatment by a random number sequence generated in Microsoft Excel™. Treatment combinations (Cer-Mg and HC or Cer-Mg and EM) were linked to a unique inclusion number. The allocation list was prepared by an investigator (SK) with no executive tasks in the trial and handed over to the VUmc pharmacy. After the enrolment of a second investigator (SAK) who had access only to the inclusion numbers, each patient was given the inclusion number and collected the creams at the pharmacy. Creams were packed in identical tubes labelled only with the 3 possible treatments (hydrocortisone, EM or Cer-Mg cream) and the body side on which to apply the cream (left or right). For safety reasons 2 investigators (TR, MFD) were given the allocation list so that in case of an adverse event medical care could be given without delay.
Intervention
Patients were randomly allocated into 2 groups. Group I was treated with Cer-Mg cream on a lesion on one side of the body and simultaneously with HC on a lesion on the contralateral side. Group II was treated simultaneously with Cer-Mg and EM (unguentum leniens, also called cold cream) contralaterally. Patients were instructed to apply one fingertip unit (approximately 1 g) of both creams twice daily for 6 weeks. Patients were instructed not to apply cream on the morning of measurements. Furthermore, patients were asked not to apply any other product on other lesions, except the study creams. Measurements were performed under the same climate conditions (21°C, controlled humidity) between September and January, by one investigator (SAK). In weeks 0, 3 and 6 the parameters were measured and samples of the SC were collected for analysis. A flow diagram is shown in Fig. 1.
Fig. 1. Randomization flow diagram. HC: hydrocortisone; EM: emollients; Cer-Mg: ceramides and magnesium; AD: atopic dermatitis.
Study material
The Cer-Mg cream (Dermalex™ Eczema, Omega Pharma, Nazareth, Belgium) contained: water, ceramide 1 (0.001%), ceramide 3 (1%), ceramide 6 II (0.5%), phytosphingosine, cholesterol, magnesium chloride hexahydrate, zeolite (the combination of magnesium and zeolites are trademarked as MagneoLiteTM), glycerol, cocoglycerides, cetyl alcohol, isopropyl myristate, emulsifiers and preservatives. The control products; hydrocortisone acetate 1% in petrolatum-cetomacrogol (HC) and unguentum leniens (EM, also called cold cream, consists of arachis oil (peanut oil), purified water, white beeswax and glyceryl monooleate) both produced by Fagron, NL, BF (Capelle aan den IJssel, the Netherlands) were, together with the Cer-Mg, packed in blinded tubes by Thiopharma (Maassluis, the Netherlands) according to the good manufacturing practice guidelines. The total lipid content of the Cer-Mg cream was 30%, of the EM 75%, and of the HC 49%.
Clinical parameters (primary outcome)
The primary outcome of the study was the comparison of the treatments based on the change in symptom severity as assessed by the difference in the SCORAD (SCORing Atopic Dermatitis) at 3 and 6 weeks from baseline. SCORAD is based on the total body surface area affected by a disease and visually apparent symptoms (erythema, oedema, excoriation, oozing/crusts, lichenification, dryness) and on 2 subjective parameters (pruritus and sleep deprivation, both measured on a visual analogue scale) (16). Due to the split-body study design a modified SCORAD (local SCORAD) was used (26). By local SCORAD, the scoring parameters were performed on the investigated skin sites and the body surface area was set to 1%.
Biophysical parameters and natural moisturizing factors (secondary outcomes)
The biophysical parameters included TEWL, skin surface pH and erythema. The measurements were conducted within a time-period of 60 min at each visit under controlled environmental conditions. TEWL was measured using a Tewameter 300 (Courage and Khazaka Electronic GmbH, Cologne, Germany) (27). Hydration was measured using a Moisture Meter SC Compact (Delfin, Inc, Kuopio, Finland). Skin pH was measured by a skin pH meter (pH900, Courage and Khazaka Electronic GmbH, Cologne, Germany) and erythema by an erythema meter (DermaSpectrometer; Cortex Technology, Hadsund, Denmark).
Natural moisturizing factors in the stratum corneum
The SC samples were collected with an adhesive tape (3.8 cm2, D-Squame, CuDerm, Dallas, Texas, USA) as described previously (12) and analysed for natural moisturizing factors (NMF) by HPLC-UV (22, 28).
Statistical analysis
Sample size was calculated using power analysis (nQuery advisor). Based on data from our pilot study (unpublished, results available on request) a difference of 5 arbitrary units (AU) (standard deviation (SD) 4.0) on the SCORAD index could be detected in a population of 39 patients (power 80%). Anticipating a drop-out percentage of 20%, we included 50 patients per group. Data analysis was performed using IBM SPSS Statistics® version 20.0. The Shapiro-Wilk test was used to check for data normality. The differences within the investigated parameters or between the 2 treatments were tested by a paired Student’s t-test (normally distributed data, data are shown as the mean value and standard error of the mean (SEM)) or a Wilcoxon signed-rank test (non-normally distributed data, shown as median value with interquartile ranges). A per-protocol analysis was performed as described in the study protocol.
RESULTS
Of 100 patients recruited between October and December 2014, 95 completed the study according to the protocol (group I: 48 patients; 16 males/32 females, median age 28.5 years (range 23.0–51.0 years) and group II: 47 patients 19 males/28 females, median age 25.0 years (range 21.0–35.0 years). Five patients were excluded during the study because of an allergic reaction to EM (n = 2), severe worsening of eczema symptoms (n = 1) or non-compliance with the study protocol (n = 2) (see Fig. 1). Due to technical failure, no reliable measurements of erythema by DermaSpectrometer could be performed; however, visual erythema was measured as a part of the SCORAD index. Furthermore, the measurement of proteins on the tapes from 3 subjects in group II could not be performed, and thus the levels of NMF in those individuals could not be determined. As the main outcome is the difference in parameter change between 2 treatments (e.g. Cer-Mg vs. HC in group I and Cer-Mg vs. EM in group II), the results will be presented separately for each group.
SCORing Atopic Dermatitis
At baseline, there was no significant difference in the (local) SCORAD between the 2 treated skin sites in either arm of the study.
Group I: HC vs. Cer-Mg. Both treatments led to clinical improvement in the test areas, as evidenced by a significant decrease in local SCORAD after week 3 and week 6 (Fig. 2A). The reduction in SCORAD from baseline (ΔSCORAD) was significantly greater for HC compared with Mg-Cer at 3 weeks; however, after 6 weeks there was no significant difference in ΔSCORAD between HC and Cer-Mg (Table I). At week 6, the ΔSCORAD amounted to –11.5 (IQR: –17.4; –5.6) for HC and –9.0 (IQR:–15.9; –5.6) for Cer-Mg.
Group II: EM vs. Cer-Mg. Cer-Mg treatment led to a significantly greater decrease in SCORAD from baseline (ΔSCORAD) compared with EM at both week 3 and week 6 (Table I). At week 6, the ΔSCORAD was –3.5 (IQR: –10.5; 3.0) for EM and –6.7 (IQR:–14.5; –2.0) for Cer-Mg.
Local pruritus (itch) intensity
Results for pruritus show a similar pattern as the SCORAD results; an extensive description can be found in Appendix SI1.
TEWL as a marker of skin barrier
Group I: HC vs. Cer-Mg. The TEWL levels after both Cer-Mg and HC decreased significantly compared with their corresponding baseline values (Fig. 2B) reflecting an improvement of the skin barrier. The decrease in TEWL from baseline (ΔTEWL) after HC and Cer-Mg was comparable and did not significantly differ at both measurement points (Table I).
Fig. 2. (A) Local SCORAD (SCORing Atopic Dermatitis). (B) Trans-epidermal water loss (TEWL). (C) Hydration and (D) natural moisturizing factors (NMF) at baseline, after 3 and 6 weeks of treatment in group I (hydrocortisone (HC) vs. ceramides and magnesium (Cer-Mg); n = 48) and group II (emollients (EM) vs. Cer-Mg; n = 47). Results are shown as medians and interquartile ranges. Significance levels as tested by Wilcoxon signed-rank test: *p < 0.05; **p < 0.01; ***p < 0.001.
Table I. Change from baseline of clinical and biophysical parameters in the treatment group I (ceramides and magnesium (Cer-Mg) vs. hydrocortisone (HC))
|
|
Cer-Mg |
IQR |
HC |
IQR |
p-valuea |
ΔSCORAD |
Week 3 |
–6.25 |
(–8.40; –1) |
–7.75 |
(–15.38; –3.63) |
0.0078 |
(AU) |
Week 6 |
–9.00 |
(–15.93; –5.63) |
–11.5 |
(–17.38; –5.63) |
0.1037 |
ΔPruritus |
Week 3 |
–1.00 |
(–2; 0) |
–1.00 |
(–4; 0) |
0.0104 |
(AU) |
Week 6 |
–2.00 |
(–4; 0) |
–2.00 |
(–4; 0) |
0.6123 |
ΔTEWL |
Week 3 |
–4.75 |
(–13.66; 1.473) |
–7.24 |
(–15.70; 2.21) |
0.104 |
(g/m2/h) |
Week 6 |
–6.28 |
(–12.20; 5.15) |
–5.19 |
(–14.36; 2.21) |
0.083 |
ΔHydration |
Week 3 |
6.95 |
(0.23; 20.03) |
3.90 |
(–1.2; 13.7) |
0.0202 |
(AU) |
Week 6 |
6.75 |
(0.83; 17.28) |
3.85 |
(–2.9; 11.23) |
0.0183 |
ΔNMF |
Week 3 |
0.01 |
(–0.15; 0.23) |
–0.02 |
(–0.18; 0.15) |
0.209 |
(nmol/µg protein) |
Week 6 |
0.08 |
(–0.12; 0.25) |
–0.10 |
(–0.23; 0.06) |
0.0015 |
ΔpH |
Week 3 |
0.00 |
(–0.20; 0.28) |
0.00 |
(–0.28; 0.40) |
0.2475 |
|
Week 6 |
0.00 |
(–0.40; 0.20) |
0.10 |
(–0.30; 0.40) |
0.024 |
aSignificance level of the difference in changes from baseline between 2 treatments (Wilcoxon signed-rank test). Significant values are shown in bold.
IQR: interquartile range; AU: arbitrary units; SCORAD: SCORing Atopic Dermatitis; TEWL: trans-epidermal water loss; NMF: natural moisturizing factors.
Group II: EM vs. Cer-Mg. Cer-Mg treatment did not lead to a significant change in the TEWL from baseline (3), while the EM treatment showed a significant increase in TEWL at 3 weeks. The change in TEWL from baseline (ΔTEWL) was significantly greater after EM compared with Cer-Mg at both time-points (Table II).
Table II. Change from baseline of clinical and biophysical parameters in treatment group II (ceramides and magnesium (Cer-Mg) vs. emollients (EM))
|
|
Cer-MG |
IQR |
EM |
IQR |
p-valuea |
ΔSCORAD |
Week 3 |
–8.50 |
(–11.5; –1.5) |
–3.50 |
(–8; 1) |
0.0058 |
(AU) |
Week 6 |
–6.70 |
(–14.5; –2) |
–3.50 |
(–10.5; 3) |
0.0056 |
ΔPruritus |
Week 3 |
–1.00 |
(–2; 0) |
0.00 |
(–1; 1) |
0.0173 |
(AU) |
Week 6 |
–2.00 |
(–3; 0) |
0.00 |
(–2; 1) |
0.0166 |
ΔTEWL |
Week 3 |
–3.48 |
(–8.24; 3.66) |
2.75 |
(–3.68; 10.07) |
0.005 |
(g/m²/h) |
Week 6 |
–3.19 |
(–8.57; 3.34) |
4.94 |
(–6.97; 12.94) |
0.0208 |
ΔHydration |
Week 3 |
3.10 |
(–3.1; 9.6) |
1.20 |
(–3.2; 6.5) |
0.0401 |
(AU) |
Week 6 |
9.70 |
(–0.7; 18.6) |
1.70 |
(–1.5; 8.4) |
0.0625 |
ΔNMF |
Week 3 |
–0.02 |
(–0.19; 0.10) |
–0.07 |
(–0.20; 0.09) |
0.9767 |
(nmol/µg protein) |
Week 6 |
–0.02 |
(–0.27; 0.21) |
0.01 |
(–0.17; 0.24) |
0.9767 |
ΔpH |
Week 3 |
0.30 |
(–0.1; 0.5) |
0.10 |
(–0.1; 0.3) |
0.5189 |
|
Week 6 |
0.00 |
(–0.2; 0.3) |
0.00 |
(–0.3; 0.3) |
0.4739 |
aSignificance level of the difference in changes from baseline between 2 treatments (Wilcoxon signed-rank test). Significant values are shown in bold.
AU: arbitrary units; IQR: interquartile range; SCORAD: SCORing Atopic Dermatitis; TEWL: trans-epidermal water loss; NMF: natural moisturizing factors.
Hydration
Group I: HC vs. Cer-Mg. Treatment with HC and Cer-Mg significantly improved skin hydration (Fig. 2C). The increase in hydration from baseline (ΔHydration) after Cer-Mg was significantly greater after Cer-Mg compared with HC at weeks 3 and 6 (Table I).
Group II: EM vs. Cer-Mg. Hydration after Cer-Mg was significantly higher than the baseline values at weeks 3 and 6 (Fig. 2C), while hydration after EM treatment improved significantly only after 6 weeks. The changes in hydration from baseline (ΔHydration) were significantly larger after Cer-Mg compared with EM at week 3 (Table II).
Natural moisturizing factors
Group I: HC vs. Cer-Mg. Treatment with Cer-Mg showed a tendency of NMF increase (p = 0.09) (Fig. 2D). In contrast to Cer-Mg, treatment with HC resulted in a significant decrease (by 22%) of NMF levels after 6 weeks. The difference in NMF change from the baseline (ΔNMF) between HC and Cer-Mg emollient was significant at week 6 (p < 0.05), (Table I).
Group II: EM vs. Cer-Mg. EM treatment showed a significant decrease in NMF at week 3 (Fig. 2D). Treatment with Cer-Mg did not influence NMF levels. No significant difference in ΔNMF could be detected between the 2 treatments (Table II).
An extensive description of pH results can be found in Appendix SI1.
DISCUSSION
The results of the present study show that the Cer-Mg cream is an effective approach in improving the clinical symptoms and skin barrier. Although all 3 treatments led to significant improvement in clinical symptoms after 6 weeks, only the HC and Cer-Mg cream reduced SCORAD by more than 8.7 units, which is considered clinically relevant (26). After 3 weeks of treatment HC showed slightly, but significantly, greater reduction in SCORAD than Cer-Mg (–7.8 vs. –6.3), while Cer-Mg showed significantly greater reduction than EM (–8.5 vs. –3.5). The subjective VAS-pruritus scale and the skin barrier function parameter TEWL showed similar results: Cer-Mg and HC showed a significantly beneficial effect, which was, however, not observed after EM treatment. Overall subjective preference slightly favoured the Cer-Mg, which might be of importance in patients’ adherence to therapy. Topical corticosteroids (TCS) are the first-line treatment for AD; however, their long-term use can lead to the deterioration of the skin barrier, which is an important aetiological factor in AD. Moreover, a recent study has shown that therapy with a potent TCS leads to a reduction in NMF levels, which play an important role in skin hydration, antimicrobial defence and skin inflammatory status (29, 30). Our study shows, for the first time, that a low-potency corticosteroid such as HC can lead to a significant reduction of NMF. A decrease in NMF has also been observed after EM treatment at 3 weeks, while Cer-Mg showed a tendency to increase NMF. This emphasizes the importance of this adverse side-effect of HC, as reduced NMF levels may contribute to the recurrent flares. The greatest improvement in SC hydration was observed after Cer-Mg cream that, similarly to HC, showed a decrease in TEWL, but in contrast to HC had no negative effect on NMF levels.
The Cer-Mg cream contains 2 components that might beneficially influence the skin barrier: ceramides (1, 3 and 6 II) and a complex of magnesium and zeolites (31). Huang & Chang (32) have shown that topical application of ceramide 1 and 3 reduces TEWL and increases hydration in sodium lauryl sulfate-irritated, thus beneficial effect of these ceramides, which are also present in Cer-Mg cream, might also have occurred in patients with AD in the present study. As the molecular size of the skin ceramides is > 500 Da, which is proposed as a molecular size cut-off for percutaneous penetration (33), the question arises whether and to which extent each of individual ceramides can penetrate across the SC, realizing that not only the amount, but also their balance is crucial for the skin barrier. Recently, Zhang et al. (34) demonstrated that topically applied ceramides are located mainly in the SC glyphs and that the penetration into the lipid layers is minimal. It is likely that penetration of ceramides through the impaired skin barrier is enhanced in AD; however, RCT studies on the penetration of various ceramides, and their efficacy in improvement of the skin barrier in AD, are lacking.
Another rationale candidate to explain the effectiveness of Cer-Mg cream is magnesium, which is known to be involved in synthesis of ceramides (23). Topical treatments with magnesium-rich Dead Sea salts showed a beneficial effect in dry and pruritic dermatoses (27). Whether the effect of the Cer-Mg cream could be assigned to the presence of ceramides or magnesium remains to be elucidated in a vehicle-controlled trial as some constituents of the vehicle in the Cer-Mg cream, such as glycerol, are also known to lead to improvement in the skin barrier (35, 36).
Strengths and limitations
In this RCT the efficacy of Cer-Mg cream was compared with that of 2 currently used therapeutic options for mild to moderate AD. In most RCTs the efficacy is compared only with either corticosteroid or OTC emollient. The double-blind, split-body design offers a well-paired comparison between 2 treatments, compensating partly for the heterogeneity of the disease severity among patients with AD. The inclusion of biophysical and biochemical parameters provides more insight into the target of the treatment (37). This study did not account for spontaneous resolution of the disease over the study period. However, as the primary aim was to compare the efficacy of Cer-Mg to the upper (hydrocortisone) and lower spectrum of recommended OTC therapy for mild to moderate AD, we did not include an untreated site. Finally, the study does not provide insight into the working mechanism of Cer-Mg, which needs to be confirmed in the separate vehicle-controlled clinical trial.
Conclusion
The present study shows that, after 6 weeks of treatment, Cer-Mg cream offers benefits over high lipid-OTC emollients and comparable clinical efficacy to hydrocortisone. In addition, in contrast to hydrocortisone, it does not influence negatively the concentration of NMF. Cer-Mg may therefore offer a non-steroid alternative for the treatment of mild to moderate AD. Furthermore, the fact that Cer-Mg might be used as a stand-alone treatment for mild and moderate AD as well as a maintenance therapy might improve adherence to AD therapy.
ACKNOWLEDGEMENTS
Funding: Omega Pharma provided study medication free of charge. However, it had no involvement in generating, analysing or processing data, nor in scientific input, and it had no input into the generation of this article.
Statement of all funding sources that supported the work: This study was supported by Omega Pharma, Nazareth, Belgium.
Conflict of interest: S. A. Koppes was reimbursed by Omega Pharma for international conference attendance.
1http://www.medicaljournals.se/acta/content/?doi=10.2340/00015555-2395
REFERENCES