Journal of Futures Studies, September 2021, 26(1): 75–82
As Researchers, We Need to Engage More into Public Outreach Towards Children in The Future
Lucie Poulet, Journal DECODER, Clermont-Ferrand, France ; NASA Postdoctoral Program, Kennedy Space Center, FL, USA
Benjamin Dalmas, Journal DECODER, Clermont-Ferrand, France ; Mines Saint-Etienne, Centre CIS, Saint-Etienne, France
Barbara Goncalves, Journal DECODER, Clermont-Ferrand, France ; Université Clermont Auvergne, Centre Michel de l’Hospital, Clermont-Ferrand, France
Camille Noûs, Cogitamus Laboratory, France
Antoine Vernay1,6, Journal DECODER, Clermont-Ferrand, France ; Université Claude Bernard Lyon 1, CNRS, ENTPE, UMR 5023 LEHNA, Villeurbanne, France
* Web Text version of each JFS paper here is for easy reading purpose only, for the valid and published context of each article, please refer to the PDF version.
Abstract
Staying informed is essential for citizens but not everyone is equipped with the right tools to distinguish scientific facts from opinions, well-founded arguments from sensationalist news. Popularization and outreach should be more focused on transmitting methods, demonstrations, and arguments rather than scientific results. It is important to keep concepts’ complexity while doing so and to target young audiences. The journal DECODER offers an outreach journal allowing direct exchange between the classroom and researchers to provide critical tools to young minds. It is a duty of researchers to engage into outreach activities, as providers of new knowledge and advocates of the scientific method.
Keywords: Student-Researcher Interactions, Criticism, Outreach, Knowledge Complexity, Scientific Method
Introduction
What is the role of researchers in the 21st century, when their messages seem lost in the glut of information? Researchers generate scientific results, published in form of academic articles, and valorized through patents and presentations at congresses; they also train their colleagues and students and, they disseminate this knowledge through public outreach. Regardless of the science field (from classical Science Technology Engineering Mathematics (STEM) fields to humanities and economics), knowledge originates from questions, leading to hypotheses being tested via empirical experiments or texts, situational analyses, resulting in answers to the initial questions.
Knowledge is thus classically defined as the understanding of a topic or the information thereabout, acquired by experience or study (Cambridge dictionary); it is the gateway to understanding our world and can be used for specific purposes facing different societal needs. Knowledge enables the comprehension of our potential and current actions in order to possibly avoid — or at least mitigate — the negative effects thereof. The prime user of this knowledge, deriving from scientific results and generated by researchers, is the scientific community itself, to further explore a question; then, authorities need and use this knowledge to meet societal challenges; besides, entrepreneurs use it to develop new products; and finally, all citizens use it to make informed decisions, have a critical look on their government’s choices, and better understand the world.
Access to knowledge derived from scientific research is crucial for all citizens in a society, and yet numerous limits do exist, preventing full and free access to everybody. Freedom of information has several political, economic and sociological stakes (Lor & Britz, 2007), and research, as a small contributor to the mass of information, shares the same ones. The authors assert that researchers’ roles in the transmission of knowledge needs redefinition. The authors argue that researchers should get involved and engage more concretely in knowledge diffusion, especially towards the younger generation. Finally, the authors discuss one way of how this can be achieved.
Knowledge: Essential and Remote
The importance of accessing knowledge
Informed choices, which citizens are bound to make on a daily basis, derive from their knowledge and information literacy on a given subject, but also from their attitude towards information, whether it is a passive “news will find me” posture or an active behavior checking different information sources (Gil de Zúñiga & Diehl, 2019). Romijnders et al. (2019), for example, showed that people who refuse vaccines have less knowledge about vaccine-preventable diseases; they perceive less risk about these diseases and their refusal is based on non-evidence arguments (Romijnders et al., 2019). Giving citizens access to (scientific) knowledge provides them with the tools needed to develop their critical thinking, i.e. efficient reading, analysis and synthesis of different resources (Bezanilla, Fernandez-Nogueira, Poblete, & Galindo-Dominguez, 2019; Ku et al., 2019). Therefore, education and awareness on critical thinking and critical literacy are crucial along with easier knowledge access and should be included in all education programs (Georgiadou et al., 2018; Berki & Valtanen, 2007).
In the last decades, citizen-science projects have emerged around the world; they show that the more people have access to scientific knowledge, the more science can progress, thanks to critical discussions (Bonney, Phillips, Ballard, & Enck, 2016). These projects and the proximity they create between the research world and the citizens also create a virtuous circle, where involved citizens always seek to learn and discover more (Pocock, Tweddle, Savage, Robinson, & Ro, 2017). Following this trend, more and more researchers have their own blog, YouTube channel or social media account to share their work, some journals like “Frontiers for young minds” enable young students to review research articles, and innovative actions like the Knowledge Democracy Initiative of the Action Research Network of the Americas (ARNA) have emerged (Rowell & Call-Cummings, 2020). Science thus becomes tangible instead of an abstract belief while enabling more transparency in science practice (Blanchard, 2011; Bobroff, 2017).
With the objectives of simultaneously fostering a wide scientific diffusion at no cost and helping young students develop their critical mind, as well as creating more direct interaction between researchers and the general public, the authors created the outreach journal DECODER — in French, to decode (each letter of this French acronym stands for Discover, bE Curious, Open-minded, Demonstrate, Experiment, Research) — written by and for junior high and high school students (Poulet, Vernay, Goncalves, Dalmas, Vernay, 2020). Students can either actively revise a short version of a published academic article — in tight collaboration with one of the co-authors — to transform it into an outreach article that can be understood by people their age, or they can set up their own experiment and write an article about it, displaying hypotheses, methods and results, while getting feedback from a researcher in the field (Fig. 1). The ambition of this journal is to propose a free alternative way of sharing science — issuing on the DECODER website students’ outreach articles based on published academic manuscripts, with the voluntary participation of original authors — while complementing already existing initiatives.
Fig. 1: Schematic view of the participation process for a researcher getting involved in DECODER. Source: the authors.
For instance, Dr. Collard, a researcher in marine biology of the Norwegian Polar Institute published an outreach version of her article (Collard et al., 2017) about microplastics in anchovies’ liver with a junior high school class in November 2020. She proposed a shortened version of her original article and the class reformulated some sentences, discussed with her about some complex topics. One of the most significant added values was the inclusion of a lexicon, which will help the audience in their understanding of the article. Another researcher, Dr. Dalmas, researcher in computer science, discussed his article (Dalmas, Fournier-Viger, & Norre, 2017) with a class of junior high school students, about an algorithm that detects past choices to predict future choices in the field of health data. This work enabled students to get familiar with algorithms and data science. They proposed the following changes: improving the clarity of the paper, the illustration that summarizes it, and even rightly highlighted a typo in a formula. Dr. Lucie Poulet, researcher in process engineering applied to space, worked with a high school class on an outreach version of her publication (Poulet, Fontaine, & Dussap, 2016) as a guide for future space gardeners on plant’s response to space environment. The students’ revision of the article led to additional photos and multiple word explanations, which made the article more accessible to a younger audience.
Restricted and too often unreachable
As Vosoughi, Roy, and Aral (2018) pointed out, fake news, often more sensational than accurate information, tend to spread much faster; in addition, scientists warning (e.g., on the global climate crisis) are regularly ignored (Washington, Lowe, & Kopnina, 2019), and scientific experts in the media are too often invited based on their overall fame, rather than on their actual expertise in the field of interest (Petersen, Vincent, & Westerling, 2019). This is in particular a consequence of a general lack of involvement in public outreach from researchers.
Public outreach and popularization in the academic world are often, in fact, solely incidental and considered unrewarding or unprofitable compared to scientific articles publication. Consequently, little training is dedicated to public outreach in researchers’ career and they are given neither the tools nor the time to effectively fulfill this role — yet an integral part of their job. In less than two years, however, the journal DECODER has gathered more than thirty authors, from three continents, eager to share their articles with a class. This strong participation rate (relative to the small communication on the project, at this stage) shows that there is an interest among researchers to get involved in outreach projects and share their knowledge.
For years, researchers have focused their public outreach efforts towards the outcomes of their research work, rather than towards the methodology they use. It is, however, hard for people to trust what they do not understand and therefore sharing knowledge per se generally performs poorly (Kahan et al., 2012).
Additionally, scientific media and academic journals have become a high-value, high-cost commodity, thus limiting their availability to the general public, but also to researchers themselves (Lawson et al., 2019). This is however another topic, not discussed in detail hereafter.
A New Hope
The emphasis on the new generation
The authors advocate that a significant part of researchers’ outreach activities should focus on the youth. Indeed, if the new generation is left without the right tools for critical thinking, fake news and conspiracy theories are likely to hit it the hardest (Petersen et al., 2019; Ku et al., 2019). Besides, young people demonstrate engagement and concerns about current global problems, such as the climate crisis. This was demonstrated by recent protests all over the world — largely supported by the scientific community — against political inaction on climate (Hagedorn et al., 2019). The developing minds of children are more prone to easily grasp and question the world’s complexity than adults, who already have existing prejudices on a wide variety of topics. A recent study exhibited that children can trigger changes in their parents’ way of thinking: even the most conservative parents became more sensitive to the climate crisis and human responsibility within it, when their children learned about climate change causes and consequences (Lawson, Sanders, & Smith, 2015).
However, there is a need to adapt the outreach methods to the new generation. More than ever, the developing technologies have changed the way to teach, to learn. Scientific evidence indicates that frequent digital technology use has a significant impact — both negative and positive — on brain function and behavior (Small et al., 2020). Extensive screen time is a risk factor in the development of attention-deficit hyperactivity disorder symptoms (Anderson & Jiang, 2018). Although uncertain, reasons behind this phenomenon might be attributed to repetitive attentional shifts and multitasking. In addition to the reduction of attentional capacities, further studies showed that despite being a booster in the information collection, the technology has limited some brain region activation to store this information in the long term, leading also to impeded brain maturation and poor verbal intelligence (Firth et al., 2019). Technology has also challenged their ability to process and/or trust the growing amount of information accessed faster than ever. On the bright side, children have also increased their multitasking skills, as well as their capacity to reason and think flexibly and to retain information over a brief period of time (Small et al., 2020). This shift in children’s cognitive abilities implies a different approach to involve them in efficient outreach activities.
DECODER allows remote outreach activities and interactions with potentially any researcher over the world, which can accommodate busy schedules of schools and researchers. However, participating researchers are encouraged to visit classes in person if they can, or organize laboratory visits, to promote real student-researcher interactions.
In order to engage the youth, a popular education approach is favored — an integrative method encouraging children to experiment, test their own hypothesis, and critically assess the outcome (Kerka, 1997). Researchers could indeed explain scientific concepts or proofs forever; if children cannot recreate them, popularization has failed. Confronting people, and children in particular, to the scientific method while giving them tools to test different hypotheses increases their confidence and attention, and it improves interactions between stakeholders — here the researchers and the children (O’Brien et al., 2013). The regular practice of critical thinking and experimental approach enables them to build stronger arguments and examine different standpoints. As Hinton recommended in his speech at the AAAI-20 invited speaker program, citing his advisor, “Figure out how you would solve the problem yourself, once you figured out how to solve it, then go read the literature” (Hinton, Le Cunn, & Bengio, 2020, 1h54min28s) As researchers, our duty is to get involved in these popular education projects to get people to feel concerned about research (Knezz, 2019), and relate to its strengths and weaknesses.
A strong scientific argument is never easy to build, and science history abounds in changes of paradigm — the Continental Drift theory by Wegener, the Theory of Evolution by Darwin and Wallace or the General Relativity by Einstein, to only name a few. These are caused by unintentional mistakes and discoveries, disagreements and debates between researchers, supporting the fact that humanity’s knowledge grows from questioning and requires collaborative work with a large array of skills (Brown, Kaiser, & Allison, 2018; Craine, 2007; Lortie, Brooker, Kikvidze, & Callaway, 2004).
Making science more accessible for all
First, following the recommendation of the European commission of July 17th, 2012 (Landes, 2014), the authors argue that publicly-funded research should be shared and made accessible free of charge to the general public. Our approach with the journal DECODER is in line with this philosophy, enabling free access to state-of-the-art research, in a comprehensive language for a large audience. The tight collaboration between one (or more) author(s) of the original research article and students avoids editorial bias and speeds up access to knowledge for everyone (Fig. 1).
Another way of making science and research results more accessible is to get back to the basics of a researcher’s role, i.e. a vector of knowledge, by embracing popularization (Bobroff, 2017).
The emphasis should be put on the process of scientific knowledge production, on the “how”, rather than on the “what”. It is crucial for a layperson to understand that scientific knowledge is the result of trials and errors (Brown et al., 2018), changes of paradigm, collaborative work (Bonney et al., 2016), requiring confrontations of theories and experimental data, rather than an opinion, shared millions of times on social media (Vosoughi et al., 2018). If the emphasis is set on scientific results only, it can be counterproductive, since strong scientific arguments could become just another opinion in people’s minds who would not think critically (O’Brien et al., 2013); it could also question the researchers’ legitimacy in the matter (Sugimoto et al., 2013).
The “teacher-learner” approach — commonly used to share knowledge — does have some advantages in the educational dimension, but it is unsuitable for popularization. It is also more and more questioned, since it only allows little opportunity for creative or problem-solving activity (Laginder, Nordvall, & Crowther, 2013). There is a crucial need to flatten the hierarchical relationship between people who possess knowledge and the ones who wish to gain it. O’Brien et al. (2013) advocate for a revolution in education by increasing inter- and transdisciplinary knowledge and to “move from science for society to science with society” (O’Brien et al., 2013, p. 54). The approach of DECODER brings together scientists and society through the collaboration of researchers with classes, and DECODER is a multi- and transdisciplinary journal. Hence, our initiative modestly aims to participate in this education revolution. It is important to specify that researchers do not possess ultimate and unquestionable knowledge. On the contrary, they are people who dig into complex problems and bring a new perspective to its resolution, but they are nonexempt to subjectivity. Therefore, good popularization including a horizontal relationship with the audience implies for researchers to face and accept constructive criticisms.
Finally, popularizing research is not about “making it simpler”; complexity should be part of the message (Morin, 2007). A recurring trend is an oversimplification of concepts (also known as “dumbing down”) and retaining only a part of knowledge, which often leads to misconceptions and a false sense of mastering a topic (also known as the “easiness effect”) (Scharrer, Rupieper, Stadtler, & Bromme, 2017; Anspach, Jennings, & Arceneaux, 2019).
Our experience with the journal DECODER demonstrated that making students work on complex concepts is a way of stimulating their curiosity and forces them to question their readings — this can only be achieved through discussions and direct exchange with the researcher.
Conclusion
An easy and straightforward knowledge access is essential for a society’s well-being and still too many obstacles remain, one of them being the lack of training of researchers in public outreach. Popularization with an emphasis on methods and reasoning, rather than only on results, is the gateway to engaging a wider audience, especially the younger generation. Therefore, researchers “going back to school” could promote critical and resilient thinking for students (Poulet, et al., 2017).
The authors agree with the editorial of Gardner and Wordley (2019, p. 1271) who “encourage [their]fellow scientists to join [them]in embracing activism”. Our action towards young people aims to meet the global momentum of researchers increasingly and actively sharing scientific methods to raise awareness. Let researchers become knowledge-lobbyists, arming future citizens with the tools they need to face the world’s complexity and upcoming global crises.
Notes
- Camille Noûs embodies the collegial nature of our work, as a reminder that science proceeds from disputatio and that the building and dissemination of knowledge are intrinsically selfless, collaborative and open.
References
Anderson, M., & Jiang J. (2018). Teens, social media & technology 2018. Pew Research Center, Internet and Technology.
Anspach, N. M., Jennings, J. T., & Arceneaux, K. (2019). A little bit of knowledge: Facebook’s news feed and self-perceptions of knowledge. Research & Politics, 6(1), 2053168018816189. Publisher: SAGE Publications Ltd.
Berki, E., & Valtanen, J. (2007). Critical and creative mathematical thinking with practical problem-solving skills-a new old challenge. Dranidis, D. and Sakellariou, I. (Eds). The 3rd South-East European Workshop on Formal Methods.
Bezanilla, M. J., Fernandez-Nogueira, D., Poblete, M., & Galindo-Dominguez, H. (2019). Methodologies for teaching-learning critical thinking in higher education: The teacher’s view. Thinking Skills and Creativity, 33,100584.
Blanchard, A. (2011). Science blogs in research and popularization of science: why, how and for whom? Moira Cockell; Jerome Billotte; Frederic Darbellay; Francis Waldvogel.EPFL Press.
Bobroff, J. (2017). Reimagining physics. Nature Nanotechnology, 12(5), 496–496.
Bonney, R., Phillips, T. B., Ballard, H. L., & Enck, J. W. (2016). Can citizen science enhance public understanding of science? Public Understanding of Science, 25(1), 2–16.
Brown, A. W., Kaiser, K. A., & Allison, D. B. (2018). Issues with data and analyses: Errors, underlying themes, and potential solutions. Proceedings of the National Academy of Sciences, 115(11), 2563–2570.
Collard, F., Gilbert, B., Compère, P., Eppe, G., Das, K., Jauniaux, T., & Parmentier, E. (2017). Microplastics in Livers of European Anchovies (Engraulis Encrasicolus, L.). Environmental Pollution, 229, 1000‑1005. doi : https://doi.org/10.1016/j.envpol.2017.07.089.
Craine, J. M. (2007). Plant strategy theories: replies to Grime and Tilman. Journal of Ecology, 95(2), 235–240.
Dalmas, D., Fournier-Viger, P., & Norre, S. (2017). TWINCLE : A Constrained Sequential Rule Mining Algorithm for Event Logs. Procedia Computer Science, 112 :205-214, ISSN 1877-0509, doi : https://doi.org/10.1016/j.procs.2017.08.069.
Firth, J., Torous, J., Stubbs, B., Firth, J. A., Steiner, G. Z., Smith, L., Alvarez‐Jimenez, M., Gleeson, J., Vancampfort, D., Armitage, C. J., & Sarris, J. (2019). The “online brain”: how the Internet may be changing our cognition. World Psychiatry, 18, 119–129.
Gardner, C. J., & Wordley, C. F. R. (2019). Scientists must act on our own warnings to humanity. Nature Ecology & Evolution, 3, 1271–1272.
Georgiadou, E., Rahanu, H., Siakas, K. V., McGuinness, C., Edwards, J. A., Hill, V., Khan, N., Kirby, P., Cavanagh, J., & Knezevic, R. (2018). Fake news and critical thinking in information evaluation. In WBILC 2018 – Proceedings International Scientific Conference “Western Balkan Information Literacy”: Alternative facts, Fake News, getting to the truth with Information Literacy, 21-22 June 2018, pages 50–71.
Gil de Zúñiga, H., & Diehl, T. (2019). News finds me perception and democracy: Effects on political knowledge, political interest, and voting. New Media & Society, 21(6), 1253–1271. Publisher: SAGE Publications.
Hagedorn, G., Kalmus, P., Mann, M., Vicca, S., Berge, J. V. d., Ypersele, J.-P. v., Bourg, D., Rotmans, J., Kaaronen, R., Rahmstorf, S., Kromp-Kolb, H., Kirchengast, G., Knutti, R., Seneviratne, S. I., Thalmann, P., Cretney, R., Green, A., Anderson, K., Hedberg, M., Nilsson, D., Kuttner, A., & Hayhoe,K. (2019). Concerns of young protesters are justified. Science, 364(6436), 139– 140.
Hinton, G., Le Cunn, Y., & Bengio, Y. (2020). AAAI 2020 Keynotes Turing Award Winners Event, https://youtu.be/UX8OubxsY8w
Kahan, D. M., Peters, E., Wittlin, M., Slovic, P., Ouellette, L. L., Braman, D., & Mandel, G. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks. Nature Climate Change, 2(10), 732–735.
Kerka, S. (1997). Popular education: adult education for social change. ERIC Clearinghouse on Adult Career and Vocational Education Columbus OH, 185.
Knezz, S. N. (2019). Drawing a new scientist: Why I come out to my chemistry class. Journal of Chemical Education, 96(5), 827–829.
Ku, K. Y. L., Kong, Q., Song, Y., Deng, L., Kang, Y., & Hu, A. (2019). What predicts adolescents critical thinking about real-life news? the roles of social media news consumption and news media literacy. Thinking Skills and Creativity, 33,100570.
Laginder, A.-M., Nordvall, H., & Crowther, J. (2013). Popular education, power and democracy: Swedish experiences and contributions. NIACE – National Institute of Adult Continuing Education.
Landes, X. (2014). Les enjeux normatifs et politiques de la diffusion de la recherche. Bénéfices sociaux et libre accès. European Journal of Social Sciences, 52, 65–92.
Lawson, D. F., Stevenson, K. T., Peterson, M. N., Carrier, S. J., L. Strnad, R., & Seekamp, E. (2019). Children can foster climate change concern among their parents. Nature Climate Change, 9(6), 458–462.
Lawson, S., Sanders, K., & Smith, L. (2015). Commodification of the information profession: A critique of higher education under neoliberalism. Journal of Librarianship and Scholarly Communication, 3(1).
Lor, P. J., & Britz, J. J. (2007). Is a knowledge society possible without freedom of access to information? Journal of Information Science, 33:387–397. Publisher: Sage Publications Sage UK: London, England.
Lortie, C. J., Brooker, R. W., Kikvidze, Z., & Callaway, R. M. (2004). The value of stress and limitation in an imperfect world: A reply to Körner. Journal of Vegetation Science, 15(4), 577–580.
Morin, E. (2007). Restricted complexity, general complexity. In Science and us: Philosophy and Complexity. Singapore: World Scientific, pages 1–25.
O’Brien, K., Reams, J., Caspari, A., Dugmore, A., Faghihimani, M., Fazey, I., Hackmann, H., Manuel-Navarrete, D., Marks, J., Miller, R., Raivio, K., Romero- Lankao, P., Virji, H., Vogel, C., & Winiwarter, V. (2013). You say you want a revolution? transforming education and capacity building in response to global change. Environmental Science & Policy, 28, 48–59.
Petersen, A. M., Vincent, E. M., & Westerling, A. L. (2019). Discrepancy in scientific authority and media visibility of climate change scientists and contrarians. Nature Communications, 10(1), 1–14.
Pocock, M. J. O., Tweddle, J. C., Savage, J., Robinson, L. D., & Roy, H. E. (2017). The diversity and evolution of ecological and environmental citizen science. PLOS ONE, 12(4), e0172579. Publisher: Public Library of Science.
Poulet, L., Fontaine, J.-P., & Dussap, C.-G. (2016). Plant’s response to space environment: a comprehensive review including mechanistic modelling for future space gardeners. Botany Letters, 163(3), 337-347, DOI: 10.1080/23818107.2016.1194228.
Poulet, L., Vernay, A., Dalmas, B., Vernay, M., Delpeuch, P., & Sinn, T. (2017). A Learning Method Based on a Mission to Mars for Primary School Children. Proceedings of the 68th IAC, Adelaide, Australia.
Poulet, L., Vernay, A., Goncalves, B., Dalmas, B., & Vernay, M. (2020). A Multidisciplinary Scientific Outreach Journal Designed for and Made by Middle and High School Students to Bring Research Closer to the Classroom. Proceedings of the International Conference on Environmental Systems.
Romijnders, K. A. G. J., van Seventer, S. L., Scheltema, M., van Osch, L., de Vries, H., & Mollema, L. (2019). A deliberate choice? Exploring factors related to informed decision-making about childhood vaccination among acceptors, refusers, and partial acceptors. Vaccine, 37(37), 5637–5644.
Rowell, L., & Call-Cummings, M. (2020). Knowledge democracy, action research, the internet and the epistemic crisis. Journal of Futures Studies, 24(4), 73-81.
Scharrer, L., Rupieper, Y., Stadtler, M., & Bromme, R. (2017). When science becomes too easy: Science popularization inclines laypeople to underrate their dependence on experts. Public Understanding of Science, 26(8), 1003–1018.
Small, G. W., Lee, J., Kaufman, A., Jalil, J., Siddarth, P., Gaddipati, H., Moody, T. D., Bookheimer, S. Y. (2020). Brain health consequences of digital technology use. Dialogues in clinical neuroscience, 22(2), 179-187. doi: 10.31887/DCNS.2020.22.2/gsmall.
Sugimoto, C. R., Thelwall, M., Larivière, V., Tsou, A., Mongeon, P., & Macaluso, B. (2013). Scientists popularizing science: characteristics and impact of TED talk presenters. PLoS ONE, 8(4).
Vosoughi, S., Roy, D., & Aral, S. (2018). The spread of true and false news online. Science, 359(6380), 1146–1151.
Washington, H., Lowe, I. & Kopnina, H. (2019). Why do society and academia ignore scientists warning to humanity on population. Journal of Futures Studies, 23(4), 17.