Which of the following is the approach in which trained professionals interpret test results?

Cited by [0]

Pros and Cons of Computerized Psychological Assessment

Computerized assessment owes much of its recent growth and status to the unique advantages that computers offer to the task of psychological assessment in comparison to clinician derived assessments. First, computers are time and cost efficient. Computerized reports can be available shortly after the completion of the test administration, saving valuable professional time.

Another advantage of using computers in psychological assessment is their accuracy in scoring, inasmuch as computers are less subject to human error when scoring [Allard, Butcher, Faust, & Shea 1995; Skinner, & Pakula, 1986].

Third, computers provide more objective and less biased interpretations by minimizing the possibility of selective interpretation of data.

A fourth advantage of computerized reports is that they are usually more comprehensive and thorough than clinicians’ reports. In a computerized interpretation, the test taker’s profile is examined in comparison to many other profiles. Therefore, test information can be more accurately used to classify the individual, while describing the behaviors, actions, and thoughts of people with similar profiles. In sum, a well-designed statistical treatment of test results and ancillary information will generally yield more valid conclusions than an individual professional using the same information [APA, 1986].

Finally, computerized test administration may be more interesting to some subjects, who may also feel less anxious responding to a computer monitor [Rozensky et al., 1986] than the more personal context of a paper-and-pencil test.

While the advantages of computerized assessment are many, this method is not totally problem-free. One major problem associated with automated administration, scoring, and interpretation is misuse by unqualified professionals. Skinner and Pakula [1986] suggest that computerized assessment may inadvertently encourage use by professionals without adequate knowledge and experience. It is important to keep in mind that the validity of the information obtained by computerized psychological assessment can be ensured only in the hands of a professional with adequate training and experience with the particular test in question. Turkington [1984] pointed out that computerized assessment, when used by those with little training or skill in test interpretation, can do more harm than good.

A second risk of the computer-assisted assessment is that mental-health professionals might become excessively dependent on computer reports, and accordingly become less active in personally interpreting test data. Computerized reports cannot take the place of important clinical observations, which provide essential information to be integrated with results from formal testing [Butcher, 1995b].

A third problem comes from the fallacy that computer-generated assessments yield information that is necessarily factual. Matarazzo [1983, 1986] cautioned professionals against the face validity of computer-generated interpretations. It cannot be assumed that computer assessments generate precise scientific statements that cannot be questioned. Computer-based conclusions are not chiseled in stone, and a critical review of such interpretation is necessary for their credible use.

Fourth, b computer statements in a computer report might not provide specific information about the test taker useful for diagnostic purposes. Practitioners should be cautious of “Barnum-type” statements that some computer reports may generate. Basing clinical decisions on this type of statement can lead to inaccurate recommendations [Butcher, 1995].

Finally, a computerized report might include statements that do not apply to every patient. It is important to keep in mind that computer reports are general descriptions of profiles. It is quite possible that individuals with similar profiles will not possess all of the characteristics identified by a particular profile. It is incumbent on the professional to ascertain the accuracy of test reports for each individual client [Butcher, 1995b].

Computerized assessment

Computerized assessment is a broad term that includes measurement via computer or tablet. Much of the research conducted thus far has centered around cognitive testing. Computerized assessment can be particularly beneficial for cognitive testing due to potential for greater recording accuracy and precision of timed tasks, easy scoring, and standardized administration without biases. As such, researchers have both developed and translated a number of assessments for cognitive impairment via computer or tablet. While the literature on computerized assessment of psychiatric conditions is limited, some suggest that this mode of assessment could facilitate discussion of issues that are less frequently disclosed to providers [e.g., alcohol use; Nemes et al., 2004].

Users of computer-based assessments report that this format is feasible. Older adults and individuals with cognitive impairment have rated computer assessments easy to use and understand [e.g., Fillit, Simon, Doniger, & Cummings, 2008]. When asked about their computerized assessment experience, older adults generally report that they believe test results reflect their abilities, though many note the concern that factors such as health condition and motor control affected their performance [Robillard, Lai, Wu, Feng, & Hayden, 2018]. Thus, it is important to broadly consider potential factors that may affect performance. While older adults see value in the computerized assessments, they may prefer that these assessments are used as an adjunct to, rather than a replacement for, care from clinicians [Robillard et al., 2018].

Assessment via tablet may present different functions than those offered on the computer. While tablets may be used for initial screening, their portability may facilitate monitoring progress over time. Older adults tend to respond positively to tablet-based assessments. In one survey, older adult participants [age range 65–88] were asked to imagine they were in a research study on antidepressants and subsequently introduced to iPad versions of the NIH Toolbox Psychological Wellbeing measure and the NIH Toolbox Cognition and Motor batteries [Lenze et al., 2016]. Eighty-five percent believed that assessment before and after treatment was important and participants found tablet-based assessments generally acceptable.

When tablet- and computer-based assessments have been adapted from paper-and-pencil tests, equivalence across tests cannot be assumed and some suggest that it is particularly important that instructions are explicit [Jenkins et al., 2016]. Considering technological familiarity and comfort are also crucial. Some suggest training for older adults to ensure sufficient understanding prior to the assessment [see Wild, et al., 2008] and ensuring that the user interface is clear. In addition, user attitudes toward the device may affect motivation and, subsequently, performance [Jenkins et al., 2016].

Assessments

I move on to Internet-based computerized assessments which basically follow the emergence of Internet therapies and have been around for more than 20 years [Buchanan, 2002]. All Internet research including clinical implementations use online questionnaires to collect outcome measures, but also for screening, epidemiological, and online psychological experiments. With regards to questionnaires, there is ample evidence showing that psychometric properties and characteristics of measures remain stable and also have advantages such as reducing the possibility skip items [van Ballegooijen, Riper, Cuijpers, van Oppen, & Smit, 2016]. Moreover, it is also possible to use stepwise procedures with, for example, screening questions leading to further questions. Online assessment procedures are increasingly incorporated in clinical practice [Zimmerman & Martinez, 2012]. One cautionary note is, however, that the format should not change between paper-and-pencil and online questionnaires in research studies [Carlbring et al., 2007]. With regards to diagnostic procedures, self-report questionnaires cannot replace diagnostic interviews [Eaton, Neufeld, Chen, & Cai, 2000], but increasingly video consultations are and have the benefit of not requiring the client to visit a clinic [Chakrabarti, 2015].

4.17.5.1 Test Development

The Ethics Code [APA, 1992] states clearly that psychologists involved in the development and provision of computerized assessment services accurately describe the purpose, development procedures, norms, validity, reliability, and applications of the service as well as particular qualifications or skills required for their use. Psychologists participating in such product development should attempt to clearly link interpretive statements to specific client scores or profiles, qualify narrative statements to minimize the potential for misinterpretation and perhaps provide some form of warning statement to alert users to the potential for misinterpretation [Hofer & Green, 1985]. At the very least, the developer might note that the clinical interpretations offered in narrative printouts are not to serve as the sole basis on which important clinical decisions are made [Matarazzo, 1986]. Adherence to the highest standard of the profession would also require developers to provide rather detailed information regarding the system's development and structure in a separate manual. Because individual users are responsible for determining the validity of any CBTI for individual test-takers, availability of such system information is critical. Bersoff and Hofer [1991] noted that in spite of the apparent conflict between the developer's proprietary interest in the product and the clinician's need to responsibly evaluate the service, open and critical review of tests and CBTIs is critical for ensuring the quality of such materials and upholding the profession's ethical code.

Advantages and challenges in early adoption

Dating from the first attempts to implement computers as cognitive assessment devices, there was an appreciation of the potential advantages as well as the challenges and cautions that surrounded computerized assessment. The advantages included standardization in test administration superior to that of the human examiner, scoring accuracy, the ability to integrate response timing into a variety of tasks in order to better assess processing speed and efficiency, expanded test metrics that capture variability in performance, the ability to integrate adaptive testing for both test and item selection, and the ability to incorporate tests not easily done with booklets or pieces of paper. The cautions included that computers could be deceptive with respect to timing accuracy and developers of automated tests had to be aware of issues related to response input methods, operating systems, program coding, and computer architecture that could affect response timing and the measurement of test performance. In the early years of automated testing, computers were relatively new devices not yet fully integrated into daily life. Hence, there were also concerns about how an individual would react to being tested on a computer. These concerns were supported by studies addressing the effects of computer familiarity on test performance [Johnson & Mihal, 1973; Iverson, Brooks, Ashton, Johnson, & Gualtieri, 2009]. The biggest limitation in adopting computerized testing for clinical assessment was that available technology did not permit the assessment of important language-based skills, including verbal memory. Algorithms for analyzing speech patterns for indications for anxiety, depression, or unusual syntax or word usage were also not available or were in the early stages of development. Constraints related to speech recognition and language analysis limited the types of tests and methods that have been implemented on computer for clinical assessment.

Universities in a globalised setting

Even though there is no agreement on the elements that comprise excellence in higher education, in the past decade we have witnessed a veritable explosion in university rankings. We can cite, for example, the Academic Ranking of World Universities [ARWU]1 published by the Center for World-Class Universities and the Institute of Higher Education of Shanghai Jiao Tong University. This system emphasises publications, citations and academic prizes, especially in science and technology. The QS World University Rankings2 relies heavily on academic peer review [which accounts for 40 per cent]. The SCImago Institutions Rankings [SIR]3 is built with data from Elsevier database Scopus. The SIR 2009 World Report ranks the best 2,000 worldwide research institutions and organisations and analyses their research performance in the period 2003–7 through five global output indicators. In turn, Webometrics Ranking of World Universities4 produced by the Cybermetrics Lab [National Research Council of Spain], offers information about more than 8,000 universities according to their web presence, a computerised assessment of the scholarly contents and visibility and impact of the whole university web domain.

Obviously, the results are not all similar because of the relative weight assigned to the indicators used, which means that we can find a single institution with widely divergent rankings depending on the list chosen. A careful statistical analysis of international ranking concludes that there is broad consensus about the first 10–12 universities, but after that the lists begin to diverge. The lack of an absolute set of performance criteria may mean that ‘world class’ standing will probably be based more on academic reputation than on a set of formal standards [Mohrman et al., 2008].

The popularity of this kind of ranking is a clear sign of the globalisation of knowledge and the internationalisation of university teaching. The more traditional comparisons among institutions within a single country have been eclipsed by observations that scrutinise a university’s position beyond the restricted political or linguistic frontiers.

In this context, networks of excellence have arisen. The International Alliance of Research Universities [IARU]5 is perhaps the network that best illustrates this phenomenon. It was set up in 2006 and includes ten leading research universities: Australian National University, ETH Zurich, National University of Singapore, Peking University, University of California at Berkeley, University of Cambridge, University of Copenhagen, Oxford University, University of Tokyo and Yale University. Equally noteworthy is the fact that several universities have set up a group that they define as the ‘Emerging Global Model [EGM]’ for the university of the twenty-first century. Mohrman et al. [2008] argue that the development of the EGM is both a response to and an influence upon the major factors in contemporary society. EGM universities look worldwide for research partners, graduate students, prospective faculty and financial resources. This group of global universities will form an elite subset in a larger universe of higher education institutions. The growth of international university associations demonstrates the interdependence of EGM universities through transnational activities.

Without downplaying the controversy over the methodology used in this kind of ranking, we believe that they are yet another element reinforcing higher education’s commitment to evaluation and quality and acting as an incentive and stimulus in the quest for excellence in institutions’ teaching and research. In European universities, the political measures aimed at intensifying research competitiveness and restructuring higher education systems have been ratcheted up in recent years. A well-known multinational organisation is the European Union’s Erasmus Mundus Programme,6 a cooperation and mobility initiative that enhances the quality of European higher education and promotes the European Union as a centre of excellence in learning around the world.

The EFQM Model of Excellence was established as a guide for rating the organisations that vied for the European Quality Award created by the European Foundation for Quality Management [EFQM].7 On 29 September 2009, the results of the revision of the EFQM Excellence Model were presented and a new version of the 2010 EFQM Model was previewed at the Annual EFQM Forum in Brussels. This model will coexist with the current model dating from 2003 throughout 2010.

Today EFQM is being accepted as a management model by organisations that are seeking institutional excellence, and it is a benchmark in Europe for excellence as its design encompasses the most up-to-date management practices within an organisation. This model is based on self-assessment and defines the parameters that must be taken into account in order to assess the maturity of the management system within any organisation.

EFQM applies the concept of quality to higher education by defining quality as the degree to which a continuum of differentiating features inherent in higher education fulfils a given need or expectation. Quality is an asset of an institution or programme that fulfils the standards preset by an accreditation agency. In order to be properly measured, this usually involves the evaluation of teaching, learning, management and results.8

The European Association for Quality Assurance in Higher Education [ENQA]9 was established in 2000 to promote European cooperation in the field of quality assurance. The idea for the association originates from the European Pilot Project for Evaluating Quality in Higher Education [1994–5], which demonstrated the value of sharing and developing experiences in the area of quality assurance. Subsequently, the idea was given momentum by the Recommendation of the Council [98/561/EC of 24 September 1998] on European cooperation in quality assurance in higher education and by the Bologna Declaration of 1999. The European Commission has, through grant support, financed the activities of ENQA since the very beginning. The third edition of Standards and Guidelines for Quality Assurance in the European Higher Education Area was published in 2009 [European Association for Quality Assurance in Higher Education – ENQA, 2009].

Likewise, the European Quality Assurance Register for Higher Education [EQAR]10 aims at increasing the transparency of quality assurance and thus enhancing trust and confidence in European higher education. EQAR will list quality assurance agencies that operate in Europe and have proven their credibility and reliability in a review against the European Standards and Guidelines for Quality Assurance [ESG]. It publishes and manages a register of quality assurance agencies that substantially comply with the European Standards and Guidelines for Quality Assurance [ESG] to provide the public with clear and reliable information on quality assurance agencies operating in Europe.

The different European Union member states have their own national and/or regional quality evaluation and accreditation agencies with the goal of contributing to improving the quality of the higher education systems through the evaluation, certification and accreditation of their programmes, faculty, institutions and services, including libraries.11

In a competitive and internationalised context, evaluation processes can spring from a regulatory requirement – either regional or nationwide – or can be conducted on the initiative of the institution itself in an effort to consolidate or accentuate the university’s prestige. The library must have solid grounding in quality management in order to adapt to the demands of the evaluations it might be subjected to in order to support the institutional strategies.

Measurement of Basic Cognitive Abilities

2.2 Studies selection

The following criteria were used for the inclusion of studies in the meta-analysis: [a] assessed any cognitive process using virtual reality and analogous classical or computerized assessment tools of the same cognitive process; [b] provided sufficient data to compute effect sizes; [c] were English-based publications.

The initial search procedure revealed 146 records. Thirty-three additional records were identified through other sources [see Fig. 1]. After removing 16 duplicates, 163 potential abstracts were inspected. We excluded dissertations, publications in other languages than English, and studies that were not focused on virtual reality and neuropsychological assessment. A total of 115 potential articles were analyzed in detail based on their full text. Studies that used computer devices but did not provide full immersion via HMDs or gesture-based video-capture systems have been excluded. Thirteen studies met the inclusion criteria and were included in the meta-analysis.