Introduction

More than three decades after the invention of the Internet, this is the most used communication space in the world today (Internet Live Stats, 2021). While the use of traditional media as an information source has decreased in recent years on a global scale, in its digital versions it has been maintained or it has increased. In fact, both online press and social media consumption has increased substantially in most countries, according to the Digital News Report (Newman et al., 2021). The 2021 version of this Reuters Institute report adds that more than 80% of users read news online and almost three-quarters do so through a smartphone.

Now, according to this same report, if there already was concern about the phenomenon of disinformation, with the appearance of the COVID-19 pandemic, it has grown. 58% of those surveyed are concerned about which news is true and which information is misleading on the internet (Newman et al., 2021).

This has redoubled the need for rigorous and reliable journalism that can both inform and educate, given how permeable we are to misinformation. The same report, but from 2018, indicates that the majority of people believe that the managers of news websites (75%) and social platforms (71%) have the greatest responsibility for solving the problem of fake news (Newman et al., 2018).

Therefore, it is relevant that the websites of the digital press, the so-called cybermedia (Díaz Noci and Salaverría, 2003), apply quality criteria not only in their journalistic content and information standards (Rivas-de-Roca et al., 2020; Romero-Rodríguez et al., 2016), but also in the organization, browsing, visual design, and ease of use of their interface. The reason is that these aspects are decisive both in the credibility of the site and in the visibility of the news.

Faced with this concern, we can find answers in the practice that deals with the evaluation of web quality, which appeared with the emergence of websites and which over the years has become an important and interdisciplinary field of study. Thus, both from the academic and professional spheres, numerous proposals have been generated for the analysis of websites, expressed in scientific publications and documents of a more technical nature oriented towards the industry. For example, Morales-Vargas et al. (2020) and Rekik et al. (2018) present reviews with more than 700 and 500 of these works on web quality, respectively.

The first studies emerged to evaluate usability or measure effectiveness, efficiency, and satisfaction (Nielsen, 2000) and, more recently, user experience (Garrett, 2011). These consider the application of experimental tests (Rubin and Chisnell, 2008) and quantitative methods from statistics (Sauro, 2010; Sauro and Lewis, 2012) and metrics (Tullis and Albert, 2013). Similarly, there is a second aspect whose methods focus on the analysis of expert professionals. Its reports can take the form of standards (Bevan, 2005), rules (Shneiderman, 2016), guidelines (Leavitt and Shneiderman, 2006), principles (Tognazzini, 2014), heuristics (Nielsen, 2020), or recommendations (Krug, 2014), among others.

On the other hand, these tools can be of general use, that is, they serve to evaluate any type of site (Codina, 2008); or correspond to sectoral analyzes, which are specialized in a specific sector. Such is the case of electronic commerce, education, health, government, tourism, or the media (Pedraza- Jiménez et al., 2016).

Examples of these latest web quality assessment instruments focused on cybermedia are found in pioneering works such as those of Abdullah and Wei (2008), based on usability heuristics;

Zambarbieri et al. (2008), which applies the eye-tracking technique; Rodríguez-Martínez et al. (2010), expert analysis aimed at analyzing the adoption of Web 2.0; Al-Radaideh et al. (2011), supported by the study of users; Chung et al. (2012), based on surveys; or the content analysis protocol proposed by Martins (2012).

This research aims precisely to examine this type of study. It seeks, in a general way, to characterize the literature on the evaluation of specialized web quality in the media sector or that related to the digital press. This, to identify their different methodologies and instruments and knowing their main authors.

And in a particular way, this work pays special attention to the web quality attributes (Hasan and Abuelrub, 2011; Olsina et al., 2006) that these tools consider and the degree of automation with which they are applied. Also if they consider among their parameters and specific indicators references to international technology and communication policies, such as the Unesco internet universality indicators (2019), the general data protection regulation of the European Union (2016), and other guidelines of informative quality.

Furthermore, it is proposed to know if these instruments take into account ethical and social aspects where the web interface has implications, such as: clickbait or misleading headlines (García Orosa et al., 2017), information privacy, accountability (Mauri-Ríos and Ramon-Vegas, 2015), the separation between advertising and journalistic content (Chung et al., 2012), sale of sponsored links (Lopezosa et al., 2019), declaration of funding sources, journalists' signature (The Trust Project, 2020), copyright, and child protection (García de Torres and Farmer, 2017), among others.

All these issues become highly relevant at this time when there is a loss of credibility in internet content due to the proliferation of hoaxes, fake news, and information disorders (Guallar et al., 2020), accentuated in times of the coronavirus (Salaverría et al., 2020). And at a time when the media industry goes through a financing crisis in search of new sources of income (Carvajal and Valero Pastor, 2018). It is in this context where the evaluation of web quality in cybermedia emerges as a discipline that can provide a set of highly useful tools. Under this hypothesis, this work presents a review and characterization of the existing studies, the instruments that they propose, and the main indicators that they contemplate.

Methodology

To fulfill the described objective, the principles of the bibliographic review with a systematic approach were used (Booth et al., 2016). Specifically, to form the document bank, a review carried out with academic databases and systematic mapping was used (Gough et al., 2017). And as a general protocol for the analysis, the SALSA framework was applied (Codina, 2018; Grant and Booth, 2009).

In the first place, the quality evaluation of digital media websites was defined as the object of study. Regarding the analysis period, it was between 2000 and 2020. The most commonly used terms in the bibliography, both in English and Spanish, were chosen as the search equation (Table 1).

These were applied to the title, abstract, and keyword fields of the main collection of Web of Science (WoS) and Scopus, the most important multidisciplinary academic databases. Others, specialized in the field of communication and information sciences, were also consulted, such as the LISTA (Library, Information Science & Technology Abstracts) collection of EBSCO Host or Emerald

Insight, together with the information resources Dialnet Plus, SciELO, Microsoft Academic, and Dimensions.

Due to its wide coverage (Martín-Martín et al., 2018), and to include not only journal articles, but also books and other technical documents (Thelwall and Kousha, 2015) –highly demanded in the professional field of web development and design–, the search engine Google Scholar was also used. From there, the citations received for each text as of August 2021 were also extracted.

As inclusion criteria, documents on web quality in cybermedia published since 2000 were considered. Studies based only on visit statistics or web analytics, documents referring only to Apps, and research focused only on the effects or the psychology of the user were excluded.

In the evaluation stage, the 532 retrieved papers were manually reviewed, examining their titles and abstracts, to verify if they met these criteria. With this, the high documentary noise detected was reduced and false positives were ruled out. Next, in the analysis phase, a bank of 83 documents was created, the information of which was supplemented with specific indicators on impact and affiliation –the latter, collected from the academic unit declared by each author–, and it was systematized into three areas referring to data from the publication, authors, and content (Table 2), based on the proposal for the analysis of scientific production in digital media by Abadal and Guallar (2018).

With this information, a comparative analysis of the different disciplines or knowledge areas that address the subject was carried out, based on sizing the presence of the communication faculties or departments. Among the works referring to quality in the media sector, 22 texts were identified as a proposal of a general evaluation protocol for cybermedia. In these, their methodology and application method were also analyzed, and the indicators considered as web quality attributes were systematized. Finally, citations to international technology and communication standards and policies were sought, as well as the presence or not in the analysis criteria of aspects of ethical, deontological, and social involvement described in the introduction.

Results and discussion

Among the main findings, it is detected that the scientific production on web quality specific to the media sector or dedicated to the digital press has shown moderate but constant growth in the last two decades. A greater concentration is observed from 2009 and 2012 stands out with ten publications.

The total number of works belongs to the academic field and, regarding the language of publication, the bank of documents is made up of 50 in Spanish, 31 in English, and two in Portuguese. Regarding the format, articles from scientific journals are by far the most predominant, except for six conference proceedings, five reports, and four doctoral theses.

According to the analysis of the declared affiliation, it was identified that 53% of the texts dedicated to cybermedia were generated in Spanish universities. Seven were generated in the United States, followed by the United Kingdom, Canada, Brazil, and Venezuela, with three each. Papers from Austria, Ecuador, Italy, Malaysia, and eight other countries were also found.

Through the systematization of the names of schools, faculties, or departments, it was possible to establish that, of the total of articles reviewed on web quality in cybermedia, more than half come from the domain of communication (58%). It is followed by business –economics, administration, marketing, and commerce– with 15%, computer sciences –computer science and programming– with 13%, library and documentation –information sciences– with 12%, and 2% that comes from other disciplines.

Of the total of works analyzed on web quality in digital media, ten correspond to review articles of theoretical or methodological aspects about the analysis of cybermedia, and nine are case studies applied to the internet press. In turn, 22 of the texts propose a protocol specialized in the evaluation of digital media websites, with a methodology and a set of explained indicators, to facilitate its application. Finally, 40 papers present research focused on the analysis of a particular parameter, such as participation, usability, or the use of multimedia.

Methods and tools

The dichotomy between expert analysis and user experience studies (UX) is evident in the research examined on web quality in cybermedia. When analyzing their application methods (Chart 1), it is detected that more than half corresponds to the first group and they propose inspections to be applied by specialized researchers or professionals with experience in website development (Pribeanu, 2009; Usability.gov, 2013).

Table 3: Cybermedia analysis protocols with the most citations in Google Scholar (GS).

Authors and year	Title	GS
Rodríguez-Martínez et al. (2010)	Cibermedios y web 2.0: modelo de análisis y resultados de aplicación	193
Rodríguez-Martínez et al. (2012)	Indicadores para la evaluación de la calidad en cibermedios: análisis de la interacción y de la adopción de la Web 2.0	139
Fondevila Gascón (2014)	El uso de hipertexto, multimedia e interactividad en periodismo digital: propuesta metodológica de ranking de calidad	76
Romero-Rodríguez et al. (2016)	Dimensiones e indicadores de la calidad informativa en los medios digitales	54
Aranyi and van Schaik (2016)	Testing a model of user-experience with news websites	52
Codina et al. (2014)	Sistema Articulado de Análisis de Cibermedios (SAAC): Una propuesta sobre el qué y el cómo para estudiar medios de comunicación digitales	44
García-Carretero, Codina, Díaz-Noci, et al. (2016)	Herramientas e indicadores SEO: características y aplicación para análisis de cibermedios	33
Aguirre Mayorga and Bernal Suárez, (2014)	Contenidos periodísticos digitales: hacia un modelo de medición de calidad	26
Sánchez-González and Alonso (2012)	Propuesta metodológica para el análisis de las tecnologías de participación en cibermedios	26
Al-Radaideh et al., (2011)	Usability evaluation of online news websites: a user perspective approach	24
Linares et al. (2016)	Interactividad, buscabilidad y visibilidad en cibermedios: sistema de análisis y resultados	15
Cristóbal Fransi et al. (2017)	Critical factors in the evaluation of online media: creation and implementation of a measurement scale (e-SQ-Media)	11
Limia Fernández et al. (2013)	Interactividad y participación en los cibermedios: una propuesta metodológica para la elaboración, registro y análisis de datos	11
Linares et al. (2015)	Interactividad en cibermedios: Propuesta nuclear de protocolo de análisis	11
García-Carretero, Codina, and Pedraza- Jiménez (2016)	Indicadores para el estudio de la visibilidad y del impacto de los cibermedios en el ecosistema digital: mapeo y caracterización de herramientas de análisis SEO online	10
Sanabre Vives (2015)	Un modelo para el análisis y concepción de sitios web: El WebSite Canvas Model aplicado a Eldiario.es	6
Hernández Soriano (2015)	E-SQ-Media, creación e implementación de una escala de medición de la calidad percibida en cibermedios	4

Among the most used techniques and tools (Table 4) in this method, we find heuristic evaluation, content analysis, checklists or questions, indexes, and articulated systems, among others. On the other hand, the detected users’ studies are mainly based on surveys, satisfaction questionnaires, usability tests, experiments, interviews, A/B tests, task monitoring, and techniques such as the think- aloud in which people are “thinking out loud” and narrating their impressions while browsing a website and using its features.

Mixed techniques that combine usability tests with perception studies are also frequently used. In these, questionnaires are applied where users later respond about their assessment of the effectiveness, efficiency, and satisfaction with the media on the Internet, and then weigh them through mathematical models with multiple factors, such as the one formulated by Cristóbal-Fransi et al. (2017).

Only four of the studies are based on direct user experimentation: Zambarbieri et al. (2008) use the eye-tracking method in reading applied to Italian newspapers; Aranyi et al., (2012) base their work on psychometric tests on news sites; while Yu and Kong (2016) perform A/B tests on web browsing on small screens.

For their part, O’Brien and Lebow (2013), propose a combination of observation and response to stimuli techniques. Methods considered included psychometric scales –user participation, cognitive absorption, system usability scales–, self-reported interest in news content, and performance metrics –reading time, browsing time, number of pages visited, and use of recommended links–; and the physiological reactions –heart rate, electrodermal activity, and electrocytogram– recorded during the interaction.

It is also worth mentioning the presence of three works on automated inspections using computational or artificial intelligence methods. Di Massa et al. (2010) use natural language processing to generate a news recommendation system based on user interest models and multimodal content analysis.

Table 4: Methodological tools present in cybermedia web quality works

Techniques		Instruments
Heuristic evaluation	27	Check-up questions	9
Survey	11	Satisfaction questionnaire	5
Usability testing	7	Scoring scale	4
Content analysis	5	Index	4
Interview	5	Psychometric scale	3
Think aloud	3	A/B test	2
Natural Language Processing (NLP)	2	Scales	1
Correspondence analysis	1	Ejectracking	1
Supervised machine learning	1	Heuristic	1
Deep learning	1	SEO indexes	1
Design thinking	1	Click count	1
User-centered design	1	Time per task	1
Experimental stimuli	1
Mixed technique	1
Models		Software
Articulated systems	11	Alexa	14
SAAC	6	Moz	4
e-SQ-Media	3	Majestic	3
WCAG 2.0	3	Ahrefs	2
eWOM	1	SEMrush	2
WebSite Canvas Model	1	eXaminator	1
		Similar Web	1
		Sistrix	1
		WebXACT	1

Likewise, Omidvar et al. (2020) use deep learning to determine the quality of the headlines of the published news through the click count and the length of stay of the visitors, to predict the headlines that best invite reading. On the other hand, Scharkow (2013) analyzes the content of German cybermedia through machine learning techniques, specifically supervised machine learning algorithms.

Finally, the use of instruments to measure specific attributes stands out, such as the WCAG 2.0 or WebXACT guides to evaluate web accessibility. Also, specialized software, such as Alexa and Similar Web to measure web traffic, or Ahrefs, Majestic SEO, Moz, SEMrush, or Sistrix, to measure the visibility and web positioning of cybermedia (García-Carretero, Codina, and Pedraza-Jiménez, 2016).

Parameters and indicators

A factor in common in almost all the publications that propose web quality assessment protocols in digital media is that they are organized into parameters (Table 5) and indicators. The first are conceptual characteristics and answer the question about what we want to study. The second, how we are going to do the study. They are the elements of analysis and allow the parameters to be “operationalized”. Their presence can be verified through an inspection using checklists or check questions (Codina et al., 2014).

Table 5: Web quality parameters in the most cited cybermedia evaluation protocols

Work	Parameters
Rodríguez-Martínez et al. (2010)	Accessibility; Visibility and popularity; Access to information; Deepening of information; Interaction tools; Personalization of information; Web 2.0 tools; Dissemination on social networks
Rodríguez-Martínez et al. (2012)	Interaction; Participation; Access to content; Personalization; Versions; Web 2.0 tools; Web 2.0 platforms
Fondevila Gascón (2014)	Hypertextuality; Text; Photography; Video; Audio; Infographic; Comments on the forum; Online interviews
Romero-Rodríguez et al. (2016)	Business areas; Socio-labor areas of the workers; Information content and final product areas
Aranyi and van Schaik (2016)	Impression; Content; Design; Information architecture; Distraction
García-Carretero, Codina, Díaz-Noci et al. (2016)	Authority (Moz); Trust Flow (Majestic); Citation Flow (Majestic); Global Rank (Ahrefs); URL Rating (Ahrefs); Domain Rating (Ahrefs); Global Rank (Alexa); Local Rank (Alexa)
Aguirre Mayorga and Bernal Suárez (2014)	Informative agenda; Depth in the management of information and documentation; Diversification in the use of journalistic genres; Multimedia; Interactivity; Hypertextuality
Al-Radaideh et al., (2011)	Usability; Content; Web design
Limia Fernández et al. (2013)	Personalization; Web 1.0; Web 2.0; Social networks
Linares et al. (2015)	Cybermedia-user relationship; User-generated content; Searchability and Browsing
Hernández Soriano (2015); Cristóbal Fransi et al. (2017)	Efficiency; System availability; Reliability and privacy; Interaction; Perceived quality; User satisfaction; User loyalty
Sanabre Vives (2015)	Technology partners; Content partners; Key Activities; Key resources; Value proposal; Customer segments; Relationship with customers; Channel; Income stream; Cost structure; Benchmarking; Competitor weaknesses; Social, local,    and mobile

Among more than 170 different parameters and indicators present in the examined studies (Chart 2), some attract special attention among the authors, who formulate specific protocols for their analysis. The most studied is that of interactivity and interaction (Cebrián-Herreros, 2009; Linares et al., 2015) as a determining factor in the quality of digital media (Hernández Soriano and Cristóbal Fransi, 2016). Besides the much-cited system to measure the adoption of Web 2.0 (Rodríguez-Martínez et al., 2012), Linares et al. (2015) propose another one based on the cybermedia-user relationship, the content generated by the latter, and searchability and browsing.

Closely linked to the previous one, is the participation indicator (Eberwein, 2019; Ksiazek, 2018), where we also find proposals for its study in particular: Sánchez-González and Alonso (2012) focus on the analysis of technologies, while Limia Fernández et al. (2013) develop a method for the preparation, recording, and analysis of data on the processes of interaction and participation in digital media.

Usability (Bevan, 2005; ISO, 2018; Nielsen, 2000) as a measure of effectiveness, efficiency, and user satisfaction is also widely addressed, both on digital newspaper websites (Al-Radaideh et al., 2011; Mariage and Vanderdonckt, 2001; Rosa and Veras, 2013) and on mobile devices (Jeong and Jung Han, 2012; Jiménez Iglesias, 2018). The same occurs with universal access or accessibility (Salas Hernández et al., 2018) and personalization.

Similarly, user experience (Garrett, 2011) has specific works such as those by Flavián and Gurrea (2006a, 2006b, 2007), who develop a line of research on the influence of web quality attributes on readers when choosing a newspaper on the internet. Rabaya Toma et al. (2018), in turn, explore reading satisfaction and its impact on the subsequent intention of users to revisit the site or recommend it.

Another analyzed parameter is the information architecture (Rosenfeld et al., 2015), with a large number of mentions of indicators such as browsing, the organization of content, and access to information, as well as the graphic design of the interface itself. However, no specific protocols are detected for its analysis.

And, of course, the content also occupies a prominent place, with the depth of the information, the reliability, and the credibility –detailed in the next section–. Also with attributes typical of digital media, such as multimedia (Guallar et al., 2010) and hypertextuality (Fondevila Gascón, 2014; Masip et al., 2010), including special indicators such as interactive infographics (Langer and Zeiller, 2017; Túñez López and Nogueira, 2017), live video transmission through social platforms (Apablaza-Campos et al., 2020; Apablaza-Campos and Codina, 2018) and the analysis of temporality (Freixa, 2020).

On the other hand, it is also worth mentioning the impact indicators, such as the search experience (Lopezosa, Iglesias-García, et al., 2020); web positioning (Lopezosa, Codina, et al., 2020; Lopezosa

and Codina, 2018); traffic (Jowkar and Didegah, 2010); and visibility on the internet (García- Carretero, Codina, and Pedraza-Jiménez, 2016; García-Carretero, Codina, Díaz-Noci, et al., 2016; Linares, Codina, Vàllez, et al., 2016), where the authors carry out a mapping and characterization of search engine optimization or SEO tools and indicators. Along the same line, Lopezosa, García- Carretero, et al. (2020) present the system for analyzing visibility in cybermedia called AVCIS, while García-Carretero et al. (2020) propose a specialized methodology for native digital media.

The strategic or performance factor in front of the audience is also addressed by Sanabre Vives (2015) in the WebSite Canvas Model with which he evaluates the Spanish media outlet Eldiario.es as a case study. Finally, the framework proposed by Serm et al. (2006), which, in a pioneering way, addresses the tasks to develop and evaluate cybermedia on mobile devices is unique.

Conclusions

Coincidentally with the works of Rekik et al. (2018) and Morales-Vargas et al. (2020), it is detected that studies on web quality in the media sector show a growing trend during the last twenty years. Especially in the discipline of communication, they are awakening interest due to their direct relationship with the sustainability of the media, although they share space with other areas of knowledge Ugras et al. (2016).

In this research, more than eighty papers have been analyzed, and about twenty that propose specific protocols to examine the quality of digital media websites have been detected. The methodologies most used in them are expert analysis through heuristic evaluation and checklists –with some authors emphasizing the need for these lists not to be limited only to questions, but to incorporate explicit expert evaluation procedures–. In contrast, user studies and computerized or automated procedures are much less frequent. This contrasts with other sectors (Allison et al., 2019) or the professional field (Rosala and Krause, 2020), where there is a greater presence of participatory and experimental methods.

As Abadal and Guallar (2018) also point out, the most studied parameter is interactivity, especially through social web tools and from user-generated content. It is followed by usability and user experience, with indicators such as accessibility, personalization, ease of use, effectiveness, efficiency, and satisfaction.

Information architecture is also subject to analysis, with browsing, organization, and access to information, as well as interface design, among many other attributes. In this regard, no specific instruments are detected for its study, as they do exist for the analysis of strategic factors and visibility or search engine positioning.

And of course, the content also ranks high. For their evaluation, the specialized instruments consider the analysis of indicators such as the depth of the information, the reliability, the multimedia, and the hypertextuality.

However, these include very few references to trust attributes, journalistic rigor, or ethical or deontological issues, and in no case are international communication and technology policies cited. This fact draws attention especially if the last years are taken into account, due to recent phenomena such as disinformation and the fact that the media industry, and in particular the press, faces a financing and credibility crisis on a global scale.

This relative lack clearly indicates a research gap. This is one of the functions of systematized bibliographic reviews as a methodological tool –like the one applied in this work–, besides presenting the most significant features of the studied subject.

In any case, the quality of digital media websites and their sectoral analysis instruments are clearly an emerging field of study, which is becoming increasingly specialized and diverse. One likely reason is the new business models of the media, once the continued decline in the sale of printed copies and the decline in advertising revenue has been noted. These two trends make studies on the digital media of cybermedia increasingly important.

Thus, in this research, besides providing a vision on the most significant aspects of web quality applied to the media –which may be useful to other researchers and press professionals–, it has been detected that in the field of cybermedia evaluation there is an interesting and wide opportunity for future work. Even more so in these times of fake news, where the need to develop comprehensive instruments that not only consider technical and strategic indicators but also trust attributes is reinforced.