Differences between the genders in ICT skills for Finnish upper comprehensive school students: Does gender matter?

Kaarakainen Meri-Tuulia

E-mail: merluo@utu.fi

 

Kivinen Antero

E-mail: antkiv@utu.fi

 

Kaarakainen Suvi-Sadetta

E-mail: susaka@utu.fi

 

Research Unit for Sociology of Education RUSE, University of Turku

FIN-20014 University of Turku

Abstract

This study examined the ICT skills of Finnish upper comprehensive school students. The data has been collected from 65 municipalities around Finland for 5455 ninth graders (mean age 15.24). ICT skills were measured using a digital, performance-based ICT skill test. The test was based on the revised Finnish national core curriculum for basic education. Based on the results, there was a small, but statistically, significant difference between the genders in the total scores on the ICT skill test. More consequential differences between the genders were found in the item level analysis. As explicit item level analysis indicated, boys tended to get better scores from more technical-oriented items, while girls got better scores from school work-oriented and social interaction-related items. The results emphasize that gender differences in ICT skills are more item specific than general. More importantly, the variation between individuals in ICT skill items was extensive and in all likelihood more influential than the gender difference as such.

Keywords: ICT skills, Genders differences, Upper comprehensive school students

1 Introduction

Ongoing digitalisation is changing every aspect of society. The growing amount of (digital) data, digital infrastructures, online artefacts and transaction systems, among numerous other things, are changing the way that people study, work, communicate, and live their everyday lives. Todays Western societies are described as digital societies, where new information and communication technologies shape the landscape of everyday living. This digital citizenship demands new kinds of literacies and digital competencies for one to be able to operate and participate in society (Gallardo-Echenique et al., 2015).

 

There is continuing debate around what the essential skills in digital societies are. The DIGCOMP framework project uses the concept of digital competence and categorises these crucial competences for information, communication, content-creation, safety and problem-solving (Ferrari, 2013). In turn Binkley et al. (2012) use the 21st century skills concept to describe the ways of thinking, working, and living in a digitalised world. Fraillon et al. (2013) apply the construct of computer and information literacy that relates to an individuals ability to use technologies to investigate, create, and communicate and participate effectively at home or school and in the workplace or society.

 

The purpose of this study is to examine the differences in information and communication technology (ICT) skills between the genders in the context of the Finnish national core curriculum for basic education. The intent of the core curriculum is to provide a common ground for the local curricula and promote equality and equity in education. It contains the guidelines for providing education as well as the objectives and key content of instruction. In the core curriculum, ICT skills are one of the seven transversal competences which are integrated into all subjects. ICT skills are considered to be an essential aspect of civic competencies and are seen both as a goal and a tool for learning. In practice, the goal is to offer every student the following skills: Understanding of the basic operations and concepts of ICT, knowledge to use ICT in a responsible, safe, and ergonomic manner; the skills to use ICT as a tool for information management and creative work; and the abilities to use ICT in interaction and networking. Overall, the aim is to offer the experiences in using different applications to understand their meanings for everyday life, communication and public influence. (FNBE, 2016.) In this study, we are particularly interested in the learning goals for actual ICT skills, instead of studying ICT as a tool for learning.

 

The research questions are: 1) Are there differences between the genders in ICT skills between upper comprehensive school students? 2) What kinds of differences exist between the genders?

2 ICT skills

2.1 Concepts

Various concepts are used to define the skills required for ICT use. Concurrently with digitalisation, terms like IT, ICT, and computer literacy have become dominant. (van Laar et al., 2017; Bawden, 2008.) In most cases, these concepts consist of a domain part (computer, ICT, Internet, etc.) and a knowledge perspective (competence, literacy, skill, etc.) (Hatlevik et al., 2015). In their systematic literature review on the different perceptions of 21st century skills, van Laar et al. (2017) made a distinction between 21st century skills and digital skills. Based on their results, 21st century skills are broader than digital skills and are not necessarily underpinned by ICT, while digital skills involve that connection. Based on the 75 reviewed articles, the authors created a framework of seven core skills and five contextual skills. The seven core skills are: Technical usage, information management, communication, collaboration, creativity, critical-thinking, and problem-solving. The five contextual skills are: Ethical awareness, cultural awareness, flexibility, self-direction, and lifelong learning.

 

In the definition offered by van Laar et al. (2017), technical usage is seen as the skill to use devices and applications. Information management skills are used to search and use information, including knowing what sources of information to use. Communication skills are used to transmit information to others. Collaboration skills are used to create social networks and be able to work as a team using ICT. Creativity skills are used to create new ideas or redefine old ones. Critical thinking skills are used to evaluate gathered information and communicate with others. Problem-solving skills are used both for finding information to solve problems and using ICT to solve those problems.

 

In spite of the diversity in the concepts, ICT skills or competences are considered essential skills for social interaction, civic participation, information retrieval and processing, academic performance, and professional success (see e.g. Hoffman & Schechter, 2016; Pagani et al., 2016; Zhong, 2011). Consequently, ICT skills should be considered necessary educational outcomes for every student in digitalised societies, and thus, a valid assessment of ICT skills is crucial (Aesaert & van Braak, 2015). In this study, the term ICT skills refers to the digital skills or competencies in line with the definition of the seven core skills offered by van Laar et al. (2017) and the learning goals for ICT skills found in the Finnish national core curriculum (FNBE, 2016).

2.2 Measuring ICT skills

In her review, Litt (2013) classified the ICT skill assessment methods as: 1) survey and self-reported measures; 2) performance/observation measures; and 3) combined and unique assessments. Surveys and self-reports seem to be the dominant methods for ICT skill assessment. In these assessments, participants respond to a question or a set of questions about their own competence level or evaluate their ability to perform specific tasks on the Internet (e.g. Heerwegh et al., 2016; Livingstone & Helsper, 2010). Qualitative studies prefer observation-based measures or interviews, which incorporate ethnographic practices. These types of studies focus on, for example, observing a persons actions during information search tasks (see e.g. Hargittai, 2002). Interviews in their turn typically utilise open-ended questions like What are you strong in? (Litt, 2013). Some performance-based studies have conducted task-based assessments and virtual environments to mirror the actual environment where the ICT skills are being used (e.g. Claro et al., 2012; van Deursen & van Dijk, 2009).

 

In their review of assessment instruments Siddiq et al. (2016) concluded that the majority of the utilised assessment instruments are based on self-reporting. Most were online tests with multiple-choice questions. Some also included questions in a dynamic format that required participants to interact with the tasks. A few had a dynamic task design that required interaction with the test environment. Most of the tests were evaluated using quantitative methodology. Siddiq et al. (2016) summarises by saying that the majority of these tests assess information search, retrieval, or evaluation, and technical skills; however, aspects like problem-solving with ICT, digital communication, and online collaboration are not covered to satisfaction.

 

It is well known that measurements based on participants own evaluations face validity problems, as people tend to overrate or underrate their own levels of competence (e.g. van Deursen et al., 2016; Aesaert & van Braak, 2015; Litt, 2013). There is, as Litt (2013) and van Deursen et al. (2016) emphasise, an urgent need for more accurate, nuanced, and reliable assessment instruments for greater generalisable and diverse samples that can capture the phenomenon in its entirety.

2.3 Gender and ICT Skills

In their longitudinal cross-sectional study of representative samples from the Dutch population on Internet skills, van Deursen and van Dijk (2015) noticed men scoring higher than women on all skill domains: Operational skills (e.g., saving files, downloading programs, using the refresh button); formal skills (e.g., being familiar with web site structure); information skills (e.g., finding information, using search booleans); and strategic skills (e.g., gaining financial benefits, and making decisions based on retrieved information). To the contrary, among students, previous research results on gender differences in ICT skills have been quite diverse.

 

In their study, Aesaert and von Braak (2015) noticed that among primary schoolers, girls perform better in tasks that relate to communication-oriented activities and deliver digital information with understandable content to the receiver. In the study by Ritzhaupt et al. (2013), middle school girls were found to be outperforming boys in all technology literacy domains. The five examined domains were: Technology operations and concepts (e.g., creating new files, locating and opening files, selecting the best device to complete a given task); constructing and demonstrating knowledge (e.g., selecting the correct printer, setting page margins within a word processing document, editing images); communication and collaboration (e.g., practical keyboard skills, using e-mail, and creating new slides within presentation software); independent learning (e.g., using print preview, deleting data in a spreadsheet, creating concept maps as a learning strategy); and digital citizenship (e.g., maintaining password security, identifying security risks, displaying an awareness of potentially inappropriate media use). Lau and Yuen (2014) found that female students in junior secondary school perceived their Internet literacy (e.g., searching information, using instant messaging, downloading files) and computer literacy (e.g., setting headers in word processing software, plotting a graph with a spreadsheet, editing a photo with image processing software) to be higher than that for male students.

 

In an earlier study, using the previous version of the ICT skill test used in this study, among Finnish upper comprehensive and secondary school level students, Kaarakainen et al. (2017) found that male students outperformed female students on tasks that required advanced information technology skills (e.g., tasks related to software installation and initialisation, information networks, server environments, programming, and database operations), while on tasks related to basic digital skills (e.g., basic operations, information seeking, word or image processing, social networking) gender differences were not significant. In the case of technology self-efficacy, males/boys often reported higher ratings than females/girls (see e.g., Huffman et al., 2013). For example, in their study on German secondary schoolers, Christoph et al. (2015) found boys self-ratings (e.g., I am able to install new programs...) and theoretical computer skills (e.g., terminological knowledge of basic computer-concepts, such as JAVA, IP-address or FTP) to be higher than those for girls, while no gender differences could be found in basic computer skills (e.g., tasks related to digital contexts within web-browsers, text processing, and e-mailing).

 

It can thus be summarised that the question of gender difference in ICT skills is complex and still largely unclear. A notable problem is the variation in the used concepts and the diverse concept operationalisations. As Ritzhaupt et al. (2013) speculated, even though girls are successful in their studies, boys may still be more proficient in other ICT-related tasks not covered in their tests. Thus, more research is needed to identify the actual dimensions of the potential gender differences within ICT skills. This current study particularly aims to clarify this issue.

3 Methods

3.1 Participants

The data was collected in Finland at the beginning of year 2017 (JanuaryMarch) as a project financed by the Finnish Prime Ministers Office (funding for Governments analysis, assessment and research activities). The participants were from 149 upper comprehensive level schools (grade 9/9) in 65 municipalities around the country, chosen based on a geographically representative sample of Finnish municipalities, as formed by the Finnish Education Evaluation Centre. Altogether, 5455 9th-graders ages 15 to 17 years were tested, and of that group 47% were boys and 53% were girls. The mean age for the participants was 15.24.

3.2 Measurement

ICT skills were measured using the online test developed in the Research Unit for the Sociology of Education (RUSE) at the University of Turku (Finland). The original ICT skill test was developed in 2013, and it was completely revised during year 2016 when the Finnish national core curriculum for basic education was renewed (2014) and implemented in schools in August 2016 (Kaarakainen et al., 2017; FNBE, 2016). The test is bilingual because both Finnish and Swedish are official languages in Finland. The software used for testing is a web application, written in PHP and Javascript and using the TinyMVC- and Bootstrap-frameworks. The application is supported by PostgreSQL database software for all data storage needs, and the test content (tasks and the specific questionnaires in each study) is included in the system as easily changeable XML-files.

 

The test consists of 18 items divided into 6 modules based on item topics (Appendix 1). The goal was to form coherent modules for a comfortable user experience. Test items were implemented in such a way that the user interface and graphics were intended to simulate common ICT applications and hence mirror real-life settings. The tested competency areas (18 items) were chosen based on the renewed Finnish national core curriculum (FNBE, 2016). The last module (requirements for the ICT study programs) was broadly based on the curricula for the ICT field of Finnish vocational institutions and the universities of applied sciences. The participants could achieve 2 points for each item, which could result in a maximum total score of 36.

 

In the ICT skill test, each item consists of multiple subtasks (16) and/or chains of actions, in which every action (selection or operation) is linked to the previous one; together they form a coherent item. In the items, a combination of close-ended questions (conventional multiple-choice, true-false multiple-choice, multiple true-false multiple-choice, and matching) (see Haladyna, 2011) and open-ended questions or questions requiring participants to interact with the test environment (input the right values or select and click the right function icons) were applied (see Siddiq et al., 2017). The majority of items can be seen as context-dependent item sets (see Haladyna, 2011), as they consist of a problem scenario that participants have to solve by choosing right actions from given options beside the progressive storyline. Some items can be seen as constructed-response questions that require the test-takers to construct or develop their own answers (Miller, 2015). In the ICT skill test, items are assessed automatically based on specified options and actions or simple text mining algorithms.

3.3 Analysis

The reliability (internal consistency) of the ICT skill test was estimated using Cronbachs alpha. Reliability refers to the extent to which a test is a consistent measure of a specific concept. It is known that scales with only two or three items tend to exhibit smaller alphas than do those with more items (Peterson, 1994). As the items of the ICT skill test can consist of only one multi-phase interactive task or just two or three subtasks, other item measures are utilised instead of alpha values for item level analysis.

 

The corrected item-total correlation is widely used to examine if any item fails to correlate with a total score. The cut-off value of .3 means that items that have item-total correlation below the threshold value are likely to be extremely easy or difficult or ambiguous, or otherwise that this item is not measuring the same construct being measured by the other items. (Nunnally & Bernstein, 1994.)

 

To analyse item difficulty, an item difficulty index was used instead of the proportion of right answers (traditional item difficulty), as that choice was better suited for complicated items (Tiruneh et al. 2017). Here the interest is not simply on how many test-takers get the item (completely) right. The formula used to compute the item difficulty index (P) was:

 

Difficulty index, P =

 

where ~fX is the total number of scores earned by all test-takers on an item, n is the number of test-takers, Xmin is the smallest item score possible, and Xmax is the highest item score possible.

 

Item discrimination is another basic measure of the validity of an item. It is defined as the ability of an item to discriminate (or differentiate) between high and low achievers. The formula used to compute the discrimination index (D) is:

 

Discrimination index, D =

 

whereandare the difficulty indexes for the highest performing (U) and lowest performing (L) groups. The widely used threshold of 27% was used to divide these upper and lower groups. (Adams & Wiemanc, 2011.)

 

In order to respond to the research questions (Are there differences between the genders? And what kind of differences are these?), a two-tailed independent samples t-test was used to analyse the statistical differences between the genders. The t-test was used to compare the sample means from two independent groups for an at least interval-scale data when the distribution was approximately normal (see e.g., Warner, 2013). Analysis of the data was first executed using total scores of the ICT skill test, and then on each item separately in order to examine the gender differences in overall performance and also for each item one by one.

4 Results

The Cronbachs alpha for the ICT skill test (all 18 items) was .86, which clearly exceeded the threshold of .7 (Nunnally & Bernstein, 1994). Results of item-level analysis are presented in Appendix 1. As that appendix indicates, the item difficulty indexes varied from .01 to .61. Low- item difficulty indexes for the last four items indicated that all programming- related items were extremely difficult for the majority of 9th-graders. Even elementary programming, which actually does not require any programming skills, but only the ability to follow and give orders was based on the given instructions. The P-values for all other items lay within the range of ~.2 and ~.6, indicating that none of these items was either too difficult (<.2) or too easy (>.8) for the participants to complete.

 

The item-total correlation varied between .22 and .65. The items related to programming, which were among the most difficult items, also had the lowest item-total correlation values. The item discrimination indexes ranged between .34 and .92, and thus were all acceptable (>.3). These values indicated that all items on the ICT skill test were able to distinguish between participants who mastered the skills required for a particular item and those who did not yet have those skills.

 

 

 

Table 1. The ICT skill test scores by items and gender differences for the students.

 

 


Altogether


Girls


Boys

The Independent Samples t-test

Item

M

SD

M

SD

M

SD

t-value

p-value

Information seeking

1.21

.45

1.29

.41

1.11

.47

15.086

.000***

Word processing

1.02

.82

1.18

.80

.84

.81

15.568

.000***

Software installation and updating

.93

.69

.90

.67

.95

.72

-2.427

.015*

Digital communication

.86

.62

.92

.60

.79

.63

7.961

.000***

Video and audio processing

.84

.68

.88

.67

.81

.70

3.770

.000***

Information security

.81

.58

.83

.57

.80

.59

2.182

.029*

Cloud services and publishing

.81

.69

.85

.68

.77

.69

4.142

.000***

Social networking

.77

.51

.80

.48

.74

.53

4.698

.000***

Image processing

.65

.57

.65

.43

.64

.49

.503

.612

Presentations

.56

.67

.63

.68

.48

.65

8.642

.000***

Spreadsheets

.51

.63

.57

.63

.45

.62

6.956

.000***

Software purchasing

.38

.43

.40

.42

.37

.43

1.920

.055

Basic operations

.36

.52

.23

.37

.52

.61

-21.579

.000***

Information networks

.34

.41

.23

.30

.46

.47

-21.104

.000***

Web programming

.15

.32

.11

.27

.20

.36

-9.556

.000***

Elementary programming

.12

.37

.09

.30

.17

.44

-7.862

.000***

Database operations

.10

.30

.07

.26

.13

.35

-7.486

.000***

Programming

.01

.13

.01

.09

.02

.16

-3.484

.000***

 

*** p < .001, highly significant

* p < .05, marginally significant

 

Table 1 presents the ICT skills test scores by items. The table is presented in descending order by the achieved average item scores of the students. The mean score for all test items for all participants was 10.45 (standard deviation 5.32), for girls 10.64 (SD 4.74) and for boys 10.24 (SD 5.92). The difference between the genders was small, but statistically significant (t-value 2.712, p-value .007). The five top scoring items for the students were information seeking (M 1.21), word processing (M 1.02), software installation and updates (M .93), digital communication (M .86), and video and audio processing (M .84). Conversely, the five lowest scoring items for the students were information networks (M .34), web programming (M .15), elementary programming (M .12), database operations (M .10) and programming (M .01).

 

There were only two items where the differences between the genders were not statistically significant, namely, image processing and software purchasing. In the case of software installation and updates, and information security, the difference between girls and boys was only marginally significant (p < .05). In the case of all the other items, the differences between the genders were highly significant (p < .001).

 

The girls significantly outperformed the boys in information seeking, word processing, digital communication, video and audio processing, information security, cloud services and publishing, social networking, presentations, and spreadsheets. The boys in their turn outperformed the girls in software installation and updating, basic operations, information networks, web programming, elementary programming, database operations, and programming. The widest gender differences in item mean scores were found in word processing (for the girls, with a difference of .34 points), basic operations (for the boys, with a difference of .29 points), information networks (for the boys, with a difference of .23 points), and information seeking (for the girls, with a difference of .18 points).

5 Discussion

In this study, the ICT skill test was introduced, and the results of the reliability and item analysis were presented to assess the quality of the test items and the test as a whole. The results indicated that the ICT skill test was a reliable instrument, and its items were mainly adequate in their difficulty level and had appropriate discrimination power. Both item difficulty indexes and item-total correlation for elementary programming, database operations, web-programming and programming items indicated the need to consider renovation of these items. It is clear that these four programming-related items are challenging for a comprehensive school final graders. However, since August 2016, programming has been integrated into the Finnish national core curriculum. In the upper grades (79/9) programming has been integrated as part of the objectives set for mathematics, while in the lower grades, the fundamentals of programming can be taught in all subjects (FNBE, 2016). Because of this these programming-related items should remain a part of the ICT skills test, as they provide the ability to track whether the desired programming skills will increase in the future or will not.

 

Based on our results, upper comprehensive level students performed best in schoolwork-oriented items, such as information seeking and word processing. Conversely, the lowest scoring items for all participants were technical-oriented items, such as basic operations, information networks, different kinds of programming, and database operations. The discovery that students performed relatively well on items related to word processing, computer security, social networking and communication is encouraging in light of the desire to achieve the learning goals of transversal competences in the Finnish national core curriculum (FNBE, 2016). On the contrary, the noticed lack of basic operational skills and knowledge deficits in information networks, spreadsheets, and software purchasing indicated the need for formal education on the very elementary use and knowledge of computers and mobile devices. It should also be noted that relatively low scores on the basic operations items, which relate to the technical usage of computers, can be explained to some degree by the fact that these items stress the use of computers and keyboards when Finnish youth today are more experienced with Smartphones or other mobile devices (see e.g., Taipale, 2014). For example, the use of shortcut keys (a subtask in the basic operations item) requires computer specific knowledge.

 

The current study answered the question on whether there were differences between the genders in ICT skills. Based on the analysis in this study, there was only a small, but still significant, difference between the genders in the total scores of the ICT skill test. Particularly, this paper sought to identify the actual dimensions of potential gender differences within ICT skills. When examining item level differences, the results are in line with previous studies (e.g., Aesaert & von Braak, 2015; Lau and Yuen, 2014), as girls were found to outperform boys, particularly on items that related to using learning-related software and tools like word processing, spreadsheets, presentation software, image processing, and items related to communication, social networking, and security issues. In contrast, in this study, boys were found to perform significantly better compared to girls on items that required more technical knowledge, like basic operations, information networks, programming, and database operations. This result is also in line with previous studies (Kaarakainen et al., 2017; Christoph et al., 2015). Above all, these discoveries emphasise that gender differences in ICT skills are particularly item specific.

 

Students success was found to be weakest on the last module (requirements of ICT study programs) of the ICT skills test. This is also the module where the boys outperformed the girls by the largest margin. This module contained items related to database operations, web programming, and computer programming, which are common areas of computer science studies in secondary and tertiary level education. It is well known that in the field of computing, a lack of diversity has existed for several decades, and female participation in computing studies has remained low (McGill et al., 2016). However, after more profound consideration, this is the module that distinctly revealed that differences between individuals outweigh the difference between genders, as even among the boys, there was a huge variation in individual performance. It is worth noticing that the standard deviations for the scores on the items in the last module were at least twice as big as the mean. In fact, there were only 19 boys and 3 girls among all the participants, who got at least half of the available points in that module. Therefore there was only a very small group of students who mastered the most technical items in our ICT skills test, and this success related more to individuals than gender as such.

 

Individual technology usage habits and experiences played a notable role in the above-mentioned phenomenon. As Aesaert and von Braak (2015) assumed, the reason behind females success in communication-oriented activities and information delivering can be found in their ICT use and experiences. Similarly, Lau and Yuen (2014) presumed that because female students tend to engage in more learning- and social networking-related activities at home, they achieve skills like information searching, instant messaging, word processing, image editing, etc., which were classified in their study as Internet and computer literacy. When it comes to more technical skills, Huffman et al. (2013) found that gender roles in particular (not just biological sex) play a large role in technology self-efficacy; in particular masculinity is a strong predictor of technology self-efficacy.

 

As van Laar et al. (2017) suggest, ICT skills are essential for performing tasks that are necessary in a broad range of occupations in a digital society. Typically ICT skills delivered in an educational context will have relevance also in 21st century digital skills for work. However, as van Laar et al., (2017) summarised, the changing labour market and job demands in a knowledge society pose serious challenges to educational systems, when they are asked to prepare students for jobs that may not yet even exist. In the Finnish core curriculum for basic education these ICT skills are decentralised under traditional school subjects (FNBE, 2016). This creates a situation where the responsibility for teaching ICT skills is fragmented between several teachers, at worst without adequately expressed and internalised common goals. This scenario increases the risk that not all students are getting sufficient digital preparedness for the future in their basic level education. All things considered, however, the notable role of individuals (leisure time) experiences and usage habits in ICT skills and digital preparedness, when combined with decentralised teaching of those skills, may compromise the ideal of equality between the genders and principally between individual students. This raises important questions to study in future research.

 

6 References

Adams, W. K. & Wiemanc, C. E. (2011). Development and validation of instruments to measure learning of expert-like thinking. International Journal of Science Education, 33(9), 12891312. DOI: 10.1080/09500693.2010.512369

Aesaert, K. & van Braak, J. (2015). Gender and socioeconomic related differences in performance based ICT competences. Computers & Education, 84, 825. DOI: 10.1016/j.compedu.2014.12.017

Bawden, D. (2008). Origins and concepts of digital literacy. In C. Lankshear & M. Knobel (Eds.), Digital literacies: Concepts, policies and practices (pp 1732). New York: Peter Lang

Binkley, M., Erstad, E., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining 21st century skills. In P. Griffin, B. McGaw, & E. Care (Eds.) Assessment and teaching of 21st century skills (Educational Assessment in an Information Age) (pp. 1766). Dordrecht: Springer

Christoph, G., Goldhammer, F., Zylka, J. & Hartig, J. (2015). Adolescents computer performance: The role of self-concept and motivational aspects. Computers & Education, 81, 112. DOI: 10.1016/j.compedu.2014.09.004

Claro, M., Preiss, D., San Martin, E., Jara, I., Hinostroza, J, E., Valenzuela, S., Cortes, F., Nussbaum, M. (2012). Assessment of 21st century ICT skills in Chile: Test design and results from high school level students. Computers & Education, 59, 10421053. DOI: 10.1016/j.compedu.2012.04.004

van Deursen, A. J. A. M., Helsper, E., J. Eynon, R. (2016). Development and validation of the Internet Skill Scale (IIS). Information, Communication & Society, 19, 804823. DOI: 10.1080/1369118X.2015.1078834

van Deursen, A. J. A. M. & van Dijk, A. G. M. (2015). Internet skills levels increase, but gaps widen: a longitudinal cross-sectional analysis (2010-2013) among the Dutch population. Information, Communication & Society, 18(7), 782797. DOI: 10.1080/1369118X.2014.994544

van Deursen, A. J. A. M. and van Dijk, J. A. G. M. (2009). Improving digital skills for the use of online public information and services. Government Information Quarterly, 26, 333340.

Ferrari, A. (2013). DIGCOMP: A Framework for developing and understanding digital competence in Europe. Luxembourg: European Union. DOI: 10.1016/j.giq.2008.11.002

FNBE (2016). National core curriculum for basic education 2014. Helsinki: FNBE.

Fraillon, J., Schulz.W. & Ainley, J. (2013). International computer and information literacy study: Assessment framework. Amsterdam: IEA.

Gallardo-Echenique, E. E., de Oliveira, J. M., Marqus-Molias, L. & Esteve-Mon, F. (2015). Digital competence in the knowledge society. Journal of Online Learning and Teaching, 11(1), 116.

Haladyna, T. M. (2011). Developing and validating multiple-choice test items. 3rd edition. London: Routledge.

Hargittai, E. (2002). Beyond logs and surveys: In-depth measures of peoples web use skills. Journal of the American Society for Information Science and Technology, 23, 12391244. DOI: 10.1002/asi.10166

Hatlevik, O. E., Ottestad, G. & Throndsen, I. (2015). Predictors of digital competence in 7th grade: a multilevel analysis. Journal of Computer Assisted Learning, 31(3), 220231. DOI: 10.1111/jcal.12065

Heerwegh, D., De Wit, K. & Verhoeven, J. C. (2016). Exploring the self-reported ict skill levels of undergraduate science students. Journal of Information Technology Education: Research, 15, 1947.

Hoffman, L. H., & Schechter, A. L. (2016). Technical skills required: how technological efficacy influences online political behavior. Journal of Broadcasting & Electronic Media, 60, 484502. DOI: 10.1080/08838151.2016.1203321

Huffman, A. H., Whetten, J. & Huffman, W. H. (2013) Using technology in higher education: The influence of gender roles on technology self-efficacy. Computers in Human Behaviour, 29(4), 17791786. DOI: 10.1016/j.chb.2013.02.012

Kaarakainen, M.-T., Kivinen, O. & Vainio, T. (2017) Performance-based testing for ICT skills assessing: A case study of students and teachers ICT skills in Finnish schools. Universal Access in the Information Society. DOI: 10.1007/s10209-017-0553-9

van Laar, E., van Deursen, A.J.A.M, van Dijk, J.A.G.M., de Haan, J., (2017). The relation between 21st-century skills and digital skills or literacy A systematic literature review, Computers in human behavior, 72, 577588. DOI: 10.1016/j.chb.2017.03.010

Lau, W. W. F. & Yuen, A. H. K. (2014). Developing and validating of a perceived ICT literacy scale for junior secondary school students: Pedagogical and educational contributions. Computers & Education, 78, 19. DOI: 10.1016/j.compedu.2014.04.016

Litt, E. (2013). Measuring users Internet skills: A review of past assessments and a look toward the future. New Media & Society, 15, 612630. DOI: 10.1177/1461444813475424

Livingstone, S. and Helsper, E. (2010). Balancing opportunities and risks in teenagers use of the Internet: the role of online skills and Internet self-efficacy. New media & society, 12, 309329. DOI: 10.1177/1461444809342697

McGill, M. M., Decker, A. & Settle, A. (2016). Undergraduate students perceptions of the impact of pre-college computing activities on choices of major. ACM Transactions on Computing Education, 16(4), 133. DOI: 10.1145/2920214

Miller, T. B. 2015. Using differential item functioning to test for inter-rater reliability in constructed response items. Theses and Dissertations. Milwaukee: University of Wisconsin-Milwaukee.

Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory. 3rd edition. New York: McGraw-Hill.

Pagani, L., Argentin, G., Gui, M. & Stanca, L. (2016). The impact of digital skills on educational outcomes: evidence from performance tests. Educational studies, 42, 137162. DOI: 10.1080/03055698.2016.1148588

Peterson, R. A. (1994). A Meta-analysis of Cronbachs coefficient alpha. Journal of Consumer Research, 21 (2), 381391.

Ritzhaupt, A. D., Liu, F., Dawson, K. & Barron, A. E. (2013) Differences in student information and communication technology literacy based on socio-economic status, ethnicity, and gender. Journal of Research on Technology in Education, 45 (4), 291307. DOI: 10.1080/15391523.2013.10782607

Siddiq, F., Gochyyev, P. & Wilson, M. (2017). Learning in Digital Networks ICT literacy: A novel assessment of students 21st century skills. Computers & Education, 109, 1137.

Siddiq, F., Hatlevik, O., E., Olsen, Throndsen, R., V. & Scherer, R., (2016). Taking a future perspective by learning from the past A systematic review of assessment instruments that aim to measure primary and secondary school students ICT literacy. Educational Research Review, 19, 5884. DOI: 10.1016/j.edurev.2016.05.002

Taipale, S. (2014). The dimensions of mobilities: The spatial relationships between corporeal and digital mobilities. Social Science Research, 43, 157167. DOI: 10.1016/j.ssresearch.2013.10.003

Tiruneh, D. T., De Cock, M., Weldeslassie, A. G. & Janssen, R. (2017). Measuring critical thinking in physics: development and validation of a critical thinking test in electricity and magnetism. International Journal of Science and Mathematics Education, 15(4), 663682.

Warner, R. M. (2013). Applied statistics. From bivariate through multivariate techniques. Los Angeles: Sage.

Zhong, Z. J. (2011). From access to usage: the divide of self-reported digital skills among adolescents. Computers & Education, 56, 736746. DOI: 10.1016/j.compedu.2010.10.016

 

Appendix 1. The Modules and items and results of item analysis of the ICT skill test.



Module/item



Description

Item Difficulty
Index, P

Item Dis-crimination Index, D

Corrected Item-Total Correlation, r

Basic use

 

 

 

Basic operations

Participants have to pair a keyboard shortcut with a correct action and choose a correct type of computer memory for present education situation.

.18

.63

.40

Information seeking

Participants have four cases where they have to choose a correct source/channel, out of three, on where to further seek information on a given topic. After this, they are presented with list of search engine results and are asked to choose relevant items related to given scenario.

.61

.55

.38

Information networks

Participants are given four network usage scenarios and have to pair them with correct data transmission technologies and then match correct descriptions of computer network-related concepts.

.17

.57

.33

Productivity software

 

 

 

Word processing

Participants are asked to edit (bold, italicise, underline and highlight) a given sample text.

.51

.72

.49

Spreadsheets

Participants are asked to fill a spreadsheet table with given information, bold a header row, and sort the table in ascending order.

.26

.79

.50

Presentations

Participants are given a general user interface view of presentation software, with essential sections marked. The task is to pair a correct name with the right section of this view.

.28

.83

.50

Social networking and communication

 

 

 

Social networking

Participants have to pair correct social networking services with four service descriptions,

define the meaning of social networking service, and choose four items out of nine that relate to the security of social networking services.

.39

.75

.59

Communication

Participants have to fill in the receiver fields, carbon copy, and blind carbon copy) of an email and add an attachment according to instructions, and identify the types of information that can be used to identify Internet users.

.43

.85

.65

Information security

Participants have to choose correct statements for secure network communications and choose from alternatives those that are related to the information security of computers in an Internet cafe abroad.

.41

.80

.65

Content creation and publishing

 

 

 

Image processing

Participants have to select correct image processing tools for cropping an image and make the person appearing in the image unrecognisable. Afterwards, participants have to choose correct image processing using related statements from given options and choosing the correct file formats for vector graphics.

.33

.67

.61

Video and audio processing

First, participants have to choose those methods that can be used to edit video footage from a single camera and then choose a right answer to the question: Which one of these alternatives is related to lossy audio compression?.

.42

.95

.64

Cloud services and publishing

In the first step, participants have to choose which of the given statements about cloud services are true. In the second step one must choose the correct YouTube-video sharing option that enables limited sharing even to those who do not have an account on YouTube. The third step is a continuation question: Can we now be certain the video does not circulate to the rest of the Internet for outsiders to see [...]?

.41

.93

.56

Applications

 

 

 

Software purchasing

Participants have to choose which matters need to be considered when evaluating the information security of mobile applications and also choose the correct definition of personal data protection from four alternatives.

.19

.58

.45

Installation and updates

In the first step, participants choose whether a statement is about an installation or an upgrade and in the second step, they choose whether a statement is related to an update or an upgrade.

.47

.92

.56

Elementary programming

Participants have to write, per instructions, a maze traversing script that leads from the starting point to the end. Then they have to write the value of a variable after the given pseudo-code has completed.

.06

.57

.40

Requirements for the ICT study programs

 

 

 

Database operations

Participants have to form an SQL-query, based on given instructions and a simple database diagram, then choose the right definition for the concept NoSQL database.

.05

.51

.20

Web programming

Participants are given three files (HTML, CSS and JavaScript) to use to create a website and the view generated by these files. Participants then answer four multiple choice questions to edit the simple web page view and the dependencies between these given files.

.08

.34

.23

Programming

The programming task requires the participants to place lines of Java code in the correct places based on given comment sections.

.01

.44

.22