Development and Psychometric Evaluation of Speech and Language Pathology Evidence-Based Practice Questionnaire (SLP-EBPQ)

Introduction

Evidence-Based Practice (EBP) is a process seeking to improve the provision of services to patients by considering the best research evidence, clinical experiences of therapists, and patient’s needs and preferences (1, 2). This systematic process helps therapists to make better clinical decisions. Despite its root in medicine, application of EBP in other fields related to health sciences has extensively increased all over the world (3-9). In addition, teaching EBP is a part of the curricula in many disciplines (10).

Today, EBP is the center of attention due to its benefits, including more effective provision of services for each individual, more accountability toward patients and their families, creation of equal treatment for clients according to scientific evidences, updating of the knowledge of the therapists, improvement in the quality of service provision, increase of the professional credibility, encouragement of adherence to ethical principles, improvement in interdisciplinary collaborations and communications, and encouragement of continuing education (5, 11-13).

As a member of the healthcare team, speech and language pathologists are required to pay more attention to EBP and make their clinical decisions based on evidence (2, 14) (15). The evaluation and review of the activities in various countries to extend the use of EBP in Speech and Language Pathology (SLP) highlight the importance of this issue (3). Scientific associations affiliated to SLP, such as Royal College of Speech and Language Therapists and American Speech-Language-Hearing Association, have also emphasized the use of EBP in SLP (16). Moreover, these associations have defined research and development in EBP as their priorities (4, 16, 17). The evaluation of the current status and documentation of the performance of speech and language pathologists about EBP could be an important step for the greater use of EBP. Accordingly, there is need to design an instrument, which can assess the status of EBP in SLP.

Today, there is a variety of questionnaires to evaluate EBP in various fields of medicine (7, 18-21). However, to our knowledge, there is no comprehensive instrument for assessment of EBP, which is specifically designed for speech and language pathologists. Meanwhile, each field or profession has its own properties, which can affect the use of EBP (22). As a result, a special SLP questionnaire must be designed in related to EBP. With this background in mind, this study aimed to develop and evaluate the psychometric properties of the Speech and Language Pathology-Evidence-Based Practice Questionnaire (SLP-EBPQ).

Methods

This methodological research was performed in two stages: 1) design of SLP-EBPQ through the review of the literature and semi-structured interviews with a group of speech and language pathologists (n=14), and 2) evaluation of psychometric properties of SLP-EBPQ by assessing its reliability and validity.

At first, the first author searched the Scopus, PubMed (Medline), and Web of Science databases as well as Google Scholar search engine for the published articles up to April 2017 in order to extract the existing questionnaires and instruments in the EBP field. Various keywords were applied to the comprehensive assessment of the present questionnaires. These keywords included ‘evidence-based practice’ in combination with ‘profile’, ‘tool’, ‘questionnaire’, ‘assessment’ and ‘instrument’. Moreover, manual search in the EBP-related articles published in well-known journals, was carried out to ensure the access to the maximum coverage of the information in this area. The inclusion criteria were: All English or Persian articles related to EBP, and access to the full text of articles. Non-English, non-Persian articles, and articles with no available questionnaire were excluded from the study. As a result, 17 articles were found and used for the item generation stage. This literature review resulted in 307 items.

Qualitative research was conducted to compile the experiences of speech and language pathologists toward EBP. In this section of the study, semi-structured interviews were conducted with 14 speech and language pathologists selected through purposive sampling. The reason for this was to achieve maximum variation in terms of years of SLP experience, education degree, workplace, clinical practice field, and types of roles.

The main questions of the interviews were 1) could you please describe your experience with EBP in clinical works? 2) What kind of EBP activities (e.g., participating in a journal club, reading articles, attending workshops and congresses, participating in continuing education programs) do you participate in? 3) What barriers do you faced in the realm of EBP? 4) What factors have you experienced as facilitators for the application of EBP? 5) Could you please explain about your experience in the application of EBP? And 6) what is the outcome of the application of the EBP in clinical practice? When data saturation was reached, the data collection stopped.

Interviews were analyzed applying conventional content analysis approach (23). To this end, the following steps were conducted: (1) transcribing the interviews, (2) deriving codes by reading the interviews word by word, (3) sorting the derived codes into categories, and (4) sorting the categories to create themes (24). Consequently, the primary questionnaire was designed based on the concept explained in the qualitative section of the study and literature review.

In total, 447 items were generated after the review of the literature and conducting semi-structured interviews. These items were evaluated by the research team and experts in several sessions to eliminate duplicated items (201 items) and merge items with similar meaning (135 items). with 111 items on five main domains, including attitude toward EBP, knowledge about EBP, use of EBP, barriers to the use of EBP and facilitators of EBP application.

The face validity of the SLP-EBPQ was determined qualitatively and quantitatively. At first, face to face interviews were conducted with 10 speech and language pathologists, and they were asked to study the questionnaire items and provide insights about the ‘relevancy’, ‘ambiguity’ and ‘difficulty’ of the items. Item Impact Score (IIS) was applied to determine quantitative face validity. In this regard, 10 SLP experts expressed their opinions about the importance of each item on the basis of the 5-point Likert scale ranging from the most important (score 5) to the least important (score 1). The IIS for each item is calculated as follows: Item Impact Score= percentage of raters who scored a score of 4 or 5 × mean score for the importance of each item. Each item with IIS scores of 1.5 or above was selected for the next stage, and those with lower scores were eliminated from the questionnaire (25, 26).

In the qualitative content validity assessment, 15 experts from the SLP faculty members (10 with PhD degree and 5 PhD students), who have experiences in EBP, evaluate the questionnaire items regarding grammar standards, use of proper phrases, necessity, importance and type of scoring, and the order of phrases in the accurate place (27). Quantitative content validity was determined using two indicators, namely content validity ratio (CVR) and content validity index (CVI). To determine CVR, the opinions of the 15 experts were used to provide insights on the necessity of each item according to the 3-point Likert Scale (1=essential, 2=useful but not essential, and 3=unessential). According to Lawshes’ Table and with regard to the participation of 15 experts at this stage, items with CVR scores of 0.49 or above were maintained (28). After this stage, the same 15 experts provided their opinions about the relevancy of the items according to a 4-point Likert scale (1=non-relevant, 2=fairly relevant, 3=relevant, and 4=completely relevant) to determine the CVI (29). The required amount of CVI for maintaining each item at this stage was ≥0.78 (29). Moreover, the mean of CVI was calculated according to the mean CVI score of all questionnaire items. At this level, the required amount for S-CVI/Ave was ≥0.9 (29). Furthermore, the Cohen's  kappa coefficient was used to provide the complementary information about the CVI, since it provides data about the agreement degree beyond the chance (29).

To analyze the items, the questionnaire was distributed among 43 speech and language pathologists. This stage aimed to evaluate the problems in the questionnaire, type of scoring, and required time to fill the questionnaire. Afterwards, the participants were asked to express their opinions about the clarity and comprehensiveness of the items.

Preliminary reliability is a type of item analysis with an emphasis on internal consistency. At this stage, the loop method was applied to evaluate the effect of elimination of each item on the increase of the internal consistency of that domain (Cronbach’s alpha). This stage is an essential step toward the evaluation, revision, and/or elimination of improper items before carrying out the construct validity. Furthermore, the correlation of each item with the domain total score and other items was assessed in order to eliminate items which have low correlation with the domain total score. Moreover, discrimination and difficulty index was calculated for the items of the knowledge domain of SLP-EBPQ to eliminate the items with the lack of necessary criteria.

Exploratory Factor Analysis (EFA) was used to explore the factors of the SLP-EBPQ. To this end, we used the principal components analysis with varimax rotation (considering the independence of the factors). If there was a missing data at this stage, it would be replaced by the median. Sampling adequacy to perform EFA was assessed through the Kaiser-Meyer-Olkin (KMO) and Bartlett’s Test of Sphericity. The sample size is adequate to perform EFA when the KMO value was above the 0.5 (30, 31). Eigenvalues above one and the scree plot method were used to decide the number of prominent factors (30).

The minimum factor load for maintaining each item in factors extracted from factor analysis was determined at 0.3. After this stage and extraction of SLP-EBPQ factors, the level of fitness was assessed for these factors and main dimensions of SLP-EBPQ. Furthermore, the comparison of the recognized groups was performed to explore the construct validity and determine the ability of the SLP-EBPQ to properly separate different groups. To this end, the educational level of the participants was divided into two categories of BSc (undergraduate) and above BSc (postgraduate). Following that, independent t-test was exploited to compare the items related to the factors of SLP-EBPQ between the two groups. The skewness and kurtosis were used to investigate the normality of the data.

Internal consistency was assessed using Cronbach’s alpha coefficient, and test-retest reliability was used to evaluate the reliability of the SLP-EBPQ. A Cronbach’s alpha coefficient of 0.7 or above was considered satisfactory (32). In order to investigate test-retest reliability, the questionnaire was completed by 30 speech and language pathologists in two stages with a two-week interval. Subsequently, the obtained scored in the two stages were compared using the intraclass correlation coefficient (ICC) with a two-way mixed model and absolute consistency type. The test-retest reliability considered excellent if the ICC was 0.75 or above (33).

To evaluate the SLP-EBPQ, the speech and language pathologists from the Iranian scientific speech therapy association were selected through random sampling technique. Inclusion criteria for the participants were: (1) a bachelor or higher degree in SLP and (2) an interest to participate in the study. The exclusion criteria included: (1) incomplete consent form, (2) partial completion of the questionnaire, and (3) no clinical practice in the last three months. Three hundred questionnaires were distributed among speech and language pathologists with the mentioned criteria to evaluate the psychometric properties of the SLP-EBPQ. To determine test-retest reliability, 30 participants completed the SLP-EBPQ again with an interval of 2 weeks based on their own interest.

All statistical analyses performed using SPSS software (version 18.0; SPSS, Inc., Chicago, IL). Descriptive statistics including frequencies, means, and standard deviations were used to conduct both item- and subscale-level analyses. P-value less than 0.05 considered statistically significant.

It should be noted that this research was approved by the Ethics Committee affiliated with Iran University of Medical Sciences, Tehran, Iran (IR.IUMS.RE1395.9221363202). In addition, the objectives of the research were explained to the participants and they were assured of the confidentiality regarding their personal information. Furthermore, participation in this study was voluntary and the subjects were allowed to withdraw from the study at any time. Finally, written informed consent forms were obtained from the participants who agreed to participate voluntarily in this study prior to the research.

Results

From 300 questionnaires distributed among the speech and language pathologists, 290 were returned. In addition, 10 questionnaires were eliminated due to incomplete information, and a total of 280 questionnaires were evaluated in terms of psychometric properties. Demographic characteristics of the participants are shown in Table 1.

The obtained results of IIS revealed that the face validity of the items was within the range of 3.4-5. Given the fact that the IIS of all items was above 1.5, none of the items were eliminated at this stage. Furthermore, some changes were applied to the items of SLP-EBPQ in qualitative assessment of content validity according to the opinions of 10 speech and language pathologists in order to ensure the comprehension of the items.

In the qualitative evaluation of content validity, some changes were applied to the items according to the opinions of the experts. Furthermore, several items were merged due to their similarity. Therefore, the number of items decreased to 107 ones.

In the quantitative evaluation of content validity, 25 items were eliminated due to CVR scores below 0.49. Nevertheless, 19 items were removed according to the opinion of the research team. Meanwhile,

Table 1. Demographic characteristics of participants

six of these items were kept due to their high importance. No item was eliminated in the CVI stage. Moreover, the questionnaire scale-level content validity index (S-CVI) was 0.952 and modified kappa coefficient was 0.79-1, which was desirable (29). After the evaluation of face and content validities, an 88-item questionnaire was prepared for the next stage.

According to the results of the pilot and item analysis stages, another eight items were eliminated at this stage. Moreover, two items from the attitude section, two items from the barriers section and one item from the section of facilitators were eliminated due to their negative impact on the Cronbach’s alpha and lack of significant correlation with other items. On the other hand, three items were eliminated from the knowledge section of the questionnaire with regard to the discrimination and difficulty index.

In the current study, EFA was performed on 80 items and the KMO coefficient was 0.79. Moreover, the value of Bartlett’s Test of Sphericity was estimated at 9937.550, which was significant at P

An oblique factor rotation detected nine latent factors. The extraction was according to scree plot visual interpretation and Kaiser’s criterion for Eigenvalues of equal to or greater than unity. The nine factors, comprising 77 of the original 80 items, explained 46.92% of the total variance. Three items were removed from the questionnaire due to their low loading on the factors. In total, nine factors were identified according to the results of factor analysis and at the end of this stage, 77 items remained in the questionnaire (Figure 1). Information related to the factors, titles, the number of items, and their percentage of variance is presented in Table 2.

According to the results of the evaluation of construct validity, individuals with an educational level above BSc had a greater attitude and more knowledge and usage of external evidence, compared to individuals with BSc. This difference between the groups was statistically significant. In addition, individuals with BSc faced more barriers while using EBP, compared to those with degrees above BSc. This difference was statistically significant, with the exception of a factor of individual obstacles. Moreover, no significant difference was observed between the groups in terms of facilitators to use of EBP (Table 3). It should be noted that the values of the skewness and kurtosis were +1.7 and +2.1, respectively.

The values of Cronbach’s alpha of various factors in SLP-EBPQ were determined within the range of 0.635-0.885 (Table 4). In addition, the results of the test-retest reliability of various factors in SLP-EBPQ were within the range of 0.814-0.966 by the evaluation of ICC, which was indicative of the proper test-retest reliability of the SLP-EBPQ. Details related to the values of ICC of various factors of SLP-EBPQ are presented in Table 4.

Table 2. Factors, items, and factor loadings for Speech-Language Pathology Evidence-Based Practice Questionnaire (n=280)

Table 3. Results of comparing scores of different factors of the Speech and Language Pathology Evidence-Based Practice Questionnaire between two groups with different degrees (n = 280)

Table 4. Cronbach's Alpha and Intraclass correlation coefficient for domains of the Speech and Language Pathology Evidence-Based Practice Questionnaire

ICC: intraclass correlation coefficient

Discussion

The SLP-EBPQ was designed according to the concepts determined in the qualitative section of the research. This questionnaire is designed for evaluation of the attitude, knowledge, and use of the EBP by speech and language pathologists and assessment of the associated barriers and facilitators of EBP. Therefore, SLP-EBPQ can be used by speech and language pathologists of various countries by considering the translation of the target language, modifications based on cultural factors, and facilities of the selected country, as well as the evaluation of the psychometric properties. In this research, face and content validities (qualitative and quantitative), construct validity (exploratory factor analysis and comparison of the recognized groups), internal consistency (Cronbach’s alpha) and test-retest reliability of the questionnaire were confirmed.

In the evaluation of the content and face validities, the improper items were eliminated from the questionnaire based on the opinion of experts and respondents, and accordingly the total number of 111 items reduced to 88 items. In addition, eight items were removed in the pilot and item analysis stage. In the end, an 80-item questionnaire with five domains was entered into the stage of construct validity evaluation.

In the development of the primary questionnaire, the items of SLP-EBPQ were categorized into five domains, including attitude, knowledge, use, barriers, and facilitators. Factor analysis results demonstrated that the items of SLP-EBPQ were in nine factors. Items in the sections of attitude, knowledge, and facilitators were each placed in a separate factor, similar to the primary questionnaire. The primary questionnaire was turned into two factors of the application of internal and external evidence. Moreover, the barrier section of the primary questionnaire was changed into four various factors (environmental, personal, barriers related to skill, education and access to evidence, and workshops and congresses barriers). Given the consistency of items in the extracted factors in factor analysis with the determined dimensions of the questionnaire in the first stage, the construct validity of SLP-EBPQ was confirmed.

In the present study, the analysis of construct validity revealed that SLP-EBPQ had an appropriate level of construct validity. Moreover, comparison of participants with BSc degree and those with degrees higher than BSc revealed a significant difference between the two groups in terms of attitude, knowledge, use of evidence, and parts of the barrier. Furthermore, the obtained results of internal consistency of the questionnaire indicated that Cronbach’s alpha values of seven out of nine factors were above 0.7, which is an appropriate value. However, the Cronbach’s alpha for two other factors of the SLP-EBPQ were within the range of 0.6-0.7, which was lower than other sections. The lower values of Cronbach’s alpha in these two factors could be related to their low number of items. In the current research, the reliability of SLP-EBPQ was evaluated through test-retest. Results obtained by two implementations of SLP-EBPQ with a two-week interval demonstrated the high stability of the SLP-EBPQ in short-term.

Compared to other questionnaires and instruments used in the EBP area for various professions, SLP-EBPQ is a self-report tool similar to other questionnaires (7, 18, 19, 21, 34, 35). In addition, SLP-EBPQ contains nine domains (attitude, knowledge, use of external evidence, use of internal evidence, environmental barrier, personal barrier, barriers related to skill, education and access to evidence, workshop and congresses barrier and facilitators), the majority of them which is similar to other questionnaires available in this field.

However, in the knowledge section, in which the knowledge of person about EBP and the associated factors are evaluated, some multiple-choice questions are used to accurately and properly assess the knowledge of individuals. Meanwhile, the knowledge of person about EBP was evaluated in previous questionnaires through self-report and self-assessment, which might have had biases (19). In addition, given the fact that qualitative studies and interviews with speech and language pathologists were used to design SLP-EBPQ, the barrier section is more comprehensive and contains 33 items placed in four factors. According to our knowledge, the items for evaluation of the barriers has never been observed in other questionnaires (18, 19), and this comprehensiveness could be the strength of the SLP-EBPQ.

The questionnaire contains 77 items that makes it a comprehensive tool on EBP. However, it is essential to increase the necessary time for completing the questionnaire, which might lead to tiresome and hasty responses of the respondents. However, given the evaluation of numerous variables by SLP-EBPQ, a decrease in items might lead to reduced content validity (19).

One of the major limitations of this study was the long time required to complete the questionnaire by the participants in various stages of validity due to the large number of items. Moreover, given the lack of similar questionnaires in EBP for Iranian speech and language pathologists, we failed to assess the concurrent validity of the questionnaire. Furthermore, similar to many questionnaires in the EBP area, SLP-EBPQ is also a self-reported questionnaire and its responses might be affected by social desirability. Despite these limitations, diversity in the domains and comprehensiveness of SLP-EBPQ, proper face and content validities, acceptable internal consistency, and high reliability are the strengths of the questionnaire.

Implications for Practice

According to the results of the current study, SLP-EBPQ can be used to evaluate knowledge, attitude, and application of EBP of speech and language pathologists and could be beneficial for the evaluation of barriers and facilitators in this regard. The SLP-EBPQ can be used for educational, clinical, and research purposes due to its acceptable level of reliability and validity and can be a beneficial instrument in EBP area. However, it is recommended to conduct further studies to evaluate the SLP-EBPQ since it is a new questionnaire.

Acknowledgments

This article was extracted from the PhD dissertation of the author in speech and language pathology supported by Iran University of Medical Sciences, Tehran, Iran. We would like to thank the Research Deputy of Iran University of Medical Sciences, experts, and speech and language pathologists who cooperated with this research.

Conflicts of Interest

The authors declare that they have no conflicts of interest.