Download ERIC ED449192 PDF

TitleERIC ED449192
File Size236.2 KB
Total Pages18
Document Text Contents
Page 1


ED 449 192 TM 032 300

AUTHOR Cowley, Kimberly S.
TITLE Evaluating the Effectiveness of a Technical Assistance

System To Build the Organizational Capacity of a High-Need
School District: One Piece of the Puzzle.

INSTITUTION AEL, Inc., Charleston, WV.
SPONS AGENCY Office of Educational Research and Improvement (ED),

Washington, DC.
PUB DATE 2000-11-00
NOTE 17p.; Paper presented at the Annual Conference of the

American Evaluation Association (Honolulu, HI, November 1-5,
2000) .

PUB TYPE Reports - Research (143) -- Speeches/Meeting Papers (150)
EDRS PRICE MF01/PC01 Plus Postage.
DESCRIPTORS Decision Making; Delivery Systems; Elementary Secondary

Education; Organizational Climate; *Organizational
Effectiveness; Pretests Posttests; Program Evaluation; Rural
Schools; *School Districts; Teacher Empowerment; Teacher
Surveys; *Teachers; *Technical Assistance

As part of a project to learn more about the technical

assistance process of helping high needs schools to develop the
organizational capacity to support effective school performance, this study
examined changes after two years of technical assistance delivery in a high
needs school district. The district was in a rural area with one elementary
school, one middle school, and one high school. Three instruments were
selected to measure the construct of organizational capacity, and these
surveys were administered to district faculty members in the 1996-1997 school
year (pretest). The instruments included a measure of school participant
empowerment, school professional community, and perceived organizational
effectiveness. This paper summarizes findings from the posttest in 1999 in
which the same 3 instruments were administered to 80 teachers to measure
changes. The high school teachers' improved scores in decision making,
self-efficacy, status, and impact suggest that improved conditions and
practices at the high school resulted in a heightened sense of empowerment.
Other differences among teachers at the three levels are discussed. As a
whole, the district appears to have expanded its organizational capacity in
teacher empowerment but made minimal progress in creating sustainable
professional learning and increasing organizational effectiveness.
Recommendations are made to improve these two areas of practice. (Contains 3
figures and 12 references.) (SLD)

Reproductions supplied by EDRS are the best that can be made
from the original document.

Page 2



Evaluating the Effectiveness of a Technical Assistance
System to Build the Organizational Capacity

of a High-Need School District:
One Piece of the Puzzle

Kimberly S. Cowley
AEL, Inc.


Paper presented at the Annual Conference of the American Evaluation Association,
November 2000, Honolulu, HI




Office of Educational Research and Improvement



.1.14his document has been reproduced as
received from the person or organization
originating it.

Minor changes have been made to

improve reproduction quality.

Points of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.

Page 9


Table 1: Cronbach Alpha Reliability Estimates by Instrument

Instrument Name Subscale Name n No.





School Participant Decision Making 78 8 .67 .50 .66 .82
Empowerment Status with Colleagues 80 6 .76 .74 .69 .76
Scale (SPES) Professional Growth 80 5 .66 .52 .76 .70

Self-Efficacy 76 12 .89 .84 .89 .93
Autonomy in Scheduling 80 3 .81 .76 .76 .83
Impact 79 5 .67 .48 .71 .86

School-Wide Shared Sense of Purpose 79 5 .82 .77 .91 .77
Professional Collaborative Activity 76 6 .71 .54 .84 .82
Community Collective Focus on 80 6 .87 .73 .95 .69
(SWPC) Student Learning

Deprivatized Practice 79 7 .80 .79 .82 .76
Reflective Dialogue 77 6 .80 .78 .89 .74

Index of Perceived
Organizational Total Scale 77 8 .87 .85 .90 .66
Effectiveness (WOE)

Data Analyses

A database was created using SPSS Windows. Individual surveys were entered and the data
file cleaned. Descriptive statistics are reported by individual schools at the subscale level only.

While this study involved a district population and not a random sample, the entire
population was not included in the analyses and has changed considerably since the pretest.
Therefore, inferential statistics were used to compare between and within group differences. One-
way analyses of variance (ANOVA) were computed to compare subscale means among the three
school levels at RCPS (elementary, middle, and high), using the Tukey post-hoc test for
comparisons. Pearson correlations were produced to examine the relationships among the SPES and
SWPC subscales and the IPOE total scale. Both Pearson and Spearman correlations were produced
to examine possible relationships among the demographic variables and the three instruments.
Independent samples t tests were computed to compare pre- and posttest findings. Matched pairs
were not conducted since almost half of the respondents had started working at their respective
school after the pretest administration. And, the focus of the comparison was on overall school
readiness for improvement, not individual gains or losses. Only significant differences (Alpha level
of .05 or less) are reported for these statistical procedures. Effect sizes, defined as "the degree to
which the phenomenon is present in the population" (Cohen, 1977, p. 9) or an "indication for
practical meaningfulness" (Fan, 1999) were also calculated for significant t tests and ANOVAs.

Page 10



Demographic Variables and Organizational Capacity

The first analysis looked at possible relationships among the demographic variables and the
three major concepts that comprised organizational capacity: teacher empowerment, professional
community, and school effectiveness. Very few demographic variables were associated with
organizational capacity to a significant degree. Professional community scores showed a low
positive correlation to the gender of the respondents (r, = .32, p < .01), indicating a slight trend of
higher scores for male respondents when compared with those for females. School effectiveness
scores showed a low positive correlation to both the respondents' age and their years of total
teaching experience (r = .37,p < .01; r= .23,p < .05). Older or more experienced teachers perceived
their schools as being more effective than did younger or less experienced teachers. As well, the
level of grades taught correlated positively with school effectiveness scores (rs= .33,p < .01), with
middle and high school teachers rating their schools as more effective than did their elementary

Overall District Findings

This section presents summaries of significant findings and effect sizes by survey and
correlations among the instruments and their subscales. See Table 2 for statistical detail by subscale,
Table 3 for significant t tests by grade level and year, and Table 4 for significant ANOVAs by grade
level. See Figures 1-3 for a visual depiction of all subscale means by a combination of year and
grade level.

SPES. The SPES instrument showed four statistically significant subscale gains for the high
school teachers from the 1997 pretest to the 1999 posttest: Decision Making, Status with
Colleagues, Self-Efficacy, and Impact. These gains ranged from 0.32 for Self-Efficacy to 0.53 for
Decision Making. Cohen's guidelines for interpreting effect sizes (1977) were used for defining the
resulting effect sizes: small = 0.2, medium = 0.5, and large = 0.8. The Decision Making subscale
gain (0.53) had an effect size of 1.29, well above Cohen's "large" descriptor; both the Self-Efficacy
and Impact subscale gains (0.32 and 0.37, respectively) had large effect sizes, as well (0.86 and
0.82). The Status with Colleagues subscale gain (0.36) was of medium size (0.61). These effect
sizes, combined with statistical significance, indicate that the subscale gains were not due to chance
and that the magnitude of the gains was substantial.

The SPES instrument also showed three statistically significant differences among the 1999
schools. The high school teachers had a significantly higher score (3.61) than the elementary
teachers (3.23) on the Decision Making subscale, with a small effect size of 0.31. Both the middle
(4.17) and high school (4.16) teachers had significantly higher scores than the elementary teachers
(3.83) for the Status with Colleagues subscale, with a small effect size of 0.39. And, the elementary


Page 17




AEL. (1995). Technical proposal for operation of the Regional Educational Laboratory in the
Appalachian region for 1996-2000. Charleston, WV: Author.

AEL. (1998). 1999 updated annual plan and budget. Submitted to the Office of Educational
Research and Improvement, U.S. Department of Education. Charleston, WV: Author.

Cohen, J. (1977). Statistical power analysis for the behavioral sciences. New York: Academic

Cowley, K. S., Nilsen, K. L., & Ceperley, P. E. (2000). Evaluation of a high-need school district's
organizational capacity for change. Charleston, WV: AEL.

Fan, X. (1999). Statistical significance and effect size: Two sides of a coin. Paper presented at the
annual meeting of the American Educational Research Association, Orlando, FL.

Klecker, B., & Loadman, W. E. (1996, April). An analysis of the school participant empower-
ment scale (Short & Rinehart, 1992) based on data from 4,091 teachers in 183 restructuring
schools. Paper presented at the annual meeting of the American Educational Research
Association, New York.

Louis, K. S., Marks, H. M., & Kruse, S. (1996). Teachers' professional community in restructuring
schools. American Educational Research Journal, 33, 757-798.

Meehan, M. L., & Cowley, K. S. (1998). Comprehensive evaluation of the 1996 interdisciplinary
teamed instruction summer institute. Charleston, WV: AEL.

Mott, P. E. (1972). The characteristics of effective organizations. New York: Harper & Row.

Nilsen, K. L. (1999). Assessing organizational capacity for change in a high-need Virginia school
district. Charleston, WV: AEL.

Short, P. M., & Rinehart, J. S. (1992). School participant empowerment scale: Assessment of level
of empowerment within the school environment. Educational and Psychological Measurement,
52, 951-961.

Western Michigan University. (2000). Draft FY 99 report: External evaluation of AEL, Inc.
Kalamazoo, MI: The Evaluation Center.


Page 18

U.S. Department of Education
Office of Educational Research and Improvement (OERI)

National Library of Education (NLE)
Educational Resources Information Center (ERIC)


Reproduction Basis



This document is covered by a signed "Reproduction Release
(Blanket)" form (on file within the ERIC system), encompassing all
or classes of documents from its source organization and, therefore,
does not require a "Specific Document" Release form.

This document is Federally-funded, or carries its own permission to
reproduce, or is otherwise in the public domain and, therefore, may
be reproduced by ERIC without a signed Reproduction Release form
(either "Specific Document" or "Blanket").

EFF-089 (3/2000)

Similer Documents