Measuring Innovativeness of Public Organizations: Using Natural Language Processing Techniques in Computer-Aided Textual Analysis

ABSTRACTWe propose a new method for measuring innovativeness in the public sector using natural language processing techniques. Our approach extends traditional content analysis techniques by combining insights from linguistic theory and recent developments in computational techniques. We develop and employ phrase-level data dictionaries (using both noun phrases and verb phrases) from organizational documents. We use letters to the board of education from a sample of New Jersey school districts to develop measures of innovativeness—one measure is based on expert assessments and the other is inductively derived. We perform rigorous tests of content validity, external validity, and predictive validity on these measures. We conclude with a discussion of implications of this new measurement approach and its potential applications to other public management contexts. ACKNOWLEDGMENTSWe are grateful to Stephan Grimmelikhuijsen, Lars Tummers, Yas Nakib, and anonymous reviewers for thoughtful critique and suggestions that have greatly improved the article. We also want to acknowledge the assistance of graduate students Jian Xie and Erica Broadus. Needless to say, any remaining errors are our responsibility.NotesOf 597 school districts, 296 school districts reported standardized test scores for the year 2012 for the high school assessment tests. We excluded 50 school districts that we previously used for developing the natural-language-processing-based data dictionary discussed earlier. Furthermore, documents for 39 school districts were available only in image format and therefore unusable for computerized content analysis. As a result, our validation sample included 210 New Jersey school districts.Normalization standardizes scores for the lengths of the documents. Normalization controls for the length of the document and ensures that construct scores generated from longer documents are not higher simply because of their length.The education system in the United States is broken up into three sectors that vary by the age of students served, the governance structure of the sector, and funding source. These three sectors are commonly referred to as early childhood education (ECE), kindergarten through grade 12 (K–12), and higher education. The US does not have a universal ECE program. Instead, students younger than five years old are served by a variety of federally funded programs (early head start, head start), state-funded programs (pre-K), and private programs. Students from age 5–18 are predominantly served by a universal state-governed school system, with considerable funding and responsibility assigned to local residents. High school graduates can attend a tuition-based private or public college, university, or technical school.Alternate specifications using quadratic and cubic terms to account for the potential for nonlinear relationship were tested and turned out to be non-significant. We gave consideration to including in the model the proportion of students receiving free lunch. However, this variable had bivariate correlation of 0.9 with percent limited English proficient and 0.8 with percent on IEP and therefore we chose to include the two variables (%IEP and %LEP) to avoid multicollinearity.Additional informationNotes on contributorsSheela PandeySheela Pandey (spandeywrk@gmail.com) is a Visiting Scholar at the Trachtenberg School, The George Washington University, and a Fellow with the Center for Organization Research and Design (CORD) at the Arizona State University. Her research interests are in nonprofit management, strategic management, and social entrepreneurship. Her research projects apply strategic management theories and concepts to nonprofit, for-profit, and public sector organizational contexts.Sanjay K. PandeySanjay K. Pandey (sanjay.k.pandey@gmail.com or skpandey@gwu.edu) is Shapiro Professor of Public Policy and Public Administration at the Trachtenberg School, the George Washington University. He is a recipient of the NASPAA/ASPA Distinguished Research Award and an elected Fellow of the National Academy of Public Administration.Larry MillerLarry Miller (larry.miller@fsw.edu) is Dean of the School of Education and charter schools at Florida SouthWestern State College; and a research affiliate at the Center on Reinventing Public Education at the University of Washington and the Edunomics Lab at Georgetown University. Dr. Miller is also a consultant with the National Center on Innovation in Education at the University of Kentucky. He received his PhD in public administration at the Maxwell School of Citizenship and Public Affairs, Syracuse University. His current research examines how public schools reallocate resources to support personalize learning. He is currently working with state and district leaders in Kentucky and New Hampshire to design new finance systems that support personalized learning.