Core IV & V: Measurements and Methods and Data

 Core Director: Jeanne Teresi, Ed.D, Ph.D.

 Co-Director: Mildred Ramirez, Ph.D.

 

Measurement and Methods Resources: For additional resources click HERE

Cores 4 & 5 Report: Measurement and Methods Cores (2006-2007)

a.) Changes in Investigators: None.

 b.) Specific Aims: The overall goal and specific aims of the Measurement and Methods Cores remain unchanged.

 c.)  Studies and Results:

Measurement and Methods Cores Seminars

A key activity of Cores 4 and 5 was the conduct of the Measurement and Methods Cores Seminars and the preparation of the Summary Grids.  The pilot investigators’ summary grids, charting the progress of the investigators were updated routinely.  The Seminars and Summary Grids comprised the cornerstone of the mechanism for Cores 4 and 5 to provide ongoing regular structured support to investigators.   

At each Seminar, the grantees presented their work, followed by questions, suggestions and recommendations provided by the faculty and fellow grantees.  Grantees from all cohorts were invited to attend the meetings; most have remained in the group for two to three years.  Mentors were also invited, and several attended selected seminars. The first Seminar meeting of the CALME renewal was held on October 31, 2002, the second on February 27, 2003, the third on May 29, 2003, the fourth on October 03, 2003, the fifth on March 04, 2004, the sixth on June 03, 2004, the seventh on October 21, 2004, the eighth on March 17, 2005, the ninth on June 9, 2005, the tenth on October 13, 2005, the eleventh on March 09, 2006, the twelfth on May, 11, 2006, the thirteenth on October 19, 2006, the fourteenth on March 1, 2007, and the fifteenth on May 24, 2007. 

After each meeting, detailed minutes (8 to 10 pages) were prepared and distributed by Dr. Ramirez.  The minutes are sent to all faculty, grantees, mentors and members of the Executive and Pilot Awards Review Committees.  Each set of minutes summarizes the recommendations of the group and provided an action plan.  The Measurement and Methods Cores comprise ten faculty members representing numerous disciplines:  health services research, sociology, epidemiology and medicine, biostatistics, mathematical statistics, measurement statistics, health economics, health care policy, computer science, community relations and institutional review board, grants management, industrial/social psychology, community psychology, and gerontology and geriatrics.

Assistance to investigators in the development of new R01's  

Numerous contacts provided by Measurement and Methods Cores faculty on behalf of investigators are documented through an Evaluation Grid. 

Training internships for pilot investigators

Several pilot investigators have benefited from training internships conducted by the Measurement and Methods Cores faculty. One pilot investigator spent one day a week learning how to perform path analysis, and published two papers as a result of the experience.  Two other pilot investigators attended several month bi-weekly training sessions aimed at instructions in measurement statistics, including basic hierarchical invariance testing using factor analyses.

Additionally, Beverly Watkins, a RCMAR scholar, received a training grant: NIA Supplements to Promote Re-entry into Biomedical and Behavioral Research: Minority Elders and Health Disparities: developing culturally sensitive measures for use in marginalized communities. She has been meeting biweekly with faculty from the Measurement and Methods Seminar.

Cross-site mentoring: Psychometics workshops at Friday Harbor

Sponsored by the University of Washington and the University of California, Davis ADRC, CALME faculty in collaboration with the Columbia and UC Davis ADRC at Friday Harbor conducted week-long psychometrics workshops in the summer of 2005 and 2006 to examine language bias in the SENAS, a major neuropsychological test. These workshops trained 20 to 30 junior investigators (from across the country and internationally) in the applications of item response theory (IRT). Several RCMAR faculty members, and current and past scholars from various sites have attended the workshops; e.g., Jennifer Manly ( Columbia), Rich Jones, Frances Yang (Columbia/Harvard), Chengwu Yang (Medical University of South Carolina), and Kala Mehta (UCSF).

Cross-site mentoring: Methodological workshops at Harvard Medical Center

Organized by Drs. Jones and Yang (of CALME), a three day workshop entitled: “Introduction to Latent Variable Methods in Measurement Research on Aging using M-Plus” has been offered in the summer of 2005 and 2006 at Harvard Medical Center. This workshop covered introduction to M-Plus and hands-on training on measurement and longitudinal models. Several RCMAR PIs from across the country attended the workshop including CALME scholars, Beverly Watkins and Earl Chambers (CALME).

       Cross-site mentoring: Visiting Scholar Program

Dr. Kala Mehta from the UCSF RCMAR, Dr. Frances Yang from Harvard University (CALME RCMAR scholar) and Dr. Chengwu Yang (MUSC RCMAR) were at Columbia and the Research Division labs of the HHAR to receive training with respect to examining measurement equivalence using item response theory ( IRT).

Cross-site mentoring: Collaborative Pilot Research Proposal for Junior Investigators

The project entitled “Detecting Differential Item Functioning in Cognitive Measures for Spanish and English-Speaking Older Adults by Acculturation and Language of Test Administration” was the first Collaborative Pilot Research Proposal for Junior Investigators awarded. The project aimed to identify DIF attributable to socio-cultural factors on measures of neuropsychological function using the Spanish and English Neurological Assessment Scales (SENAS) dataset through three DIF approaches: PARSCALE/DIFdetect, MULTILOG/DFIT, and Mplus/MIMIC. The team Junior Investigators are: Frances M. Yang, PhD (PI, Columbia), Kevin Heslin. PhD (UCLA), Kala Mehta, ScD (UCSF), Chengwu Yang, ABD (MUSC). Members of all RCMAR sites are participating as team mentors and advisors: Ron Hays, PhD (UCLA); Richard N. Jones, ScD (Columbia); Rafael A. Lantigua, MD (Columbia); Dan Mangus, PhD (UCDavis); Leo Morales, MD, PhD (UCLA); Katja Ocepek-Welikson, PhD (Columbia); Mildred Ramirez, PhD (Columbia); Anita L. Stewart, PhD (UCSF); Jeanne Teresi, EdD PhD (Columbia); Barbara C. Tilley, PhD (MUSC); Jack Goldberg, PhD (Colorado); and Tom Templin, PhD (Wayne State/Michigan). This project was successfully completed, and a paper submitted to the journal, Health Services Research (HSR) in June of 2007.

Protocol for evaluating measures for cultural sensitivity

Working with the Measurement and Methods Cores from the other five RCMAR sites, core 4-5 investigators at CALME developed a resource website to provide outside investigators with access to reviews of measures and guidelines for selection of culture-fair measuresis being developed.  The CALME site is focused on cognitive assessment measures. It contains abstracts of review articles discussing measurement issues across several domains. It includes measurement evaluation grids (MEGs) for cognitive measures. It also includes a section on measurement equivalence presenting abstracts and MEGs with expert reviews in the areas of: differential item functioning, and domains such as: depression, quality of life, general health and physical functioning. It provides a compilation of intermediate and advanced-level references on measurement and methods. It will be expanded to include MEGS in the areas of physical functioning and pain. The MEGS can be found in the CALME website (http://salk.cpmc.columbia.edu/dept/calme or at http://www.research-hhar.org/SubMeasure/).

Scale construction and analyses of data

The psychometric properties of measures must be adequate for minority as well as for majority populations in order for valid across-group comparisons to be made.  Item response theory ( IRT) was used to examine the psychometric properties of a cognitive screening measure used in several epidemiological surveys among Latino, African-American and White, non-Latino elderly.  Estimates of precision (reliability), examined across several values of θ (the estimate of degree of cognitive impairment), were good (.70's to .90's) in the range representing most respondents.  Overall reliability, although adequate, was lower among the White non-Latino and high-education groups relative to other ethnic/race and education subgroups.  Differential Item Functioning (DIF) was examined using several area- and model-based tests.  Tests of the magnitude of DIF showed the measure to be relatively free of DIF for the racial/ethnic group and education groups examined.  However one item, related to remembering the telephone number, was more difficult for Latinos than for other ethnic/racial groups.  Several other items evidenced mild DIF, and one item (difficulty remembering words or names) was a poorly discriminating item.  The subjective memory items, intended to provide additional information at the earlier, "borderzone" stages of cognitive impairment, did provide more information at the mild to moderate levels of impairment, although maximum information was not provided at these borderzone ranges.  This work was published in 2000; a cross validation is currently being prepared.

Teresi, J.A., Kleinman, M., Ocepek-Welikson, K., Ramirez, M., Gurland, B., Lantigua, R., Holmes, D.  Applications of item response theory to the examination of the psychometric properties and differential item functioning of the CARE Dementia Diagnostic Scale among samples of Latino, African-American and White non-Latino Elderly.  Research on Aging. 2000, 22, 738-773.

   Analyses of differential item functioning of patient reported outcomes

Statistical Coordinating Center (SCC) for the roadmap project, Patient Reported Outcomes Information System (PROMIS)

This is part of the NIH roadmap initiative that is designed to stimulate and standardize clinical research across NIH programs and grants dealing with patient reported outcomes. The SCC will perform item response theory, item banking and computer assisted testing. Both Dr. Teresi and Dr. Ron Hays (UCLA RCMAR) are PROMIS investigators. Dr. David Cella, the director of the statistical coordinating center to PROMIS and Dr. Bryce Reeve (a measurement statistician and project officer from NCI) have collaborated with the RCMAR projects, contributing an article on item banking and computerized adaptive testing to the special issue of Medical Care. Five areas will be the focus of the PROMIS effort; there is overlap with respect to PROMIS and CALME in two areas: physical function and affective suffering.

Teresi, J., Ocepek-Welikson, K., Kleinman, M., Cook, K.F., Crane, P.K., Gibbons, L.E., Morales, L.S., Orlando-Edelen, M., Cella, D. (In press) Evaluating measurement equivalence using the item response theory log-likelihood ratio (IRTLR) method to assess differential item functioning (DIF): applications (with illustrations) to measures of physical functioning ability and general distress. Quality Life Research.

Teresi, J.A., Fleishman, J.A. (In press) Differential item functioning and health assessment. Quality Life Research.

Reeve, B.B., Hays, R.D., Bjorner, J.B., Cook, K.F., Crane, P.K., Teresi, J.A, Thissen, D., Revicki, D.A., Weiss, D.J., Hambleton, R.K., Liu, H., Gershon, R., Reise, S.P., Lai, J-S., Cella, D. Psychometric evaluation and calibration o health-related quality of life item banks: Plans for the patient-reported outcomes measurement information system (PROMIS). Medical Care, 2007; Vol. 45, No. 5, Suppl. 1, S22-S31.

Publications and dissemination related to differential item functioning and cross-cultural comparisons (joint efforts with other RCMAR sites)

Continuing work begun by CALME investigators in 1997, the CALME RCMAR commissioned and published several articles focusing on state-of-the art methodologies for cross-cultural evaluation of measures.  Members of the RCMAR measurement and methods cores faculty contributed several relevant methodological articles.  Additionally, an article from the CALME RCMAR was contributed to the two volume series edited by Mutran and Sudha (UNC RCMAR) devoted to health disparities research and published in Research on Aging.  

Ramirez, M., Ford, M., Stewart, A., Teresi, J. (2005) Measurement issues in health disparities research. Health Services Research, Vol. 40, No. 5, pp. 1640-1657

Publication of a volume of peer-reviewed articles related to measures for use with elderly, ethnically diverse populations

The following review articles were published in the first RCMAR-sponsored publication, edited by Skinner (External Advisory Board of CALME), Teresi ( CORE V Director), Holmes (Executive Committee of CALME), Stahl (NIA Project Officer) and Stewart (UCSF RCMAR).  The issue of Mental Health and Aging, 2001 (volume 7) was devoted to methodological and review articles of measures for use among ethnically diverse populations.  Contributions were from leading scholars in the area of measurement in this area. Guest associate editors were notable contributors to health disparities research.  Many of the contributors and associate editors were from RCMAR sites.  This volume was recently published as a book “Multicultural measurement in older populations.” Articles from RCMAR affiliates are listed below:

Measurement in Older Ethnically Diverse Populations: Overview of the Volume

John H. Skinner, EdD, Jeanne A. Teresi, EdD, PhD, Douglas Holmes, PhD, Sidney M. Stahl, PhD, and Anita L. Stewart, PhD

Some Methodological Guidelines for Cross-Cultural Comparisons Jeanne A. Teresi, EdD, PhD, and Douglas Holmes, PhD

Statistical Methods for Examination of Differential Item Functioning (DIF) With Applications to Cross-Cultural Measurement of Functional, Physical and Mental Health Jeanne A. Teresi, EdD, PhD

Acculturation, Ethnic Identity, Socioeconomic Status and Social Support Acculturation: Measures of Ethnic Accommodation to the Dominant American Culture John H. Skinner, EdD

Social Support: Clarifying the Construct With Applications for Minority Populations Elizabeth J. Mutran, PhD, Peter S. Reed, MPH, and S. Sudha, PhD

Performance of Cognitive Tests Among Different Racial/Ethnic and Education Groups: Findings of Differential Item Functioning and Possible Item Bias Jeanne A. Teresi, EdD, PhD, Douglas Holmes, PhD, Mildred Ramirez, PhD, Barry J. Gurland, MD, and Rafael Lantigua, MD

Cognitive Assessment Among Minority Elderly: Possible Test Bias Mildred Ramirez, PhD, Jeanne A. Teresi, EdD, PhD, Stephanie Silver, MPH, Douglas Holmes, PhD, Barry Gurland, MD, and Rafael Lantigua, MD

Cross-Cultural Assessment of Geriatric Depression: A Review of the CES-D and GDS Ada C. Mui, PhD, Denise Burnette, PhD, and Li Mei Chen, MSW

Applying Health Locus of Control and John Henryism Active Coping Theories to Older African American Adults Marvella E. Ford, PhD, Deanna Hill, MPH, Ameera Butler, BS, and Suzanne Havstad, MA

Use of Health-Related Quality of Life Measures in Older and Ethnically Diverse U.S. Populations Anna M. Napoles-Springer, PhD, and Anita L. Stewart, PhD

Publication/editing of a special issue devoted to measurement methods

Teresi (Columbia RCMAR), Stewart (UCSF RCMAR), Morales (UCLA RCMAR) and Stahl (NIA) edited a special volume of articles published in Medical Care, November 2006, Vol. 44, No. 11.  These articles are examples of qualitative and quantitative methods for use in examining cross-cultural invariance. Following is the Table of Contents of the volume:

SPECIAL ISSUE OF MEDICAL CARE ON MEASUREMENT IN A MULTI-ETHNIC SOCIETY

Preface

The National Institute on Aging's Resource Centers for Minority Aging Research:  Contributions to measurement in research on ethnically and racially diverse populations (Sidney M. Stahl, Anneliese A. Hahn, NIA)         

 Overview of the Volume:  Editors summary

Measurement in a multi-ethnic society: Overview to the special issue (Jeanne A. Teresi, Anita L. Stewart, Leo Morales, Sidney Stahl)

 Qualitative Issues and Approaches

Overview of Qualitative Methods

Overview of qualitative methods in research with diverse populations: Making research reflect the population (Anna M. Nápoles-Springer, Anita L. Stewart, UCSF RCMAR)

            Conceptual Issues

Deconstructing race and ethnicity: Implications for measurement of health outcomes (Jennifer J. Manly, Columbia RCMAR)

   Qualitative Methods

                        Review of Methods  

Methods and frameworks for cross-cultural measurement (Timothy Johnson, University of Illinois: Chicago)

                        Examples and Applications                                           

Using cognitive interviews to develop surveys in diverse populations (Anna M. Nápoles-Springer, Jasmine Santoyo, Helen O’Brien, Anita L. Stewart, UCSF RCMAR)

Commentary: Narrative and the fundamental limitations of quantification in cross-cultural research (Ronald J. Angel, University of Texas: Austin) 

Commentary: The use of qualitative methods to improve quantitative measures of health-related constructs (Neal Krause, University of Michigan)

Quantitative Issues and Approaches

Overview of Quantitative Issues and Methods

Overview of quantitative measurement methods:  Equivalence, invariance and differential item functioning in health applications (Jeanne A. Teresi, Columbia RCMAR)

Classical Test Theory (CTT) and Item Response Theory (IRT)

Classical test theory (CTT) (Robert F. DeVellis, University of North Carolina, former RCMAR)

Classical test theory and item response theory analyses of multi-item scales assessing parents' perceptions of their children's dental care (Ron D. Hays, Julie Brown, Lorraine U. Brown, Karen L. Spritzer and James J. Crall, UCLA RCMAR) 

Factorial Invariance:   Concepts and Examples

An essay on measurement and factorial invariance  (William Meredith and Jeanne A. Teresi) UC Berkeley and Columbia RCMAR)

Do self report instruments allow meaningful comparisons across diverse population groups?  Testing measurement invariance using the confirmatory factor analysis framework(Steven E. Gregorich, UCSF)

Differential Item Functioning (DIF)

Differential item functioning (DIF) and the Mini-mental State Examination (MMSE):  Overview, sample, and issues of translation (Mildred Ramírez, Jeanne A.Teresi, Douglas Holmes, Barry Gurland and Rafael Lantigua, Columbia RCMAR)

Differential item functioning on the Mini-Mental State Examination:  An    application of the Mantel-Haenszel and standardization procedures (Neil J.    Dorans, Edward Kulick, Educational Testing Services)

DIF analysis with ordinal logistic regression techniques:  DIFDETECT and    difwithpar (Paul K. Crane, Laura I. Gibbons, Lance Jolley, Gerald van Belle,    University of Washington)  

Identification of measurement differences between English and    Spanish    language versions of the Mini-Mental State Examination:  Detecting differential    item functioning using MIMIC modeling (Richard N. Jones, Harvard &    Columbia RCMAR)

Identification of differential item functioning using item response theory    and the    likelihood-based model comparison approach: Application to the Mini-Mental    State Examination (Maria Orlando, RAND, David Thissen, University of North    Carolina, Jeanne Teresi, Marjorie Kleinman, Katja Ocepek-Welikson, Columbia    RCMAR)

Item and scale differential functioning of the Mini-Mental State Exam assessed using the differential item and test functioning (DFIT) framework (Leo S. Morales, UCLA RCMAR, Claudia Flowers, University of North Carolina, Peter Gutiérrez, UCLA, Marjorie Kleinman, Jeanne Teresi, Columbia RCMAR)

Summary of Advantages and Disadvantages of DIF methods

Different approaches to differential item functioning in health applications: Advantages, disadvantages and some neglected topics (Jeanne A. Teresi, Columbia RCMAR)

Commentaries

Comments on methods for the investigation of measurement bias in the Mini-Mental State Examination (Roger E. Millsap, Arizona State University)

When does measurement invariance matter? (Denny Borsboom, University of Amsterdam) 

Good practices for identifying differential item functioning (Ronald K. Hambleton, University of Massachusetts)

Item Banking and computerized adaptive testing

Item banks and their potential applications to health status assessment in diverse populations (Elizabeth A. Hahn, David Cella, Rita K. Bode, Richard Gershon, Jin-Shei Lai, Northwestern University)

Commentary:

Special issues for building computerized-adaptive tests for measuring patient-reported outcomes: The National Institute of Health’s investment in new technology. (Bryce B. Reeve, National Cancer Institute)

Epilogue:

Assessing and understanding measurement equivalence in health outcome measures: Issues for further quantitative and qualitative inquiry (Coleen A. McHorney, John A. Fleisman)

Measurement and Methods Symposium at the Gerontological Society Meetings in 2006

A symposium (linked to the GSA measurement and methods interest group) was organized around the RCMAR collaborative study. Four presentations related to the project were presented by junior investigators (F. Yang (Columbia), K. Mehta (UCSF), Heslin (UCLA) and C.Yang (MUSC) in collaboration with senior mentors (Jones, Kleinman, Ramirez, Teresi, Welikson (Columbia), Tilley (MUSC), Hays, Morales (UCLA), Stewart (UCSF) and Mungas (UC-Davis ADRC), and external advisors, Goldberg (Colorado) and Templin (Michigan/Wayne State). Four methods were presented, together with a summary.

Pre-Conference at GSA 2005: Approaches for Developing/Adapting/Evaluating Health-related Concepts and Measures in Diverse Older Populations

The Pre-conference at GSA 2005 was co-sponsored by the Research in Quality of Care (RESQCARE) Formal Interest Group, co-directed by Jeanne Teresi; the Gerontological Society of America's Task Force on Minority Issues; National Institutes of Health; and the National Institute of Aging. This Pre- conference consisted in a full-day workshop where methodological approaches used in minority health and health disparities research were discussed. Methods for testing the equivalence of surveys were presented; and approaches involving the integration of qualitative and quantitative methods in an iterative manner to develop new, equivalent measures across groups were reviewed. Members from all the RCMAR sites participated as presenters. Columbia RCMAR presenters included Jeanne Teresi on item response theory and Rich Jones on multiple indicators, multiple cause (MIMIC) analyses.

      RESQCARE/RCMAR Symposium at GSA 2004

The following symposium at the Gerontological Society of America (GSA) meetings was sponsored jointly by formal interest group Research in Quality of Care (RESQCARE) Chaired by Jeanne Teresi and by the RCMARs. Members from three of the RCMAR sites ( Columbia, UCLA, UCSF) and the NIA project director, Dr. Stahl participated. Following is a summary of the symposium.

CULTURAL DIVERSITY, CULTURAL COMPETENCE AND QUALITY OF CARE

J. Teresi (Research Division, The Hebrew Home for the Aged at Riverdale, 5901 Palisade Ave., Riverdale, NY 10471, & Columbia University, RCMAR)

M. Calkins (IDEAS Institute, Kirtland, Ohio)

PARTICIPANTS:

M. RamRrez (HHAR-Research Division, Riverdale, NY) "Cultural diversity and quality of care: Demoralization and attitudes toward residents among certified nursing assistants (CNAs) in relation to job stressors and work resources”

V.A. Parker a,b, D.A. Tyler b, S.M. Geron c (Department of Veterans Affairs a , Boston University School of Social Work c, Boston University School of Public Health b, Health Services Department, Boston, MA) “Intervening in organizational cultural competence”

A.L. Stewart, 1 A.M. Nápoles-Springer, 2 S. Gregorich, 2 J. Santoyo, 1,2 ( 1Center for Aging in Diverse Communities, 2Institute for Health & Aging, UCSF, San Francisco, CA.) “Measures of interpersonal processes of care (IPC): Psychometric invariance in Latino, African American, and White patients”

R. Weech-Maldonado (Department of Health Policy & Administration, The Pennsylvania State University, PA) “Cultural Competency Assessment Tool for Hospitals: An Application of the Culturally and Linguistically Appropriate Services ( CLAS) Standards”

DISCUSSANTS:

S. Stahl (Behavioral and Social Research Program, National Institute on Aging/National Institutes of Health, Bethesda, MD)

Updating of intermediate/advanced measurement and methods reading/reference list

This list was updated and is continuing to be kept current

 d.)  Significance: Unchanged

 e.)  Plans:

The Measurement and Methods Cores (MMC) plans to continue with the quarterly mentoring meetings.  The MMC will continue the effort to create a website that can be a resource to NIH investigators for seeking measures that are relatively more culture-fair. The MMC will continue to populate the CALME measurement website with Measurement Evaluation Grids and reviews of instruments. The MMC will continue to work with the other RCMAR Measurement Cores in the preparation of a grant proposal to continue the cross-site collaboration study initially funded through RCMAR. The MMC will continue to provide hands-on-training at the Research Division labs. Finally, the MMC will continue to advance the science of measurement and methods through (1) ongoing work with members of the Columbia and New York University biostatistics departments and Drs. Luchsinger, Carrasquillo, Palmas and Manly in the development of methods for longitudinal analyses of health and cognition; (2) ongoing collaboration with PROMIS roadmap staff across the country in creating items banks and computerized adaptive testing applications for use in health assessment of minority and majority, older and younger cohorts.

Measurement and Methods Resources: For additional resources click HERE