In this paper, we describe a large-scale, long-term study of a National Science Foundation undergraduate research program via the Computer and Information Science and Engineering (CISE) Research Experiences for Undergraduates (REU) Evaluation Toolkit project. The overarching research questions are: what attitudinal shifts occur for undergraduate research experience student participants, and do experience perceptions differ based upon student demographics? Results across two cohort years showed statistically significant gains for several key program level outcomes. However, intent to go to graduate school decreased in both cohorts. Gender and ethnicity differences were observed. Women and underrepresented minority group students had significantly higher scores for several factors. These findings are discussed in context of culturally relevant pedagogy for undergraduate research programs. This study is unique in its scale and scope, providing a depiction of student outcomes across multiple cohorts of a national sample including over 58 computing research sites and more than 900 undergraduate students.