Evaluations
 Evaluations are an important tool in building and maintaining organizational effectiveness. In nonprofit organizations, they also play another critical role. Because limited resources are devoted to the organization, DREAM has an obligation to determine the value to society that it produces. This obligation starts with the Board of Directors and runs through the staff and every volunteer. Without a fair-market determination of value through charging families for our services, we are reliant on evaluations to demonstrate our impact.
DREAM has taken on many evaluation initiatives and has developed a comprehensive and longitudinal evaluation system for the following target groups:
- Children: exploring the impact that DREAM has on its participants' well being.
- Mentors: understanding the impact DREAM has on its student mentors as well as exposing the influence that the mentors have on their DREAM partners.
- Parents/Caregivers: showing the influence that DREAM has on the families and neighborhoods involved in the program, as well as the affect of DREAM on the child participants as perceived by parents or caregivers.
- Alumni: understanding any long term influences DREAM has on its mentors.
Listed below are our annual Impact Studies, a summary of our evaluative methods and goals, and results from surveys collected before the implementation of our current system.
Annual Impact Studies
DREAM, with the help of Evaluation Consultant Cynthia Char has completed its second impact study utilizing surveys from 2004 and 2005. We are very excited about the positive impacts the program is having and hope that you will find this information valuable. You can download and read the entire report here...
 DREAM 2003-2004 Evaluation Report!
Children
This set of evaluation instruments has been developed for The DREAM Program to help the organization in their assessment of participants’ needs, and of the program’s impact on individual participants, families and communities. The instruments are designed to accommodate the regular "fall through spring" school calendar cycle of the DREAM program.
Two different types of evaluation instruments have been developed. The first is a "New DREAMer" background information survey for all children and youth beginning the program. It contains a variety of questions concerning an individual’s current interests, activities, and level of engagement in their school and community, and provides baseline information and a "snapshot in time" profile for each individual. Specifically, there is an open response section, a self-efficacy questionnaire and a future aspirations questionnaire. This survey will be administered orally by his/her DREAM mentor or another member of the DREAM staff, as a "getting to know you" interview. The questionnaire will be administered some time during the first four weeks of an individual’s entry into DREAM. For children who are already involved in DREAM, the form will be administered this fall (Fall 2004) to gather information on the current status of participants.
The second type of instrument developed is a post-program survey, administered in late spring towards the end of the DREAM fall-spring program cycle. This questionnaire is designed to assess DREAM’s program impact on child participants, including the areas of self efficacy and self-confidence, competence, interpersonal connections and relationships, and healthy behaviors. A core set of items are drawn from instruments suggested by the Vermont Mentoring Partnership, with a follow up section that mirrors the future aspiration questionnaire of the “New DREAMer” survey as well as some DREAM specific open response questions. Of central interest is the extent to which DREAM has encouraged a broader set of physical, social and psychological experiences in the world, an expanding view of both present opportunities and future possibilities in the world, and increased sense of self, self efficacy, and group affinity. There are two versions of this post-program survey: one for young children and one for older children. These surveys are administered in late spring, approximately nine months after the start of the fall program.
The information is gathered at the individual DREAM sites across the state. Once collected, all data is stored at the central DREAM office. Findings are reported either as a group aggregate, or with individual pseudonyms. Annual results are written up in a formal Impact Study, which are available for download above.
Mentors
Oftentimes the positive impact DREAM has on its child participants overshadows the good that the organization is doing for its mentors. As truly student run programs, there are a wealth of opportunities available to DREAM mentors to learn and hone transferable skills as well as to reap personal satisfaction from their efforts.
Mentors are responsible for governing their local programs, which means learning to handle such logistics as recruiting, fundraising, driving, and planning trips. The mentors also learn to effectively communicate with parents, community members, college administrators, peers, and children. Perhaps most importantly, the mentors continually build on their ability to be future parents and caregivers!
Through all their activities, it is perhaps the most rewarding to watch our mentors dream big. They continue to enthusiastically stretch the organization. In the past few years, mentors have brought children on "High Adventures" (summer trips) to California, Colorado, Washington, and Utah, and are planning a trip this coming summer to Alaska. With the Central Office available to support mentors’ endeavors, it is obvious that their pursuit of DREAM’s mission teaches them valuable, practical skills.
In addition to transferable skills, DREAM provides mentors with a forum in which to develop stable and supportive social networks amongst their peers, have personal satisfaction in their lives, and overall have a richer and happier experience at college. Mentors explore themselves and the world alongside the children. It is not uncommon for a mentor to send an email out to a Local Program’s email list like this recent note:
"Hey DREAMers, I have to say that this weekend has been the best weekend of my life, seriously. I have never hung out with such cool people for such a long time and had so much fun doing good for the world. But then again, I do DREAM, that is what I have come to expect every Friday.... and Saturday, and Sunday...........why not... and Monday through Thursday...."
DREAM created an online mentor survey to assist its Local Programs in gathering ideas and feedback, and also to assess the impact that DREAM is having on its mentors. The survey, collected every January, focuses on unique questions dictated by individual programs as well as general questions formulated from "Ends" set forth by DREAM’s Board of Directors. The "Ends" state that mentors will:
- have lasting and positive mentoring and peer relationships,
- gain an enduring caring for and commitment to the DREAM families,
- have a higher quality of experience at college than they would have had without DREAM,
- develop transferable skills that assist them in being catalysts of change, and
- have ownership in their local programs.
Here are examples of the mentor survey:
Findings are reported either as a group aggregate, or with individual pseudonyms. Annual results are written up in a formal Impact Study available above for download, exposing both DREAM’s significance in the mentors’ lives and also target areas in need of work. Data pertinent to individual programs is forwarded to the program’s chairs. Some additional results from past surveys and write-ups are available for download below:
- 2004 Mentor Survey Report
- 2003 Mentor Survey Report: In the spring of 2003, before our annual survey was instituted, mentors in the Elm Street DREAM program conducted their own mentor survey. The results of the survey demonstrate the mentors’ perceptions of the organization, its effectiveness, and improvements that could be made. Highlights from this survey are below:
The survey also asked questions about the mentors’ motivation for DREAM. The results from this set of questions illustrate the mentors’ excitement for the program, and also provide an indication of the variety of DREAM-related activities in which mentors engage beyond the Friday program:
Finally, the survey provided mentors with insight into their own program. A few of the major trends that they identified include:
- A strong feeling of empowerment among mentors: we are realizing our ability to make a positive impact on the lives of DREAMers.
- A desire for more one-on-one time during Friday meetings
- More discipline and organization for Fridays and in general
- Need for faster matching-up of new mentors with kids
- More variety of DREAM activities (Fridays and otherwise)
In the fall of 2002, DREAM conducted an internal evaluation for strategic planning purposes that we called the "Local Program Exploration." The effort centered around focus groups, interviews, and discussions with mentors, parents, and children in the various Local Programs. Since then, the "Local Program Exploration" has been incorporated into the annual mentor survey and solicits questions directly from the chairs of each program. The following recommendations came out of the 2002 Local Program Exploration:
Parents and Neighborhoods
DREAM is managed by both mentors and parents, and as such provides additional opportunities for parents to be involved in their children’s lives and to share ideas on how to strengthen their neighborhoods. Parents are invested members of the program, making many important contributions, and appreciate the work the program does in their communities. One DREAM father spoke to the value he places on the program to a group of mentors and fellow parents:
"I don't know if it's like this for every parent here, but I have a really good relationship with my kids' mentors. I mean, if they came to my house and asked to borrow my TV and I was watching the Super Bowl, I would give them my TV."
DREAM surveys the parents of its child participants to both corroborate findings from its child-surveys and also to determine what affects the program is having on parents themselves and their larger neighborhoods. Specifically, DREAM hopes that the parents:
- have positive, healthy interactions with their neighbors, and
- are actively engaged in the running of DREAM.
The parent survey, collected every March, is distributed to two groups of community parents from demographically similar neighborhoods. One group of parents are parents of DREAM children and the other group, the survey’s “control” group, do not have DREAM in their neighborhood. The survey packet given to DREAM parents asks the same set of general questions as the “control” survey, and also asks a series of DREAM focused questions. In addition to these DREAM focused questions, there is an optional page that provides direct feedback to the Vermont Children’s Forum, a youth advocacy organization.
Here are examples of both the “control” and DREAM parent surveys:
Findings are reported either as a group aggregate, or with individual pseudonyms. Annual results are written up in formal Impact Studies available for download above, exposing DREAM’s significance in the parents’ lives, the worth of DREAM to their children, and target areas in need of work.
In addition to these studies, there are some significant results from past surveys. In August of 2002, DREAM performed its first round of surveys with parents. Surveys were collected from the eight families involved in the newly-formed DREAM program at the Elm Street Apartments in Winooski. The results from these surveys were extremely encouraging.
Below is a chart illustrating the results on some sample questions.
The survey also collected anecdotal information from parents. Below are a sample of some of their comments:
"The program really seems to help foster the children's' self-esteem and teaches them how to build positive peer relationships. The mentors are positive role models for all kids."
"They can't wait until Friday when they can get out and do different things and enjoy the one on one time also."
"All the children have nothing but positive things to say about the program."
"This is the best program for my kids because they’re always bored—it’s lifted their self-esteem, gave them something to look forward to. I really hope they continue this program because with a large family sometimes I don’t get the time with them and DREAM is wonderful! I really hope they continue."
To see more about the Elm Street Parent Survey conducted in August of 2002, please peruse these survey results:
Alumni
Coming soon...
Daring to DREAM
 In the 2002-2003 academic year, Katherine Meyer, a senior at the University of Vermont, conducted an evaluation of DREAM for her honors thesis. She was advised by Heather Bouchey, an Assistant Professor in UVM's Department of Psychology. The thesis was titled "Daring to DREAM: An Evaluation of a Local Mentoring Program" and centered around a survey conducted with 80 children from three Local Programs. The surveys were conducted in two times with the same group of children, in October and March, to gather time-series data. Mentors were also surveyed to determine if certain mentor characteristics influenced the program's impact on children.
The study set out to determine the effect of DREAM on characteristics such as the children’s self-efficacy, self-esteem, likelihood and interest in future travel experiences, and perception of their own academic ability. The study faced many obstacles such as an abbreviated timeframe, a small sample size, and the lack of a control group, but we were excited to have a great start on developing a comprehensive evaluation system for DREAM.
Katherine’s study was presented at a conference of the American Psychological Association that took place in August, 2003.
|
