CPB GRANT EVALUATION PLAN

1. EVALUATION STRATEGY
   
   
   1.1. Identify model to be used
   1.1.1. The Concerns Based Adoption Model (Hall, 1979)
   1.1.2. Joint Venture: Silicon Valley Network Indicators
   1.1.3. Others?
   1.2. Process
   1.2.1. Establish community baseline
   1.2.2. Identify key indicators.
   1.2.3. Each project team sets goals and objectives in terms that define certain measurable outcomes for these key indicators.
   1.2.4. Collect data on key community indicators.
   1.2.5. Monitor external events that may have an impact on the indicators.
   1.2.6. Analyze data and report at three points:
   1.2.6.1. Initiation (establishes baseline)
   1.2.6.2. Interim (mid-point for course correction)
   1.2.6.3. Final (Assess project impact and final outcome)
   
   
2. CRITERIA FOR MEASURING EFFECTIVENESS OF THE PROJECT TOWARD REACHING GOALS DEFINED.
   
   
   2.1. Sustainability Measurements
   2.1.1. Level of community ownership
   2.1.2. Match of project outcome to high-priority community goals
   2.1.3. Community awareness of projects' accomplishments
   2.1.4. Flexibility and responsiveness to marketplace and technology changes
   2.1.5. Use of community strengths
   2.1.6. Effectiveness as unifying/catalyzing organization
   2.1.7. Accountability demonstrated to community
   2.1.8. Focus vs diffusion of efforts
   2.2. General parameters, all projects
   2.2.1. Stakeholder satisfaction
   2.2.2. Difference in stakeholder attitude resulting from specific project components
   2.2.3. Penetration in the targeted stakeholder communities
   2.2.4. Level of use of new information and analysis tools
   2.2.5. Evidence of successful application to problem-solving and consensus-building (around environmental issues in particular)
   2.2.6. Technical evaluation of tools selected and implemented
   2.2.7. Rate of increase in quantity of information available to general public
   2.2.8. User evaluation of quality and usefulness of information
   2.3. Key community-building indicators.
   2.3.1. Citizen participation levels.
   2.3.2. Civic participation, by population
   2.3.3. Government-government contacts.
   2.3.4. Use of on-line forums.
   2.3.5. Number of virtual organizations formed, and their nature.
   2.4. Volunteer management system indicators
   2.4.1. Number of volunteers recruited (and volunteer hours contributed) for this and other civic efforts through the volunteer management system
   2.4.2. From volunteer's point of view
   2.4.2.1. Accessible from Internet/specifically with web interface
   2.4.2.2. Privacy/security for personal data
   2.4.2.3. Ease of use
   2.4.2.4. Can see volunteer opportunities
   2.4.2.5. Can create self-profile and have likely opportunities flagged (DCN volunteer)
   2.4.2.6. Or can register and be personally notified by Volunteer Center of matches (necessary for screening, etc.)
   2.4.2.7. Can apply for volunteer opportunities on-line
   2.4.2.8. Application is easy; two-tiered process (basic information first, details later)
   2.4.2.9. Can easily update information (skills, etc.)
   2.4.2.10. Once application submitted, volunteer hears back quickly from app.person
   2.4.2.11. Can enter volunteer time worked
   2.4.2.12. Volunteer time is tracked, and credited
   2.4.2.13. Volunteer credits can be used for other services (catalogue for selection?)
   2.4.2.14. Can communicate by e-mail with others on project
   2.4.2.15. Is recognized for work
   2.4.2.16. Doesn't have to have a computer to participate in on-line system (access or alternate modes)
   2.4.3. From volunteer coordinator/Executive Director's point of view
   2.4.3.1. System provides way to post both specific and general volunteer opportunities
   2.4.3.2. Committee chairs can post their own volunteer opportunities
   2.4.3.3. Volunteers can enter their own data (see 2.1.5)
   2.4.3.4. System provides tools to develop sense of community and camaraderie
   2.4.3.5. System helps coordinator quickly and easily keep track of volunteers' names, addresses, phones, skills, hours served, birth dates, schedules, status, educational background
   2.4.3.6. System provides help in matching volunteers' skills to opportunities and needs (standard and ad hoc queries)
   2.4.3.7. System supports scheduling (automatically generates schedules, based on the assignments allows override of the schedule where necessary)
   2.4.3.8. For standard "roles," system identifies who is assigned to which role, the status of the assignments -eg requested, placed, start dates, end dates, frequency, comments about the assignment
   2.4.3.9. System automatically collects time worked
   2.4.3.10. System provides verification (through chair, etc.) of time worked
   2.4.3.11. System provides justifiable valuation of time worked
   2.4.3.12. System creates necessary matching contribution reports for grants
   2.4.3.13. System creates other reports (standard as well as on-the-fly; labels for mailings; automataic e-mail messages; need specifications for standard reports)
   2.4.3.14. Tracking data is reasonably secure (limited access)
   2.4.3.15. System tracks who is "supervising" volunteer (intern requirement)
   2.4.3.16. System tracks orientation, training steps taken, and volunteer achievements
   2.4.3.17. System helps identify where volunteer found out about opportunities, best practices, etc.
   2.4.3.18. Platform supported on/open architecture/standards/database, etc.
   2.4.3.19. Portability (CPB requirement if developed in-house)
   2.4.3.20. Security
   2.5. GIS Project indicators
   2.5.1. Qualitative
   2.5.1.1. How is GIS being used (before and after)?
   2.5.1.2. How have decision processes changed?
   2.5.1.3. Are better decision being made (ownership, external assessment, citizen involvement, etc. - need to define how to assess decision as "better").
   2.5.1.4. Recorded user comments (unsolicited testimonials)
   2.5.2. Quantitative
   2.5.2.1. User demographics and their application of the tools.
   2.5.2.2. Obtain baseline data from other agencies about use of GIS.
   2.5.2.3. How many places did you have to go to get data before project; how many after? (fragmentation of sources)
   2.5.2.4. Web form questionnaire (hits, uses, reactions)
   2.5.2.5. Number of hours/miles saved in obtaining data after vs before.
   2.6. Document Database Project indicators
   2.7. Integrated Document Database/GIS indicators
   2.8. Community Outreach and Demonstration Centers Project indicators
   2.8.1. Numbers and categories of community members attending workshops, and their evaluation of the workshops
   
   
3. EXTERNAL FORCES THAT WOULD AFFECT REACHING THE GOALS DEFINED (THOSE NOT WITHININ THE PROJECT'S CONTROL)
   
   
4. DATA TO BE COLLECTED
   
   
5. EVALUATION TECHNIQUES AND INSTRUMENTS
   
   
   5.1. Surveys
   5.1.1. General population survey
   5.1.2. Target population survey
   5.1.3. Service provider survey (volunteer agencies)
   5.1.4. Key informant survey
   5.2. Secondary analysis of existing data to set baseline
   5.2.1. Relevant Indicators (justify relationship to goals)
   5.3. Review of managerial and administrative records
   5.4. Group Processes
   5.4.1. Town meetings
   
   
6. ANALYSIS TO BE PERFORMED
   
   
7. HUMAN SUBJECTS PROTOCOL
   
   [Need to get ok from Office of Vice Chancellor for Research on Human Subjects protocol, since the University is involved]