SATPC0030901 Tab 04 6 Evaluation Criteria -UPDATED.pdf

Audience Research and Content Evaluation for the

Earth Information Center Exhibition

Evaluation Basis:

Excellent: A thorough and compelling proposal of exceptional merit that fully responds to the objectives of the solicitation, as documented by numerous or significant strengths and with no major weaknesses.

Good: A competent proposal representing a credible response to the solicitation, with strengths and weaknesses, essentially balancing each other.

Poor: A seriously flawed proposal, having one or more major weaknesses that constitute fatal flaws. Not technically acceptable.

Evaluation Factors:

1. Formative evaluation approach: The Offeror shall describe its approach and ability to conduct formative evaluations both on-site and via virtual platforms. Approach shall describe representative audience selection and recruitment, test methodologies, qualitative and quantitative metrics to be used, and reporting and recommendation practices.

2. Summative evaluation approach: The Offeror shall describe its approach and ability to conduct summative evaluations both on-site and via virtual platforms. Approach shall describe representative audience selection and recruitment, test methodologies, qualitative and quantitative metrics to be used, and reporting and recommendation practices.

3. Technical Experience: The Offeror shall describe its understanding of and experience working with museum audiences and communicating complex ideas. Technical experience is demonstrated by including large data visualizations, virtual user engagement, storytelling, and personal narratives through the lens of earth science communication.

4. Subject matter expert availability: The Offeror shall describe its ability to obtain subject matter experts aligned with the technical area and provide availability of key personnel.

5. Past performance: The Offeror shall provide relevant experience and past performance in developing similar audience and content evaluation. Substantive evidence of successful participation in similar developments should be included which describes quantity of prior work, methodology, scope and complexity, and quality of deliverables.

Evaluation Factors:

1. Formative evaluation approach: The Offeror shall describe its approach and ability to conduct formative evaluations both on-site and via virtual platforms. Approach shall describe representative audience selection and recruitment, test methodologies, qualitative and quantitative metrics to be used, and reporting and recommendation practices.

Excellent

• Online and onsite focus groups (5 or more in total)

• Description of how online and onsite focus groups will be organized, moderated, and compensated.

• Use of feedback tools for onsite focus groups (e.g., card sorting, ethnographic camera studies)

• Substantial iterative feedback built into the approach between the evaluators and the EIC team and content design team at the formative stage (e.g., on-site intercept testing, in situ observation of traffic flow, pre-, during, and post- interviews, user testing of UI/UX design elements, anonymous feedback capture boards, user driven prototyping).

• Synthesis and processing of the data and insights from the formative interview using design-based tools (e.g experience or empathy mapping, personas, journey maps) and quantitative and qualitative takeaways.

Good

• Online and onsite focus groups (5 or less).

• Description of how online and onsite focus groups will be organized OR moderated OR compensated.

• Online and onsite focus groups use direct questions more than feedback tools.

• Minimal iterative feedback on the results with the EIC team and content design team at the formative stage.

• Deliverable of formative testing includes quantitative and qualitative takeaways.

Poor

• Either online OR onsite focus groups, but not both.

• No description of how online and onsite focus groups will be organized, moderated, and compensated.

• No description of iterative feedback on the results with the EIC team and content design team at the formative stage.

• No use of feedback tools in the online/on-site focus groups

• No use of design-based tools. Either qualitative or quantitative takeaways, but not both.

2. Summative evaluation approach: The Offeror shall describe its approach and ability to conduct summative evaluations both on-site and via virtual platforms. Approach shall describe representative audience selection and recruitment, test methodologies, qualitative and quantitative metrics to be used, and reporting and recommendation practices.

Excellent

• Use of surveys, in-person interviews, and observations of visitor engagement.

• Survey approaches involve a random sample of visitors collected over at least 2 months.

• Surveys follow up with visitors before entering the exhibit, immediately after, and 2 weeks or more after visiting the exhibit.

• Surveys will be collected from 200 participants or more.

• Interviews will include a sample of more than 20 persons.

• Observations of visitor engagement will be both quantitative and qualitative.

Good

• Use of 2/3: surveys, in-person interviews, and observations of visitor engagement.

• Survey approaches involve a random sample of visitors collected over at least 1 month.

• Surveys follow up with visitors in two out of the three time periods: before entering the exhibit, immediately after, and 2 weeks or more after visiting the exhibit.

• Surveys will be collected from 100 participants or more.

• Interviews will include a sample of more than 10 persons.

• Observations of visitor engagement are quantitative or qualitative.

Poor

• Use of 1/3: surveys, in-person interviews, and observations of visitor engagement.

• Survey approaches involve a random sample of visitors collected for less than 1 month.

• Surveys follow up with visitors only one out of the three time periods: before entering the exhibit, immediately after, and 2 weeks or more after visiting the exhibit.

• Surveys will be collected from less than 100 people.

• Interviews will include a sample of less than 10 persons.

• There are no observations of visitor engagement.

3. Technical Experience: The Offeror shall describe its understanding of and experience working with museum audiences and communicating complex ideas. Technical experience is demonstrated by including large data visualizations, virtual user engagement, storytelling, and personal narratives through the lens of earth science communication.

Excellent

Experience with 4 out of 4

• Experience working with Earth science data and concepts with past clients.

• Discussion of experience working with and evaluating large data visualizations

• Discussion of virtual user engagement strategy (e.g., online tools for video interviewing, UI/UX testing)

• Discussion of experience crafting and evaluating storytelling (mini-documentaries, podcasts, interactive learning experiences)

Good

Experience with 3 out of 4

• Experience working with Earth science data and concepts with past clients.

• Discussion of experience working with and evaluating large data visualizations

• Discussion of virtual user engagement strategy (e.g. online tools for video interviewing, UI/UX testing)

• Discussion of experience crafting and evaluating storytelling (mini-documentaries, podcasts, interactive learning experiences)

Poor

Experience with 2 or less

• Experience working with Earth science data and concepts with past clients.

• Discussion of experience working with and evaluating large data visualizations

• Discussion of virtual user engagement strategy (e.g. online tools for video interviewing, UI/UX testing)

• Discussion of experience crafting and evaluating storytelling (mini-documentaries, podcasts, interactive learning experiences)

4. Subject matter expert availability: The Offeror shall describe its ability to obtain subject matter experts aligned with the technical area and provide availability of key personnel.

Excellent

5-6 out of 6 below are part of the team

• Visitor experience/ Public engagement specialist

• Data visualization expert

• Design expertise

• Science education/ content development specialist

• Data analysis expertise

• At least one member of the team has a PhD or research background

Good

3-4 out of 6 below are part of the team

• Visitor experience/ Public engagement specialist

• Data visualization expert

• Design expertise

• Science education/ content development specialist

• Data analysis expertise

• At least one member of the team has a PhD or research background

Poor

1-2 out of 6 below are part of the team

• Visitor experience/ Public engagement specialist

• Data visualization expert

• Design expertise

• Science education/ content development specialist

• Data analysis expertise

• At least one member of the team has a PhD or research background.

5. Past performance: The Offeror shall provide relevant experience and past performance in developing similar audience and content evaluation. Substantive evidence of successful participation in similar developments should be included which describes quantity of prior work, methodology, scope and complexity, and quality of deliverables.

Excellent

• Experience working with the Smithsonian Museum of Natural History

• Experience evaluating at least 3 other museum exhibits.

Good

• Experience working with other Smithsonian museums and science museums.

• Experience evaluating at least 2 other museum exhibits.

Poor

• Experience working with either the Smithsonian or another science museum.

• Experience evaluating at least 1 other museum exhibit.