Attachment J.14_ECMP Demonstration_Scope Objectives and Evaluation Criteria.docx

ECMP

Request for Proposal ECON Attachment J.14 – ECMP Demonstration – Scope, Objectives, and Evaluation Criteria Offerors may combine two or more scenarios into a single scenario wherever it makes sense to do so. As long as all the required scenarios are demonstrated, the Offerors are free to combine them into a smaller number of scenarios.

Some scenarios need to be demonstrated to an OCC audience on the scheduled demo date. Other scenarios, because they will involve potentially long-running programs, will be executed offline by the Offeror’s team, with documentary evidence of the scenario’s execution to be brought to the scheduled demo and provided to the OCC audience. Scenarios that can be executed and documented offline are identified within the table below. Offerors should assume that any scenario that does not include a reference to offline execution in its supporting details will, by default, be demonstrated live in front of an OCC audience.

Table J.14–1 ECMP Demonstration – Scope, Objectives, and Evaluation Criteria

· Database schema

· Evidence of the Input / Output (I/O) throughput achieved [footnoteRef:1] [1: Please note that the performance level achieved by the Offeror in its demonstration will be used as the performance level to determine the baseline for system throughput documented In the OCC’s SLA. This level should be, at a minimum, 75 MB/seconds. The OCC expects the Offeror to demonstrate achievable performance during the demo, rather than a level of performance that cannot be sustained when the solution is implemented and operated.]

· If available: an algorithm or formula from which the OCC can extrapolate an estimate of the size of the storage for the total solution The OCC will provide the full LPC data set as a SAS 9.2 file with an index. The Offerors should load this into their respective databases to create the baseline environment. Then Offerors will receive multiple small zip files (containing ASCII files) that they will need to explode and load into the existing data environment. This scenario simulates the monthly cycle of updating an existing set of data with new sets.

The OCC will provide the Offerors with the description of two business problems that need to be solved using this data. One problem will require that a large amount of data—but not all of the columns—be used. The other problem will require that a large number of columns—but not all of the data—be used. These two types of problems are typical of how the economists use this data.

Objectives=1) to demonstrate the extraction, transformation, and loading of the data; 2) to document the proposed database’s ability to achieve the OCC’s required level of I/O throughput; and 3) to understand how the proposed database deals with data compression and storage overhead.

This scenario can be executed offline, in advance of the scheduled demo date, with evidence of the scenario’s execution and throughput levels documented by the deliverables described in the “Deliverables from Offeror” column.

1. To what extent was the function demonstrated with little or no customization of a Commercial Off-the-Shelf (COTS) tool? (That is, can this function be supported “out of the box”?)

2. Was the scenario executed completely, partially, or not at all?

3. Were all deliverables associated with this scenario provided?

4. Did the Offeror demonstrate a correct understanding of the two business problems utilized to frame this scenario?

· Apply security permissions to loaded data file

· Report of the metadata associated with the loaded data

· Executed queries against the metadata The OCC wishes to see the GUI displaying the data that was created at the time the LPC data was loaded (in Scenario #1). During the demo, the Offerors will be asked to assign one or more new metadata elements to the existing set of metadata. Once the additional metadata is added, the Offerors will be asked to query the metadata to display individual elements.

Objective=to demonstrate the features of the metadata management tool and provide insight into its robustness and ease of use

1. To what extent was the function demonstrated with little or no customization of a COTS tool? (That is, can this function be supported “out of the box”?)

2. Was the scenario executed completely, partially, or not at all?

3. Were all deliverables provided?

4. Does the metadata management tool appear to be robust? Does it seem to be relatively easy to use?

1. A clear and concise graphical depiction and narrative explanation of the flow of data during the promotion process

2. An opportunity to view the GUI(s) that would be utilized by end-users, data managers, and system administrators Objectives = 1) to convey an understanding of the OCC’s data-promotion requirements; 2) to articulate that understanding clearly in the discussion of the promotion process; and 3) to demonstrate that the promotion process is not unduly time-consuming or difficult.

1. To what extent was the function demonstrated with little or no customization of a COTS tool? (That is, can this function be supported “out of the box”?)

2. Did the Offeror provide a clear, concise explanation of the promotion process?

3. Did the Offeror show the interface to be used to perform the promotions?

4. Does the Offeror appear to understand the OCC’s data-promotion requirements?

5. Was the promotion process deemed to be relatively intuitive and user-friendly?

· loaded data from Stata

Objective = To show the OCC how Stata can communicate with the proposed database via an ODBC connection. If the Offerors do not have an ODBC connection between Stata and their database, they should instead demonstrate how data can be extracted from the database in a format that Stata can use and then execute a program against that data.

1. To what extent was the function demonstrated with little or no customization of a COTS tool? (That is, can this function be supported “out of the box”?)

2. Was the scenario executed completely, partially, or not at all?

3. Did the Offeror appear to understand the business question/problem to be addressed in this scenario?

Objective = To show the OCC how Matlab can communicate with the proposed database via an ODBC connection. If the Offerors do not have an ODBC connection between Matlab and their database, they should instead demonstrate how data can be extracted from the database in a format that Matlab can use and then execute a program against that data.

1. To what extent was the function demonstrated with little or no customization of a COTS tool? (That is, can this function be supported “out of the box”?)

2. Was the scenario executed completely, partially, or not at all?

3. Did the Offeror appear to understand the business question/problem to be addressed in this scenario?

· SAS logs with option “Full STimer” turned on

· Audit log for the user role executing the job

· Evidence of I/O throughput achieved[footnoteRef:2] [2: Please note that the performance level achieved by the Offeror in its demonstration will be used as the performance level to determine the baseline for system throughput documented In the OCC’s SLA. This level should be, at a minimum, 75 MB/seconds. The OCC expects the Offeror to demonstrate achievable performance during the demo, rather than a level of performance that cannot be sustained when the solution is implemented and operated.]

The OCC will provide the Offerors with the description of two business problems that need to be solved using the LPC data loaded during Scenario #1. One problem will require that a large number of rows—but a small number of columns—be used. The other problem will require that a large number of columns—but a small number of rows—be used. These two types of problems are typical of how the OCC economists use this data.

The OCC will provide a program for the Offeror to utilize in support of this scenario. The OCC would like to see the following included in this scenario:

· An external table merge. (This table should be simple in structure and small in size, e.g., a table of the U.S. states).

· The execution, at a minimum, of PROC FREQ and PROC DOWNLOAD during the remote submit.

· The execution of at least three (3) other programs accessing the same data set while the primary job is running.

Objectives = 1) to document the proposed database’s ability to achieve the OCC’s required level of I/O throughput; 2) to review how contention for resources is handled by the proposed solution; and 3) to demonstrate the user experience associated with executing an analytical job in a local SAS session and via a remote SAS submit.

This scenario will have both a live component and an offline component. While the execution of the program sto demonstrate the level of throughput and the resolution of contention can occur offline, with documentation provided of the performance results, the OCC wishes to see a live demonstration of the user interface via which the user will perform the local SAS session and the remote submit. For the purposes of the live demonstration, the Offeror may use a small subset of the LPC data set, as well as an abbreviated portion of the SAS program.

1. To what extent were the functions demonstrated with little or no customization of a COTS tool? (That is, can these functions be supported “out of the box”?)

2. Was the scenario executed completely, partially, or not at all?

3. Was the remote submit process deemed to be relatively intuitive and user-friendly?

· Remove a job from the processing queue

· Show an abnormally ended job and demonstrate its impact on the job queue

· Promote a job in the processing queue to run sooner than originally planned

2. Was the scenario executed completely, partially, or not at all?

3. Was the explanation of the logic associated with assigning, removing, interrupting, and promoting jobs deemed to be clear and concise?

4. Were the stops and re-starts of the interrupted jobs “clean”?

5. Were other jobs negatively impacted by the removal or interruption of a job in the queue?

6. Did the promotion of the job occur without negatively impacting other jobs?

2. Was the scenario executed completely, partially, or not at all?

3. Did the solution respond to the need for more CPUs, memory, and/or storage without causing the job (or other jobs in the queue) to interrupt or fail?

4. To what extent did the process of allocating additional CPUs, memory, and/or storage happen automatically? If human intervention was required, what was the level of effort involved for the system administrator and/or end user?

5. Were the correct types of notifications, alerts, and/or dashboard updates generated in a timely fashion?

7. Was the scenario executed completely, partially, or not at all?

8. Was there any negative impact to the jobs running on the server when the server was taken offline?

9. Were any other servers negatively impacted by this server being taken offline?

10. Were the correct types of notifications, alerts, and/or dashboard updates generated in a timely fashion?

2. Was the scenario executed completely, partially, or not at all?

3. Was there any negative impact associated with the addition of the server to the demo environment?

4. Were the correct types of notifications, alerts, and/or dashboard updates generated in a timely fashion?

· Remove a user and demonstrate that this user can no longer log in to the solution

· Assign permissions to a user and demonstrate the associated access available to him/her

· Revoke permissions from a user and demonstrate that he/she no longer has access to the parts of the solution to which access was previously given

6. Was the scenario executed completely, partially, or not at all?

7. Were the processes to add/remove users and assign/revoke permissions deemed to be relatively intuitive and user-friendly?

8. Did the assignment and revocation of permissions function properly to ensure the appropriate levels of access for the user?

· Escalate the ticket through the different support tiers

· Convert the ticket to a change request

2. Was the scenario executed completely, partially, or not at all?

3. Does the Offeror’s demonstration of this functionality indicate an understanding of the ECMP requirements for help-desk services?

4. Was the entry and escalation of the ticket deemed to be relatively intuitive and user-friendly?

5. To what extent are the two tools (to support help desk and change management) integrated?

Office of the Comptroller of the Currency 1 September 13, 2011