Q and A for MA HIX IES PMT RFR 32822.docx

COMMONWEALTH OF MASSACHUSETTS

EXECUTIVE OFFICE OF HEALTH AND HUMAN SERVICES

ONE ASHBURTON PLACE, 11TH FLOOR

BOSTON, MASSACHUSETTS 02108

QUESTIONS AND ANSWERS

regarding the

REQUEST FOR RESPONSES for PROJECT MANAGEMENT and TESTING (PMT) SERVICES

IN SUPPORT OF THE

MASSACHUSETTS HEALTH INSURANCE EXCHANGE PROGRAM

Document #: 22LCEHSHIXPMTRFR

Q & A Issued: March 28, 2022

This document contains responses to questions received regarding the Request for Responses for Project Management and Testing (PMT) Services in support of the Massachusetts Health Insurance Exchange Program issued by the Executive Office of Health and Human Services (EOHHS) on February 9, 2022. Answers have been provided to those questions which are of general interest and relevant to the preparation of a Response to the Request For Responses.

QUESTION 1: Please clarify the role of the Security Architect.

RESPONSE: This position is responsible for ensuring that the PMT Vendor complies with all of its own responsibilities regarding security and compliance. It is not responsible for other vendors’ adherence to their respective responsibilities regarding security and compliance. The PMT Vendor is not responsible for the security of the MA HIX infrastructure.

QUESTION 2: Will the Commonwealth consider more than one Project Manager position?

RESPONSE: The Commonwealth requires that Key Personnel include a Project Manager. The Bidder may propose additional Key Personnel as it deems appropriate or necessary, and may propose other staff positions (not designated as Key Personnel) as it deems appropriate or necessary.

QUESTION 3: Are all key positions expected to be full-time positions?

RESPONSE: All Key Personnel positions are expected to be full-time positions as set forth in Attachment A, Model Contract, Section 2.2.A.5.c.

QUESTION 4: Are Vendor staff expected to be onsite at the Commonwealth’s facility, on either a full-time or hybrid schedule?

RESPONSE: The Commonwealth does not require that PMT Vendor resources be on-site, except as set forth in Attachment A, Model Contract, Section 2.1.C. The Bidder may propose that some or all of its staff be located on-site on either a full-time or hybrid schedule.

QUESTION 5: What is the page limit for responses to Section 5.5.I of the RFR?

RESPONSE: The page limit for section 5.5.I, including Knowledge Transfer, is 10 pages.

QUESTION 6: The RFR requests submissions of sample plans. Do these samples count toward page limits where such limits are specified?

RESPONSE: No, copies of example plans will not count toward page limits.

QUESTION 7: Please clarify the scope of responsibility of the PMO.

RESPONSE: The PMO is responsible for both managing the scope of the Contractor’s Testing and Validation Unit, and for providing project management services for the HIX Program as set forth Attachment A, Model Contract, Section 2.6.

QUESTION 8: Please clarify the scope of responsibility of the Quality Assurance (QA) Lead.

RESPONSE: The QA Lead is responsible for ensuring that all Contractor Deliverables, including without limitation Test Deliverables, are complete, comprehensive, and accurate. While this position is responsible for internal Contractor QA only, the role of the PMO unit itself is broader than internal QA, as set forth in Attachment A, Model Contract, Section 2.6.

QUESTION 9: Can the Anticipated Contract Effective Start Date be moved up to allow for more time between the estimated Operational Start Date and 1 February 2023? (RFR, p. 59 and Section 8 Timeline) RESPONSE: The Anticipated Contract Effective Date is based on certain assumptions regarding the anticipated duration of post-award negotiations, and state and federal review and approval cycles; therefore, the Commonwealth is not currently in a position to commit to an earlier Contract Effective Date.

QUESTION 10: Is there any flexibility in the number and nature of references provided?

RESPONSE: The Commonwealth will evaluate the references provided in the context of a Bidder’s entire Proposal and in accordance with the evaluation criteria set forth in the RFR.

QUESTION 11: Is there an incumbent vendor currently performing PMT services?

RESPONSE: Testing and Validation Services are currently provided by the Current Vendor. A subset of PMO services are currently provided by the Current Vendor, a subset are currently performed by Commonwealth staff, and a subset are new services not currently being performed.

QUESTION 12: Please clarify the types of testing that the PMT Vendor is responsible for.

RESPONSE: Appendix C, Testing Types, to Attachment A (Model Contract) to the RFR sets forth the scope of testing responsibilities of the PMT Vendor.

QUESTION 13: Please clarify the scope of PMT Vendor responsibilities regarding Accessibility Testing.

RESPONSE: The Contractor is expected to perform accessibility testing services as described in Appendix C to Attachment A.

QUESTION 14: Did the Commonwealth intentionally omit Appendix B, Software Testing Life Cycle (“STLC”), from the procurement artifacts?

RESPONSE: The Commonwealth is requesting that Bidders propose an STLC as part of their responses. Appendix B to Attachment A refers to the STLC that is proposed by the Successful Bidder (as further negotiated by the parties as needed) and approved by the Commonwealth.

QUESTION 15: Will the STLC that the PMT Vendor creates (Attachment A, Model Contract, Section 2.7.B.1) replace what is currently being used by Optum or will the PMT Vendor utilize an existing STLC?

RESPONSE: The Commonwealth is requesting that Bidders propose their own STLC which, if agreed to by the Commonwealth, will replace existing processes.

QUESTION 16: Does EOHHS have expectations on any specific testing artifacts or deliverables from a test team?

RESPONSE: Bidders are asked to propose an STLC consist with contract requirements as set forth in Section 2.7.B of the Model Contract, which, if agreed to by the Commonwealth, will be incorporated as part of the Contract. Bidders should include their proposed testing artifacts and deliverables within their proposed STLC.

QUESTION 17: Does EOHHS have a preferred approval workflow expectation when it comes to defining when testing is complete?

RESPONSE: Proposed exit criteria should be defined as part of the Bidder’s proposed STLC.

QUESTION 18: Regarding the transitioning of ALM, section 5.4.F states: “It is the intent of EOHHS to transition to a Commonwealth-controlled instance of ALM or other tool prior to the Contract Effective Date.” (RFR p. 25 and Section 5.4.F).

a. Would the PMT Vendor need to assume any licensing agreements, or will those reside with the Commonwealth?

b. Will the Commonwealth be assuming all maintenance and support of the application and environment?

c. If another tool is selected does the Commonwealth have any preferred applications (besides ALM) that would be proposed?

RESPONSE: The Commonwealth will assume all licensing and maintenance and support responsibilities. At this time the Commonwealth is not actively considering another tool; however, if the Bidder proposes an alternative tool or solution, the Commonwealth will give such proposal due consideration.

QUESTION 19: Please clarify the anticipated post-procurement vendor mix.

RESPONSE: As described in Section 2.2 to the RFR and Attachment C, Reference Material:

· Project Management and Testing and Validation Services will be provided by the Contractor selected as a result of this procurement.

· Operations and Technical Services, similar in scope to that as set forth in appendix H, will be provided by a vendor selected as a result of a forthcoming procurement.

· The MA HIX Application Vendor, Optum, will continue to provide the license to WebInsure, maintenance and support for WebInsure, custom enhancements to the MA HIX Application, and maintenance and support of existing and new customizations.

· The Infrastructure Vendor, Smartronix, will continue to provide infrastructure managed services and certain infrastructure-related services in support of release management, operations, and security.

QUESTION 20: Will the Commonwealth consider additional contract terms to be evaluated and negotiated after contract award?

RESPONSE: By submitting a bid in response to this RFR, the Bidder acknowledges that it accepts all of the terms of the RFR (including but not limited to Attachment A, Model Contract). The Commonwealth may choose to negotiate additional terms with the Apparent Successful Bidder and may accept or reject such additional terms in its sole discretion.

QUESTION 21: HIX PMT RFR, section 5.4.G Other Programmatic Suggestions, pg. 27. This section states that “Any proposed additional functionality shall not be included with the overall Program or Cost Response.” To the extent bidder’s additional functionality or activities may have additional cost, where should this cost be reflected?

RESPONSE: An additional tab has been added to the Cost Response Worksheet for this purpose.

QUESTION 22: HIX PMT RFR, section 7.19 Incorporation of RFR/Order of Precedence, pgs. 53-54, HIX PMT RFR Attachment A - Model Contract, Section 5.33, Order of Precedence, pg. 95. These two sections have different documentation as to what comprises the contract. Could the Commonwealth please align with the correct order of precedence and documentation that is in scope?

RESPONSE: Amendment 1 to the RFR reflects the correct Order of Precedence, which shall control.

QUESTION 23: Please clarify the PMT vendor’s responsibility with respect to reviewing 3rd-party vendor deliverables.

RESPONSE: The PMT vendor is not responsible for reviewing the content of 3rd-party vendor deliverables, except as needed to inform direct PMT-responsibilities. However, the PMT vendor is responsible for ensuring compliance from all vendors with the deliverable lifecycle and therefore the PMT vendor will review deliverables for their adherence to program approved DEDs (deliverable expectations documents).

QUESTION 24: HIX PMT RFR Attachment A - Model Contract – Section 2.6M model contract, pgs. 46-47. Please confirm that incident response is only for production environments.

RESPONSE: Incident response is only for production environments.

QUESTION 25: Please confirm whether the PMT Bidders will need to provide a quote for LoadRunner, which is listed in Appendix G as being currently provided by the current vendor.

RESPONSE: If the Bidder proposes an alternative to LoadRunner, the Bidder should provide a quote. Otherwise, the Bidder should assume that the Commonwealth will be responsible for licensing LoadRunner directly.

QUESTION 26: HIX PMT RFR Attachment A - Model Contract, Model 2.7.D.3, pg.57. Please confirm that development support would be the responsibility of the Application Vendor.

RESPONSE: Development support will remain the responsibility of the MA HIX Application Vendor.

QUESTION 27: If a Bidder proposes its own or another 3rd party vendor solution for Additional Services as described in Section 5.4.F of the RFR, will the Commonwealth consider licensing terms that are term-based rather than perpetual?

RESPONSE: As stated in Section 5.4 F.1.c., and 5.4 F.2.b, The Bidder should assume that EOHHS would procure the license and any needed third-party vendor services. If a Bidder proposes a third-party vendor solution, the Commonwealth will evaluate its proposed licensing terms in the context of the Bidder’s overall Response. Bidders should make clear in their Responses any proposed proprietary or third-party components that will require a licensing agreement, and provide the terms of any such agreements.

QUESTION 28: HIX PMT RFR Attachment A - Model Contract 5.6 Deliverables and Intellectual Property Rights. 5.6(B) and (C) both include reference to Contractor property as developed by Contractor for clients other than the Commonwealth. Will the Commonwealth amend the definition of Contractor Property so that Contractor Property includes property developed by Contractor for other clients, including Commonwealth, where development was not charged to Commonwealth or funded by the Commonwealth, but instead Contractor Property was licensed to clients, and where such development could be prior or concurrent to the performance of such services?

RESPONSE: Bidders are expected to comply with the legal terms as embodied in the Model Contract. If a Bidder takes exception to any such term, the Bidder should clearly note this and propose specific alternative language in its Response. The Commonwealth will evaluate Bidder’s Response in its entirety in accordance with Section 6 of the RFR.

QUESTION 29: HIX PMT RFR Attachment A - Model Contract, MSA, Section 2.6 R.10., pg. 50. Can EOHHS please provide additional clarification on what type of independent audit of MA HIX EOHHS would like the PMT vendor to perform annually (e.g., CMS MARS-E, SOC, etc.)? Also, please confirm this activity is only for one audit annually, not for each audit EOHHS has done on an annual basis.

RESPONSE: The Commonwealth is removing the requirement outlined in Model Contract, section 2.6.R.10 (see Amendment 1 to the RFR).

QUESTION 30: HIX PMT RFR Attachment A - Model Contract, Section 3.2, I.3, pg. 70 & Section 2.7.A.3, pg. 52. Please confirm that the Contractor can use their own automation framework for building test automation.

RESPONSE: The Commonwealth may consider a Bidder’s test automation framework provided that such framework can be transferred to the Commonwealth in full upon request and at no additional cost to the Commonwealth.

QUESTION 31: Attachment A - Appendix A, Key Performance Measures, Table 3.b, TV07, pg. 13. Please confirm that configuration, infrastructure, and third-party vendor-related outage issues will be considered as Valid for the purposes of calculating this metric.

RESPONSE: Configuration, infrastructure, and third-party vendor-related outages will not be considered valid for the purpose of calculating this metric.

QUESTION 32: Please confirm that if PMT Vendor testing is conducted in parallel with UAT, nonconformities found during UAT will be excluded from the denominator for this metric.

RESPONSE: It is assumed that Contractor testing phases will be complete prior to UAT. There may be circumstances in which the Commonwealth allows Contractor testing to run parallel to UAT, in which case the Commonwealth may consider modifying this metric.

QUESTION 33: There are two KPMs that appear to be misnumbered. Can the Commonwealth provide an updated table with correct numbering?

RESPONSE: Appendix A, Key Performance Measures, has been updated (see Amendment 1 to the RFR) as follows:

· PM-003 Key Personnel has been renumbered PM-04 Key Personnel, and

· TV10 Test Automation has been renumbered TV09.

QUESTION 34: Based on Attachment C - MA HIX Reference Material, can EOHHS please confirm that for 2-3 major releases every year the PMT vendor should account for a.) three cycles of test executions in each major release, and b.) full regression test execution for each cycle? Also, can EOHHS please confirm that the execution of full regression tests will be required for all minor releases too? (Attachment A - Appendix A, Key Performance Measures, Table 3.b, TV10, pg. 14.)

RESPONSE: Attachment C – MA Reference Material provides a point-in-time snapshot of aspects of the current MA HIX Program, the MA HIX Application, and the MA HIX/IES System. Bidders should not make any assumptions regarding the number of test cycles per major release based on information provided in that document. It is the Commonwealth’s expectation that the regression test suite will be appropriately identified and executed for every release.

QUESTION 35: HIX PMT RFR Attachment B – Cost Response Worksheet. Please clarify that the Additional Services tab of the Cost Response Worksheet should reference RFR Section 5.4 F and not section 5.5. and that each subsection of this tab should reference 5.4.F.1, 5.4.F.2 and 5.4.F.3.

RESPONSE: The Cost Response Worksheet has been revised to reflect the correct section references (see Amendment 1 to the RFR).

QUESTION 36: HIX PMT RFR Attachment B – Cost Response Worksheet. On the “Start Up” tab, there are two different Contract Effective Dates – December 2022 and July 2022 - please provide the Contract Effective Date Bidders should use.

RESPONSE: The Cost Response Worksheet has been revised to reflect an anticipated Contract Effective Date of September 1, 2022 (see Amendment 1 to the RFR). The Instructions on the Start Up tab have been revised to remove the request to prorate Month One costs.

QUESTION 37: HIX PMT RFR Attachment B – Cost Response Worksheet: What accommodations, if any, will the Commonwealth make to ensure the confidentiality of pricing agreements between the Bidder and third-party vendors?

RESPONSE: The Commonwealth recognizes the potential sensitivity regarding the disclosure of third-party pricing agreements, as there does not appear to be a public records exemption for such information. Nevertheless, Bidders should make their best efforts, consistent with any contractual non-disclosure obligations that they have assumed, to comply with all of the Response Requirements set forth in Section 5. Notwithstanding the foregoing, the Commonwealth expects that Bidders proposing to pass through third party technology costs will provide the relevant rates or costs in their response.

QUESTION 38: May Bidders propose Key Personnel roles that differ from those listed in Section 5.4.C of the RFR?

RESPONSE: Bidders may propose additional Key Personnel, and/or may propose an organizational structure and staffing model that includes Key Personnel positions that address the roles and responsibilities of Key Personnel as outlined in the RFR.

QUESTION 39: Would it be acceptable to phase in Key Personnel between contract effective date and full operational start date?

RESPONSE: As set forth in Attachment A, Model Contract, Section 2.5.B.2.a, “The Contractor shall ensure that all Key Personnel have been fully onboarded and dedicated full-time to the Contract no later than thirty (30) Calendar Days following the Contract Effective Date, or such other date as agreed to in writing by EOHHS.” Bidders may propose a phase-in schedule for Key Personnel in accordance with the terms of Section 5.6.

QUESTION 40: Does winning this work preclude a vendor from winning the referenced Operational and Technical Services (OTS) contract?

RESPONSE: No, such a vendor would not be precluded from consideration in the OTS procurement.

QUESTION 41: Regarding the experience requirements in 5.3.D of the RFR, can the experience of the Bidder and proposed Material Subcontractor together be used to meet the experience requirement?

RESPONSE: Per Section 5.3.D, “Experience of proposed Material Subcontractors may count toward requirements 2-4 if the proposed Material Subcontractor will be performing the same or similar functions for this HIX/IES System engagement. If a requirement is to be met by a Material Subcontractor, this must be clearly stated in the response.“ QUESTION 42: Would the Commonwealth accept a statement that the financial information required in Section 5.3.G of the RFR would be provided upon contract award in lieu of inclusion in the Response?

RESPONSE: The Commonwealth’s evaluation of Business and Programmatic Responses, as set forth in Section 6.3, includes evaluation of the Bidders’ financial stability, as reflected in its responses to Section 5.3.G.

QUESTION 43: Would the Commonwealth consider waiving Section 5.3.G of the RFR with respect to Material Subcontractors?

RESPONSE: Suggested Response: The Commonwealth’s evaluation of Business and Programmatic Responses, as set forth in Section 6.3, includes evaluation of the Bidders’ and their Material Subcontractors’ financial stability, as reflected in its responses to Section 5.3.G.

QUESTION 44: Can EOHHS release the current Functional and Technical Roadmap and Strategic Release Schedules?

RESPONSE: The Commonwealth is not able to share the current functional or technical roadmaps with Bidders. We anticipate that 2 major (July and December) and two minor (August and February) releases will be provided by the Application vendor on an ongoing basis. Major releases consist of 10-20 functional items as well as 20+ previously identified defects (or modifications to resolve operational workarounds). Minor releases generally consist of approximately 15 previously identified defects (or modifications to resolve operational workarounds).

QUESTION 45: Please provide details on the HIX test environments used for UAT.

RESPONSE: The following environments are utilized for UAT: INT0, INT1, TT4, DEV2, and STG0. (See Attachment C, Reference Material, for more information about these environments.) In UAT, functional testing is done to ensure software is behaving as designed and to meet requirements. This also includes testing with different integration points like FDSH HUB interface testing for the SSN document verification process. The Commonwealth UAT teams perform validations of the Softheon system, but health insurance carrier testing is performed outside of UAT. UAT also includes MMIS Interface testing and formulary search interface testing (Checkbook API service).

QUESTION 46: What tool does the HIX Program use for defect and test case management?

RESPONSE: The HIX Program currently uses HP Application Lifecycle Management (ALM) for requirement, test case, and defect management.

QUESTION 47: Is EOHHS looking to automate (as opposed to manual) any type of testing like regression testing?

RESPONSE: Yes. Please see Attachment A, Model Contract, Sections 2.6.I.1.g; 2.7.A.3; 2.7.B.1.b,xi; 2.7.B.1.c.iii; and 2.7.B.7.

QUESTION 48: Can EOHHS confirm access to the database or datasets is available if needed for testing purposes?

RESPONSE: This is confirmed.

QUESTION 49: Does EOHHS see a need for test scripts to support UAT efforts along with QA technical support during UAT?

RESPONSE: Development of test scripts to support UAT is not specifically requested; however, the Bidder may propose this as a value-added service in response to Section 5.4.G of the RFR.

QUESTION 50: Please clarify the PMT Vendor’s Project Management responsibilities with respect to Release Management.

RESPONSE: The PMO is responsible for the project management functions of release management. Technical services in support of release management will be provided by the Operations and Technical Services vendor, with additional support provided by the Infrastructure and Application Vendors (if needed).

QUESTION 51: Please provide additional information about the MA HIX infrastructure.

RESPONSE: Please refer to Attachment C, Reference Material, for additional information about the MA HIX infrastructure.