MDR_Brand_Name_JA_Redacted.pdf

1 DTIC – Master Data Repository Brand Name Justification

May 4, 2015

BRAND NAME JUSTIFICATION AND APPROVAL FOR

MarkLogic Server, Maintenance, and Professional Services

1. AGENCY and CONTRACTING ACTIVITY:

Requiring Agency:

Department of Defense (DoD) Defense Technical Information Center (DTIC) 8725 John J. Kingman Road, Suite 0944 Fort Belvoir, VA 22060-6218

Contracting Activity:

Department of Defense Washington Headquarters Services (WHS) – Acquisition Directorate 1155 Defense Pentagon Washington, DC 22301-1155

2. NATURE/DESCRIPTION OF THE ACTION:

This action requests the approval for a Brand Name Justification for the purchase of MarkLogic Server licenses, maintenance and professional services.

The Defense Technical Information Center (DTIC) has the requirement for a software applica-tion and support services to implement a Master Data Repository. DTIC’s goal is to consolidate, unify, manage, control, search, analyze, and disseminate scientific and technical data using a sin-gle tool. Consolidating the data into a single tool will ease and facilitate analysis, management, and control. A single tool is necessary to replace the numerous legacy technologies; reduce the number of required skill sets; reduce risk by eliminating many processes, interfaces, and points of integration; and gain efficiencies by applying policies, features, validation rules, and data quality improvements once rather than in multiple data flows.

The tool must also provide better functionality than some of DTIC’s existing technologies, namely:

• Storage that can quickly ingest diverse structured and unstructured data without time-consuming preliminary data model standardization.

• Access control, integrated with the data store that will ensure the security of its infor-mation but can be more responsive to changing policies and new data sets.

DEPARTMENT OF DEFENSE

WASHINGTON HEADQUARTERS SERVICES

1155 DEFENSE PENTAGON

WASHINGTON, DC 20301-1155

Acquisition Directorate

2 DTIC – Master Data Repository Brand Name Justification

• Search that supports simple and complex queries across structured and unstructured data, delivers fast, accurate, and precise results, can quickly respond to requests for new fea-tures, and is integrated with semantic technologies.

• Semantic capabilities for storing and querying Resource Description Framework (RDF) triples.

• Analytic and visualization capabilities that can be quickly implemented across minimally normalized data to offer new insights on Department of Defense science and technology research.

• Input systems that can be easily spun up for new data calls and whose data can be inte-grated, searched, controlled, and analyzed in the same system.

Initial tool and services to develop it will be on premise at DTIC in its development, staging and production NIPR (Nonsecure Internet Protocol Router) environments with potential of moving to a cloud. The system will also be deployed to SIPR (Secret Internet Protocol Router) staging and production environments. Service is required in CY 2015.

The Government anticipates awarding a firm fixed price contract for the MarkLogic perpetual license, maintenance, and professional services with a total estimated value of $X. The Period of Performance (PoP) is as follows:

Annual Maintenance: Base period of 1-year with two (2) 1-year options Professional Support Services: Base period of 6-months with three (3) 6-month options

3. DESCRIPTION OF SUPPLIES/SERVICES:

This procurement is for purchase of MarkLogic Server licenses, maintenance, and professional services.

The MarkLogic server licenses will allow DTIC to run the software on its NIPRnet and SIPRnet production and staging environments. The number of cores estimated for the license is based on the amount of data DTIC requires the system to store, the predicted rate of usage, and the desired response time of the system. The licenses are perpetual licenses. The one-time cost for the perpetual licenses will provide savings over the long term and will allow DTIC to run the software in perpetuity, eliminating annual renewals and their concomitant risk of interruption of service.

The Maintenance fee will enable DTIC to obtain patches, updates, and new versions of the software from the vendor. These patches, updates, and new versions typically address future security vulnerabilities, bugs, and new features.

Professional services are on-site software developer services to build DTIC’s instance of MarkLogic, harvest and ingest DTIC’s data, implement access control policies, provide search, analytics, semantics, and input services, and integrate with DTIC systems. On-site professional services are needed because DTIC lacks NoSQL (Not Only Structured Query Language) experience generally and MarkLogic expertise specifically. Professional services will work closely with, and provide training for, DTIC’s internal implementation team to grow DTIC’s in-

3 DTIC – Master Data Repository Brand Name Justification house NoSQL and MarkLogic expertise with the goal of DTIC’s team becoming self-sufficient for maintenance of and enhancement to the system.

These licenses and services will provide storage, access control, search, analytics, input platform and semantic platform, for DTIC data and products. In the past, DTIC has had separate contracts for these functions and in other cases has built custom solutions. DTIC has not previously offered the semantic capability.

Search – DTIC currently has a contract (HQ0034-14-F-0325) for the Google Search Appliance.

This is not a perpetual license, requiring the appliances and licenses to be repurchased every few years. The Google Search Appliance lacks several features that have proven to be crucial to our users: truncation and wildcards, full Boolean logic and nesting across fields and full text, accurate and precise result counts, and the ability to return all items in a results set for further analysis, instead of just the first 1000 items. MarkLogic does offer these features and is unique in that it combines these features with the capabilities to meet the rest of DTIC’s requirements (storage, semantics, control, etc.). This contract, since it applies to the NIPR and SIPR environments, would eliminate DTIC’s reliance on Google Search Appliance for those environments. At the same time, DTIC is exploring other options to replace the Google Search Appliance on the Public environment where its requirements are not as stringent. This contract was competitively awarded using a limited sources brand name justification.

Storage – DTIC currently uses Oracle (W91QUZ-04-A-0001 - DP05) and MySQL (HQ0034-15- P-0069) to store much of its content. Additional content is stored on file systems. Moving nearly all of DTIC’s content into MarkLogic will eventually greatly reduce our use of those technologies. Initially, only the storage for archived content will be replaced by MarkLogic. But as each input system is replaced or updated, the local storage for the content at DTIC will be replaced by MarkLogic. The additional cost savings this achieves is in replacement of the extensive and sometimes duplicative ETL (extraction, transformation, load) processes currently used to move data from the input storage systems to the search, analytics, and access control systems. By using MarkLogic as the principle store of data, and as the search, access control, and analytics tool, the cost of maintenance and the associated risk of those many ETL processes is eliminated. The data will be stored in the same system that manipulates and publishes the data.

Both of these contracts were competitively awarded using limited sources brand name justification. The Oracle contract was awarded using the DoD Enterprise Software Initiative, Enterprise Software Agreements where there are multiple awardees on the contract.

Access Control – DTIC currently uses an in-house custom built application to enforce access control of collections, individual documents, field level data elements, and services for its primary content. This custom application controls the content in only some dissemination products and for none of the secondary content because of the large work effort required for any update. This also greatly hinders accommodating new collections and changes in policy.

MarkLogic Server would replace this custom built application and enable consistent access control across the bulk of DTIC content. The MarkLogic Pilot at DTIC demonstrated the speed with which DTIC’s access control policies could be implemented within the tool. This reduces risk by implementing a system that DTIC can maintain, and facilitates responsive modifications to changes in policy.

4 DTIC – Master Data Repository Brand Name Justification

Analytics – DTIC is often asked to evaluate analytics tools and is currently testing several. Each has been hampered by the necessity of repeatedly pulling the relevant DTIC data from multiple storage systems with diverse schemas into yet another storage system. MarkLogic will reduce the work effort for these and similar efforts by pulling the data together once into a consistently up-to-date data library with access control rules and a search web service for secure and easy reuse of the content. MarkLogic supports analytical queries and visualization tools giving DTIC the option of building simple custom charts from MarkLogic queries, or laying a third party tool in front of the database, Input - DTIC is currently running several different input systems. It is working with Defense Logistics Agency’s Document Automation and Content Services (DACS) to build a robust replacement document input and management system for its primary content which will feed into the Master Data Repository. However, DTIC also maintains and often builds custom input applications for smaller collections. These products require rapid development. Typically, this has led to a proliferation of isolated collections with duplicative applications. MarkLogic Server will enable rapid development of input systems for minor collections with the added benefits of consolidated search and storage. This capability will be used for new secondary collections and to replace input systems for existing secondary collections.

4. STATUTORY AUTHORITY: FAR 13.501(a)(1)(ii) Acquisition under the authority of the test program for commercial items at 41 U.S.C. 1901. Brand Name feature is essential to the Government’s requirements, and market research indicates other companies’ similar products, or products lacking the particular feature, do not meet, or cannot be modified to meet, the agency’s need.

5. DEMONSTRATION OF CONTRACTOR’S UNIQUE QUALIFICATIONS:

The proposed source is MarkLogic or one of its resellers. MarkLogic has been determined to be the only product that as (A) is a NoSQL solution, (B) is a single tool and can fulfill all six primary requirement areas and a number of non-functional requirements detailed in the Performance Work Statement (PWS) (C-H). This determination was based on a pilot project, market research and a Request for Information (RFI).

A. NoSQL

DTIC resolved on NoSQL document store technology to meet the need to quickly ingest and integrate diverse structured and unstructured document data for search, analysis, and control.

Typically, with a relational database system, data from diverse schemas must first be normalized into a single comprehensive schema before it can be combined into a single database to support input, search or access control at the data element level. This is an intensive effort that can take months. With NoSQL systems, data with diverse schemas, or no schema, can be immediately ingested as is. Priority data elements can then be mapped as needed for specific use cases after ingest without having to map all of the data. This capability was demonstrated during a pilot implementation of MarkLogic. DTIC currently maintains over 20 different collections and since the pilot has been given responsibility for several more. Because of the current dependency on relational databases, it has not been able to combine any more than a handful of these collections http://www.documentservices.dla.mil/pdf/Document_Automation_and_Content_Services.pdf

5 DTIC – Master Data Repository Brand Name Justification for search or analytics. Because each new collection has a unique schema, they, too, have been set up in isolated databases adding to the challenge of integrating, controlling and disseminating the data. A NoSQL document store will enable DTIC to immediately ingest and serve any collection delivered to the agency and ensure that any new data call will from the start be part of the master repository.

B. Single Tool

The single tool requirement comes from DTIC’s experience managing many data sets cross-pollinating multiple dissemination products. A single tool would realize DTIC’s goals to:

• Reduce the complexity and risk of its architecture by replacing a mix of many commer-cial and custom built tools with a single system

• Decrease the workload and number of required skill sets to maintain its systems by shrinking the number of technologies used.

• Diminish the workload required to integrate multiple tools when a single tool will suffice.

• Lessen the workload and risk of creating and maintaining many Extract, Transform, Load processes to load the same data into separate tools for input, storage, access control, search, analytics, and semantics. Some of the responses to the RFI proposed a combina-tion of tools that would require multiple pipelines for reloading the data into each tool and maintaining consistency across the copies of data.

• Ensure the access controls rules are applied once, rather than duplicating the effort to im-plement and maintain the policies in different technologies for each component. Ensure the access control rules automatically and consistently control data in all states: input, storage, search, semantics, analytics.

• Enable rapid development of solutions for new collections by offering a tool that permits immediate search, control and analytics once data is loaded, instead of reconfiguring each separate functional tool to accommodate every new collection.

MarkLogic Server’s ability to function as a single tool has been proven through the Pilot project.

It’s uniqueness as a single tool covering all requirements was demonstrated through the market research and RFI.

C. Storage DTIC’s requirement is for storage that can quickly ingest diverse structured and unstructured data without time-consuming preliminary data model standardization. This is described above under A. NoSQL. MarkLogic Server’s abilities as a NoSQL document store are not unique among the few vendors of NoSQL document stores, but MarkLogic was the only product to offer these features in combination with DTIC’s other requirements.

D. Access control DTIC’s requirement is for role-based access control of search and analysis queries at collection, document, and metadata element levels based on rules for controlling Scientific and Technical Information and user attributes. The system also must be easily configurable to accommodate new collections and policy changes. While this capability is not unique, MarkLogic was the only product to offer these features in combination with DTIC’s other requirements.

E. Search

6 DTIC – Master Data Repository Brand Name Justification

DTIC’s requirement is for a search system that supports simple and complex queries across structured and unstructured data, delivers fast, accurate, and precise results, can quickly respond to requests for new features, and is integrated with semantic technologies. None of the other NoSQL document systems considered in the market research were considered by industry to be search engines or enterprise search systems. Responses to RFI corroborate the conclusion that no other NoSQL, by itself, could fulfill DTIC’s search requirements. While there are certainly search engines that can meet the search requirements, they would require integration with a data store solution and often separate semantic and access control systems.

F. Semantics

DTIC’s requirement is for a system to store ontologies and RDF triples of DTIC data and offer search and SPARQL (Simple Protocol and RDF Query Language) queries of RDF triples integrated with the search of metadata and full text. RDF and SPARQL are used to semantically link data to create new composites of data, discover new relationships, and facilitate exploring.

Very few NoSQL products claimed to support RDF and SPARQL. Search engines typically search RDF data as a separate data store. MarkLogic was the only product shown to natively store and support RDF and SPARQL, offer a unified search of standard data and RDF, and combine these capabilities with DTIC’s other requirements.

G. Analytics and visualization DTIC ‘s requirement is for a system that supports statistical and analytical queries , support visualization of data through charts, graphs, and maps, and support third party Business Intelligence tools. These capabilities are not unique except in combination with DTIC’s other requirements. For example, some of the RFI responses proposed only a business intelligence tool without consideration for the storage and input requirements.

H. Input DTIC’s requirement is for a system that supports input, editing, and management of data by users and internal staff. The requirement includes potential for workflows, ACID (atomicity, consistency, isolation, durability) transactions and multiversion concurrency control (MVCC).

While much of the data DTIC manages is scientific and technical, it also manages and validates data for the DoD Comptroller and the DoD Acquisition community. DTIC is requiring ACID and MVCC to ensure it upholds the same level of data integrity for those communities in a NoSQL tool as it does in a relational database management system. Responders to the RFI showed that only two NoSQL products support ACID transactions and, of those two, only MarkLogic meets the rest of DTIC’s requirements.

From the above, procuring a different brand would not meet DTIC requirements and would not be cost effective to the government.

6. FEDBIZOPPS ANNOUNCEMENT/POTENTIAL SOURCES:

An RFI was posted to FBO on December 9, 2014 to locate qualified, experienced and interested businesses capable of providing a software application and support services to implement a Master Data Repository as described in the PWS. See Section 8 for specific details.

7 DTIC – Master Data Repository Brand Name Justification

On April 8, 2015, the Government posted a Special Notice (Notice of Intent – Brand Name Only Acquisition) on FedBizOpps to determine if any firm could provide a MarkLogic equivalent product. This notice closed on April 13, 2015 with no responses to the Special Notice.

7. DETERMINATION OF FAIR AND REASONABLE COST:

The Contracting Officer will ensure that the resulting contract award price is fair and reasonable per FAR 13.106-3(a)(2), by comparing the contractor’s proposal to: 1) the pricing found through market research; 2) the independent government cost estimate; and 3) the historical pricing paid under similar contracts at AD for like services.

8. MARKET RESEARCH/EFFORTS TO OBTAIN COMPETITION/INCREASE COM-

PETITION:

MarkLogic Server was found to be the only product that was a single tool, was a NoSQL document store, and that met all the requirements. This determination was based on a pilot project, market research and an RFI.

Pilot Project

MarkLogic conducted a pilot project at DTIC in 2013. The pilot’s goals were:

1. Ingest data from a variety of repositories (e.g. MySQL, Oracle, XML, PDFs) with diverse schemas.

2. Improve search functionality with consistent search results, Boolean logic, truncation, a search language that supports simple and complex searches, filtering, clustering, brows-ing, and semantic features.

3. Apply a standard and supportable access control system across DTIC data.

4. Map relationships across related data from diverse schemas, facilitating analytics, visual-ization, and finding data gaps.

The pilot lasted 8 weeks and was performed by 1.25 MarkLogic full-time equivalents (FTEs) with marginal support from DTIC staff. At the end of the eight weeks, the pilot had completed all of its goals including:

• Ingesting DTIC’s target content within first two weeks

• Demonstrating search functionality at end of first two weeks that exceeded capabilities of

DTIC’s current systems.

• Implementing complex access control rules that DTIC has spent a year developing in its legacy system

• Visualizing results of analytic queries of data that had never before been integrated at

DTIC

• Demonstrating several use cases above and beyond the initial goals

Nine DTIC staff members monitored the pilot and evaluated the product according to 40 criteria.

Evaluating each feature on a scale of 1-10, average score across all criteria by the nine evaluators was 9.28. The pilot demonstrated that a NoSQL technology, and MarkLogic specifically, could meet DTIC’s set of requirements.

8 DTIC – Master Data Repository Brand Name Justification

DTIC’s experience with the tool was corroborated through discussions with three MarkLogic customers: Special Operations Command (SOCOM), Network Enterprise Technology Command (NETCOM), and Defense Information Systems Agency (DISA).

Market Research

DTIC staff performed market research by researching and comparing the available NoSQL technologies and submitted a Master Data Repository Market Research Report to WHS. Ten products were considered: MarkLogic, MongoDB, Virtuoso, Couchbase, CouchDB, Cloudant, FoundationDB, Oracle NoSQL, Cassandra, HBase. Products were evaluated based on criteria corresponding to some of the primary functional and non-functional requirements in the PWS.

Information for the evaluation came from:

• review of technical documentation on the vendors’ web sites

• product comparisons in industry articles and on web sites such as http://db-engines.com/en/ and http://www.bigdataanalyticstoday.com/

• descriptions in Wikipedia

• review of Forrester and Gartner surveys o Gartner Magic Quadrant for Operational Database Management Systems, 2014 o Gartner Magic Quadrant for Enterprise Search, 2013 o Forrester Wave: NoSQL Document Databases, Q3 2014

Analysis showed that, aside from MarkLogic, no other NoSQL tool on the market covered the primary requirements areas identified by DTIC. There were many NoSQL vendors and some open source solutions, but only a few offered NoSQL document stores. Other vendors offer NoSQL key-value and column-wide data stores which will not meet DTIC’s needs for a document repository. From the information reviewed, none of the vendors combined NoSQL storage with access control, enterprise search, and native semantic capabilities.

RFI

An RFI was posted to FBO on December 9, 2014 to locate qualified, experienced and interested businesses capable of providing a software application and support services to implement a Master Data Repository as described in the PWS.

Three DTIC division chiefs and the information architect reviewed each response against the broad requirement categories and specific high priority requirements from the PWS. The team assessed technology solutions and approaches separately while considering DTIC’s strategic mission and goals. The team reached consensus on each assessment.

Thirteen responses to the RFI were received. DTIC also considered two responses to an earlier announcement posted on NASA Solutions for Enterprise Wide Procurement (SEWP). Of the fifteen responses received, three met the Government’s technical requirements while also http://db-engines.com/en/ http://db-engines.com/en/ http://www.bigdataanalyticstoday.com/

9 DTIC – Master Data Repository Brand Name Justification meeting their need for all-in-one NoSQL solution. All qualified responses firmly recommended the same enterprise solution (MarkLogic). This confirmed DTIC’s earlier market research (above) which found that there was only one NoSQL tool (MarkLogic) that meets all of DTIC high priority needs, and covers the majority of the agency’s requirements.

DTIC recommended pursuing an acquisition strategy that would enable early implementation of a Master Data Repository using a NoSQL single tool that meets the technical requirements.

Based on the findings of the market research and the RFI, MarkLogic is the only tool that meets DTIC’s technical requirements. The findings indicated that there are at least three vendors capable of providing the MarkLogic server thus ensuring sufficient competition.

9. OTHER FACTS:

N/A

10. INTERESTED SOURCES:

MarkLogic Droidan i_SW

11. ACTIONS TAKEN TO REMOVE BARRIERS TO COMPEITION: (FAR 6.303-

2(b)(11))

The market research found at least three vendors capable of providing the MarkLogic server thus ensuring sufficient competition. Technical personnel within DTIC will continue to evaluate the market for a commercial tool that will fulfill the Government’s need. If another MDR using a NoSQL single tool is found to meet the technical requirements then the Government can compete future requirements without a brand name restriction.

10 DTIC – Master Data Repository Brand Name Justification

CONTRACTING OFFICER CERTIFICATION:

I certify that this justification is accurate and complete to the best of my knowledge and belief.

Signature:

Robin Castoldi Contracting Officer WHS-AD

Name and Title Organization

REQUIREMENTS CERTIFICATION:

I certify that the facts and representations under my cognizance which are included in this justification and which form a basis for this justification are complete and accurate.

Andrew Pedrick Technical POC DTIC

AGENCY COMPETITION ADVOCATE APPROVAL:

I have reviewed this justification and approve it, as required by FAR 13.501(a)(2)(ii).

Wendy Schermerhorn Competition Advocate WHS-AD