Data Analytics Tool_Software -- Contractor Capability Supplemental Worksheet.xlsx

Capability Statement Supplement

3 = Visual pipeline: Same as 1, plus the solution includes rich visual tools to integrate and transform data from multiple sources and formats 1 = Coding: The solution provides coding APIs and can also use external visual tools to integrate data sources and prepare an analytical data sets 0 = The solution provides coding and coding APIs to prepare analytical data sets.

Workbench: Feature engineering What capabilities exist to assist data scientists in feature engineering activities such as feature selection, feature creation, missing values, outliers, and other data preparation methods? 5 = Auto feature creation: Same as 3, plus the solution provides built-in feature selection/creation/dim-reduction capabilities and an automated variable selection/creation operator that does not require analysts to specify an explicit technique.

3 = Automated data prep: The solution has built-in data preparation capabilities that don't require analysts to specify an explicit technique (e.g., handling of categorical variables, missing values, outliers, normalization).

1 = Operators: The solution has a broad set of feature engineering capabilities that can be specified by the data scientist.

0 = The solution does not provide any automatic or semiautomatic data preparation/feature selection/creation/dim-reduction capabilities.

Workbench: Modeling What capabilities are available to allow data scientists to build custom pipelines to train and evaluate models using a drag-and-drop, configuration wizard, and/or coding interfaces? (This excludes automation features; automation is a separate criterion.) 5 = Ambidextrous: The solution has a seamless ability to move from one model development interface to another (e.g., it autogenerates a workflow from a wizard or code from a workflow) across the most common modeling techniques.

3 = Visual pipelines and wizards: Double-handed: The solution supports the end-to-end modeling life cycle in both a visual environment as well as an instantiated notebook-based environment.

1 = Code-first with some visual interface elements: The solution supports the end-to-end modelling life cycle for all common modeling techniques in at least one interface.

0 = The solution does not support all common modeling techniques in any one type of interface.

Workbench: Visualization What data and/or model visualization tools are available for data science and or business teams? 5 = The solution provides the capability for data scientists to create interactive apps that can pull, process, and run models against new data based on user input.

3 = The solution automatically generates relevant model performance visualizations (e.g., lift, ROC charts).

1 = The solution provides a drag-and-drop ability to create all standard chart types.

0 = Programming is required for users to generate data visualizations.

Workbench: Collaboration What capabilities are available to allow data science teams to collaborate among themselves and with business stakeholder teams? 5 = Integrated: Same as 3, plus the solution offers integrated collaboration features and version control (VC) for non-data scientist stakeholders.

3 = Asset sharing: Same as 1, plus the solution has built-in capabilities to clone, abstract, and/or reuse project workflows and other assets.

1 = Documentation: A workbench provides specific features to annotate workflows, document projects, and other assets.

0 = The solution does not provide any specific collaboration features.

Workbench: Automation What capabilities are available to allow data scientists and/or others to automate and/or schedule key steps in the model building life cycle or other processes? 5 = Full: The solution includes capabilities to automatically generate models by analyzing a user specified data set with very little other human input, including Natural Language Processing (NLP) and automatically utilizing unstructured data.

3 = Partial: Same as 1, plus the solution includes capabilities to automate some parts of the model-building process, such as data prep, feature engineering, algorithm selection, and/or hyperparameter tuning.

1 = Workflow: Users can accomplish automation by building arbitrary workflows/pipelines.

0 = The solution offers no specific automation features.

ModelOps: Deployment What capabilities exist to use trained models and logic/rules for batch and/or real-time production applications? 5 = Edge: Same as 3, plus the solution is directly integrated with one or more internet-of-things (IoT) platforms for deployment within edge devices.

3 = Embedded: Same as 1, plus models can be embedded in applications, databases, or other execution platforms following standard formats, such as PMML, PFA, and/or ONNX.

1 = Callable: The model is exposed as an API callable from an external application 0 = The vendor has no model deployment solution.

ModelOps: Monitoring What capabilities exist to monitor the efficacy of models while they are deployed in production applications? 5 = Business: Same as 3, plus the solution includes a built-in model monitoring feature with comprehensive metrics to periodically analyze the accuracy of the model on new data and definable business key performance indicators (KPIs).

3 = Retraining: Same as 1, plus the solution includes a built-in model monitoring feature to periodically analyze the accuracy of models and send models alerts such as for drift, etc.

1 = Custom: Data scientists can build and deploy arbitrary workflows/scripts/programs with the model to monitor models.

0 = The solution has no production model monitoring features.

ModelOps: Staging What capabilities exist to stage and test new/replacement models in production applications? 5 = Switching: Same as 3, plus the solution provides capabilities for rules-based switching for models in production.

3 = Split: Same as 1, plus the solution provides built-in features that streamline the routing of traffic to different models in production (e.g., for A/B testing).

1 = Basic: The solution provides common model evaluation features, and production model candidates can be tested against existing models with programming or scripting.

0 = The solution does not provide any model testing features.

ModelOps: Collaboration What capabilities exists to smooth the hand-off of models to DevOps, application development teams, or other teams that use built models? 5 = Integrated: The solution is integrated with continuous delivery and release automation tools.

3 = Model repository: Same as 1, plus the solution includes a model repository and visual management tool that development plus operations (DevOps) pros can use to access and deploy models.

1 = Scheduler: The solution includes a built-in capability to define, schedule, and manage any arbitrary job.

0 = No specific capabilities exist for operations collaboration.

Algorithms: Open source What, if any, open source algorithms/methods are included and usable within the vendor's solution workbench? What, if any, contributions does the vendor make to the open source community for algorithms/methods? 5 = Deep: Same as 3, plus the solution's workbench abstracts open source deep learning libraries/packages algorithms in operators and/or wizards.

3 = Encapsulate: Same as 1, plus the solution's workbench abstracts and encapsulates open source libraries/packages algorithms in operators and/or wizards.

1 = Programming: Algorithms in open source libraries/packages can be called from within the workbench using a coding interface in programming languages R and Python.

0 = The solution cannot leverage open source algorithms.

Algorithms: Unique What, if any, algorithms/methods are implemented exclusively for use within this vendor's solution workbench? 5 = Specialized: Same as 3, plus the solution offers numerous additional specialized unique algorithms/methods, including deep learning, forecasting, and others methods compared to most vendors in this evaluation.

3 = Comprehensive: The solution offers hundreds of unique algorithms/methods.

1 = Standard: The solution offers a satisfactory but lower number of unique algorithms/methods relative to other vendors in this evaluation.

0 = The platform offers no unique algorithms.

Architecture: Scalability What architectural design maxims are employed by the vendor's solution to support the training of large data sets, concurrent workbench users, and low-latency model scoring/inferencing? 5 = Auto elastic: Same as 3, plus the solution monitors workloads and can automatically provision and deprovision compute resources within upper limit boundary.

3 = Distributed extensive: Same as 1, plus significant unique algorithms, processing pipelines, and other methods are implemented to be distributed across multiple computers.

1 = Distributed engines: Some algorithms and processing pipelines can be pushed to one or more distributed processing engines, such as Apache Spark, Apache Flink, and H2O.ai.

0 = Partitioning framework: Algorithms and processing pipelines can be partitioned manually or using a partitioning framework to distribute workload across multiple computers.

Architecture: Infrastructure What are the infrastructure requirements and/or options for deploying the vendor's solution on-premises and/or cloud? 5 = The vendor's workbench can run on-premises or as a managed service on a public cloud infrastructure and includes/can utilize desktop tools for offline use.

3 = The vendor's workbench can run on-premises and instantiated as a machine image on a public cloud infrastructure.

1 = The vendor's workbench can run only on a public cloud infrastructure or only on a single-user desktop/workstation computer.

0 = The vendor's workbench deployment options are unknown.

Security What types of security atributes are built into the tool? 5 = Fully capable of assigning user roles based on groups, LDAP and integrated with CAC SSO. Can also mask sensitive code. Has achieved IA compliancy/STIG compliant.

3 = Partially capable of handling LDAP and SSO. Can also mask sensitive code. Has achieved IA compliancy/STIG compliant.

1 = Moderate hierarchial provisioning built in 0 = Little to no user/role/groups structure provisioning built in Governance How well does the product integrate with data governance/data management features? 5 = Fully capable of standing alone or intergrated as part of data centric solution to include enforced governance rules, data catalogue, user/roles management, and entity resolution 3 = Highly capable of integrating into existing data governance tools through direct connections, extracts, etc.

1 = Moderate integration with key data governance concepts such as keyword identification, metadata repository 0 = Little to no data governance or data management capabilities/integration built in Compatibility How compatible is the platform with our primary DW, ETL, BI, and analytics toolsets? 5 = Complete compatibility with all of our primary ETL, DW, BI, and analytics tools 3 = Compatible with our mostly all of our tools, such as Netezza, HANA, SAP, IBM, and SAS 1 = Moderate compatibility with our key technologies in use with DISA, PAX, and ERP 0 = Little to no compatibility with our key technologies in use with DISA & PAX data centers or with ERP.

Accelerators What, if anything, does the vendor solution provide to help data science, application development, and/or business teams to speed the development of models and/or applications for horizontal and/or vertical business domains? 5 = Ready apps: Same as 3, plus the vendor offers a substantial number of industry-specific applications that are in use by customers relative to other vendors in this evaluation.

3 = Starter apps: Same as 1, plus the vendor offers some industry-specific applications that are in use with current customers.

1 = Sample code: Same as 0, plus the vendor offers sample code for a few business use cases.

0 = Documentation: The vendor only offers online documentation as community support.

Community What, if anything, does the vendor solution provide to help customers of its solution share best practices, applications, code, pipelines, Q&A, discussions, etc., with other customers? 5 = The vendor supports large-scale community outreach and sponsors large, regular events for the data science community.

3 = The vendor has active community outreach and sponsors regular events for the data science community.

1 = The vendor offers online support with limited data science community outreach and events.

0 = The vendor's community is primarily geared toward outbound content (e.g., blogs, documentation).

3 = The vendor has sufficient financial, technical, and management resources, and it is increasing market adoption.

1 = The vendor has sufficient financial, technical, and management resources, but it lacks momentum given its experience in market.

0 = The vendor has management and/or financial turbulence.

Solution road map What is our level of confidence in the vendor's road map in terms of making the solution more competitive for enterprise buyers compared to other vendors in this evaluation? 5 = The vendor's road map will significantly increase its ability to increase adoption compared with other vendor solutions.

3 = The vendor's road map will keep pace with the market.

1 = The vendor's road map will fill gaps in the solution, but not enough to keep pace with the market.

0 = The vendor's road map is insufficient, and its solution will become less competitive in the market.

Implementation support What is our level of confidence in the vendor's ability to smoothly implement and support its current and future enterprise customers? 5 = The vendor has a proven global technical services and support organization that can handle current customers and future growth, and it has significant resources and expertise to handle multiple, simultaneous complex enterprise implementations.

3 = The vendor offers both technical services and a support organization that can handle current customers and support growth plans.

1 = The vendor provides both technical services and a support organization that can handle current customers, but it does not have much capacity to handle growth.

0 = The vendor does not have a technical services and support organization.

Pricing and acquisition What options are available to prospective customers to use the vendor's product (e.g., subscription, license, free, etc.)? How can prospective customers calculate the cost of using the vendor's solution based on their needs? 5 = The vendor's free open source or community/trial version is available for production deployments.

3 = The vendor offers pricing options including both term and perpetual.

1 = The vendor offers one option of either term or perpetual.

0 = The vendor's solution can only be acquired as part of a bundle of the vendor's other products.

Partners What deep, strategic partnerships has the vendor forged with other technology, services, business, or other companies such as systems integrators, independent software vendors (ISVs), and/or channel partners? 5 = The vendor's partnerships significantly increase adoption of its solutions compared with others in this evaluation.

3 = The vendor's partnerships help it keep pace with the market.

1 = The vendor's partnerships fill gaps in the its solution, but not enough to keep pace with the market.

0 = The vendor's partnerships are insufficient, and the vendor's solution will become less competitive in the market.

3 = The vendor has more than 3 DOD agencies as current customers 1 = The vendor has more than 1 DOD agencies as current customers 0 = The vendor has fewer than 1 DOD agencies as current customers or no DOD customers