Investigative and Intelligence Analytical Software Specifications.docx

Investigative Intelligence

Capabilities Specification

Table of Contents

Introduction and Scope

The scope of this document is to define the requirements related to the architecture, the functionalities and performance of the Siren Platform (hereinafter referred to as “The Solution”).

Definitions

The terms used in this document as "shall", "should" and "will" have the following meanings:

· "Shall" is used to specify a mandatory requirement.

· "Should" is used to describe desirable features not part of the supply.

· "Will" is used to identify factual statements intended to provide clarification or provide supporting information.

The terms "TBD" and "TBC" used in this document, have the following meanings:

· "TBD" Identifies a requirement (or a parameter of the requirement) not yet defined.

· "TBC" Identifies a requirement not yet confirmed.

For structured or unstructured data, it is meant:

· "UNSTRUCTURED data" means those data in a typically text format that does not have one default template and / or that are not organized by default (common data formats of unstructured texts are DOCX, PDF, HTML, TXT, etc.).

· "STRUCTURED data" means those data typically in tabular or matrix format. They have a predefined template and / or are organized in referential mode (common forms of data structured are XLSX, CSV, JSON files).

Requirements

Database Requirements

1. The Solution shall store data in a centrally accessible repository.

a. This centrally accessible repository will be an ever-growing asset containing data for analysis and data resulting from analysis.

2. The Solution shall support real-time analysis of the data as it is ingested.

3. The Solution shall store profiles, posts, and connections between each dataset.

4. The Solution shall use an Elasticsearch cluster as the main data repository.

5. The Solution shall support a data integration layer that requires no additional hardware or middleware above the existing Elasticsearch cluster.

6. The Solution shall support Logstash input plugins to ingest data.An input plugin enables a specific source of events to be read by Logstash.The table below lists the input plugins that are available.

7. The Solution shall support Avatica for access to JDBC datasources.

8. The Solution shall provide the option to create and maintain a local, optimized, physical copy of remote data as an Elasticsearch index.

9. The Solution shall support the execution of queries across all datasets connected to it, including data joining between different indices.

10. The Solution shall support distributed joins between indices, delegating the execution of the queries to the local platforms or individual Elasticsearch nodes whenever possible.

11. The Solution shall extend the Elasticsearch DSL with a join query clause which enables a user to execute a join between Elasticsearch indices.

12. The Solution join capability shall operate on top of an in-memory distributed computing layer which scales with the number of nodes available in the cluster.

13. The Solution shall support a semi-join that can be used to filter one set of documents, A, based on a second set of documents, B. A semi-join between the two sets of documents, A and B, returns the documents of A that satisfy the join condition with the documents of B. This is equivalent to the EXISTS() operator in SQL.

14. The Solution shall support an inner join that enables the “projection” of arbitrary fields (including script fields and document’s scores) from a set of documents, B, and “combines” them with a set of documents, A. The projected fields and associated values of a document from set B are mapped to all of the documents from set A that satisfy the join condition. The result of the join is the set of documents, A, augmented by the projected fields from the set of documents, B.

15. The Solution join shall allow a combination of nested joins using Boolean operators (conjunction, disjunction, negation) to create complex query plans.

16. The Solution data model shall allow the definition of multiple bi-directional and uni-directional relationships across two indices, based on a specific field of each index.

17. The Solution join filter shall enable the filtering of one set of documents (the target) with another one (the source) based on shared field values.

18. The Solution shall support the virtualization and querying of remote Elasticsearch clusters.

19. The Solution shall have a reflection layer that enables the caching of data from external databases within Elasticsearch.

20. The Solution shall have a distributed join layer that enables the execution of join operations at scale.

21. The Solution shall have a join-caching layer that enables the caching of the most common join results for faster execution times.

Ingestion and Export Requirements

1. The Solution shall provide PDF, PNG, CSV, and XLS exports, which can be scheduled, triggered by conditions, or on demand.

2. The Solution processing engine must support at least the following structured data types:

a. CSV.

i. Large CSV import and export (GB+ capacity).

b. Excel.

c. Export from databases obtained via JDBC.

d. Feed and Syndication formats.

e. JSON.

f. PDF.

g. REST services.

h. RSS.

i. SOAP services.

j. Streams of real-time data coming from the Web (e.g., Twitter stream metadata).

k. XML.

3. The Solution processing engine must support at least the following unstructured data types:

a. HTML.

b. OpenDocument.

c. JSON.

d. RTF.

e. Text.

f. File - uploading local files by:

i. Drag and drop process.

ii. Upload button.

iii. Provide a remote folder path. Supported file extensions are summarized above.

4. The Solution shall support multi-vendor searches across two or more internal and external data sources in a single user interface.

5. The Solution shall ingest data from two or more internal or external data sources without requiring the user to leave the Solution.

6. The Solution shall display relationships across multiple internal and external data sources in the graph browser.

7. The Solution shall support a data model for describing relationships that exist across two or more internal and external data sources.

8. The Solution shall support API or Bulk Data integration to external data sources including:

a. Cellebrite

b. Nuix

c. OpenText

d. Q6 Cyber

e. SearchLight Cyber

f. Skopenow

g. Webz.io

h. DarkOwl

i. Anti-Human Trafficking Intelligence Initiative

j. Esri

k. ICEYE

l. Spire Global

m. Planet

n. Global Fishing Watch

o. Dow Jones

p. Moody’s

q. Sayari

r. OpenCorporates

s. Kharon

t. Dun & Bradstreet

u. Refinitiv

v. ShadowDragon

w. SpyCloud

x. Chainalysis

y. Thomson Reuters Clear

z. LexisNexis Accurint

aa. Whooster

ab. NLETS

ac. NIBIN

ad. Pipl

Elasticsearch Enhancement Requirements

1. The Solution shall provide the following enhancements to the traditional Elasticsearch data model:

a. Data is ingested as the entities they represent.

b. A Data Model that dynamically joins entities to answer different questions.

c. The ability to support a limitless number of combinations to combine data and answer questions that were not defined beforehand.

Artificial Intelligence and Automation Requirements

1. The Solution shall provide a Machine Learning module based on Elasticsearch Machine Learning.

2. The Solution shall provide machine learning analysissupporting at least the following use cases:

a. Anomaly Detection.

b. Forecast Future Behavior.

c. Predicting numerical values with Regression.

d. Predicting classes with Classification.

3. The Machine Learning module shall connect to and query data from the Elasticsearch cluster.

4. The Machine Learning module shall be able to securely authenticate to Elasticsearch and keep TLS encryption for data in transit.

5. The Machine Learning module shall be able to execute live detections periodically or on historical data.

6. The Machine Learning module shall allow users to define a time period of the data to be used for training directly from the UI.

7. The Machine Learning module shall be deployed in a containerized environment to support the deployment pipeline.

8. The Machine Learning module shall allow automatic hyperparameter optimization based on a training loss and validation loss threshold.

9. The Machine Learning module shall be able to store configurations and generated neural networks of each model created.

10. The Solution shall provide an infrastructure to build the following Machine Learning use cases:

a. Anomaly detection.

b. Bot classification.

c. Entity resolution.

d. Fake News Detection.

e. Image tagging.

f. Object detection.

g. Outlier detection.

h. Prediction.

i. Geolocation predictive analytics.

ii. Implicitly propose hidden correlations amongst data.

iii. Predictions of responses by entities and outlooks based on available information.

i. Recommendations.

j. Relationship identification.

Natural Language Processing (NLP) Requirements

1. The Solution shall support natural language processing (NLP) capable of extracting locations, events, dates, amounts, sentiment, and salience (prominence), as well as certain relations.

2. The Solution shall support NLP with multilingual capabilities and support for transliteration.

3. The Solution shall support a user interface for editing the NLP knowledge baseand rule sets.

4. The Solution shall support annotations to override the NLP definitions.

5. The Solution NLP shall be composed of multilingual lexicons, complex linguistic rules, and gazetteer data.

6. The Solution shall support tokenization by recognizing document encoding and code block segments, identifying word boundaries for whitespace and non-whitespace delimited languages, along with word boundaries for left-to-right and right-to-left ordered languages.

7. The Solution shall be ISO 639 compatible and recognize languages for each segment or portion of a document to allow for processing of multiple languages or mixed code within a document.

8. The Solution shall support extensible entity extraction.

9. The Solution shall support relationship extraction that identifies the most important Predicate Subject Object (PSO) triples with the ability to extend to as many as needed for a given use case.

10. The Solution shall support location extraction with a built-in gazetteer which provides intelligent latitude and longitude.

11. The Solution shall support multi-vector sentiment analysis that identifies Polarity, Mood, Aspect, Intensity for documents and Salience of PERSON entities.

12. The Solution shall provide the English gist translation of non-English documents to quickly assess the general meaning and whether a document needs to be translated or not.

Entity Resolution Requirements

1. The Solution shall be able to resolve multiple records that reference a single entity concept, like a person or a company, to a single unified version, making a distinction between records that represent the same entity or are simply related due to a certain characteristic.

2. The Solution shall support entity resolution through fuzzy matching and non-monotonic reasoning.

3. The Solution shall provide the ability to see the detailed records for resolved entities in a table.

4. The Solution shall provide the ability to see the connection between resolved entities in a graph.

5. The Solution shall check for name mistakes or possible name duplications.

6. The Solution shall provide the ability to create normalization rules, to correct and equalize the semantic output of the entity, to merge equivalent entities having different names (for instance nicknames or abbreviated names).

User Interface Requirements

1. The Solution shall support the definition of data models directly through the UI, based on Elasticsearch indices.

2. The Solution shall provide simplified executive dashboards for decision makers and stakeholders.

3. The Solution shall support dashboards that allow the examination of all data available through the main data repository.

4. The Solution’s dashboards shall support the navigation to a related dashboard, filtering the target dashboard content based on a relationship defined in the data model.

5. The Solution shall support a local data model to be defined for each dashboard, inheriting the relationships from the Solution’s main data model.

6. The dashboard local data model shall be created from one primary index, which then connects to further indices.

7. The dashboard local data model shall keep the integrity of the queried data across all indices, with filters being applied to any of the indices available in the dashboard based on the relationship defined.

8. The Solution shall support scriptable actions to be coded into the dashboards, in JavaScript syntax, to automate interactions with visualizations.

9. The Solution’s dashboards shall be composed of multiple independent visualizations that are updated based on the data model.

10. The Solution’s visualizations shall be interactive, enabling users to select parts of the visualization to be applied as a filter to the dashboard.

11. The Solution shall allow visualizations to display data of any one of the data sets available in the dashboard.

12. The Solution shall allow the user created visualizations to be re-used across multiple dashboards.

13. The visualization shall be able to display the Elasticsearch query executed to produce its content.

14. The Solution shall provide at least the following type of visualizations:

a. Record Table.

i. The Record Table shall display documents matched by a query in Elasticsearch.

ii. The Record Table shall allow the results to be paginated.

iii. The Record Table shall allow the attribution of aliases to the table columns.

iv. The Record Table shall allow custom HTML templates to be created to display the data.

v. The Record Table shall allow the results displayed to be exported in CSV or JSON format.

b. Graph.

i. The Graph visualization shall provide integrated link analysis.

ii. The Graph visualization shall allow all links created as relationships in the data model to be explored as a graph.

iii. The Graph visualization shall use the same back-end platforms that are powering the dashboards, without a need for a separate graph store.

iv. The Graph visualization shall support all back-end platforms to which it is connected.

v. The Graph visualization shall summarize intermediate nodes in a single edge, even when there are millions of middle nodes, through link aggregation – based on Elasticsearch Term or Significant Term aggregation.

vi. The Graph visualization shall allow nodes to be grouped on an ad hoc basis, according to user selection or category grouping criteria, for example: nodes can be grouped by a specific metadata attribute.

vii. The Graph visualization shall support the ability to create a filter set by using the big data relational search capabilities in the dashboards and to apply those filters on a graph to control graph relationships expansion.

viii. The Graph visualization shall support end-users to define rule-based styles on it, changing at least: node colors, node sizes, node labels and node into edges.

ix. The Graph visualization shall allow the rule-based graph styles to be cascaded and individually switched on or off at any time.

x. The Graph visualization shall allow data with geographic location to be displayed in a map, while still displaying links across the visible nodes.

xi. The Graph visualization shall support an animated graph evolution feature which shows changes in data over time and across geographic locations.

xii. The Graph visualization shall allow users to create and modify records and relations between them directly from the Graph interface.

c. Analytic Table.

i. The Analytic Table shall allow the results of a query in Elasticsearch to be summarized in at least the following types of aggregations:

1. Date Histogram.

2. Histogram.

3. Range.

4. Date Range.

5. IPV4 Range.

6. Terms.

7. Filters.

8. Significant Terms.

9. Geohash.

d. Markdown.

i. The Markdown visualization shall allow the creation of custom text to be displayed according to Markdown syntax formatting rules.

e. Metric.

i. The Metric visualization shall display one or more individual numbers based on a specific aggregation criterion.

f. Goal.

i. A Goal visualization displays how an aggregated metric progresses towards a predefined fixed goal.

g. Goal and Gauge.

i. A Gauge visualization displays where an aggregated metric falls based on a predefined range.

h. Pie charts.

i. Enhanced Coordinate Map.

i. The Coordinate Map shall display geographic areas overlaid with circles or squares representing the data determined by buckets of a specific aggregation.

ii. The Coordinate Map shall allow free, paid, or self-hosted tile-maps to be provided.

iii. The Coordinate Map shall allow the addition of point layers based on a geo-point type field of an Elasticsearch index.

iv. The Coordinate Map shall allow the addition of standard or tiled WMS layers.

j. Region Map.

i. The Region Map visualization shall allow metrics of an aggregation to be grouped based on geographical areas (polygons), represented in a color scale.

ii. The Region Map shall allow the definition of which polygons will be used to aggregate the data.

k. Time Series.

i. The Time Series visualization shall allow the user to combine an infinite number of aggregations and pipeline aggregations to display complex time series data.

l. Tag Clouds.

i. The Tag Cloud visualization shall display the most frequent or significant words in an Elasticsearch text field, based on an aggregation.

m. Topic Clustering.

i. The Topic Clustering visualization performs significance and clustering analysis on a field’s values.

ii. Topic Clustering highlights significant terms (topics) whose frequency in documents increases when current filters and search queries are applied.

iii. The Topic Clustering panel is divided into separate cells, each representing a term. The cell size represents the number of documents it matches, while its color represents its relevance to current queries/filters.

iv. The Topic Clustering visualization will cluster together mutually significant terms, forming groups that denote 'areas of interest' of the text corpus (large set of structured texts).

v. The following interactions with Topic Clustering using mouse or touch will be supported:

1. Pan the view by click-and-drag/tap-and-drag

2. Zoom in or out using the mouse wheel/pinch-zoom gestures

3. Zoom out to initial view and close all cells with the ESC key

4. Open a cell with a double-click/tap

5. Close a cell by double-clicking its header

6. Expanding and collapsing cells

7. Double-click/tap a cell to open it; this loads its significant sub-terms and displays them recursively. It also puts the cell term in a white header at the top of the cell.

vi. Once loaded, cells are automatically opened and closed depending on your current level of zoom. However, you can still open a cell explicitly by double-clicking on it. Conversely, you can close it by double-clicking on the white cell header.Loaded cells open/close automatically as you zoom in/out.

vii. Tooltip - Hovering over a cell displays details such as the number of documents it represents and its relevance/significance score:

viii. You can click on Filter corpus just on "<topic>" to apply the topic as a new global filter. Clicking on Extract sub-topics calculates sub-topics relevant to the highlighted topic; it is equivalent to double-clicking on the cell.

ix. Live Filter - Clicking a cell selects it. In a dashboard, this automatically applies a live filter matching the cell’s term to other visualizations in that dashboard.The live filter does not apply to the visualization itself, or other Topic Clusterings in the same dashboard, which retain their UI state.

For example, you can pair a Topic Clustering with a Record Table next to it. Selecting a cell updates the table and displays associated document samples.

n. Heatmap Chart.

i. The Heatmap Chart shall provide a graphical representation of data where the individual values contained in a matrix are represented as colors.

ii. The Heatmap Chart shall allow the definition of different color scales.

iii. The Heatmap Chart shall allow the results of each cell value to be represented as a percentage.

o. Line, area, and bar charts.

p. Scatter Plot.

q. Radar chart.

r. Box Plot.

s. Bubble diagram.

t. Horizontal Bar Chart.

u. Dimensional Correlation Chart.

i. The Dimensional Correlation Chart shall have one primary dimension based on an aggregation and multiple secondary dimensions based individual metrics of such aggregation.

ii. The Dimensional Correlation Chart shall display scatter plots for each combination of two neighboring dimensions, based on the sequence of metrics the user defined.

iii. The Dimensional Correlation Chart shall display lines showing the different combinations of records across all the defined dimensions, showing higher concentrations as thicker lines.

iv. The Dimensional Correlation chart shall allow filters to be applied to each individual dimension.

v. Multichart.

i. The Multichart visualization shall allow multiple charts to be selected based on a list in a single visualization in the dashboard.

w. Parallel Lines Chart

i. Arranges several metrics on parallel columns.

x. Timeline

i. Visualize events in chronological order.

y. Visual Builder

i. Visualize time series data using pipeline aggregations.

z. Timelion

i. Compute and combine data from multiple time series data sets.

aa. Controls

i. Create interactive controls for easy dashboard manipulation.

ab. Relational Navigator

i. Provide navigation between relationally-connected dashboards.

Analysis Requirements

1. The Solution shall support heterogeneous data analytics.

2. The Solution shall allow the ingestion of large volumes of streaming data, with thousands of records per second.

3. The Solution shall provide shortest path analysis across multiple indices and index patterns without requiring the use of an external graph database.

4. The Solution shall provide common communicator analysis across multiple indices and index patterns without requiring the use of an external graph database.

5. The Solution shall provide the capability to analyze and aggregate data ingested from different input sources such as:

a. Real-time and near real-time signal and sensor readings.

b. Structured data coming from public or private sources.

c. Data from Image Analysis.

d. Data from Semantic Analysis.

e. Heterogeneous data gathered because of different intelligence activities (e.g., LI, HUMINT, Social Network Analysis and Siren).

6. The Solution shall provide the following benefits for the end user:

a. Data browsing and correlation, ranging from the identification of simple concepts, entities, and relationships, to the possibility to correlate data with third party sources (e.g., sensor reports, flights list).

b. Display of complex relationship networks. The ability to easily show complex relationship networks associated to extended periods of time, to create visualizations which, highlight the entities who should be worthy of interest.

c. Adaptive graphical interface. A flexible and powerful graphical interface that can be easily and automatically exported into official reports.

7. The Solution shall provide the ability to analyze events over a timeline.

8. The Solution shall provide the ability to analyze progression of a theme over a timeline.

9. The Solution shall provide the ability to analyze trends over a timeline.

10. The Solution shall provide the ability to detect non-obvious relationships.

11. The Solution shall support lenses to provide graph metrics including page rank, centrality, betweenness, closeness, and eigenvector centrality.

Search and Link Requirements

1. The Solution shall support searching functionalities and link analysis to actively support:

a. Keyword search, by leveraging on a full-text-indexing of the ingested contents.

b. Taxonomy and custom taxonomy search.

c. Timeline search (e.g., search in time frame).

d. Geo-spatial search (e.g., search in specific areas of the world).

e. The ability to navigate the resulting semantic relationship networks.

f. The ability to filter data according to specific time windows and spatial coordinates.

g. The ability to selectively ignore subsets of data and nodes which are not relevant to the investigation in progress.

h. The ability to have analysis-based summaries, for example, on the correlation frequency among entities.

i. The ability to analyze the relationships and the interactions among entities defined by users, and others which are automatically proposed by the system.

j. The ability to highlight indirect relationships and interactions among entities.

k. The ability to search all records from separated Elasticsearch indices using a Global Search Panel.

l. The ability to drag and drop results of the Global Search onto the graph browser.

m. The ability to access the dashboard associated with the entities obtained from Global Search.

Integration and API Requirements

1. All data queried by The Solution shall be available to be queried by other external applications through REST APIs.

2. The Solution architecture shall be designed for modularity and scalability to support integrations with multiple applications.

3. Siren shall provide assistance integrating ELK plugins and filters for data processing.

4. The Solution shall provide the ability to export a graph to i2 Analyst Notebook.

5. The Solution shall support Web Services invocations in at least:

a. Dashboards.

b. Graphs.

c. Alerts.

d. Scripts.

6. The Solution must allow web service calls triggered by alerts to facilitate automation and integration with other applications.

7.

8. The Solution shall provide an API to the federation layer as an extension of the core Elasticsearch API.

9. The Solution federation layer shall support the following APIs:

a. Search API - Allows you to execute a search query and get back search hits that match the query.

b. Scroll API - Allows the pagination of search hits.

c. Multi search API - Allows the execution of several search requests within the same API.

Alerting Requirements

1. The Solution shall support an alert capability that generates alerts and reports, based on logic conditions applied to the data.

2. The Solution shall support the following alert logic conditions:

a. Simple scheduled queries and threshold alerts.

b. Complex, multi-event triggered alerts.

3. The Solution shall support the setup of alerts through:

a. Dashboards.

b. Alert creation interface.

4. The Solution shall support alert actions including:

a. Generating emails.

b. Sending PDFs.

c. Calling APIs to execute actions.

5. The Solution shall support scripted alert templates that can be activated by operators for use cases such as spatial proximity, geo-fencing, or multiple triggers.

Alert Structure

1. The Solution shall create alert watchers using the following structure:

a. Trigger Schedule - When and how to run the Watcher

i. The schedule defines a set of constraints that must be met to execute a saved watcher. Any number of constraints can be added to a single schedule, and multiple rules can be combined to achieve complex intervals, programmed using simple text expressions using the Node.JS later module.

b. Input Query - What Query or Join Query to execute

i. The input parameter is the key element of a watcher and defines a dynamic range index query feeding the circuit. The input field accepts any standard Elasticsearch query including server-side scripts in supported languages and fully supports the Siren Join capabilities out of the box.

c. Condition - How to conditionally analyze response

i. The condition block is the "entry gate" into the processing pipeline of a Watcher and determines its triggered status.

1. On true condition, the pipeline will proceed further.

2. On false condition, the pipeline will stop (no action will be executed) until its next invocation.

ii. Never condition - Use the never condition to set the condition to false. This means the watch actions are never executed when the watch is triggered. Nevertheless, the watch input is executed. This condition is used for testing. There are no attributes to specify for the never condition.

iii. Compare condition - Use the compare condition to perform a simple comparison against a value in the watch payload.

iv. Array compare condition - Use array compare to compare an array of values.

v. Script condition - A condition that evaluates a script. The scripting language is JavaScript. It can be as simple as a function expecting a Boolean condition or counter.

vi. Anomaly Detection - Simple anomaly finder based on the three-sigma rule.

1. Dynamic detection of outliers/peaks/drops:

2. Static detection for known ranges/interrupts:

vii. Range Filtering - Use for getting documents which have a value in between some values. For example, get only the documents which have values from 45 to 155 inside the “Amount” field.

d. Transform - How to Adapt or Post-Process data

i. A transform processes and changes the payload in the watch execution context to prepare it for the alert watch actions. No actions are executed in the case that the results are empty after transform processing.

1. SEARCH TRANSFORM - A transform that executes a search on the cluster and replaces the current payload in the watch execution context with the returned search response.

2. SCRIPT TRANSFORM - A transform that executes a script (JavaScript) on the current payload and replaces it with a newly generated one. Its uses include:

a. Converting format types.

b. Generating new payload keys.

c. Interpolating data.

d. Create new payload property.

e. Filter aggregation buckets.

3. CHAIN TRANSFORM - A transform that executes an ordered list of configured transforms in a chain, where the output of one transform serves as the input of the next transform in the chain.

e. Actions - How to Notify users about this event

i. Actions are used to deliver any results obtained by a Watcher to users, APIs, or new documents in the cluster. Multiple Actions and Groups can be defined for each. Supported actions:

ii. EMAIL - Send Query results and message using Email/SMTP.

iii. EMAIL HTML - Send Query results and message using Email/SMTP using HTML body.

iv. WEBHOOK - Deliver message to remote web API.

v. WEBHOOK USING PROXY - Deliver message to remote API using Proxy.

vi. SLACK - Deliver Message to #Slack channel.

vii. REPORT - Take a website snapshot and send it using Email/SMTP.

viii. CONSOLE - Output Query results and message to Console.

Alert Watcher Wizard

1. The Solution shall provide a wizard to help create alert watchers using a step-by-step sequence.

Custom watchers

1. The Solution shall support an additional list of use case-specific watcher types that are created from the dashboard. These types of watchers are called custom watchers.

2. A custom watcher created from a dashboard inherits its search criteria, including the search pattern, search query, and filters. Each custom watcher type applies specific processing and trigger conditions.

3. The Solution shall support at least the following customer watcher types:

a. New results: Alerts when there are new results for the given search.

b. Geo fence: Alerts when there are new results within a geographical area.

c. Proximity: Alerts when two entities are closer or further than a specified distance. This type must be enabled manually as it is not applicable to all data sets. The Solution provides support for enabling this.

Alert Examples

1. The Solution shall enable automation of recurring questions (as queries) by using Watchers. Examples of the types of alerts that shall be supported by the Solution are:

a. HIT COUNT PER HOUR

i. QUESTION: How many hits does index X receive hourly?

ii. WATCHER: Query index and return count of hits in last hour.

iii. ACTION: Notify with number of Hits per hour.

b. METRIC THRESHOLDS

i. QUESTION: Are any of my monitored metrics surpassing a certain value?

ii. WATCHER: Query index and type for specific values, aggregated by an arbitrary field.

iii. ACTION: Notify with aggs bucket details every time a threshold is surpassed, or spike anomaly detected.

c. BLACKLISTS HITS

i. QUESTION: Are any of my users trying to reach blacklisted destinations?

ii. WATCHER: Query firewall logs comparing destination IPs to a blacklist.

iii. ACTION: Notify admin using email if any IP >= 10 matches returned.

d. FAILED LOGINS

i. QUESTION: Are there recurring failure attempts authenticating users on my network?

ii. WATCHER: Query Active Directory logs for login failures in last hour and compare to user index.

iii. ACTION: Notify admin using webhook if >= 10 matches returned.

e. LEAK DETECTION (chain)

i. QUESTION: Are there any public leaks about my data I was not aware of?

ii. WATCHER: Query for user emails included in published leaks ingested from third parties.

iii. ACTION: Save hits in secondary result Index. Notify using email if leak was not known in a secondary Watcher.

Collaborative Workspace Requirements

1. The Solution shall support workspaces, allowing configurations to be segregated from other workspaces for at least the following:

a. Searches on large Elasticsearch indices, meant to be shared across use cases (e.g., "articles-*" to find all the documents in indices starting with "articles-").

b. Searches on small indices (i.e., ingested from a CSV), meant to be used only in a particular workspace.

c. Data models (relations between the fields in the data returned by the searches).

d. Dashboards and visualizations.

e. Entity tables and searches.

f. Entity identifiers.

g. The data model visual layout.

h. Saved Graphs.

i. Scripts.

j. Alerts.

k. Webservice results.

2. The Solution shall provide the capability to migrate segregated workspace resources as above into indices meant to be shared across use cases.

a. The Solution shall restrict the capability to migrate segregated resources to specific roles/groups.

3. The Solution shall allow a user to name their workspace.

4. The Solution shall allow a user to establish a nickname for their workspace, which can be used by other users to reference and access their workspace.

5. The Solution shall support two types of workspaces

a. Master workspace - a set of dashboards and searches around a specific index or a set of indices e.g., "all phone calls and dashboards to ask specific questions". These would typically be created by administrators to "wrap" large important indices in Elasticsearch clusters.

b. Investigation workspace - These are created for a specific investigation e.g., "Case A". These might include (incorporate) one or more Master workspace, but also allow the analyst to save details pertaining to the specific case.

6. Within a workspace the user will be able to ingest data specific to the workspace and modify the data model specific to that workspace.

Workflow Management Requirements

1. The Solution shall integrate with a 3rd party workflow system such as JIRA.

2. The Solution shall support the following integrations with JIRA:

a. Select an active Jira ticket from the SirenSolution.

b. Export dashboard information from Siren Solution to a Jira ticket.

c. Search for Jira tickets in Siren Solution by ticket ID, description or data.

d. Access Siren Solution via a hotlink from a Jira ticket.

e. Perform standard Jira functions such as assignment and escalation.

Extensibility Requirements

1. The Solution shall be developed based on open-source technologies and be easily extended.

2. The Solution shall offer the ability to write scripts for custom templating, transformations, workflows, and visualizations including:

a. Additional graph functions (as administrative JavaScript extensions).

b. Additional graph lenses (visual transformations, as JavaScript extensions).

c. Graph styling configurations (analyst runtime scriptable).

d. Record visualization templates (analyst runtime scriptable).

e. Scripted data columns and automation scripts (dashboard navigations, status changes, and so on).

f. The Solution shall be able to be deployed in production by using common distribution bundles using DevOps scripts are provided to facilitate deployments and updates to installations that have a large or variable number of nodes.

Templating and Reporting Requirements

1. The Solution shall support a record viewer page that can be customized with templated scripts to show a view which can be created dynamically for each entity type or subsearch.

2. The Solution shall be able to generate formatted reports from user-defined templates.

3. The Solution shall be able to generate reports in PDF and DOCX formats.

Security Requirements

1. The Solution shall support enterprise-level access control at the index, record, and field levels.

2. The Solution shall support the following Elasticsearch security options:

a. Elasticsearch subscriptions (formerly X-Pack).

b. Search Guard.

c. Open Distro Security for Elasticsearch.

3. The Solution shall support LDAP and Active Directory authentication and authorization.

4. The Solution shall be able to establish individual and role-level access control for individual UI components, such as dashboards, visualizations, and saved search.

5. The Solution shall implement restrictions on data access.

6. The Solution shall provide end-to-end encryption from the user interface down to inter-cluster communications, and Elasticsearch shall itself be encrypted.

7. The Solution shall provide authentication at least through the following methods:

a. Active Directory/LDAP.

b. OpenID.

c. JWT.

8. The Solution shall enable user authorization through roles provided by the authentication method.

9. All data utilized by The Solution shall be encrypted, including when it is in transit between Elasticsearch and client components or between different Elasticsearch nodes.

Security Architecture and Auditing

High-Level Diagram of Sirenuggested Security Architecture

1. The Solution shall support role-based access control on Elasticsearch indices, documents, and fields.

2. The Solution shall support role-based access control on JDBC tables mapped to Elasticsearch indices.

3. The Solution shall support full integration with Search Guard add-ons for field-level security, document-level security, and configuration management.

4. The Solution shall support TLS on transport layer and REST API.

5. The Solution shall support pluggable authentication methods (LDAP, Kerberos, JWT, etc.).

6. The Solution shall support adaptive user interface with per-role access rules.

7. The Solution shall support field level anonymization for improved compliance with GDPR and other Personally Identifiable Information protection standards.

8. The Solution shall support the following audit logging:

a. AUTHENTICATED - A user has been authenticated successfully

b. GRANTED_PRIVILEGES - represents a successful request to Elasticsearch.

c. FAILED_LOGIN - the provided credentials of a request could not be validated, most likely because the user does not exist, or the password is incorrect.

d. MISSING_PRIVILEGES - an attempt was made to access Elasticsearch, but the user does not have the required permissions.

e. BAD_HEADERS - an attempt was made to spoof a request to Elasticsearch with the Solution’s internal headers.

f. SSL_EXCEPTION - an attempt was made to access Elasticsearch without a valid SSL/TLS certificate.

g. SG_INDEX_ATTEMPT - an attempt was made to access The Solution internal user and privileges index without a valid admin TLS certificate.

9. The Solution shall support logging the actions that are performed by the logged-in user.

10. The Solution shall log the following types of session data:

a. Actions that are performed by the user as they are interacting with the application. For example, a log entry is created when a user opens a dashboard.

b. Access to system configuration objects. For example, a log entry is created when a user requests a configuration object for particular dashboard.

c. All data-related search queries. For example, a log entry is created when a visualization on a dashboard is fetching the data to render.

d. All queries that obtain the document count. For example, a log entry is created when a query is used to fetch the count of documents visible on a dashboard.

e. A log entry is created each time a user attempts to export data as JSON or CSV.

f. A log entry is created each time a user send request from Dev tools console.

g. All internal data-related search queries. For example, an internal log entry is created when the system periodically issues a request to fetch license information.

image1.png