PL 006 TRA MANAGED FILE TRANSFER.docx

TRA04.06 – Managed File Transfer (MFT) Services

TRA04.06 – Managed File Transfer (MFT) Services Version 0.14Prepared by:

South Carolina Department of Health and Human Services (SCDHHS) Enterprise Services (ES)

8/3/2017

Table of Contents

Table of Figures

Table of Tables

Table of Code Examples

Table of Syntaxes

Table of Standards

SCDHHS expressly restricts the distribution of this document to include only those SCDHHS staff, SCDHHS processing environment contractors, or any entity given explicit access to this document with SCDHHS executive or management approval.

Intended audience The SCDHHS Technical Reference Architecture (TRA) audience includes SCDHHS staff, trading partners, and third-party vendors who will be implementing, integrating, managing, or operating the SCDHHS Medicaid Enterprise System (MES). The SCDHHS TRA provides the reference model by which technologies and components of the SCDHHS MES will be measured for development and implementation.

Introduction Despite the advent of service-oriented architecture (SOA) and message-oriented middleware (MOM), a large portion of enterprise data exchange takes place by transferring files between systems. Most enterprises use File Transfer Protocol (FTP) server and client technology to transfer files. However, the basic file transfers using, for example, FTP or Hypertext Transfer Protocol (HTTP), which were adequate when introduced, do not address present-day rigorous enterprise requirements such as, control over sensitive information, global visibility, and integration with other enterprise systems.

Managed file transfer (MFT) applications use FTP as their foundation and add software components and communication protocols to provide services, such as the following.

Reliably transferring data using secure protocols, including SFTP, FTPS, and HTTPS.

Automatically transferring data, for example, transferring data based on events, schedules, file names, etc.

Easily integrating with other technologies in the environment, such as the enterprise service bus (ESB).

Monitoring file transfers.

Logging processes, errors, and audit trails.

This document examines MFT services and their use to exchange data in the South Carolina Medicaid Enterprise System (MES) environment. See the TRA04.02 – Web Services, TRA04.05 – Extract, Transform, and Load (ETL) Processing, and TRA04.07 – Electronic Data Interchange (EDI) supplements for information on other methods to exchange data. See the TRA06 – Technology Products Portfolio supplement for the technologies approved for use in the MES.Figure 1. Managed file transfer in the MES

MFT in the MES Architecture The MES uses a service-oriented architecture with system functionality implemented into loosely coupled components and subsystems. (See the TRA02 – Shared Services Service-Oriented Architecture (SOA) supplement for a description of the architecture.) The collection of tools and applications that integrate the MES modules data includes MFT. Figure 1. Managed file transfer in the MES highlights integration points where MFT services could transfer data files in a secure, efficient, and reliable manner.

The following examples describe some of the integration points.

The MFT application monitors a trading partner external file server directory and transfers files placed in that directory to the ESB.

An MES component, such as the member eligibility system, places a file in the MFT embedded file server, and MFT transfers the file to the ESB.

An MES application, such as the ESB, sends a file to MFT, and notifies a trading partner that the file is available to pull from the embedded file server.

Figure 2. Detailed view of MFT in the MES shows more details about the use of MFT in the MES. MFT can transfer files to and from other the integration hub applications, such as the ESB and ETL, and between trading partners and the MES. The file transfers occur using industry standards, such as FTP, SSH File Transfer Protocol (SFTP), and Hyper Text Transfer Protocol Secure (HTTPS).

Figure 2. Detailed view of MFT in the MES MFT also facilitates transfer of files to and from applications outside the MES. The right side of .

Figure 2. Detailed view of MFT in the MES shows that the source system can place files on a server external to the MES, where MFT can push/pull the files to the appropriate MES application. Source systems can also push/pull the files from the file server embedded in the MFT.

Registration The TRA04 – Integration Hub supplement describes the inventory of services available through the MES, and the considerations for setting up a service. Setting up a service to transfer files using the MFT requires an additional step—completing the MFT registration. Only registered users can access the MFT application.

The users of the MFT application fall into two broad categories:

File server users include:

· Individuals or systems that send and receive files from the MES, such as claims processing agencies, bills processing agencies, or state agencies.

· MES applications (for example, ETL and ESB) that perform specialized tasks.

Application console users perform design or administrative tasks, such as designing interfaces, scheduling transfers, monitoring transfers, etc.

TRA04.06-S1. Users cannot self-register for access to the MFT application.

Administrators have access to register users; users cannot self-register for access to the MFT application.

File Server Users File server users access the MFT embedded file server to send and receive files. These users log into the application using FTP client software. The users can also access the file transfer application programming interfaces (API) provided by the MFT to programmatically upload/download the files.

Upon registration, SCDHHS provides each file server user with the following:

Unique username and password Directories with appropriate access rights to read and/or write Required disk space in the directories Encryption or decryption keys (if applicable) Application Console Users The MFT application grants the appropriate access to application functionality based on the user role, for example, administrative users typically have access to read, write, edit, and delete files, while non-administrative users only have read access.

MFT Process Standard The standard for MFT processes includes a structure for process steps and operational considerations.

Process steps include: Figure 3. MFT process

Authentication and authorization Receipt of the file on the file server Notification of the file transfer to the intended recipient (target) Receipt of the file by the target Archival of the transferred file Functionality common to the process steps include:

Capturing process events in logs and audit trails Notification of events Exception handling Secure data transmission Encryption/decryption Compression/decompression Operational considerations include:

High availability Automation of file transfers Naming conventions of folders and files processed by the MFT Administrative access to monitor and manage selected file transfer instances Aggregating MFT activities into configurable reports Authentication and Authorization TRA04.06-S2. The MFT service will adhere to the TRA01.02 – Access Control and Identity Management supplement.

A managed file transfer user must be properly authenticated and authorized to use the MFT service. Authentication refers to the process by which the username and password (or authentication token) is validated. Authorization is the process by which the MFT application determines which activities the logged in user is allowed to perform. See the TRA01.02 – Access Control and Identity Management supplement for more information about authentication and authorization.

TRA04.06-S3. MFT user access authorization will be defined by user roles.

Read and write access to the directories in the embedded file server are defined based on the role of the user. Directories and access rights (read, write, delete) can be associated to the roles so that a user with a certain role has access rights to all the folders associated with that role. The MFT console supports configuration of access rights and granting of those access rights based on the role of the user in the organization.

The MFT console enables the system administrators to assign roles and responsibilities to the MFT users and the access rights are granted based on the role of the user in the MFT application.

File Receipt After authentication, MFT allows a user/application to place a file in the embedded FTP server. The method the user/system employs to place the file in MES depends on the source system. The methods include:

· Manually using FTP client—the user logs in to the FTP (sFTP, FTPs) file server using a desktop client and drags the file(s) from the local machine to the FTP server.

· Manually using web client—the user drags the local file into the server using a browser or thin client. The MFT solution supports this mechanism in the MFT console. The roles and responsibilities of the user define access to this feature.

· Source-initiated program code—the batch processes that generate the files automatically place the files in the FTP server. Programmers usually design the process to dynamically define the location of the FTP server. The batch process code automatically logs in using the credentials of the system and places the file in the given location.

· MFT-initiated web service—the MFT application initiates a web service request to a service hosted by the source application. The source application sends a response message with inline payload or a reference to an FTP location. Then, the file is transferred to the target location.

Target Notification The MFT application can send an email or an SMS notification to the user upon successful completion of file transfer. The MFT application enables administrators to configure the recipients and the message format of the notification.

In most of the cases, the files are processed as soon as they are received or at a specified interval of time. For this reason, the target notification is not required.

Target Receipt and File Processing The MFT application transports the file to a target location, for which the MFT application has valid credentials to access. To obtain access, register the MFT application as a user with the target location.

The target location could be any of the following types:

FTP, sFTP – if the target location is a remote FTP server (not embedded in the MES MFT), the MFT application must obtain the following parameters of the target location:

· Host name

· Folder/directory to receive the files

· User who has access to the target

· Password

· Control port of target

· Authentication type, such as SSL, password, or public keys Remote network location—if the target location is a local machine or a network drive, the MFT application must obtain the network address of the system and the folder where the file needs to be placed.

Web service/ SOA—if the target location is a web service, the MFT application sends the file either as an inline payload or as a reference to the embedded FTP server. To invoke this service, the MFT application needs to know the endpoint URI of this service. For more information about the endpoint URI, refer to the TRA 4.02 Web Services supplement.

File Archival TRA04.06-S4. The MFT application will hold archival copies of files exchanged in the MES for a predefined period.

After transferring the file to the target location, the MFT application archives the file. If a file is lost or corrupted during transfer, the MFT application can resend the archived file. Having an archived file eliminates reproducing the original file, which may not be easy to do if the state of the source database has changed since the creation of the lost or corrupted file.

The MFT application places file archives in a predefined location based on the directory naming convention standards described in Section 4.7.3. File-Naming Conventions.

TRA04.06-S5. Access to archived copies of files will be allowed based on the archive access administrative process.

TRA04.06-S6. The archived files will be stored in a compressed format.

The archive file directories could grow in size very quickly, and therefore, archived files are stored in a compressed format.

MFT Functionality An MFT application performs more than just transferring files; it provides a rich set of built-in functionality that improves the overall control, security, and reliability of file transfers. This section describes MFT application functionality.

Process Logging and Audit Trails Process logs and audit trails help investigate and troubleshoot potential failures and errors in enterprise data interfaces. In the event of a missing file or an error in the file transfer process, the operations staff can utilize audit trails to help track and narrow down the point of failure.

The logs and audit trails also demonstrate regulatory compliance for safe handling of files by the MFT application. Documenting user activity and the resources users access provides information that is helpful in investigating data leaks.

TRA04.06-S7. The MFT application will log process and user activity data.

Log data must be both comprehensive and easy to analyze. The MFT application captures extensive data in a standard format, such as an SQL database or a structured file. Two types of audit data are critical—the job and file data and user activity data.

Data elements captured in job and file data include:

File name Source location Target location Transfer initiating system (MFT/source name) Initiating procedure name Initiation time Completion time Size of the file Transfer status The data elements captured in user activity include:

Usernames Log-in time Session length Transfers initiated Folders accessed Files read/updated/deleted TRA04.06-S8. The MFT application will maintain an audit trail of file transfers.

The MFT solution maintains an audit trail of file transfer across the enterprise to serve the following purposes: security, diagnosis, and regulatory compliance. Audit logs will:

Adhere to all regulatory requirements on an ongoing basis Be regularly reviewed Ensure non-repudiation Be appropriately retained Notification The MFT application can send notifications when specified events occur, such as the delivery or failure to deliver a file.

TRA04.06-S9. The MFT application will send notifications on certain events.

The MFT application sends notifications to preconfigured mailing lists upon occurrence of certain events.

The system administrators will turn on and turn off such notifications to events such as successful file transfer, file read, and file deletion Exception Handling TRA04.06-S10.The MFT application will adhere to the enterprise error-handling guidelines.

The MES provides a uniform enterprise-wide exception-handling mechanism with a repository of error codes and descriptions. In the event of any failure in the file transfer process, the MFT application handles the exception according to the enterprise error-handling guidelines.

The goal of the error-handling procedure in the MFT application is to move the error-causing file to a dedicated folder and to send notification(s) to the source system with the information such as:

· File name

· File size

· File format

· File encoding

· Target location

· Reason for failure

· Error code

· Error description Reasons for a file delivery failure include:

· The target location is not available.

· The target web service is not responding.

· The target location cannot decrypt the file.

Security A major goal of the MFT application is to ensure that the sensitive and proprietary data moves safely without interception or alteration. Section 3.2. Authentication and Authorization describes the key aspect of MFT application security—authentication. This section describes additional standard methods of secure data transmission. These standards allow the receiving applications to trust the content in the files and to process them directly without additional validations or manual interventions. The TRA01.02 – Access Control and Identity Management supplement also describes security requirements.

TRA04.06-S11. The MFT will support only HTTP/S and S/FTP file transfer protocols.

The MES supports only the following secure protocols to exchange the files internally as well as with external trading partners.

Description

TRA04.06-S12. The MFT application will use Pretty Good Protection (PGP) encryption for data security.

In addition to using secure protocols during transmission, the MFT application can provide another level of security for the data rest using Pretty Good Protection (PGP) encryption. SSL FTP and sFTP encrypted sessions are suitable for protecting data when in transit; however, when that data lands on the destination FTP server, the data may not be inside a firewall and could be exposed.

PGP allows a user to automatically or manually encrypt the files. In brief PGP, allows users to create a PGP key pair, a combination of public and private keys to encrypt and decrypt the files. While the public key can be distributed among the interested parties, the private key must be kept secret with the user.

The MFT application uses the private key to encrypt the file and send it to the destination. The target decrypts the file prior to processing.

Compression/Decompression TRA04.06-S13. The MFT application will be able to use standard compression techniques.

The MFT application can use standard compression techniques, such as Zip, to compress large files before sending the files to a target application.

TRA04.06-S14. The MFT application will support compressing multiple files into one file.

The MFT application supports compressing multiple files into one file before sending across to target applications, reducing network traffic.

Operational Considerations A number of operational considerations ensure a successful and consistent MFT platform.

High Availability The MFT solution is implemented in clusters to support high availability and to ensure the SCDHHS meets service-level agreements (SLA) with trading partners. If one MFT application web server fails, other servers continue to support the file transfer operations.

TRA04.06-S15. Only one MFT server will access/process a file.

The high availability configuration ensures that at any point, only one server is accessing and processing a particular file.

TRA04.06-S16. If an MFT server fails during a file transfer, the transfer will resume/restart when the server restarts.

If a server fails during the file transfer, the transfer is resumed/resubmitted when the server restarts, and thus, there is never a loss in the file transmission.

TRA04.06-S17. The MFT application will not constrain file transfers based on capacity.

The MFT application is configured with no limitations or exceptions in terms of file sizes, usages, users, etc.

TRA04.06-S18. The MFT application will ensure timely data delivery.

The MFT application must ensure that it delivers data on time and within the SLAs established with the trading partners.

Automation The MFT application is highly integrated with the ESB and has the ability to invoke processes in the ESB, which helps automate the execution of batch processing based on file transmission.

TRA04.06-S19. The MFT application will support event-driven automation.

The MFT application initiates a particular action at the occurrence of a particular system event. For example, the MFT application watches for the arrival of a file to a specific folder and upon receiving the file, sends a message to the ESB to initiate the execution of corresponding processes and workflows.

TRA04.06-S20. The MFT application will support scheduling file transfers.

The MFT application processes files at a pre-defined interval. For example, if a trading partner sends claims throughout the day to the MFT application, instead of picking up a claim for processing each time a file arrives, the MFT application can be configured to send all claims for that trading partner to the processing application at a specified time each day.

TRA04.06-S21. The MFT application will support name-based routing.

The MFT application is configurable to write rules, based on the file names (supporting regular expressions and wildcard characters), regarding which systems are notified and/or to which system files are transmitted. For example, a trading partner sends three kinds of files, each of which needs to be processed by a different application. The MFT application sends the files appropriately for processing based on the name of each file.

TRA04.06-S22. The MFT application will support out-of-the-box (OOTB) integration.

The MFT solution has OOTB integration capabilities to integrate with other technology and tools used in the MES. For example, the MFT application has OOTB functions/drivers to connect to SOA, ODI, OSB, and business-to-business (B2B). OOTB integration not only ensures near seamless file-based data exchange between various applications, but also minimizes development team design and research effort.

TRA04.06-S23. The MFT application will support custom code.

In addition to sending and receiving files, the MFT application performs pre-processing and post-processing actions on the file system. Some of the complex pre- and post-processing actions are not configurable using the OOTB tools. For this reason, the MFT application supports configuration using custom code in Java or any other standard programming language. Examples of processing actions include:

Adding headers and footers to files Validating file content Invoking separate applications in the event of success or failure of file transmission File-Naming Conventions The MES defines file-naming conventions for all files passing through the enterprise. File-naming conventions not only help identify the contextual details of the file, but also help automate the processing of files by the MFT application and other file processing systems, such as the ETL and ESB.

TRA04.06-S24. Folders and files processed by the MFT application will follow naming conventions.

Naming conventions for the file server folders, or directories indicate where the file is sent and who has access to it.

Directory File Name Conventions Syntax 1. Directory file name provides the naming convention for directories.

Syntax 1. Directory file name Root/Mount/Environment/Status/TradingPartner/Direction Code example 1. Directory file name shows an example folder name.

Code example 1. Directory file name Oracle/acfs/mft/prod/active/maximus/inbound Table 1. Directory file name component definitions describes the file name components.

Table 1. Directory file name component definitions

EDI File Name Conventions EDI has two different file naming conventions—one for outbound files and one for inbound files. Inbound messages use the format shown in Syntax 2. Inbound EDI message file-naming convention and Syntax 3. Outbound EDI message file-naming convention shows the convention for outbound EDI messages.

Syntax 2. Inbound EDI message file-naming convention FromParty_ToParty_DoctypeName_DoctypeRevision_Timestamp.extension Code example 2. EDI inbound message file name example shows an example of an EDI inbound message file name.

Code example 2. EDI inbound message file name example Maximus_scdhhs_834_5010a_2016101612:12:12.pff Outbound messages use the format shown in Syntax 3. Outbound EDI message file-naming convention.

Syntax 3. Outbound EDI message file-naming convention FromParty_ToParty_DoctypeName_DoctypeRevision_MsgID_Timestamp.extension Code example 3. Outbound EDI message file name example shows an example of an EDI outbound Code example 3. Outbound EDI message file name example Scdhhs_mmis_834_5010a_2016101612:12:12.pff Table 2. EDI file name component definitions defines the file name components.

Table 2. EDI file name component definitions

Non-EDI File Name Conventions Incoming and outgoing files processed through ESB and ETL use the format shown in Syntax 4. Non-EDI file name conventions.

Syntax 4. Non-EDI file name conventions SourceSystem_FileType_Version_Timestamp.extension Code example 4. Non-EDI file name example shows an example of a non-EDI file name.

Code example 4. Non-EDI file name example MMIS_Member_v1_2016101612:10:10.txt Table 3. Non-EDI file name component definitions provides definitions for the non-EDI file name components.

Table 3. Non-EDI file name component definitions

Monitoring Dashboard The MFT application provides a monitoring dashboard to view real-time transfer sessions and user statistics. This dashboard graphical view contains information about the file transfer instances, such as the source, target, transfer status, size, etc. Typically, an administrator accesses the monitoring dashboard in the event of unusual application behavior.

The monitoring functionality also provides the application administrators a means of control to start, pause, resume, or cancel file transfers. Control of file transfers is especially useful when the administrative discretion is needed while transferring important files, very large files, and a large volume of files.

Reporting Instead of using audit logs in raw form to data-mine the relevant information, the MFT application generates reports that aggregate information, such as time stamps, participating applications being invoked, file names, file sizes, destinations, etc.

TRA04.06-S25. The MFT application will support configurable reporting capabilities.

The MFT application includes on-demand and on-schedule reports, interactive monitoring, automated report creation, and appropriate access control of these reports to the users.

Use Cases This section examines some uses of MFT services in the MES and explores design options.

Receiving Files Inbound to the MES This example shows the path a file takes when sent by an external application to an MES module. Security considerations and the technical ability of all the parties involved determine whether to place the file on an external SFTP server where the MFT application retrieves the file (option 1), or to place the file directly in the embedded MFT server (option 2).

Option 1: Retrieve File from the External Server Figure 5. Use case to place the file on an SFTP server outside the MES illustrates the first option, where an application not in the MES, such as the ASO or DASO, places the file in an external server. The two parties agree on the directory location in the file server during the design stage of the interface. The MFT application monitors the file server location and when the MFT application detects a file, the MFT application brings the file to the server embedded in the MFT application. Then, the MFT application sends the file to the appropriate MES application.

Business reasons to choose this option include:

The external application refuses to part with their current practice of placing all outgoing files in a particular server directory.

The MES security team denies access to the external system for MFT file server.

Figure 5. Use case to place the file on an SFTP server outside the MES Option 2: Place File on the MFT File Server Figure 6. Use case to place file on the MFT file server illustrates the second option where the external application places the file directly in the MFT file server. The MFT application then sends the file over to internal applications, such as ETL. Option 2 eliminates one file hop (additional step) and the need to archive the file in the skipped location.

Business reasons to choose this option include:

The MFT application does not have access to the external file server.

A new trading partner agrees to place the file in the MFT file server.

Figure 6. Use case to place file on the MFT file server Sending Files from the MFT Application to the ESB Figure 7. Use case for using the ESB to send the file to an application illustrates the case where the ESB and an application each expose a web service. The ESB web service allows systems to send files for ESB processing. The application web service also enables/allows the partnering systems to send files for application processing. In this option, the MFT application sends the file to the ESB using the web service, and the ESB sends the file to the EDI (via a web service).

A business reason to choose this option is an enterprise design decision to route the processing of all files through ESB to leverage integration capabilities.

Figure 7. Use case for using the ESB to send the file to an application Handling Files in Web Service Calls This use case explores how the MFT application sends a file to the ESB. The options include sending the file directly (that is, embedded as part of the XML), or indirectly (that is, a reference to the actual SFTP file location).

Option 1: Send the File Directly to the ESB In Figure 8. Use case for sending the file directly to the ESB, the MFT application sends the file using the web service exposed by the ESB. The web service schema contains a placeholder for the file to be sent in binary format. In this case, the ESB consumes the XML request message sent by the MFT application, and the ESB retrieves the file content from the message.

Business reasons to use this option include:

Design simplicity.

The ESB can process the file directly without making an additional call to the MFT application.

Figure 8. Use case for sending the file directly to the ESB Option 2: Send the File Location to the ESB In Figure 9. Use case for sending the file location to the ESB, the MFT application also uses the web service exposed by the ESB. However, in this case, the request message contains only the reference to the file, that is, the SFTP location of the file. The ESB then reads the file directly from the SFTP location.

Business reasons to use this option include:

The ESB passes the file location to another application without having to process it internally, thereby avoiding unnecessary network traffic.

If a large number of files arrive for processing at the same time, the MFT application payloads with embedded files can congest the network. Sending the file location eases the traffic.

Provides the ESB an option to de-batch the file processing; that is, in the event of multiple files, the ESB can take all the file locations and chose to process them, a few files at a time. This helps reduce the load on batch processors.

Figure 9. Use case for sending the file location to the ESB Acronyms

Referenced Documents

8/3/2017 SCDHHS - Confidential Page 6 of 6 image1.png image2.png image3.png image4.png image5.png image6.png image7.png image8.png image9.png image10.png image11.png image12.png image13.png image14.png image15.png