Attachment III - Evaluation Criteria (Dec 7).docx

EVALUATION FACTORS FOR AWARD

The Government intends to use a price/non-price tradeoff process to award a firm-fixed price purchase order to the responsible offeror whose quotation represents the best value to the Government. The Government reserves the right to make an award to other than the lowest-cost/priced offeror or to the offeror with the highest technical rating if the Contracting Officer determines that doing so would result in the best value to the Government. The Contracting Officer intends to award without discussions, but reserves the right to initiate discussions, if necessary.

Quotations that are unrealistic in terms of technical or schedule commitments, or may be deemed to be reflective of an inherent lack of technical competence, or indicative of a failure to comprehend the complexity and risks of the proposed work and may be grounds for rejection of the quotation as being technically unacceptable. Failure to provide the information required to evaluate a quotation could result in rejection of the quotation as being technically unacceptable.

Two Phase Evaluation

The Government will evaluate quotations in two phases as follows. The Contractor must be capable/willing of participating in Phase I and Phase II in order to be considered for award:

Phase I – Evaluation of Technical Volume

NIST will perform an initial technical evaluation to determine that the offeror has clearly demonstrated its technical capability to meet a subset of seven (7) of the minimum specifications identified under the minimum requirements. This subset includes specifications 1, 6, 8, 14, 15, 19, and 37. These specifications will be used to determine if the Contractor can participate in Phase II: Atifact Testing.

This initial evaluation is only to determine whether the quotation demonstrates an offeror’s capability to meet specifications 1, 6, 8, 15, 15, 19, and 37. Quotations that do not clearly demonstrate the offerors compliance to this subset of the minimum specifications will not be considered further for award. The offeror must meet all the subset minimum specifications identified under this phase to participate in Phase II. It is recommended to provide system drawings or sketches to demonstrate the capability to meet or exceed specifications 1, 6 and 19; component specification sheets to demonstrate the capability to meet or exceed specifications 8, 14, and 15; and a detailed written statement to demonstrate the capability to meet or exceed specification 37.

If a Contractor is found to meet the subset of seven (7) of the minimum specifications the Contractor shall be accorded the ability to participate in Phase II. For those participating in Phase II, a full technical capability and past performance evaluation will be completed while the artifact testing is being completed This evaluation shall include a review of the Contratctor’s proposal for the following factors:

Factor A – Technical Capability: The Government will evaluate the offeror’s technical capability. Evaluation of technical capability will be based on the information provided in the quotation. NIST will evaluate whether the offeror has demonstrated that its proposed system and services meet or exceed all minimum requirements. Quotations that do not demonstrate the proposed system and services meet all requirements may not be considered further for award. Specifically, if documentation submitted by an offeror does not indicate whether its proposed system and services meet a certain minimum requirement, NIST may determine that it does not. NIST will give preference to offerors who demonstrate that their proposed system and service exceeds the requirements of the Specifications. Specifically, NIST will give stronger consideration to the following specifications (i.e. listed in descending order of importance): 8, 19, 37, 6b, 35b, 6a, 33c, 30, 7a, 31, 33b, 35a, 33a.

Factor C – Past Performance: The evaluation of past performance information will consider the extent to which the offeror’s past performance demonstrates the capability and capacity to provide high quality services within the scope of the SOW. The term “offeror” here is defined as the prime contractor and any of its primary subcontractors. The Government may evaluate Past Performance Information Questionnaires that are submitted, and any past performance information that the Offeror provides in its technical quotation. The Government may also use data obtained from other sources. The Government will consider the relevance of the Offeror’s previous contracts to the subject requirement, and the Offeror’s performance on the previous contracts completed or substantially completed within the last five years. Offerors without a record of relevant past performance or for whom information on past performance is not available, will not be evaluated favorably or unfavorably on Past Performance, and will be assigned a Neutral rating. The Past Performance Information Questionnaire can be found as an attachment to the solicitation.

Phase II – Artifact Testing

Offerors that are found to meet the initial evaluation of the subset of seven (7) of the minimum specifications from the Phase I evaluation will be sent two (2) NIST-provided artifacts – Artifact A and Artifact B. Offerors will be required to perform the measurements identified below.

Offerors shall have two weeks from the date of receipt of the NIST artifact to perform the specific process testing, as specified by the Government. Additional time requests will be considered for those offerors who need to ship the artifact out of the country for measurement. Any request for additional time beyond the two weeks shall be provided as part of the Contractor’s solicitation response. Offerors shall return the NIST artifact to NIST, via overnight delivery. All shipping charges for transporting the Artifacts to the offeror and return costs shall be paid by the Government. Artifact A will be calibrated by NIST before and after each offeror; Artifact B will be visually inspected for damage before and after each offeror. Offerors shall provide an electronic copy of the measurement test results to the NIST Contract Specialist and Contracting Officer.

In the event that an offeror does not have testing access to a system that has identical configuration to the system being proposed to NIST, these measurements shall be performed using a system that has the same software, source, detector and basic construction as the system that would be provided. Any set-up and/or calibration must be reproducible at NIST at the time of Contractor installation. All participation of artifact testing shall be voluntarily by the Contractor and shall incur no additional cost to the Government.

ARTIFACT A

Artifact A is an aluminum structure, of approximately 63 mm diameter and 65 mm height, containing twenty-four 3 mm nominal diameter fused silica spheres – nine each on a plane on the top and bottom of the structure and four on the sides of the cylinder. No CAD model will be provided.

Artifact A measurement instructions For Artifact A, two (2) measurements shall be performed. For these measurements, the offeror may choose the parameters that would achieve the highest performance, provided the positioning requirements listed for the two measurements are met and a circular trajectory is used. If the system has a dual tube configuration, the offeror may select which tube is used.

Position Requirements for Artifact A:

Let d be the source to rotation axis distance and D be the source to detector distance, then for Artifact A:

Artifact A measurement 1 position: Position the stage and detector such that the maximum possible area of the detector is filled. If there is more than one setup by which this can be achieved, the offeror shall select the setup with the lowest magnification.

Artifact A measurement 2 position: Move the detector as close to the source as possible. With the detector fixed at this position, move the stage as close to the detector as possible. For example, d = 200 mm, D = 370 mm, fills about 35% of a 300 mm x 300 mm detector.

Reporting requirements for Artifact A:

The offeror shall provide comprehensive information (in .pdf, .xlsx, or .docx format) of the measurement technique and user-selectable parameters, and reconstruction settings for each of the three (3) measurements (two of Artifact A and one of Artifact B), including but not limited to the following:

· Voltage, current, and power

· Prefilter material/thickness

· Frame rate or exposure time

· Number of frames per projection

· Number of projections

· Source-to-detector distance, source-to-object distance, and other geometry information

· Detector pixel pitch and other detector related information

· Detector binning

· Reconstruction algorithm type and reconstruction filter type (if applicable)

· Reconstruction binning

· All corrections applied

· Voxel size

· Sample positioning and a picture of the setup

· Support material and fixture

· The reconstructed images (with associated meta data) that can be opened in CT analysis software such as VG Studio Max 3.4. Optional stacks of .tif (or .tiff) files are also encouraged if not already provided in the format.

Additional required and optional reporting requirements for each of the two (2) measurements of Artifact A

· The center coordinate (in a common reference frame), size and form of all spheres, provided in Excel format (.xls, .xlsx) or Excel-compatible format. The surface points used in the evaluation of the lengths, size and form should evenly cover the hemisphere opposite from where the sphere is attached to the aluminum structure. An initial non-linear least-squares sphere fit (allowing the sphere center and sphere diameter to vary) should be done for each sphere, after which any point whose residual exceeds 3 times the standard deviation of all the residuals of the surface points may be removed. A second non-linear least-squares sphere fit (allowing the sphere center and sphere diameter to vary) should be performed on the remaining surface points. The center-to-center lengths, sphere sizes, and sphere forms should be derived from this second sphere fit. The spheres required for the above referenced testing shall be provided by the Contractor. NIST will not provide any additional artifacts for the above referenced testing.

· The first image of each CT scan and a second image, acquired with the reference object in the same position as the first image of the scan, which is acquired immediately following the CT scan (in some cases, this may be the final image of the scan). All images should be submitted in a format that uses lossless compression and can be opened using ImageJ (https://imagej.net/Importing_Image_Files).

· [optional] The offeror may additionally supply a specification for the maximum expected error in the size, form, and/or distance between sphere centers. The greater of either the maximum calculated errors in the submitted measurements or the manufacturer-supplied specification will be used in the evaluation.

ARTIFACT B

Artifact B is a complex structure with internal features that was produced using Laser Powder Bed Fusion (LPBF) metal Additive Manufacturing (AM) technique. It is approximately 35 mm x 12 mm x 8 mm in size. No CAD model will be provided.

Artifact B Measurement For Artifact B, the one (1) measurement shall be performed. The offeror may choose the XCT scanning setup and parameters to achieve the highest spatial resolution and image quality with the following requirements:

· Entire sample is measured

· Reasonable total XCT acquisition time (time to acquire the projections without reconstruction) is used per scan (< 1 hr)

· Any trajectory may be used

The offeror shall minimize imaging artifacts such as cone beam artifact, beam hardening artifact, ring artifact, streaking artifact, and scattering artifact (if applicable) through advanced acquisition techniques or correction algorithms. Imaging artifact refers to any feature which appears in an image and which is not present in the original imaged object (not to be confused with the test artifacts).

Reporting requirements for Artifact B The offeror shall provide comprehensive information (in .pdf, .xlsx, or .docx format) of the measurement technique and user-selectable parameters, and reconstruction settings for each of the three (3) measurements (two of Artifact A and one of Artifact B), including but not limited to the following:

· Voltage, current, and power

· Prefilter material/thickness

· Frame rate or exposure time

· Number of frames per projection

· Number of projections

· Source-to-detector distance, source-to-object distance, and other geometry information

· Detector pixel pitch and other detector related information

· Detector binning

· Reconstruction algorithm type and reconstruction filter type (if applicable)

· Reconstruction binning

· All corrections applied

· Voxel size

· Sample positioning and a picture of the setup

· Support material and fixture

· The reconstructed images (with associated meta data) that can be opened in CT analysis software such as VG Studio Max 3.4. Optional stacks of .tif (or .tiff) files are also encouraged if not already provided in the format.

Additional reporting for Artifact B ASTM E1695-20 standard (Standard Test Method for Measurement of Computed Tomography (CT) System Performance) test parameters using the provided cylinder: Plots of Modulation Transfer Function (MTF) curves and MTF10 values (limiting frequency at 10% modulation) at three slice planes as described in ASTM E1695-20 6.2.3.

Basis for Award – After completion of Phases I and II, the results from the Technical Evaluation in Phase I and the results of the Artifact Testing will be reviewed to make an award determination. The following evaluation factors will be used for this determination. Factor A is the most important factor, Factor B is the second most important factor, Factor C is least important factor.

Factor A – Technical Capability Factor B – Artifact Testing Results Factor C – Past Performance Factor D – Price

In determining best value, all non-price factors, when combined, are more importance than Factor D – Price.

Factor A – Technical Capability: The Government will evaluate the offeror’s technical capability. Evaluation of technical capability will be based on the information provided in the quotation. NIST will evaluate whether the offeror has demonstrated that its proposed system and services meet or exceed all stated specifications. Quotations that do not demonstrate the proposed system and services meet all requirements may not be considered further for award. Specifically, if documentation submitted by an offeror does not indicate whether its proposed system and services meet a certain specification, NIST may determine that it does not.

As shown in the Statement of Work, some specifications denote an opportunity for “Stronger Technical Consideration”. Specifically, NIST will give stronger consideration to the following specifications (i.e. listed in descending order of importance): 8, 19, 37, 6b, 35b, 6a, 33c, 30, 7a, 31, 33b, 35a, 33a. NIST will give preference to offerors who demonstrate that their proposed system and service exceeds the requirements of the Specifications.

Factor B – Artifact Testing Results:

NIST will perform an evaluation of the offerors test results from measurements of Artifact A per the below:

1. Deviations from NIST-calibrated values will be calculated for offeror-provided distances between sphere centers and sphere sizes. NIST-calibrated values will not be shared with offerors so that the results will reflect the performance of the system when measuring an uncalibrated workpiece. NIST will give stronger consideration to the extent that the calculated deviations are less than 20 m for inter-sphere distance and sphere size and the extent that the offeror-provided sphere form is less than 30 m.

2. NIST will perform an evaluation of the shift, in mm, of Artifact A’s position relative to the detector plane from the beginning to the end of the scan by comparing the first image to the last image and using the source-to-detector distance, source-to-object distance and pixel size to calculate the magnification.

NIST will perform an evaluation of the offerors test results from measurements of Artifact B per the below:

1. For the presence and severity of imaging artifacts by checking for apparent imaging artifacts that affect surface determination and general image analysis

2. For adequacy of the data quality for surface determination for dimensional measurement tasks by evaluating the data in a software capable of applying surface determination

3. For detectability of small lattice and pore features. NIST will give stronger consideration to offerors who demonstrate the average of the three slice planes total image unsharpness values (1/MTF10) less than 70 µm to the extent of reducing the value.

Factor C – Past Performance: The evaluation of past performance information will consider the extent to which the offeror’s past performance demonstrates the capability and capacity to provide high quality services within the scope of the SOW. The term “offeror” here is defined as the prime contractor and any of its primary subcontractors. The Government may evaluate Past Performance Information Questionnaires that are submitted, and any past performance information that the Offeror provides in its technical quotation. The Government may also use data obtained from other sources. The Government will consider the relevance of the Offeror’s previous contracts to the subject requirement, and the Offeror’s performance on the previous contracts completed or substantially completed within the last five years. Offerors without a record of relevant past performance or for whom information on past performance is not available, will not be evaluated favorably or unfavorably on Past Performance, and will be assigned a Neutral rating. The Past Performance Information Questionnaire can be found as an attachment to the solicitation.

Factor D – Price: The Government will evaluate the Offeror’s proposed pricing information to determine the extent to which the pricing is fair, reasonable, and consistent with or discounted from prevailing market pricing information.

For non-small business offerors, the Government will further evaluate the extent to which offerors identify and commit to small business and historically black college or university and minority institution performance of the contract, whether as a joint venture, teaming arrangement, or subcontractor.

image1.png image2.jpeg