Attachment_4_Technical_Requirements.pdf

Attachment 4: Technical Requirements Page 1 of 6 Sourcing Event ID: 0000011726

Attachment 4: Technical Requirements RFQ for Products, Software, and Services powered by Emerging Technologies

Sourcing Event ID: 0000011726

Introduction The purpose of this document is to outline the technical requirements that Proposers must comply with at each phase of working with the City as part of this Emerging Technologies RFQ Pool (“RFQ pool”):

Required acknowledgements to join the Emerging Technologies RFQ pool

Requirements for Experimentation Short-Form contract (Step 1)

• Sandbox Requirements

• Prototype Testing Requirements

• Security and Privacy Assessment

• Data Requirements

Requirements for Long-Form Production Contract Amendment (Step 2)

• Cybersecurity

• Data requirements

• AI Transparency

• Public-Facing Content

To the extent not already covered, the City reserves the right to incorporate these requirements into short-form contracts and/or long-form contract amendments entered into with Proposers.

Required acknowledgements to join the Emerging Technologies RFQ pool When applying to join the Emerging Technologies RFQ pool, Proposers must acknowledge that they have reviewed the entirety of these Technical Requirements and agree to abide by them at the different phases of experimentation and contracting. Proposers will complete this acknowledgment as part of their responses to the RFQ Response Form.

Proposers will be asked to acknowledge that they understand and accept:

• Contractual terms and conditions. When applying to join the RFQ pool, Proposers must review and acknowledge that they substantially agree to the City’s standard https://digital.forms.sf.gov/et_rfq

Attachment 4: Technical Requirements Page 2 of 6 contract terms for a software solution (Attachments 2, 3a, and 3b). This includes terms and conditions related to accessibility, cybersecurity, City network/systems integration, insurance, and indemnity provisions. The flexibility to negotiate these terms will be limited.

• Cybersecurity. Proposers must acknowledge that they will complete the City’s cybersecurity review prior to signing a full contract (see page 3).

• Data requirements. Proposers must acknowledge that they will meet all Data Requirements in this document.

• AI Transparency requirements: Proposers must acknowledge that they will provide the department with the information needed to ensure compliance with San Francisco Administrative Code Chapter 22J, including responses to the disclosure items in Section 22J.3(b)(1)-(-16).

• Sandbox requirements. Proposers must acknowledge that all prototyping and testing of AI/Emerging technology products under the RFQ pool must be conducted in a sandboxed environment that is logically isolated from all City production systems and data.

• Public-facing content requirements. Proposers must acknowledge that they will meet all standards and requirements when creating public-facing content, portals, or systems for the City.

Requirements for Experimentation Short-Form contract (Step 1)

Sandbox Requirements

• Proposers must conduct all testing in a sandbox environment that mirrors production security and access controls. All data used or generated must be public and/or synthetic, and must be deleted or returned to the City at the conclusion of testing.

• This ensures that City production systems and data are protected during evaluation.

No production data may be used in the sandbox environment without explicit City approval and appropriate data use agreements.

• Proposers cannot use City data to train or fine-tune models without explicit written consent from the City.

• All data generated by the pilot will be owned by the City and the Proposer must support the City’s ability to export or delete City data from the AI system upon request.

• Proposer must disclose any third-party AI services or sub processors used within the product (e.g., OpenAI, Azure AI).

https://codelibrary.amlegal.com/codes/san_francisco/latest/sf_admin/0-0-0-71391 https://codelibrary.amlegal.com/codes/san_francisco/latest/sf_admin/0-0-0-71391

Attachment 4: Technical Requirements Page 3 of 6

Prototype Testing Requirement

• Proposers must work with departments to define and document a testing approach that assesses overall system performance across key dimensions, including accuracy, completeness, fairness and bias, safety, security, and robustness. Methods and outputs must be shared with departments and be fully auditable. Testing should be conducted using representative benchmark datasets that include a range of typical and edge cases, and may incorporate methods such as automated evaluation and/or subject matter expert review.

• Proposers may be asked to complete disparate impact testing across demographic groups before deployment, and if bias is identified, create and implement a remediation plan.

Security and Privacy Assessment

• Proposers must assess and document privacy and security risks associated with the solution, including how training data, user inputs, and any connected data sources are handled. This includes documenting data collection, storage, retention, access controls, and applicable legal or policy requirements.

Data Requirements

• Document all City systems of record their product will read from or write to, including the data elements accessed.

• Document all data outputs (tables, datasets, model outputs) that their product produces that are intended for storage in the City’s Unified Data Platform (UDP) in Snowflake.

• Synthetic data must not contain PII and should be representative enough of production data to meaningfully evaluate system performance.

Requirements for Long-Form Production Contract Amendment (Step 2)

Cybersecurity

The City’s Cybersecurity Risk Assessment (“CRA”) is performed for each entity manufacturing the product, performing technical functions related to the product’s performance, and/or accessing City’s networks and systems – whether those activities are performed by the prime https://media.api.sf.gov/documents/CRA_Assessment_Guide_for_City_Suppliers.pdf

Attachment 4: Technical Requirements Page 4 of 6 contractor, a subcontractor, and/or a reseller. As part of this process, firms are required to submit one of the following documents to the City:

• SOC-2 Type 2 Report: Report on Controls at a Service Organization Relevant to Security, Availability, Processing Integrity, Confidentiality or Privacy;

• ISO/IEC 27001: The international standard for establishing, implementing, maintaining, and continuously improving an Information Security Management System (ISMS), ensuring systematic risk management of information security controls;

• CSA STAR L2: A cloud-specific certification from the Cloud Security Alliance that requires a third-party audit (attestation or certification) based on the Cloud Controls Matrix and ISO/IEC 27001, offering enhanced assurance and transparency beyond self-assessment;

• FedRAMP: A U.S. federal government-wide program that standardizes security assessment, authorization, and continuous monitoring for cloud services, enabling “authorize once, reuse across agencies” compliance aligned with FISMA;

• StateRAMP: A nonprofit-led program modeled after FedRAMP, tailored for state and local governments, providing a standardized, NIST SP 800-53-based cybersecurity authorization process for cloud service providers serving public-sector entities;

• NIST 800-53: A U.S. government publication offering a comprehensive catalog of security and privacy controls for federal information systems (organized in control families), widely adopted across sectors to manage confidentiality, integrity, and availability risks;

• HITRUST CSF: A certifiable framework harmonizing over 60 standards and regulations (e.g., HIPAA, PCI DSS, ISO, NIST) into a unified set of tailored security and privacy controls with a prescriptive, risk-based assurance model;

• PCI DSS Level 1: The highest compliance tier of the Payment Card Industry Data Security Standard, required for organizations processing over 6 million card transactions annually, mandating an annual audit by a Qualified Security Assessor and quarterly vulnerability scans; or

• City’s Cyber Risk Assessment Questionnaire: Proposer’s responses to a City’s Cyber Risk Assessment Questionnaire, which will be provided to Proposers who are selected for a short-term contract.

Once submitted, the report will be evaluated by the City to identify existing or potential cyber risks to City. Should such risks be identified, City shall afford the firm (Proposer) an opportunity to cure such risk within a period of time deemed reasonable to City. Such remediation and continuing compliance shall be subject to City’s on-going review and audit https://www.iso.org/standard/27001 https://cloudsecurityalliance.org/blog/2024/03/23/csa-star-level-2-all-about-star-attestations-and-certifications https://www.fedramp.gov/ https://govramp.org/ https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final https://info.hitrustalliance.net/cybersecurity-framework?utm_term=hitrust%20csf&utm_campaign=roi_ga_search_ams_br_why-ht&utm_source=adwords&utm_medium=ppc&hsa_acc=2724012343&hsa_cam=21184066499&hsa_grp=188666518009&hsa_ad=775856391524&hsa_src=g&hsa_tgt=kwd-309398207488&hsa_kw=hitrust%20csf&hsa_mt=e&hsa_net=adwords&hsa_ver=3&gad_source=1&gad_campaignid=21184066499&gbraid=0AAAAACedCYB6JRpcWDzzPKHXrbEWW5cF0&gclid=EAIaIQobChMI3pb3zo_SkwMVyB6tBh3lDBRkEAAYASAAEgJCXfD_BwE https://www.pcisecuritystandards.org/standards/pci-dss/?utm_source=chatgpt.com

Attachment 4: Technical Requirements Page 5 of 6 through industry-standard methodologies, including but not limited to on-site visits, review of the entities’ cybersecurity program, penetration testing, and/or code reviews.

Data requirements

• Integrations must use City-approved APIs or data connectors. Direct database access is not permitted without explicit approval.

• Proposers must provide a data flow diagram showing how data moves between the AI product and each system of record.

• All integration points must support role-based access control (RBAC) aligned with City identity management standards.

• Proposers must maintain an up-to-date data dictionary for any fields read from or written to City systems of record.

• Proposers must provide audit logging for all read and write operations to City systems of record, with logs retained for a minimum of 12 months.

• Data pipelines into UDP (Snowflake) must use City-approved integration patterns (e.g., Fivetran, dbt, or direct UDP [Snowflake] connectors). Ad hoc manual exports are not acceptable for production use.

• Proposers must provide schema documentation for all data written to UDP (Snowflake), including field names, types, and definitions.

• Data written to UDP (Snowflake) must include provenance metadata (e.g., source system, timestamp, pipeline version).

• Proposer must support incremental loading patterns to avoid unnecessary full-table refreshes unless full refresh is technically required.

• Data must be written to City-designated UDP (Snowflake) databases and schemas.

Proposers must not create ad hoc schemas without City approval.

• Proposer must define and document data quality checks performed prior to writing data to UDP (Snowflake) (e.g., null checks, referential integrity).

• Synthetic data used for testing must be generated or approved by the City and must meet City-defined guardrails on similarity to production data. Proposers must document the methods used to generate synthetic data and make that documentation available to the City upon request.

AI Transparency

• Comply with the city’s AI transparency ordinance and work with departments to address 22 questions that will be published on the city’s open data portal.

Attachment 4: Technical Requirements Page 6 of 6

• Proposer must work with departments to implement a process for continuous or periodic monitoring of AI model outputs for bias, disparate impact, and performance drift after deployment. The plan must include timelines for remediation and a process for notifying the City when a bias issue has been identified and resolved. Monitoring metrics and performance data will be tracked internally and made available to the public through an up-to-date AI Use Case Card (i.e., a publicly accessible document published on SF.gov that provides transparency about an AI system's purpose, functionality, and performance), consistent with Chapter 22D Open Data Policy.

Public-Facing Content

• Digital accessibility is a cornerstone of the City’s values so we ensure all residents and visitors can engage with the City regardless of disability status. As such, we require that all selected firms (Proposers) adhere to the requirements of the Americans with Disabilities Act and the Web Content Accessibility Guidelines (WCAG) 2.1 Level AA. At this point in the process, Proposers must attest that any public-facing technology they provide to the City is compliant with these standards; in certain cases, firms may also be asked to provide documentation of compliance

• Public-facing portals and websites must comply with the City’s Domain Registration and Management Policy and the SF.gov Subdomain Standard.

• Custom portals should have a clear, short, plain-English third-level name that describes their purpose. This must be appended to the appropriate official dot-gov domain (e.g., purpose.sf.gov) as a third level name.

• All custom portals must be clearly marked as official websites of the City and County of San Francisco. Moreover, SaaS portals should offer clear indicators of their official status wherever possible. This can be through a cross-link within the portal to an official dot-gov City webpage and consistent use of City seals and department logos.

• SaaS portals should use "sanfrancisco" or "sf" in the second or third-level subdomain, following the naming convention of the SaaS provider. Departments must avoid "sf" where it could be ambiguous, such as other organizations that use the same abbreviation on the same platform.

https://codelibrary.amlegal.com/codes/san_francisco/latest/sf_admin/0-0-0-13982 https://www.sf.gov/domain-registration-and-management-policy?preview=true&ts=20250206153444 https://www.sf.gov/domain-registration-and-management-policy?preview=true&ts=20250206153444 https://www.sf.gov/sfgov-subdomain-standard