Brian_Foo_Proposal_and_Budget.pdf

Citizen DJ LOC Innovator in Residence Program Project Proposal, LCCIO19D0052

Applicable interest areas: Artistic innovation, Creative computing

Brian Foo hello@brianfoo.com 551-265-9804 Website: brianfoo.com

Submitted: August 15, 2019 Last updated: August 31, 2019

BAA - 030ADV22R0041

mailto:hello@brianfoo.com http://brianfoo.com/

Table of Contents Volume 1: Technical Proposal 2

1.1 Introduction 2

1.1.1 Background 2

1.1.2 Phases of Work 2

1.1.3 Technological Innovation 4

1.1.4 Outcomes and Impact 4

1.1.5 Library Staff Time 5

1.2 Technical Approach 6

1.2.1 Project Schedule 6

1.2.2 Testing Plan 9

1.3 Capabilities and Experience 9

Volume 2: Price Proposal 11

Volume 3: Supplemental Information 12

3.1 SAM Registration 12

3.2 Intellectual Property 12

Appendix A: Visual references 12

Appendix B: Mock-ups 14

Appendix C: Referenced projects 16

BAA - 030ADV22R0041

Volume 1: Technical Proposal

1.1 Introduction

Citizen DJ is a project to enable hip hop music creation using public domain audio and video materials from the Library’s collections that may include oral histories, field recordings, government film, or ephemeral video.

1.1.1 Background

Since its beginnings in the 1970s, hip hop has become today’s dominant worldwide music genre and cultural movement. At the center of this movement is the DJ, or Disc Jockey, whose role is to discover, transform, and collage disparate and often obscure source materials from current and past cultures to create wholly new, relevant, and infectious music.

The golden age of hip hop was said to be in the late 80s to early 90s when DJs had nearly universal creative freedom to collage from found sounds. This small window of time produced landmark albums such as Public Enemy’s Fear of a Black Planet and De La Soul’s 3 Feet High & Rising, both considered to be culturally significant and selected for preservation in the National Recording Registry. These albums were dense and intricate sonic collages composed of hundreds of found sounds. However, the increasing popularity of hip hop in the following decade gave rise to high profile lawsuits and excessive restrictions on how audio could be sampled. Today, collage-based hip hop as it existed in the golden age is largely a lost and very expensive artform.

I believe if there was a simple way to discover, access, and use public domain audio and video material for music making, a new generation of hip hop producers can thrive, invent new sounds, and connect listeners to materials that might otherwise be hidden from public view.

1.1.2 Phases of Work

There are four phases of work in this project with a concrete deliverable for each phase.

I go into more detail in section 1.2.1 (“Project Schedule”) .

1. Research and discovery

I will work with Library staff to identify and access media collections that are in the public domain or available under a permissive license. Since recorded music is notoriously difficult to use for derivative works, I will consider all genres of recorded sound such as oral histories, field recordings, government film, and ephemera. I will also leverage the expertise of the staff to identify culturally significant or underused collections that may be of interest to the American public. This phase may also include requests for digitization of materials that only exist in physical formats.

The deliverable of this phase would be a public website that (1) outlines what types of materials can be used for music making and their restrictions, and (2) provides a curated list of collections that contain audio that can be used for music making. For each collection, I will provide a brief description of what it contains, how it can be accessed, what restrictions exist for derivative works, and some audio samples.

2. Access and processing

I will work with Library staff to access media assets from specific collections in bulk. This can be done via an API, FTP server, or manual file transfer. Once I obtain these assets, I will run them through a workflow of computer scripts that divide the audio into individual clips or samples. These audio samples will be the building blocks for music creation, where each sample will have a distinct sonic "pulse" as in a single note of a musical instrument or a single syllable in the case of speech. Each sample will then be analyzed for musical features such as loudness, pitch, timbre, and clarity. I may use machine learning to measure each sample’s similarity to a traditional instrument, so I can say that a sample is “drum-like” or “string-like.”

The deliverable of this phase would be a repository of audio sample packs with each sample labelled with a controlled vocabulary of categories and tags. These sample packs will be in a standard format that can be used in most mainstream music creation software. The computer scripts that generated these sample packs will also be published as open source software with documentation.

3. Music creation tools

I will implement a basic interface that can be used on the web, desktop, or tablet.

The functions of the interface will include: (1) searching and browsing audio collections from the previous two phases, (2) a simple sequencer for creating music loops and sequences, and (3) exporting musical sequences to audio files.

Since there are many existing music production software, this interface will focus on searching for and discovering sounds within large public audio collections.

The design will embrace the diversity and scale of the audio collections and encourage serendipitous moments of juxtaposing sounds from disparate sources in harmonious and surprising ways. It will also be designed for users who may not be familiar with music production software. Each audio sample will have clear and appropriate attribution and will be linked to its source if possible.

See Appendix B for mock-ups.

4. Music production

Lastly, I will use the interface from phase three to produce a conceptual music album made up of about 6 to 10 tracks. I hope for the tracks to feature different collections that resonate with me during my research from phase one. Each music track will also include documentation containing links to its sampled source materials.

1.1.3 Technological Innovation

I believe the core technological innovation lies in two areas. The first area of innovation lies in the workflow and open source software that processes and analyzes audio in bulk during phase two. One can think of this as indexing large collections of audio clips using sonic properties such as loudness, pitch, and timbre in addition to standard metadata. This enables the second area of innovation: providing new ways of searching and browsing large audio collections. For example, this can enable searching by a sample sound file or visualizing an entire audio collection as a point cloud organized by sonic similarity.

This would be comparable to the recent innovations of search, discovery, and visualization of large image collections where we can combine the strengths of human-created metadata, computer vision, and machine learning techniques. However, such innovations for working with large audio collections are still in their very early stages and largely not accessible to the general public.

1.1.4 Outcomes and Impact

The goal of this project is to open up public audio collections as source material for creativity and innovation for the general public. There are an increasing number of resources for access and reuse of public domain images and texts, such as Wikimedia, Flickr Commons, and Project Gutenberg, which drives creativity and innovation in the visual arts and literature. However, recorded sound has remained largely restrictive as creative source material from both a technical and legal point of view.

The four phases of this project map directly to four outcomes that reflect the entire lifecycle of creativity. The first phase will surface rich collections of recorded sound in the public domain that were previously unknown, difficult to find, or difficult to access by the general public. The second phase will provide the building blocks for the development of tools for search and discovery within large audio collections. The third phase will give creators, researchers, and educators tools for searching, browsing, and creating music with large audio collections. The fourth and last phase will demonstrate what is possible when audio collections are open and accessible. The product itself will be made freely available for reuse, thus completing the lifecycle of creativity.

In addition to being a creative tool and resource for the music community, I can see Citizen DJ compelling for use in secondary school classrooms. Sample-based music production can provide both an outlet for self expression and a way to actively engage students with history and culture. This process can also cultivate the activity of primary research and reinforce students’ relationships with libraries that often get lost after graduation.

I can envision a natural next step for this project is working with communities, classrooms, and practitioners to incorporate public audio collections into their activities, curriculum, or practice.

1.1.5 Library Staff Time

The majority of library staff time is needed in phases one and two. During phase one, I would like to leverage the expertise of staff to identify collections of audio that can be good candidates for this project. I would like to identify collections that:

1. Is in the public domain or available under a permissive license. In some cases due diligence may be required to declare a collection public domain.

2. Has cultural significance, is relevant, and may resonate with the general public.

3. Is underutilized. This may be due to a variety of factors including a lack of metadata, is in an inaccessible format, or a lack of promotion.

There may be identified collections that only exist in a physical format. If it is an option, a request for digitization may be made to make those materials available in a digital format and perhaps made available online.

During phase two, I will need help from staff to access the assets for each of the identified collections in phase one. If there is already a process or service with documentation for this, then staff time would be minimal for this step. However, if assets need to be manually moved to an accessible network, server, or drive, I will need staff time to facilitate this.

I will need minimal staff time for the remaining phases which may include providing feedback for the design and information architecture of the interface, organizing public events, and promotion of the resulting products.

1.2 Technical Approach

Note: Based on feedback from the concept paper, I reassessed and revised my estimates for the project schedule to reflect technical and creative uncertainties present in each phase.

1.2.1 Project Schedule

There are four primary milestones that map directly to the four phases of work in this project. A progress review and demonstration will occur at the end of each phase of work. A longer talk, retrospective, and listening event will be given at the end of the project that will be open to the public. I will coordinate with the Library to promote the products of this project to the general public throughout the process.

1. Phase One: Research and discovery

a. Duration: 13 weeks (3 months)

b. Completion time: 3 months from start date

c. Deliverables:

i. A public static website that contains:

1. An introduction to this project, its goals, and its approximate schedule

2. A list of resources related to learning about copyright, fair use, public domain, and other topics related to usage rights

3. A list of public audio and video collections, their descriptions, how they can be accessed, how they can be used, and sample audio clips

ii. A mechanism for receiving feedback or suggestions from peers, Library staff, and the general public

iii. A talk and demonstration that will introduce myself, this project, and highlight some of my findings thus far

d. Quality requirements:

i. This website should be immediately accessible and useful to the general public , particularly creators, educators, and researchers who use audio materials in their work.

ii. The navigation, organization, and writing should be clear and use plain language.

iii. The technology used should accommodate easy maintenance and content management.

2. Phase Two: Access and processing

a. Duration: 13 weeks (3 months)

b. Completion time: 6 months from start date

c. Deliverables

i. An open source code repository containing computer scripts for accessing, analyzing, and processing large audio collections

ii. Documentation for using code repository

iii. Sound sample packs generated from computer scripts and collections identified in Phase One

iv. A mechanism for receiving, tracking, and responding to technical issues

v. A talk and demonstration of how one can take an audio collection as input and create music using the resulting sample packs and free music production software

d. Quality requirements:

i. An experienced programmer should be able to use the computer scripts in the code repository using just the documentation provided

ii. An experienced programmer should be able to generate new sound sample packs using different audio collections

iii. The sound sample packs should conform to industry standards and be able to be imported and used in mainstream music production software such as GarageBand

3. Phase Three: Music creation tools

a. Duration: 13 weeks (3 months)

b. Completion time: 9 months from start date

c. Deliverables

i. A simple interface that:

1. Allows users to easily visualize and explore large audio collections

2. Allows users to easily generate, edit, and export music sequences using samples from public audio collections

3. Is designed for a broad audience, including those unfamiliar with music production

4. Is built using open source Web technologies

5. Can be used on a mobile, tablet, or desktop device

ii. Instructions for use

iii. A mechanism for receiving usability feedback

iv. A talk and demonstration of generating a song using this interface and public audio collections. This will be followed by a hands-on creation session where the audience is invited to create music.

d. Quality requirements:

i. A user with little to no experience with music production can create a song (good or bad) using this interface with minimal time investment or instruction. For example, have an option to generate a random beat using specific collections. See Appendix B for mock-ups.

ii. A musician can discover new sounds and export them for use in their own projects

iii. A user can quickly find the original source of any sample, understand how it can be used, and cite if necessary

4. Phase Four: Music production

a. Duration: 13 weeks (3 months)

b. Completion time: 12 months from start date

c. Deliverables

i. 6 to 10 songs produced using the interface and audio collections from the previous phases and released under a CC0 license

ii. A static web page that describes the collections used for each song and links back to the original source

iii. A mechanism for playing and downloading the songs

iv. A listening event and accompanying talk

d. Quality requirements:

i. Songs should entirely use source material in the public domain or available under a permissive license

ii. Songs should meet my personal quality standards as a creative professional from an aesthetic and production point of view

iii. Songs should thoughtfully highlight and reference the source collections

1.2.2 Testing Plan

Usability and accessibility are important features in all public-facing products I create.

Much of the user testing and technical quality assurance testing will be focused in phase three, during the design and development of the music creation tools. Because the interface will be web-based, I can continuously iterate based on new feedback.

I plan to test the interface on all major browsers, operating systems, and devices, including assistive technologies such as screen readers. I intend for this app to be WCAG 2.1 AA compliant. I also plan to have usability tests with a diverse group of potential users of various ages, backgrounds, and relevant professions such as amateur musicians, musicologists, educators, and students.

1.3 Capabilities and Experience

Given my computer science, music production, interaction design, and library experience, I believe I am capable to successfully execute the proposed project. I have over two decades of computer science and design experience and have worked in cultural institutions for the past seven years. I have a record of executing professional and personal projects of various scale including a digital collections platform, a permanent museum exhibit, a full-length music album, and a number of grant-funded projects of all sizes. I would like to highlight four past projects that demonstrate my ability to complete each of the four phases of this project.

https://www.w3.org/TR/WCAG21/

While working at the New York Public Library, I designed and developed the Digital Collections platform which gives public access to over 800 thousand digital items. I also was part of the Library’s public domain release of over 180 thousand items in 2016. For that release, I generated tools for visualizing those items as well as an interactive experience that visualized trips one could take using The Green Book collection, which was part of the public domain release. All of these platforms, tools, and experiences were designed and developed in collaboration with the Library’s curators and librarians.

This work demonstrates my commitment to the public domain and libraries and my ability to collaborate with library staff (phase one).

Around the same time, I executed a personal project called Data-Driven DJ, which is a 10 song album produced entirely from public datasets and open source software that I wrote. It has been featured multiple times on public radio and will be exhibited in the Museum of the City of New York later this year. This project demonstrates my ability to compose and produce music (phase four), my ability to execute large-scale part-time projects, and my commitment to open source software and public documentation.

Between 2017 and 2018, I designed and developed a permanent exhibit about climate change at the American Museum of Natural History (where I am currently employed).

This exhibit is composed of a 36 foot wide media wall that displays climate sceince visualizations and nine interactive stations that allow for visitors to interact with climate data. The interactive applications are built using open source web technology that runs on the Web, on physical displays, and mobile devices. I will use similar technologies for phase three of this project. The exhibit is now the most visited section of the Museum by school groups. This project demonstrates my ability to build robust, public-facing interfaces and applications (phase three). I would also like to note that I am on the Museum’s accessibility and digital committees, so I am committed to making accessible and inclusive products.

Lastly I would like to highlight a recent personal project called Moving Archives, which is a series of films that visualize thousands of hours of footage from public moving image archives and collections. This project required me to design a workflow and write open source code for accessing, analyzing, processing, and visualizing thousands of hours of media. Many of the tools I created for this project can be directly applied to accessing, analyzing, and processing audio materials. This project demonstrates my ability to work with large collections of media (phase two.)

https://digitalcollections.nypl.org/ https://digitalcollections.nypl.org/ https://www.nypl.org/research/collections/digital-collections/public-domain http://publicdomain.nypl.org/pd-visualization/ http://publicdomain.nypl.org/greenbook-map/ http://publicdomain.nypl.org/greenbook-map/ https://digitalcollections.nypl.org/collections/the-green-book https://datadrivendj.com/ http://brianfoo.com/work/amnh/climate_wall.html http://brianfoo.com/work/amnh/climate_wall.html http://movingarchives.brianfoo.com/films/

Volume 2: Price Proposal Note: Based on feedback from the concept paper, I reassessed and revised my estimates for the price proposal to reflect technical and creative uncertainties present in each phase.

The following table outlines the schedule and price for each milestone, including costs of materials and travel requirements. I plan to primarily work remotely but will be on-site for about three days per month on average for the twelve month residency.

This proposal assumes I will be working part-time at a rate of about 60 hours per month.

Depending on the start date, I may be able to increase this rate of work. In any case, I ensure I will have completely dedicated time (720 hours) to this project and deliver on time as outlined here and in the Project Schedule.

Phase One Phase Two

Phase

Three

Phase

Four Total

LABOR

Duration (months) 3 3 3 3 12 months

Hours worked per month 60 60 60 60

Total hours worked 180 180 180 180 720 hours

Work rate (per hour) $50 $50 $50 $50

Work rate (per month) $3,000 $3,000 $3,000 $3,000

Total amount (labor) $9,000 $9,000 $9,000 $9,000 $36,000

TRAVEL & MATERIALS

Travel costs

Trips per month 1 1 1 1

Travel cost per trip $100 $100 $100 $100

Lodging cost per trip $200 $200 $200 $200

Total amount (travel) $900 $900 $900 $900 $3,600

Materials costs

Hardware $2,000 $0 $1,000 $0

Website/asset storage and hosting $0 $100 $100 $100

Total amount (materials) $2,000 $100 $1,100 $100 $3,300

Total amount (travel and materials) $2,900 $1,000 $2,000 $1,000 $6,900

Total amount $11,900 $10,000 $11,000 $10,000 $42,900

In summary, the total cost of the project is $42,900 inclusive of labor, travel, and materials completed over a 12 month residency. Since this uses a part-time rate, the estimate is equivalent to about 4 months of full time work.

Equipment costs include (1) hardware for storage and transfer of large collections of media, and (2) tablets for testing and demo-ing the music creation interface. I have also included the cost of (3) storing and hosting the digital assets and interface on the Web during this period. The travel costs assume round trip train tickets from New York City and staying for an average of three days per trip.

Volume 3: Supplemental Information

3.1 SAM Registration

Registered as: Foo, Brian

DUNS: 117158839

CAGE: 8D8D5

3.2 Intellectual Property

All technology I will use for this project will be open-source under a permissive license such as Apache, BSD, or MIT License.

All source material will be in the public domain or created under a permissive license such as a Creative Commons license. Attribution will be given for all used material and linked to the original source if possible. If necessary and possible, I will request library staff to help vet materials that have unclear or ambiguous usage rights.

Appendix A: Visual references The following are references to existing projects that will inform my design for the interface in Phase Three.

Bird Sounds Thousands of bird sounds visualized using machine learning https://experiments.withgoogle.com/bird-sounds By Manny Tan & Kyle McDonald https://experiments.withgoogle.com/bird-sounds

This open source interface and code base demonstrates how a large audio collection can be analyzed, organized, visualized, and interacted with in an intuitive and efficient way.

The Infinite Drum Machine Thousands of everyday sounds, organized using machine learning https://experiments.withgoogle.com/drum-machine By Manny Tan & Kyle McDonald https://experiments.withgoogle.com/drum-machine

This open source interface and code base demonstrates how a large audio collection can analyzed and visualized based on its music-making potential and used to create music in a very simple way.

Tones.fm https://www.tones.fm/ A simple, open, collaborative music-making platform

A very simple, intuitive, and responsive web-based interface for creating musical patterns and sequences. It is published under a CC license.

Appendix B: Mock-ups These are mock-ups of three example screens for the interface in phase three. These are meant to convey basic functionality rather than imply the final design.

https://www.tones.fm/

This is an example of a start menu. The goal would be to reduce friction for any user, regardless of music production background. The first option will be to immediately generate a drum look using a random sampling of public audio collections. In this way, a first time user can hear unique music within seconds.

This is an example of of five random samples that make up a random drum pattern. Each row represents a specific audio sample, and each column is 1/16th of a measure. The interface will be as minimal as possible to begin with. The source of each audio sample and its containing collection are featured prominently.

BAA - 030ADV22R0041

https://www.dropbox.com/s/ozozo6sfw44df87/mockups_01_menu.png?dl=0 https://www.dropbox.com/s/yz4kmv88neu1vme/mockups_02_loop.png?dl=0

When you are exploring an audio collection, a matrix of audio clips are visualized based on sonic similarity. As you drag your mouse/finger, the clips play immediately. The linked source of each clip is shown with a timestamp as you drag or click. On the left, you can filter/highlight the audio clips using a combination of human generated metadata (where and when available) and the clips’ sonic properties generated in phase two.

Appendix C: Referenced projects These are personal and professional projects I referenced in this proposal. You can find my entire portfolio at http://brianfoo.com/

● NYPL Digital Collections: https://digitalcollections.nypl.org/

● NYPL Public Domain Release: http://publicdomain.nypl.org/

● Visualizing the Public Domain Release:

http://publicdomain.nypl.org/pd-visualization/

● Navigating the Green Book: http://publicdomain.nypl.org/greenbook-map/

● Data-Driven DJ: https://datadrivendj.com/

● AMNH Climate Change exhibit: http://brianfoo.com/work/amnh/climate_wall.html

● Moving Archives: http://movingarchives.brianfoo.com/films/

● Code repository for accessing, analyzing, and processing large media collections: https://github.com/beefoo/media-tools

BAA - 030ADV22R0041

https://www.dropbox.com/s/4bavh3y5uwicpuq/mockups_03_explore.png?dl=0 http://brianfoo.com/ https://digitalcollections.nypl.org/ http://publicdomain.nypl.org/pd-visualization/ http://publicdomain.nypl.org/pd-visualization/ http://publicdomain.nypl.org/greenbook-map/ https://datadrivendj.com/ http://brianfoo.com/work/amnh/climate_wall.html http://movingarchives.brianfoo.com/films/ https://github.com/beefoo/media-tools