Today

• warm-up: Exquisite Corpse
• review grid-based works
• camera
• sound
• BONUS: translation

Warmup: Exquisite Corpse

Exquisite corpse, also known as exquisite cadaver (from the original French term cadavre exquis), is a method by which a collection of words or images is collectively assembled. Each collaborator adds to a composition in sequence, either by following a rule (e.g. “The adjective noun adverb verb the adjective noun.” as in “The green duck sweetly sang the dreadful dirge.”) or by being allowed to see only the end of what the previous person contributed. –source

The drawing game was invented by the surrealists a century ago.

Rules:

1. Our sketches will be 600 pixels high by 400 pixels wide.
2. Create a function with the name head(), body() or legs().
3. Our sketch’s canvas will be divided into thirds. The head section should be drawn only in the top rectangle of the canvas 400pixels wide by 200pixels tall, e.g. (0,0,400,200). The body section is in the middle, e.g. (0,200,400,200). The legs in the bottom rectangle, e.g. (0,400,400,200).
4. Be sure to place your initials before any global variables you add, and place those at the top. Load any external images in the preload, and set any randomness in the setup.
5. We will mix these together, so the initials at the beginning of our variables will ensure that we don’t overwrite each other’s variable names.

Sound

So far, we have only been using programming to create visual artworks. But p5.js also features a sound library, called p5.sound that allows us to work with sound in our sketches. Sound adds an extra dimension, potentially helping to place us more fully in an immersive experience. It can also serve just as accompaniment, or as a fundamental part of our work.

Working with sound is quite similar to the process of working with imported images.

1. Load the sound into the editor
2. Create a global variable to hold the sound file.
3. Load the sound file in your preload function (or use a callback function that runs only after your sound loads).
4. NEW: begin playback of that particular sound file somewhere using .play().

let mySound;

function preload() {
  mySound = loadSound('doorbell.mp3');
}

function setup() {
  mySound.play();
}

We don’t play mySound.play() in our draw() function because we only want the sound to play once. If we place it in draw, it will trigger a new playing of the sound every time the draw() function is called.

A few other sound-specific methods:

• isLoaded() will return the boolean true if the sound file has loaded already.
• isPlaying() will return true if the sound file is currently playing. This is really useful for toggling sound on and off. Use an if statement to detect if a sound file is playing, and then choose to play it if it’s currently false.
• setVolume() between 0.0 (silence) and 1.0 (full volume)
• .play() starts playback of a specific sound file
• .stop() does what you think it does
• .loop() begins looping playback of a soundfile
• .pause() pauses the file. Start it again with .play()

playMode

And lastly, an important one:

• playMode(), which has three potential arguments, restart or sustain or untilDone. By default, the playMode is sustain. In sustain mode, each time play is called, the program will play the full sound file from start to finish, which means any previously-started playing of the same sound file will be heard simultaneously until it has ended playing. In restart mode, if the sound file has already been playing and you call play on that sound again, it will stop the previous and start playing the soundfile from the beginning again. Lastly, if playMode is untilDone and you call play on a soundfile, it will only start playback if it’s not currently playing.

• These and additional sound file commands/methods can be found in the p5Sound reference.

//example of toggling a song on or off when you press the mouse
var song;

function preload(){
  song = loadSound('freebird.mp3');
}

function setup() {
  createCanvas(720, 200);
  text("click to play",50,50);
}

function mousePressed() {
  if (song.isPlaying()) { //remember, this is the same as writing if (song.isPlaying() == true){ };
    // .isPlaying() returns a boolean true in this instance
    song.stop(); //so stop playing it
  } else { //otherwise, it wasn't playing
    song.play(); //so start playing it!
  }
}

Note:

Web browsers have just recently begun to be set to ignore autoplaying sound and video on the web, even if we’ve specified that we want a sound file to start playing in the setup(). They have done this to try to curb on the amount of auto-playing sound that plagued the internet, particularly advertisements. To play audio going forward, you’ll have to have a user click first.

To get around this, many programmers are now adding a button or instructions to ‘click to play’ and then have your [[audio]] start playing once the user has clicked in the window.

Simple example here

Video

Loading video works almost entirely the same as loading audio. Both are called ‘media elements’ in a web browser.

1. Load the file in your code editor
2. Create a global variable to hold the media file
3. Preload it in that variable in the preload (or if you load it somewhere else, you’ll need to add a callback function to run after it successfully loads)
4. You can play the video with play

var vid;
function preload() {
  vid = createVideo('toothy.mp4');
}

// This function is called when the video loads
function setup() {
  vid.play();
}

example code

• Note: To position the video element on the page, you specify position with .position(x,y);.
• More info on createVideo options.

Image capture

Dan Rozin

Our laptops have built-in webcams, and we can access them with p5.js. The video we get from the camera we call a capture.

example code

let capture;
function setup() {
  createCanvas(400, 400);
  capture = createCapture(VIDEO);
  capture.hide();
  imageMode(CENTER);
}
function draw() {
  background(50);
  image(capture, mouseX, mouseY, 160, 120);
}

Multidimensional arrays

Arrays can be multidimensional. So far we have talked about a 1-dimensional array.

Each element in the array has a single number that assigns where it is location. 0 is the first element in the array. 1 is the next element. And so on.

A chessboard is 2-dimensional. There is a X, Y position for every space on the chessboard. So we need to give two numbers to identify a space in this 2-dimensional array.

Pixel Array

Our pixel array is even more complex. For each individual pixel in our canvas sketch, there is a X and Y location.

We can grab and change a pixel individually with get() and set().

Additionally, p5js gives us a pixels array. It is very long, and it corresponds to all of our canvas pixels.

The pixel array is 1-dimensional but our canvas is 2-dimensional.

pixels = [_, _, _, _, _, _, _, .........]

How does this work? Each pixel has a number for its red, green and blue value as well as its alpha (transparency). So the very first pixel, the pixel located at 0,0 in our grid are the first four values in the pixel array.

pixels = [R, G, B, A, ..........]

The formula for going from a 2d grid to a one-dimensional array is (x + y * width) * 4.

Playing with pixels

example code

//getting started with p5.js
function setup() {
  createCanvas(320, 240);
  pixelDensity(1);
}

function draw() {
  background(51);

  loadPixels();
  for (var y = 0; y < height; y++) {
    for (var x = 0; x < width; x++) {
      var index = (x + y * width)*4;
      pixels[index+0] = x;
      pixels[index+1] = 0;
      pixels[index+2] = y;
      pixels[index+3] = 255;      
    }
  }
  updatePixels();

}

Homework

Review

• Watch The Pixel Array on The Coding Train on YouTube
• Create Capture to get a live image from the webcam
• Painting With Pixels

Create an experimental camera

Make your own experimental camera application or artwork. You can work with the pixel grid array, or add special features such as a frame, shapes on your face, or something else. The goal is to make an interactive artwork using the camera as a primary mode of interacting with the viewer. Consider the entire web browser as your canvas.

Requirements:

• for full points the sketch must be built to be shown fullscreen, e.g. make use of createCanvas(windowWidth, windowHeight);
• your work should be a fundamental shift or experimental re-use of a camera, not just a simple display of the camera’s video feed. What role does your work have?: surveillance, utility, playfulness, abstraction or something else?