# p5.touchgui
***A Mouse & Multi-Touch GUI Library for p5.js***

*by [Carlos "L05" Garcia](https://github.com/L05) ([@L05_](https://twitter.com/L05_) | [L05.is](https://L05.is))*

-----

[*p5.js*](https://p5js.org) is a wonderfully accessible way for students and creative minds alike to learn computer programming and create interactive art and experiences. 

[*p5.touchgui*](https://github.com/L05/p5.touchgui) is intended to extend *p5.js* and make it easy to add canvas-based buttons, sliders, and other GUI (graphical user interface) objects to *p5.js* sketches, enabling users to focus on quickly iterating ideas with easily created GUI objects that work with both mouse and multi-touch input.

## > Process

### How did this start?

As an artist and immersive experience designer, I have been researching and developing multi-person interactive experiences for 8 years, experimenting with all kinds of input sensors and mechanisms to create some [pretty cool stuff](https://L05.is).

A couple of years ago, as I was learning node.js and WebSockets, I started to experiment with *p5.js* as a way of creating simple, mobile touch-screen interfaces that would enable audience members to control games and installations with their phone. In this process, I created a small toolkit of p5-based GUI elements and a node.js server template for multi-person interactive games. Check out an example of one such experience [here](https://i.vimeocdn.com/video/706438411.webp?mw=1100&mh=619&q=70).

*Sidenote: my next *p5.js* project is to package and release a multi-player node.js server and client template along with a tutorial!*

[![Space Dander Example Video](https://i.vimeocdn.com/video/706438411.webp?mw=1100&mh=619&q=70)](https://vimeo.com/274410221)

During my graduate studies at UCLA's Design Media Arts department, I had the wonderful opportunity to work as a teaching assistant for Lauren McCarthy and Casey Reas, who both taught *p5.js* in their classes. I noticed that students often had difficulty with GUI elements in their projects; building a simple button in *p5.js* is an excellent learning exercise if the primary focus of an assignment, but it can be a blocker if the assignment goals are to express some larger creative vision.

Having already created my own personal toolkit of project-specific GUI elements, I realized that with some work, I could adapt it into a more generalized tool that may be useful to students, artists, and educators. And so the process began...

### Research
My first step in working on *p5.touchgui* was to research GUI design methodologies and examples. I looked at the current state of UI within *p5.js*. Users must either use DOM elements via *p5.dom* or create their own canvas-based GUI objects with *p5.js*. 

I performed a number of preliminary tests with DOM elements and multi-touch screens, and it seemed DOM elements were not well suited to multi-touch environments: the performance was often slow and touch quality inconsistent. DOM elements seemed sufficient for simple interactions but not for more dynamic experiences such as touch-screen controllers.

While *p5.dom* gives users access to DOM elements, it still feels a bit clunky to use. In my classroom experiences, *p5.dom* often seemed a pain point for many students trying to use DOM elements in their sketches.

As my focus shifted towards designing canvas-based GUI tools, I became especially interested in learning about [Immediate-Mode](https://en.wikipedia.org/wiki/Immediate_Mode_GUI) and [Retained-Mode](https://en.wikipedia.org/wiki/Retained_mode) approaches to Graphical User Interfaces. I first learned of these categories from reading [Doeke Wartena](http://www.doekewartena.nl/)'s description of his Processing GUI library, and [this video](https://www.youtube.com/watch?v=Z1qyvQsjK5Y) by Casey Muratori on Immediate-Mode GUIs was particularly illuminating. I decided to dive into creating *p5.touchgui* as an Immediate-Mode GUI.

### Immediate-Mode p5.touchgui

I first implemented a version of *p5.touchgui* using an Immediate-Mode approach, and in many ways, it felt simple and easy-to-use. The GUI Objects were not stored anywhere; they were fed a variable when drawn, and they modified that variable by reference based on user input.

Here's what the code looked like in the Immediate-Mode version:

```javascript
let x;

function setup() {
    createCanvas(400, 400);
    createGui();

    x = {val: 0.5};
}

function draw() {
    background(220);
    startGui();

    if (button("Button 1", 50, 50)) {
        print("Button 1 pressed.");
    }

    endGui();
}
```
[See it in action here!](https://editor.p5js.org/L05/sketches/7yMYNgJyM)

**Pros**
* Accessing GUI object states in the `draw()` loop seemed like an exciting idea for teaching beginners.
* No GUI object states had to be stored, retained, or synchronized.
* It was entirely data driven.

**Cons**
* Required the creation of JavaScript objects in order for pass-by-reference to work. This is an advanced concept to have to introduce to beginners.
* Would be difficult to offer custom styling.
* Multi-touch input would not work well with a strictly Immediate-Mode GUI. This was obviously kind of a deal breaker.

### p5.touchgui as Retained-Mode GUI (that mimicks IMGUI)

While I really liked some aspects of Immediate Mode GUI (IMGUI), I realized a purely IMGUI approach would not offer the functionality necessary to deliver the primary goals of *p5.touchgui*:

* Easy to use
* Easy to teach
* Multi-touch support
* Performant

I began to ask myself, how can I adapt some features of IMGUI (such as having access to GUI object states) into a more flexible and comprehensive GUI system?

I ended up creating a Retained-Mode GUI that mimicked some behaviors of an IMGUI. For instance, users can access GuiObject member variables such as `button.isPressed`, which returns a `Boolean` value describing if the button is in fact pressed.

By creating callback event handlers specifically for the library, I was also able to account for both mouse and multi-touch input. These custom handlers not only enable expected mouse behavior such as hover states and multi-touch control of GUI objects, but they don't interfere at all with the built-in *p5.js* event callbacks (e.g. *mousePressed()*, *touchStarted()*, etc.). Building the input processing around event-based handlers also made it significantly more performant than calculating inputs during every single frame.

By switching to a Retained-Mode approach, I was able to begin implementing a style system for GuiObjects, building a small set of presets, and providing user-accessible functions for modifying object styles.

Here's what the code looks like now:

```javascript
let gui;
let b;

function setup() {
  createCanvas(400, 400);
  
  gui = createGui();
  b = createButton("Button", 50, 50);
}

function draw() {
  background(220);
  drawGui();

  if(b.isPressed) {
    print(b.label + " is pressed.");
  }
}
```
[See it in action here!](https://editor.p5js.org/L05/sketches/YjR7CxKg-).

## > Next Steps

There are a few areas in which I look forward to further developing *p5.touchgui*.

### Accessibility

I learned at the *p5.js Contributors Conference 2019* that *p5.js*, and by extension *p5.touchgui*, is very inaccessible to screen readers. On further reflection, this is not surprising given that *p5.js* is essentially a Canvas object, a buffer of pixels. This pixel buffer and the elements within it inherently carry none of the data  screen readers are able to access in DOM elements.

I am interested in working with the *p5.js* community to find ways of making both *p5.touchgui* and *p5.js* more accessible, whether through providing off-screen shadow DOM structures or through solutions such as the Accessibility Object Model (AOM).

### Style

Style is one area where *p5.touchgui* needs more work; as it turns out, it's hard to build a style system that balances customizability with simplicity! The presets are a good start, and it is possible to stylize individual objects, but the number of lines of code it would take to build out a larger interface still seem extraneous.

Some features I think would help:
* *Group styling* - i.e. determining the style of an entire object class. This is a high priority feature to be added to the library.
* *Simplified style properties* - the properties currently get pretty granular, which on one hand gives a lot of color control and on the other means that a non-trivial number of properties need to be set. I am very interested in the simplery way Material UI has primary and secondary color choices, for instance.
* More presets!

### Additional GuiObjects and Features

There are a number of additional features on which I would like to work:

* More GuiObjects!
    * Radio Button
    * Color Picker
    * Button Bank
    * Slider Bank
    * Range Slider
    * Text Field
* Tags (grouping GuiObjects by tag)
* Selective Input Lock (right now all GuiObjects default to input lock)
* Migrate reference documentation to Docsify model.
* More examples!

You can also check out my [development notes](https://github.com/L05/p5.touchgui/blob/main/docs/NOTES.md) if you'd like to read a more informal summary of process and interests.

## > Support

Please use *p5.touchgui* and give me feedback!
* If you **use it in a project**, what works and doesn't work?
* If you **teach it in a class**, what works and doesn't work?

Any questions pertaining to this project may be communicated via Issues on the [p5.touchgui GitHub repository](https://github.com/L05/p5.touchgui). Simply create a new Issue and either assign or tag me in the conversation with [@L05](https://github.com/L05). For anything else, don't hesitate to get in touch at carlosga@gmail.com!

## > Acknowledgements

I'd like to give an extra special thanks to my GSOC mentor Yining Shi for her support, guidance, understanding, and encouragement throughout the project. I'd also like to thank Lauren McCarthy for talking through some of the pedagogical applications of *p5.touchgui* with me during its development. Finally, thank you to all of the contributors and community members of *p5.js* who make it such a fantastic resource and community!
