clippy

Clippy started off as a series of experiments with Microsoft Office PowerPoint software that the University has licensed. Having only used a version of PowerPoint that had a digital assistant, known as clippy, I was delighted to find that it had been upgraded with a machine learning makeover, and thus providing a serendipitous encounter with one of the themes I am exploring – banal AI.

It is not lost on me (and kind of the point) that my practice-led research begins by exploring the ubiquitous PowerPoint (and images of cats). One of the things I am examining how the Alt-Text function of the AI works, where it automatically generates image descriptions with high or low levels of confidence. As the process Microsoft uses is not open source, to do this, I am uploading images that have various effects incrementally added to them such as noise and blur.

A second element responds to the information sent to and from Microsoft’s servers, using packet sniffer technology to attempt an examination of what is sent back and forth. Using the service, Microsoft write that the service improves the more people use it (part of another theme on free work). The description by the company, as well as interaction with their staff via their website refer you back to the 15,000 word user agreement with any enquiry as to the information retained.

The use of PowerPoint in this context reminded me to some of the most infamous PowerPoint slides – both for their poor design quality as well as the revelation of a vast embedded telecommunications spying program – that were released in 2013 by Edward Snowdon – at the time working as a United States National Security Agency contractor. Known as PRISM, the classified slides were leaked to journalists at The Washington Post and The Guardian.

I am recreating some of the PRISM slides (pictured later) and using a second Microsoft Assistant AI that suggests design ideas, in a video that examines Microsoft’s part in the PRISM program, and the PowerPoint tool itself.

---

Catfish

Microsoft PowerPoint Office Assistant Alt-Text automaticallly generated descriptors for an image.
Image 1: A close up of a cat. Image 2: A group of fish in the water (Image 1 with 500% monochromatic noise added).

---

Microsoft PowerPoint Office Assistant Alt-Text automaticallly generated descriptors for Image 1, overlaid with additional 100% noise corresponding to each line.

---

A View of the City From a Window

A View of the City From a Windows uses PowerPoint’s AI automatic description feature to describe an image of a cat as it is modified from a pixellation effect in increments of 10, beginng at 1000x.

---

A Pair of Footprints in the Snow

A Pair of Footprints in the Snow is an animation using PowerPoint’s AI automatic description feature to describe an image of a cat as it is modified by introducing monochromatic noise as an image modifier. Each iteration of the effect is numbered. The process was run through twice, giving different outcomes in the latter half as noise makes it increasingly difficult for both human and machine to ‘interpret’ the image.

---

A White Wall With A Hole In It

A White Wall With A Hole In It is an animation using similar processes of using PowerPoint to automatically describe the image presented as noise experiments above, but using gaussian blur as the image modifier. Each iteration of the blur effect is numbered. The process was run through twice, giving different outcomes in the latter half where more blur occurs.

The Alt-Text is spoken by two LOVO AI commercial AI voiceover software ‘voice skins’ – one given the name Jess Thomas, described as female, young adult, explainer, news, classy, distant, and the other, given the name Ed Bolinger, described as male, middle-aged, games, narration, low-pitched, informative.

As the results differed on each run through of the PowerPoint, Jess and Ed end up having a low level argument. When attempting to recreate the discrepancies a third and fourth time using the same steps, the result reverted to those voiced by Ed.

---

A different Ed

Microsoft have been an early signatory of the PRISM program, as can be seen by the classified slide above. How much did Clippy know?

Further banal AI Powerpoint experiments can be found at all at see.