The Photon Experiment

I while ago shared a notebook that showed how I was starting to use JAX for image processing. I finally completed enough work on my image processing library (Photon) to make some images that are interesting enough to share. Here are a few samples:

Artwork by author

These images above are heavily downscaled so you can't really see how they were put together. These crops give a better idea of their construction.

When I started this project I wanted to learn JAX, but I also wanted to test a hypothesis around how computer code can be structured to allow something that I call "Function Synthesis" - where code can be rapidly composed/assembled by simply collecting together a group of functions and then allowing the software to piece together the individual functions to achieve a goal. Photon is not yet at the stage where all of this happens by magic, but the code is structured to allow for much more automation later.

This notebook is an example of how I put a collection of renderer functions together to produce the sample images above. It explains how Photon mimics some of the creative process that I use for conventional painting.

#%% md ## Choose a collection of "stipple marks" Since Photon aims to mimic the painting process, it has to understand some basic technique. In its current primative form it understands stippling quite well. It is also pretty good at flat washes, optical blending, masking and layering. If developed further it could probably pull off other techniques like hatching, splatter, scumbling and sgraffito. Stippling is a somewhat expressive technique as the individual marks are visible as "brush strokes" in the final image. Like your handwriting, everybody's stipples are different. Photon uses examples of your own stipples to help you express what you want to express. This is a preview of a sample of some of the stipple marks that I made by hand for this example.

Does this use Dall-e, MidJourney or Stable Diffusion? No. Photon uses some of the same underlying technology as these deep learning image generators, but it’s approach is totally different. Pretrained models like Dall-e accept text prompts and apply style by drawing inspiration from the millions of images they looked at in training. They understand what a dragon looks like, what a fly swatter looks like, and are really good at figuring out how a dragon wielding a fly swatter may have been painted by Monet. Photon is not pretrained at all so it doesn’t know a dragon or a fly swatter from its own armpits and has never heard of Monet. Photon aims to mimic the personal style of each user. It is still in its infancy, but in the fullness of time, Photon could learn people’s personal artistic style by understanding patterns in the way that each users’ preferred renderings deviate from their references. With each round of rendering, Photon intentionally makes lots of deviations. Saying that Photon colours outside of the lines is an understatement. As part of its deliberate clumsiness it misreads reference images or even blatantly ignores them, it chooses colours unwisely and it miss-registers marks. You will hate the results of some of these blunders, but you may also like some of them. When you offer feedback on its clumsy accident-ridden renderings, it aims to start to understand patterns in how your visual style deviates from the reference. Once it understands these patterns it can start making more decisions that cause more happy accidents and less unhappy accidents. There is no language model in Photon. If you really want to produce a dragon wielding a fly swatter in the style of Monet, you would have to draw a reference sketch of the scene and you would have to influence Photon’s rendering choices (such as palette, choice of marks, handling of contrast) to get it to produce “Monet-like accidents”. This would encourage Photon to make more Monet-like accidents for you in the future. Photon is really not intended to mimic Monet or anybody else though. Its goal is to allow people to express their individuality. Other than Photon’s ham-fisted approach to painting there is something else that is different about it. It produces giant images. The pretrained models are limited by the size of publicly visible images and also impacted by the computational cost of training. Photon’s process naturally produces very large images because it is more expressive when you use a large “brush” and there is no real limit on brush size. The only thing that limits image size in Photon is available memory. The little thumbnails visible in this notebook don’t reveal the scale well, but these images are huge. They could easily produce a mural if transferred to a wall. Do computers and AI have the potential to ruin real art? This is a question that has been weighing on my mind while I have been working on the Photon experiment and seeing how much attention Dall-e and MidJourney are getting. One part of me wants to think that anything that allows people to express themselves creatively is a good thing. There is another part of me that thinks that anything that makes something that should be treasured for its rarity commonplace reduces its value. I would argue that over the course of civilization, the most valuable artefacts the human race has produced are the arts - visual arts, creative writing and music. Many fellow technologists would disagree with this statement, but I see technology as largely ephemeral. Each bit of past progress is only relevant today as inspiration for later technologies. The technologies we hold dear today will be soon forgotten when they too are replaced. Art is not like this. Significant art accomplishments have the potential to treasured for generations. The doom and gloom scenario is that computers and AI can churn out so many “good enough” images so quickly and so cheaply that the average consumer/business will not require the services of human artists to decorate their houses, virtual presences and entertainment. If this happens, it implies a lot less demand for human artists. A lot less demand for human artists to produce everyday artwork also implies less capacity to produce new works of significance. This doom and gloom scenario is made worse by the fact that once people are surrounded by mass produced visual artefacts, they will become numb to visuals - whether created by human or computer. I don’t know whether this doom and gloom scenario will play out, but one way to test it is by looking at advances in music over the last few decades. I think it is fair to say that the most significant musical artworks were produced centuries ago and that these artworks are less appreciated by the general population than they were in the past. It is also fair to say that music is much more accessible than it was before. What is more important to humanity? Easy accessibility of lots of music or the capacity to produce works of great historical significance? Has the level of accessibility to music made us numb to it? If it has, it hasn’t stopped us consuming it, but do we get as much out of the experience as we used to when it required more effort to obtain and consume? If computers and AI make it even easier to produce and publish music would that improve the experience or degrade it? Photography is another test. As photography became more and more accessible to more and more people, the quality of images rose and so did the quantity. Surely quality and quantity is a good thing? Before we were surrounded by quality images, quality images stood out and made an impression. This was important in the commercial world because it allowed companies to make a strong statement about their products. It is even more important in the realm of fine art photography. A truly successful fine art image allows you as the viewer to experience something that you have never felt before. Again speaking personally of my own experience, I find myself truly moved by an image less often than in the past and I attribute this to numbing by an overload of high quality images.