Intense Development

There are periods of time during a project when I don't even want to sleep. Others around me get very annoyed. But when I come out the other end, something magical can be seen. This is partly because I, thankfully, work in the realm of computer graphics. And partly because I'm a visual person who can imagine a visual result that others can appreciate.

And it's all in the demo.

There is no sleight of hand in a demo. Not when people are to be impressed. But sometimes people just don't get the value in what you construct. This is where you have to educate them, to show them the value, to connect it to something they can understand. You have to make all that obsessive development time mean something.

You need to become tolerable again.

I have talked about where ideas come from. About the different frames of mind we can be in. About how to foster creativity in the first place. But, once you get the idea and reason out how it can be implemented, there is a new phase that needs to be explored. How does this process unfold, this intense development? How does the large feature or the complex technique get implemented? How can we, as mere humans, even manage something like this? What tools do we use to do the seemingly impossible? What parts of our brains do we have to use to accomplish our goals?

Organization

The best method to tackle a large project is to get organized. I do this by taking notes, drawing pictures, and building tools.

I have found that some of the best notes to take are these:

• new ideas or features that you would like to explore
• problems that need to be resolved
• places to look when updating to some new arrangement of the code

For most people, the note-taking process is a hassle. But you really need to start taking those notes to accomplish a project that is so big you can't keep it all in your head!

When drawing a picture, sometimes a flowchart is useful. Here we have the basic step in constructing a Laplacian pyramid. The objective is to decompose the step into smaller operations, a process known as top-down decomposition.

Here the basic step gets split into reduction, expansion, and difference substeps.

The reduction step is the process of converting an image into another image that is half the size in both width and height. And one which thus does not contain any of the highest-frequency information in the original image. The expansion step is the process of resizing the half-sized image back into full size. This image will be blurrier than the original by definition. The difference step is the process of determining the differences between the original full-sized image and the blurred full-sized image. These differences form the highest frequency detail in the image.

This step can be repeated to create a quarter-sized image and a half-sized detail image.

So not only is the image decomposed into various frequency bands, but the process of decomposing the image has also been decomposed into steps!

Rational Processes

Using your rational mind is partly deduction, and partly experience. For instance, when you implement a gradient operation, experience tells you that the center of a line has a zero gradient, and either side of the line has a non-zero gradient. As a practical demonstration of this, consider the Painter brush stroke. It is from an airbrush at high opacity with a 2 pixel diameter: a typical thin line.

If you compute the gradient using a Sobel technique, each 3x3 neighborhood of the image is convolved with two 3x3 kernels. There are variations on this theme, but usually the kernels will look something like this:

 1  2  1                   -1  0  1
 0  0  0      and       -2  0  2
-1 -2 -1                  -1  0  1

The first kernel is for computing gradients in the y direction (horizontally-oriented edges) and the second gradient is for computing gradients in the x direction (vertically-oriented edges).

Convolution means multiplying each element of the kernel with  corresponding pixel in the actual neighborhood in the image and forming a sum of the products.

You do that for both kernels, producing two sums, which you can imagine to be the x and y value of a vector field. The gradient is simply the magnitude of that vector.

The result of this is a gradient like you see here. Notice that the center of the line has an empty space in it, corresponding to a zero edge.

My rational mind already knows this through experience. So this means that if I want to use the gradient as a mask, and process the center pixels of the line, I will have to do something to fill in the center of the gradient. Like an annealing operation (a blur followed by an increase of the contrast or exposure of the gradient).

A rational mind mixed with the ability to visualize is probably the best way to get image processing operations done the quickest. But there are times when visualizing is not enough. We must see the intermediate results and check that they are being produced correctly and effectively. This brings us to the next technique: building tools.

Building Tools For Visualizing and Debugging

Any process in image processing, no matter what it is, will have intermediate results. There will be a blurred buffer, morphology applied to something, a gradient, a vector field, some representation that needs to be visualized. And we may need to verify that each step is being accomplished correctly, or verify that the step is even doing what we imagined it would, and is thus useful in the process of finding a solution.

So we need to construct a tool to see the intermediate results, to study them, to inspect them, and to debug their construction when your idea of what they should look like does not match what you get.

I have done this time and time again with large projects I have worked on, and it has enabled me to make much faster progress on a large project. And with a tool such as this, it becomes another thing: your demo environment. Not only can you see what's happening, but others can as well.

In order for a demo to come off smoothly, your implementation has to be fast as well. This means that you will need to implement selective update, and also you will need to make it go as fast as possible through optimization.

It doesn't matter what kind of project you are working on. You will always need to demo to justify your continued work. You will need to show progress. You will need to convince people that it can be done.

Tool construction (a testbed with demo capability) is your best tool to accomplish this!

Choosing the Best System to Build On

When constructing an image processing tool that involves steps, intermediate results, complex staging, or heavy computation, you need to choose a system to build it all on top of. For my purposes, I am considering a Macintosh as my system platform. But there are APIs and methodology that apply to any task.

Core Image is a good API for image processing, when your result is constructed one pixel at a time. It can allow you to utilize a GPU or a multi-core CPU to get the job done, and it can render the task of constructing a pass on your data into a simple thing. This is highly desirable when you have a lot of passes to construct. Core Image kernels are pretty easy to construct. You can reference any number of source images, but you may produce only one pixel in the destination image. This conceptually works pretty easy for blurs, color operations, compositing operations, and even transitions. You can build Core Image filters on top of your operations, and their parameters are entire images. And settings for your operations.

OpenGL is a good system for doing computation and presenting that computation inside a texture on screen. When this texture is transformed in 3D, as in "onto a 3D object" then this is the ideal API to accomplish the task. OpenGL may also be used for computing results on 2D flats that are presented using an orthographic projection. The computation can occur using almost any OpenGL operation or it can occur using a fragment program. This is conceptually the same as Core Image, so there is not much value in going the OpenCL route unless textures are going to be transformed in 3D.

OpenCL is a good system for doing arbitrary computation using the GPU and the CPU. You can support multiple output buffers as well as multiple input buffers. This means that come simulation operations are easier. Also, things like scatter and gather to and from planar color formats are much more natural. For instance, conversion of RGB to YCC where the Y is kept separate from the CbCr information can be supported very easily. One RGB image input, two images, one Y ands the other CbCr output.

Multi-core CPU computation is another good method to get things done fast. Here you can use Grand Central Dispatch to easily queue your computation on multiple CPUs. It has never been easier.

The Dangers of Obsession

You can get buried in a project. It can overcome you. This can have a very confusing effect. Unless you disentangle yourself from it for a while and take a step back, you run the risk of becoming irrevocably lost.

Back in my Caltech days, there were those people who were interested in Dungeons and Dragons (D&D). This sometimes resulted in people becoming obsessed with the rule systems and the immersive game-play.

And sometimes people just got lost. First they forgot to shower, neglecting their basic cleanliness. Then they showed the effects of malnutrition: the endless supply of Coke and little white powdered-sugar donuts. They started talking about fifth-level clerics and trolls. They always carried those little clear twelve- and twenty-sided dice around with them. And one day they didn't come to class. And never appeared again.

These were good, perhaps weak-willed people who were casualties of war. The war against obsession.

Yet I also saw people get obsessed in technical and scientific matters. These were called grad students. They would work on their thesis obsessively, disappearing into a dark cave until they came out with something hard and shiny like a diamond. I observed that obsession had its value, it seems.

Buried in Complexity

You can add more and more to a program over a period of many months. This is called add-on programming. And it can lead to another problem: complexity. A haphazard programmer can continue to kludge up a piece of code using branching and questionable data structures. This can lead to spaghetti code: twisty passages all alike.

The only solution to this problem is rethinking it: it must be rewritten. There is no other way if it is to be modified in the future. If you were adding more and more stuff to it, then this is a virtual certainty. At this point it is time to develop the right control structures and data structures to render the solution in the most effective and extensible way.

Immersive Programming

At some point you will need to debug what you have created and make it work. This requires total immersion. The better you have organized your code, the easier it will be to understand the processes it uses and thus to figure out which steps are correct and which are incorrect. This is the process of debugging.

It's like putting your head into the code and visiting codeland.

One thing is sure: you better have your head on straight when you debug a large project the first time. This will be when your organization and rethinking of control and data structures will pay off.

SOmetimes when debugging a project it becomes clear that there is a logic flaw in the code. This can be a small one, like an off-by-one error, or some statements that are out of order.

Or it can be a very large problem indeed. One with huge ramifications for the code.

My advice is to fix it before going any further, no matter how sweeping the implied changes are.

To Sum It All Up

Once you have been through fifty or so large projects, you begin to see patterns much more clearly. Perhaps you can profit from some of the patterns I have found, and some of the cautionary tales.

All I know is that I mostly had to learn these things the hard way.

Sigh.

My Early Painter Art

As Painter's creator, I was also its first user. You might remember the reason I created Painter: to satisfy my own desire to draw. The first benefit of this fact to Painter was that I was never really satisfied with the tool set.

And I had nobody to blame but myself in the early days before John Derry came to Fractal Design.

So I became a constant user. There was a demand for imagery in press kits, so I had to contribute what I could, and quick! As it happens, the first tools I successfully simulated were colored pencils. So in my early art, I shaded using Painter's colored pencils (mostly because they built up from lighter colors to darker, more saturated colors). I was already pretty good at shading, and I was pleased to see my style come right through.

This lonely-looking example was drawn in the earliest days of Painter. It exhibits my shading style about as good as any other image I can show.

I really needed the ability to clean up edges and to create sharper-looking items in the images I was drawing. Aside from creating chalk, charcoal, and felt pens, I created a little thing called the frisket tool. To an artist, a frisket is a mask that can help to guard some parts of the image from being touched. It is often used with airbrushes. Perhaps I should have called it a selection. That's what it really was. But unlike most selections, this one was anti-aliased, with a clean edge. This came from the benefits of having high-resolution information data for the brush stroke, which in turn came from the Wacom tablet. You could restrict your paint strokes to the inside of the frisket area, or to only the outside of the frisket area. It was convenient to be able to switch from one side to the other.

So I started using friskets in my work as well. This led to some images that often were used in screen shots in the early reviews of Painter. This image, done in red ochre using Painter's colored pencil, was drawn while looking at a real number 2 pencil and a Bic ballpoint pen. Notice the water droplet on in the lower right corner, which is drawn with a frisket and a soft cover brush in black and white.

It only took three or four passes with a soft brush to create the shading in the water droplet. I like to create catchlights on the edges of the objects I render this way. Still, it looks kind of naive to me now. Part of the shading style I had been using since high school!

On the very next day, I got a little more ambitious and did my best to create a cloud scene. I went back to mountains and big water droplets, which were a part of my visual signature at the time. Of course, the color picker worked in those days, but if you remember, it was a triangle with a hue slider at the bottom. Later I replaced this with a hue ring because I needed more accuracy in hue selection (particularly between red and yellow).

This scene ended up in several press reviews of Painter 1.0, mostly because it was in the screen shot in our press kit. The mist in the background was done with the eraser tool, which actually worked by subtracting density, a very different thing than overlaying white, which was how every other eraser worked as far as I could tell. The technology behind the colored pencil, the density-subtraction eraser, and the felt pens is covered by my '620 patent.

Another image to make it into the press kit at the time (for Painter 1.2) was the Splat image. This image showed how you could cut down on the density inside a frisket before shading it with a soft cover brush (which would now be known as a digital airbrush). Looking back at it, my control over the opacity of the brush was amateurish. But when you compare the results with any other paint program of its day, Painter's control over textures was really what made it stand out in versions 1 and 1.2. The texture technology was also covered in the '620 patent. I still remember trying to describe it to our patent attorney Jeffrey Hall. The ability to dim the area inside the frisket testified to one of the first effects that was added to Painter: fill. In its earliest form, it was actually a feature of the paint bucket. The transparency in the cover inside the frisket was the result of partial undo. That gave me the ability to fade back any operation I had just done. I think it was even called fade.

Soon thereafter, I invented a new kind of technology, that allowed the user to smear the canvas in color. This technology was radically faster than a pixel-by-pixel smear brush because the entire brush dab was given the same color. The color pick-up from the image was controlled by two mysterious color well parameters, called bleed (which controlled how much the colors on the canvas would be picked up by the brush) and resaturation (which controlled how much the current color continued to affect the color the brush deposits). If you set resaturation to 0, you would get pure smear. Then you could adjust bleed to control the length of the smear.

This Apple image was another image that graced the Mac press at the time. Painter 1.0 (really just known as Painter) was a Mac-only product. Painter 1.2 was released for both Mac and Windows simultaneously.

With the Just Add Water brush, my intent was to provide the feeling of smearing charcoal with a fingertip. For those of you who remember ImageStudio, you will remember that the smear brush icon was a fingertip. As it happened, ImageStudio (sold by Letraset from 1987 through 1990) was my first artist program, with charcoal and the fingertip. But there was no paper texture.

These mountains are very much a reminder to me of my style before Painter using real media. For these images, I put in the color using colored pencils, and then used Just Add Water to get the smeary look. I'm still waiting for the first iPad app that can get this look, by the way.

One day, I set up a 1024x1024 canvas and started to draw a large-format image. My subject was my left hand, always a convenient model. The background was just an ad-lib theme, kind of like a 3D fingerprint.

As I worked on it, I used a smaller-radius smear brush to even out the colored pencil. And I settled on a cubist style. You can see my style: relentless contrast improvement by shading.

This was the first larger image that I did; the rest of the images were 640 pixels wide or so (excepting the branch image at the top, which was an experiment).

This one also caught the eye of a few reviewers and graced the pages of MacWeek and other press. Again, as one of the only artists using Painter before its release, I could play to a captive audience!

But there were more brushes to create, and little time to do it. I got this idea that one stroke could be orchestrated to create multiple strokes that overlapped, and the VanGogh Brush was born. This was a "stroke of genius" according to several magazine reviewers that used the hackneyed phrase over and over again.

Here we see a Grainy Flat Cover brush being used using the VanGogh setting to create a faux VanGogh painting, complete with crows. Yes, I observed several VanGogh paintings in the Arles period to create this piece. Yes, I looked at the color sets he was using. This one made it into a few magazine art contests as well.

I was quite proud of Painter. And I was (naively) talented enough to have enough chops to test my own clever brushes as well...

Soon enough, it was time to ship a product and choose the packaging. Hal Rucker and Cleo Huggins showed me the Painter Can, amongst three other designs. Tom, Steve, Lee, and I all were brave enough to prefer the paint can approach, and the rest was history. I set about creating the image that would grace the front of the paint can itself.

The image came from a photograph by Rucker/Huggins. It took hours and hours because it was done at 300 dpi resolution, and so was 2400 pixels square. I did some brightness/contrast manipulation on the image and proceeded to paint onto it using a Grainy Hard Cover brush in VanGogh mode. The color was set to bleed 100% from the image, and resaturation was set to 0. In this way, I could draw in the colors from the image.

After it was completed, I went over it with a bit of a new effect, Surface Texture. There was a mode where the color variations in the image could be used to create the height field used for shading in surface texture. This image was used for the Painter can until Painter 4, in which John Derry did a mosaic rendition of the can.

When I first saw the paint can, I did this image showing the form as a basic shape. It used some grainy cover brush to put in the colors, and Just Add Water to smear them out again. No friskets were used in the production of this image. Or harmed!

At one point in 1994, John Derry and I thought up the slogan Painter can! This signified that Painter was a production tool and that Painter was up to nearly any design task you needed to get done. But it was nixed because it was too confusing. Steve Guttman responded "Hmm. Painter can! ... and it's in a can. Hmm." OK I got it, Steve! Hey, we were just being playful...

Actually, the can wasn't my first attempt at creating the VanGogh look on a real image. That has the distinction of being a landscape scanned from a slide on a Nikon slide scanner. I liked this image because it showed actual chiaroscuro.

I worked on the color variation used in the VanGogh brush. It uses the exact same brush and color pick-up technique as was used for the Painter can image. But I didn't apply any surface texture to this image after I was done painting it.

It was in the press kit, but I don't think it ever got used by the press.

Then came the time for going on the road. At Comdex in 1992, we were showing Painter 1.2 on Windows 3.1 for the first time. Tom Hedges and I got to meet Bill Gates and Steve Ballmer at that show, if I remember correctly. I doubt we were well-behaved.

Anyway, as the story goes, I was sitting at the Fractal Design booth at a huge monitor, sitting in a high chair with a Wacom tablet in my lap before the show started on the first day. It was in the Sands, I think, which was not the premiere venue at all. And I started drawing using a Grainy Hard Cover brush in a manner that I might have drawn with colored pencils. But this time, I was freely choosing colors every ten seconds while I drew. This actually drew a bit of a crowd of onlookers.

This has become one of my favorite modes for drawing: with many colors in an abstract color style that merely emphasizes the form without being too literal. I don't drink coffee, so I'm sure I wasn't drawing from life.

My usual technique of using white lines to delineate the shapes was used when contrast was lacking. I had to say it was fun to attract a crowd, and this gave me some ideas that were used later on in Fractal Design's booths. And, really, I didn't know how dumb my demos were until I saw John Derry demo Oasis at Macworld 1991. Now, that was a pro!

The next day, I created another painting using the same style. This time it was a vase. And kind of a crooked one at that. This time, as you can see, I was a bit more careful about which colors I chose.

And the same thing happened the second day: it attracted a crowd. Remember, I was a Mac developer at a Windows show. So I figured that our product was probably going to be a bit irrelevant to the Windows market. Boy, was I wrong.

Little did I realize just how hungry that market was for image editing and painting programs. Up to that point, paint programs were just pixels. Now paint programs were art.

And, with my naive style, I was lucky most Windows people were quite tolerant when they saw what an application could do. So it was even more groundbreaking on Windows!

When I got back to Aptos, I took that image and turned it into something quite different by using Just Add Water, again.

Now the colors took on the lilt of the California Plein Air style. If the colors used for the table were used for the sky, I'm quite sure it would have been better.

Oh, well.

As time went on, we secured more artists to do the demos, and the requirement for me to create example art was severely curtailed.

The talents of Chelsea Sammel, Corinne Okada, Rhoda Grossman, Diane Margolin, and Elise Huffman changed that by providing quite a stable of artists to draw from.

So, dear reader, that is the early history of my art on Painter. I have to say, my own style advanced considerably as time went on. Particularly when John Derry introduced me to scratchboard. But that is a story for another day.

My self portrait, shown to left, has a real tilt to it, which perhaps serves to indicate what an overbearing a**hole I could be in those days. Yes, I had an iconic beard, not much hair on top, and a pony tail. And what's with those glasses? The look was quite unusual for a CEO then. At least the investors thought so.

Hmm. Never ended up in the press kit. No wonder!

The Illustrated Evolution of Painter UI

Most of the comments I have gotten about Painter's UI over the years have been about how much stuff there is in it. We did implement features about twice as fast as we could organize them, I think. This is a history of that process. If you need someone to blame for its complexity, blame me. If you need someone to praise for how well we handled the complexity, praise John Derry. The images come from a Japanese web site. The commentary is mine, of course.

Painter and Painter 1.2

Both Painter 1 (just called Painter) and Painter 1.2 use this kind of splash screen, which was a rectangular selection grabbed from the original art of the paint can which I painted for Painter 1. This artwork is found on the cans for Painter 1 and Painter 2, as well as the Painter 1 poster. It was painted in August of 1990 at my Mac IIfx workstation using a Wacom tablet and Painter 0.9, an early version of Painter.

Note the ugly dark-green Fractal Design logo (which I designed, BTW). OMG.

The Painter 1.2 user interface shows the very start of our UI. A palette each for tools, brushes, colors, and papers. For controlling brushes behavior, there are the brush size, brush behavior, and expression palettes. For selections in Painter (called Friskets in early versions) there was the frisket palette and the fill palette, which also helped control the paint bucket. Finally, a correction window was more a modal approach to the brightness and contrast of the image.

Note the color picker was a triangle, but with a hue slider at the bottom. The brush icons were a bit inconsistent, and featured gigantic images.



Painter 2.0

Painter 2.0 saw the influence of John Derry. John had a much more literal approach to splash screen design, as you can see. Also, note the inclusion of John Derry and Bob Lansdon as authors of Painter. Bob did work on the watercolor brush technology. Note also the trademark on Natural-Media.

The design of the splash screen is heavily influenced by the CRC Handbook of Chemistry and Physics, which features the same color scheme and gold press. The Fractal Design logo has changed, meanwhile, to the "Big FD" as we called it. The real Fractal Design logo has color paint strokes inside it. Here it was decided that the color paint strokes would ruin the effect John was trying to achieve. For the new logo, John Derry designed the shapes and I designed the brushy fill. Later, Cleo Huggins produced a high-resolution PDF version.

For Painter 2.0, we added Brush Looks, which were presets that we used as a tool to cut through the complex interface for the brushes A brush look was the combination of a brush, all its modified settings, and a paper texture. You could design the icon for the brush in the Brush Look Designer, another palette for trying out a brush and its settings before you went to use it. Most of the features added to Painter 2.0 were brush capabilities, including things like angled tips for the brush, clone location variability, control over the smeariness of a brush using Resaturation and Bleed. The random grain brush strokes capability allows the grain to move around with each dab of the brush that was deposited to the canvas, creating a much better textured look.

We supported tear-off brush variants, brush looks, and textures, so the screen space could be minimized for a user working on a specific task. Incredibly, we added scripts to Painter 2.0. You could play them back at a higher resolution. The brush stroke designer was driven by a script for a brush stroke that could play back with the new brush settings whenever you changed them.

I created two special effects that were unlike any other program at the time. The first was Apply Marbling. This allowed you to create marbled textures like you get when you drag rakes and tines through the surface of a liquid that has spots of ink on it. The second was Apply Lighting. This allowed you to light an image by conical light sources to achieve spot light and other natural lighting effects.

A new brush, called the New Brush Model can be seen in the Brush Behavior palette. This brush was characterized by a cylindrical arrangement of bristles, some portion of which could touch the canvas during any one point of the brush stroke. The diagrams for how this worked are shown in Creativity and Painter, Part 1. They are the yellow pages.

Painter 3

With Painter 3, John and I introduced the world to the image hose.Our new product manager, Steve Guttman, and two new programmers, Priscilla Shih and Shelby Moore are credited as well. We show the real Fractal Design logo here in all its colorful majesty as well. A reference to the '620 patent is seen at the bottom. This patent was for tiled textures interacting with a brush, and Lambert/Beer build-up, which enabled color pencils, charcoal, and felt markers.

John Derry's splash screen used a rocky texture background and the image hose to make it look more natural than it ever had before. The golden letters theme was maintained, though.

The Painter 3 interface was a radical departure from its predecessors, showing the organizational influence of John Derry. Toolbox buttons were now sculpted like the keys of a retro typewriter. The palettes are now gray-backed and the icons are much smaller. The numerous brush icons are now located in a drawer-like structure that can be opened up and closed. We first started toying with drawer-based interfaces in Dabbler. Each of the palettes has 5 graphically-represented sections. Friskets have been renamed to Paths, at the request of Steve Guttman. Our implementation of Layers, initially called Floaters, was debuted in Painter 3 as well. The Nozzle section in the Brush Controls helped the user choose the kind of images that are used with the image hose. With Painter 3, each action you did is now recorded in a script, so it can be played back, even at different resolutions. Color Sets, Gradations, and Weavings have been added. I added multiple undo as well. Painter 3 (including version 3.1) was clearly the single largest change to Painter yet.

An important UI feature was the ability to tear off art materials, like the color palette or the paper palette. Yet they were also nicely organized. This tear-off feature worked with any palette that contained icons.

Oh, and you could make a drop shadow for a floater with the push of a button.

Painter 4

Now comes Painter 4 in 1995. In Skagen, Denmark in the summer of 1994, I coded a mosaic tool, based on the polygonal boolean operations. In this model, one vector-based mosaic tile can be used to clip another. Outsetting operations allowed me to enforce grout requirements. The influence of the mosaic tool is seen in the splash screen, produced by John Derry. The gold letters are now bevel-edged. A dappling of light and shadow graces the image. The venerable Glenn Reid, Vahe Avedissian, and Christy Hall (now Christy Brandt) joined as authors.

Sorry, this UI screen shot is in Japanese. But you can see that individual art materials are torn off into their own palettes. There is a new art material: patterns. There are additional tools for Shapes editing and for shape creation. The gradations palette allowed rotation of the color ramp, and it also allowed different topologies for the ramp: linear, polar (radar), circular, and spiral. Scripts got their own palette, and it had a transport deck for playing scripts back like you might play back a video. Note that the Shapes tools got their own little mini-section in the tools palette.



Painter 5

In 1996, with Painter 5, we added several new kinds of brushes, including liquid metal, fire, and dodge and burn. The influence of these brushes is seen in the splash screen, created by John Derry. You can tell it is his by the painted mark of a hand, which is one of his signatures. Scott Cooper and Erik Johnson are now on the author team.

We added many new kinds of brushes to Painter 5. There were brushes for blurring and sharpening, for bulging or pulling the image around. There were brushes for fire, glow, dodge, and burn. There were liquid metal and transparent water droplet brushes, complete with shines. You could add grain with a brush and change the hue and saturation of a section of the image, while leaving the luminance alone, for easy tinting effects. A relief brush allowed you to draw the surface texture of the image separately from the color of the image, and you could even twirl an area of the image directly. The various distortion effects were supported internally by a vector field through which the image was advected for display, kind of like Kai's Power Goo.

In the UI, you will notice that the icons and the color picker and other screen objects are laid into the gray background rather than having a drop shadow as they did in Painter 4. This further cut down on screen real-estate, and provided less conflict for the eyes. Some of the icons went to half-height also.



Painter 5.5

In Painter 5.5, we added a web-safe palette, and some features for cutting your image into sub-rectangles for easier inclusion into a web page. This is especially useful when some parts of the page design use flat colors or have less detail and can thus be compressed more effectively than others. The additional web features of Painter 5.5 was the effort of some of our new Ray Dream personnel (François Huet, Damien Saint-Macary, and Nicolas Barry) and Scott Cooper.

Meanwhile, I was working on a completely new brush model for the next version.

You also see a new logo on the splash screen, the trefoil knot of Metacreations. This company was the result of the merger between MetaTools and Fractal Design. I began spending more time talking with Kai Krause about new interfaces and features.



Painter Classic was a feature-limited version of Painter, kind of like Painter 4. It was a lower-cost product and designed to be bundled with scanners, cameras, and tablets.

Painter 6

When the feature set for Painter 6 was being discussed, I brought up my brush testbed. This project was my skunkworks for new brush models, where I had employed object-oriented programming using C++. In the new testbed, I had implemented several kinds of new physically-modeled brushes. The first was the multi-bristle brush. With this brush, each bristle had its own path and could allow for splay of the bristles. In addition, each bristle had its own color as well as its own internal model for picking up color from the image. The bristles were initial organized in a bristle bundle, which was based on an irregular tight-packing of spots. The look of the bristle bundle was based on the "monkey eye receptor" model, since the bristles were really in an organic cellular arrangement. To make this packing, I had a special tool for computing bristle arrangements that used reverse gravity to create a very regular spacing indeed. The arrangements were additionally computed in a fold-over square that could tile the plane with this arrangement. So I could compute brushes with an arbitrary number of bristles more economically.

Each bristle path was processed in space by using an exponentially-damped signal of (x, y) pairs. This led to a more physically-based model of the bristles, and gave each bristle a little bit of individuality. If I were to do this again today, I would probably start worrying about the capillary effects between the bristles, since that's more like the way the paint is actually transported through the brush.

I also developed a spatter airbrush model that had a physically-modeled trajectory for each speck of paint. And a proper concentration of trajectories. And to top it all off, it used the Wacom tablet and stylus to drive it as realistically as if you were using a real airbrush. I was quite proud of this brush.

Another brush I developed was a brush where images could be warped along the path of the brush. At this time, we were working with Alex Hsu of Creature House, whose Expression product we sold for a while. I couldn't be outdone by Alex, of course, and so I implemented this brush to warp pixels and do it using mip-maps so there were no undue aliasing artifacts. It could work with masked tiled images so you could draw chains and other cool figures quickly. The width of the stroke could be controlled by pressure with ease.

The marketing and development of Painter 6 are detailed in Creativity and Painter, Part 2. The war for Painter 6!

Metacreations' new three-hump logo was not my favorite, and it was the first logo I didn't have a hand in. Fortunately it was short-lived!



In the Painter 6 UI, we finally tired of not being able to access the deep features of the brush controls in a way that allowed simultaneous tuning. Therefore, we went to an expanding list, with disclosure arrows. On the front of a closed-down item there was a name and sometimes a picture of the art material or a swatch of color indicating what was the current color. We also put a salient parameter on the surface of the closed-down list item, such as opacity or the name of the current script. This could act as an important indicator when the list item is closed. Also, at the right end of each close-down list item is a pop-down menu of commands for that area of the product. This was another important feature for users.

Closed-down list items really cut down on the complexity of the program. they also let the users open up only the features they needed for the task at hand.

This wound up the most significant advance in Painter's interface, the product of John Derry and myself.



Painter 7

In 2000, Painter was bought by Corel and Tom, John, and I consulted for them for a year and a half, while helping to construct Painter 7. You will notice that Tom's name migrated mysteriously to the start of the splash screen credits. I can tell you I didn't have a hand in it; neither did I complain. After all, I no longer owned the product. As we constructed some really interesting features for the product, numerous Corel engineers started the process of taking over the code, and moving it over to Mac OS X.

Now, I'm not sure why they chose to label it under the procreate brand, nor why they put a bunny on it. At the time I found it hard to care, mostly because it was now their product. But I certainly didn't skimp on my enthusiasm for new features.

My Liquid Ink feature was based on a potential height field that metaballs were added into. It actually kept the height field in floating point. Because of the physically based viscosity and surface tension model, it was possible for blobs of ink to join together like two water droplets. this same technology was used for water droplets and also reflective liquid metal ink. What differed was the rendering of the color. Water was transparent, allowed for an amount of refraction, and had a shine model that used environment maps. Because the height field was floating point, it could be easily shaded like a ray-traced bump map. The Liquid ink used a reflection map for its coloring. This was based on a normal calculation from the height field, also from 3D bump maps.

My Watercolor implementation in Painter 7 used several layers to simulate the paint that was still wet, how it traveled, and its interaction with the paint grain. This became the most realistic form of watercolor available yet commercially. It used cellular automata to transport the dyes with liquid while it was wet, and interaction with the paper grain dictated where the ink would dry first, or get absorbed first.

If I had to do this again today, I would model individual paper fibers and use their capillary action to dictate where the liquid dye moves. There is just so much compute power available today; it's dying to get used for something like this.

My Woodcut feature allowed you to algorithmically select areas of the picture that were similar to a given color, process them to model the stickiness of the silk screen ink, and do a wood block stamp of that color onto a separate layer. This would allow you to create several "prints" on top of each other, like carved linoleum pressings.

Tom developed an improved PSD import, crafted continuous zoom, and I think he also worked on the perspective grid tool (though I'm not sure).

Much of my effects and brushes for Painter 7 were perfected with the assistance of John Derry, who contributed more than his share of good ideas, as usual.



The interface for Painter 7 did not change much, except to become "lickable" like the rest of the early Mac OS X aqua interface. Topologically, though, nothing actually changed that I can recall.



Painter 7 was the last version of Painter that I contributed to. It had been a long 11 years since the start. But Painter was a very fun project to work on!

Thanks

And it was always a pleasure to work with the Fractal Design folks! ...and the Meta folks! ...and the Corel folks!

To think it all started in my house in September of 1990 and it might never have seen the light of day had Letraset not been such a messed-up place.

Nonetheless, blessings to Marla Milne, Martin Dowzell, Brian Cohen, David Taylor, and Mike Popolo who worked at Letraset in the early days. I have many good memories of that time.

Creativity and Painter, Part 3

The visual design of Painter 2 was  fantastic process with John Derry on board at Fractal Design.

What's in a Name?

We notably called it Painter 2.0, even though later versions of Painter dropped the ".0". Our new product manager for Painter 3, Steve Guttman, maintained that people usually didn't want to buy the ".0" version, instead waiting for the dot release.

But before Steve arrived, the marketing designs for Painter and Sketcher were essentially the Mark and John show. And we went wild, usually.

In between Painter versions 1 and 2 there were several other products released. These included Painter 1.2, which brought Painter onto Windows 3.1 by using Altura's cross-platform framework. Also, Sketcher, a lower-cost grayscale version of Painter specifically designed for laser printers, was famous as a product that shipped in a cigar box. It seems that many artists used old cigar boxes to store their pens, pencils, brushes, and erasers. Tom Hedges spearheaded those projects while I busily constructed the new features for Painter, and did a few of the press tours along with Karen Stagnaro (now Karen Bria), Cori Garnero (now Cori Tuck), Jennifer Andrew, and Dawn Hannah (now Dawn Bercow). I should also mention that Fractal Design released additional texture libraries for Painter, in particular Really Cool Textures.

So Hot, So Cool

For Painter 2, we decided to choose a theme for the marketing of the product. So one day John and I sat down and started dreaming up a catchphrase. Eventually we ended up with So Hot, So Cool. This meme actually had a few good associations with us, and we began to draw. My specialty was the graphic form, so I came up with a few sketches you see to the left.

We tried a reverse-out styling at the lower left: too obvious. And the Clockwork Orange letters at the top: too strange. Hexagonal and interlocking letters: naaaah! But in the center are cleaner designs that come much closer to the letterforms we ended up using.

Note at the top left there is a recipe for some graphic I did giving each part of the catchphrase a different look. Landmass, Corrugations, and Hatching were all texture names. The numbers next to them are all scale factors. Akzidenz Grotesk is a font from the Berthold foundry. We ended up using that font for the catchphrase letters on the poster.

The Burning Ice Cube

We actually went a bit farther with the design when we realized that a burning ice cube graphic could sum it all up in one insane visual theme. Now that's creativity! John executed some designs to show the So Hot, So Cool theme that subsequently went into ads and also onto the Painter 2.0 can. The visuals of fire and ice, hot and cool colors, were incorporated into the design. A new thing appeared as well: a sticker. In Santa Cruz, where we came from, stickers were all the rage with skateboard companies, and so John created a spot color design for the burning ice cube. A stop sign theme with the word HOT replacing STOP. A knob with the word COOL on it was the symbol for the other side. And John put one of his visual signatures in the center, a hand print. This is a symbol for human expression and mark-making: the first thing a child does is put their mark on something.

When it came to rendering the burning ice cube, we all had our own ideas. But John had a really interesting one: stained glass. I knew this was destined to draw attention. The irreverence of the burning ice cube meme mixed with religious icons. Yeah we were a bit on the edge.

When we introduced the product at the MacWorld expo in San Francisco, I have to admit we did place a few of these stickers around the city, in the spirit of the skateboard sticker.

Perhaps the strongest criticism of our new look came from the New York press, according to Karen Bria. They commented that it looked too west coast. Well, I think that was our goal actually.




The Painter 2.0 Poster

The poster was a difficult project, with four artists working for days on each image. Steve Manousos had the responsibility of gathering up the artwork, and he generated high-resolution comps of each poster image, which I still have on hand. By the way, this was in addition to being responsible for writing the manual and setting it in Quark XPress. And also generating the four-color transparencies for the printer. John Andrew supervised that part of the process.

John Derry created a  burning ice cube that was a famously a stained glass rendition of the theme. His first version of it, which ended up in some screen shots that were distributed to, amongst other magazines, InfoWorld, was a simpler version of the theme, but can illustrate the direction he was taking for the final piece.

The final poster images were to be executed at 300 dpi and were thus gigantic in those days. Nonetheless, with Tom Hedges' new virtual memory capability, Painter 2 was up to the task, even in its beta versions, which were what we were using to complete the poster images.

John Derry was clever enough to observe existing stained glass so he could see how the lead was inserted to provide structure and support as well as how the glass shapes were limited to what could actually be made by actual glass cutting techniques. For the final poster, John exceeded our expectations by including hints of a blurred background behind it, and by giving each kind of glass its own texture, which came from the kinds of textured glass he could observe being used in real stained glass.

His final version, shown to right, with the beveled-glass teardrop at the bottom, shows a rectangular leading.

The ultimate tribute was payed to this piece by a real stained-glass artist that called John up and asked him about the piece. They had talked for quite a while before the artist realized that the piece wasn't an actual photograph of a stained-glass creation. OMG.

Such is the consummate level of John Derry's art, his accomplishments.

I was really lucky to have met him, much less to have worked with him.

I could say the same of Tom Hedges. And the three of us were the central nexus of Fractal Design. When we were in Aptos, on Spreckels Drive, we shared a large office space with three desks.

When we moved to Scotts Valley, we continued that tradition. We were the formula for Painter. Not to minimize the influence of our wonderful employees. At that time, much screen shot artwork and example pieces, for instance, came from our employees, like Michael Cinque. The collection of artwork and prints were compiled and bound by Cori Tuck and in many cases printed by Steve Manousos.

Next on the Painter 2 poster was the piece by Bill Niffenegger, shown to the right. I guess I thought it was kind of a Norse treatment when I first saw it.

It's in a rock cave, like a well, with a pool at he bottom. It's dripping into it as it burns. The flames seem to me to be like ears or Viking horns. I think that may be because the ice has a face. If you look close, you can see that the flames actually start inside the ice as a kind of flow. Then they magically come out through the ice.

We liked Bill's treatment, and as a result, we also used him to create the cover of the Painter 2.0 Companion, which also shipped with Painter 2.0 in the can.

It's a scene in a comfy living room in front of the fireplace, with a man reading something, perhaps its a self-reference. More about that in a minute.

The third image on the Painter 2.0 poster came from Hal Rucker. Now, Hal, at Rucker/Huggins Design, was the design shop that presented the paint can idea to us. We had four options to choose from, but we chose Hal's can. They also designed the Sketcher cigar box.

It turned out that his poster image, shown to the left, was quite remarkable.

To me, it looked a bit like a meteor. But what made the piece was the internal colorings and shadings of the ice cube itself, as it picked up on the hot and cool colors.

It is distinguished from the other poster images by its remarkable continuity and realism. The ice cube is rendered solely by its reflections and refractions. It shows the result of observation of real ice cubes, I think.

It is funny, but when I Google for burning ice cube today, I see a lot of images that are visually quite similar to our original burning ice cubes.

Before releasing Painter 2.0, we found out that Styx had used a burning ice cube for the cover of Equinox. So we definitely weren't the first to think of the theme. But it went so well with So Hot, So Cool, that we just could' help ourselves. Our biggest fear was that someone would come up with So Hot, So Cool, So What! But that didn't happen. Perhaps they were taking us seriously...

And the fourth Painter 2.0 poster image came from me, Mark Zimmer. After John did the sticker, I wanted to so a spot color rendition that showed the effect of refraction, the distortion and convection of the fire. And I wanted the cube to be half-melted in a reflective pool of water.

My image, shown to the right, used glass distortion to warp out the straight edges of the refractions inside the ice cube. The flames were pulled and re-sculpted several times. With each effect I applied, the edges got soft, so I had to redraw each of the edges many times to get the effect I wanted. The reflective pool was also fun to draw. And why I had the flames weave in and out of each other in 3D, I will never know.

I applied a waviness to the reflection of the cube in the water below.

The whole image seems kind of naive to me these days.

Painter 2.0 wasn't complete without the Painter 2.0 Companion. This was illustrated by John and myself, and written as a tutorial by Karen Sperling.

There was so much work in those days. And when Painter 2.0 was all done, I began to take the wraps off my latest clever invention, that was to become Painter/X2. Layers!

Image Retouching

I have done my share of image retouching and adjustment, but, being in the industry, I have seen other people do the same, often to great effect. I will now tell a few stories of what I have actually seen. I just wish I still had the images!

Tom's Chipped Tooth

The earliest case of image retouching I can remember was in 1987, when Tom Hedges and I (who made up the partnership Fractal Software) were working on ImageStudio. Tom was working on scanner drivers and I was working on the main user interface, brushes, and things like floating selections, which were influenced by Bill Atkinson's MacPaint. Since Tom was working on scanners, he was the source of images we worked with. So some of the first images he worked with were of himself and his family back home in Kansas City. One was a nice smiling portrait of himself and another was of his mother Georgeanne, his brother Chris, and himself. His father had died some years earlier of ALS. Our marketing partner in those days was Letraset, and our product manager was Marla Milne. Marla had a bit of a prankster streak in her, and we thought that was good, because it went well with our senses of humor. Now, Tom had a chipped front tooth in those days and it was his visual signature when he smiled. So Marla, once she got ahold of Tom's images, took his chipped tooth and applied it to both his mother and his brother in the family portrait.
It was hilarious, diabolical, and irreverent, like something out of Mad Magazine!

She brought it to our tiny 660-square-foot office behind Piggy market (which its now the Palm Deli) in Aptos. OMG was Tom pissed when he saw it! I could barely contain myself and prevent myself from laughing when she showed me the picture. I wish I still had the picture, but I suspect Marla was clever enough to delete it once she saw Tom's response.

And I'll be damned if Tom didn't have that tooth fixed a week later.


Electric Paint

At Fractal Software, in 1989, once we got into the design and implementation of ColorStudio, we made some friends in Hollywood. One of those was a really smart Australian guy named Tony Redhead, who ran Electric Paint. Electric Paint was a Hollywood service bureau that did image retouching with Quantel Paintbox XL and it also had some video bays. Each room was set up with a specific design for the artist (on a lower level with a full-sized workstation and a humongous CRT, running the Quantel Paintbox XL) and the client (on a higher level and with a comfortable couch). The art director usually traveled between the two levels and managed the relationship. This was my first exposure to the artist/art director/client relationship. And Electric Paint played it perfectly.

Officially, I was working on a software connection from Quantel to ColorStudio, so files could be moved in and out and the advantages of both platforms could be used. Tony was big on that concept.

Once, I was ushered into the artist studio, and introduced to Marlo Bailey, who was a professional retoucher using the big Quantel box. On the screen was a picture of Paula Abdul that was to grace the front of a TV Guide. I was suitably impressed at the subtle and not-so-subtle work that was being done on the image. In particular, with a flick of keys, I could see Miss Abdul's rear end change from big to small. I found it to be hilarious: so this is what the Hollywood clients wanted.

I could just imagine a client in the couch above calmly asking, "could we make that ass smaller?". OMG!

The were telling me that it was even more interesting when one model was used for the body and the star's head was placed on top. They called it a head swap. Apparently this was done quite often.

Portrait Mix and Match at Fractal Design

John Derry was telling me one day in 1994 how funny it was to to head swaps and we got the idea of using a soft mask to move someone's face onto someone else's head. Of course Painter was up to the task and I got the idea of snapping a portrait of all the personnel at Fractal Design with an Apple QuickTake camera. We were careful to use the same lighting and face orientation for each person. We had a wonderful time swapping faces from person to person.

We also found that making the face 5-10% smaller or larger and recompositing it back onto the same face had the effect of creating an outrageous caricature. Good times!

Net Painter

At Fractal, we were interested in collaborative painting, and I pitched this to Tom Hedges in 1996. He spent a huge amount of time working on a version of Painter for Internet collaboration we called Net Painter internally.

It was based on a low-level protocol called UDP, that TCP/IP is based on. At the time, Tom worked with a friend named Stuart Cheshire to figure out the protocol and the main means of transmitting the data and achieving reliability. It's odd, because UDP means "Unreliable Data Protocol". They explained that you have to first assume it's unreliable, and then verify the messages that got sent.

Net Painter was to allow the painters to capture the moment and draw, and then others could interact and work on the same document. It had a stoplight protocol visually, so you could understand what your status was. Also, it had a list of users that you could see, like users in a chat list. When their stop lights went yellow, it would indicate that they wanted to draw. And text would come from each user as well, so you could have a conversation. Not necessarily in a chat mode, but, really asynchronously. I used it many times, and we used it across the ocean on occasion.

So it became an environment for collaboration across the entire world through the Internet, which was rather low bandwidth back then. To implement it, we sent script items across the net, like a recording in Painter. Painter at that time had already begun to record every nuance of your artistic process, so you could always play back your session, remember? This was actually part of Net Painter: a secret plan.

But Tom was always having difficulties back then with his lymphoma and also relationship problems with his wife. And this project, though it worked, was never fully realized I think.

So Net Painter was kind of a whiteboarding application.

We wanted to be able to ship files across as well, so we could share more complex things.

I believed the idea of a stoplight from Tom's Net Painter was stupid, though. Why bother to have something that halts creativity, when you can be spontaneous! This is the way I always wanted it to be, with two people or more working at once. It was a curious prototype.

Sure, the time lag can make collaboration hard to do in real time. And that can cause other issues of confusion as well. But I think that most of the time, this will not be true. It is for the same reason that most source changes on a multi-developer system can be disambiguated and merged back in. Only on a painting, which is two-dimensional, the collisions might indeed be less frequent.