Sunday, February 5, 2012

Texture, Part 2

The Texture application was a secret application I created at Fractal Design for creating paper textures and for testing out new technologies. But it fell into disrepair, and so it hasn't worked since 1996. But I have been resurrecting it under Mac OS X, and it is just now starting to come back to life. In part 1, I explained what speckles are. They are a basic area of technology used by Texture. Here in part 2, I will give you an overview of another area: FFTs.

The Spatial and Frequency Domains

For instance, the FFT window has a section for the spatial domain and another section for the frequency domain. The spatial domain is the one in which we can open and save textures. The frequency domain is used for applying sophisticated filters and other operations to.

You can apply filters to the frequency domain, and see the filtered result in the spatial domain in real time.

But, you may be asking yourself, what is an FFT? An FFT is a Fast Fourier Transform: a fast way of transforming spatially-varying information into its frequency-domain equivalent and back again.
Above you see a diagram of the process. Texture information is imported into the spatial domain. A forwards transform converts that data (using an FFT) into frequency data in the frequency domain. Also, a backwards transform converts the frequency domain data back to the spatial domain (also using an FFT). In the window above, you see the True-Lizard texture and its frequency-domain representation.

The frequency domain is organized so that the center of the square has the bias of the data (you can think of that as the average of all the pixels into just one gray level). Around it are a small number of pixels corresponding to the biggest wavelengths. The farther you go out from the center, the smaller the wavelength of the corresponding spatial data. It's kind of like an inside-out representation of the texture.

Operating in the frequency domain has several advantages. You can easily filter the data to create different versions of it that are constrained to a limited number of wavelengths.

Another basic advantage of the FFT in particular is that it works really well with wrap-around data. It's just made to work with tiling textures. All of the outputs of the FFT are tiling textures by definition! And, generally, they are powers of two in length, so most of the textures you see here are 256 x 256. But you can easily construct 512 x 512 and 1024 x 1024 textures, and larger. Well, there are some constraints, but these properties do make FFTs ideal for processing wrap-around tiling textures.

So, how do you filter in the frequency domain? Convolution is the best way. And that works by multiplying in the frequency domain. You can multiply two frequency domain FFTs to convolve their textures. Or you can just multiply by a spot function.

Here is the spot function window. You can define the size, hole size, the softness of the edge, and you can even define spots that are elongated and at different angles.

In this case, I have dialed up a small spot, and in the frequency domain section, I have checked Filtered and so when I adjust the spot, I can see the spatial domain image change in real time.

This is particularly cool, and this also means that I can turn the lizard texture into a leopard-skin texture with just a slider-move.

You see that the texture closely resembles the original. The spot with a hole in it makes a bandpass filter that limits the spatial frequencies of the texture, but totally preserves its wrap-around quality.

Another thing that works easily in the frequency domain is the creation of fractals. In fact, I have a Make Fractal button in the frequency section.

In the frequency domain this means creating a 1/f frequency spectrum, and randomizing the phase. You can also use the Make button in the spatial domain section to make a spot and then randomize its phase in the frequency domain section.

Either way, this creates a perfect wrap-around fractal, which you see here. To get different fractals, just randomize its phase. Filter a fractal, and you can limit its spatial frequencies. This means you can make it appear to be composed of several smaller clouds instead of one large cloud.

Here is the result of filtering it using a highpass filter (where the inside of the spot is black and the outside of the spot is all white to the edge).

The remarkable thing is that it's the same fractal. You can see some of the features are still in the filtered result. Really the only difference is that the slowest-varying features are removed from the fractal.

Now that's cool!

The neat thing about fractals is that they also can make excellent paper grains.

When I filter this same fractal using a highpass filter so that only the smallest features get through, I end up with a pretty good paper texture just like that.

So, you can see how I can create a large number of paper textures with grains of arbitrary size and softness from one pattern. And remember, all I have to do is randomize the phase in the frequency domain, and I have another one just as good as the previous one, but with entirely different features.

I was quite clever in the Fractal Design days, but it appears that I am actually much more clever now because I have considerably more knowledge of the frequency domain. You see, it all makes much more sense to me now than it did then.

For instance, now I have arranged it so I can adjust a slider to change the filter and thus the look of the texture in real time. I couldn't do that before.

So part of the fun is that I am using Cocoa on Mac OS X to make the UI work. And it's much faster to write that now. And part of it is that computers are a bit faster than they used to be. Maybe about 100 times faster, and with multiple cores too. And GPUs to get things to run really fast. That's what I do for a living now.

Which is why Texture is coming up to speed at an amazing pace, considering how long it has been since it worked.

Anyway, back to the frequency domain.

So, I cal also construct a lowpass filter. By making a spot that's white on the inside and black on the outside, and using that to filter he frequency domain. This allows me to create softer versions of something. Which can be quite useful.

For instance, I can load an existing texture, thick-knit. This texture was made using an interesting technology for one of the future parts of the Texture posts. That technology was called slice-and-dice, although it was based on crossfades made of Voronoi cells on a tile.

It made for quite complex programming, but it produced some of the more interesting textures I produced.

This texture has a lot of tiny details, and if I apply a subtle lowpass filter to the details, I get a slightly different result. Because I can change the edge softness of the spot, I can also control how hard or soft the lowpass filter's edge is. This is extremely useful in achieving specific visual looks.

In this case, a very natural blur is imparted. But with none of the ringiness of a Gaussian blur.

Real or Imaginary?

The FFT operates on complex-valued data. This means that each pixel consists of two floating-point numbers: the real part and the imaginary part. Each complex number is of the form:

a + bi

where i represents the square root of -1. That's how I can alter the phase of the information independently. When I say that the phase is randomized, I mean that, in the frequency domain, I have to convert the complex number (a,b) into a polar coordinates (R, theta), put a random value into the phase angle theta, and finally convert it back to Cartesian coordinates.

When I'm all done with the operation, and I've transformed it back to the spatial domain, then the real value becomes the texture value.

So, when I randomize the phase of the low passed thick-knit pattern, I get the texture you see to the right. It's kind of like a shag carpet. But some of the features, like the tilt of the texture come through. This is called anisotropic texture, which means that it favors some angles. Like the original think-knit texture.

Texture Synthesis From Geometry

In Part 1, I showed how speckles can be used to create cellular patterns called Voronoi tesselations, tree-like patterns called spanning trees, and now I have mentioned crossfade sets.

Here is a diagram for how these work together with textures to create even more textures.


In the next part, I will show how crossfade sets can generate the most interesting patterns by reference through the cellular arrangements derived from speckles.

8 comments:

  1. This is very interesting, Mark. I managed to create a set of Grain offset scripts for Painter, using the info. you wrote in an old Tech note document;
    http://www.jitterbrush.com/?p=734
    Other idea I had for the Paper textures palette were;

    1. to internally and temporarily be able to apply Painter's Glass Distortion filter to the currently selected paper texture (with an option to save as a new texture). Example using default Woodgrain texture with glass distortion;
    http://www.jitterbrush.com/images/Sim_Woodgrain_Distort.jpg

    2. to be able to stack 2 or 3 textures as layers from the Papers palette, with each layer taking advantage of transparency and independently moveable. The final texture output applied to the canvas via. a grainy variant would be a combination of all the textures in the stack. By experimenting with this on normal layers, I discovered that sliding just one texture over another (preferably high contrast, open grained texture) produced a significant number of new textures based on the layer overlay position relative to the one below.

    ReplyDelete
  2. David,
    Of course Texture can use glass distortion to offset a texture using itself or using another texture. This is how some of the textures in Painter were created. Offsetting a grain for combination with itself is a technique for generating anisotropy, or an effect that can make a texture appear to be shaded from a given direction. Like offsetting one direction to highlight and offsetting the opposite direction to shadow. Stacking textures is useful for additive combination. For instance, you can combine two relatively prime sized textures to make a larger texture with a huge repeat pattern not discernible to the eye. Actually that's one reason I use speckles: the repeat pattern is much less obvious.

    You have good ideas and obviously you have the mind for it. It's always nice to meet others who have thought similar thoughts.

    --Mark

    ReplyDelete
  3. Obviously much cruder than your application, below is a screenshot of a random macro texture generator which I made in Painter IX (again from info. found in your Tech note pdf). This is a Painter scripted interface, with the scripts ran via a custom palette (right in screenshot).

    First a new 8x8 pixel document is created, next I select the 1-Pixel Pen (Black). I can then choose all black or 50% Value variability prior to selecting one of the 'Auto Clone' presets on the second row, which uses Painter's ability to randomly place one or several dabs on the canvas (left), prior to Select All and Capture Paper. Centre is resulting paper created;

    http://www.jitterbrush.com/images/macro_tex_gen.jpg

    David

    ReplyDelete
    Replies
    1. Very interesting! I had a technique that I started using in ColorStudio called sequential random addition (SRA) to create textures which was just to use random dabs of a brush into a tiling pattern. But I found it limiting as to the kinds of textures it could produce. Nonetheless, several of Painter's textures were generated using this technique. Each dab could be a shaped object, or one tilted at a random angle, or even a ring, which was useful for creating cratered textures.

      Painter also features the ability to define pattern, which makes the image into a pattern. This gives you the ability to auto-clone into the pattern without generating an unsightly repeat.

      Some have asked for paper textures with half-drops like Painter's patterns can have, and that brings up the subject of all the planar tiling groups. For instance, a hexagonally-tiled paper texture might be interesting.

      But, for now, I think my work on Texture will concentrate on bringing the slice-and-dice method of creating textures back to life. You're going to like that one.

      Delete
    2. Yes, the Define Pattern is a very useful feature. If I use Define Pattern initially, then Auto Clone using a suitable captured Dab variant, I can achieve a seamless tile for capturing as a paper.

      Another technique I have experimented with is using custom variants to sample an underlying area of existing imagery already on the canvas, then 'spray' the sampled fragment in the same or different location to create a new fragment-based texture by multiple dab application (not using the Image Hose). Brush Loading = checked, Bleed and Resat 0%. The trick is to set Opacity to pressure so that the loaded imagery is not applied to the canvas whilst the stylus tip is dragged to a new destination area (using the minimum of stylus contact pressure), prior to the sampled imagery being applied. Unfortunately there seems no way to temporarily/permanently hold the loaded pixel data to the brush tip, thus this is lost when the stylus is lifted from the tablet surface.

      I am looking forward to trying out your Texture application, should it become publicly available in the future.

      David

      Delete
    3. Well, you have hit the nail on the head. To use a random pattern through a brush to generate textures is cool (even one from a clone that is positionally jittered, which produced the first random edges in Painter in 1991-2). To load he image onto the brush and paint with that is more like the image hose. But creating one is complex and time-consuming.

      What you ask for is a kind of temporary image hose. And that is one of the things Texture is designed to do. I was consciously keeping a set of interesting technologies out of the product so I could make textures back in the 90s. Imagine that!

      Well, this technology is slowly coming back, and it's the basis of slice-and-dice textures.

      A crossfade mask set is derived from the Voronoi tessellation (that cellular partitioning derived from a speckle, shown in Texture, Part 1) and these crossfade masks can each hold a piece of an image to place within them. The reason the word crossfade is used is because each cell has a soft edge based on a certain crossfade distance.

      The cool thing is how they get placed. As you will see in the upcoming Texture, Part 3, a z-buffering approach works best in producing visually interesting textures.

      Delete
  4. Folks, I have a theory where humans are here to learn and write the user's manual. We'll cure mortality while enabling our own abilities like, sprouting an extra appendage or changing density. New fun started yesterday to learn that frequencies shape matter from energy, period. So...

    As anything can be articulated mathematically and music holds hands with math, I've been wondering about fractals based on notes and timing. Now, I'm thinking that the fractals from frequencies could just blow us away with an advantaged understanding we never might imagine.

    I'm a mathematical moron; true. I saw fractals for the first time and appeared to go catatonic with wonder and wondering. Y'all seem to know your way around them and I wonder if any of you might be onto this already. I'd love to see what there is and anything future-held, too.

    Please respond knowing I'm grateful for folks smart where I'm pretty lacking to play with. Have an excellent 2013; the year that wasn't supposed to happen! Thank you, Marg

    ReplyDelete
    Replies
    1. I called my company Fractal Design for several reasons. I was well-versed in fractals. I am a mathematical generalist, so I can see that the fractal concept crosses over into several areas. I like to model real-world things. And thus fractals are a very important tool for people like me.

      It is remarkable how many mathematicians are also musicians. I think music is a way to round off the sharp corners of a rational mind. And it is a good emotional release.

      I have tried fractal music in the mid-80s and found it rather uninteresting. However, much music satisfies the self-similarity requirement of fractals. Note that very few musicians (outside of J.S. Bach in Art of the Fugue) really used the same theme at different time scales. Usually there is some variation in the theme's expression, as in the romantic era's theme and variations. the repetition of a theme in music is rarely literally copied. There are usually some changes required to make the theme acceptable to the ears.

      I suspect that it is possible to create a kind of fractal based on discrete events or elements that is more amenable to the expression of music. I bet we haven't even scratched the surface of this kind of fractal.

      If we ever hope to write the user's manual for the universe, we will have to conquer its mysteries. This means understanding how various fields interrelate, like gravity, magnetism, and the two nuclear forces. For instance: how does gravity propagate through normal space? We know that it propagates at the speed of light and so there must be some medium in the vacuum of space that supports it. This is the dark energy concept: the energy in normal space that is always there, matter or not.

      These discoveries may be a ways off. We still don't even have a closed form for the fields that surround our own bodies, for instance. The qi concept.

      I think the universe is naturally built from pattern and disorder; both concepts interest me. An extension of the Texture application into three dimensions, for instance, could take a lifetime to write and several to explore. For instance, trees could be expressed as three-dimensional patterns, or worlds.

      Some food for thought!

      --Mark

      Delete