Follow by Email

Wednesday, February 1, 2012

Where Do Ideas Come From?

Sometimes I have too many ideas. The problem becomes: which ideas might pan out, and how do they interrelate? But for most, it's more a question of: where do ideas come from? Let's talk about the various places.

Spontaneous Head Combustion

Sometimes an idea just pops into your head. I think this is because brains are always processing in the background. I had a calculus teacher in high school, Mr. Pearson, who said that the solution to a particular differential equation he had been working on would sometimes come to him while shaving.

Though that was a fairly nerdy thing to say, he wasn't far off my point.

The process of spontaneous idea generation is related to why we dream: the brain always wants to exercise in periods of disuse. When shaving or driving, your brain is not fully engaged. This means there are cycles that can be put to use. We will return to this later. But some people dream in ways that are connected to stuff they are working on. During my most productive periods when I get a cool idea and then rush to implement it (especially a very complex one that will probably take several days of constant work) I am told that I sometimes talk in my sleep: I'm mumbling variables and data structures. This definitely happened to me when I was working on Mosaics in Painter.

Hmm. Maybe I work too much!

But a simple way to use this is to think about a problem before you go to sleep, and your mind may come up with a solution while you sleep. In fact, I used to listen to music in my head right when I was going to sleep. I liked to listen to Beethoven's 9th Symphony in this state. Not really listening, per se. Actually just hearing it in my mind, every note and phrase. It was great practice for musical composition.

Sketch It Up

I like to sketch things out, and get ideas that way too. I guess it's because I'm a visual person. But, for me, lots of graphical ideas work themselves out on paper. A doodle can quickly become something larger.

I was working on borders and corners one day when I realized, as many people have, that you could represent the corners as four image pieces and the edges as another four image pieces. And the edges could be stretched or repeated. So I drew one of my favorite things, and I have reproduced it at left. I suppose it's a good example of three-dimensional thinking.

I draw in guide lines that showed the corner pieces and the edge pieces succinctly. Then I realized that these corners and edges could be filled with pretty much any image.

So i drew cylinders with pipes interconnecting them, like you see at right here. I realized that even shadows could be part of these clipped images. Note: this 3 x 3 partitioning is used to create buttons and other UI elements, also. In fact, that was what I was using it for at the time.

But I didn't want the UI to be constrained to be a rectangle. No matter! These pieces will still work, as long as all the corners are squarish. For instance:

At right, I have shown something that can be made out of the usual eight pieces, just rearranged in a different fashion from the typical arrangement I have shown so far.

This goes to show that it doesn't matter how you design it. If you take a little care, that design can service a much larger class of objects.

This design really hinged on two things: the corners and the edges. So I concentrated on the corners. What could the corner possibly be? A number of three-dimensional things popped into my head, and I sketched them up.

Here are a number of possibilities that occurred to me, drawn as corners. To left: a gabled approach that exhibits shading and has pyramidal corners. To right: that same approach, but with different edges. The edges can have their own articulations as well.  There are plenty of three-dimensional things that can grace the corner of a design.

Here, I explore the use of cubes and balls. To right I considered using a stake at each corner and a taut piece of rope in between. And knots on the rope in between the corners.

Of course, there are a number of articulations on the edge that can be used in addition to knots and spikes. Really, any repeating line pattern can be employed along the edge.
Here are some line patterns: wavy lines, beads, and twisted rope. Of course braids and yarn motifs are useful as well. But plain patterns, like stripes and diagonal "candy" stripes are also useful.

Exercising Your Brain

When I am otherwise not mentally engaged, I use that time for practice. Sometimes I do something tedious like factoring five-digit numbers in my head. You know, I like prime numbers. For instance, today I passed a house on San Jose-Soquel Road that had a number of 24,769. (I look on the right side as I'm driving home and there are often odd numbers like this one, which means they won't be divisible by 2.) I'll cut to the chase and tell you that it's 17 x 31 x 47. It took me about 4 minutes to determine that, performing long division in my head. Then, when I got home, I checked it using Siri.

It may be tedious, but it does help to keep my brain sharp when it comes to numbers. Another activity I do while driving is writing songs. I compose the melody and lyrics simultaneously. When I get an idea, I record it into Voice Notes, an iPhone app. I will go over the lyrics a few times and record a verse or a refrain.

This process keeps my creativity working while I would otherwise be bored. Exercising the right side of my brain. I used to turn on the radio while driving. I don't do that any more. I let my mind fill the silence.

I also got a great idea for a novel while driving. So I used Voice Notes to get it all down.

And, by the way, it might be a good idea to pull over, if you get too involved in an idea while driving!

Know Your Subject?

When I am wracking my brain for ideas on a particular thing, I can easily come to a standstill. It is at moments such as this that I take a step back. Specifically, this is called meta-level thinking. This process can actually be impeded by knowing too much about a subject. You can get into the mindset of thinking that it's all been said and done.

When writing a novel, it can be as simple as taking a step back and considering a character's arc. When writing music, this can manifest itself as rearranging the structure. Or adding in a full break or another device that so often makes songs interesting to our ears.

The Beatles practiced this. In We Can Work It Out, for instance, there is a section that breaks into triplets in an otherwise foursquare piece. On the back side of Abbey Road, they ran all the songs together into a single homogenous piece, a structural device.  In Getting Better, one of the verses becomes one long melody instead of several broken up melodies. On All You Need Is Love, the verses contain an extra half-measure per phrase, which helps to offset the otherwise foursquare melody (the Beatles called it the Dirge), the device of changing the time signature.  Or just having two authors can create a cognitive dissonance that can make a piece remarkable. On I've Got A Feeling, arguably the last composition that Lennon and McCartney wrote together (unless you consider the back side of Abbey Road to be a single piece), their contributions are quite different, and improve the song immeasurably. It goes on and on.

When programming, it helps to look at the problem in a different way. This can be a meta-level approach when you look at multiple ways of solving the problem, or you find a way to combine what would otherwise be different methods. This is particularly good when there is something you are missing.

Sometimes it's just a matter of employing multiple modes of thought at the same time.

Ideas Travel In Groups

When finding another person that is likewise creative, it is often possible to have a huge idea session. This has happened to me countless times with various people, including Tom Hedges and John Derry, the co-authors of early Painter.

A single session can generate hundreds of good ideas, I have found. When you get onto an interesting or fruitful subject, the result can be a veritable treasure trove of ideas.

This may be the Lennon and McCartney effect: the influence of two brains on creation is synergistically better than if the two created separately.

I have experienced this effect with John Derry on several occasions. Together we thought up Painter's image hose, for instance.

Combining Expertise

When you become an expert on one subject, it can help to give you the ability to solve problems in that subject. But is thorough research a ticket to stagnation?

I have found that it helps to be an expert on several subjects and then to combine the knowledge between the subjects. I'm a pretty good programmer, for instance, and I know how to draw. Combining them produced Painter.

Anyway, it's hard to ignore the profound influence of general thinking: knowing a reasonable amount about several subjects and thinking about how they relate.

Grist For the Mill

I'm not sure why it is, but I am equally sure that this is true: when we have emotional experiences, we are driven to create. This is typified by Lord Byron clutching his forehead at the top of the stair: the tortured artist. Or by Michelangelo undergoing great pains to complete the Sistene Chapel's frescoes: the agony and the ecstasy. Is it a cliché that only through pain can we create?

No, I don't think that it is. After hitting a patch of black ice and crashing my car in Denmark on the night of December 28, 1995, I sat down and wrote 7 really good poems. Sure, I write song lyrics, but poems? I was in a rare mood that evening after some wonderful "Good Samaritan" Danes stopped to pick me up and kindly delivered me to my Hotel in Copenhagen. Note that this near-death experience is not chronicled in my post Handling Serious Events. It seems I have more than my share of such events.

But, seriously, how many love songs are there? How many my baby done left me songs are there? I tell you: there's something to this theory of emotional distress and creativity. It's grist for the mill.


  1. Man right on the mark, Marc! I have so many ideas just out of the morning shower, or even more so while on a 5 mile aerobic run (full court press of the gray matter works wonders some how).

    A famous fantasy artist friend of mine talks about honing the blade of his imagination by feeding it raw goods, inspirational material. He refers to it creating a large bin of index cards upon which to connect the dots between. As I get older, I can really relate to this as this large collection allows me to often look and see potential where others look and find none. It is not an IQ thing, but a lifelong focus on collecting maternal upon which to draw from.

    I have very much enjoyed working with John Derry over the past couple of years, much as you have noted in your posts.

    Thanks for sharing!

  2. sorry for the typo on the name.

  3. No prob Jerry.

    Ideas are of course part and parcel to any product. Look for Texture, Part 2; coming soon. It will show the importance of being able to operate on textures in the frequency domain.

    Most of this stuff I had forgotten because it was literally filed away in the garage. But when I got it all out recently, I found that there was so much cool stuff that I couldn't in any good conscience continue to keep it completely private. And it reminded me of a few stories...

    John's a trooper, no doubt.

    There are too many things to do from day to day. Too many ideas to act on, usually.

    Maybe the hard part is picking and choosing; deciding which to implement.

  4. I used to squirrel away ideas so I would always have something to fall back on, but now I just can't see myself having time to act upon all of them. So I share them, but often without this sort of full time background search for a solution to overcontrained problems, others can look but not see if you know what I mean. So then you have to overcome the seeing is believing stage. This can be highly distracting, as you probably know sometimes when there is pressing work, letting a cool idea of the leash can be a joyful but all consuming thing. Sometimes it is infuriating that demos are required to turn the tide, a visual aide to the mentally myopic, unfortunately part of overcoming the human condition.

    I have had many of those "where did those last miles just go" moment while driving and letting ideas off the leash. As you noted we don't multitask very well. It is actually why I run around a trail in my subdivision instead of on the streets, because I completely forget about my surroundings, and immerse myself in some visual space in pursuit of solving a problem. Odd how many times I have come up with something really cool 2 miles in and panic wondering if my 51 year old brain can keep in the short term pile until I get back. I have cut a few runs short due to such things.

    Sometimes you can relate to Micheal Jacksons statements about not being able to get songs out of his head, and similar statements from others that had big breakthroughs after insane bouts of work in isolation (iteration). I often wake up and can't go to sleep because that idea I planted before nodding off pops up front and center.

    So Mark I end up using lots of metaphors to draw people into a familiarity with my ideas, but even then sometimes it can be a bit like doing the "jack and jill" version to make a sales job. I have to say that is what I liked about working in a small company, as I could just let it off the leash and run without permission.

    Funny about the iBook/iPad thing and education. As you have noted, the first canvas for many an artists is the margin of a textbook. Creative pros are born there. I hate to see that go away in a search for the best way to educate a fictitious average student. Words are a recent invention, sortof the first compression technique, unfortunately with a one to many mapping that leads to ambiguity. I don't think about image processing as a string of greek letters, but in 4 space. It is a far more intimate association, as you have alluded to in your recent posts.

    Keep up the great work, I wish I had the time to blog as it might be an outlet for those ideas that don't quite have a home.

    1. Jerry.

      I'm looking at Texture now and it is kind of liberating to be able to revisit this after so many years.

      I'm a firm believer in the "others can look but still not see" aspect of ideas. Some of us have experience and enough knowledge to see the usefulness of a bit of knowledge. Others have different motives.

      Demos are required to get people on the same page, or even within the same chapter. And demos are all about the marketing, so we get good at that too. Still, you may have to convince finance, so you need to be able to develop a mini-business plan so people can see the light.

      But to me, ideas are their own motivation. And, in another sense, I developed a innate sense of what could be useful to the artist, and simply motivated myself to appease that sense. Early at Fractal, bringing on John Derry was an admission that I was on the right track but that I needed professional help. John had the demo chops as well as the artistic sense and the broad knowledge of the struggle to move traditional media onto the computer.

      When I have an idea in the wild, I used to use a notebook. I have dozens of full notebooks here (and its making a mess!). Nowadays I either use my iPhone or my iPad to take down the idea. Sometimes its just a voice note.

      Friends have turned me on to idea collection programs like Evernote and Penultimate, also.

      At Fractal Design, I used to get in trouble for taking one of the ideas and running with it. This left Tom Hedges and others to finish (debug, clean up UI, handle things like undo) my work. There was some resentment. But, as CEO, people trusted me to do what's best. With that responsibility, I almost always knew the right way to go. With projects like X2 (layers) and X3 (3D paint on power PC) I usually could convince others that I was on a righteous, holy quest. And I made hay with it when it came time to show new stuff to the crowds at conferences, MacWorld, and internationally as well.

      Luckily, responsibility and the requirement to do what's right has served me well in the big company also. The idea, the tangent: it must pay off and in a visual way.

      And by the way I wouldn't worry about a student drawing and sketching. They will find an app for that. If schools proscribe that kind of dreaming it would be a crime, for sure. And its all in the art of taking notes, anyway. If you don't take notes, you don't retain. So I can't see drawing going away anytime soon. I doubt that books can replace teaching.

      Ideas and the mind: I will have to post on this concept. How ideas are stored in the mind and how we "see" them. When implementing, I just "know" that something would be cool. I know exactly what kind of tool I can build that will enable me to get much, much farther in development.

      When I develop, I create my own tools. And this enables me to create things much faster, and with much greater certainty and accountability. I'm not talking about writing a text editor. But when I do a particular complex process, I can visualize the intermediate buffers directly on screen and in the proper ways to see what is happening and where bugs may lie.

      That becomes increasingly difficult when it comes to GPU work.

      But, as in a demo, when they see what I'm up to, they immediately get it. So there's a bit of marketing in the process as well. And it helps keep me accountable. There's nothing worse than a manager who has to say "I just can't quite figure out what he's doing".

      And, blessed with a high degree of intelligence (from my manager) and the right kinds of tools, I can simultaneously avoid that situation and also make fast progress. And, it makes it easier for a manager to throw ideas into the mix, which I always appreciate.

      For me, blogging is effortless because I have had a good teacher. I have a sense of what to write as well, since I first take notes on what I want to say.

      This blog is about 70% archeology as well: digging up old ideas from the past.


    2. I have no problem the demo part, and totally grok the need for them. Once I helped a world class track coach attempt to pen a book. His native language was portuguese, and he had a way of categorizing training that at first struck me as odd. Now it makes frames a sharp point about the particulars of being human and adaptation to a stimulus. He would refer to weight training as resistance of weights, pool training - resistance of water, interval training - resistance of speed. So when you have so many cool ideas, sometimes it is daunting to realize the scope of what is involved to deliver those ideas given the runway left before you. Sort of the resistance of the imagination when in full flight. As an inventor you have to overcome and account for that cognitive friction new ideas will encounter.

      As far as weeding out bad ideas, pain is natures way of making an idea sink into long term memory fast. Having experienced the result of working on things that interested me but not customers, I must say nature has seared that the desire to focus on things that will be adopted into the gray stuff pinned between my ears. Funny how some lessons can't be taught, but only learnt via a dose of pain.

      In any event, it is a great position to be in. And as the commercial goes stay thirsty my friend.

      BTW, I love thinking about the benefits, and designs around a gpu, especially when framing where no man has gone before scenarios versus simply speeding up old school ideas. I can't wait for grand unification of address's spaces;-)

    3. Of course, I have been doing GPU work since 2004. Now, OpenCL is my preferred approach, because it can handle a much greater set of computational topologies.

      There will always be faster and slower memory. Local memory, shared by a set of GPU work units seems to be the current way of looking at memory and bandwidth prioritization.

      I like that OpenCL can operate on CPUs and GPUs both, giving me some degree of portability and persistence of code.

      Cognitive friction - that's a nice concept!

  5. There is a theory I remember from my study of AI, that the brain uses a combination of gradient search for local minima of an N-dimensional solution space, and then us simulated annealing to converge on the global minima. When we do something unrelated or rest, it pops our solution engine out of the rut at a local minima, and randoming starts walking downhill from a different point in the solution space.

    1. I doubt brains are nearly as ordered as a Monte-Carlo gradient search. When inactive, it's just looking for something to do. My hypothesis is that our brains are motivated to work on whatever it obsesses on. If you are used to thinking about it, then your brain will just continue to do it.

      Like if you press your arm against a wall. Hold it there long enough and, once you move away from the wall, your arm magically floats up. So, likewise, your brain is in a notch thinking about something. And it automatically falls into the same notch once at rest.

      When it comes to dreamed solutions, I believe the process is similar to taking a step back and using meta-level thinking. One thing is sure: dreams are not random. Our own brains make them.

    2. The following is quite speculative on my part. The trail to answers for the puzzle solving during sleep start just above the nose. We have two forward facing eyes that image in stereo. We have incisors all the more to chew you with my dearie;-)

      My jack russell is a riot to watch as he sleeps, he growls, he digs, he barks, not unlike what Mark described. I do the same thing, sometimes even while awake my wife will inquire who I am having a conversation with at work, as my lips are moving (no phone is in hand;-). My Jack Russell will spend the entire day chasing down squirrels, and digging for moles. His nose down in a hole, and that little clipped tail just a wagging in the air. Same thing with my cat, which delivers uneaten squirrels, birds, and lizards upon my doorstep on occasion. Quite often I can catch him, steeped down, moving forward stalking some bird, and then with a quick twitch on the hind quarters lurch forward. You don't have to watch much animal planet to realize most of a young predators waking moments are spent honing hunting, or pecking order skills. That fact that our species is still here given our lack of speed and agility, point us on the right trail. Why do sports phycologist hypnotize clients to have the visualize the perfect throw, catch, tackle, or jump? As we know now practice makes perfect, and it does not matter that much whether it is mental or physical as both rewire the brain. So what is safer, more energy conserving, practicing for real, or simply mentally? What keeps the cat engaged in what is mostly a long odds situation, how about the jack russell who rarely catches anything? There is joy to be found in the stalking, a bit of dopamine I would guess. It is a reward system for something as unrewarding as stalking prey and mostly coming back empty handed.

      As Mark points out if you create a physical mental model of a virtual problem domain, your brain has an easier time with it. My guess is we are simply hijacking old wiring, and substituting some ancient prey with with an inanimate object made real to the mind via visualization. By being the geeks that we are, and by a laser focus, our brains stalk that solution to some image processing puzzle. We ache to put corner the solution and stick a spear in it. We anticipate that feeling. I hate to say it but this displaced behavior probably leads to our booming economy, where almost everyone finds more joy in the shopping than the actual item delivers once it is brought home.

      The best human interfaces tap into this behavior, imo. We keep the focus on the visualization of the modern prey, not on the cognitive or mechanical friction of using the combination of hardware and software. This gets right back to what Mark said about driving. We don't multitask very well, in fact we have a built in interrupt mechanism (brain farts) which experts tend to think is a survival skill to look about on occasion to spot prey or predators while engaged in some narrowly focused head down task.

      Fun stuff to ponder!

    3. Yes, I think I also talk to myself when I am engrossed in code. Any number of people have found it annoying from time to time. It seems to be automatic.

      Often programmers personify their code. It becomes an agent of their personality. They talk about "I'm doing this" when it's really their code that does the work.

      Lots of programmers enjoy the FTW moment when it all comes together. I can think of several occasions, after writing a fairly complex program, when it worked the very first time I ran it. And that's a rush. Like a hole-in-one.

      Arguably, rational thought is all about predatory instinct.

      I worry about people who say we need to evolve. Who don't realize where we came from.

      I worry about people who think first contact with aliens is going to be a good thing. I agree with Stephen Hawking on this particular subject. Because evolution is all about survival.

    4. I once joked with our former plenotics expert who often recited specifics about the evolution of vision, we would know by the placement of their eyes if they came in peace or not (stereo vision - not so good - run for the hills - pano vision - break out the salad buffet and a sigh of relief). He also mentioned that the reasons lizards do those crazy pushups is to fake depth images from a two camera perspective;-)

      I have also heard that they doubt aliens would be that different from us, meaning it takes too much glucose to fuel bipedal, and stereo vision/audio. The brain is the largest consumer of glucose per weight of any organ. It is an amazing cheating supercomputer with a built in realtime healing brush.

      Other fun topics....

      Software as a dopamine drip (what road did the iPod start a journey upon)?

      Identifying your inner squirrel (don't be to amazed by the hoarders, as we all hoard it is a survival trait that has served us well, but is also now displaced ) Mark, you and I hoard random ideas about image processing, and graphics.

    5. If I had more than one working eye I would have made a great predator. As it is, the lack of visual processing I'm my left eye makes it possible for my right brain to use the extra cycles for something else, I suspect.

      I am a hoarder, Jerry! Particularly when it comes to harvesting my own ideas. I write them down so I can use them later like a squirrel stores nuts.

      And all the notes I keep are making •me• nuts!

  6. Sorry about all the typos and poor grammar, as I often shoot from the hip and knock off a toe;-) My brain never learned to stalk text or grammar, guess it was not on the Serengeti being stalked or stalking my predecessors.

  7. Pick a very difficult problem that you can't readily solve, such as "what is dark matter". Think about it nonstop until it is solved. Can't always be done. Sometimes we need to take a break and do other things for a while. Or we need to add more information to our knowledge set. The mind is searching its available experience and knowledge space for the solution. Sometimes randomizing that process a bit, yields discovery.

    1. When I get stumped on a problem, I usually just go play piano.