Proper subject extraction, background removal, and background painting

[ofnuts]

Beginners very often wonder why a faint border remains around sharp items such as letter or logos when they remove or bucket-fill the background around them.

The explanation: when a border isn't a straight vertical or horizontal, it will look jagged /pixellated, if done with plain pixels because pixels on a computer display are big enough to be visible. To avoid this "anti-aliasing" is used: instead of having only two colors for the pixels (subject color or background color), pixels along the edge of the subject are made partially colored or partially transparent:

   

When you wand-select the background or bucket-fill, you specify a threshold that says how close are the colors you want to select. If the threshold is low, you don't select the anti-aliasing pixels, so you leave a faint border uncolored or uncut:

   

But if you use a high threshold, you apply the color (or select) bluntly to the anti-aliasing pixels so they disappear and you are left with a jagged border:

   

So, obviously, these anti-aliased pixels are the most precious part of your picture and you should never, ever, use any processing that destroys them. And to do so, there is only one solution: draw the shape on a transparent layer. This way, the anti-aliasing is done using partially transparent pixels... The next post will show why this is is important.

[ofnuts]

Now, if you want the background color to blend smoothly, you just have to paint *under* these pixels, and their partial transparency will do the rest... And since where the pixels are fully opaque the underlying layer won't show, the best way to paint the background around the subject is to add a layer below and fill it with the color:

   

With some combination of colors this may create a slight visually dark outline (it is the case above). This can be alleviated by using the "Levels" tool on the alpha channel of the layer on top, to alter the transparency of the anti-aliased pixels. By making them a bit more transparent (drag the middle cursor to the left), the dark outline disappears before the jagged look sets in:

   

[ofnuts]

Now, the $1000 question is: you have this nice logo already on a colored background, how do you get it on a transparent layer with the proper anti-aliasing?

In other words, how do you replace the border pixels with mixed colors by partially transparent pixels? Well, Gimp has a magical tool for this, which is called "Color to alpha" (the "alpha channel" is the channel, that, next to Red/Green/Blue ones, controls the opacity of a pixel). Here is how you can use it:

First, you use the Magic Wand to select the background, with a low threshold (you will notice that only the palest anti-aliasing pixels get included in the selection):

   

Then you grow the selection by one pixel. This makes it include all the anti-aliasing pixels(*):

   

Then, you apply "Colors/Color to alpha", using the background color as the reference color (in the picture below I put a white layer under the transparent layer to better show the recovered transparency on the border pixels):

   

And you can repaint that underlying background, to show how well this works:

   

Now, all this was done using several layers, but sometimes it's more convenient to work on just one layer. And Gimp also allows this, using two specific paint modes, "Color erase" and "Behind" that are complementary.

- The "Color erase" mode does exactly the same thing as Color-to-alpha, it replaces the painted color by transparency. Bucket-filling the selection in "Color erase" mode give identical results to Color-to-alpha. Like Color to alpha is is best used with a selection that covers the anti-aliasing pixels but protects the rest. In the image below the yellow brush strokes magically produce semi-transparent blue pixels:

   

- The "Behind" mode produces the same visual result as painting on a layer below the active layer. The paint is fully applied, where the active layer is transparent, partially applied where it is partially transparent, and no painting occurs where the layer is fully opaque. In this mode you can just paint around your subject and the painting will magically blend into the opaque parts:

   

(*) The smart reader will ask: why do we grow the selection instead of selecting everything? It's because Color-to-alpha faces an ambiguity. There are several ways to produce a given color, depending on the opacity: a light gray on white can be either:

• a fully opaque light gray
• a somewhat transparent dark gray
• a very transparent black

And color-to-alpha always takes the solution that assumes the maximum transparency (it will choose the very transparent black above). So if you select everything 1) any area inside the object that will have a color close to the background will become quite transparent, and even the whole subject can become slightly transparent. The "grow-the-background-selection" method prevents this. This is also why you should use the Magic Wand to select the background and not the Color selection tool, since the latter could select part of the subject.

[ofnuts]

One last thing... you can of course wonder how you can repaint the subject and keep these precious anti-aliasing pixels partially transparent... it happens that there is something for this called "alpha lock". When this box is checked (see red circle below), the alpha channel of the whole layer remains unaffected by paint operations. This mean that if you want to apply one single color you can bucket-fill the whole subject layer, the background will remain transparent, and the anti-aliasing pixels will get the new color while remaining partially transparent:

   

[Pl3th0r4x]

Awesome! It worked and it looks so good.

Basically you just:

1. Use the wand and select the background
2. Grow by 1 pixel
3. Color>Color to Alpha

Mine was on a transparent layer, and I chose the color white, and I am assuming since the original background color was white, this is why it cleared up the smudges? Well excellent work Ofnuts just what we needed!

[2backbar]

This is not proper extraction, this is stupid workarounds cause fill bucket doesnt do its job and its laughable in 2012 year that nbody took care of it yet.Now people are forced to use lahyers to do simple stuff which takes one click in photoshop, thats just pathetic.