When you look at cartoons they have a distinct art style to them. One of the main points of the art style is the black lines around objects. In reality there are no black lines outlining everything but for this style it is used. This begs the question how do you detect if a fragment is an edge or not. Well have no fear math is here. To start us off we have to acknowledge that this is a post processing effect and it comes after you have shaded your scene. For my example i am doing cell shading for my lighting and making sure that i am rendering the color output and the normal output for the scene as this will be needed in the effect. Along with the normal map for the scene you will need to get the depth map of the rendered scene. This is done with Frame Buffer Objects and should be handled accordingly. Once your first lighting shader outputs the scene color, normals and depth textures you can start working on Sobel edge detection.
In your fragment shader you are going to want to make two functions that will do the Sobel algorithm. the algorithm is based on the current fragment and sampling all other pixels around it. You check in the vertical and horizontal to see if passes a certain threshold. If there is enough of a different between the fragments then it is an edge else it is not. That is the Edge detection in simple terms, but what does that look like?
// return 0 if edge and 1 if no edge
float Sobel_Horiz(sampler2D image, vec2 tc){
vec2 ps = pixelSize;
vec2 offset[6] = vec2[](vec2(-ps.s, -ps.t), vec2(ps.s, -ps.t),
vec2(-ps.s, 0.0 ), vec2(ps.s, 0.0 ),
vec2(-ps.s, ps.t), vec2(ps.s, ps.t) );
vec3 sum = vec3(0.0);
sum += -texture2D(image, offset[0] + tc).rgb;
sum += texture2D(image, offset[1] + tc).rgb;
sum += -2.0*texture2D(image, offset[2] + tc).rgb;
sum += 2.0*texture2D(image, offset[3] + tc).rgb;
sum += -texture2D(image, offset[4] + tc).rgb;
sum += texture2D(image, offset[5] + tc).rgb;
float lenSqu = dot(sum, sum);
return (lenSqu < 1.0 ? 1.0 : 0.0);
}
float Sobel_Vert(sampler2D image, vec2 tc){
vec2 ps = pixelSize;
vec2 offset[6] = vec2[](
vec2(-ps.s, -ps.t), vec2(0.0, -ps.t), vec2( ps.s, -ps.t),
vec2(-ps.s, ps.t), vec2(0.0, ps.t), vec2( ps.s, ps.t) );
vec3 sum = vec3(0.0);
sum += -texture2D(image, offset[0] + tc).rgb;
sum += -2.0*texture2D(image, offset[1] + tc).rgb;
sum += -texture2D(image, offset[2] + tc).rgb;
sum += texture2D(image, offset[3] + tc).rgb;
sum += 2.0*texture2D(image, offset[4] + tc).rgb;
sum += texture2D(image, offset[5] + tc).rgb;
float lenSqu = dot(sum, sum);
return (lenSqu < 1.0 ? 1.0 : 0.0);
}