Destroy all the bricks by clicking them in groups of the same color. The more you get at once, the higher the score.
The only way to remove a single brick is to zap it with a magic wand.