It doesn't actually increase the base damage, it just causes all the affected ships to 'share' damage. It's an empathic effect - a target ship takes regular full damage, and all malice affected ships will take a percentage (up to 25% at level 3 malice) of that damage as well.
It doesn't multiply or boost the damage of an attack, just massively increases the total dps (basically give every ship that fires at a maliced target pseudo-splash damage to the other maliced ships).
It's been described before elsewhere, but I'll repeat here. To use it, basically just use malice, then immediately use cleansing brilliance on a malice affected target (be quick, since malice lasts 20 seconds and cleansing brilliance lasts 8, and you want all 8 seconds done while the targets are still under the effect of malice).
To maximize the effect, make sure your Radiance is already facing the targets (so that it doesn't have to turn first, which takes a good amount of time). Use malice in the middle of an tightly packed area, so as to catch as many ships under its effect as possible. Fire cleansing brilliance at a target which:
has a lot of health remaining (more than 2000 shields/hull left)
has a relatively low shield mitigation max
is in line and behind with multiple other target ships
This way, the target will not die before sharing all damage, and the damage is not blunted by mitigation. Additionally, EVERY ship that is damaged under malice shares the damage, so the narrow AoE of cleansing brilliance multiplies the damage shared depending on how many ships are hit by it.
Someone said that the ideal targets are heavy cruisers, since they have a lot of health, but not as high a mitigation value as capital ships. The main issue is that they tend to be up front, so don't allow much AoE spillover.