To add this feature in our module we had to:

1) add a new field max_discount in the table salesrule

2) observe event salesrule_validator_process

        <salesrule_validator_process>
        <observers>
            <amasty_rules_model_observer>
                <type>singleton</type>
                <class>amrules/observer</class>
                <method>handleValidation</method>
            </amasty_rules_model_observer>
        </observers>
        </salesrule_validator_process>
3) keep track of all discounts given to item by rule_id (just remember them in a variable)

protected function _limitMaxDiscount($r, $rule, $itemId, $quote)
{
    if ($rule->getMaxDiscount()==0) {
        return $r[$itemId];
    }
    if (isset($this->_ruleDiscount[$rule->getId()])) {
        $this->_ruleDiscount[$rule->getId()]['base_discount'] += $r[$itemId]['base_discount'];
    } else {
        $this->_ruleDiscount[$rule->getId()]['base_discount'] = $r[$itemId]['base_discount'];
    }
    if ($this->_ruleDiscount[$rule->getId()]['base_discount'] > $rule->getMaxDiscount()) {
        $r[$itemId]['base_discount'] = $r[$itemId]['base_discount'] - ($this->_ruleDiscount[$rule->getId()]['base_discount'] - $rule->getMaxDiscount());
        $r[$itemId]['discount'] = $quote->getStore()->convertPrice(
            $r[$itemId]['base_discount']
        );
    }
    return $r[$itemId];
}
4) when apply next discount to an item, check if it is in the range.