NEW YORK—A judge's rejection of a settlement between federal regulators and Bank of America Corp. over controversial bonus pay highlights concern among insurers and policyholders that lavish executive compensation could fuel substantial claims against corporate directors and officers, observers say.

On Sept. 14, U.S. District Court Judge Jed Rakoff rejected a $33 million fine the Securities and Exchange Commission proposed to levy against Bank of America in connection with $3.6 billion in bonuses paid to employees of Merrill Lynch just before it was acquired by the Charlotte, N.C.-based bank. The SEC said a financial report from Bank of America misled shareholders into believing bonuses would not be paid, but it declined to sue bank executives, citing a lack of evidence that their deceit was intentional.

But Judge Rakoff took the unusual step of rejecting the settlement and forcing the case to trial, calling the settlement unjust because the fine would have been paid by the corporation and its shareholders, the alleged victims of the wrongdoing, rather than by executives. The development has added to concerns among D&O liability attorneys and others that, with heightened public awareness about executive compensation, the issue could produce significant D&O liability claims.

“This is the kind of stuff that keeps me awake at night,” said Steve Shappell, managing director of Aon Financial Services Group in New York and Denver.

Mr. Shappell and others say plaintiffs attorneys already have sent many companies “demand letters,” insisting corporations investigate the compensation package of certain executives. Experts say such letters are a legal precursor to shareholders filing a derivative lawsuit on behalf of the corporation against its own directors and officers.

“We're going to see a substantial spike in financial executive compensation cases,” Mr. Shappell said. “This is perhaps the beginning of a trend—short-term, hopefully—of executive compensation being a big issue for the plaintiffs bar.”

Companies already are spending millions of dollars responding to “dozens and dozens” of demand letters and are bracing for derivative litigation to follow, said Ann Longmore, New York-based executive vp for D&O with Willis HRH's executive risks practice.

Such derivative lawsuits likely would allege that the company lost money in the form of exorbitant bonuses collected by certain executives and that directors and officers breached their fiduciary duty to properly manage compensation packages. Executives forced by a settlement or judgment to pay back bonus money likely would not be covered by insurance, under the premise that they are paying back what never rightfully was theirs.

But defense costs for a lawsuit against directors and officers typically would be covered under a D&O liability policy's Side B coverage, Mr. Shappell said. Settlements or judgments in such derivative suits typically would be Side A claims because they often are non-indemnifiable under corporate and state laws, experts say.

Many D&O insurers have written coverage broadly at competitive prices because they believed the chances of derivative suits were remote, Ms. Longmore said.

“Right now carriers are braced for what this might mean,” Ms. Longmore said. “They'd thought that with the Disney decision a few years back, that it'd kind of put these questions to bed.”

In 2005, the Delaware Chancery Court found that directors for Burbank, Calif.-based Walt Disney Co. did not breach their fiduciary duty in awarding a $130 million severance package to former President Michael Ovitz in 1997. The Delaware Supreme Court upheld the ruling in 2006, finding the board made a rational business decision.

But, Ms. Longmore said, much of the new derivative litigation over executive compensation could take place outside of Delaware. She also said it cost Disney and insurers tens of millions of dollars in legal fees to win the case.

“From an insurance carrier perspective, tens of millions of dollars to get to a win is not always a "high-five moment,'” she said.

John McCarrick, a New York-based partner at Edwards Angell Palmer & Dodge L.L.P., said courts often are reluctant to second-guess the business decisions of boards—even on decisions that look foolish in retrospect—as long as the board deliberated appropriately. He said the claim that directors breached their fiduciary duty because of conflicts of interest or self-dealings—for example, approving an executive's compensation as part of a package that included their own compensation—is more likely to succeed.

But Mr. McCarrick said he expected the increased attention on executive pay to prompt regulatory actions by the SEC and state attorneys general more than shareholder litigation.

Brokers say executive compensation-related litigation is exactly the type of exposure D&O liability insurance is intended to cover, so underwriters could never exclude it. But, Mr. Shappell said underwriters across the board are asking more questions about executive compensation.

“Is this going to allow insurance companies to charge more premium?” Mr. Shappell said. “I think the short-term answer is no...(But) if we have a substantial spike in frequency, that translates very quickly into the insurance marketplace and the price of D&O insurance.”