(Reuters) — An ex-Royal Bank of Scotland (RBS) employee is suing the U.S. Justice Department and Securities and Exchange Commission (SEC) for records that could relate to a bounty he says he is owed under a post-crisis whistleblower program.

Victor Hong alleges the U.S. agencies "capriciously" and "in bad faith" flouted the law when assessing whether he was due a payout for information he provided to probes into the British bank's mis-selling of mortgage bonds in the run-up to the 2007-08 financial crisis, court filings show.

In August 2018, RBS agreed to pay $4.9 billion to end the probes, which were led by the Justice Department. No portion of that fine has ever been deemed eligible for a reward under the whistleblower program, public records show.

The Justice Department and the SEC have not said publicly why the RBS settlement was not deemed eligible for a potential tipster reward.

The SEC declined to comment while the Justice Department did not respond to requests for comment.

The Justice Department has objected to being named as a respondent in the case saying in a Jan. 30 filing that the SEC was “the only appropriate respondent.”

Mr. Hong's move is an example of the disputes that can result from what from campaigners say is the opaque nature of the whistleblower program, set up after the financial crisis revealed a swath of improper and illegal activity across Wall Street.

The program, which promises cash rewards should information lead to regulatory action, receives thousands of tips a year. Some advocates say only a token number of whistleblowers receive compensation and that the decision-making process over who should be rewarded is shrouded in secrecy.

"I'm sorry to say Mr. Hong's experience is not atypical," said Tom Devine, legal director at the Washington-based Government Accountability Project.

“The bounty has been an excellent boost for law enforcement, but the programs are using whistleblowers rather than rewarding them."

SEC Chairman Jay Clayton has said in the past that the program is sometimes constrained by rigid rules and that a pending proposal to tweak those rules aims to get more money into the hands of whistleblowers.

Since it made its first payout in 2012, the whistleblower program has resulted in more than $2 billion in penalties and $387 million in rewards, SEC data shows.

Currently, the SEC can reward tipsters whose original information leads to a penalty exceeding $1 million, with rewards of between 10% and 30% of the fine.

The SEC may also pay a reward in a related enforcement action brought by the Justice Department or other agencies, provided that settlement was based on the same information the tipster originally gave to the SEC.

The SEC may deny claims for a range of reasons, including that the tip did not offer new information or prompt any action. Mr. Hong has supplied evidence and testimony to other legal actions against the bank, but Reuters could not determine what role, if any, the information Mr. Hong provided played in the RBS settlement.

Over the past 18 months, Mr. Hong has tried unsuccessfully to access SEC and Justice Department records that could shed light on why the agencies determined that the 2018 RBS settlement was not eligible for a reward, according to court filings.

On Monday, he filed a motion with the U.S. Second Circuit Appeals Court seeking to compel the SEC and the Justice Department to produce the full set of records relating to the RBS settlement and the decision to deem it ineligible for a reward.

A former managing director of risk management at RBS's U.S. operations, Mr. Hong resigned after two months on the job after discovering the mispricing of billions of dollars of mortgage bonds.

He first approached the SEC and Justice Department offering information in December 2007, after leaving the lender the previous month, according to the filings.

He later met with Justice Department officials in December 2014, and provided testimony and documents on RBS misconduct, according to the filings, which include copies of subpoenas, Mr. Hong's formal SEC tip, and emails with Justice Department officials.