(Reuters) — Royal Bank of Scotland P.L.C. will pay $5.5 billion to settle one of two major U.S. investigations into allegations it mis-sold mortgage-backed bonds that it needs to resolve before the British government can sell its shares.

The Edinburgh-based bank said on Wednesday it agreed to settle the lawsuit with the U.S. Federal Housing Finance Agency that accused it of mis-selling $32 billion of mortgage-backed securities before the global financial crisis.

Analysts had previously estimated RBS would have to pay between $3.5 billion and $5 billion to settle the case.

"It is never a great experience for a CEO to be writing such a large check ... but unfortunately that is the price we are paying to get this organization into a much better shape," RBS Chief Executive Ross McEwan told reporters.

Mr. McEwan has been trying to clean up RBS's balance sheet and end an array of legal cases so the government can sell the more than 70% stake in the bank it obtained via a £46 billion ($59.6 billion) bailout during the financial crisis.

RBS said it will get a reimbursement of about £581 million ($752.3 million) from other parties and the fine is largely covered by existing provisions.

The bank, whose shares were broadly flat at 1400 GMT, said it will record a charge of £151 million ($195.5 million) in next month's result related to the fine.

RBS has moved closer to resolving some of the major barriers to its recovery in recent months.

This included paying £1 billion ($1.29 billion) to settle with shareholders who claim they were misled before its near collapse in 2008, while the government has backed a new plan to resolve its E.U. state-aid requirements.

No Justice Department update

Mr. McEwan said there was no update on talks to settle what is expected to be the biggest fine in the bank's history with the U.S. Department of Justice over claims it mis-sold mortgage bonds.

Edward Kirth, an analyst at KBW, said in a note to clients RBS will probably have to pay a further $10 billion to settle with the DOJ and other legal cases.

RBS still has about $3.7 billion put aside to resolve lawsuits over mortgage-backed securities, packaged and sold before the U.S. housing meltdown and financial crisis in 2008.

The case against RBS is the biggest and last of 18 lawsuits the FHFA filed in 2011 over about $200 billion in mortgage-backed securities that various banks sold to U.S. mortgage giants Fannie Mae and Freddie Mac.

The FHFA previously recovered $17.87 billion to resolve most of those lawsuits, including $5.83 billion from Bank of America Corp. and $4 billion from JPMorgan Chase & Co.

The FHFA contended that in buying the mortgage-backed securities from RBS, Fannie Mae and Freddie Mac relied on false and misleading statements contained in offering documents, leading them to suffer massive losses.

In September 2008, the U.S. government appointed the FHFA as conservator for Fannie Mae and Freddie Mac after seizing the mortgage financiers. Lehman Brothers Holdings Inc. went bankrupt one week after that seizure.

The British government has said it will not resume selling its stake until the bank settles its U.S. fines and resolves its state aid requirements.

Taxpayers currently face a £29.2 billion ($37.81 billion) loss on the value of the government's shares in RBS, according to the Office for Budget Responsibility, Britain's independent budget watchdog.