(Reuters) — The Chicago Board Options Exchange will pay a $6 million penalty and take "major remedial measures" to settle civil charges that it failed to properly enforce short sale rules, U.S. regulators said on Tuesday.

The financial penalties against the exchange operator and C2 Options Exchange, an affiliate, are the first ever against a U.S. exchange for violating the duty to self-police a marketplace, the U.S. Securities and Exchange Commission said.

Some of the problems the SEC uncovered at CBOE are related to the inadequate self-policing of optionsXpress, a firm now owned by Charles Schwab that was ordered by an SEC judge last Friday to pay a penalty and disgorgement for illegally selling shares it did not own. The firm is considering appealing the judge's ruling.

CBOE settled the case without admitting or denying the charges.

Under the settlement, CBOE must implement a mandatory training program for surveillance staff, hire a chief compliance officer and other compliance staff, and increase its regulatory budget. CBOE also had to retain outside counsel and consultants in a "bottom-up" review of its practices.

In a statement, a spokeswoman said the exchange worked "proactively" with the SEC and voluntarily launched its own "exhaustive internal assessment" of its regulatory and compliance practices.

"All actions either required or recommended by the SEC, as well as those resulting from our rigorous self-review, have been or are now being implemented. This settlement marks a significant step in putting the SEC matter behind us," the spokeswoman said.

In April 2012, two months after the probe was disclosed, CBOE made some changes to its compliance team, hiring former SEC enforcer Alexandra Albright as chief compliance officer and promoting regulatory services division vice president Margaret Williams to the newly created role of deputy chief regulator.

In its complaint, the SEC revealed fresh details about "egregious" conduct at CBOE that went well beyond failing to detect the alleged misconduct at optionsXpress.

After optionsXpress became the subject of its investigation in December 2009, the SEC says, CBOE took the "unprecedented step" of assisting optionsXpress by providing information and even edits to the firm's so-called "Wells" notice submission.

A Wells notice is the name of the document the SEC sends to a firm or individual when it plans to recommend bringing charges. Recipients of Wells notices are given a chance to explain why the SEC should not file a lawsuit.

"Even more troubling, the information and edits provided by CBOE resulted in the member firm providing the commission with inaccurate and misleading information," the SEC said.

"When questioned by Enforcement Division staff about the underlying matter, CBOE failed to disclose that it had assisted the member firm with its Wells submission."

According to the SEC, problems began at CBOE in 2008 after it transferred its oversight function to monitor compliance with abusive short-selling rules from one department to another.

After the switch, the SEC says, CBOE failed to take action against any firm for short sale rule violations, which require people making short trades to buy or borrow stock they don't own to ensure delivery within three days of a trade.

The SEC said staff at CBOE did not have a basic understanding of the rules governing short sales and were ill-prepared to investigate complaints about possible short-selling violations by optionsXpress.

Under federal securities laws, exchanges have responsibilities as "self-regulatory organizations" to monitor and oversee their markets.

"CBOE's failures in this case were disappointing. The public depends on SROs to provide a watchful eye on their exchanges and market activities occurring through them," said Daniel Hawke, the head of the SEC's market abuse specialized enforcement unit that brought the case.

Andrew Stoltmann, a Chicago-based securities lawyer, said the allegations against the exchange are a major black eye. "The regulatory oversight is nothing short of embarrassing for the organization," he said in a statement. "It is the latest evidence that self regulation simply does not work."

This marks the third time in history the SEC has imposed financial penalties on exchanges, and the fourth major case against an exchange since 2011 as part of a broader crackdown.

Last year, the SEC fined the New York Stock Exchange $5 million after it allegedly gave certain customers an "improper head start" on trading information.

Then in May, the SEC imposed the largest ever financial penalty against an exchange in its case against Nasdaq OMX , in which the SEC alleged the exchange operator made missteps in Facebook's initial public offering.

In those prior cases, however, the charges stemmed from misconduct on the business side of the operation, as opposed to the regulatory side in the CBOE case.