"The bid did not attract minimum shares needed to succeed so we are going to cancel the bid," Caryle said in a statement.

Carlyle teamed up with JAG's chairman and CEO Tetsuo Yamashita in November to launch a management buyout. But the activist fund backed by veteran investor Yoshiaki Murakami made a counterbid.

JAG shares surged following Carlyle's bid, and increased further after City Index Eleventh, the fund backed by Murakami, flagged a counterbid in mid-January, prompting Carlyle to raise its offer.

Carlyle's revised offer included 33 billion yen for JAG shares and a 15 billion yen cash payment for JAG businesses that Carlyle had planned to buy. The scheme also allowed the CEO to get control of the rest of JAG businesses for 2.5 billion yen.

City Index Eleventh, which built up a 20.5% stake in JAG, last week launched its own bid to buy the remainder at 1,210 yen per share, topping Carlyle's offer of 1,200 yen.

The investment arm had said Carlyle's offer was too low and allowed the CEO to gain significant benefit at the expense of minority shareholders.

JAG shares closed at 1,270 yen on Wednesday, more than double Carlyle's initial offer of 600 yen.

City Index Eleventh's bid will run until March 22.

($1 = 104.5800 yen)

(Reporting by Junko Fujita; Editing by Stephen Coates)

By Junko Fujita