Lose one's marbles lose one's faculties or common sense (informal, recorded from the early 20th century).
Marble Arch a large arch with three gateways at the NE corner of Hyde Park in London, near the site of which Tyburn gallows once stood. Designed by John Nash, it was erected in 1827 in front of Buckingham Palace and moved in 1851 to its present site.
pick up one's marbles and go home in informal North American usage, withdraw petulantly from an activity after having suffered a setback.
See also Elgin Marbles.