The site had housed the Hospital de la Misericordia in Spanish times, and later a Debtors’ Prison, but was now derelict, and on it Aaron built an elegant three-storey house for himself and his family. Later additions, to the North especially, have ruined its symmetry, but many original features remain in its interior, and the Mayor’s Parlour was lovingly restored a few years ago by the Gibraltar Heritage Trust.

Aaron was a hospitable man: North African notables spent months at a time in his house, and in the 1820s he opened his doors to at least twenty liberal refugees from the oppressive regime of Fernando VII, including General Quirogas and Sir Robert Wilson, who was to return, many years later, as Governor of Gibraltar (1842-48).

Cardozo’s wife had died in 1820, and he spent less and less time in Gibraltar after that, living in Portugal or London, where he died in 1834. By that time, the building, administered by his nephew and heir, Isaac Cardozo, was let to the Gibraltar Garrison Club as their club-house. Over the next few years, Balls and meetings of various societies were held there, including the Scientific Society which later announced the discovery of the Gibraltar Skull.

In 1839, Isaac let the building again. It became the Club House Hotel, and Gibraltar’s leading hotel, run initially by a widow, Mrs Crosbie. Balls continued to be held ther, as well as concerts. By 1844, however, it had deteriorated, and Thackeray, who stayed there, called it “mouldy and decrepit”, but it improved under later owners and housed many eminent guests, including Prince Adalbert of Prussia, who had been wounded when his ship, the Dantzig, had been fired upon off the coast of Morocco.

In 1868 the hotel was leased to John Ansaldo. At that time it was being used by sporting clubs for their meetings, including the Calpe Hunt, the Jockey Club, and the Garrison Cricket Club.

In 1875, Ansaldo gave up the hotel, and Isaac sold the building to Pablo Antonio Larios, a businessman of Spanish origin. Pablo promptly made it available to Queen Victoria’s third son, the Duke of Connaught, who was on a tour of duty on the Rock. The Duke lived there for some months. When he left, Pablo Antonio Larios moved in with his family.

When Pablo Antonio died, after a riding accident, his son Pablo inherited the building, by now known as Connaught House. Finding it too small for his needs, he erected an extension to the north, which in its present form gives the building its lop-sided look.

In 1920, Larios, by now the Marques de Marzales, had fallen on hard times, and sold the building to the Government for £39,500. The intention was to convert the building to a general post office, but funds were lacking, and only the ground floor housed a parcels post office.

In 1924, the first City Council was elected, and Cardozo’s building became the City Hall, by which name it is still known, although the City Council has long given way to the Government of Gibraltar.