Lady Ffolliot of Four Oaks Hall died in 1744, and the Hall was sold to Simon Luttrell of Luttrellstown in Ireland. Simon Luttrell was an ambitious politician in need of an English country house, and he settled at Four Oaks Hall with his wife and eight children. He was a courtier, created Baron Irnham of Luttrellstown in 1768 and Earl of Carhampton in 1785. He was a Member of Parliament from 1755, and spent most of his time at court, where his rakish behaviour earned him the title “King of Hell”.

Simon Luttrell’s daughter Ann was a beauty. In 1765 she married Christopher Horton of Catton Hall (a few miles north of Tamworth), where her portrait still hangs. By 1768 Ann was a widow, described by Horace Walpole as a young widow of twenty-four, extremely pretty and well made, and remarkable for the great length of her eyelashes, which veiled a pair of most artful and coquettish eyes. The Honorable Mrs. Horton attracted the attention of Henry, Duke of Cumberland, younger brother of King George III; their marriage in 1771 caused a scandal, as the king only heard of it a month after the event by a letter which he received from his brother, saying that he was married to Mrs. Horton, and had gone off with her to Calais.

Lady Ann, Duchess of Cumberland, soon became a leading society hostess at her own house in Hertford Street, Mayfair, and a favourite of Queen Charlotte. She suffered a shattering blow in 1799 when news reached her of the death of her elder sister, Elizabeth, at the age of 62. Lady Elizabeth Luttrell had been a constant companion to Lady Ann, but, not being a beauty, was not so popular, “being a coarse, unprincipled woman, devoured by a love of play”, according to one source. Gambling brought about Elizabeth’s downfall in 1797, when she only escaped prison by paying fifty pounds to get married to a baker; she then went abroad to escape the disgrace. A scandalous report circulated that she was convicted of picking pockets and condemned to clean the streets of Augsberg chained to a wheelbarrow, and poisoned herself.

Four Oaks Hall was sold to Dr. Gresley in 1778, putting an end to the riotous house parties which had been held there; one such party features in a play by W.D.Scull published in 1897 entitled “Bad Lady Betty”.