Lord Fife was the father of three children by a Keith woman of humble origin from whom they were taken at an early age and placed under the direction of William Rose, Fife’s factor. James was a handsome youth on whom his father lavished affection, making him his constant companion, purchasing his advancement in the army, and giving him a small estate in Aberdeenshire and voting qualifications in Elgin and Banff. In 1779, when Fife’s friend James Harris, minister at St. Petersburg, was created K.B., Duff acted as his proxy at the installation, and for that purpose was himself knighted. His grandmother Lady Fife thereafter referred to him caustically as ‘the nominal knight’.
At the general election of 1784 Fife gave up his Banffshire seat to his son, and himself stood for Elginshire. From Fife’s uncertain politics and Duff’s connexion with the Duke of York,
Nominated on 29 Mar. to the select committee on the Nairnshire election, he inadvertently caused a discussion on procedure by absenting himself through a misunderstanding. His apology to the House is his only recorded speech.
I was never more distressed than I am at present respecting my conduct in Parliament. The Prince of Wales has sent to me, requesting I will give him my support in case of any attempt of restricting him of his natural rights. I have answered that I will not give my countenance to any measure where he is personally concerned. I am afraid Lord Fife will act contrary to this, but it is impossible for me to give offence to a young Prince coming to the throne, under whose protection as a military man I so immediately stand. I shall act in this ... to the best of my judgment and abide by the consequences.
In a further letter Duff described the dispute with his father:
On my informing him of my difference of sentiment on the present state of politics, my disapproving of Mr. Pitt’s conduct, and resolution not to take any active part in opposition to the Prince of Wales, every art of persuasion and flattery of which he is master were put in practice to dissuade me from it, but, if I could not bring myself to vote for Mr. Pitt, imploring me, as the greatest service I could do him, to resign my seat. I took two days to consider on the subject of our conversation. I then wrote him that, anxiously wishing to do everything in my power I consistently could, I, in compliance with his request, was willing to vacate my seat.
Despite his father’s ‘numerous professions of friendship and affection’, Duff was not surprised when the following day Fife deliberately avoided him in the street. All communication between them ceased; and Fife immediately nominated James Ferguson of Pitfour for Banffshire. Duff wrote to Rose: ‘I could never have thought of holding a place of that nature when I was totally debarred from having an opinion of my own, not even where I think my interest is concerned.’ After the King’s recovery Duff, at Rose’s suggestion, wrote to Lord Findlater justifying his parliamentary conduct,
Cordial relations being now restored, Fife proposed that Duff should succeed Sir James Grant in Banffshire. But as Dundas was unsympathetic,
He died 5 Dec. 1839.
