The seat of the Laird (Baron, later Viscount) of Lochinvar was at Kenmure (Kenmuir) Castle near New Galloway, the site being occupied from late in the 11th Century until 1900. Kenmure Castle was the seat of the West March Gordons of Lochinvar. The real 'Young Lochinvar' of Sir Walter Scott's famous poem was said to be the Laird of Lochinvar, William de Gordon of Kenmure. Unfortunately, there is no evidence to support the story. Sir William Gordon of Lochinvar was laird of Lochinvar and Kenmure, his eldest son being John of Lochinvar, but the name of his wife is not known. William may have died in 1455, having passed a charter of lands (a will) to his family in 1450. The Gordons of Kenmure were a cadet branch of the Aberdeenshire family, possibly descended from the 14th century Sir Adam Gordon. They became the most powerful family in the GlenKens (Valley of Loch Ken and the River Dee).
Sadly, the lairdship was declared dormant in 1847, the McEwan family then owning the estate. The contents of Kenmure Castle were sold on the American market, in 1900. The Castle was later de-roofed to avoid paying 'roof tax'. The Ewart Library in Katherine Street, Dumfries, holds some documents; the only other local evidence of the vanished Gordons of Lochinvar. The documents span the period from 1507 to 1858, - sadly, long after the period of William Gordon.
The ruins of Kenmure Castle stand amongst trees on a wooded hillock about a mile south of New Galloway, a bare 5 miles from the Lochinvar Hotel at the north end of Loch Ken. The position would have been naturally suitable for a motte-and-bailey fortress of the mid 1000s, later offering good foundations and a commanding position for several rebuildings in stone.
From the Ordnance Gazetteer of Scotland: A Survey of Scottish Topography, Statistical, Biographical and Historical, edited by Francis H. Groome and originally published in parts by Thomas C. Jack, Grange Publishing Works, Edinburgh between 1882 and 1885:
Kenmure Castle, a seat in Kells parish, Kirkcudbrightshire, 5 furlongs above the head of Loch Ken, and ¾ mile S by E of New Galloway. It stands on a high, round, isolated mount, which, till one observes the rock that crops out on its S side, might be taken for artificial; and it seems of old to have been surrounded by a fosse, supplied with water from the river Ken. Approached by a noble lime-tree avenue, and engirt by well-wooded policies and gardens with stately beech hedges, it forms a conspicuous feature in one of the finest landscapes in the South of Scotland. The oldest portion, roofless and clad with ivy, exhibits the architecture of the 13th or 14th, but the main building appears to belong to the 17th, century. The interior is interesting, with its winding staircases, mysterious passages, and heirloom collection of Jacobite relics and portraits - the sixth Viscount Kenmure (painted by Kneller in the Tower of London), Queen Mary, James VI. (by Zuccaro), 'Young Lochinvar' (by Lely?), etc. When or by whom the original portion of the pile was built, is a matter not known. In early times, and even at a comparatively recent date, it suffered much from the ravages of war, having been burned both in the reign of Mary and during the administration of Cromwell. Originally, it is said to have been a seat or stronghold of the Lords of Galloway; and John Baliol is reported to have made it his frequent residence, nay even to have been born within its walls. On the other hand, the lands of Kenmure and Lochinvar are said to have been acquired in 1297 from John de Maxwell by Sir Adam Gordon, whose sixth descendant was the first Earl of Huntly (see Gordon Castle), whilst his tenth, in the younger line, was created Viscount Kenmure. Thus the Gordons of Lochinvar or Kenmure claimed strictly the same stock as the Gordons of the north; and, after settling down at Kenmure, they gradually acquired, by grant, purchase, or marriage, the greater part of Kirkcudbrightshire. They were distinguished by the confidence of, and their attachment to, the Stuart sovereigns. Sir John Gordon of Lochinvar was a steadfast adherent of Mary, and ran serious hazards in her cause. In 1633 his grandson, Sir John Gordon (1599-1634), was raised by Charles I. to the peerage under the title of Viscount Kenmure. This nobleman combined attachment to the house of Stuart with unflinching fidelity in the profession of the Presbyterian religion; and, much as he is known for the honours conferred upon him by Charles, he is greatly better known for his intimacy with John Welsh and Samuel Rutherford. In 1715, William, the sixth Viscount, took an active part in the Rebellion, and next year was beheaded on Tower Hill in London, entailing upon his family the forfeiture of the title. His descendants, however, having bought back the estates from the Crown, endeavoured, by serving in the army, to atone for their ancestor's error, and distinguished themselves by patriotic concern for the interests of their tenants, and for the general welfare; and, in 1824, they were restored by act of parliament to their ancient honours in the person of John Gordon (1750-1840), the forfeited Viscount's grandson. He was succeeded by his nephew, Adam, a naval officer, who displayed great gallantry on the American lakes during the war of 1813, and at whose death in 1847 the peerage became extinct. Kenmure Castle passed to his sister, the Hon. Mrs Bellamy-Gordon, owner of 14,093 acres in the shire, valued at £4230 per annum. John Lowe (1750-98), the author of Mary's Dream, was a son of the gardener at Kenmure Castle, at which Queen Mary is said to have rested in the course of her flight from Langside, and which was visited once by Robert Burns.—Ord. Sur., sh. 9, 1863. See pp. 163, 174-177 of M. Harper's Rambles in Galloway (1876); and p. 302 of R. Chambers' Popular Rhymes of Scotland (edn. 1870).