An ancient friary, dating back to the 11th Century, once occupied this land. In the 14th Century, the Old Rectory was built, possibly by Thomas de Alkham, who died in 1356. He served as the Rector of Southfleet from 1323 and became the chancellor of the Rochester Diocese in 1327. 
The parsonage house is one of the most ancient edifices of the kind in the diocese. It is built of stone, and the windows are large with pointed arches and stone munitions resembling those of a church. The porch has a strong arch, and the whole has a most venerable and ecclesiastical appearance. It had much more so till the front of it was lately plaistered over and whitewashed. The gothic windows altered and sashed, which has taken much from the ancient beauty of it. Some windows on the south side next to the yard still retain their old form.
The Rector in 1893 was J H Hazell; I am unsure whether he was Rector throughout the whole of the 1890s.
Since the 1950s, the Old Rectory has been divided into two residences: Friary Court and the Old Friary. In August 1952, it was granted a Grade II listing.
Local lore suggests that during medieval times, three nuns (though some accounts say just one) were bricked up in the cellar and left to starve after being caught in an indiscretion with some monks. There are also reports of a monk's body found in a tomb beneath rubble near the west door of the church, with an inscription on the tomb stating that the monk had been excommunicated.
Another tale involves a female nurse whose body was believed to be walled up in the cellars. it was reported that in the mid-19th Century, while caring for a severely ill patient, the nurse allegedly attempted to tamper with the patient's will in her favour. 
**HAUNTINGS**
A stained glass window commemorates a visit by the Bishop of Rochester Thomas Legh Claughton (Born 6 November 1808—died 25 July 1892) in 1874 to exorcise a ghost. Between 1891 and 1898, numerous ghostly encounters were reported, including sounds of rustling paper, footsteps, starched clothing movement, and doors opening and closing. 
According to Anna Dubuis in an article entitled 'The ghosts and legends of North Kent', which appeared in the Gravesend Reporter on Wednesday 31 October 2012 ', Old Rectory in Southfleet has a ghostly nun' David Mills in a News Shopper Website article called 'NORTH KENT Is where you live haunted' (30 August 2009) gave the following information regarding the haunting:
The ghost, known as the Brown Lady and believed to be one of the nuns, reportedly haunts the Monk Room. Described as a short, plump woman dressed in a brown habit, she is said to have appeared partially at times. In 1874, the Bishop of Rochester unsuccessfully attempted to exorcise the spirit in this room.
Regarding the nurse, The article mentions a nurse who was, according to local legend wal, led up in the cellar, but this story may be confusing with the nun's tale; 
A servant recounted the following experience: "I stood at the top of a small stair just outside the room, and to my surprise, saw a nurse in a clean staff uniform standing at the end of the passage. Feeling nervous about her presence, I timidly walked down the three stairs and along the passage toward her. As I approached, she began to walk forward as if to meet me, then gently drifted backwards, all while facing me, until she reached the door of a small room at the end of the passage. To my surprise, she vanished back through the door."
It is said that there are references to possible paranormal experiences as late as 1920.