Several of Joseph's letters survive, shedding some light on the family history and supporting the claim by Philip Stewart that the family descends from the royal Stewarts:

" White Creek, Aug. the 28, 1818

" I have received your letter and I am sorry to hear of your sickness.
"By all accounts of our descent, we are of the royal house of the Stewarts. My Father was John the eldest son of Robert my Grandfather, who was obliged to fly to Ireland when they were newly married. My Granny was sent to Edinburgh and he was born there. As far as I can learn they belong to the House of White Rose and not altogether separated from the House of Black Hall. My Grandfather's family's names was John and Robert and their sister's name was Juleyan, Samuel the youngest. My Grandfather had a good estate in Scotland when he fled from it. King William would do nothing about it, neither would Queen Anne, but when King George came to the crown under their Uncle Samuel Stewart, by the help of the Duke of Argyle, recovered it. That must be the estate you mention. I was informed that Uncle Samuel died without issue, left no heirs. The way that I came to know of our descent was by old Father James Stewart of Colrain. You may remember young James, who married at last Margaret Anderson your cousin. That descent was from White Rose, for he himself belonged to Black Hall. He had a catalogue of the house of Stewarts for many hundred years, but son Alex carried it away with him to Pennsylvania. I did not know all this until after my father's death. This I knew they belonged to the Rose party, by reason of the high etseem they had for Charles the 1st who had many good properties.
"My father's eldest son Charles who is your uncle, and my Father and your grand Uncle Robert would never own the last pretender or any of the race by reason of his spurious birth. No man dare assail the name of Stewart that was if he would not forfeit his life. It gives you the reach, James the 1st had two sons James and Robert, James the 3rd had two sons James and Robert. This is the whole I can give you at present. I am afraid you can't read for since I got that fall at your house I could never hold a pen to write straight. I would beg you would acquaint me of your proceedings therefore I rest
"Your father
Joseph Stewart."

The following letter was addressed to a person, name unknown, who evidently had written him a letter of inquiry:

"White Creek, March 15, 1819.

"Dear Sir,

"I received yours of the 16th February last, informing me that the heirs of Elizabeth Forsyth that she married a Stewart. My Father's name was John Stewart, the eldest son of Robert Stewart. My mother's name was Elizabeth Forsyth. My grandmother's name was Forsyth, her Christian name forgot. My great-grandfather's name was, I believe, Walter. My grandfather's name by my mother's side was either William or James, which I cannot tell."

The above letter seems to have been closed abruptly and never received by the person addressed, it being a choice relic now in the possession of Joseph's descendants (Severance, B. Frank. Genealogy and biography of the descendants of Walter Stewart of Scotland and of John Stewart who came to America in 1718 and settled in Londonderry, N.H. Greenfield, Mass. : T. Morey & Son, 1905, pg. 31-35).