• Ghana had achieved its independence from the UK in 1957, and Mr Mugabe had come to "to find himself" and experience the dream of an independent African state.
• Indeed, Zimbabwe granted Ghanaians visa-free entry, and the arrangement still exists.
In our series of letters from African writers, journalist and former Ghana government minister Elizabeth Ohene writes that Zimbabwe's ex-President Robert Mugabe, who has died aged 95, had a special place in the hearts of Ghanaians.
We in Ghana called Robert Mugabe our in-law. It was in Ghana that he met his first wife, Sarah Francesca Hayfron. He married her in 1961, and she became known as Sally Mugabe.
She was one half of a set of female twins and she was teaching in Takoradi city in Ghana's Western Region when Mr Mugabe met and fell in love with her.
Ghana had achieved its independence from the UK in 1957, and Mr Mugabe had come to "to find himself" and experience the dream of an independent African state.
The unofficial story was that he had received training on how to wage the struggle for independence in what was then Rhodesia.
Those were heady days in the newly independent Ghana and the Hayfron family was educated, political, well-connected and part of high society.
The father was a well-trained horticulturist and his home was famous for well designed gardens.
Sarah Francesca Hayfron was certainly quite a catch for the then unknown and struggling Robert Mugabe.
As it turned out, his new wife turned out to be a veritable political partner and activist, when he took her back to his home country.
Mugabe's extra-marital affair
The story of the couple's struggles in Rhodesia - including their incarceration - have become the stuff of legend.
In the midst of it all, and when Mr Mugabe was in prison, their son, Nhamodzenyika, died of malaria in Ghana.
This was to have far reaching consequences because by the time husband and wife were together again, they could not have another child.
In 1987, Mr Mugabe started an affair with his secretary, Grace Marufu, 41 years his junior, because, according to him, he felt he had to provide grandchildren for his mother. It was not because of his desires.
Sally stayed by her man, continuing as a political activist and earning the well-deserved title of Amai, or Mother, by the time Mr Mugabe became prime minister in 1980, and First Lady when he became president in 1987.
We in Ghana felt we had a special status in newly independent Zimbabwe and it was suddenly the place to go.
The 1980s were not good times in Ghana and many top professionals - including medical doctors, university professors, and judges - went to live and work in Zimbabwe.
They were anxious for Zimbabwe to learn the lessons from Ghana, and other African countries, which had attained independence earlier and were struggling politically and economically.
They admired the orderliness of the Zimbabwean towns and cities, and their hosts admired their self-confidence.
But on a reporting trip to Zimbabwe in 1987, I found that many of the Ghanaians were expressing anxieties about Mr Mugabe's decision to transform his position from prime minister to president. They said they feared for Zimbabwe's future.
Zimbabweans were welcoming to Ghanaians and I remember arriving at the main airport in the capital, Harare - now known as the Robert Gabriel Mugabe International Airport - and the immigration officer smiling when he saw my Ghanaian passport and saying: "Welcome, our in-law."
By the beginning of the 1990s, some tensions were surfacing as the emerging Zimbabwean black middle class began to resent the presence of Ghanaian professionals.
But the respected in-law position remained. Indeed, Zimbabwe granted Ghanaians visa-free entry, and the arrangement still exists.
We in Ghana silently cheered Sally's firm but tender hold on her husband and it was widely felt that she was his moral anchor, who kept him on the straight and narrow.
Sally's death in January 1992 was felt keenly in Ghana and a large delegation went to her funeral in Harare.
Our son-in-law looked bereft and he gave a most touching and powerful eulogy at the burial ceremony.
Once Grace was installed as the new Mrs Mugabe in 1996, we knew our status had changed.
Many of the Ghanaians left, hardly any new ones went and Zimbabwe disappeared from the destination map of the ever-footloose Ghanaian.
'Unsentimental and unforgiving'
In 2017, one of the last surviving members of the group of Ghanaian émigrés - Roger Korsah, a retired judge of the Supreme Court - died, almost bringing down the curtain on that chapter.
But Zimbabwe continued to have a special place in the hearts of Ghanaians.
We can never be as rude about Zimbabwe as we can be about other countries, and when we hear of its political and economic difficulties, the feeling is one of dismay and bewilderment.
In Ghana, your in-law is your in-law, and deserves respect, no matter what happens.
I have to admit though that the last time Mr Mugabe came to Ghana, to attend our 60th independence anniversary on 6 March 2017, the younger generation was only interested in poking fun at him for seeming to fall asleep during the ceremony.
It was, sadly, a different Robert Mugabe from the man who had married our Sally; now at 90-something and into his 37th year as leader of Zimbabwe, young Ghanaians on social media were unsentimental and unforgiving.
Now that he is gone, we will have to go and perform the necessary rites at the funeral; he was our son-in-law, and that cannot change.