Boyle and his wife Caitlin Coleman were snatched by the Haqqani network - a group allied to the Taliban - while hiking in Afghanistan in 2012.
"We're sitting as hostages with a lot of time on our hands", Mr Boyle told The Associated Press in an email. Boyle said the middle child, 2-year-old Noah, is having the most hard time adjusting.
The couple and their children were freed by a Pakistani military operation and returned to Canada on Friday.
Mr Boyle said his eldest son is particularly fascinated with his discovery of a flushing toilet, but their middle child, Noah, is having a hard time adjusting.
Boyle also said that he thought one of the captors, who told him the new US president was Donald Trump, was joking.
Speaking at Toronto Airport, Mr Boyle, 34, said the family's captors killed their baby daughter and raped his wife.
He said his captor mentioned it before he was forced to make a "proof-of-life" video.
Boyle gave some insight into the conditions the family faced, explaining that they were moved to three different prisons over the five years.
Trump blames Cuba for diplomat attacks
Administration officials had said they did not know who was responsible and that Havana was cooperating in an investigation. State Department, the attacks have caused health problems in 22 American diplomats and intelligence personnel.
He has described the first as remarkably barbaric, the second as more comfortable and the third as a place of violence in which he and his wife were frequently separated and beaten.
He also emailed two photos of his son Najaeshi Jonah Makepeace Boyle and said the boy began "raiding the first refrigerator of his life".
Meanwhile, Mr Boyle denied reports he had had refused to board a plane that was preparing to take the family home.
At this point, Ms Coleman has declined to be interviewed - she simply nodded to confirm her identity when an Associated Press reported asked.
Mr Boyle, a former call centre worker, said all five family members - including their sons Jonah and Noah and newborn daughter Grace - were stuffed into the boot of a auto.
"The stupidity and evil of the Haqqani network's kidnapping of a pilgrim and his heavily pregnant wife engaged in helping ordinary villagers in Taliban-controlled regions of Afghanistan was eclipsed only by the stupidity and evil of authorising the murder of my infant daughter", he said.
Mr Boyle was previously married to the older sister of former Guantanamo Bay prisoner Omar Khadr.
The Canadian-born Omar Khadr was 15 when he was captured by U.S. troops following a firefight and was taken to the USA detention centre at Guantanamo Bay.
AP reports one official saying the marriage had nothing to do with the abduction which was being treated as a "a frightful coincidence".