The novel alternates between two timelines: the present-day story of Sara Jenkins, who runs a vintage furniture and home décor shop called Bits and Pieces in New Orleans, and the past story of her grandmother, Mags (then called Margaret), whose secret history comes to light after Mags's death.
In the opening chapter, seventy-two-year-old Mags sits on a cedar bench in the garden of The Hideaway, a bed-and-breakfast she runs on Mobile Bay in Sweet Bay, Alabama. She considers telling Sara the story of her life but decides to wait. She feels a tightness in her chest and dismisses it. Shortly after, Sara receives a call from Mags's lawyer, Vernon Bains: Mags has died of a heart attack. Sara grieves alone, studying an old photo of herself, her mother Jenny, and Mags, sensing a complexity in her grandmother she had never noticed.
The novel flashes back to January 1960. Margaret, a young society wife in Mobile, finds a note from her husband, Robert Van Buren, covering for his affair with a woman named AnnaBelle Whitaker. Margaret has already packed to leave. Her father gives her money, assuming she will return, but she drives across Mobile Bay and stumbles upon The Hideaway, a fading Victorian bed-and-breakfast run by a widow named Mrs. Evelyn DeBerry. Margaret checks in under a false name. Among the artists and bohemians living communally in the house, a quiet woodworker named William Cartright catches her eye. Margaret begins helping around the house, shedding the trappings of her old life: stockings, bobby pins, pearl necklace, and wedding ring. She and William become a couple, and he takes her to a secluded cove he owns on the bay, where he hopes to build a house one day.
In the present, Sara drives to Mobile for the will reading. She reunites with The Hideaway's four longtime residents: Dot Ingram, Mags's best friend; Bert Ingram, Dot's husband and the house cook; Major Gregg, a gruff army veteran; and Glory Gregg, Major's wife and the house hairdresser. The will leaves The Hideaway to Sara, with instructions to renovate and live there during the work. A letter from Mags explains her wish for the house to be restored. A small key falls from the envelope. Major is angry, but Dot defends the decision.
Sara assesses The Hideaway: The house has beautiful bones but is in serious disrepair. After a call with Allyn, her employee and closest friend, who assures her he can manage Bits and Pieces, Sara decides to stay. She hires Coastal Contractors, run by Charlie Mack and Crawford Hayes, a thoughtful man passionate about restoring old houses. While cleaning the attic, Sara discovers a locked metal box that the small key opens. Inside she finds a 1957 photo of a young Mags, a hand-carved miniature house with a skeleton key on the underside, a diamond ring, torn pieces of a note signed "Love, William," and a postcard from Mags's mother reading, "You made the right choice." Sara is confused: Her grandfather's name was Robert, not William. She recognizes the same skeleton key on the garden bench and on Mags's headstone, inscribed, "You hold the key to my heart."
In the past, Mrs. DeBerry disappears, leaving Mags a note asking her to take over The Hideaway. Mags uses her father's money to pay down Mrs. DeBerry's debts and asks the residents to contribute financially. William carves a new sign for the house and builds Mags a miniature house representing the home he plans to build in the cove. He begins engraving a skeleton key into every piece of furniture he makes, a symbol of his love for Mags.
Mags's father tracks her to Sweet Bay and insists she return to Mobile, revealing that Robert has come back and is experiencing episodes related to his war service. William overhears the conversation on the back porch. Before Mags can speak to him, he leaves in the night, writing that he is leaving to spare her the pain of choosing. He leaves behind a diamond ring. Mags is devastated.
In the present, Dot reveals the central secret: William, not Robert, was the biological father of Jenny, Sara's mother. Mags becomes pregnant before William's departure. Her parents pressure her to stay with Robert, who moves into The Hideaway under strict conditions: separate bedrooms and no discussion of the past. Jenny is born with William's fair coloring. Mags endures life with a man she does not love, hoping William will return. After a confrontation years later, Mags forgives Robert, realizing they were all trapped by the expectations of their era. Robert eventually leaves, returning to AnnaBelle. Mags embraces the eccentricities Sara grew up seeing, deliberately rejecting the propriety that had constrained her life. She writes a letter to Jenny explaining who her real father was but never delivers it. When Jenny and her husband Ed Jenkins die in a car wreck, Mags loses her last connection to William.
Sara's relationship with Crawford deepens. She discovers a map marking the location of William's cove, and Crawford drives her there. A white-haired man arrives at The Hideaway; Sara recognizes him as the figure she noticed at Mags's funeral. He introduces himself as William Cartright. He explains that he came back a year or two after leaving but saw Mags, Robert, and a little girl together and assumed the child was theirs. Sara does not immediately reveal the truth. Later, over lunch, she tells William that Jenny was his daughter. William is overcome with emotion and learns that Jenny died in a car accident.
The Hideaway faces a new threat when Sammy Grosvenor, a local developer who has long coveted the waterfront property, invokes eminent domain, a legal process allowing the government to seize private property for public use. He has convinced the mayor that condominiums will benefit the town. A newspaper article generates community support, with neighbors sharing stories of how Mags helped them, but Sammy delivers a court order: thirty days to vacate. The residents begin planning to leave. Sara says painful goodbyes and returns to New Orleans.
In New Orleans, Sara tries to resume her old life. Allyn warns her not to shut Crawford out, telling her she has changed and has love and family in Sweet Bay regardless of the house. A week later, Mr. Bains calls: Sammy has withdrawn his claim.
Sara rushes back to find that William sold his cove, the land he kept for fifty years, to Sammy in exchange for releasing his claim on The Hideaway. Crawford discovered William's ownership of the cove while searching old land records for a way to save the house. Sara tells the residents to unpack and reconciles with Crawford.
By December, The Hideaway has reopened as a functioning bed-and-breakfast. Crawford's team has completed the renovations and built a small cottage on the adjacent lot, which Sara turns into a shop called Lost and Found, selling vintage furniture, including pieces made by William. Sara and Crawford are in a committed relationship, and William has become part of their extended family. Sara reflects that Mags drew her back to Sweet Bay, and the house and family she found there saved her from a successful but solitary life.
The novel's final chapter returns to Mags nine months before the present. She visits the cove one last time, reflects on William and on the life she lived, and acknowledges she has known real love and true beauty. Sitting on her cedar bench, she watches the sunset and decides she has had it just about as good as it gets.