This children's novel follows Gregory, a nine-year-old boy known in his neighborhood as "the Chalk Box Kid" for the detailed chalk garden he once drew on the walls of a burned-out chalk factory behind his house.
As summer vacation begins, Gregory dreads the weeks ahead. His mother, a restaurant cook, and his father, a bank guard, will be at work every day, leaving him home with his Uncle Max, who spends most of his time watching television and playing guitar. Gregory's friend Ivy, a talented artist from his school, arrives looking for her little brother Richard, who has wandered off again. They search the neighborhood and find Richard sitting comfortably under an orange tree in the backyard of Uncle Pancho, an elderly man who lives alone in the last house on the street. Though he is not actually anyone's relative, everyone in the neighborhood calls him Uncle Pancho. The old man has been watching over Richard and telling him stories.
After returning Richard home, the children go back to visit Uncle Pancho, bringing a loaf of bread from Ivy's mother. The old man tells them about growing up in Mexico with his siblings, describing how they took goats to pasture, played on a swinging bridge of rope and vines over a river, and listened to a beautiful lady singer who once performed just for him. Gregory announces he wants to spend his vacation painting Uncle Pancho's house, and Ivy offers to help, noting her father, Mr. Brimm, is a house painter with plenty of supplies.
The next morning, Mr. Brimm gives the children a sample of yellow paint on a piece of wood. Uncle Pancho holds it against the house and approves the color. He then reveals that government men told him the previous year that a new freeway would cut through the end of the street and demolish his house, a home he built with his own hands. Bob the barber assured him it was a mistake, so he stopped worrying. Uncle Pancho also shares a painful personal story: His wife grew tired of poverty and left, taking their young son José. Uncle Pancho followed them but lost their trail and eventually gave up the search.
Gregory, moved by everything the old man has shared, proposes painting pictures of Uncle Pancho's life on the exterior walls of the house. Uncle Pancho reacts with sudden anger, believing Gregory wants to make his house an object of ridicule, and storms inside.
Gregory spends the rest of the day brooding in his chalk garden. That evening, he visits the nearby plant nursery owned by Mr. Hiller, the man who originally inspired Gregory's chalk garden. Mr. Hiller, an old friend of Uncle Pancho's, suggests the old man simply did not understand Gregory's intentions and offers to speak with him.
The next morning, Uncle Pancho appears at Gregory's chalk garden. He apologizes, explaining that Mr. Hiller told him Gregory is the Chalk Box Kid. After examining the chalk drawings on the walls, Uncle Pancho recognizes Gregory's genuine artistic ability and understands the boy's intentions were sincere. Gregory proposes a compromise: They will first paint the house yellow, and then he will paint one picture. If Uncle Pancho does not like it, Gregory will paint over it. The old man agrees.
The three children scrape the old paint and spend a week painting the house yellow, using supplies provided by Mr. Brimm. Gregory plans the pictures with Ivy: Gregory will handle the large compositions, Ivy will contribute faces and small details, and Richard will fill in grass and sky. Mr. Hiller brings tubes of paint for Gregory to mix. Gregory paints the first mural: a portrait of the young Pancho lined up with his brothers and sisters. Some passersby criticize the work, but Gregory remains confident. When Uncle Pancho finally looks, he recognizes himself and his siblings immediately and urges Gregory to paint more.
Over the following weeks, Gregory covers the house with scenes from Uncle Pancho's stories: the swinging bridge, the mountains of Mexico, the house of the lady singer, and the little roads. For the final painting, Gregory depicts Uncle Pancho standing with a grown-up version of José by the orange tree, expressing hope that father and son might reunite. Uncle Pancho is deeply moved, acknowledges Gregory as a good friend, and resolves not to give up hope of finding José.
While the children work on the last painting, a man from the state arrives and confirms the freeway is coming through. The house will be torn down. Uncle Pancho initially refuses to believe the news but then falls silent, looking suddenly old and tired, and retreats inside. Ivy and Richard leave, but Gregory stays behind to finish the painting, which he considers his best work ever.
Miss Cartright, Gregory's art teacher, visits and is amazed by the murals. When she learns the house must be demolished, she vows to fight and devises a campaign under the slogan "Save the Picture House," including a television appearance. Ivy is too shy to go on camera, so Miss Cartright presses Gregory to represent the project. Mr. Hiller encourages Gregory by noting that the publicity might help José see the story and come forward.
Gregory and Uncle Pancho appear on the television show. Gregory describes how he, Ivy, and Richard painted the story of Uncle Pancho's life, and Uncle Pancho tells the audience about coming from Mexico to find his son. Uncle Max writes a song about the Picture House but is not allowed to perform it on the show. The announcer declares the house must be preserved. The next day, crowds fill the street to see the paintings, but José does not appear. Uncle Pancho tells Gregory that when his son comes, they will recognize each other.
Miss Cartright meets with the freeway officials but returns in tears: They refuse to change their plans. Gregory and Mr. Hiller deliver the news to Uncle Pancho, who has already heard. The old man despairs, saying the effort was for nothing and José never came. Mr. Hiller tells him it was not for nothing, offers to be a son to the old man, and proposes moving Uncle Pancho's house to the lot behind the nursery, where there is room for two houses.
Workers lift the house onto wheels and tow it through the streets at night, setting it down beside Mr. Hiller's house. A celebration takes place around the relocated Picture House, with food, music, and dancing. Uncle Pancho dances with the crowd, his face looking almost young again. Mr. Hiller makes a speech, noting that a telephone pole knocked off a bit of paint during the move and Gregory repaired it, then invites Gregory to address the crowd. Gregory reflects that he did not know what to do at the start of vacation but found a meaningful project, calling it his best vacation ever. Uncle Max plays his guitar and sings his song, ending with lines that christen Gregory "the Paint Brush Kid." Gregory is surprised by how well his uncle can sing.