I think that has more to do with technology and the attention economy than anything else. Working class people used to read books a lot more than they do now. Then along came TV (aka the idiot box) to soak up those free hours. Now it’s all Facebook, Instagram, TikTok, YouTube, Netflix.
I doubt working class people spent their evenings reading high-brow books. Magazines, cheaper novels, things that don’t demand much mental investment after 8+ hours of work have drained your energy and left a little for chores.
Families that could live on a single income may have had more time, but if that has reduced, it may well because a single income often can’t sustain a whole family any more.
TV didn’t magically create a need for mindless entertainment. It may have supplanted other recreational activities, but it couldn’t replace e.g. meeting up for a drink and a nice chat unless the convenience of it outweighed the loss of social activity.
They might not have read Joyce but I can guarantee they were reading Steinbeck, Hemingway, Poe, Whitman, Dickinson, Twain, Vonnegut, Lee, Salinger, Frost.
All the novels and poetry in the American canon, the stuff high school students groan about having to read today, were once bestsellers in their day. You don’t get to be a bestseller back then by selling only to millionaires.