we need to figure out what consciousness is

Nah, “consciousness” is just a buzzword with no concrete meaning. The path to AGI has no relevance to it at all. Even if we develop a machine just as intelligent as human beings, maybe even moreso, that can solve any arbitrary problem just as efficiently, mystics will still be arguing over whether or not it has “consciousness.”

Edit: You can downvote if you want, but I notice none of you have any actual response to it, because you ultimately know it is correct. Keep downvoting, but not a single one of you will actually reply and tell us me how we could concretely distinguish between something that is “conscious” and something that isn’t.

Even if we construct a robot that fully can replicate all behaviors of a human, you will still be there debating over whether or not is “conscious” because you have not actually given it a concrete meaning so that we can identify if something actually has it or not. It’s just a placeholder for vague mysticism, like “spirit” or “soul.”

I recall a talk from Daniel Dennett where he discussed an old popular movement called the “vitalists.” The vitalists used “life” in a very vague meaningless way as well, they would insist that even if understand how living things work mechanically and could reproduce it, it would still not be considered “alive” because we don’t understand the “vital spark” that actually makes it “alive.” It would just be an imitation of a living thing without the vital spark.

The vitalists refused to ever concretely define what the vital spark even was, it was just a placeholder for something vague and mysterious. As we understood more about how life works, vitalists where taken less and less serious, until eventually becoming largely fringe. People who talk about “consciousness” are also going to become fringe as we continue to understand neuroscience and intelligence, if scientific progress continues, that is. Although this will be a very long-term process, maybe taking centuries.