I often hear from others and even find myself speaking about "the implications of such and such" or how "what so and so did was illogical", and it struck me suddenly how our language carries around in it a very particular and peculiar understanding of how humans deal with their world. That understanding sees us as computers, which given new information, calculate out the logical implications that flow from it and add them to some store of facts in our minds. This might sound like a modern understanding, but it's among the most ancient. It's the exact same way the Ancient Greeks, like Plato, believed humans ought to and must really fundamentally work. Those beliefs were a result of a religious reverence for the Idea and one of the primary tools associated with it - Logic.
Computers, though a recent invention, are a direct implementation of that ancient understanding. They have been fruitful in their usefulness to our culture, and have also brought along and deepened an unquestioning adoration for that understanding. The weight and heft of that fruit seems to block any passage of the light of evidence (and ironically, logic or reason) from reaching those who adhere to that understanding. The failure of computers to deliver an artificial intelligence via that logical understanding - even with decades of top dollar DARPA funding for artificial intelligence labs cross America's top universities, initiated from the very first years of modern computing - hasn't shifted that adoration out of our culture. It is so ingrained in our language, that it's hard to talk about any other kind of understanding. It would be fair to say that we don't have any other understanding that has diffused itself into our culture enough to contrast it with. Without that, there's a vacuum that makes it hard to speak about any other view intelligibly. So I'll present here another understanding that's been worked through in the last 100 years or so (yes, rather old, but at least not 2k+ years):
Part of what we call "growing up" is finding out how to make sense of the world. Via an inborn inclination towards sense and language, we absorb a particular breakdown of what matters in the world and how to do things from how our families and the wider culture they're embedded in have already been doing all that. We make their sense our own. This sense gives us an understanding, before we even step into a room or situation, of what is and isn't possible. What should happen when (I see someone I don't know at this party - I should introduce myself and "shake" their hand), what things are and function as (that big puffy thing against the wall, I can put my butt on it if I want to rest - I can "sit" on that "couch") - all of which aren't some natural consequence of the structure of the situation and the objects in it. Rather, they make sense based on the culture I've grown into. Someone who grew up in some other culture will have a different sense. They might consider it odd to be perched up so high on a "couch" to rest and would prefer to squat should they feel tired.
When something new or surprising happens, my sense of the world immediately changes. Via the presence of this new element of the world, things relate to each other differently when I "look" at them (looking with my eyes, or simply attending to them - putting them in the front of my mind). The parts of the world that are "closest" to me then, immediately show themselves to me in the world opened up by that new sense. If it suddenly starts raining, I put my hood up right away. The rain drops are acting directly on me and annoying - they are "close" and have grabbed my attention. My hood fits into my culture's "how to handle rain" sense, and now it's raining, so I use it. At the same time, I know my little brother is at a soccer game right now, and before the rain started I was already feeling guilty that I couldn't make it. Even though the game is across town, the realization that he'll have to play in the rain immediately dawns on me and I feel sorry for him. He was already "close" to me in this sense. Meanwhile, I left my windows wide open at home. I would be able to remember that perfectly well if asked specifically about it, but I don't think of that at all when the rain starts. The windows aren't close to me in any sense at the moment, and my guilt, plus whatever I'm up to instead of going to the game cloud out any chance that I'll realize that my apartment is open to the elements, until I get home.
Someone who believes (explicitly or implicitly) that humans deal with the world fundamentally by logical calculation might see me as forgetful, or at least see a "better" or "more alert" person as someone who would have seen the problem of the open windows as soon as the rain started and would have rushed home. Moreover, I'd expect believers in computer-based AI to think of a good AI as one that would also realize this problem immediately, plus any other important implications from the rain, given its store of facts. But that doesn't fit how we can plainly see humans work, in their everyday, basic dealings with life. When we sometimes say in exasperation that people are illogical, we're exactly right. I don't think that's a bad thing at all.
In fact, perhaps this particularity of how people work - that they have attention focused on a limited area within a wider world of sense - is a necessary part of the structure of anything we'd recognize as intelligent that can actually exist in our world. Let's consider that structure a little more: Each person is centered in their perspective on the world, and based on whatever they're up to and dealing with at the moment, certain things are close (in the sense I alluded to earlier), and others are far. You're able to act intelligently in as much as you give your attention to things close at hand and act on them in ways that make sense. When you don't pay attention to things close at hand, and instead let your attention be diffuse around you, we say you have our head in the clouds. You can't deal properly with the world without focused attention. Instead of flood lights on everything at once, you laser in on a few things for a while. In every day life, you don't calculate out implications to understand what's going on - you simply see the world as making sense directly. When a change happens, you see a new world that makes sense directly.
But where does Logic fit in then? It's a tool, which you can use to see further, given some facts and focus on something you want to know. Had I gone beyond the obvious scene that rain + a soccer game produces in the sense of the world I have, I may have been able to deduce that the game was probably cancelled. But that's simply not the default operating mode for people. Perhaps a computer with vast amounts of compute and storage could do all the logical computations implied by new information, regardless of attention or focus? That was the implicit assumption in the first generation of AI, and some ongoing efforts like Cyc. But even with exponential compute growth and clever algorithms, that showed itself to be an unrealistic model and failed (predictably, by philosophical investigations that inspired this).
Though this logical understanding of human dealings lives on in our language and subtly in our sense of ourselves, it's dead in intelligence research for the most part. The understanding of how humans deal that's currently assumed in intelligence research and applications is a younger cousin of the ancient Logical one. Instead of implication, for those who're immersed in it, it evokes a language of priors, probability, and statistics. This model has also been rather fruitful for our culture, and it's also inherited and grown a religious adoration that just as stubbornly resists reason and evidence. Like all dogma historically, the Logical and Statistical understandings blind us, as a culture, to views of the world and ourselves that can not only take us closer to an understanding of reality, but can transform us. In sticking to them, I suspect we aim specifically to not be transformed by our technology - to not have to change fundamentally. To get more of the same thing we've always suspected, but bigger and better. Just keep scaling. Put 7 trillion into it, why don't you?
We won't move forward until we can let go of that. Until we're willing to truly broaden our horizons and be moved and awed by the world and each other again.
I have purposely tried to avoid the very language I'm talking about throughout this thing. No "this implies" or "such and such is implicit". Let's see what it looks like when English carries these thoughts without its Logic related idioms.
No, the "attention" in transformers is not this. There is no world of sense in that. And no, embeddings are not a world of sense. Too much to go into here, but I can discuss elsewhere if you'd like.
Another example to drive home the independence of physical distance and this sort of distance: for someone who wears glasses, when they're working fine they are the last thing on their mind. They are so physically close that they're sitting on your face, but they are much further from your attention than what you're looking at across the room.
Cyc is an epic moonshot of a project to put common sense knowledge and then some into explicit, logical, computer consumable form. They've been at it for 40 years. As far as I've heard, intelligence is still not forthcoming from that approach. See https://en.wikipedia.org/wiki/Cyc for more.
See Alchemy and Artificial Intelligence , by Hubert Dreyfus. Note the date (1965). He had a lot to say, on the topic of the relationship between computers and intelligence / human being. Most of it is derived (as is the "other understanding" I presented here) from Martin Heidegger's insights, which he is far far better at presenting to a contemporary, English speaking audience. Lots of insightful material from him easily accessible via google.
Again, because every child has this. And then as we grow up and gain sense, dogma and certainty come along for the ride, in our culture. Seems to me truly new ideas have historically accompanied a child-like sense of awe and wonder. Can't be very curious without that.