The concept of the self is closely related to archaic notion of the soul, which is an important vehicle of socially constructed organized religions (ancient social institutions designed to accumulate wealth and power by exploiting ignorance, superstitions and fear of death of its subjects and followers).
The ultimate non-existence of the self could be proved by the same chain of argumentation educated people dismiss the notion of a soul today as a mere social creation - a meme. The last 300 years of philosophy and science were, to a large extent, was the effort to discard religious dogmas and replace them by more adequate approximation to the truth.
At the time of the Buddha the only reliable method of discovering of the nature of reality was to trust our senses more than dogmas and be very precise with language use. This is the essence of the method of introspection, which, arguably, has been used by the Buddha to gain his insights about the nature of the mind.
Turning his attention inward (to put it in a modern language - he let his intellect observe its own working) he observed and classified mental phenomena (processes of the mind) which arise and fade in his own mind. After analyzing the nature of these processes he concluded that each one of these are impermanent (transitory) and specialized, which means not general enough to govern the whole observable human behavior.
This result of so-called mindfullness or self-awareness - when one aspect of the mind is tried (to a very limited extent) to observe the other aspects within the whole of brain's activity is, arguably, the best "philosophy of mind" has been done so far, and predates modern science of psychology.
One of the implications of observations made by the Buddha transformed by him into profound insights is the illusory nature of what we call "our self". He taught that this self is mere an appearance, a set of aggregates, like cooking spices wrapped in a banana leaf. Another famous metaphor is a process of pealing of a piece of onion, when by removing layer by layer of what we call nowadays habits, memories, personal experiences, social and cultural conditioning, no substantial self, no permanent core could be found.
This notion of an illusory nature of what we call self is the most fundamental concept of the Buddha's philosophy, which relates to the concept of Maya which predates him. Maya, according to some of Hindu mystics (which is another name for the seekers after the absolute truth, or god), is a veil which obstructs or view of what is - of reality as it is. Arguably, it is due to inevitable interference of some parts of the brain with another, especially so-called language area - a seat of linguistic abstractions, which produces its own constructs, which, it seems, looks from the other areas of the brain as a valid and accurate perceptions. The glimpse of such possible dynamics has been demonstrated in the set of experiments with hemisphere-split patients.
What is the mark of true genius is the use intuition - so called non-verbal knowledge of the body to guide ones search for the closest approximation of the truth. Modern scientists will tell us that the body has information about the nature of the environment it has been shaped by, encoded in the physical structure of the specialized sensors and brains areas. The Buddha has no such understanding, but he paid attention to intuitions from his own brain. Nowadays we would formalize this process as a heuristic-guided search.
Thus he, presumably, got the insight about a brain being an aggregate of highly specialized areas and the phenomena of the mind which is accessible to introspection by so-called primordial awareness turned inward, is mere a bunch of parallel processes which arise and fade on-demand to serve a particular low-level or high-level function which is a part of a observable behavior.
It is quite easy to mess everything up with the modern terminology and modern views based on decades of research in the field of cognitive psychology, nevertheless it is easy to observe that the insights about the fundamental principles about the nature of the mind, described by the Buddha, are still hold. His method of direct observation instead of mere abstract speculation and practice as the way of testing his hypothesis (insights) is a precursor of the scientific method we use today, giving that each one of his disciples is free to replicate test and validate his insights and conclusions.