Why don't we remember being babies?

The inability to remember your first few years of life is called infantile amnesia. But why does it happen?