That phrase, spiritually dead, is not found in the bible (except the New Living Translation, I think). In other words, it's a concept that we regularly insert into passages to help them make sense based on our understanding of how things work. But is it a correct concept?
The word is not there but the concept certainly is.
Genesis 2:17 - God warns Adam, "But of the tree of the knowledge of good and evil you shall not eat, for in the day that you eat of it you shall surely die." This death refers to a spiritual separation from God, as Adam and Eve did not physically die immediately but were cast out of God's presence.
Isaiah 59:2 - "But your iniquities have separated you from your God; and your sins have hidden His face from you, so that He will not hear." This highlights the relational separation caused by sin.
Ephesians 2:1-2 - "And you He made alive, who were dead in trespasses and sins, in which you once walked according to the course of this world..." Paul describes believers as having been spiritually dead before their salvation.
John 5:24 - Jesus says, "Most assuredly, I say to you, he who hears My word and believes in Him who sent Me has everlasting life, and shall not come into judgment, but has passed from death into life." The transition here is from spiritual death to life.
Colossians 2:13 - "And you, being dead in your trespasses and the uncircumcision of your flesh, He has made alive together with Him, having forgiven you all trespasses."
Let's see if it is helping. First, you agree with me that when Paul speaks of death coming through Adam, that it at least includes the physical death of all men, right?
It is mortality (i.e. physical death) that has been inherited from Adam.
Some don't agree with me, and maybe you also, but I think that if Adam and Eve had never eaten of the wrong tree, they could have lived, physically, forever.
I do agree. Mankind was designed originally to be immortal. Indeed, had they eaten of the Tree of Life they would have been immortal in spite of their rebellion, which is the reason why God gives for removing them from the Garden.
Yet, the evidence presented was that all death is the result of Adam's sin, including physical death, right?
Not directly. Our physical mortality is directly inherited from Adam but we are not born spiritually dead as the Catholics and Calvinists both teach.
Deuteronomy 1:39 "Moreover your little ones and your children, who you say will be victims, who today have no knowledge of good and evil, they shall go in there; to them I will give it, and they shall possess it."
Isaiah 7:16 "For before the Child shall know to refuse the evil and choose the good, the land that you dread will be forsaken by both her kings."
Matthew 19:14 "But Jesus said, ‘Let the little children come to Me, and do not forbid them; for of such is the kingdom of heaven.’"
Romans 7:9 "I was alive once without the law, but when the commandment came, sin revived and I died."
Ezekiel 18:20 "The soul who sins shall die. The son shall not bear the guilt of the father, nor the father bear the guilt of the son."
Or are some of those people between Adam and Moses still alive somewhere (please ignore Enoch for a moment, because he's a singularity)?
I.e., if physical death is part of what came from Adam's sin, then we would expect that everybody between Adam and Moses, including infants, would die physically. And that's exactly what we see. And according to you, if Paul was referring to spiritual death, then he wasn't telling the truth, because there would have been perhaps many infants who never died spiritually, yet Paul said "all between Adam and Moses "died".
The first thing I would say here is the the word "all" almost never means "every single one". It is one degree or another of hyperbole nearly every time it's used.
Secondly, your interpretation here would directly contradict Paul's own teaching as I quoted above in Romans 7:9.
Thirdly, God is just and He is the One who has been wronged when someone rebels. It would, therefore, not be beyond His right to bring someone from this life to the next whenever He sees fit to do so, whether that equates to a normal physical death or not. Generally, it is appointed unto man, once to die and then the judgement but there does seem to be at least two exceptions to that; Enoch and Elijah but even they may yet die physically. Some believe them to be the two witnesses spoken of in Revelation 11:3-12.