The team finished second but the final handover between Daniel Talbot and Adam Gemili, 18, was not completed in time.
Gemili said: “Maybe I went early or too hard. It’s disappointing as we could have been in contention in the final.”
Favourites Jamaica rested Usain Bolt but still won heat one, while, in the second, the United States recorded the fastest overall qualification time.
Britain were disqualified in Beijing four years ago and have now made a mess of baton changes in five of the last six major championships.
Talbot said: “I don’t know what happened. I just couldn’t catch Adam. I’m really gutted and it’s devastating.”
Christian Malcolm, who ran the first leg for Britain, described the outcome as “unfortunate”.
He added: “It’s one of those things that happens but it’s a fantastic performance to run the time we did. Daniel has come in at the last moment. This is a big stadium and it can be overwhelming.”