This map shows Ipswich as it had probably developed by as early as 725AD. There do not seem to have been any defences needed until after the Danes arrived, c 879.
Ipswich in 1086
By 1086 Ipswich had at least 12 churches, and several of these were outside the defences, located in the undefended suburbs. There was an early Norman castle built somewhere west of St Mary Elms, but it had probably been demolished by 1176.
Sources:
"A History of Ipswich" by Robert Malster, published by Phillimore, 2000 AD