Downriver from the Millennium Bridge is Southwark Bridge. Dating from 1921, it's the least-used bridge in central London, because there's no straight-on access to the city for motor traffic at its north end. However, bikes can slice straight through a series of bike/ped crossings along Queen St as far as the Guildhall.
The bridge also has a wide separated cycle lane on each carriageway, making it a pleasant bike crossing.
In Fruiterer's Passage, the riverside pedestrian alleyway that ducks underneath the bridge on the north side, there are some tiled murals (below right). They depict the bridge as it used be, as seen from a building site which due to health and safety was evidently a top-hat area.
From here it's all Dickensian alleys and wharves along either bank, underneath and past Cannon St railway bridge. The north bank is narrow walkways with access alleys taking you up to street level. The south bank is more touristed, twisting and cobbled, taking you past Vinopolis, the Clink (which gave its name to the traditional sound made by toasting glasses), and the Golden Hinde (a replica of the ship that Sir Francis Drake used to take bowls to the world in exchange for tobacco, potatoes and slavery); if you want to avoid steep steps to access London Bridge you have to go round the far side of Southwark Cathedral. Either way, London Bridge is under half a mile away.