Going up to town and down to Scotland

You always go up to London (or up to town, which is non U but was U In the 17th century) and down to everywhere else except when you matriculate at Oxford and Cambridge. This apparently confuses Americans.