If you are using a website to lookup the location of your IP and find that the location does not match the server that you are connected to then the geolocation data from the website is inaccurate/out of date. These websites purchase geolocation data from specialized providers who collect this information from various sources.
For geolocation providers determining the country based on an IP is relatively accurate (95%-99% percent) since the country is required information when an IP range is allocated and IP registrars supply that information for free. However, determining the physical location down to a city is more difficult and less accurate because there is no official source for the information and ISP's often base IP addresses in a city where the company is basing its operations. Accuracy rates for cities is 50-80% according to dnsstuff.com. If you would like a more accurate tool to determine the location try using - http://www.yougetsignal.com/tools/visual-tracert/, however even this tool can be inaccurate.