Finding where a photo was taken using Google Vision

Scanning photos taken by grandparents on a trip to Europe in 1960 is quite fascinating. Seeing the places they visited, some which I have also had the opportunity to visit as well, is a nice way to remember them. However, the photos presented a bit of a challenge- while some had handwritten captions mentioning the location they were taken, most lacked this information.

Modern digital cameras embed metadata into an image file with details such as the date/time taken and geolocation. These bits of information make it easier to know where a photo was taken. So with these analog photos- I have only a black and white image of an unknown place.

1960’s photos meet Artificial Intelligence

Luckily, modern image recognition techniques can identify landmarks. Google’s Vision API can identify an image’s location by using AI smarts to compare them to a vast image dataset. As more images are fed, Google Vision “learns” more about landmarks and the objects contain within the images. While the API is targeted for application developers, Google provides a web site in which you can upload an image and it returns this information as well.

Google Vision API Website
The Google Vision API provides a way of identifying landmarks from photos. In this example, the service correctly identified the photo as taken at the Buen Retiro Park in Madrid, Spain – https://cloud.google.com/vision/

In mobile devices, Google Lens also uses this technology to recognize landmarks and present relevant information.

Google Lens
Google Lens uses Google Vision to identify landmarks from images.

 

Adding location information to the image file metadata

After positively identifying the landmark and geographical location where the photo was taken using Google Vision, I turn to GeoSetter for adding the missing geolocation information as well as captions to the image file metadata. Adding this information allows other applications and services to use it as well.

 

GeoSetter
Adding geolocation information to an image file using GeoSetter – http://www.geosetter.de/en/main-en/

 

OneDrive
Applications or services as OneDrive can use geolocation data to display the location on a map. The caption is also displayed showing the name of the monument as it was identified with the help of Google’s Landmark Recognition engine.
2018-05-22 (1).png
Geotagged photo shown in Windows Photo Gallery – the added metadata of the location taken is shown as a Geotag.

Related:

Moving away from Windows Photo Gallery Geotags

Windows Photo Gallery “Geotags” was a nice feature for adding location information to photos. It is no longer supported and has caused some angst among some users. So I am writing this post to provide a way to access and maintain the geo location information in your photos.

First, a quick explainer on how geotags are saved in image files and used to work in Windows Photo Gallery (WPG) – As you may know, photo files such as JPG can store additional information called metadata. The best analogy for understanding what metadata is by imagining a printed picture with written information written on the back. There are multiple ways (formats) of writing this information known as EXIF, IPTC, XMP. Think of it as paper forms with fields (such as date, time, caption, locations) to fill in, which are added to a photo.

WindowsPhotoGallery_Geotag
A photo taken at the Seattle Central Library shown in Windows Photo Gallery. Placing the mouse over the geotag would reveal all the geotag information.

The fields which Windows Photo Gallery uses for geotags are: IPTC Core and Extension Location Fields (Location, City, State/Providence, Country) and EXIF GPS Latitude, GPS Longitude.

Here is the general logic behind Windows Photo Gallery’s Geotags (in order):

1. If a photo already contains Location, City, State/Province and Country fields filled in, WPG would use the information it has to display as the Geotag. It would display the IPTC Extension fields as the first choice (if available) and the IPTC Core “legacy” fields (if available) as the second option.

2. If a photo has GPS Latitude and Longitude, but no City, State/Province, or Country information in the fields described in (1), then WPG would consult a Microsoft web service to determine the “Geotag” based on latitude and longitude, this process is called reverse geocoding. It will not however, write the “Geotag” information back to the file, unless they are edited manually by the user.

3. If you, the user, edit a Geotag in WPG manually, it will be saved back to the file and that information becomes the “geotag”. However, here it is where it gets a bit tricky- There are multiple Location, City, State/Province, Country fields among the different metadata standards (IPTC and within XMP). WPG uses the “IPTC Extension- Location Created” fields to store the “Geotags” info within the photo.

IMG_3394
Windows Photo Gallery “Geotag” rename dialog box. When edited, the information is saved back to the file to the IPTC Extension “Location Created” fields.

Here is an example of the metadata from one of my photos. You will note multiple fields which may seem to contain redundant information, but the ones WPG would write to are just four which start with “Location Created”. These fields are detailed in the 2008 IPTC Extension Spec, further discussion on how this information is handled by certain applications can be found on the Metadata Working Group Spec.

Location Created City Seattle
Location Created Country Name United States
Location Created Province State Washington
Location Created Sublocation Seattle Central Library

Given that Microsoft decided to discontinue WPG last year, the service which determines the geotags is no longer working, so holdout users may find these fields empty or have trouble adding new geotags. Hence, exiftool, Lightroom or other photo management applications will only be able to show you geotag information if it was written within the file.

GeoSetter to the Rescue

As a workaround, I rely on a freeware application called GeoSetter (which internally uses exiftool). Geosetter uses a service from Geonames.org, in a similar fashion which WPG did, to aid in finding location info as well it allows users to edit them manually. I exclusively have been using GeoSetter for some time, which unlike WPG it will always save the information back to the photo file. GeoSetter also allows one to edit the latitude and longitude info, which I was unable to do so in WPG.

GeoSetter_Edit_Location
GeoSetter’s Edit Data dialog allows users to edit EXIF GPS as well as IPTC Location fields.

However, as good as the Geosetter is, there is a slight catch – Geosetter by default does not read nor write to the newer 2008 IPTC Extension format which Windows Photo Gallery writes to. This is not just an issue with Geosetter, other photo management tools, like XnView MP, still primarily use the IPTC Core “legacy” Location fields, instead of the newer IPTC Extension Location fields. The IPTC Photo Metadata User Guide, explains the reasoning of creating the IPTC Extension fields. Luckily, Geosetter is versatile enough to provide a means for a workaround.

To make GeoSetter read any existing WPG Geotag – You will need to copy any saved “geotag” information from the IPTC Extension, to the equivalent IPTC Core fields using an exiftool command. In a previous post I described how to use exiftool for accessing Windows Photo Gallery. You will only need to run this step once on your photos from the command line. I advise you to make a backup copy of your files before executing it.

exiftool *.jpg -"Location<LocationCreatedSubLocation" -"City<LocationCreatedCity" -"Province-State<LocationCreatedProvinceState" -"Country<LocationCreatedCountryName" -r -overwrite_original
-r means that it will execute recursevly, so that all subfolder are included.
-overwrite_original means that exiftool will not create a backup of the file.

Again, this will only work on file which WPG saved the geotag information to the file, otherwise the fields will be empty.

To configure GeoSetter write back information to the newer format when an edit is made– In the GeoSetter application menu, go to File | Settings. In the Settings dialog, go to the “ExifTool” tab. Ensure that the “Use Additional Exiftool Commands after GeoSetter command” checkbox is enabled and add this text to the text field underneath:

-execute -"LocationCreatedSubLocation<Location" -"LocationCreatedCity<City" -"LocationCreatedProvinceState<Province-State" -"LocationCreatedCountryName<Country" -"LocationCreatedCountryCode<CountryCode"

 

GeoSetter_Exiftool_Settings
GeoSetter’s Settings dialog. Adding additional exifool commands allow custom actions such as copying field values to another. In this case location information is copied to IPTC 2008 Spec fields.

 

If you are still a die hard Windows Photo Gallery fan, you will find that with these tips the geotag information populated with what you was entered by GeoSetter. While the reverse geocoding capability may be gone, you can still use WPG sort by GeoTag and Search functionality as before. Still, looking forward, I would avoid editing geotags in WPG and rely instead on GeoSetter as a replacement for this functionality in order to avoid the problems previously described.


References:

Related Posts:

Accessing Windows Photo Gallery Metadata using Exiftool

With the demise of support to Windows Photo Gallery, it may be a good time to plan an app migration. I have seen various posts and comments on this, particularly after all the time and effort you may have spent tagging photos. The app which is meant to replace Windows Photo Gallery is the Windows Photos App. Unfortunately, the Windows Photos app does not support Captions, Descriptive Tags, People Tags nor GeoTags and to date I have not found any information which points that those features will be supported in the future.

So do not fret, as one of the pluses of using Windows Photo Gallery is that most of this information is stored into the image file and is accessible in Windows Explorer as well as any other apps which support photo metadata standards. Only in some instances this data is not stored to the file itself, such as “Flags” or if the image file in PNG or is set to “read-only”.

Understanding where the metadata is stored

Windows Photo Gallery stores much of this information as photo metadata using standards such as EXIF, IPTC IIM, and XMP (IPTC Core, IPTC Extension, Dublin Core and Microsoft Schemas) . I will not delve into the specifics of these standards but it is worth knowing to which fields the information is mapped to. In some occasions some information may be stored in multiple fields. For example, WPG captions information is stored in the XMP:Title, XMP:Description as well in other IPTC and EXIF fields.

The following table lists some of the photo information displayed in Windows Photo Gallery and the corresponding metadata fields from where the information is read and written to. I created this table from plain observation, if you find any other behavior let me know.

For reading and writing to these tags I highly recommend the excellent Exiftool command line utility. It may be daunting for some users to use the Windows Command line so I will try to be as specific as possible.

Reading Windows Photo Gallery Metadata using Exiftool

Installing Exiftool

  1. (For Windows Users) Download the stand-alone Windows Executable from the Exiftool site.
  2. Unzip the exiftool(-k).exe and rename it exiftool.exe
  3. Copy the exiftool.exe to your c:\windows folder or to a folder contained in the PATH variable so it is accessible in any directory/folder. Detailed installation information can be found here.

Using Exiftool

  1. Open your command prompt in a folder containing photos either by running the Command Prompt shortcut in the Windows App Menu or by right mouse clicking on a folder while pressing the <Shift> key  and selecting “Open Command Window here”.
  2. Confirm you have the current exiftool version by typing into the command prompt:
    exiftool -ver
  3. Pressing <enter> will return the version number.

Just typing the exiftool and the image file name will bring a host of information.

Example:

exiftool IMG_1234.JPG

 

Say you wish to view the Windows Photo Gallery Captions of all the jpg photos in a folder, just open the command prompt in that folder and enter:

exiftool *.jpg -xmp-dc:title

 

View all the WPG Descriptive tags:

exiftool *.jpg -xmp-dc:subject

 

Additional information on using Exiftool can be found in the following links:


Exporting Windows Photo Gallery information to a File

This example will produce a file named “PhotoMeta.txt” which will contain a table with the filepath, filename, date and time photo was taken, camera maker, camera model, wpg caption, wpg descriptive tags, and wpg people tags contained in the current directory and its subdirectories.

Example:

exiftool -T -Directory -Filename -DateTimeOriginal -Make -Model  -xmp-dc:Title -xmp-dc:Description -xmp-dc:Subject -RegionPersonDisplayName -r *.jpg > PhotoMeta.txt

Using the “-T” option will output tab-delimited results. The “-r” option is used to include all the subdirectories contained within the specified “Pictures” directory. You can later open this table in Excel or import it to a database such a Microsoft Access for quick analysis.

Using IPTC location identifiers to link your photos to knowledge bases

In 2014, IPTC released an update to its photo metadata standard which supports the adding of “location identifiers” to photos. In simplest terms, a url can be added to the photo metadata which links the photo to geolocation identifiers (in the form of a URI) provided by services like WikiData, Foursquare, Facebook, DBpedia, Geonames, among others- a form of linked data

Wikidata_LocationIDWhy is this interesting?

As more cameras are GPS enabled it aides in establishing the “where” the photo was taken. A common example, is when you take a photo with your smartphone and share it on Instagram. During the upload process, Instagram provides a way to add the location where the photo was taken. Instagram uses the GPS data of the photo and looks up locations from the Facebook Places database (Instagram switched from using Foursquare to Facebook Places after being acquired by Facebook). After your photo is uploaded Instagram the photo is “linked” to that location. You can quickly see the photos you and other took in that same location. This IPTC standard simply allows you to save that association within the file.

Now, the benefits from this may not become entirely clear now but if such a “location identifier” is already included within the photo it could bring with it some benefits. From the previous example, it could ease the upload process since Instagram or any other photo service may be able to deduce quickly the location of a photo. I am not sure of any photo services which does this yet, but for example, Flickr for some time has implemented something similar using machine tags

In Flickr if you add a tag to a photo pointing to a foursquare location in the form of foursquare:venue=, Flickr recognizes the location and adds a map with the foursquare location.

A Smarter Search

Say you attend a basketball game in Madison Square Garden, and take some photos. Your smartphone recognizes where you are and “tags” the photo with the WikiData entry for “Madison Square Garden” (Wikidata Q186125, Foursquare VenueID). You upload a photo to an online photo sharing service, such as Google Photos and perform a search for “basketball game” or the team which played on that date. Given that the Wikidata identifier is associated with the type of venue and the type of games played there, a search engine with some “smarts” can deduce that a basketball game was played on the date you took that photo. Similarly, if Google Photo employs some of its image recognition on the photo it can have some additional information to make a better analysis of the image.

Also, a location may have different names and may be spelled differently in several languages. Linking a location to a Knowledge Base such as Wikidata allows a search engine to better understand the context of a photo.

So you see, the IPTC Location identifiers opens up some interesting scenarios and it is quite useful for photo archivists, librarians or a home user which may also be looking for ways to link photo to similar knowledge bases.

Adding Location Identifiers with Exiftool

Using Exiftool it is easy to add Location Identifiers. Simply use the LocationCreatedLocationId or LocationShownLocationID tags as follows:

exiftool <filename> -"LocationCreatedLocationID+=https://foursquare.com/v/madison-square-garden/4ae6363ef964a520aba521e3"
*The += will add values to the Location Created List.

Geotag Photos with Foursquare / Swarm / Instagram and Facebook Check-Ins

Following up on my previous post on Geotagging Photos with Foursquare / Swarm Check-Ins, I stumbled upon an interesting scenario- How about a solution in which you could use your Foursquare/Swarm check ins, but the location information from Foursquare, Facebook, and Instagram.

I did some research and found possibly the best app for managing such data- the aptly named PinDrop. With Pindrop you can aggregate your Foursquare, Facebook, and Instagram locations as well as add your own. The locations, or pins you add on the app can be either shared or kept private. This is great for when travelling and if you want to keep track of the places you visited, but don’t necessarily wish to share it with your friends.

Now, for the frosting on the cake – you can export this information to a KML file which you can load into Geosetter. So far you can only export the data by visiting the website settings section. With the KML file you can more easily append the location place names to your photo’s metadata.

Now, combine this app with Geotag Photos Pro, which can record your location for easier location tagging, and you have a pretty interesting photo geotagging solution.

Happy geotagging!

Geotagging Photos with Foursquare / Swarm Check-Ins

Your are on vacation, so take a lot of photos with your camera and mobile phone. You keep your friends and family up to speed on your whereabouts by checking in on Foursquare / Swarm.

Now you arrive home and want to organize the photos you took on your PC. If you like to geotag your photos and update the photo metadata geo locations here is a helpful tip.

You can use the foursquare location names of your check-in and add them to the location fields of the photos you took.

1. Export your Foursquare check-in history in kml format  – Visit http://foursquare.com/feeds. Right mouse click on the kml feed link and select “Save As”. Save the file to your PC.

2. Load the kml file in Geosetter (http://geosetter.de/en)-  On the “Tracks” pane, click on the open track track files button, and load the track to GeoSetter. Your foursquare locations will appear as dots on the map.

3. Assign Position and/or Name to Photos – Load the photos you which to add the geographic information and click on the waypoints from the loaded locations. You can add the position information as well as the location name to the photo metadata.

If you then load the photos on applications such as Windows Photo Gallery or share the photos on sites (such as Flickr) which can read this metadata, the information will be shown.

foursquare Logo 800x800 viewer