Data and Basemap Creation for MobileMap

<none>

Creating Base Map Data for MobileMap

MobileMap utilizes two broad categories of data - operational data and base map data.  Operational data are stored in ArcGIS Feature Services and contain the data model for collecting and editing data.  These data are uploaded and downloaded from the feature service when a user is online.  Base map data, on the otherhand, are read-only data layers that provide background imagery and related data (e.g., ownership boundaries, streams, roads, etc.).  MobileMap supports a range of file formats for base map data, including Esri Tile Packages (TPK), Esri Compact Tile Caches (CTC), MBTiles databases (MBTiles), Shapefiles (SHP) and Comma Separated Value (CSV) files.  The most common formats used in MobileMap are TPK and SHP, and creation of these files are described in the sections below.

TPK Creation

Esri Tile Packages (TPK) are tiled images stored at multiple zoom levels.  The zoom levels follow the standard Esri/Google/Bing tiling system which is best described here .  This tiling system uses the Web Mercator projection and zoom levels range from 1 (lowest resolution, whole earth stored in 4 tiles that are 256 pixels x 256 pixels each) to 23 (highest resolution, each pixel represents 1.87 cm of the Earth's surface).  There are several ways to create TPKs, including in ArcMap, ArcGIS Pro and MobileMap.  The sections below describe the process for creating TPKs in ArcMap and MobileMap.

NOTE:  The TPK format is very similar to Esri's lesser known Compact Tile Cache (CTC format) mentioned above.  In fact, a TPK is merely a Zipped folder containing a CTC, along with some related metadata.  In the early days of of MobileMap (circa 2013) we routinely converted TPK file extensions to .zip, unzipped them, and extracted the CTC.  There are small performance gains from using these unzipped data.  In recent years, with the advances in mobile device hardware, we have not found this step to be necessary.  If you are using very large TPKs and seeing a performance issue when loading or panning base map data, reach out to our support team for instructions on how to create and use CTCs instead of the easier to use TPK format.

Creating TPKs in ArcMap

NOTE:  As of ArcMap version 10.2, it is no longer possible to create tile packages (TPKs) that contain ArcGIS Online hosted base maps.  Prior to ArcMap 10.2 (e.g., ArcGIS 10.1) it was possible to include these data as inputs to tile packages.

Steps for creating a TPK in ArcMap

  • Open a map document in ArcMap.  Change the coordinate system of the data frame (“Layers” in the example below) to WGS 1984 Web Mercator (auxiliary sphere).  This option is found under Projected Coordinate Systems -> World -> WGS 1984 Web Mercator (auxiliary sphere). 

           

 

  • Add base map data as needed from existing imagery or other inputs.  Note that raster data that are not from streaming services should be projected into WGS 1984 Web Mercator (auxiliary sphere) coordinate system before adding them to the map when creating TPKs. Vectore data, including points, polylines, polygons from Shapefiles or Feature Classes, do not need to be re-projected as long as the data frame projection has been set to WGS 1984 Web Mercator (auxiliary sphere).  ArcMap will project vector data as needed (on-the-fly) when building the TPK.
  • Set the symbology (colors, display scale levels, labels, etc) for all layers.
  • Add metadata to the Map Document Properties (File -> Map Document Properties) Summary and Description boxes (this is required for the Create Map Tile Package Tool) and save the map.

         

  • Open the ‘Create Map Tile Package’ tool.
  • Select the map document you want to cache (make sure the data frame shows the entire area you want to cache) and that you have saved the MXD since your last change
  • Select the name for the output file (it will be a tpk file format)
    • We recommend using a naming convention that includes the zoom level (see next step) as well as the source or type of data (e.g., Oregon_NAIP_2017_L18.tpk)
  • Set the level of detail to the desired level. Level 16 to 17 is standard for forestry work, while a higher level like 19 might be more appropriate for wetlands work.  
  • Click OK to run the tool. 

           

  • After the tool completes its process, navigate in Windows Explorer to the folder where you saved the output .tpk file.
  • Follow steps below to store the TPK you created into the correct directory structure for use in MobileMap.

Creating TPKs in MobileMap

Note: This functionality is not available if your organization uses Portal for ArcGIS.

MobileMap includes a Download Base Map tool as a convienient method for obtaining base map data, including imagery and topo maps.  When downloading TPKs from the Download Base Map tool, MobileMap is accessing the 'export' function from Esri's base map services.  This functionality is limited to a maximum number of tiles, and there are additional practical limits on total file size of an exported TPK file.  Keep in mind that the size of a TPK is a function of the extent and the maximum zoom level.  For large base maps (e.g., high resolution imagery for a whole county or state) use ArcMap and an imagery source that you have local access to (e.g., a download of NAIP data).  If you need high resolution data for a demo or small project, use this approach and a high zoom level (e.g., 19) but limit the spatial extent of your download to a very small area.  If you want to have a large basemap just to serve as a frame of reference, use a large extent but limit the maximum zoom level to a very low level (e.g, 6 or 8).  While there is no hard rule on the maximum extent or zoom level combinations that are supported, we have found that TPKs larger than 100 MB in size do not always create or download property.  To be safe, try to limit the size of downloaded TPKs to under 50 MB.

In addition to the caveates above, there is one additional problem with TPKs generated via this tool - they include a maximum zoom level parameter in their metadata.  This parameter causes the TPKs to disappear if you zoom in beyond that zoom level.  So, rather than getting pixelated or blurry, they simply disappear.  In many cases this is not the desired behavior.  TPKs created in ArcMap (process described above) do not have this limitation.  If you want to use the Download Base Map tool in MobileMap but do not want your TPK to disappear at higher zoom level, there is an optiona step that can be performed to remove this maximum resoluton parameter from the TPK file, but it is a bit complicated.  See the optional instructions at the bottom of this section for more details.

To create TPKs in MobileMap use the Download Base Map tool by following these steps:

  1. Sign in to ArcGIS (Actions > SIGN IN TO ARCGIS)
  2. Open the Dwnload Base Map tool (Actions > Download Basemap)
  3. Select the layer you would like to download from the dropdown list
  4. Pan and zoom the map to the desired extent
  5. Select the maximum zoom level that you would like to download.
  6. Select DOWNLOAD
  7. When the download completes, note the directory where the file was downloaded.  Typically this will be the Download folder on your internal storage
  8. Use a file manager (e.g., MyFiles, Astro, etc.) to copy the TPK to a map cache's TPK folder (see instructions below for more information about file structure)

Optional - If you want to use the Download Base Map tool in MobileMap but do not want your TPK to disappear at higher zoom level follow the steps below to enable "overzoom" within a TPK:

  • After step 7 above, and before saving your file into it's final destination you will need to modify the TPK
    • Copy the TPK to a PC where you have access to a Zip file creator application (e.g., 7-Zip) and a text editor
    • Rename the TPK file extension from *.tpk to *.zip
    • Unzip the file using any unzip tool (default Windows capability fine for this step)
    • Navigate within the resulting folder to \servicedescriptions\mapserver
    • Open the mapserver.json file in a text editor and do the following
      • Search for the text "maxScale"
      • Wherever this property exists and is not set to zero (typically only in 1 place, the 6th instance of "maxScale", the last property in the "contents" property) it will be set to the map scale that cooresponds to the zoom level you selected when building the TPK.  For example, if you selected zoom level 15, you will find one or more instances of  "maxScale": 18055.95 which indicates that the maximum map scale is 1:18,055.
      • Replace all instances of this with the map scale for zoom level 23, using: "maxScale": 70.53.  This will tell MobileMap (or any Esri software) that this TPK should be desplayed at all resolutions, instead of hiding these data at resolutions higher than the resolution they were created from.
      • Save the modified file mapserver.json
    • Re-zip the folder using your Zip application (e.g., 7-Zip), setting to no compression (e.g., compression format = 'storage').  This step is important, we don't want to actually compress the data, we just want to contain it into a single zip file.  If compression is used the resulting file will not be readable by MobileMap or any Esri software.
    • Replace the *.zip file extension back to *.tpk
    • Copy the zip file to the correct destination folder in MobileMap (e.g. sdcard/MobileMap/myproject/TPK) and test to verify that you can now "overzoom" the TPK

 

MobileMap Directory Structure

  • The folder structure for MobileMap can be created directly on your mobile device, or can be created on a PC, then copied onto the device
  • Start by creating a parent MobileMap data folder called MobileMap.
  • Within the MobileMap folder create one subfolder for each project or 'map cache'.  Some users create new map cache folders for each geographic area where they work.  Other users create different map cache folders for different types of work, based on the base map data they like to have displayed when performing that work.  Still others use a single map cache for all of their work.  The strategy will depend on the volume of data that you need, the number of distinct geographic areas where you work, the degree to which larger work areas are separated into regions or other groups, etc.  For this example we will create two map caches, one for Washington and the other for Oregon.
  • For each map cache folder (e.g., sdcard\MobileMap\region1), create two subfolders, TPK and SHP.
  • Add shapefiles to the SHP folder and TPKs to the TPK folder. Note that Shapefiles must be projected into WGS 1984 Web Mercator (auxiliary sphere) to display correctly in MobileMap.

         

  • Repeat the steps above to create additional map cache folders and add in the necessary Shapefiles and TPKs
  • Optionally, create an additional map cache named 'Default' and include the same two subfolders: TPK and SHP
    • The default map cache can be used to store any data that should be used as a base map for all projects.  Typically a very large extend but low resolution TPK is stored in sdcard\MobileMap\Default\TPK to provide a common base map for all MobileMap use.  An example might be the entire continent at zoom level 6, or an entire state at zoom level 8
  • Once the data are correctly organized, copy the MobileMap folder from your desktop to the root folder of your SD Card. Removing the SD card from your device and using a USB to micro SD card adapter in your PC will be the fastest way to transfer data to the SD card, although data can also be copied over a USB cable
  • Below are examples of a typical file structure on an SD card for MobileMap.  The example below has two map Caches (OR and WA), each with 2 TPKs, and 2 Shapefiles. 

MobileMap Root Directory

Map Cache Directory (OR or WA)

SHP Folder Structure

TPK Folder Structure

 

CREATING AND PUBLISHING A FEATURE SERVICE

 

ARCGIS MAP DOCUMENT

  • Open a blank or existing ArcMap document. (This can be any ArcMap version 10.0+)

  • Change the Data Frame (Layers) coordinate system to WGS 1984 Web Mercator (auxiliary sphere).

  • Add all Feature Classes. These feature classes will be templates for your field data layers. 

  • Each feature must be projected to WGS 1984 Web Mercator (auxiliary sphere).  (MobileMap and ArcGIS Online only reads WGS 1984 Web Mercator (AS) projection)

 

DATA MODEL

The data model for MobileMap is defined in an ArcGIS Feature Service, which is typically created by designing a data model using an ArcGIS File Geodatabase and then publishing that feature service from ArcMap or ArcGIS Pro.  Data modeling in ArcGIS is a complex topic and requires some knowledge of Esri data types, Coded Value and Range Domains, required fields, etc.  MB&G recommends that you start with a template that we provide, and then modify that as needed to incorporate your specific data requirements. 

  • General Approach
    • Check the MB&G Template for your industry specific Data Model. MB&G recommends using (or at least starting with) a standard Data Model and Domains for a successful MobileMap experience.
    • Turn ON Editor Tracking for all layers in ArcMap (or ArcGIS Pro) prior to publishing your feature service.  Editor tracking is required for MobileMap to properly sync with the feature service.  Editor tracking must be enabled on all layers prior to publishing - it is not possible to enable editor tracking in ArcGIS Online for use with MobileMap.
    • To turn on Editor Tracking 
      • In ArcCatalog, naviagte to the geodatabase which contains all feature classes. 
      • Right click each feature class below the database and Select 'Manage'. 
      • In the 'Manage' menu, select 'Enable Editor Tracking'
  • Recording the user that created or last edited a feature
    • ArcGIS Online's Editor Tracking functionality will maintain some information about the ArcGIS User who created and last edited a feature.  It is important to note a few caveates with this capability
      • ArcGIS Online will record the date/time that the feature was saved to ArcGIS Online, not when it was saved in the field.  If a user collects or edits on a Monday, but doesn't upload until Friday, ArcGIS will record the date/time of creation or modification as Friday
      • Republishing a feature service with existing data in the File Geodatabase will change all created user and created dates to the user who is publishing the service and the date/time that the service was published, overwriting the original user and date/time for each record
      • To address these limitations, MB&G recommends using seperate fields for managing this important information
        • Observer - typically a 3-4 character Text field that records the initials of the user collecting or editing the data.  There are two settings in MobileMap (Settings > Data Model Settings > Observer Field, Settings > Data Model Settings > Observer Initials) which can be used to set the name of this field (the default is 'Observer' but other values like 'Cruiser_ID' can be used for the field name) and the initials of the device user.
        • Created - Date field that MobileMap will use to record the data/time at which a new feature was created in MobileMap.  This field will NOT be overwritten by ArcGIS when data are uploaded.  The name of this field can be changed in MobileMap settings (Settings > Data  Model Settings > Custom Created Date Field Name, default value is 'Created').
        • Edited - Date field that MobileMap will use to record the data/time at which an existing feature was edited in MobileMap.  This field will NOT be overwritten by ArcGIS when data are uploaded. The name of this field can be changed in MobileMap settings (Settings > Data  Model Settings > Custom Edited Date Field Name, default value is 'Edited').
        • Created_Source - Text field that MobileMap will use to record the MobileMap version that was being used when a new feature was created in MobileMap.  This field can be very useful if muliple applications are being used to create data (e.g., ArcMap, ArcGIS Pro, ArcGIS Online, MB&G Spatial CMS, MB&G InventoryManager) as it will identify if MobileMap was the application that created a particular feature.  It can also be useful in debugging if issues arise during data collection, as data managers can see if an issue is only occuring on a certain version of MobileMap.  It can also help device managers keep track of what versions of MobileMap are being used in the field.  The name of this field can be changed in MobileMap settings (Settings > Data  Model Settings > Created Source Field Name, default value is 'Created_Source').
        • Edited_Source - Similar to Created Source, this Text field records the the MobileMap version that was being used when an existing feature was last edited. The name of this field can be changed in MobileMap settings (Settings > Data  Model Settings > Edited Source Field Name, default value is 'Edited_Source').

SYMBOLOGY

Getting the correct Symbology for MobileMap is a critical step to view your data. MobileMap only handles Simple symbols types for Point, Lines and Polygons. 

For point features, 

  • In your Map Document, double click on the point symbol to open symbol selector window. 
  • Select  'Edit Symbol'

  • After clicking on Edit Symbol, ensure the type is set to 'Simple Marker Symbol'. 
  • Select the style of the symbol from the drop down for Style. 

 

 

  • Once the Symbol is selected, Click on save.
  • Similar to points, lines must use Simple Line Symbols and Polygons must use Simple Fill Symbols.
  • Similar steps are to be performed for Lines and Polygons. (Note that Lines and Polygons can only have solid or dotted Style).

 

PUBLISHING TO A FEATURE SERVICE

Once the symbology, projection, data model and Editor Tracking is set and saved, follow the steps below to publish the map document as a feature service in ArcGIS Online. 

  • Make sure to zoom out to an extent that covers all of the area for the relevant data to be published. For example, if you have data which consists of all counties in California, zoom out on the map to show California, Oregon and Mexico.
  • Once zoomed out, save the map document and do not zoom back in before publishing.
  • Ensure that you are logged in to your ArcGIS Account in ArcMap. To login, go to File -> Sign In, or go to www.arcgis.com to create an account.
  • Share the feature service by navigating to File -> Share As -> Service.  Select ‘Publish a service’ and name the service (pick a short name, all lower case with no spaces). Work your way down the left panel.
    • Under Capabilities, check Feature Access and uncheck Tiled Mapping.  
    • Under Feature Access, check the boxes next to Create, Delete, Query, Sync, and Update.  (Figure 11)
    • Under Item Description, add text in the Summary and Tags. 
    • Under Sharing, choose the level of Sharing.  Selecting Everyone makes the feature service a ‘public’ service, meaning anyone could potentially find the service if they know what to search for on ArcGIS Online.  Selecting your organization limits the sharing to just your group. (If you select your organization, you will need to put your ArcGIS Online user name and password in the settings of MB&G MobileMap).  
  • Finally, select Publish in the upper right corner.

 

  

 

   

 

Your feature service is now published.  Log in to ArcGIS Online to review and confirm it is as expected.

Users may now access the feature service using MobileMap. Detailed instructions for accessing feature services in MobileMap can be found in the Quick Start Guide

https://mbgmobilemap.com/quick-start-guide-standard

 

OTHER WAYS TO GET BASEMAP IMAGERY

 

- DOWNLOADING IT WITHIN MOBILEMAP

- HIGH RES IMAGERY DOUBLE CACHING

 

Contact MB&G for details on how to obtain basemap imagery from MobileMap or double caching

  • Phone
    • Call our dedicated MobileMap support line +1 503-290-0542
    • If no one answers, please leave a detailed message
    • Voice mail checked regularly during work hours, Monday - Friday, 8am- 5pm (PST)
  • Email