Thursday, October 9, 2014

Using ArcREST to Export Hosted Feature Services

THIS POST IS FOR RELEASE ARCREST 2.0 AND WILL NOT WORK PERFECTLY FOR ARCREST RELEASE 3.0.

When you upload a feature class to ArcGIS Online (AGOL), you have the option to publish those feature classes. Now let's assume you have enabled editing on that data set. The question is how do you pull down the data so you can have an updated version in office? Well luckily for us, you have ArcREST to automate this task! ArcREST can be found here (www.github.com/Esri/ArcREST) if you do not have it.

This example will show how to automate the download of a hosted feature class as a shapefile to a centralized storage location.

The workflow is as follows:

  1. Connect to the AGOL site
  2. Export the item to a shapefile
  3. Download the item to disk
  4. Erase the exported item

import arcrest
import uuid
import os
import time
#   Inputs
#
itemID = "THE ITEM ID TO EXPORT"
username = "SOME USERNAME"
pw = "MY PASSWORD"
url = "http://www.arcgis.com/sharing"
filePath = r"c:\temp"
#   Logic
#
#  Create security handler
shAGOL = arcrest.AGOLTokenSecurityHandler(username, pw)
#  Connect to AGOL
org = arcrest.manageagol.Administration(url=url, securityHandler=shAGOL)
# Grab the user's content (items)
content = org.content  
usercontent = content.usercontent(username=username)
#  Create a export item with a random name using UUID
#  and export it to a zipfile
#
result =  usercontent.exportItem(title="%s" % uuid.uuid4().get_hex(),
                                     itemId=itemID,
                                     exportFormat=exportDataAs,
                                     exportParameters=None)

exportedItemId = result['exportItemId']
jobId = result['jobId']
exportItem = content.item(itemId=exportedItemId)
#   Ensure the item is finished exporting before downloading
#
status =  usercontent.status(itemId=exportedItemId, jobId=jobId, jobType="export")
    
while status['status'].lower() == "processing":
    time.sleep(3)
    status =  usercontent.status(itemId=exportedItemId,
                                 jobId=jobId,
                                 jobType="export")
filePath = exportItem.itemData(f="json", savePath=filePath)
#   Erase the exported item to clean up
#   AGOL
#
usercontent.deleteItems(items=exportItem.id)
del exportItem
print 'finishe!'

So what we have is a quick little script that downloads a file to the temp folder.  The key thing to look at is the while part of the code.  When perform actions like publish, generateFeatures, export, and createService, they are performed asynchronously on the system.  This means that you need to check if they are finished before performing further actions on the items.  The status method returns a dictionary of values, and you need to ensure that the value is not equal to 'processing', or any further actions like itemData() will return invalid information or data.  In the case of the itemData() you will download an empty zip file.

Enjoy and happy coding.

17 comments:

Anonymous said...

Its not working for me.! also tried with Featureservice and layers but nothing getting cracked! kindly help

Anonymous said...

Is this line correct:
org = arcrest.manageagol.Administration(url=url, securityHandler=shAGOL) ?
My python offers me no "manageagol"...

Andrew said...

The package was renamed. This example was from version 1.0 of the package. You need to use manageorg instead of manageagol.

Anonymous said...

Do you know a possibility to get the itemID by an tag? I tried content.getItemID, but my syntax is always invalid.

Andrew said...

You could do a query on the organization and search by tags.

This would give you back a list of items. Inside the item JSON, there is an item id.

Caleb said...

This is exactly what I'm looking for but am receiving the following error when "import arcrest":

Traceback (most recent call last):
File "", line 1, in
import arcrest
File "C:\Python27\ArcGIS10.2\lib\site-packages\arcrest\__init__.py", line 1, in
import agol
File "C:\Python27\ArcGIS10.2\lib\site-packages\arcrest\agol\__init__.py", line 4, in
import helperservices
ImportError: No module named helperservices

Caleb said...

Apparently the issue with the help services has been resolved!
https://github.com/Esri/ArcREST/issues/61
Thanks anyways

Caleb said...

Unfortunately, I still haven't gotten this script to work. I am now receiving the following:

NameError: name 'exportDataAs' is not defined

I'm new to ArcREST and any insight would be appreciated.

Caleb said...

Sorry for the barrage of posts. I'm new to ArcREST and have been slowly figuring things out. I replaced exportFormat=exportDataAs with exportFormat= "File Geodatabase" and changed exportItem.itemData(f="json"... to f="zip". The script downloads a zip file but it is only 1kb. Any thoughts? Thanks.

Andrew said...
This comment has been removed by the author.
Andrew said...

Can you post your code and issue here so I can track it?
https://github.com/esri/arcrest/issues
Thank you

Caleb said...

I've posted code and issues to github. Thanks!
https://github.com/Esri/ArcREST/issues/62

Anonymous said...

Is there a reason you do not just use a layer file? I created a layer file that points to the hosted feature service then use that in any arc tools I want (like append). Seems to work great.

STEPH LONG said...

content.usercontent results in an error of:

exceptions.AttributeError: 'Content' object has no attribute 'usercontent'.
It is not a drop down option. Only 2 proxy calls, self, and securityHandler.

I am using "manageorg" as was suggested above.


Anyone else able to assist?

Anonymous said...

I encounter the same issue as @STEPH_LONG. Is anyone available to assist us? Thanks.

Jane W said...

usercontent.exportItem results in the following error. I tried user.ExportItem but same problem.


AttributeError: 'function' object has no attribute 'exportItem'

Any help appreciated.

Jane

gisbert said...

I was unable to get this script to work but found https://github.com/Esri/ArcREST/blob/master/samples/create_replica_portal_item.py
which works well and does the same thing; including the picture. However the script at github doesn't cope well when having multiple pictures in one record.