Showing posts with label ArcGIS Online. Show all posts
Showing posts with label ArcGIS Online. Show all posts

Tuesday, April 28, 2015

ArcREST Basics - ArcGIS Online and Query()

In this weeks ArcREST basics series, I am going to go into the how to connect to your organization on ArcGIS Online (AGOL) using Python, and perform some basic queries using the manageorg sub-package in ArcREST.

First we start by building off the previous lesson and create a security handler object for AGOL.  If you missed a deeper discussion on how to handle security, please see this post.

One of the most common tasks to perform when working with your site, is to query it to see what content you have.  There are two ways to query a site:

  1. With Credentials - this means a token will be appended on the end of the query search, and users can find both public and non-public items if your users has the proper permissions to do so.
  2. Without Credentials - this means only find public items shared with everyone.
In the example below, a query is being performed with credentials being given.  This means case #1 will be applied.

from arcrest.security import AGOLTokenSecurityHandler
from arcrest.manageorg import Administration
if __name__ == "__main__":
    username = "username"
    password = "password"
    proxy_port = None
    proxy_url = None    
    securityHandler = AGOLTokenSecurityHandler(username, password,
                                               proxy_url=proxy_url,
                                               proxy_port=proxy_port)
    siteObject = Administration(securityHandler=securityHandler,
                                proxy_url=proxy_url,
                                proxy_port=proxy_port)
    results = siteObject.query(q="< some query string >")


This returns a Python dictionary object as shown below:

{
   "query" : "type:feature class",
   "total" : 12345,
   "start" : 1,
   "num" : 10,
   "nextStart" : 11,
   "results" : [ ...list of items... ]
}

Let us examine the results object.  It's a dictionary with many valuable key/value pairs.  Since this query was performed using the default values, this means that the query will begin with the first item and go to the 10th item (num).  The key nextStart tells you where the start value needs to be if you want to page each result.  So the next time you pass the query in order to get all the results you need to set the start = 11.  Manually this seems quite daunting task, but luckily for us, we have looks.  There are many way to create loops to perform this task, but in this example, a while loop will be used to walk the results and put them in a single list object.


def findContentByDate(admin, q):
    """
       finds items based on a query (q)

       Inputs:
    
          admin - manageorg.Administration object
          q - query string
    """
    start = 1
    count  = 0
    nextStart = 0
    num = 100
    results = []
    while nextStart != -1:
        query = admin.query(q=q,
                        sortOrder="desc",
                        sortField="modified",
                        t=None,
                        start=start + (count * num),
                        num=100)
        results = results + query['results']
        nextStart = query['nextStart']
        count += 1
    return results
Now we have all the items based on a query in a single Python list. This is very helpful in cataloging a site/org or even good for finding publicly shared items.

AGOL has it's own query language much like the google or other search engines.  This documentation can be found here (http://doc.arcgis.com/en/arcgis-online/reference/search.htm).  It is a must read for anyone who wants to query AGOL or Portal effectively.

Though the samples shown are created to work with Portal, then same example can be used to query a Portal site.

If you want to learn more, please visit the ArcREST Github Page.  Help us make it better and post comments, suggestions and improvements in the Issues section.  Or better yet, fork the repo and submit a pull request.

Happy Coding and Enjoy!

Wednesday, April 8, 2015

ArcREST - Bigger and Better

I've been working hard on ArcREST (https://github.com/esri/arcrest) these days, and the package is catching on I think.  Over the last couple of months, ArcREST has been shown at both Esri's Federal Developer Summit and at the Esri Developer Summit in California.  I truly think the effort is beginning to pay off, and users are reaping the hard work put into this package.

Simply put, this is a toolbox to manage, manipulate, and control your online GIS presence.  Portal, ArcGIS Server, and ArcGIS Online can now be controlled through Python!  The modules present in the Python package are like hammers, nails, and screw drivers.  They provide the framework to do greater things.  By themselves, they do not seem like much, but together you can build your GIS house through the common scripting language of the science community.

I am constantly looking for feedback on this package, so sign up on github, and test it out with a work flow you need in these environments.  If you find a bug, have a question, or need help, please post it in the issue area of github.  

Thank you everyone!


Monday, November 17, 2014

Workflow for Adding Files in Parts for Portal or ArcGIS Online

Python 2.7.5 is not very good at handling big files.  Actually it stinks especially if you need to upload them via a multi-part post.  There are many 3rd party solutions, but I found issues with both poster and requests with the tool belt add-in.  They simply didn't work.  Luckily for us, the folks who designed the AGOL/Portal REST API provided an alternative way of uploading large files.  The method is called Add Item Part.  The documentation can be found here: http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Add_Item_Part/02r300000094000000/.

When trying to figure out how to use this function, you also need to read the Add Item method as well to get how Add Item Part should be used.  The workflow is as follows:
  1. Call Add Item with no data (file data) and pass in multipart=True along with the filename=.extension.
  2. Break the file into parts, but they must be over 5 MBs except for the last chunk which signals an end of file. (I break everything into 50 MB chunks because I don't want to perform tons of POST calls)
  3. Call Add Item Part by using a POST.  Pass in the parameters partNum, which is a unique integer from 1 to 10,000.  You must pass that in, or each part will be overwritten.  Also, pass in the file via multi-part POST.
  4. Now call Commit function to tell the server, hey I'm done.  This is an asynchronous call, so don't forget to check the item's status before moving on, or you will not be able to update the item's properties.
    • If you look at your 'My Content' page on AGOL, you'll notice and item with everything stated as 'null' for the title, type, etc.. All you can do is delete the item right now.  This makes step 5 very important
  1. Update the item using the updateItem REST call.  Here we state the type, title, tags, etc.. all the good stuff you need to know so users can access the data.  

Great now we have the item in AGOL or Portal.  Simple right?  Not really, but it's a great way to get around Python's 2.7.5 annoying memory issue.  I haven't tested this on x64 python or python 3.4.  Hopefully some of my readers will post a comment letting me know if the memory issues with StringIO/cStringIO still exist when performing large file POSTs.


The add item part is in ArcREST today!

Get ArcREST: http://www.github.com/Esri/ArcREST
Also vote for my idea for a GUI builder for Python Add-Ins.

Thursday, October 9, 2014

Using ArcREST to Export Hosted Feature Services

THIS POST IS FOR RELEASE ARCREST 2.0 AND WILL NOT WORK PERFECTLY FOR ARCREST RELEASE 3.0.

When you upload a feature class to ArcGIS Online (AGOL), you have the option to publish those feature classes. Now let's assume you have enabled editing on that data set. The question is how do you pull down the data so you can have an updated version in office? Well luckily for us, you have ArcREST to automate this task! ArcREST can be found here (www.github.com/Esri/ArcREST) if you do not have it.

This example will show how to automate the download of a hosted feature class as a shapefile to a centralized storage location.

The workflow is as follows:

  1. Connect to the AGOL site
  2. Export the item to a shapefile
  3. Download the item to disk
  4. Erase the exported item

import arcrest
import uuid
import os
import time
#   Inputs
#
itemID = "THE ITEM ID TO EXPORT"
username = "SOME USERNAME"
pw = "MY PASSWORD"
url = "http://www.arcgis.com/sharing"
filePath = r"c:\temp"
#   Logic
#
#  Create security handler
shAGOL = arcrest.AGOLTokenSecurityHandler(username, pw)
#  Connect to AGOL
org = arcrest.manageagol.Administration(url=url, securityHandler=shAGOL)
# Grab the user's content (items)
content = org.content  
usercontent = content.usercontent(username=username)
#  Create a export item with a random name using UUID
#  and export it to a zipfile
#
result =  usercontent.exportItem(title="%s" % uuid.uuid4().get_hex(),
                                     itemId=itemID,
                                     exportFormat=exportDataAs,
                                     exportParameters=None)

exportedItemId = result['exportItemId']
jobId = result['jobId']
exportItem = content.item(itemId=exportedItemId)
#   Ensure the item is finished exporting before downloading
#
status =  usercontent.status(itemId=exportedItemId, jobId=jobId, jobType="export")
    
while status['status'].lower() == "processing":
    time.sleep(3)
    status =  usercontent.status(itemId=exportedItemId,
                                 jobId=jobId,
                                 jobType="export")
filePath = exportItem.itemData(f="json", savePath=filePath)
#   Erase the exported item to clean up
#   AGOL
#
usercontent.deleteItems(items=exportItem.id)
del exportItem
print 'finishe!'

So what we have is a quick little script that downloads a file to the temp folder.  The key thing to look at is the while part of the code.  When perform actions like publish, generateFeatures, export, and createService, they are performed asynchronously on the system.  This means that you need to check if they are finished before performing further actions on the items.  The status method returns a dictionary of values, and you need to ensure that the value is not equal to 'processing', or any further actions like itemData() will return invalid information or data.  In the case of the itemData() you will download an empty zip file.

Enjoy and happy coding.

Friday, July 26, 2013

Always Tip Well Because...

Always tip that pizza driver well, because you never know if they are a GIS major or not.  Besides being funny, this is a very interesting ArcGIS Online map.  It shows concentrations of pizza consumption in the college town of Shippenburg, PA.  It also allows people in that town to see how much your neighbors are tipping.  Better tip well, you do not want to be the cheap guy on the block.

http://www.arcgis.com/home/item.html?id=f86568bc807b49cd86e437194ccd2bca

I wonder if the map creator is the driver and if now he will get more tips!

Enjoy

Thursday, February 21, 2013

Query Feature Service By Object IDs (Python)

To query a feature service using object ids, you need to perform a POST.  For those who don't know the difference between a "GET" and "POST" here is a quick blurb based on the HTML specifications:
"POST" means that former means that form data is to be encoded (by a browser) into a URL while the "GET" means that the form data is to appear within a message body. 

Even simpler, "GET" is basically for just getting data where a "POST" may involve anything, like updating data, sending, or creating data.

Using python, you can perform both "GET" and "POST" methods, but since we need a "POST" to query by IDs, here is a simple example.  Please note that most feature services limit 1000 features being returned, so if you want to grab all of the features from a feature service, you'll have to perform multiple queries to get all the features back in JSON format.

Example:


import urllib2
import urllib
import urlparse
import httplib

def query_by_objectid(url, objectIDStart=0, objectIDEnd=1001):
    """ performs a POST operation where the query is called using
        the object id method.  If a feature service has more than
        1000 records, use this method to get a range of features
        from the feature service.

        Inputs:
           :url:  - string of feature service URL
           :objectIDStart: - integer of the start whole number
           :objectIDEnd: - end range whole number
        Returns:
           returns string JSON of query
    """
    url = url + '/query'
    start = int(objectIDStart)
    end = int(objectIDEnd)

    objectIDs = ",".join([str(x) for x in range(start, end)])
    headers = {"Content-type": "application/x-www-form-urlencoded",
               "Accept": "text/plain"}
    parameters = {'objectIds' : objectIDs,
                  'f' : 'json'}
    urlparams = urllib.urlencode(parameters)
    parts = urlparse.urlparse(url)
    h = httplib.HTTPConnection(parts.netloc)
    headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}
    h.request('POST', parts.path, urlparams, headers)
    r = h.getresponse()
    return r.read()

if __name__ == "__main__":
    url = 'http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/TaxParcel/AssessorsLiveLayers/MapServer/1'
    print query_by_objectid(url=url,
                            objectIDStart=0,
                            objectIDEnd=5)

So we have a simple "POST" example. This sample should work with both ArcGIS Server 10.1 and ArcGIS Online feature services.

Please support my idea of having a python GUI by voting it up here.

Enjoy

Friday, April 22, 2011

Fun With ArcGIS Online

I've been playing around with ArcGIS Online, it's very interesting way of sharing complex geo-spatial information. 

Here is a simple example:

View Larger Map