The Road Ahead for ArcREST

If you are going to San Diego this year, you can see me in action presenting in the demo theater about ArcREST.  I'm going to do some demos, talk about ArcREST basics, and discuss the road ahead.  In turn you can ask questions and learn something new.

Here are the details:
12:30 PM - 01:15 PM : Using the ArcREST Python Package on Tuesday 7/21/2015.

As we approach Esri's UC 2015, I wanted to layout what is coming for the next version of ArcREST for those who cannot attend my presentation.

Here is a brief list of what is coming very soon:

• Ntlm, PKI and LDAP security handlers
• improved backend handling of GET and POST functions
• cleaner code
• additional geometry service functions
• additional query options

and much more...

The biggest focus is supporting ArcGIS server, portal or online sites that do not use the standard username/password token based security.  

The ArcREST team is also taking requests for additional functions that we may have missed or didn't include.  The project is built around you the user, so help out and log an enhancement request or an issue here. (https://github.com/esri/arcrest/issues).  

ArcREST Publishing Part 2 - SD Files

The easiest and quickest way to get data into AGOL/Portal is to publish from a service definition file, or SD file.  SD files are compressed files using 7-zip.  They contain all sorts of information about how your service behaves, what it's name is, the service's data, etc...  It's a complete package or all in one file for publishing services.

You can automate the creation of service using Python or AO, but I am not going to dive into that topic.  Instead I am going to show how you can easily publish the SD file using ArcREST to your site (AGS, AGOL or Portal).

This example is going to focus on solely the publishing of SD files to AGOL.

import arcrest
import json
import time
#--------------------------------------------------------------------------
if __name__ == "__main__":
    username = "someadminuser"
    password = "reallysecurepassword"
    sd_file = r"C:\temp\1\DemoPublishing.sd"
    proxy_port = None
    proxy_url = None

    #   Connect to the site
    #
    sh = arcrest.AGOLTokenSecurityHandler(username, password)
    admin = arcrest.manageorg.Administration(securityHandler=sh,
                                             initialize=True)
    content = admin.content

    uc = content.usercontent(username)
    ip = arcrest.manageorg.ItemParameter()
    ip.title = "DemoPublishing"
    ip.type = "Service Definition"
    ip.tags = "SD, publishing, example"
    ip.typeKeywords = "Data, Service, Service Definition"

    res = uc.addItem(itemParameters=ip, filePath=sd_file)
    itemId = res['id']
    # Publish the SD File
    #
    resPublish = uc.publishItem(fileType="serviceDefinition",
                                publishParameters=None, itemId=itemId)
    jobId = resPublish['services'][0]['jobId']
    serviceItemId = resPublish['services'][0]['serviceItemId']
    flUrl = resPublish['services'][0]['serviceurl']
    status = uc.status(itemId=serviceItemId, jobId=jobId)
    while status['status'].lower() == "processing":
        time.sleep(2)
        status = uc.status(itemId=serviceItemId, jobId=jobId)
    print 'service published.'
    print 'Bonus Demo - update some additional properties on a feature service'
    flAdmin = arcrest.agol.FeatureService(url=flUrl, securityHandler=sh).administration
    # Bonus 1- Enable edits Only
    updateDefinition = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete","allowGeometryUpdates":true,"editorTrackingInfo":{"enableEditorTracking":false,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
    print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition))
    # Bonus 2- Enable edits with export data - adds 'Extract' to capabilities
    updateDefinition2 = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete,Extract","editorTrackingInfo":{"enableEditorTracking":false,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
    print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition2))
    # Bonus 3- Enable Sync on services - Adds 'Sync' to capabilities
    updateDefinition3 = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete,Sync,Extract","editorTrackingInfo":{"enableEditorTracking":false,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
    print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition3))
    # Bonus 4- Enable editor tracking on service - set enableEditorTracking = True
    updateDefinition4 = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete,Sync,Extract","editorTrackingInfo":{"enableEditorTracking":true,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
    print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition4))
    print 'finished!'

I added some additional information to the example here.  The basic example ends at the print 'service published' part.  The additional code below shows how to enable the editing, export data, sync or track edits.  You can do this all in call, but I wanted to show how you enable each one at a time because you may only want to enable one function over another.  Notice how I get the feature service URL from the publish() then I take that URL and create a Feature Service object from the agol.FeatureService Class.  Since my user is an administrator and owner of the service, I can access the back end function to enable the additional properties.  On the Feature Service class, there is a property called administration which takes you to the hostingservice.AdminFeatureService class which allows users to update the service definitions on a hosted feature service.

I threw a lot out there, and I hope it helps!

Previous posts on publishing:
Part 1 - Publish CSV files can be found here.

Listing Installed Python Packages

Here is a quick two liner bit of code to see what Python packages you have installed.  I am running this from the command line. Here is the example:

>>> import pip
>>> print sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])
['arcrest==2.0.100', 'astroid==1.3.2', 'boto==2.38.0', 'cffi==1.1.0', 'colorama=
=0.3.3', 'cryptography==0.9', 'enum34==1.0.4', 'idna==2.0', 'ipaddress==1.0.7',
'libarchive==0.4.3', 'logilab-common==0.63.2', 'matplotlib==1.3.0', 'nose==1.3.6
', 'numpy==1.7.1', 'pip==7.0.0', 'pyasn1==0.1.7', 'pycparser==2.13', 'pylint==1.
4.0', 'pylzma==0.4.6', 'pyopenssl==0.15.1', 'pyparsing==1.5.7', 'pyzillow==0.4.0
', 'setuptools==17.0', 'six==1.9.0', 'wheel==0.24.0']

Enjoy!

See me at Esri's User Conference 2015

I'm going to be at UC 2015 presenting about ArcREST in the demo theater.

12:30 PM - 01:15 PM : Using the ArcREST Python Package on Tuesday 7/21/2015.

Hope to see you all there!

ArcREST - New Geoprocessing Model

Today ArcREST has a geoprocessing (GP) update.  The changes are minor, but the results are big.  The GP objects has been streamline so all the properties are the same.

Example of GPObject in Action:

import arcrest
# convert a feature class to a GPFeatureRecordSet
gpfrs = arcrest.ags.GPFeatureRecordSetLayer.fromFeatureClass(r"c:\temp\grid.gdb\sample", paramName="parameterName on GP tool")

The above code shows converting a feature class into a GPFeatureRecordSet object that can be given to a task that asks for that type of parameter.

Working with GP services is easy as well.

Example:

import arcrest
if __name__ == "__main__":
    sh = arcrest.AGSTokenSecurityHandler(username="user",
                                         password="password",
                                         token_url="http://site:6080/arcgis/admin/generateToken"
                                         )
    url = "http://site:6080/arcgis/rest/services/gp/scriptmv/GPServer"
    gp = arcrest.ags.GPService(url=url,
                               securityHandler=sh)
    for task in gp.tasks:
        if task.name.lower() == "":
            # submit the job
            job = task.submitJob(inputs=None)
            # wait till the job finishes
            while job.jobStatus != "esriJobSucceeded": pass
            # get the job results
            results = job.results # do something here.

Here we accessed a GP service and found a tool we wanted to run.  Since we are running the process async, submitJob() was used.  The function has additional options, like submitting the result via POST verse GET, etc...

Enjoy

ArcREST Publishing Part 1 - CSV Files

One of the main purposes of ArcGIS.com is that is allows you to share information.  Sometime you want to quickly stage content, or you want to mirror your site in another location.

So in this post, I will show you how to do some publishing.  Publishers can create feature services as well as tiled map services.

Feature services can be created using input files of type csv, shapefile, serviceDefinition, featureCollection, and fileGeodatabase.

The AGOL/Portal REST has some great bullet points to remember when publishing items:

• CSV files that contain location fields, (ie.address fields or X, Y fields) are spatially enabled during the process of publishing.
• Shapefiles and file geodatabases should be packaged as *.zip files.
• Tiled map services can be created from service definition (*.sd) files, tile packages, and existing feature services.
• Service definitions are authored in ArcGIS for Desktop and contain both the cartographic definition for a map as well as its packaged data together with the definition of the geo-service to be created.
• Use the Analyze operation to generate the default publishing parameters for CSVs.

Now that we have what we can publish, it is time to start coding!

This post will cover publishing CSV files.

CSV files require a bit more steps then the other publishing work flows because not only do you have to upload your data, you should analyze it in order to get a correct set of default publishing properties.  Luckily, ArcREST can ease the pain.

import arcrest

if __name__ == "__main__":
    username = "some account"
    password = "some password"
    csv_file = r"C:\Users\andr5624\Desktop\book1.csv"

    sh = arcrest.AGOLTokenSecurityHandler(username, password)
    admin = arcrest.manageorg.Administration(securityHandler=sh)
    content = admin.content
    usercontent = content.usercontent(username=username)
    ip = arcrest.manageorg.ItemParameter()
    ip.title = "Sample CSV"
    ip.type = "CSV"
    ip.tags = "tag1,tag3"
    res = usercontent.addItem(itemParameters=ip,
                              filePath=csv_file)
    itemId = res['id']

    #  Now you need to analyze the item to publish it
    #
    featureContent = content.featureContent
    analyzeParams = arcrest.manageorg.AnalyzeParameters()
    analyzeResult = featureContent.analyze(itemId=itemId, analyzeParameters=analyzeParams)
    #  Published based off the analyze function
    #
    if 'publishParameters' in analyzeResult:
        pp = analyzeResult['publishParameters']
        publishParams = arcrest.manageorg.PublishCSVParameters(name=pp['name'],
                                               locationType=pp['locationType'],
                                               layerInfo=pp['layerInfo'],
                                               latitudeFieldName=pp["latitudeFieldName"],
                                               longitudeFieldName=pp['longitudeFieldName'])
        print usercontent.publishItem(fileType="csv",
                                      publishParameters=publishParams,
                                      itemId=itemId)

Let's discuss the above code. Here we access AGOL and add an item for our user.  Once uploaded, the code, using the item id, analyzes the CSV to get a set of default publishing parameters.  These parameters are used to populate the PublishCSVParameters class in order to publish the CSV file correctly.

Pretty cool.  ArcREST masks a complex work flow into a couple of short lines of Python code.

ArcREST Basics - Creating a Group

Managing your portal or AGOL site can be a tough task, and ArcREST is here to help.  Often within your organization, individuals need new groups or maybe you need to create a set of default groups.  ArcREST allows individuals to create a list of group then put users into that group easily.  

The Pointy Hair Boss (http://www.andrewlipson.com/)

So here is the situation.  You boss wants you to create a group called 'Publishing Review'.  This group will contain all your users must be in this group.  The point of this group is that non-administrator accounts cannot publish to the public, and any data within this group will be reviewed by administrators on a given cycle and published if the information meets the organizations standards.

Let's begin!

import arcrest
#--------------------------------------------------------------------------
def getAllUsers(portalId, admin):
    """
       returns all the users for a given AGOL

       Inputs:
          portalId - unique id of the portal
          admin - manageorg.Administration object
       Output:
          returns a list of the users
    """
    start = 1
    num = 100
    portals = admin.portals(portalId=portalId)
    count = 0
    nextStart = 0
    results = []
    while nextStart > -1:
        users = portals.users(start=start + (num * count),
                              num=num)
        results = results + users['users']
        count += 1
        nextStart = users['nextStart']
        del users
    return results
#--------------------------------------------------------------------------
if __name__ == "__main__":
    #   Connect to the site
    #
    sh = arcrest.AGOLTokenSecurityHandler(username, password)
    admin = arcrest.manageorg.Administration(securityHandler=sh,
                                             initialize=True)
    community = admin.community
    portals = admin.portals()
    groups = community.groups
    portalId = portals.portalId
    #   Create Group
    #   Basic group inputs
    access = "org"
    groupTitle = "Pre Publishing Group"
    groupTags = "Publishing;Service Management"
    description = "This group allows users to request publishing services to public."
    # search to see if the group exists. If not, create the group, else use the ID 
    searchResult = community.getGroupIDs(groupNames=groupTitle)
    if len(searchResult) == 0:
        groupId = community.createGroup(title=groupTitle,
                                tags=groupTags,
                                description=description,
                                snippet="",
                                phone="",
                                access=access,
                                sortField="title",
                                sortOrder="asc",
                                isViewOnly=False,
                                isInvitationOnly=False,
                                thumbnail=None)['group']['id']
    else:
        groupId = searchResult[0]
    # Get all the site's users and add them to the group
    users = [user['username'] for user in getAllUsers(portalId=portalId, admin=admin)]
    if len(users) > 0:
        groups.addUsersToGroups(users=",".join(users), groupID=groupId)

So let's review the code.  First we connect to our site.  Next we grab all the objects we will need to work with in order to make the magic happen.  Since the topic is groups, there is a need to work with the community functions as well as some portal functions.  This will give us all the access we need.  After we do all of that, we need to know the group ID, this is found by either creating the group or if it exists, getting the site to return the information to us.  Using the community object, call the getGroupIDs() returns the IDs for a group or groups as a list.  If the list if < 1 then create the group using the createGroup() else use the search result Id to add the users.

To get all the users, I created a function that will go and grab all the users from the portal object.  All you need to pass is the portalId and the administration objects and I detailed list of users will be returned.  Since in this sample we only need username, we can just parse out what we need and move on as shown above.

Hope this have been a helpful post, keep on rocking, and post your questions/comments/enhancements on the ArcREST page: http://www.github.com/Esri/ArcREST