Better Workflows in ArcREST

ArcREST is a project I started, and continue to work on.  It is currently on version 2.x and is evolving daily.  The primary focus from the administration side of things, it to make life easier to manage object without knowing all the ins and outs  or URLs for every server that relates to ArcGIS Online Site or an internal portal.

Starting today, the package will include a property that will allow users to directly access the site's hosting servers in either the federated environment or on ArcGIS Online.  To access this, you must be an administrator user of your site.  You create the manageorg.Administration object, and then you can navigate around your whole system.  This object has a new function called hostingServer().  It has an optional parameter called portalId which represents the organizational identifier.  If the user does not know it, a None can be passed and the package will try and discover the id.

The results are as follows:

• If the site is ArcGIS Online, a list of hostedservice.service.Services objects is returned
• If the site is Portal, a list of AGSAdministration objects is returned

Lets look at some sample code:

 import arcrest
if __name__ == "__main__":
    username = "A USERNAME"
    pw = "PASSWORD"
    adminUrl = "https:///portal/sharing/rest
    tokenUrl = "https:///portal/sharing/rest/generateToken"
    sh = arcrest.PortalTokenSecurityHandler(username=username, password=pw, 
                                            org_url=adminUrl,
                                            token_url=tokenUrl)
    print sh.token
    admin = arcrest.manageorg.Administration(url=adminUrl,
                                             securityHandler=sh)
    servers = admin.hostingServers()
    for s in servers:
        if isinstance(s, arcrest.manageags.AGSAdministration): 
            print s.info.fullVersion
            print s.resources
        elif isinstance(s, arcrest.hostedservice.Services):
            print s.currentVersion

Depending on what information you provide for the manageorg.Adminstration function, you will get back different object.  If you pass in a AGOLTokenHandler, and connect to AGOL, you will get back one object, and if you pass a Portal reference, you will get back the federated and hosted server information.


This is an effort to make managing AGOL and Portal sites easier for administrators.  I hope this helps everyone out there!

Enjoy,

A

So What's Not in ArcGIS Pro 1.0 for ArcPy?

Not every tool is in ArcPy at Pro 1.0.  If you are porting tools, it is important to remember this.  A complete list of tools not in ArcGIS Pro 1.0 can be found here (http://pro.arcgis.com/en/pro-app/tool-reference/appendices/unavailable-tools.htm).

Lots of functions are missing, but that doesn't mean you shouldn't try out Pro.  It's a wonderfully fun product, and the 3D really makes your data pop out.  I also feel that the function layouts are more intuitive, so it's easy to get started.

Moving from the 2.x framework of Python can also be daunting as well.  There is a tool for analyzing your scripts, toolboxes and/or models in Pro found here. (http://pro.arcgis.com/en/pro-app/tool-reference/data-management/analyzetoolsforpro.htm)

You'll get back a nifty little report telling you what will not work.  It's very helpful in trying to find where the fails will occur.

Check them out and happy coding!

Check out ago-assistant

Ever want to get into the nuts and bolts of AGOL or Portal and see the JSON behind the scenes?  Well check out ago-assistant.

This application was developed by a follow co-worker, and it's awesome!  It's a nice GUI based application that gives you fine grained access to your site's information.

I find this tool coupled with fiddler can help you easily debug your applications, or better help you understand the ArcGIS REST API that performs the magic.

You can grab it from github here: https://github.com/Esri/ago-assistant

Enjoy and check it out.

A

Convert Runtime Database to File Geodatabase

New at 10.3, you can convert a runtime database to a file geodatabase.  The tool copies the contents of a runtime geodatabase into a new file geodatabase including the table names, etc...   It's very simple to use:

import arcpy
arcpy.CopyRuntimeGdbToFileGdb_conversion(r"D:\data\RuntimeGDBs\delta.geodatabase", 
                                         'D:\data\copiedGDBs\deltaFGDB.gdb')

Here we copied a .geodatabase file (runtime database) to a file geodatabase.  Now the data captured in the runtime db can be used in desktop or anything else that can't directly access runtime libraries.

Enjoy

Converting PDF to TIFF (10.3) using ArcPy

New at 10.3, is a handy tool many of us in the GIS world have wanted for a long time.  That is converting geo-referenced PDFs to Tiff files.
The help describes this tool as:

Exports an existing PDF file to a Tagged Image File Format (TIFF). If the PDF has georeference information, the TIFF can be a GeoTIFF. These TIFFs can be used as a source for heads-up digitizing and viewing in ArcMap. Both GeoPDF and ISO standards of georeferenced PDFs are supported. 

It is very straight forward to use:

import arcpy
arcpy.PDFToTIFF_conversion(in_pdf_file="C:/temp/sample.pdf", 
                           out_tiff_file="C:/temp/sample.tif", 
                           pdf_password="", 
                           pdf_page_number="1", 
                           pdf_map="Layers", 
                           clip_option="NO_CLIP", 
                           resolution="250", 
                           color_mode="RGB_TRUE_COLOR", 
                           tiff_compression="LZW", 
                           geotiff_tags="GEOTIFF_TAGS")

For this sample, I just created a map with one of the AGOL imagery layers in a blank ArcMap layout and exported it to PDF with the GeoReference information embedded.  This creates a TIFF that can be used for additional spatial data in the ArcMap session.

Enjoy,

A

Workflow for Adding Files in Parts for Portal or ArcGIS Online

Python 2.7.5 is not very good at handling big files.  Actually it stinks especially if you need to upload them via a multi-part post.  There are many 3rd party solutions, but I found issues with both poster and requests with the tool belt add-in.  They simply didn't work.  Luckily for us, the folks who designed the AGOL/Portal REST API provided an alternative way of uploading large files.  The method is called Add Item Part.  The documentation can be found here: http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Add_Item_Part/02r300000094000000/.

When trying to figure out how to use this function, you also need to read the Add Item method as well to get how Add Item Part should be used.  The workflow is as follows:

1. Call Add Item with no data (file data) and pass in multipart=True along with the filename=.extension.
2. Break the file into parts, but they must be over 5 MBs except for the last chunk which signals an end of file. (I break everything into 50 MB chunks because I don't want to perform tons of POST calls)
3. Call Add Item Part by using a POST.  Pass in the parameters partNum, which is a unique integer from 1 to 10,000.  You must pass that in, or each part will be overwritten.  Also, pass in the file via multi-part POST.
4. Now call Commit function to tell the server, hey I'm done.  This is an asynchronous call, so don't forget to check the item's status before moving on, or you will not be able to update the item's properties.

• If you look at your 'My Content' page on AGOL, you'll notice and item with everything stated as 'null' for the title, type, etc.. All you can do is delete the item right now.  This makes step 5 very important

5. Update the item using the updateItem REST call.  Here we state the type, title, tags, etc.. all the good stuff you need to know so users can access the data.  

Great now we have the item in AGOL or Portal.  Simple right?  Not really, but it's a great way to get around Python's 2.7.5 annoying memory issue.  I haven't tested this on x64 python or python 3.4.  Hopefully some of my readers will post a comment letting me know if the memory issues with StringIO/cStringIO still exist when performing large file POSTs.

The add item part is in ArcREST today!

Get ArcREST: http://www.github.com/Esri/ArcREST

Also vote for my idea for a GUI builder for Python Add-Ins.

Searching for User Content on AGOL Using ArcREST

ArcGIS Online (AGOL) and Portal organizations are great, but managing the whole organization's content can be tough.  Here is a brief example on how you could search an site (in our case AGOL) and look at what a given user has shared publicly.

Searching is fairly straight forward, and do not require any security logins to perform.  You can see this by just going to www.arcgis.com and typing something into the search box.  For this example though, I am going to use login credentials just in case I needed to perform an administrative task down the line.

if __name__ == "__main__":
    username = "MY USERNAME"
    password = "MY PASSWORD"
    url = "http://www.arcgis.com/sharing"
    proxy_url = None#"127.0.0.1" # for fiddler
    proxy_port = None#"8888" # for fiddler
    q2 = "owner:esri"
    securityHandler = arcrest.AGOLTokenSecurityHandler(username,
                                                       password,
                                                       proxy_port=proxy_port,
                                                       proxy_url=proxy_url)    
    admin = arcrest.manageagol.Administration(url=url, 
                                              securityHandler=securityHandler,
                                              proxy_url=proxy_url,
                                              proxy_port=proxy_port)    
    res = admin.query_without_creds( q=q2, start=1, num=1) # Want to find only public item on owner:esri
    items = []
    total = int(res['total'])
    steps = int(total / 100)
    if (total % 100) > 0:
        steps += 1
    step = 1
    while step <= steps:
        res = admin.query_without_creds( q=q2, start= 1 + ((step-1)*100), num=100)
        for r in res['results']:
            print r # do something with item!
            del r
        del res
        step += 1    

The item is returned from the query_without_creds() as a dictionary.  It contains information about each item that is unique to that item.

The script isn't doing much with the items that it finds, but it does show how you can loop through the owner's items (esri).  If you could administer the owner's item, you could change those public items to private.

To get a better sense of what you cannot and can query, and how the syntax works, I would check out this (http://doc.arcgis.com/en/arcgis-online/reference/search.htm).  It shows the advanced search syntax in great detail.

You can get ArcREST here (www.github.com/esri/ArcREST) if you do not have it.

Happy searching!