ArcREST - Bigger and Better

I've been working hard on ArcREST (https://github.com/esri/arcrest) these days, and the package is catching on I think.  Over the last couple of months, ArcREST has been shown at both Esri's Federal Developer Summit and at the Esri Developer Summit in California.  I truly think the effort is beginning to pay off, and users are reaping the hard work put into this package.

Simply put, this is a toolbox to manage, manipulate, and control your online GIS presence.  Portal, ArcGIS Server, and ArcGIS Online can now be controlled through Python!  The modules present in the Python package are like hammers, nails, and screw drivers.  They provide the framework to do greater things.  By themselves, they do not seem like much, but together you can build your GIS house through the common scripting language of the science community.

I am constantly looking for feedback on this package, so sign up on github, and test it out with a work flow you need in these environments.  If you find a bug, have a question, or need help, please post it in the issue area of github.  

Thank you everyone!

Better Workflows in ArcREST

ArcREST is a project I started, and continue to work on.  It is currently on version 2.x and is evolving daily.  The primary focus from the administration side of things, it to make life easier to manage object without knowing all the ins and outs  or URLs for every server that relates to ArcGIS Online Site or an internal portal.

Starting today, the package will include a property that will allow users to directly access the site's hosting servers in either the federated environment or on ArcGIS Online.  To access this, you must be an administrator user of your site.  You create the manageorg.Administration object, and then you can navigate around your whole system.  This object has a new function called hostingServer().  It has an optional parameter called portalId which represents the organizational identifier.  If the user does not know it, a None can be passed and the package will try and discover the id.

The results are as follows:

• If the site is ArcGIS Online, a list of hostedservice.service.Services objects is returned
• If the site is Portal, a list of AGSAdministration objects is returned

Lets look at some sample code:

 import arcrest
if __name__ == "__main__":
    username = "A USERNAME"
    pw = "PASSWORD"
    adminUrl = "https:///portal/sharing/rest
    tokenUrl = "https:///portal/sharing/rest/generateToken"
    sh = arcrest.PortalTokenSecurityHandler(username=username, password=pw, 
                                            org_url=adminUrl,
                                            token_url=tokenUrl)
    print sh.token
    admin = arcrest.manageorg.Administration(url=adminUrl,
                                             securityHandler=sh)
    servers = admin.hostingServers()
    for s in servers:
        if isinstance(s, arcrest.manageags.AGSAdministration): 
            print s.info.fullVersion
            print s.resources
        elif isinstance(s, arcrest.hostedservice.Services):
            print s.currentVersion

Depending on what information you provide for the manageorg.Adminstration function, you will get back different object.  If you pass in a AGOLTokenHandler, and connect to AGOL, you will get back one object, and if you pass a Portal reference, you will get back the federated and hosted server information.


This is an effort to make managing AGOL and Portal sites easier for administrators.  I hope this helps everyone out there!

Enjoy,

A

So What's Not in ArcGIS Pro 1.0 for ArcPy?

Not every tool is in ArcPy at Pro 1.0.  If you are porting tools, it is important to remember this.  A complete list of tools not in ArcGIS Pro 1.0 can be found here (http://pro.arcgis.com/en/pro-app/tool-reference/appendices/unavailable-tools.htm).

Lots of functions are missing, but that doesn't mean you shouldn't try out Pro.  It's a wonderfully fun product, and the 3D really makes your data pop out.  I also feel that the function layouts are more intuitive, so it's easy to get started.

Moving from the 2.x framework of Python can also be daunting as well.  There is a tool for analyzing your scripts, toolboxes and/or models in Pro found here. (http://pro.arcgis.com/en/pro-app/tool-reference/data-management/analyzetoolsforpro.htm)

You'll get back a nifty little report telling you what will not work.  It's very helpful in trying to find where the fails will occur.

Check them out and happy coding!

Check out ago-assistant

Ever want to get into the nuts and bolts of AGOL or Portal and see the JSON behind the scenes?  Well check out ago-assistant.

This application was developed by a follow co-worker, and it's awesome!  It's a nice GUI based application that gives you fine grained access to your site's information.

I find this tool coupled with fiddler can help you easily debug your applications, or better help you understand the ArcGIS REST API that performs the magic.

You can grab it from github here: https://github.com/Esri/ago-assistant

Enjoy and check it out.

A

Convert Runtime Database to File Geodatabase

New at 10.3, you can convert a runtime database to a file geodatabase.  The tool copies the contents of a runtime geodatabase into a new file geodatabase including the table names, etc...   It's very simple to use:

import arcpy
arcpy.CopyRuntimeGdbToFileGdb_conversion(r"D:\data\RuntimeGDBs\delta.geodatabase", 
                                         'D:\data\copiedGDBs\deltaFGDB.gdb')

Here we copied a .geodatabase file (runtime database) to a file geodatabase.  Now the data captured in the runtime db can be used in desktop or anything else that can't directly access runtime libraries.

Enjoy

Converting PDF to TIFF (10.3) using ArcPy

New at 10.3, is a handy tool many of us in the GIS world have wanted for a long time.  That is converting geo-referenced PDFs to Tiff files.
The help describes this tool as:

Exports an existing PDF file to a Tagged Image File Format (TIFF). If the PDF has georeference information, the TIFF can be a GeoTIFF. These TIFFs can be used as a source for heads-up digitizing and viewing in ArcMap. Both GeoPDF and ISO standards of georeferenced PDFs are supported. 

It is very straight forward to use:

import arcpy
arcpy.PDFToTIFF_conversion(in_pdf_file="C:/temp/sample.pdf", 
                           out_tiff_file="C:/temp/sample.tif", 
                           pdf_password="", 
                           pdf_page_number="1", 
                           pdf_map="Layers", 
                           clip_option="NO_CLIP", 
                           resolution="250", 
                           color_mode="RGB_TRUE_COLOR", 
                           tiff_compression="LZW", 
                           geotiff_tags="GEOTIFF_TAGS")

For this sample, I just created a map with one of the AGOL imagery layers in a blank ArcMap layout and exported it to PDF with the GeoReference information embedded.  This creates a TIFF that can be used for additional spatial data in the ArcMap session.

Enjoy,

A

Workflow for Adding Files in Parts for Portal or ArcGIS Online

Python 2.7.5 is not very good at handling big files.  Actually it stinks especially if you need to upload them via a multi-part post.  There are many 3rd party solutions, but I found issues with both poster and requests with the tool belt add-in.  They simply didn't work.  Luckily for us, the folks who designed the AGOL/Portal REST API provided an alternative way of uploading large files.  The method is called Add Item Part.  The documentation can be found here: http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Add_Item_Part/02r300000094000000/.

When trying to figure out how to use this function, you also need to read the Add Item method as well to get how Add Item Part should be used.  The workflow is as follows:

1. Call Add Item with no data (file data) and pass in multipart=True along with the filename=.extension.
2. Break the file into parts, but they must be over 5 MBs except for the last chunk which signals an end of file. (I break everything into 50 MB chunks because I don't want to perform tons of POST calls)
3. Call Add Item Part by using a POST.  Pass in the parameters partNum, which is a unique integer from 1 to 10,000.  You must pass that in, or each part will be overwritten.  Also, pass in the file via multi-part POST.
4. Now call Commit function to tell the server, hey I'm done.  This is an asynchronous call, so don't forget to check the item's status before moving on, or you will not be able to update the item's properties.

• If you look at your 'My Content' page on AGOL, you'll notice and item with everything stated as 'null' for the title, type, etc.. All you can do is delete the item right now.  This makes step 5 very important

5. Update the item using the updateItem REST call.  Here we state the type, title, tags, etc.. all the good stuff you need to know so users can access the data.  

Great now we have the item in AGOL or Portal.  Simple right?  Not really, but it's a great way to get around Python's 2.7.5 annoying memory issue.  I haven't tested this on x64 python or python 3.4.  Hopefully some of my readers will post a comment letting me know if the memory issues with StringIO/cStringIO still exist when performing large file POSTs.

The add item part is in ArcREST today!

Get ArcREST: http://www.github.com/Esri/ArcREST

Also vote for my idea for a GUI builder for Python Add-Ins.