tag:blogger.com,1999:blog-33091463413577830352024-03-12T20:15:00.507-07:00Another GIS BlogPython, .NET, C++, GIS, and Computer MysticismAndrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comBlogger312125tag:blogger.com,1999:blog-3309146341357783035.post-2527455657599465022018-02-08T04:00:00.000-08:002018-02-08T04:00:20.973-08:00Esri Developer Summit 2018See me live and in-person at the Esri DevSummit 2018!<br />
<br />
Tomato's optional.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2V00Phq7AmI0PxdvNlXwEwWYQaTEuRIDD91jAfLngvF8D1VJQzljLbCCStFkRixYNsrzCaZLVAPMhEMOp3iqsjcjhwIHhtmXi4erms_Yf2fnOtFcjXPM7Oee3jQb0zwdLvH-DpGz9VDxV/s1600/march_dev_summit_pres.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="765" data-original-width="920" height="532" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2V00Phq7AmI0PxdvNlXwEwWYQaTEuRIDD91jAfLngvF8D1VJQzljLbCCStFkRixYNsrzCaZLVAPMhEMOp3iqsjcjhwIHhtmXi4erms_Yf2fnOtFcjXPM7Oee3jQb0zwdLvH-DpGz9VDxV/s640/march_dev_summit_pres.JPG" width="640" /></a></div>
<br />
Like Pokemon, you gotta catch them all.<br />
<br />
<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-37433446161763206402017-07-10T02:00:00.000-07:002017-07-10T02:00:30.804-07:00ArcGIS API for PythonSorry for not posting folks, but I've been busy working on the ArcGIS API for Python. <br />
<div>
<br /></div>
<div>
For those who don't know I wrote ArcREST with another colleague of mine, Mike. Since the announcement of the ArcGIS API for Python, we have decided to retire ArcREST, though if pull requests are submitted to fix critical issues, we will do our best to merge them. </div>
<div>
<br /></div>
<div>
So what does that leave the people with to work with the REST API? There is a great package from Esri, the ArcGIS API for Python. Get it <a href="https://developers.arcgis.com/python/guide/install-and-set-up/" target="_blank">here</a> (https://developers.arcgis.com/python/guide/install-and-set-up/). </div>
<div>
<br /></div>
<div>
It has tons of great features for everyday users, content producers and administrators. Another benefits is that you get the new Spatial DataFrame! </div>
<div>
<br /></div>
<div>
The Spatial DataFrame (SDF) is built on top of the Panda's DataFrame to display 2-D data with labelled columns. The SDF can pull data from services and local feature classes. It is a quick way to merge your online content to local content, and republish it out in a new service or save as a feature class.</div>
<div>
<br /></div>
<div>
At v1.2 of the API, we added support for ArcGIS Server, and enhance the Enterprise ArcGIS and ArcGIS Online support. </div>
<div>
<br /></div>
<div>
I highly recommend you start moving the ArcGIS API for Python. </div>
<div>
I look forward to seeing and hearing what you do with the Python API. </div>
<div>
<br /></div>
<div>
If you want to see me at Esri User Conference 2017, I will be presenting the following sessions:</div>
<div>
<br /></div>
<div>
<table>
<tbody>
<tr style="height: 23px;">
<td style="height: 23px;">Session </td>
<td style="height: 23px;">Location </td>
<td style="height: 23px;">Date/Time </td>
</tr>
<tr style="height: 23px;">
<td style="height: 23px;"> Administering ArcGIS Enterprise and ArcGIS Online with Python</td>
<td style="height: 23px;">SDCC - Ballroom 06 D </td>
<td style="height: 23px;"> 07/11/2017 - 8:30 am - 9:45 am</td>
</tr>
<tr style="height: 23px;">
<td style="height: 23px;"> Automating Enterprise GIS Administration using Python</td>
<td style="height: 23px;">SDCC - Demo Theater 11 </td>
<td style="height: 23px;"> 07/11/2017 - 4:30 pm - 5:15 pm</td>
</tr>
<tr style="height: 23px;">
<td style="height: 23px;"> Administering ArcGIS Enterprise and ArcGIS Online with Python </td>
<td style="height: 23px;">SDCC - Ballroom 06 C </td>
<td style="height: 23px;"> 07/13/2017 - 8:30 am - 9:45 am</td>
</tr>
</tbody>
</table>
<br />
<br />
<b>Tomato throwing is optional.</b><br />
<b><br /></b>
<b><br /></b>
See ya there</div>
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-46928703999186859962016-12-14T07:48:00.003-08:002016-12-14T07:48:45.814-08:00Microsoft Compiler for Python 2.7Doesn't everyone hate this message:<br />
<br />
<blockquote class="tr_bq">
Microsoft Visual C++ 9.0 is required (Unable to find vcvarsall.bat).</blockquote>
<br />
I sure do, and I solved it by downloading a helpful program from Microsoft! Don't believe me, just google it! Install Microsoft's compiler for Python 2.7 from here: https://www.microsoft.com/en-us/download/details.aspx?id=44266 and most of the pip installs should work!<br />
<br />
Enjoy<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-48391398605296683162016-12-01T04:09:00.002-08:002016-12-01T04:09:25.637-08:00So long VBA and Thanks for all the MemoriesMicrosoft has stopped providing fixes or troubleshooting for VB and VBA. Esri just announced the same in the 10.5 release. It's time to update that old code. <br />
<br />
Some options moving forwards for your old desktop applications are:<br />
<br />
<ul>
<li>.NET</li>
<li>Python</li>
</ul>
<br />
It <b>IS</b> time to re-evaluate the application and see how it fits into the other development frameworks. <br />
<br />
<br />
VB/VBA is officially dead.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://backofpack.files.wordpress.com/2016/04/solong.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://backofpack.files.wordpress.com/2016/04/solong.jpg" width="320" /></a></div>
<br />
<br />
Check out the article <a href="https://blogs.esri.com/esri/arcgis/2016/11/14/arcgis-desktop-and-vba-moving-forward/" target="_blank">here</a>: (https://blogs.esri.com/esri/arcgis/2016/11/14/arcgis-desktop-and-vba-moving-forward/)<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-46717295001558747062016-09-19T03:50:00.002-07:002016-09-19T03:50:50.838-07:00Configuring Juypter Notebook Startup FolderBy default when you install jupyter notebook (formally iPython), the product will point to Window's My Document folder. I find this to be less than optimal because My Documents can contain a mishmash of various documents. To change the start up directory, there is a run time option where you can specify a folder, but that is not a permanent solution. A better solution is to create a configuration file.<br />
<br />
After Jupyter is installed (I used anacoda's distribution of Python 3.5), navigate to the folder containing the jupyter.exe<br />
<br />
<ol>
<li>Type the following: <code> jupyter notebook --generate-config </code></li>
<li>This will generate an entry in your user profile: <code>~/.jupyter</code></li>
<li>Edit the jupyter_notebook._config.py file and find <code>c.NotebookApp.notebook_dir</code></li>
<li>Uncomment the entry and enter in your path</li>
<li>Save any file changes and start jupyter</li>
</ol>
<br />
The ipython notebooks should now be saved in your new directory.
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-35732704269278581632016-08-19T05:00:00.000-07:002016-08-19T05:00:07.412-07:00Panda Dataframe as a Process Tracker (postgres example)Sometimes you need to keep track of the number of rows processed for a given table.<br />
<br />
Let's assume you are working in postgres and you want want to do row by row operations to do some sort of data manipulation. Your user requires you to keep track of each row's changes and wants to know the number of failures with the updates and the number of successful updates. The output must be in a text file with pretty formatting.<br />
<br />
There are many ways to accomplish this task, but let's use Pandas, arcpy.da Update Cursor, and some sql.<br />
<br />
<br />
<code>
<span style="font-family: monospace;">#--------------------------------------------------------------------------</span><br />
<span style="font-family: monospace;">def create_tracking_table(sde, tables):</span><br />
<span style="font-family: monospace;"> """</span><br />
<span style="font-family: monospace;"> creates a panadas dataframe from a sql statement</span><span style="font-family: monospace;"><br /></span>
<span style="font-family: monospace;"> Input:</span><br />
<span style="font-family: monospace;"> sde - sde connection file</span><br />
<span style="font-family: monospace;"> tables - name of the table to get the counts for</span><br />
<span style="font-family: monospace;"> Ouput:</span><br />
<span style="font-family: monospace;"> Panda Dataframe with column names: Table_Name, Total_Rows and</span><br />
<span style="font-family: monospace;"> Processed</span><br />
<span style="font-family: monospace;"> """</span><br />
<span style="font-family: monospace;"> desc = arcpy.Describe(sde)</span><br />
<span style="font-family: monospace;"> connectionProperties = desc.connectionProperties</span><br />
<span style="font-family: monospace;"> username = connectionProperties.user</span><br />
<span style="font-family: monospace;"> sql = """SELECT</span><br />
<span style="font-family: monospace;"> nspname AS schemaname,relname,reltuples</span><br />
<span style="font-family: monospace;"> FROM pg_class C</span><br />
<span style="font-family: monospace;"> LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)</span><br />
<span style="font-family: monospace;"> WHERE</span><br />
<span style="font-family: monospace;"> nspname NOT IN ('pg_catalog', 'information_schema') AND</span><br />
<span style="font-family: monospace;"> relkind='r' AND</span><br />
<span style="font-family: monospace;"> nspname='{schema}' AND</span><br />
<span style="font-family: monospace;"> relname in ({tables})</span><br />
<span style="font-family: monospace;"> ORDER BY reltuples DESC;""".format(</span><br />
<span style="font-family: monospace;"> schema=username,</span><br />
<span style="font-family: monospace;"> tables=",".join(["'%s'" % t for t in tables])</span><br />
<span style="font-family: monospace;"> )</span><br />
<span style="font-family: monospace;"> columns = ['schemaname','Table_Name','Total_Rows']</span><br />
<span style="font-family: monospace;"><br /></span>
<span style="font-family: monospace;"> con = arcpy.ArcSDESQLExecute(sde)</span><br />
<span style="font-family: monospace;"> rows = con.execute(sql)</span><br />
<span style="font-family: monospace;"> count_df = pd.DataFrame.from_records(rows, columns=columns)</span><br />
<span style="font-family: monospace;"> del count_df['schemaname']</span><br />
<span style="font-family: monospace;"> count_df['Processed'] = 0</span></code><br />
<code> count_df['Errors'] = 0<br />
<span style="font-family: monospace;"> return count_df</span></code><br />
<br />
<br />
Now we have a function that will return a dataframe object from a SQL statement. It contains 3 fields; Table_Name, Total_Rows, and Processed. Table_name is the name of the table in the database. Total_Rows is the length of the table. Processed is where you are going to modify every a row gets updated successfully. Errors is the numeric column where if an update fails, the value will be added to.<br />
<br />
So let's use what we just made:<br />
<br />
<span style="font-family: monospace;">count_df = </span><span style="font-family: monospace;">create_tracking_table(sde, tables)</span><br />
<span style="font-family: monospace;">for table in tables:</span><br />
<span style="font-family: monospace;"> with arcpy.da.UpdateCursor(table, "*") as urows:</span><br />
<span style="font-family: monospace;"> for urow in urows:</span><br />
<span style="font-family: monospace;"> try:</span><br />
<span style="font-family: monospace;"> urow[3] += 1</span><br />
<span style="font-family: monospace;"> urows.updateRow(urow)</span><br />
<span style="font-family: monospace;"> df.loc[df['Table_Name'] == '%s' % table, 'Processed'] += 1</span><br />
<span style="font-family: monospace;"> except:</span><br />
<span style="font-family: monospace;"> </span><span style="font-family: monospace;">df.loc[df['Table_Name'] == '%s' % table, 'Errors'] += 1</span><br />
<br />
The pseudo code above shows that whenever an exception is raised, 'Errors' get 1 added to it, and when it successfully updates a row 'Processed' gets updated.<br />
<br />
The third part of the task was to output the count table to a text file which can be done easily using the <code>to_string()</code> method.
<br />
<br />
<span style="font-family: monospace;">with open(<path>, 'w') as writer:</path></span><br />
<span style="font-family: monospace;"> writer.write(count_df.to_string(index=False, col_space=12, justify='left'))</span><br />
<span style="font-family: monospace;"> writer.flush()</span><br />
<br />
So there you have it. We have a nice human readable output table in a text file. <br />
<br />
Enjoy<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-40583056581959985712016-08-03T05:00:00.000-07:002016-08-03T05:00:07.767-07:00More on Pandas Data Loading with ArcGIS (Another Example)Large datasets can be a major problem with systems that are running 32-bit Python because there is an upper limit on memory use: 2 GB. Most times programs fail before they even hit the 2 GB mark, but there it is. <br />
<br />
When working with large data that cannot fit into the 2 GB of RAM, how can we push the data into DataFrames?<br />
<br />
One way is to chunk it into groups:<br />
<br />
<code>
#--------------------------------------------------------------------------<br />
def grouper_it(n, iterable):<br />
"""<br />
creates chunks of cursor row objects to make the memory<br />
footprint more manageable<br />
"""<br />
it = iter(iterable)<br />
while True:<br />
chunk_it = itertools.islice(it, n)<br />
try:<br />
first_el = next(chunk_it)<br />
except StopIteration:<br />
return<br />
yield itertools.chain((first_el,), chunk_it) </code><br />
<br />
This code takes an iterable object (has next() defined at Python 2.7 or __next__() for Python 3.4) and makes other iterators of size n where n is a whole number (integer).
<br />
<br />
Example Usage:<br />
<code><br /></code>
<code>
import itertools<br />
import os<br />
import json<br />
import arcpy<br />
import pandas as pd<br />
</code>
<br />
<code><br /></code>
<code>with arcpy.da.SearchCursor(fc, ["Field1", "Field2"]) as rows:</code><br />
<code> groups = grouper_it(n=50000, iterable=rows)</code><br />
<code> for group in groups:</code><br />
<code> df = pd.DataFrame.from_records(group, columns=rows.fields)</code><br />
<code> df['Field1'] = "Another Value"</code><br />
<code> df.to_csv(r"\\sever\test.csv", mode='a')</code><br />
<code> del group</code><br />
<code> del df</code><br />
<code> del groups</code><br />
<code><br /></code>
This is one way to manage your memory footprint by loading records in smaller bits.<br />
<br />
Some considerations on 'n'. I found the following effects the size of 'n': number of columns, field length, and data types.<br />
<br />
<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-36213568097737841842016-07-27T05:00:00.000-07:002016-07-27T05:00:46.880-07:00Reading Spatial Data Into a Pandas DataframeAt 10.4.x scipy is included in your basic python install, which is great!<br />
<div>
<br /></div>
<div>
Working with Pandas DataFrame can make life easy, especially if you need to do it quickly.</div>
<div>
<br /></div>
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 12px; line-height: 12px;">
import arcpy
import pandas as pd
import sys
#--------------------------------------------------------------------------
def trace():
"""
trace finds the line, the filename
and error message and returns it
to the user
"""
import traceback
tb = sys.exc_info()[2]
tbinfo = traceback.format_tb(tb)[0]
# script name + line number
line = tbinfo.split(", ")[1]
# Get Python syntax error
#
synerror = traceback.format_exc().splitlines()[-1]
return line, __file__, synerror
with arcpy.da.SearchCursor(r"d:\temp\scratch.gdb\INCIDENTS_points",
["OBJECTID", "SHAPE@X", "SHAPE@Y"]) as rows:
try:
df = pd.DataFrame.from_records(data=rows,
index=None,
exclude=None,
columns=rows.fields,
coerce_float=True)
print ((df.columns[1], df.columns[2]))
print ((df[df.columns[1]].mean(), df[df.columns[2]].mean()))
except:
print trace()
</span>
</span></pre>
<br />
<div>
<br /></div>
<div>
Like normal, you create an arcpy.da cursor, then pass that generator into the DataFrame's from_records(). Once the data is loaded, like in my example, you can perform operations on the frame itself. For example let's say you needed the mean location of points. This can be quickly done by loading in all the location XY columns (SHAPE@X and SHAPE@Y) and performing a mean call on each column.<br />
<br />
With this method you can't control the chunksize when loading the data, so be careful of your memory.<br />
<br /></div>
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-5636977172046561312016-06-10T04:39:00.002-07:002016-06-10T04:39:24.682-07:00ArcREST is now on PyPiInstalling ArcREST just got easier because you can use pip.<br />
<br />
It's easy as:<br />
<br />
<code> pip install arcrest_package </code><br />
<code><br /></code>
<br />
Enjoy to much fanfare.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/08T6Y7Sm_Eg/0.jpg" src="https://www.youtube.com/embed/08T6Y7Sm_Eg?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-78093473342208803882016-02-19T04:27:00.001-08:002016-02-19T04:27:52.603-08:00ArcREST 3.5.3 Help Now OnlineArcREST help documents has been updated with the latest release last week!<br />
<br />
It can be found <a href="http://esri.github.io/ArcREST/index.html" target="_blank">here.</a> (http://esri.github.io/ArcREST/index.html)<br />
<br />
As always check out the project here: http://www.github/com/Esri/ArcREST<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-31243641910704515372016-02-19T04:25:00.001-08:002016-02-19T04:25:13.747-08:0010.4 is ReleasedCheck it out, 10.4 is <a href="https://blogs.esri.com/esri/arcgis/2016/02/18/arcgis-10-4-is-here/" target="_blank">here</a>!<br />
<br />
enjoy<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-89169281214514956612016-01-07T05:30:00.000-08:002016-01-07T05:30:22.976-08:00Opendata Added to ArcRESTI am very proud to say that the open data REST has been added to ArcREST. OpenData sites hosted on ArcGIS.com allow groups to share authoritative information to users with just a few clicks on their site.
Here is a simple usage example:
<br />
<br />
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: Courier New, Courier, monospace;"><span style="font-size: 12px; line-height: 12px;">import arcrest
url = "http://opendata.arcgis.com"
opendata = arcrest.opendata.OpenData(url=url)
#Search by Query
searchResults = opendata.search(q="parcels")
print (searchResults)
</span></span></pre>
<br />
The other big thing that you can do with the open data API is export information:<br />
<br />
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: Courier New, Courier, monospace;"><span style="font-size: 12px; line-height: 12px;">import arcrest
url = "http://opendata.arcgis.com"
itemId = "f59603825818413f87d9d819c3acff88_0"
opendata = arcrest.opendata.OpenData(url=url)
item = opendata.getDataset(itemId=itemId)
print (item.export(outFormat="kml", outFolder=r"c:\temp4"))
#supports: 'shp', 'kml', 'csv', and 'geojson' in the outFormat parameter.
</span></span></pre>
<br />
Hope you enjoy searching and exporting the Open Data site!<br />
<br />
ArcREST can be found <a href="https://github.com/Esri/ArcREST" target="_blank">here</a>.<br />
ArcREST Issues should be logged <a href="https://github.com/Esri/ArcREST/issues" target="_blank">here</a>.<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-24661024848642765432015-11-30T11:40:00.000-08:002015-11-30T11:40:10.960-08:00Unique Values from NumPy ArrayHere is a quick example of how to get a unique value list from a numpy array.
<br />
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: "courier new" , "courier" , monospace;"><span style="font-size: 12px; line-height: 12px;">from __future__ import print_function
import numpy as np
values = [['a', 'b', 'c'],
['e', 'b', 'd'],
['a', 'b', 'c']]
np_array = np.array(values)
print (np.unique(np_array[:,1], return_index=False))
# should see: ['b']
</span>
</span></pre>
Enjoy!<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-90860589075088211782015-11-06T05:00:00.000-08:002015-11-06T05:00:00.709-08:00Extending Python 3.x (online resource)Who doesn't love C++/C, I sure do....<br />
<br />
Here is a great guide to developing all your c/c++ extension <a href="https://docs.python.org/3.4/extending/" target="_blank">here</a> (https://docs.python.org/3.4/extending/).<br />
<br />
Happy Coding!<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-35779672389047486052015-11-06T04:00:00.000-08:002015-11-06T04:00:01.453-08:00Great Blog to Follow (Dan Patterson's GeoNet Blog)If you have used any ArcGIS software over the years, and asked a question on the Esri or GeoStackExchange Forums you have probably encountered Dan Patterson. He has a great blog, which is full of knowledge any programmer, cartographer or GIS analyst should know.<div>
<br /></div>
<div>
I don't know the man, but I love reading his posts. I am sure you will too.</div>
<div>
<br /></div>
<div>
The web address is: https://geonet.esri.com/blogs/dan_patterson</div>
<div>
<br /></div>
<div>
Enjoy.</div>
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-9269184458913635912015-10-27T05:00:00.000-07:002015-10-27T05:00:01.822-07:00ArcREST Help DocumentationArcREST finally has hosted documentation. Check it out <a href="http://esri.github.io/ArcREST/index.html" target="_blank">here</a> (<a href="http://esri.github.io/ArcREST/index.html" style="background-color: white; box-sizing: border-box; color: #4078c0; font-family: 'Helvetica Neue', Helvetica, 'Segoe UI', Arial, freesans, sans-serif; font-size: 16px; line-height: 25.6000003814697px; text-decoration: none;">http://esri.github.io/ArcREST/index.html</a>)!<br />
<br />
Keep on coding.<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-79341047595600977842015-08-31T03:27:00.004-07:002015-08-31T03:27:57.048-07:00The Walking Metadata (hermes example)The <a href="http://www.github.com/esri/hermes">hermes</a> Python package was released (v1) last week, and now I'm going to show how to read the values instead a given metadata file from a feature class.<br />
<br />
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: Courier New, Courier, monospace;"><span style="font-size: 12px; line-height: 12px;">import hermes
if __name__ == "__main__":
fc = r"c:\data_and_maps\usa\census\blkgrp.sdc\blkgrp"
paperwork = hermes.Paperwork(dataset=fc)
data = paperwork.convert()
for k,v in data.iteritems():
print k, v
</span></span></pre>
<br />
In this example, I access the dataset, in this case a feature class. To convert the metadata to a dictionary, call the convert(). Then you can use the dictionary iterator object to walk the data.<br />
<br />
This example is just going to show something like:<br />
<br />
<blockquote class="tr_bq">
metadata {some dictionary value}</blockquote>
<br />
<br />
Enjoy!<br />
<br />
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-2329800151833266962015-08-28T17:05:00.001-07:002015-08-28T17:05:57.670-07:00Work with metadata check out hermes So you have to work with metadata? Or you love paperwork, or you just hate using the Python XML packages? Give hermes a try. It is named after And inspired by Hermes Conrad from Futurama.
It is a simple set of tools to translate the Xml to a Python dictionary.
Check it out here: http://github.com/Esri/hermes
Fork it, follow it, improve it.
I'll be posting a couple of usage examples over the next few weeks.
I look forward to everyone's feedback.<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-34785706100329499582015-08-03T12:30:00.000-07:002015-08-03T12:30:00.668-07:00ArcREST (v3 beta) manageorg updateOn the current master branch, the ArcREST team is up to some good old fashion reworking in the hopes of making life a bit easier when working with portal and AGOL. <br />
<br />
In today's push (8/3/2015). The big update is the manageorg sub package is changed to make the user's content iterable. <br />
<br />
Let's check out an example:<br />
<br />
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: Courier New, Courier, monospace;"><span style="font-size: 12px; line-height: 12px;">import arcrest
admin = arcrest.manageorg.Administration(url=url,
securityHandler=sh)
user = admin.content.users.user("ARandomUserAccount")
for item in user.items:
print item.id, item.access, item.owner, item.ownerFolder
for k in item:
print k
</span></span></pre>
You'll notice two really interesting things. We are accessing a site with a token handler sh, then we use the for loop syntax to loop through all the items in the root folder. Also, the UserItem object, which is returned from the iterator off of user.items also returns Key/Value pairs from the raw JSON. This means you can get really find grained information from the object, not just what is stubbed out in the class objects.
I would like some feedback on this. It was tons of code changes, and wanted to make sure I didn't drop any functions anyone really needed.
Enjoy!
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-81556989638119569022015-07-24T07:00:00.000-07:002015-07-24T07:00:05.524-07:00A Big Thank You And ArcREST Documentation Update!I want to thank everyone for attending my session. It was great to hear what you are doing, and what you want to do with ArcREST, as well as just meeting all of you. It was exciting to know that I was able to make your workday a little easier and hopefully you can REST a bit better knowing that this package is there to help you out!<br />
<br />
I have a lot more work flows to hammer out and ideas to get working on now that I spoke to the users!<br />
<br />
While flying back to my home, I pushed out new documentation for the v3.0.0 ArcREST, which can be found <a href="https://github.com/Esri/ArcREST/tree/master/docs" target="_blank">here</a> (https://github.com/Esri/ArcREST/tree/master/docs) If you clone the repository, the HTML documentation is pulled down. So update your forks, or re-pull the code to get the latest documentation. I must admit though, I am not the best with Sphinx and Latex, so it's bear bones. <br />
<br />
Please feel free to contact me on the issue's page <a href="https://github.com/Esri/ArcREST/issues" target="_blank">here</a>, or submit a pull request to help make the help documentation look better.<br />
<br />
Thank you for attending Esri's UC 2015, attending the ArcREST demo theater presentation, or just talking to me at the developer island. It was a lot of fun to interact with everyone and see all the awesome things people are doing.<br />
<br />
Keep on mapping!<br />
~A<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-32187698333673510522015-07-20T13:33:00.002-07:002015-07-20T13:33:41.697-07:00ArcREST Exciting UpdateThe ArcREST team or people.. released v2 final and moved v3 to the main branch on the ArcREST github page. <br />
<br />
Why is this exciting? We we continue to take all the feedback we get, especially the comments found here: http://github.com/Esri/ArcREST/issues and improve the package. <br />
<br />
Some v3 highlights and what is coming when we finalize v3:<br />
<br />
<ul>
<li>New security handling (PKI, IWA, LDAP)</li>
<li>AGOL metadata support (adding/updating, downloading, deleting)</li>
<li>Improved back end code (you won't see it, but it's there)</li>
<li>Standardized error handler</li>
<li>10.4 support</li>
<li>Python 3 support(separate branch)</li>
<ul>
<li>This will be built on top of Python 3.4+</li>
</ul>
</ul>
<div>
That's some of the what's coming/what's new!</div>
<div>
<br /></div>
<div>
If you are going to the Esri's 2015 User Conference in San Diego, I will be presenting on ArcREST</div>
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-49783208242688424422015-07-20T13:32:00.001-07:002015-07-20T13:32:58.793-07:00UC 2015 and ArcRESTThe Esri User Conference has started with a flurry of great presentations on what ArcGIS Pro will bring. The first half of Monday's Plenary showed that 3D is here and not a niche market anymore. We live in a 3d world so it is only natural that GIS is moving in that direction. Pro is officially released at v1.1 with a new sdk to modify the program even further. On the web side, smart mapping was shown. For non-cartographic folks out there this is BIG. Mapping that helps you present data properly and logically. Even if you design maps daily, it's nice to have smart mapping there to assist you. <br />
<br />
On the ArcREST front, I am presenting on Tuesday @ 1230 in demo theater 11. <br />
<br />
Hope to see you all there.<br />
<br />
<br />
<br /><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-83471230010594185002015-06-30T06:10:00.000-07:002015-06-30T06:10:32.233-07:00The Road Ahead for ArcRESTIf you are going to San Diego this year, you can see me in action presenting in the demo theater about ArcREST. I'm going to do some demos, talk about ArcREST basics, and discuss the road ahead. In turn you can ask questions and learn something new.<br />
<br />
Here are the details:<br />
<span style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 12px; line-height: 16.7999992370605px;"><b>12:30 PM - 01:15 PM : Using the ArcREST Python Package on Tuesday 7/21/2015.</b></span><br />
<br />
As we approach Esri's UC 2015, I wanted to layout what is coming for the next version of ArcREST for those who cannot attend my presentation.<br />
<br />
Here is a brief list of what is coming very soon:<br />
<br />
<ul>
<li>Ntlm, PKI and LDAP security handlers</li>
<li>improved backend handling of GET and POST functions</li>
<li>cleaner code</li>
<li>additional geometry service functions</li>
<li>additional query options</li>
</ul>
<div>
and much more...</div>
<div>
<br /></div>
<div>
The biggest focus is supporting ArcGIS server, portal or online sites that do not use the standard username/password token based security. </div>
<div>
<br /></div>
<div>
The ArcREST team is also taking requests for additional functions that we may have missed or didn't include. The project is built around you the user, so help out and log an enhancement request or an issue <a href="https://github.com/esri/arcrest/issues" target="_blank">here</a>. (https://github.com/esri/arcrest/issues). </div>
<div>
<br /></div>
<div>
<br /></div>
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-89059963932792905442015-06-09T03:00:00.000-07:002015-06-09T03:00:00.455-07:00ArcREST Publishing Part 2 - SD FilesThe easiest and quickest way to get data into AGOL/Portal is to publish from a service definition file, or SD file. SD files are compressed files using 7-zip. They contain all sorts of information about how your service behaves, what it's name is, the service's data, etc... It's a complete package or all in one file for publishing services.<br />
<br />
You can automate the creation of service using Python or AO, but I am not going to dive into that topic. Instead I am going to show how you can easily publish the SD file using ArcREST to your site (AGS, AGOL or Portal). <br />
<br />
This example is going to focus on solely the publishing of SD files to AGOL.<br />
<br />
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: Courier New, Courier, monospace;"><span style="font-size: 12px; line-height: 12px;">import arcrest
import json
import time
#--------------------------------------------------------------------------
if __name__ == "__main__":
username = "someadminuser"
password = "reallysecurepassword"
sd_file = r"C:\temp\1\DemoPublishing.sd"
proxy_port = None
proxy_url = None
# Connect to the site
#
sh = arcrest.AGOLTokenSecurityHandler(username, password)
admin = arcrest.manageorg.Administration(securityHandler=sh,
initialize=True)
content = admin.content
uc = content.usercontent(username)
ip = arcrest.manageorg.ItemParameter()
ip.title = "DemoPublishing"
ip.type = "Service Definition"
ip.tags = "SD, publishing, example"
ip.typeKeywords = "Data, Service, Service Definition"
res = uc.addItem(itemParameters=ip, filePath=sd_file)
itemId = res['id']
# Publish the SD File
#
resPublish = uc.publishItem(fileType="serviceDefinition",
publishParameters=None, itemId=itemId)
jobId = resPublish['services'][0]['jobId']
serviceItemId = resPublish['services'][0]['serviceItemId']
flUrl = resPublish['services'][0]['serviceurl']
status = uc.status(itemId=serviceItemId, jobId=jobId)
while status['status'].lower() == "processing":
time.sleep(2)
status = uc.status(itemId=serviceItemId, jobId=jobId)
print 'service published.'
print 'Bonus Demo - update some additional properties on a feature service'
flAdmin = arcrest.agol.FeatureService(url=flUrl, securityHandler=sh).administration
# Bonus 1- Enable edits Only
updateDefinition = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete","allowGeometryUpdates":true,"editorTrackingInfo":{"enableEditorTracking":false,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition))
# Bonus 2- Enable edits with export data - adds 'Extract' to capabilities
updateDefinition2 = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete,Extract","editorTrackingInfo":{"enableEditorTracking":false,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition2))
# Bonus 3- Enable Sync on services - Adds 'Sync' to capabilities
updateDefinition3 = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete,Sync,Extract","editorTrackingInfo":{"enableEditorTracking":false,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition3))
# Bonus 4- Enable editor tracking on service - set enableEditorTracking = True
updateDefinition4 = """{"hasStaticData":false,"capabilities":"Query,Editing,Create,Update,Delete,Sync,Extract","editorTrackingInfo":{"enableEditorTracking":true,"enableOwnershipAccessControl":false,"allowOthersToUpdate":true,"allowOthersToDelete":true}}"""
print flAdmin.updateDefinition(json_dict=json.loads(updateDefinition4))
print 'finished!'
</span></span></pre>
<br />
<br />
<br />
I added some additional information to the example here. The basic example ends at the print 'service published' part. The additional code below shows how to enable the editing, export data, sync or track edits. You can do this all in call, but I wanted to show how you enable each one at a time because you may only want to enable one function over another. Notice how I get the feature service URL from the publish() then I take that URL and create a Feature Service object from the agol.FeatureService Class. Since my user is an administrator and owner of the service, I can access the back end function to enable the additional properties. On the Feature Service class, there is a property called administration which takes you to the hostingservice.AdminFeatureService class which allows users to update the service definitions on a hosted feature service.<br />
<br />
I threw a lot out there, and I hope it helps!<br />
<br />
Previous posts on publishing:<br />
<a href="http://anothergisblog.blogspot.com/2015/05/arcrest-publishing-part-1-csv-files.html" target="_blank">Part 1 - Publish CSV files can be found here.</a><div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.comtag:blogger.com,1999:blog-3309146341357783035.post-47218198566595614572015-06-08T04:00:00.000-07:002015-06-08T04:00:08.532-07:00Listing Installed Python Packages Here is a quick two liner bit of code to see what Python packages you have installed. I am running this from the command line. Here is the example:<br />
<br />
<pre class="bbcode_code" style="background-color: #f2f6f8; background-repeat: repeat no-repeat; border: 1px inset; direction: ltr; overflow: scroll; padding: 6px;"><span style="color: #333333; font-family: Courier New, Courier, monospace;"><span style="font-size: 12px; line-height: 12px;">>>> import pip
>>> print sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])
['arcrest==2.0.100', 'astroid==1.3.2', 'boto==2.38.0', 'cffi==1.1.0', 'colorama=
=0.3.3', 'cryptography==0.9', 'enum34==1.0.4', 'idna==2.0', 'ipaddress==1.0.7',
'libarchive==0.4.3', 'logilab-common==0.63.2', 'matplotlib==1.3.0', 'nose==1.3.6
', 'numpy==1.7.1', 'pip==7.0.0', 'pyasn1==0.1.7', 'pycparser==2.13', 'pylint==1.
4.0', 'pylzma==0.4.6', 'pyopenssl==0.15.1', 'pyparsing==1.5.7', 'pyzillow==0.4.0
', 'setuptools==17.0', 'six==1.9.0', 'wheel==0.24.0']
</span></span></pre>
<br />
Enjoy!
<div class="blogger-post-footer">Copyright AJC</div>Andrewhttp://www.blogger.com/profile/16991441455885757621noreply@blogger.com