Working with numpy's Structured Array and numpy.dtype

In my previous post, I showed how to quickly get access data (2007/2010) into a file geodatabase without creating an ODBC connection in ArcCatalog/Map using pyODBC.  You might have noticed that I used numpy to create a table pretty easily, but you might be wondering what are the dtypes?

numpy.dtype are data type objects that describe how the bytes in a fixed-size block of memory are seen by the system.  So is the data a string, number, etc...  It describes the follow aspects of data:

1. Type of data (integer, float, object, ...)
2. size of data (number of bytes)
3. describes an array of data
4. names of fields in the record
5. if the data is a sub-array, it's shape

Getting the dtype formats for common python data types if fairly easy in python. The numpy.dtype() will return the proper format for almost any python data type:

>>> print numpy.dtype(str)
|S0

For array protocol type strings, there are various data types supported:

'b'	Boolean
'i'	(signed) integer
'u'	unsigned integer
'f'	floating-point
'c'	complex-floating point
'S', 'a'	string
'U'	unicode
'V'	anything (void)

(source: numpy help documents)

This allows you to specify thing like string length.
Example:

>>> dt = numpy.dtype('a25')  # 25-character string

After you know what your data types are, you will want to associate these types with the fields in the records.  I prefer to use the optional dictionary method where there are two keys: 'names' and 'formats'.  You would then pass this information to create your 'structured array'.

Example:

>>> dt = numpy.dtype({'names': ['r','g','b','a'],

     'formats': [numpy.uint8, numpy.uint8, numpy.uint8, numpy.uint8]})

>>> colors = numpy.zeros(5, dtype = dt)

>>> print colors

[(0, 0, 0, 0) (0, 0, 0, 0) (0, 0, 0, 0) (0, 0, 0, 0) (0, 0, 0, 0)]

There are many ways to declare dtypes for your data, and you can read them all here.
More on structured arrays in numpy can be found here.

Enjoy

Using pyODBC to Connect To Access with ArcPy

pyODBC allows users to access any data source with the proper ODBC driver installed on the system.  This is very convenient, and helpful because at 10.1 using ArcPy only you cannot create ODBC connection.

You can download pyODBC here.  Grab the correct python version and install it.

Once installed, let's try to access a 2010-2012 MS Access Database.

I create a dummy database called 'db_text.accdb' and create a table called 'Addresses' with 5 text fields: 

1. name
2. street
3. town
4. country
5. zipcode

After that I populated two rows with dummy data.  Once you have some dummy data, let's move forward.

import pyodbc

import arcpy

import numpy

arcpy.env.overwriteOutput = True

accb = r"C:\temp\db_test.accdb"
access_con_string = r"Driver={Microsoft Access Driver (*.mdb, *.accdb)};Dbq=%s" % accb
cnxn   = pyodbc.connect(access_con_string)
cursor = cnxn.cursor()
cursor.execute("select * from Addresses")
rows = cursor.fetchall()

dts = {'names': ('ID','name','street', 'town', 'country', 'zipcode'),
       'formats':(numpy.uint8, 'S255','S255','S255','S10','S10')}

array = numpy.rec.fromrecords(rows, dtype=dts)

arcpy.da.NumPyArrayToTable(array, r"c:\temp\scratch.gdb\outTable")

Here we connected to the MS Access database using the drivers installed on my local system. Next I performed a simple query to return all records from the table. With the new da.NumPyArrayToTable() at 10.1, I want to convert the pyodbc rows to a table that can be used in ArcMap. I convert the list of tuples (rows object) to a numpy.array object. After the conversion, I fire off the arcpy function and now I have a table on disk.

For a complete listing of 10.1 numpy functions, check out: http://resources.arcgis.com/en/help/main/10.1/index.html#/What_is_the_data_access_module/018w00000008000000/

There you can learn about extending tables, convert feature classes to numpy arrays, and rasters as well.

Enjoy

Dynamic Layers and Workspaces

An interesting and exciting aspect that I really like at 10.1 is the use of dynamic map services to display data from a work space on disk.

The exciting aspect is this:

• Dynamic Rendering
• Add 1:n layers to a map service

Where is gets interesting is this, in ArcPy, you cannot update symbology for feature classes.  So essentially if you wanted to add a feature class to a map document, you have to accept the default symbology, or apply the symbology from an already existing set of layer files (.lyr).  Not real dynamic, and at least I feel, a horrible choice.  

Leveraging the web map REST specifications, as a developer you could create a map object (mapping.MapDocument object) and export the display to PNG, PDF, JPG, etc...  The downside is that at 10.1, it appears you cannot save the MapDocument object to disk and view the dynamic layer in the TOC.  This will throw an error when you open the map document file.

So what is this good for you are probably thinking?  It's true you cannot view dynamic layers in 10.1, but the ability to use dynamic layers gives data driven pages the much needed shot in the arm to add layer on the fly from a web application.  The new printing services for ArcGIS for Server leverages this concept except the out of box tool only produces one export at a time, where as a data driven page tool would produce multiple exports from a single input.

To get this concept to work, you would have to do the following:

import arcpy

from arcpy import mapping

json = arcpy.GetParameterAsText(0)
results = mapping.ConvertWebMapToMapDocument(json)

mxd = results.mapDocument

mapping.ExportToPDF(mxd, r"c:\temp\export.pdf")

Here we created a new map object from a JSON string input and dumped it out to a PDF.  (JSON not included here)

ConvertWebMapToMapDocument() has some other very interesting functions/properties.  It can incorporate existing map document templates, and even extract Point, Lines and Polygon graphics to feature classes when passed as JSON to the function.

You can find more about ConvertWebMapToMapDocument here: http://resources.arcgis.com/en/help/main/10.1/index.html#//00s30000006n000000.  In addition to the function you can also find examples of web map JSON so you can quickly start playing around with the functions.


Please support this idea of having GUI designer in python built in with python add-ins.

It can be found here. (direct link: http://ideas.arcgis.com/ideaView?id=087E00000004SmHIAU)

Enjoy

Great Learning Python Resource

Hey Everyone,

The Internet is filled with great resources for learning programming, almost too much, but here is a great resource: http://learnpythonthehardway.org/book/

This site hosts a e-book called 'Learn Python The Hard Way' and I highly recommend if you are starting out programming, or want to just a refresher course on python that you take a look.

There are also courses on C, CLI, SQL, Ruby, and Regex.

You might learn something new, so give it a try.

As always, I encourage my reader to please support my idea on the ArcGIS Ideas Site to allow GUI integration with ArcPy.  Please vote it up here: http://ideas.arcgis.com/ideaView?id=087E00000004SmHIAU

Enjoy