Wednesday, May 28, 2014

Playing with GeoPackages - Shrinking the Database

Awhile back Esri announced the support of the GeoPackage, which is just a sqlite database that holds vector data, and soon raster data too.

When you add and remove data to the sqlite database, it grows but never shrinks.  Think of it like a river that rises and falls, but the water line always remains.  This waterline is the space used on your hard drive.

To reduce the footprint of the database, you can run the VACUUM command to shrink the data base.
import sqlite3
import arcpy
SQLITE_FILE = r"c:\temp\example.gpkg"
conn = sqlite3.connect(SQLITE_FILE)
conn.execute("VACUUM")
conn.close()

Pretty simple to reduce the size of the database.  What VACUUM does is reconstruct the database from scratch. This will leave the database with an empty free-list and a file that is minimal in size. Note, however, that the VACUUM can take some time to run and it can use up to twice as much temporary disk space as the original file while it is running.

Enjoy

Wednesday, April 30, 2014

10.2.x - Getting Data Extents as Feature Classes (4 Methods)

Over the past couple of years, I have posted many posts on creating polygon extents from data.  Today, I am going to provide 4 ways to get the extent of a polygon.
  1. Use the 'Minimum Bounding Geometry' tool to create 'ENVELOPE' geometry types with or without the 'Group Option'.  This tool is available at the 'Advanced' level, so if you don't know how to/want to use python, this is your go to option to use.  The output is a feature class.
  2. For whole feature class extent use the arcpy.Describe(), and write it out to disk
    import arcpy
    from arcpy import env
    env.overwriteOutput = True
    fc = r"c:\States.shp"desc = arcpy.Describe(fc)
    extent = desc.extent
    pts = [arcpy.Point(extent.XMin, extent.YMin),
           arcpy.Point(extent.XMax, extent.YMin),
           arcpy.Point(extent.XMax, extent.YMax),
           arcpy.Point(extent.XMin, extent.YMax)]
    array = arcpy.Array(items=pts)
    poly = arcpy.Polygon(array)
    
    arcpy.CopyFeatures_management(poly, r"%scratchgdb%\way1")
    
  3. Method 1 for individual features: Use a cursor with extent XMin, YMin, XMax, and YMax, and arcpy.CreateFeatureClass()
    import arcpy
    from arcpy import env
    env.overwriteOutput = True
    fc = r"c:\States.shp"
    out_fc = r"%scratchgdb%\way2"
    if not arcpy.Exists(out_fc):
        arcpy.CreateFeatureclass_management(out_path="%scratchgdb%",
                                            out_name="way2", geometry_type="POLYGON",
                                            spatial_reference=arcpy.SpatialReference(4326))
    icur = arcpy.da.InsertCursor(out_fc, "SHAPE@")
    with arcpy.da.SearchCursor(fc, "SHAPE@") as rows:
        for row in rows:
            extent = row[0].extent
            pts = [arcpy.Point(extent.XMin, extent.YMin),
                   arcpy.Point(extent.XMax, extent.YMin),
                   arcpy.Point(extent.XMax, extent.YMax),
                   arcpy.Point(extent.XMin, extent.YMax)]
            array = arcpy.Array(items=pts)
            poly = arcpy.Polygon(array)
            icur.insertRow([poly])
            del array
            del poly
            del pts
            del extent
            del row
    
  4. Method 2 for individual features: Use a cursor objects and arcpy.CopyFeatures()
    import arcpy
    from arcpy import env
    env.overwriteOutput = True
    fc = r"c:\States.shp"
    out_fc = r"%scratchgdb%\way3"
    with arcpy.da.SearchCursor(fc, "SHAPE@") as rows:
        polys = []
        array = arcpy.Array()    
        for row in rows:
            extent = row[0].extent
            array.add(extent.lowerLeft)
            array.add(extent.lowerRight)
            array.add(extent.upperRight)
            array.add(extent.upperLeft)
            array.add(extent.lowerLeft)
            polys.append(arcpy.Polygon(array))
            array.removeAll()
            del row
        del array
        arcpy.CopyFeatures_management(polys, out_fc)
        del polys
    
Here are 4 examples on how to create extents using python or the built in system tools in ArcToolbox.
 Enjoy

Tuesday, April 29, 2014

Convert Your Custom Object to a Dictionary

Creating your own classes is a great in python.  It's easy and straight forward, and allows python users to mask repetitive code in our scripts, and create custom packages.

Let's assume that you are working with JSON, and you want to create a bunch of custom classes that return the data stored inside of them back as dictionaries.  This can be easily done by implementing the __iter__() in the class you are designing.

The function __iter__() returns an iterator object.  The object is required to support the iterator protocol.

For example, take the class below:

import json
class test(object):
    _a = None
    _b = None
    def __init__(self, a,b):
        self._a = a
        self._b = b
    #----------------------------------------------------------------------
    @property
    def a(self):
        """"""
        return self._a
    #----------------------------------------------------------------------
    @property
    def b(self):
        """"""
        return self._b
    #----------------------------------------------------------------------
    def __str__(self):
        """ returns object as string """
        o = {}
        for k,v in self.__iter__():
            o[k] = v
        return json.dumps(o)
        
    #----------------------------------------------------------------------
    def __iter__(self):
        """ iterator generator for public values/properties """
        attributes = [attr for attr in dir(self)
                      if not attr.startswith('__') and \
                      not attr.startswith('_')]
        for att in attributes:
            yield (att, getattr(self, att))

Here __iter__() and __str__() are implemented, along with two properties 'a' and 'b'.  From the user's standpoint, they are probably only interested in the public properties, so in the __iter__(), the properties without '__' and '_' are returned in a key/value pair object.  __str__() returns the data as JSON using the json library built into python.  __str__ is another built in class that is called when you do either a print or print str().

The result if you print out as a string is: {"a": 1, "b": 2}
For the dictionary: {'a': 1, 'b': 2}

The first is a string and the second is dictionary.

Enjoy

Friday, March 21, 2014

Searching a List of Dictionaries

Often when I work in Python, I get JSON text which I convert using the JSON module in python.  This is great for me because I now have lots of data in dictionaries and lists.  Dictionaries are easy to use because everything is in a Key/Value format.  I view it as the data has nice neat labels.

Sometimes there is a need to search for data, and if that dictionary is in a list of dictionary items, how can you easily check if that item exists within a given list of dictionaries?  Enter the built-in function called any.

Python's 2.7.x help describes any as:
Return True if any element of the iterable is true. If the iterable is empty, return False.
Any it is a short cut for:
def any(iterable):
    for element in iterable:
        if element:
            return True
    return False 

Now let's use the function.  Assume we have a given list, aListofDicts, that contains n number of dictionaries defined as such:
aListofDicts = [{"alpha": "a"}, {"alpha": "b"},
               {"alpha": "joker"},{"alpha": 1.2}]

To search this data, one would off the cuff have to write the above function to find the value in an iterable, but any gives us the magic shortcut because no one wants to write long scripts.
>>> print any(d['alpha'].lower() == 'a'.lower() for d in aListofDicts)
True
>>> print any(d['alpha'].lower() == 'tank'.lower() for d in aListofDicts)
False

So what happened in the above code?  We used generator expression, which is just a big word that creates an object that we can iterate on, ie a list.  That is this part:
aListofValues = [d for d in aListofDicts]
 Next we appends a conditional to store a list of boolean values (True/False)
aListofValues = [d[''] == "
This gives you a list of True/False values instead of a list of the actual dictionary or dictionary values.  This means if you have 1 million items in your list, you will have 1 million True/False values.  Not very efficient.

Now enters the any function:
exists = any([d['alpha'] == "
As soon as the list hit that True, if it's in position 1 or position 1 million of the list, it will stop the iteration, thus saving time and memory.  Hooray!

Happy Coding!


Wednesday, March 12, 2014

ArcGIS Pro and Python

At the Esri DevSummit 2014, it was announce that Python 3.4 will be the future of the language.

Goodbye 2.7.x and hello 3.4.

Most programmers will probably want to brush up on Python 3.x, so check out Dive into Python 3! http://getpython3.com/diveintopython3/

Enjoy


Monday, February 10, 2014

SQL Fiddle - Your Database Scratchpad

Ever want to test something out on your database but did not want to mess everything up. Now is your chance, check out: sqlfiddle.  Just go to this site and select your database.  You can then create a quick schema and dummy data and bam! Test away.


Enjoy

Tuesday, February 4, 2014

Add Geometry Attributes - ArcGIS 10.2.1 Tool

New at 10.2.1 is a simplified ways of adding geometry properties to a feature class.  The Add Geometry Attributes geoprocessing tool is a short cut from using the calculate field's geometry calculations.

Let's use the old way first:

    fc = r"some datasource"
    arcpy.AddField_management(fc, fieldName1, "DOUBLE", 
                              fieldPrecision, fieldScale)
    arcpy.AddField_management(fc, fieldName2, "DOUBLE", 
                              fieldPrecision, fieldScale)
 
    # Calculate centroid
    arcpy.CalculateField_management(fc, fieldName1, 
                                    "!SHAPE.CENTROID.X!",
                                    "PYTHON_9.3")
    arcpy.CalculateField_management(fc, fieldName2, 
                                    "!SHAPE.CENTROID.Y!",
                                    "PYTHON_9.3")
So here we create the field and then perform a calculation on the data set to get the X/Y centroid of the data.

At 10.2.1 you can clean this up and just do the following:

fc = "some dataset"
arcpy.AddGeometryAttributes_management(fc, "CENTROID")
With just two lines of code, we did what took multiple line previously to 10.2.1.

The Add Geometry Attributes tool is a time saver, and makes life just a tad easier.  It's a convenient tool that allows for quick addition of geometry attributes.

Check out more about this tool here.

Enjoy