Wednesday, August 3, 2016

More on Pandas Data Loading with ArcGIS (Another Example)

Large datasets can be a major problem with systems that are running 32-bit Python because there is an upper limit on memory use: 2 GB.  Most times programs fail before they even hit the 2 GB mark, but there it is.

When working with large data that cannot fit into the 2 GB of RAM, how can we push the data into DataFrames?

One way is to chunk it into groups:

#--------------------------------------------------------------------------
def grouper_it(n, iterable):
    """
    creates chunks of cursor row objects to make the memory
    footprint more manageable
    """
    it = iter(iterable)
    while True:
        chunk_it = itertools.islice(it, n)
        try:
            first_el = next(chunk_it)
        except StopIteration:
            return
        yield itertools.chain((first_el,), chunk_it) 


This code takes an iterable object (has next() defined at Python 2.7 or __next__() for Python 3.4) and makes other iterators of size n where n is a whole number (integer).

Example Usage:

import itertools
import os
import json
import arcpy
import pandas as pd


with arcpy.da.SearchCursor(fc, ["Field1", "Field2"]) as rows:
     groups = grouper_it(n=50000, iterable=rows)
     for group in groups:
         df = pd.DataFrame.from_records(group, columns=rows.fields)
         df['Field1'] = "Another Value"
         df.to_csv(r"\\sever\test.csv", mode='a')
         del group
         del df
     del groups

This is one way to manage your memory footprint by loading records in smaller bits.

Some considerations on 'n'.  I found the following effects the size of 'n': number of columns, field length, and data types.




Wednesday, July 27, 2016

Reading Spatial Data Into a Pandas Dataframe

At 10.4.x scipy is included in your basic python install, which is great!

Working with Pandas DataFrame can make life easy, especially if you need to do it quickly.


import arcpy
import pandas as pd
import sys
#--------------------------------------------------------------------------
def trace():
    """
        trace finds the line, the filename
        and error message and returns it
        to the user
    """
    import traceback
    tb = sys.exc_info()[2]
    tbinfo = traceback.format_tb(tb)[0]
    # script name + line number
    line = tbinfo.split(", ")[1]
    # Get Python syntax error
    #
    synerror = traceback.format_exc().splitlines()[-1]
    return line, __file__, synerror

with arcpy.da.SearchCursor(r"d:\temp\scratch.gdb\INCIDENTS_points",
                           ["OBJECTID", "SHAPE@X", "SHAPE@Y"]) as rows:
    try:
        df = pd.DataFrame.from_records(data=rows,
                                       index=None,
                                       exclude=None,
                                      columns=rows.fields,
                                      coerce_float=True)
        print ((df.columns[1], df.columns[2]))
        print ((df[df.columns[1]].mean(), df[df.columns[2]].mean()))


    except:
        print trace()




Like normal, you create an arcpy.da cursor, then pass that generator into the DataFrame's from_records().  Once the data is loaded, like in my example, you can perform operations on the frame itself.  For example let's say you needed the mean location of points.  This can be quickly done by loading in all the location XY columns (SHAPE@X and SHAPE@Y) and performing a mean call on each column.

With this method you can't control the chunksize when loading the data, so be careful of your memory.

Friday, June 10, 2016

ArcREST is now on PyPi

Installing ArcREST just got easier because you can use pip.

It's easy as:

pip install arcrest_package


Enjoy to much fanfare.


Friday, February 19, 2016

ArcREST 3.5.3 Help Now Online

ArcREST help documents has been updated with the latest release last week!

It can be found here. (http://esri.github.io/ArcREST/index.html)

As always check out the project here: http://www.github/com/Esri/ArcREST


10.4 is Released

Check it out, 10.4 is here!

enjoy

Thursday, January 7, 2016

Opendata Added to ArcREST

I am very proud to say that the open data REST has been added to ArcREST. OpenData sites hosted on ArcGIS.com allow groups to share authoritative information to users with just a few clicks on their site. Here is a simple usage example:

import arcrest
url = "http://opendata.arcgis.com"
opendata = arcrest.opendata.OpenData(url=url)
#Search by Query
searchResults = opendata.search(q="parcels")
print (searchResults)


The other big thing that you can do with the open data API is export information:

import arcrest
url = "http://opendata.arcgis.com"
itemId = "f59603825818413f87d9d819c3acff88_0"
opendata = arcrest.opendata.OpenData(url=url)
item = opendata.getDataset(itemId=itemId)
print (item.export(outFormat="kml", outFolder=r"c:\temp4"))
#supports: 'shp', 'kml', 'csv', and 'geojson' in the outFormat parameter.

Hope you enjoy searching and exporting the Open Data site!

ArcREST can be found here.
ArcREST Issues should be logged here.


Monday, November 30, 2015

Unique Values from NumPy Array

Here is a quick example of how to get a unique value list from a numpy array.
from __future__ import print_function
import numpy as np

values = [['a', 'b', 'c'],
          ['e', 'b', 'd'],
          ['a', 'b', 'c']]
np_array = np.array(values)
print (np.unique(np_array[:,1], return_index=False))

# should see: ['b']

Enjoy!