I am a newcomer to ArcGIS and Python so I would greatly appreciate any and all input on this issue. What I am trying to do is very simple: map 400K business locations using lat/lon coordinates onto a shapefile of California census tracts, and then export the data from the spatial join into a text file so that I have a spreadsheet containing the census tract that each business falls into. The input file of lat/lon data is a csv with 400K records, 5 variables and is ~15MB.
I have successfully completed this task interactively in ArcMap 10.2.1, but I would like to have a record of my process for future replication, hence my foray into ArcPy. In ArcMap, this process takes a few minutes. However, my ArcPy script takes multiple hours to complete. It appears to be the CopyFeatures step that requires significant computer resources and so I am not including ArcMap details or ArcPy code beyond this step. Note also that I have run my full ArcPy script on a subset of my data (the first 20 records of my 15MB file) and it works fine and finishes in just a few seconds. Given that, I know the issue is likely with the size of my file, but I have not been able to determine or successfully implement a work around to decrease the memory usage of my code.
Are there any ways I can modify my script to decrease processing time for this simple task? I'm hopeful there's a way to make this as efficient with ArcPy as it is with ArcMap. Please let me know if you have any ideas of how to do this.
ArcMap steps:
# Import modulesimport arcpy, csvimport osfrom arcpy import env# Set environment settingsenv.workspace = "C:/Users/ccwright/Desktop/GIS_workspace"arcpy.env.overwriteOutput = True# Set local variables:in_table = "infousa_latlon_small.csv"x_coords = "longitude"y_coords = "latitude"out_layer = "lat_lon_layer"feature_class = "lat_lon_feature_class.shp"census_tracts_shp = "census_tracts_CA_only/US_tract_2000_CA_only.shp"spatial_join = "spatial_join.shp"spatial_join_output_txt = "spatial_join_data.txt"# Set coordinate system to WGS-1984:WKID = 4326 sr = arcpy.SpatialReference()sr.factoryCode = WKIDsr.create()env.outputCoordinateSystem = sr# Process: Make XY Event Layerarcpy.MakeXYEventLayer_management(in_table, x_coords, y_coords, out_layer, sr, "")# Process: Copy Featuresarcpy.CopyFeatures_management(out_layer, feature_class, "", "0", "0", "0")
أكثر...
I have successfully completed this task interactively in ArcMap 10.2.1, but I would like to have a record of my process for future replication, hence my foray into ArcPy. In ArcMap, this process takes a few minutes. However, my ArcPy script takes multiple hours to complete. It appears to be the CopyFeatures step that requires significant computer resources and so I am not including ArcMap details or ArcPy code beyond this step. Note also that I have run my full ArcPy script on a subset of my data (the first 20 records of my 15MB file) and it works fine and finishes in just a few seconds. Given that, I know the issue is likely with the size of my file, but I have not been able to determine or successfully implement a work around to decrease the memory usage of my code.
Are there any ways I can modify my script to decrease processing time for this simple task? I'm hopeful there's a way to make this as efficient with ArcPy as it is with ArcMap. Please let me know if you have any ideas of how to do this.
ArcMap steps:
- Import shapefile containing census tracts
- Add data -> choose shapefile
- Import and prepare lat/lon data for spatial join
- File -> Add Data -> Add XY Data -> select my 15GB csv -> X Field (longitude) -> Y Field (latitude) -> Coordinate System (WGS 1984)
- Right click on XY data -> Data -> Export data -> All Features -> Output feature class is a .shp saved in my GIS workspace -> upon completion of this step, a pop up displays asking whether want to add to map as layer and I choose Yes (don't think this is significant however)
- File -> Add Data -> Add XY Data -> select my 15GB csv -> X Field (longitude) -> Y Field (latitude) -> Coordinate System (WGS 1984)
# Import modulesimport arcpy, csvimport osfrom arcpy import env# Set environment settingsenv.workspace = "C:/Users/ccwright/Desktop/GIS_workspace"arcpy.env.overwriteOutput = True# Set local variables:in_table = "infousa_latlon_small.csv"x_coords = "longitude"y_coords = "latitude"out_layer = "lat_lon_layer"feature_class = "lat_lon_feature_class.shp"census_tracts_shp = "census_tracts_CA_only/US_tract_2000_CA_only.shp"spatial_join = "spatial_join.shp"spatial_join_output_txt = "spatial_join_data.txt"# Set coordinate system to WGS-1984:WKID = 4326 sr = arcpy.SpatialReference()sr.factoryCode = WKIDsr.create()env.outputCoordinateSystem = sr# Process: Make XY Event Layerarcpy.MakeXYEventLayer_management(in_table, x_coords, y_coords, out_layer, sr, "")# Process: Copy Featuresarcpy.CopyFeatures_management(out_layer, feature_class, "", "0", "0", "0")
أكثر...