This might not be an ideal question for GIS.SE and a word like 'long' cannot be defined easily, but pleae read the stuff below, and let me know if you have any suggestions.
For the first time I am not just executing a few lines of code but I am working on a very complex script that creates way too many intermediate outputs that all need to be acessed again later on. Ideally I would simply throw these otputs into memory, but after some time Arc keeps spitting out strange outputs (basically, the inputs; it is not doing anything) - can in_memory be too overloaded? I don’t know. Anyways, during development, with a reduced dataset, things like listing feature classes seemed to take too long for what they were doing, but the time spent was acceptable, but now, using real data (much larger) it is these things that truly kill my script. For example, calling arcpy.ListFeatureClasses() in a workspace with 400 shapefiles, takes at least 10 minutes. I cannot tell how long it truly takes, as I lost patience and force-crashed my script. So, my question is, what exactly is ArcGIS doing here that takes so incredibly long?
I am now using the os module to achieve the same affect. See here:
import osallSingleNonRoadPolys = []for file in os.listdir(r'D:\data\allSingleNonRoadPolys'):if file.endswith(".shp"): print(file) allSingleNonRoadPolys.append(file)These few lines of code append all files ending with .shp (as a string) in a folder to a list, and the whole thing takes less than a second (for 400 shapefiles). You can then achieve the same effect as ListFeatureClasses by simply prepending the workspace to the filename, each time you need to use a dataset, as seen here:
for shp in allSingleNonRoadPolys: arcpy.someTool(workspace + '\\' + shp)Why would I even keep using ListFeatureClasses() and what makes it take so long? Does it do anything incredibly useful that cannot be achieved in any other way?
أكثر...
For the first time I am not just executing a few lines of code but I am working on a very complex script that creates way too many intermediate outputs that all need to be acessed again later on. Ideally I would simply throw these otputs into memory, but after some time Arc keeps spitting out strange outputs (basically, the inputs; it is not doing anything) - can in_memory be too overloaded? I don’t know. Anyways, during development, with a reduced dataset, things like listing feature classes seemed to take too long for what they were doing, but the time spent was acceptable, but now, using real data (much larger) it is these things that truly kill my script. For example, calling arcpy.ListFeatureClasses() in a workspace with 400 shapefiles, takes at least 10 minutes. I cannot tell how long it truly takes, as I lost patience and force-crashed my script. So, my question is, what exactly is ArcGIS doing here that takes so incredibly long?
I am now using the os module to achieve the same affect. See here:
import osallSingleNonRoadPolys = []for file in os.listdir(r'D:\data\allSingleNonRoadPolys'):if file.endswith(".shp"): print(file) allSingleNonRoadPolys.append(file)These few lines of code append all files ending with .shp (as a string) in a folder to a list, and the whole thing takes less than a second (for 400 shapefiles). You can then achieve the same effect as ListFeatureClasses by simply prepending the workspace to the filename, each time you need to use a dataset, as seen here:
for shp in allSingleNonRoadPolys: arcpy.someTool(workspace + '\\' + shp)Why would I even keep using ListFeatureClasses() and what makes it take so long? Does it do anything incredibly useful that cannot be achieved in any other way?
أكثر...