I have a CSV containing user locations (has lat, long, userid, timestamp). I have a countries shapefile (has country name, geometry - got this from thematicmapping.org). I am trying to append a country attribute to each user point. Here is the script that I managed to write:
import csvimport pandas as pdimport rtreeimport fiona as fifrom shapely.geometry import shape, mapping, Pointwith open("outtest.csv", 'w') as outtest: with fi.open("TM_WORLD_BORDERS.shp",'r') as countries: idx=rtree.index.Index() for feat in countries: fid=int(feat['id']) geom=shape(feat['geometry']) idx.insert(fid,geom.bounds) with open("userloc.csv", 'r') as userloc: reader = csv.DictReader(userloc) for row in reader: point = Point(float(row['longitude']), float(row['latitude'])) for j in idx.intersection(point.coords[0]): if point.within(shape(countries[int(j)]['geometry'])): outtest.write(row['userid']+','+row['timestamp']+','+row['latitude']+','+row['longitude']+','+countries[int(j)]['properties']['NAME']+','+countries[int(j)]['properties']['ISO2']+','+"\n")I am exporting the results to a csv file now and will use the file for further analysis in Spark/Python. This seems to be pretty slow. Considering I have to do this for close to a billion data points, it seems like this might take a very long time. Can you please suggest improvements to this code/workflow?
أكثر...
import csvimport pandas as pdimport rtreeimport fiona as fifrom shapely.geometry import shape, mapping, Pointwith open("outtest.csv", 'w') as outtest: with fi.open("TM_WORLD_BORDERS.shp",'r') as countries: idx=rtree.index.Index() for feat in countries: fid=int(feat['id']) geom=shape(feat['geometry']) idx.insert(fid,geom.bounds) with open("userloc.csv", 'r') as userloc: reader = csv.DictReader(userloc) for row in reader: point = Point(float(row['longitude']), float(row['latitude'])) for j in idx.intersection(point.coords[0]): if point.within(shape(countries[int(j)]['geometry'])): outtest.write(row['userid']+','+row['timestamp']+','+row['latitude']+','+row['longitude']+','+countries[int(j)]['properties']['NAME']+','+countries[int(j)]['properties']['ISO2']+','+"\n")I am exporting the results to a csv file now and will use the file for further analysis in Spark/Python. This seems to be pretty slow. Considering I have to do this for close to a billion data points, it seems like this might take a very long time. Can you please suggest improvements to this code/workflow?
أكثر...