GEOACE is a US-based full-service geospatial firm that specializes in streamlining workflows and empowering users through the use of location. Recently, one of GEOACE’s local government clients turned to GEOACE for a tool that could provide a daily refresh of a county's parcel layer in their ArcGIS Online account, which is available for consumption only as a REST service. That parcel data also needed to be spatially joined with several of the client's data layers.
Furthermore, they wanted a backup of their entire ArcGIS Online (AGOL) organization feature service library as they didn’t have one. Without an available server/computer to run the processes or store the data, GEOACE had to use one of their machines or turn to the cloud.
Creating and formalizing a GIS data processing pipeline
The solution created by GEOACE is a data automation pipeline that fits in a docker container and is designed for the cloud. A data pipeline is a method in which raw data is ingested from various data sources, transformed, and then ported to a data store, such as a data lake or data warehouse, for analysis. A Docker container image is a lightweight, standalone, executable package of software that includes code, runtime, system tools, system libraries, and settings. ArcGIS API for Python is a Python library for performing GIS visualization and analysis, spatial data management, and GIS system administration tasks within the Esri ecosystem.
GEOACE has a lot of local government clients that rely on county-level data and want to merge it with their own data. The cloud is the obvious solution here as these clients often do not have a local machine to run the solution on, or they ask Chief Technology Officer Aaron Laver to spin something up for them. With so many clients having the same needs, Laver figured it would make sense to create and formalize a configurable data pipeline that automates certain GIS data processes, with deployment included, so that others can also use it.
Turning to the cloud
GEOACE’s data pipeline is built on open-source software and uses the freely available ArcGIS API for Python when necessary. “My goal was to make something in a standalone Docker container that could be deployed through the cloud”, explains Laver. “This would save costs for not having to run a virtual machine all the time, and only having to pay for CPU consumption. As cloud providers offer free tiers that take up to millions of requests for free, we went that route to manage these.”
Laver explains that the data pipeline is about taking source data, transforming it in some way, and then exporting it into the client organization either via ArcGIS Online or PostgreSQL. “Our client needed the source data, which are county parcels, then needed to merge that data with multiple different layers, and finally needed to store it in their ArcGIS Online organization for inter and intradepartmental workflows. We opted to create a separate step for the data transformation side of things by taking the county data REST service and putting its data in our PostgreSQL database. From there, we perform spatial joins on the target layers of the client data, which then goes back from PostgreSQL to ArcGIS Online.”
The solution is meant to be an enabler for organizations that rely on both ArcGIS Online and open source workflows with a need to move agency data from one to the other. “Our data pipeline goes both ways. We try to be agnostic toward which solution our clients go with, and the goal is to make people’s lives easier, no matter the solution.”
The road ahead: a Kickstarter campaign
After creating the solution for the client, Laver published a LinkedIn post describing the problem and solution. He was floored by the response he received from that post. What it conveyed to him is that for many people it’s a real pain point to keep an abundance of data up to date to a point where it’s still valuable.
Also announced in the LinkedIn post was the decision to make the data pipeline solution open source. The company will launch a Kickstarter campaign soon to make the solution open source and take it a step further, with newly added features. “This way, our solution is made more easily accessible to organizations with minimal resources, where a single person is responsible for many tasks, including GIS.”
The Kickstarter website shows the different components of GEOACE’s cloud-native data automation software, RESTerville, which has been developed specifically for geospatial workflows. The feature roadmap shows completed features and others that are in the pipeline. Once complete, the brand new toolkit will be made public and open source forever.
