ArcGIS Online

Updating the ArcGIS Online Components Database View

Summary

We have an ETL (code) that uses the component_arcgis_online_view Moped database view to push data about project components to an AGOL feature service with point, line, and combined_geometry layers (feature service endpoint). This describes the workflow of adding, updating, or removing fields in this service and the coordination it takes with the DTS Geospatial team.

There are two downstream dependencies of this database view that can break:

Code updates

The database view and ETL GraphQL query must be updated with the new or updated field names.

Here is an example pull request: https://github.com/cityofaustin/atd-moped/pull/1357.

Adding a new field

  • Stage code changes for new field

  • Submit a service request for the GIS team to add new field

  • Release or patch to introduce new field into ETL

  • Tell the Data Science team about this update so they can update the PowerBI dataflow

Renaming an existing field

  • Stage code change for field rename

  • Submit a service request for the GIS team to add new column name. We keep the old column name to keep the ETL compatible and working with both the new and old field names

  • Release or patch to introduce new field into ETL

  • Request GIS team remove the old field name now that it is unused

  • Tell the Data Science team about this update so they can update the PowerBI dataflow

Removing an existing field

  • Stage code change for field removal

  • Release or patch to remove column from ETL

  • The ETL will not break with the changes up to this point. The next step can be performed anytime after removing the fields from the code to clean up the AGOL dataset.

  • Submit a service request for the GIS team to remove the field

  • Tell the Data Science team about this update so they can update the PowerBI dataflow

Links

ArcGIS Online Feature Service settings

Before making any changes to a feature service, check with the Geo team and let them know what we are trying to accomplish.

Updating settings

Settings on the feature service level (the AGOL side) can be updated using the Admin Service Catalog endpoint described here. The only setting that we have changed so far is the xssPreventionInfo setting to place the responsibility of sanitizing invalid HTML on the AGOL side.

To update this setting, go to https://services.arcgis.com/0L95CJ0VTaxqcmED/ArcGIS/rest/admin/services/Moped_Project_Components/FeatureServer/updateDefinition?token=.

You will need to fetch a token using the same mechanism we use in our Moped <> AGOL ETL code. Using an API development tool like Postman or Insomnia can be helpful for this, and it will require the same secrets used in the code.

Once a token is retrieved and place in the URL, the UI can be used to make a POST request through the form with new configuration in JSON format placed in the input labeled Update Service Definition.

To update configuration, you should include only the configuration that you are trying to update. Including unneeded configuration will produce errors. See this esri post for more context.

To update the sanitization config, the following JSON was used.

Original configuration shown in the Update Definition input

{
  ...the rest of the config,
  "xssPreventionInfo" : {
    "xssPreventionEnabled" : true, 
    "xssPreventionRule" : "InputOnly", 
    "xssInputRule" : "rejectInvalid"
  },
  ...and a bunch more config
}

Updated configuration sent in input

{
  "xssPreventionInfo" : {
    "xssPreventionEnabled" : true, 
    "xssPreventionRule" : "InputOnly", 
    "xssInputRule" : "sanitizeInvalid"
  }
}

Once you press the Update Definition button to send the payload, you will see a success message or an error if the JSON is invalid.

Links

PreviousDataset documentationNexteCapris

Last updated

Was this helpful?