# Open Data Portal - Socrata Integration ### Adding New Fields to an Existing Socrata Dataset This doc describes the process for adding new fields to an existing Socrata dataset whose source is a Knack application. #### 1. Locate the API View Locate the API view in the Knack application that serves as the source for the Socrata dataset. Each of our Knack applications will have a series of views (aka, pages) which we call "API Views" and are purpose-built for publishing data from Knack to external sources (Socrata, ArcGIS Online, etc.). If you're unsure which view is the source for a data set, you'll need a developer to review the ETL script which publishes the Knack data to the destination system. They will be able to confirm the page and scene which serves as the source. The majority of our Socrata and AGOL publishing scripts live in the `atd-data-publshing` repo, and are defined in [this configuration file](https://github.com/cityofaustin/atd-data-publishing/blob/master/transportation-data-publishing/config/knack/config.py). #### 2. Field Naming Field naming: In Knack parlance, field names are called "Labels". There are very few restrictions in Knack around how you can name a field. It can have spaces, numbers, mixed case, etc. This is because Knack maintains it's own unique field identifiers which you as a builder cannot modify, e.g. `field_101`, etc. Crucially, we use the user-defined Knack field label as the field name in the downstream system. In Socrata, these field names are called the "API Field Names", and they must be lower case, with no spaces or special characters. In ArcGIS Online, these fields may be upper or lower case, with no spaces or special characters. As such, the field "label" of any Knack field which is to be published to an external system should follow these same conventions. Typically, we follow the practice of labeling these fields in upper case with an underscore in lieu of spaces. *Our Knack > Socrata publishing script will convert any Knack field name to lowercase to match Socrata's expected field format.* #### 3. Update the Knack View Once you've located the API view in the Knack builder, first verify (1) that the fields are named appropriately (as described above) and (2) if the columns you want to include in the destination dataset are already present in the view. If they fields are not present, add them. Otherwise, continue to the next step. #### 4. Update Socrata Dataset 1. Login to [datahub.austintexas.gov](https://datahub.austintexas.gov/) and navigate to the `Primer` page of the dataset you wish to modify. Scroll down to the`Columns in this Dataset`section and verify that the fields you wish to add are not present in the dataset. To add new fields, click on the blue `Edit`button in the top right corner of the Primer page. 2. Click on the button below `Review & Configure Data`. From the configuration editor, click the `Add Column` button. Fill out the form fields as follows: 3. **Display Name:** A human-friendly field label 4. **Field name:** this is the "API Field Name" which should exactly match the Knack field label (*with the exception that value should be lower case. See note in step #2.*) 5. **Description:** a detailed description of what values the field contains, the purpose it serves, etc. 6. **Source Column:** Select "No Source column" 7. **Type:** choose the appropriate field type. 8. Click `Save`to save your changes. Return to the Primer page, and click the blue `Update` button in the top-right of the page to publish the dataset changes. #### 5. Refresh the Dataset In order to refresh the Socrata dataset with values for the new columns, you should run a batch update in the Knack builder to set the modified date of all records to the current date. This will enable the integration script to update the destination datasets the next time it runs. If your dataset contains more than \~2k rows, ask a dev to manually run the dataset publishing script with a `--replace` command. Contact a dev for help. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://atd-dts.gitbook.io/atd-knack-operations/knack-integrations/open-data-portal-socrata-integration.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.