Using GitHub Actions to get notified when an API response (or web page) changes

TL;DR: A customizable GitHub Action workflow that uses cURL, jq, and Twilio to notify you via text message when a web page or API response changes.
3 minute read

Given all the uncertainty in the global supply chain these days, I was recently in the dark as to when an eagerly-expected purchase would arrive. Luckily the manufacturer had a hidden API you could call that indicated where in the supply chain the product was as it made its journey around the globe. Better still, as it got closer to delivery, the response also included its estimated delivery date.

Rather than smashing refresh on the endpoint compulsively, I realized I could use GitHub Actions to regularly call the API on my behalf and report back with any changes. These types of easy-on-the-surface projects all too often harbor the risk of over-engineering, so I purposely kept things simple, using cURL to fetch the data, jq to parse the response, and GitHub Action’s native caching library to maintain state between runs. The implementation was surprisingly easy to implement as a no/low code solution with only a handful of lines of YAML, and it ended up working better than expected.1

If you find yourself needing to track when an API response (or web page) changes - delivery related or otherwise - you’re welcome to follow my approach. I used Twilio’s SMS action (along with a free trial account) to send a text message when the response changed, but there are a number of existing community Actions available that could be used for notifications, from GitHub Issues, to app-based mobile push, to Slack - you name it.

Setup

To set up the notify-on-changes GitHub Action to track changes to your own API or web page response, just follow these steps:

  1. Save the below snippet as a YAML file in the .github/workflows/ folder (for example, .github/workflows/check.yml) of a public or private GitHub repository.
  2. Swap out the <URL YOU WANT TO CHECK HERE> placeholder, for the actual API endpoint or web page URL you want to check.2
  3. To parse out the JSON response so that you can include a specific field or fields from the response in the notification you send yourself, modify the jq query. Otherwise, you can remove the “Parse data” step entirely.
  4. If using Twilio to receive notifications via text message, follow the setup instructions, saving the keys and phone numbers as repository secrets and customizing the message for your use case. Otherwise, swap out the “Notify if data has changed” step to use your own notification mechanism.3
  5. Commit and effortlessly receive notifications any time the response changes.

Example workflow

Here’s the annotated YAML for the workflow I used, in its entirety:

on:
  schedule:
    # Run every hour, on the hour. This can be customized to checking as frequently as every 5 minutes.
    - cron:  '0 * * * *'

name: Check for changes

jobs:
  check:
    runs-on: ubuntu-latest
    steps:

      # Use cURL to fetch the given URL, saving the response to `data.json`
      - name: Fetch data
        run: curl "<URL YOU WANT TO CHECK HERE>" -o data.json

      # Optionally, use `jq` to pull one or more fields from the JSON to include in the SMS message
      - name: Parse data
        id: parse_data
        run: echo '::set-output name=someField::'$(jq -r '.someField' data.json)
          
      # Compare the response to the previous run, using a hash of the response as the cache key
      - name: Fetch Cache
        id: cache
        uses: actions/cache@v2
        with:
          path: data.json
          key: ${{ hashFiles('data.json') }}
      
      # If there was not a cache hit (meaning the response changed), notify me via text message
      # See https://github.com/twilio-labs/actions-sms for setup instructions
      # You could use a different notification action here, so long as you include the `if` condition below
      - name: Notify if data has changed
        uses: twilio-labs/actions-sms@v1
        if: steps.cache.outputs.cache-hit != 'true'
        env:
          TWILIO_ACCOUNT_SID: ${{ secrets.TWILIO_ACCOUNT_SID }}
          TWILIO_API_KEY: ${{ secrets.TWILIO_API_KEY }}
          TWILIO_API_SECRET: ${{ secrets.TWILIO_API_SECRET }}
        with:
          fromPhoneNumber: ${{ secrets.from_phone }}
          toPhoneNumber: ${{ secrets.to_phone }}
          message: "There's been a change! someField is now ${{ steps.parse_data.outputs.someField }}."

Conclusion

I hope that my unnecessarily obsessive product tracking can help others to track status changes or generally save others from needlessly hitting refresh repeatedly for other reasons. If you do adapt the workflow for your own creative use, I’d love to hear about it. Happy change tracking!

  1. In fact, the purchase arrived ahead of schedule, but I don’t think I can give the workflow credit for that. ↩

  2. There’s no reason it couldn’t be an authenticated endpoint that you’re checking, just be sure to store your API key as a repository secret, to keep the secret out of your codebase. ↩

  3. If you use a different notification mechanism, be sure to include the if: steps.cache.outputs.cache-hit != 'true' conditional, to ensure that the notification is only sent if the response has changed. ↩

Originally published December 15, 2021 | View revision history

If you enjoyed this post, you might also enjoy:

benbalter

Ben Balter is the Director of Engineering Operations and Culture at GitHub, the world’s largest software development platform. Previously, as Chief of Staff for Security, he managed the office of the Chief Security Officer, improving overall business effectiveness of the Security organization through portfolio management, strategy, planning, culture, and values. As a Staff Technical Program manager for Enterprise and Compliance, Ben managed GitHub’s on-premises and SaaS enterprise offerings, and as the Senior Product Manager overseeing the platform’s Trust and Safety efforts, Ben shipped more than 500 features in support of community management, privacy, compliance, content moderation, product security, platform health, and open source workflows to ensure the GitHub community and platform remained safe, secure, and welcoming for all software developers. Before joining GitHub’s Product team, Ben served as GitHub’s Government Evangelist, leading the efforts to encourage more than 2,000 government organizations across 75 countries to adopt open source philosophies for code, data, and policy development. More about the author →

This page is open source. Please help improve it.

Edit