Meta Surf Forecast

• Meta Surf Forecast
  • Purpose
  • Developer Setup
  • Adding Spots
  • Data Sources
    • Surfline
      • New API (v2)
        • Responses
        • Requests
      • Old API (v1)
    • MagicSeaweed
    • Spitcast
  • The Magic
    • Surf quality ratings
    • Timestamps
  • TODO

Purpose

Pull data from Surfline, MagicSeaweed & Spitcast APIs to display an aggregated surf forecast.

Developer Setup

1. Install dependencies using Homebrew: brew bundle
2. If on Linux: pg_ctl -D /home/linuxbrew/.linuxbrew/var/postgres start
3. gem install bundler -v '>= 2.0.2'
4. bundle
5. yarn
6. cp config/database.yml.example config/database.yml
7. bin/rails db:setup
8. Grab some Surfline v2 data: SURFLINE_EMAIL=xxx SURFLINE_PASSWORD=yyy bin/rails surfline_v2:update
9. Grab some MagicSeaweed data (requires a valid API key): MSW_API_KEY=xxx bin/rails msw:update (replace xxx with your key)
10. Refresh the materialized Postgres view that collates all forecast data into one table: bin/rails database_views:refresh
11. bin/foreman start -f Procfile.dev
12. Open http://localhost:5000
13. Any changes you make to view files will auto-reload the browser

Pull requests welcome, especially around new data sources/better data visualization (see TODO for suggestions)

Adding Spots

Contributing new spots is easy! Make sure you're signed into your Github account and edit the seeds file:

1. Create a new Region/Subregion if necessary. For example, Los Angeles is created like so:

   CA = Region.find_or_create_by(name: 'California')
   LA = Subregion.find_or_create_by(name: 'Los Angeles', region: CA)
   LA.timezone = 'America/Los_Angeles'
   LA.save!

   You can get valid timezone names from this list.
2. Use this tool to draw a bounding box around the area you want to find spots for.
3. Choose CSV from the dropdown at the bottom & copy the coordinates string.
4. Run rails console, then run SpotFinder.new('{string}').formatted_spots.
5. Copy the output and paste it into the spots array in seeds.rb, then make sure to assign each spot to the right subregion & delete extraneous fields (msw_name, match_type, distance).
6. It's strongly encouraged to add all spots for a particular county or region rather than just a single one. Be a pal!
7. Submit a pull request and I'll get it on the site ASAP!

Use the following as a template. Delete the lines for surfline_v2_id, msw_id, etc, if that spot doesn't exist on that particular site.

  {
    name: 'County Line',
    lat: 34.051,
    lon: -118.964,
    surfline_v2_id: '590927576a2e4300134fbed8',
    msw_id: 277,
    subregion: LA,
  },

Data Sources

Surfline

New API (v2)

Responses

Surfline's new API is undocumented but easy to reverse engineer using their new website's code. Thankfully its structure is much more sane than the old API.

Requests

https://services.surfline.com/kbyg/spots/forecasts/{type}?{params}

For reference, I believe kbyg stands for "Know Before You Go," which is their tagline.

Anywhere there is an optimalScore the value can be interpreted as follows:

However, I have never seen a score of 1 in any of their API responses (only 0 or 2), which is unfortunate when it comes to granularity of ratings. Hopefully this changes in the future.

Old API (v1)

Surfline's old API is undocumented and unauthenticated, but was used via javascript on their website, so it was fairly easy to reverse-engineer. However, they have updated their site & apps to use the new API, and it appears that they've stopped including some critical data in the responses for the old API, so it's disabled in this app for now (and probably forever).

It returned JSON, but with a very odd structure, with each item that is time-sensitive containing an array of daily arrays of values that correspond to timestamps provided in a separate set of arrays. For example (lots of data left out for brevity):

"Surf": {
  "dateStamp": [
      [
        "January 24, 2016 04:00:00",
        "January 24, 2016 10:00:00",
        "January 24, 2016 16:00:00",
        "January 24, 2016 22:00:00"
      ],
      [
        "January 25, 2016 04:00:00",
        "January 25, 2016 10:00:00",
        "January 25, 2016 16:00:00",
        "January 25, 2016 22:00:00"
      ]
    ],
  "surf_min": [
      [
        2.15,
        1.8,
        1.4,
        1
      ],
      [
        0.7,
        0.4,
        0.3,
        0.3
      ]
    ],
}

Requests are structured as follows:

https://api.surfline.com/v1/forecasts/{spot_id}?{params}

This is a breakdown of the params available:

MagicSeaweed

MagicSeaweed has a well-documented JSON API that requires requesting an API key via email. This was a straightforward process and they got back to me quickly with my key.

I've asked MagicSeaweed a few questions and added their responses below:

• "Our API provides 5 days of forecast data, with segments of data provided for each 3 hour interval during that 5 day time span."
• "Our data is updated every 3 hours."

Spitcast

As of February 2020, the Spitcast API has been shut down. I'm waiting to hear back about whether their API will ever come back...

Spitcast only provides a list of API endpoints, but the data is sanely-structured JSON so it's pretty easy to parse.

I've asked Jack from Spitcast a few questions and added his responses below:

• To get more than the default 24 hour forecast for a spot, add dcat=week to the querystring.
• Why does the site show a size range, but the API only returns one size value? "I actually take the API number and create the max by adding 1/6 the height (in feet), and then create the min by subtracting 1/6 the height."
• All possible values for shape:
  • Poor
  • Poor-Fair
  • Fair
  • Fair-Good
  • Good

The Magic

Surf quality ratings

All of the forecasting services (including Surfline v1 vs v2) use different systems for rating waves. I've attempted to normalize them all to a 0-5 (6-point) scale as best as possible, which is perhaps easier to understand when mapped onto the commonly-used Poor-Good scale (some throw Epic in there at the top end, but I went with Very Good):

• 0 => Poor
• 1 => Poor - Fair
• 2 => Fair
• 3 => Fair - Good
• 4 => Good
• 5 => Very Good

Each forecasting service is massaged onto that scale as follows:

• MagicSeaweed: integer fadedRating (0-5) & solidRating (0-5). I simply subtract fadedRating (which is essentially the negative effect of wind) from solidRating.

• Spitcast: ratings in text form:

  • Poor
  • Poor-Fair
  • Fair
  • Fair-Good
  • Good

  I massage these by assigning them a number from 0.5-4.5

• Surfline v1: decimal ratings (0-1) for up to 6 different swells at each spot, as well as an optimalWind boolean. I take the max swell rating at any given time for that spot, multiply it by 5, and then halve it if the wind is not optimal.

• Surfline v2: integer optimalScores for both swell & wind (0-2). Adding these together gives a 0-4 scale, and adding 0.5 puts it on the same 0.5-4.5 scale as Spitcast.

Timestamps

It took me a long time to land on a solution here, but I've finally settled on storing all timestamps in the database in the spot's local time. This defies Rails convention, but makes intuitive sense. If you pull up the forecasts table and look at the timestamp, that's the actual local time at that spot that it's forecast for (even though Rails & Postgres both think it's being stored in UTC). This is typically the format that the forecasting service gives it to us in, and what users want to see it in, so there's no point in doing all sorts of fancy conversion when it should be the same all the way through the pipeline. Now, you may ask, why am I still using the Rails default of TIMESTAMP WITHOUT TIMEZONE, and the answer is that shockingly enough, TIMESTAMP WITH TIMEZONE doesn't actually store timezone data!

TODO

• Figure out a way to convey forecast certainty in charts (ie: most forecasts are in agreement, or they disagree by a wide margin)
• Explore lazy-loading components
• Explore SSR for possibly faster browser paint
• Fetch & display tide/wind/water temperature data from NOAA (they actually have a decent API!)
• Improve charts:
  • Fix timestamp formatting.
  • Account for min/max size forecast. Currently charts just reflect the max.
  • Display forecast quality ratings. Perhaps color each bar different depending on how good the rating is. Surfline also has an optimal_wind boolean that is being crudely integrated into the display_swell_rating method - improvements welcome.
• Fetch & display recent buoy trends that are relevant to each spot to give an idea of when swell is actually arriving.
• Refresh data on a schedule based on when new data is available (refreshing all forecast sources hourly)
• Support multiple timezones as opposed to Pacific Time only
• New Surfline API
• Stop manually seeding the db and figure out a way to pull all spots from each data source and automatically associate them to a canonical spot record (probably using geocoding)
• Dark Theme
• Remove asset pipeline & process CSS w/ webpacker