Refreshing pre-aggregations

Refresh worker responsibilities, key environment variables for scheduled builds, and troubleshooting when refresh intervals fall behind.

Pre-aggregation refresh is the process of building pre-aggregations and updating them with new data. Pre-aggregation refresh is the responsibility of the refresh worker.

Configuration

You can use the following environment variables to configure the refresh worker behavior:

• CUBEJS_REFRESH_WORKER (see also CUBEJS_PRE_AGGREGATIONS_BUILDER)
• CUBEJS_PRE_AGGREGATIONS_SCHEMA
• CUBEJS_SCHEDULED_REFRESH_TIMEZONES
• CUBEJS_DB_QUERY_TIMEOUT
• CUBEJS_REFRESH_WORKER_CONCURRENCY (see also CUBEJS_CONCURRENCY)
• CUBEJS_SCHEDULED_REFRESH_QUERIES_PER_APP_ID
• CUBEJS_DROP_PRE_AGG_WITHOUT_TOUCH
• CUBEJS_PRE_AGGREGATIONS_BACKOFF_MAX_TIME

Troubleshooting

Refresh scheduler interval error

Sometimes, you might come across the following error:

{
  "message": "Refresh Scheduler Interval Error",
  "error": "Previous interval #2 was not finished with 60000 interval"
}

It indicates that your refresh worker is overloaded. You probably have a lot of tenants, a lot of pre-aggregations to refresh, or both.

If you’re using multitenancy, you’d need to deploy several Cube clusters (each one per a reduced set of tenants) so there will be multiple refresh workers which will work only on a subset of your tenants.

If you’re using Cube Cloud, you can use a Multi-cluster deployment that would automatically do this for you.