sql - BigQuery handling dependencies and retries in query scheduling

Question

I'm looking for some best (simplest;)) practices here. I have Google Analytics data that is send to BigQuery on a daily basis. I have a query running on a daily basis that uses the data from the previous day's table. However, I can't be sure this table and the data is there at the time the query runs and I'd like to check if it does. If it isn't there I want to retry later. Ideally I have some monitoring/alerting around this as well.

Of course this can be done within the Google Cloud in many ways, I'm looking for some best practices how others do this? I'm used to working with Airflow, but using Composer just for this seems a bit over the top. Cloud Run would be an option and I'm sure there are others. Also I've seen this question discussing how to handle a dependency in SQL, I'm just not sure if I could have it retry using just SQL as well?

EDIT: I've got the check for the table working in SQL. I guess I just have to see if BigQuery has a way to build in delay like 'WAITFOR'

question from:https://stackoverflow.com/questions/66060360/bigquery-handling-dependencies-and-retries-in-query-scheduling

Answer 1

Waitting for answers