Ready Airflow

Welcome to Ready Airflow’s documentation!

Pattern Registry

Before creating new operators, utilities, DAG scaffolds, or SQL patterns, check: - docs/patterns/registry.md

Operators

The following are custom operators that are used throughout the project. They are organized by the type of operator they provide. In general, operators should be created if they contain central logic that is used in multiple places. If the logic is only used in one place, it should likely be kept in the DAG itself.

BaseTransformOperator

This operator is used to transform data. It is used in the transform task of the BaseDAG.

class operators.base_transform.BaseTransformOperator(source_conn_id, dest_conn_id, dest_schema_name, dest_table_name, truncate=False, source_sql='', if_table_exists='replace', query_limit=0, *args, **kwargs)

Base class to transform data from one connection to another. Uses sqlalchemy and pandas. Defaults to SQL Server. :type source_conn_id: :param source_conn_id: the source onnection ID :type source_conn_id: str :type source_sql: :param source_sql: the source SQL statement. Should return the data to be

saved to the destination. (templated)

Parameters:

• dest_conn_id (str) – the destination connection ID

• dest_schema_name (str) – the name of the destination schema

• dest_table_name (str) – the name of the destination table

• truncate (bool) – whether to truncate the destination table

• if_table_exists (str) – what to do if destination table exists. Can be fail, append, replace. Default fail.

• query_limit (int) – how many rows to pass in each chunk

CHECK_TABLE_EXIST = "SELECT EXISTS (SELECT FROM pg_tables WHERE schemaname = '{}' AND tablename = '{}');"

DEFAULT_QUERY = 'SELECT * FROM {}.{}'

DROP_QUERY = 'DROP TABLE {};'

TRUNCATE_QUERY = 'TRUNCATE TABLE {};'

static check_result_exist(result)

wrapper for db differences - depending on what the CHECK_TABLE_EXIST query does, may need to do this differently

check_truncate_table(dest_conn, dest_schema_table)

delete all contents of the given table before writing if necessary

chunk_query(source_conn, dest_conn, dest_schema_table)

chunk the query result to obey memory limits

df_to_sql(df, dest_conn, dest_schema_table, if_exists=None)

helper for repeated code - write as a txn

execute(context)

main call method - creates two engines and feeds them to pandas

get_dest_hook()

add dest hook that is able to make a sqlalchemy connection

get_source_hook()

add src hook that is able to make a sqlalchemy connection

template_fields: Sequence[str] = ('source_sql', 'dest_table_name', 'dest_schema_name')

transform_query(source_conn, dest_conn, dest_schema_table)

read the entire query in a df then write it to a db

LinearIssuesToS3Operator

This operator is used to ingest Linear issues from its API into S3.

class operators.linear_issues_to_s3.LinearIssuesToS3Operator(*args, **kwargs)

Custom operator to fetch issues from Linear API. Store as parquet files in S3.

• fetches linear_api_key from Airflow connections

• s3_bucket_name is default

• s3_key is default – partition data in S3 but when creating Athena table, use the latest data

  (i.e., table in Athena will always point to the latest data in S3 – overwritten)

execute(context)

Get the Linear API key from Airflow connections

Return type:

None

static format_date(date_str)

Formate date string for readability if it exists

Parameters:

date_str (str) – The date string to format

Returns:

Formatted date string or None if input is empty

Return type:

str or None

format_issue()

Format issue data for display with safe attribute access

Parameters:

issue (dict) – The issue data to format

Returns:

A dictionary with formatted issue data

Return type:

dict

static get_all_linear_issues(api_key)

Fetch ALL Linear issues with pagination and retry logic.

Parameters:

api_key (str) – Linear API key

Returns:

List of dictionaries containing all issues

Return type:

List[Dict[str, Any]]

static safe_get(obj, *keys, default=None)

Safely get a value from a nested dictionary

Parameters:

• obj (object) – The object to extract the value from

• keys (str) – The keys to traverse the object

• default (object) – The default value to return if the key is not found

Rtype obj:

object

Returns:

The value at the specified keys or the default value

Return type:

object

ui_color: str = '#FFFFFF'

static write_to_s3(issues)

Write the list of issues to S3 as a parquet file. Create table in Athena with the latest data.

Parameters:

issues (List[Dict[str, Any]]) – List of issue dictionaries to write to S3

Returns:

None

Return type:

None

CQACollectGeometryOperator

This operator is used to invoke a CQA collect api call.

CQAGetLocationId

An operator to get the location id from CostQuest API from longitude and latitude

class operators.cost_quest.cqa_get_location_id.CQAGetLocationId(query, longitude_col_name, latitude_col_name, verbose=True, *args, **kwargs)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

list[int]

fetch_location_id(longitudes, latitudes, token, retries=3)

Return type:

int

get_api_token()

Return type:

str

p(message)

Print a message if verbose is True

Return type:

None

store_longitudes_latitudes()

Return type:

None

ui_color: str = '#A6E6A6'

CensusDownloaderOperator

An operator to download census shape files from the Census FTP server and store them in S3 and Athena.

class operators.census.census_downloader.CensusDownloaderOperator(vintage, state_abbr, geolevel, partition_cols, datalake_db='shoreline', use_ftp=True, *args, **kwargs)

DATA_PROD_SUFFIX = 'r_census'

DATA_PROD_VERSION = 'v1'

FTP_SERVER = 'ftp.census.gov'

HTTP_SERVER = 'https://www2.census.gov'

download_file(url, retry_count=3, retry_sleep=5)

download_using_http(url, retry_count=3, retry_sleep=5)

Download Census TIGER shapefile with retry logic and error handling.

Args:

url: URL of the file to download output_dir: Directory to save the file max_retries: Maximum number of retry attempts

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_file_details_from_url(url)

get_file_list(directory)

get_ftp_client()

static get_s3_prefix(vintage, geolevel)

Return type:

str

static get_table_name(vintage, geolevel)

Return type:

str

get_urls(shp_type, year, stat_abbr='')

Current tiger shape fils

process_zip_file(downloaded_file, current_file, update_table=False)

remove_temp_file(temp_file)

ui_color: str = '#A6E6A6'

BaseDBTOperator

Execute dbt run on the remote server.

class operators.dbt.dbt_base.BaseDBTOperator(select=None, exclude=None, project_directory=None, vars=None, target='dev', profile='athena', profile_directory=None, verbose=True, skip_deps=False, *args, **kwargs)

Initializes the BaseDBTOperator

Parameters:

• select (Optional[List[str]]) – List of dbt model selectors to include in the run. if this list is None, it will run all models. if it is not empty, it will run only the models that match the selectors. if it has .sql extension, it will be removed.

• exclude (Optional[List[str]]) – List of dbt model selectors to exclude from the run. if this list is None, it will run all models. if it is not empty, it will exclude only the models that match the selectors. if it has .sql extension, it will be removed.

• project_directory (str) – Path to the dbt project directory. it is usually the folder where dbt_project.yml is located. if this is not set, it will be assigned based on the dag folder. os.path.join(context[“dag”].folder, “dbt_project”)

• vars (Optional[dict]) – Dictionary of dbt variables to pass to the command. for example, if you want to pass a variable to the dbt model, you can use this parameter. it will be passed to the dbt command as –vars ‘{“key”: “value”}’. if this is not set, it will not pass any variables. we have a series of default variables which you can use, like vars={“target_grapi_schema”: “arkansas”, “target_grapi_database”: “postgres_grapi_dev”}. For the complete list of variables, see the dbt project documentation on confluence or see airflow/dbt/README.md

• target (Literal["dev", "prod", "stage"]) – name of target (database connection details; this is default entry in outputs). https://docs.getdbt.com/docs/core/connect-data-platform/profiles.yml and https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles: target is for default target, outputs is for additional targets. This is the target environment for dbt, which can be “dev”, “prod”, or “stage”. It is used to determine which dbt profile to use. If you are in a development environment, you should use “dev” target. If you are in a production environment, you should use “prod” target. If you are in a staging environment, you should use “stage” target. This will be validated in the get_target method. in the airflow dev environment, only “dev” target is allowed. If you try to use “prod” or “stage” target, it will raise an AirflowSkipException.

• profile (Literal["athena", "game", "grapi", "default"]) – dbt profile name. This is the profile name that dbt will use to connect to the database. It can be one of the following: “athena”, “game”, “grapi”, or “default”. If you are not sure which profile to use, you can use “athena” profile. The profile determines how dbt connects to the database and which credentials to use. To see all the profiles available, you can check the airflow/dbt/shared/profiles_sample.yml file or the dbt project documentation on confluence.

• profile_directory (str) – Path to dbt profiles directory. You probably don’t want to change this unless you have a custom profiles.yml file. By default, it will be set to ~/.dbt/profiles.yml on the dbt vm.

• verbose (bool) – Enables verbose logging and adds –debug and –log-path flags.

• args (Any) – Additional arguments for BaseOperator.

• kwargs (Any) – Additional keyword arguments for BaseOperator.

Default select:

None

Default exclude:

None

Default project_directory:

None

Default vars:

None

Raises:

AirflowSkipException – If a non-dev target is used in a dev environment.

Default target:

dev

Default profile:

athena

Default profile_directory:

None

Default verbose:

True

NOTE: super.execute() should be called in the execute method of the subclass. it will set the job_id and target_path and self.project_director attributes.

apply_default_vars()

Applies default variables to the operator.

This method is called to set default variables for the operator. It can be overridden in subclasses to provide custom default variables.

Parameters:

context (Context) – Airflow task context.

Return type:

None

assign_project_folder(context)

Assigns the project folder based on the DAG folder if not set.

Parameters:

context (Context) – Airflow task context containing DAG information.

Return type:

None

build_command(dbt_base_command)

Builds the dbt command to be executed on the server.

Parameters:

dbt_base_command (str) – The base dbt command (e.g., “dbt run”, “dbt test”).

Returns:

The complete dbt command with arguments.

Return type:

List[str]

custom_operator_name = 'BaseDBTOperator'

exclude: Optional[List[str]]

List of dbt model selectors to exclude from the run. If this list is None, it will run all models. If it is not empty, it will exclude only the models that match the selectors.

execute(context)

Executes the dbt operator logic.

Performs setup, runs dbt deps and dbt debug, and logs relevant information. NOTE: this method should be called in the execute method of the subclass. it will set the job_id and target_path and self.project_director attributes.

Parameters:

context (Context) – Airflow task context.

Raises:

AirflowSkipException – If target or project_directory is not set.

get_job_id(context)

Retrieves the job ID from the context or generates a new one.

Includes the task_id in the seed so that parallel tasks within the same DAG run get unique target paths and don’t collide on artifacts.

Parameters:

context (Context) – Airflow task context.

Returns:

The job ID.

Return type:

str

get_target()

Validates and returns the dbt target environment.

Ensures non-dev targets are blocked in development environments.

Parameters:

target (str) – The dbt target environment.

Returns:

The validated target.

Return type:

str

Raises:

AirflowSkipException – If a non-dev target is used in a dev environment.

get_target_path()

Generates a unique temporary path for dbt run output.

Returns:

Path in the format /tmp/dbt-target/<job_id>.

Return type:

str

job_id: str

Unique identifier for the dbt job run, used to create a unique target path.

notify_slack(message)

Return type:

None

on_kill()

Override this method to clean up subprocesses when a task instance gets killed.

Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up, or it will leave ghost processes behind.

Return type:

None

profile: str

The dbt profile name that dbt will use to connect to the database. It can be one of the following: “athena”, “game”, “grapi”

profile_directory: str

Path to dbt profiles directory. You probably don’t want to change this unless you have a custom profiles.yml file.

project_directory: str

Path to the dbt project directory. It is usually the folder where dbt_project.yml is located.

run_dbt_debug()

Runs dbt debug command to check if the connection is working.

Return type:

None

run_dbt_deps()

Runs dbt deps command to install the dependencies.

Return type:

None

run_dbt_refresh_sources()

Runs the script that gets all tables and write the sources.yml file.

Return type:

None

run_ssh_command(command)

ssh command to connect to the server

Return type:

str

select: Optional[List[str]]

List of dbt model selectors to include in the run. If this list is None, it will run all models. If it is not empty, it will run only the models that match the selectors.

property subprocess_hook

Returns hook for running the bash command.

target: str

The target environment for dbt, which can be “dev”, “prod”, or “stage”. It is used to determine which dbt profile to use.

template_fields: Sequence[str] = ('project_directory', 'target', 'profile', 'profile_directory', 'vars', 'select', 'exclude')

ui_color: str = '#9933FF'

ui_fgcolor: str = '#FFFFFF'

vars: Optional[dict]

Dictionary of dbt variables to pass to the command. For example, if you want to pass a variable to the dbt model, you can use this parameter.

verbose: bool

Enables verbose logging and adds –debug and –log-path flags.

DBTRunOperator

Execute dbt run on the remote server.

class operators.dbt.dbt_run.DBTRunOperator(dry_run=False, full_refresh=False, skip_deps=False, **kwargs)

Execute dbt run on the remote server.

custom_operator_name = 'DBTRunOperator'

execute(context)

Executes the dbt run command on the remote server.

Runs the parent class’s execute method, builds and runs the dbt run command with specified models and variables, logs compiled SQL on failure, and cleans up the target directory.

Parameters:

context (Context) – Airflow task context.

Raises:

Exception – If the dbt run command fails, after logging compiled SQL.

Returns:

Dictionary containing job details including job ID, target path, command run, and models executed.

template_fields: Sequence[str] = ('project_directory', 'target', 'profile', 'profile_directory', 'vars', 'select', 'exclude', 'dbt_dry_run', 'full_refresh')

ui_color: str = '#9933FF'

ui_fgcolor: str = '#FFFFFF'

DBTTestOperator

Execute dbt test on the remote server.

class operators.dbt.dbt_test.DBTTestOperator(**kwargs)

Execute dbt test on the remote server.

Parameters:

• models – list of model names (without .sql) to test; defaults to all

• vars – dict of vars to pass into dbt

custom_operator_name = 'DBTTestOperator'

execute(context)

Executes the dbt operator logic.

Performs setup, runs dbt deps and dbt debug, and logs relevant information. NOTE: this method should be called in the execute method of the subclass. it will set the job_id and target_path and self.project_director attributes.

Parameters:

context (Context) – Airflow task context.

Raises:

AirflowSkipException – If target or project_directory is not set.

ui_color: str = '#DB0058'

ui_fgcolor: str = '#FFFFFF'

DBTDocsOperator

Execute dbt docs generate on the remote server.

class operators.dbt.dbt_docs.DBTDocsOperator(select=None, exclude=None, bucket='463002504035-datahub', s3_prefix='{env}/data_docs/dag={dag_id}/docs/{version}', *args, **kwargs)

Airflow operator to generate dbt docs, package them, and upload to S3.

custom_operator_name = 'DBT Docs Operator'

execute(context)

Execute dbt docs generate and upload to S3.

Return type:

None

red_datahub_recipe(context)

Return datahub recipe for the operator.

Return type:

Dict[str, Any]

replace_dbt_paths(obj, dbt_project_root)

ui_color: str = '#FFB433'

ui_fgcolor: str = '#000000'

BedrockInvokeOperator

This operator is used to invoke a Bedrock model.

class operators.bedrock_agent.BedrockInvokeOperator(s3_prefix, dataset_name, input_dataset, mode='overwrite', *args, **kwargs)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_client()

static get_table_name(dataset_name)

invoke_agent(input_text, client, sessionId)

ui_color: str = '#A6E6A6'

BedrockPDFExtractorOperator

This operator is used to extract text from a PDF using a Bedrock model.

class operators.bedrock_pdf_extractor.BedrockPDFExtractorOperator(model_id, output_s3_prefix, output_athena_dataset_name, output_schema, input_files_s3_path, input_files_s3_bucket_name, mode='overwrite', inference_config={'maxTokens': 4096, 'temperature': 0.7, 'topP': 0.9}, *args, **kwargs)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_llm()

invoke_llm(input, llm, sessionId)

ui_color: str = '#A6E6A6'

BedrockAgentOperator

This operator extends BedrockInvokeOperator to invoke a Bedrock agent.

class operators.bedrock_agent.BedrockAgentOperator(agent_id, alias_id, s3_prefix, dataset_name, input_dataset=None, *args, **kwargs)

get_client()

invoke_agent(input, client, sessionId)

ui_color: str = '#A6E6A6'

BedrockFileSummaryOperator

This operator is used to summarize files using a Bedrock model.

class operators.bedrock_agent.BedrockFileSummaryOperator(model_id, file_s3_path, s3_bucket_name, user_prompt='\nsummarize the following document. Each summary should be in this format:  Title, Summery, Key points, Published date, Author, Source (link). Only return the results WITHOUT saying anything else ".\n', inference_config={'maxTokens': 4096, 'temperature': 0.7, 'topP': 0.9}, *args, **kwargs)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_client()

invoke_agent(conversation, client)

ui_color: str = '#A6E6A6'

BedrockModelOperator

This operator extends BedrockInvokeOperator to invoke a Bedrock model directly.

class operators.bedrock_agent.BedrockModelOperator(model_id, s3_prefix, dataset_name, input_dataset=None, inference_config={'maxTokens': 4096, 'temperature': 0.7, 'topP': 0.9}, *args, **kwargs)

get_client()

invoke_agent(input, client, sessionId)

ui_color: str = '#A6E6A6'

BedrockAutomationBaseOperator

Base operator for Bedrock automation tasks.

class operators.bedrock_automation_base_operator.BedrockAutomationBaseOperator(input_pdf_key, output_uri, blueprint, blueprint_version, blueprint_stage, max_wait_time=120, check_interval=10, **kwargs)

check_job_status(job_id)

create_bedrock_client()

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_data_automation_profile_arn()

invoke_data_automation_async(payload, region='us-east-1')

Call AWS Bedrock InvokeDataAutomationAsync API using the correct headers. Remove this after boto3 is updated

post_process(context, process_model_output, response)

process_model_output(response)

submit_automation_job()

uri_to_json(uri)

wait_for_job_completion(job_id, max_wait_time=120, check_interval=10)

BedrockDataComparisonOperator

This operator is used to compare data using a Bedrock model.

class operators.bedrock_data_comp.BedrockDataComparisonOperator(source_path, target_path, source_sql, target_sql, key_columns=[], column_mappings=None, semantic_equivalents=None, model_id='us.anthropic.claude-sonnet-4-5-20250929-v1:0', user_prompt='', gdrive_folder_id=None, slack_webhook_conn_id=None, slack_channel='#amazon-q', postgres_conn_id='postgres_default', comparison_threshold=0.85, sample_size=1000, inference_config=None, git_repo_path='/home/ubuntu/airflow/dags/bedrock/data_comparison_agent/SQL/', mode='overwrite', *args, **kwargs)

Enhanced Bedrock operator for intelligent data comparison Extends existing Bedrock operators with specific comparison capabilities

analyze_content_differences(source_df, target_df, schema_analysis)

Use Bedrock to analyze actual content differences including distinct values

Return type:

Dict

analyze_df(sample_df, path, sql, data_origin)

Return type:

dict

commit_sql_to_git(sql)

Commit generated SQL to git repository in a new branch

Return type:

None

execute(context)

Main execution method

generate_comparison_sql(schema_analysis)

Generate SQL for data comparison based on schema analysis

Return type:

str

generate_markdown_report(results)

Generate comprehensive markdown report

Return type:

str

get_llm_client()

Get ChatBedrockConverse client for structured output

invoke_bedrock_for_schema_analysis(source_schema, target_schema)

Use Bedrock to analyze schema differences with structured output

Return type:

Dict

load_sample_data(path, sql)

Load data from S3 or PostgreSQL

run_query(data_origin, sql)

send_slack_notification(results, report_url)

Send Slack notification with feedback request

Return type:

None

template_fields: Sequence[str] = ['source_path', 'target_path', 'key_columns', 'column_mappings', 'semantic_equivalents']

ui_color: str = '#7FB3D5'

upload_results_to_gdrive(results_df, report_content)

Upload results DataFrame to Google Drive using GDriveUtil

Return type:

Optional[str]

BedrockDataCompAnalyzer

AI analysis helper module for data comparison with structured output types.

AI analysis helper for Data Comparison Agent. Provides structured output types and analysis functions using Bedrock.

class operators.bedrock_data_comp_analyzer.ComparisonSummaryResult

Final comparison summary from AI analysis.

ai_determined_keys: list[str]

ai_determined_keys_confidence: float

ai_mapping_rationale: str

ai_semantic_mappings: dict[str, str]

key_findings: list[str]

recommendations: list[str]

severity: str

summary: str

class operators.bedrock_data_comp_analyzer.KeySuggestionResult

Result from AI key suggestion.

confidence: float

reasoning: str

suggested_keys: list[str]

class operators.bedrock_data_comp_analyzer.SchemaMatchResult

Result from schema matching analysis.

exact_matches: list[str]

semantic_matches: dict[str, str]

source_only: list[str]

target_only: list[str]

operators.bedrock_data_comp_analyzer.analyze_schema_matches(source_columns, target_columns, model_id=None)

Use AI to detect semantic column mappings.

Return type:

SchemaMatchResult

Args:

source_columns: List of source column names target_columns: List of target column names model_id: Optional model override

Returns:

SchemaMatchResult with exact and semantic matches

operators.bedrock_data_comp_analyzer.generate_comparison_summary(schema_diff, row_counts, duplicates, row_level_diff, custom_prompt=None, key_suggestion=None, schema_matches=None, model_id=None)

Generate final AI summary of comparison results.

Return type:

ComparisonSummaryResult

Args:

schema_diff: Schema comparison results row_counts: Row count comparison duplicates: Duplicate detection results row_level_diff: Row-level differences custom_prompt: Optional additional analysis instructions key_suggestion: AI key suggestion if used schema_matches: AI schema matches if detected model_id: Optional model override

Returns:

ComparisonSummaryResult with findings and recommendations

operators.bedrock_data_comp_analyzer.get_llm_client(model_id=None, temperature=0.1)

Get ChatBedrockConverse client for structured output.

operators.bedrock_data_comp_analyzer.suggest_primary_keys(source_columns, target_columns, sample_data, model_id=None)

Use AI to suggest primary key columns.

Return type:

KeySuggestionResult

Args:

source_columns: List of source column names target_columns: List of target column names sample_data: Sample data with distinct value counts model_id: Optional model override

Returns:

KeySuggestionResult with suggested keys and confidence

WebsiteScraperOperator

This operator is used to scrape websites and update a knowledge base.

class operators.bedrock_scraper.website_scraper_operator.WebsiteScraperOperator(sitemap_url, s3_prefix, s3_bucket_name, data_source_id, sitemap_class_type='GeneralSiteMapReader', knowledge_base_id=None, kendra_index_id=None, export_format='pdf', *args, **kwargs)

Airflow operator for scraping websites using sitemaps and updating knowledge bases.

This operator performs the following tasks: 1. Reads and parses a sitemap from a given URL 2. Converts web pages to PDF or HTML format 3. Stores the converted files in S3 4. Updates the specified knowledge base with the new content

Attributes:

ui_color (str): Color code for the operator in Airflow UI sitemap_url (str): URL of the sitemap to process s3_prefix (str): Prefix for S3 storage path s3_bucket_name (str): Name of the S3 bucket for storage data_source_id (str): Identifier for the data source sitemap_class_type (str): Type of sitemap reader to use (“GeneralSiteMapReader” or “GovtrackReader”) knowledge_base_id (str): ID of the knowledge base to update kendra_index_id (str): ID of the Kendra index export_format (str): Format to export pages as (“pdf” or “html”)

execute(context)

Handles the website scraping logic:

• Initializes the appropriate sitemap reader

• Parses the sitemap to extract URLs

• Processes each URL by converting it to PDF or HTML and saving to S3

• Handles checking for existing files to avoid duplicate processing

logger = <Logger operators.bedrock_scraper.website_scraper_operator (INFO)>

ui_color: str = '#A6E6A6'

CreateSchemaPostgresOperator

This operator is used to create a schema in Postgres.

class operators.create_schema_postgres.CreateSchemaPostgresOperator(postgres_conn_id, schema_name, if_not_exists=False, *args, **kwargs)

Create a schema in a Postgres database. :type postgres_conn_id: :param postgres_conn_id: the Postgres connection ID :type postgres_conn_id: str :type schema_name: :param schema_name: the name of the schema to create :type schema_name: str :type if_not_exists: :param if_not_exists: whether to add the IF NOT EXISTS clause to the SQL statement.

Default is False.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('postgres_conn_id', 'schema_name')

ui_color: str = '#e4e3ef'

CSVToPostgresOperator

This operator is used to load a CSV into Postgres.

class operators.csv_to_postgres.CSVToPostgresOperator(csv_path, postgres_conn_id, schema_name, table_name, file_encoding='utf-8', if_table_exists='fail', header='infer', skip_row_count=0, compression='infer', column_rename=None, field_names=None, chunk_size=0, sep=',', normalize=False, sheet=None, index_col=None, engine=None, stop_col=0, *args, **kwargs)

Operator to read a CSV file and save it in a Postgres database :type csv_path: :param csv_path: path to CSV file. For S3 use s3://<bucket>/<prefix>/<file> format. :type csv_path: str :type postgres_conn_id: :param postgres_conn_id: the Airflow connection ID to use to store data in Postgres. :type postgres_conn_id: str :type schema_name: :param schema_name: the Postgres schema name :type schema_name: str :type table_name: :param table_name: the Postgres table name within the schema :type table_name: str :type file_encoding: :param file_encoding: default to utf-8. Use ISO-8859-1 if CSV file contains chars outside of utf-8. :type file_encoding: str :type if_table_exists: :param if_table_exists: default to fail for data safety. Can also be replace or append. :type if_table_exists: str :type header: :param header: how to handle the first row of the CSV. Default infer. Can also be None. :type header: str :type skip_row_count: :param skip_row_count: number of rows directly after the header row (if any) to skip. Default 0. Useful when there are 2 or more header rows. :type skip_row_count: int :type compression: :param compression: the compression scheme of the CSV file. Default infer. Can also be gzip, bz2, zip, xz, None :type str :type column_rename: :param column_rename: map of {'oldname':'newname'}. Default None. :type dict :type stop_col: :param stop_col: column number to stop at, if theres a bunch of duplicate columns :type integer

check_table(dest_conn, dest_schema_table)

delete all contents of the given table before writing if necessary

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

handle_df_transform(df, conn, if_table_exists)

helper to write csv

ui_color: str = '#A6E6A6'

AthenaCTASOperator

This operator is used to create a table in Athena.

class operators.ctas.AthenaCTASOperator(prefix, query, target_table, partition_by=[], bucket='463002504035-shoreline', source_db='shoreline', target_db='shoreline', workgroup='airflow-dev', clean=True, data_source='AwsDataCatalog', bucketing_info=None, partition_key=None, partition_val=None, column_comments=None, table_description=None, *args, **kwargs)

An operator to execute CTAS (Create Table As Select) operations in Athena.

This operator wraps awswrangler’s create_ctas_table functionality to create new tables from SELECT queries in Athena. It handles cleaning up existing tables and managing partitioning/bucketing configurations.

Parameters:

• prefix (str) – The S3 prefix where the table data will be stored

• query (str) – The SELECT query to use for creating the new table

• target_table (str) – Name of the target table to create

• partition_by (list[str]) – Columns to partition the table by. Defaults to [].

• bucket (str) – S3 bucket name. Defaults to Config.DATA_LAKE_BUCKET.

• source_db (str) – Source database name. Defaults to Config.DATA_LAKE_DB.

• target_db (str) – Target database name. Defaults to Config.DATA_LAKE_DB.

• workgroup (str) – Athena workgroup name. Defaults to Config.DEFAULT_WORKGROUP.

• clean (bool) – Whether to remove existing table before creation. Defaults to True.

• data_source (str) – Data source name. Defaults to “AwsDataCatalog”.

• bucketing_info (tuple[list[str], int]) – Tuple of (columns, bucket_count) for table bucketing.

• partition_key (str) – Partition column name for partition-aware mode. When set (along with partition_val), the operator will drop the partition and INSERT INTO if the table exists, or CTAS on first run.

• partition_val (str) – Partition value for partition-aware mode (templatable). Used with partition_key for drop_athena_partition and row-count logging.

• column_comments (Dict[str, str]) – Optional dict mapping column names to their descriptions. If provided, updates Glue table metadata after CTAS completes.

• table_description (str) – Optional description for the table. If provided, updates Glue table metadata after CTAS completes.

execute(context)

Execute the CTAS operation.

When partition_key and partition_val are set, operates in partition-aware mode: drops only the target partition (if table exists) and INSERT INTO, or CTAS on first run. This allows multiple states to coexist.

Return type:

None

Args:

context: Airflow context dict

Raises:

AirflowException: If there’s an error during execution

template_fields: Sequence[str] = ('prefix', 'query', 'target_table', 'table_description', 'partition_val')

ui_color: str = '#89CFF0'

PostgresCTASOperator

This operator is used to create a table in Postgres using CREATE TABLE AS SELECT.

class operators.ctas.PostgresCTASOperator(query, target_table, target_db, target_schema=None, clean=True, *args, **kwargs)

An operator to execute CTAS (Create Table As Select) operations in Postgres.

This operator wraps the Postgres CREATE TABLE AS SELECT statement to create new tables from SELECT queries in Postgres.

ATTENTION: This operator is DESTRUCTIVE and will remove the existing table before creating a new one if clean=True. <3

Parameters:

• query (str) – The SELECT query to use for creating the new table

• target_table (str) – Name of the target table to create

• target_db (str) – Name of the target database. Defaults to Config.DEFAULT_DB_POSTGRES.

• target_schema (str) – Target schema name. Defaults to “data”.

• clean (bool) – Whether to remove existing table before creation. Defaults to True.

execute(context)

Execute the CTAS operation.

Return type:

None

Args:

context: Airflow context dict

Raises:

AirflowException: If there’s an error during execution

template_fields: Sequence[str] = ('query',)

ui_color: str = '#BEEFED'

RedshiftCTASOperator

This operator is used to create a table in Redshift using CREATE TABLE AS SELECT with transaction support and Redshift-specific distribution/sort key options.

class operators.redshift.redshift_ctas.RedshiftCTASOperator(query, target_table, redshift_conn_id, database=None, target_schema=None, diststyle=None, distkey=None, sortkey=None, clean=True, column_comments=None, table_description=None, grant_select_to=None, *args, **kwargs)

Execute CTAS (Create Table As Select) operations in Redshift with transaction support.

Wraps DROP TABLE + CREATE TABLE AS in a single transaction for atomic table replacement. Supports Redshift-specific distribution and sort key options.

Note: The query parameter must be a single SELECT statement without a trailing semicolon. If clean=False and the table exists, the operation will fail.

Parameters:

• query (str) – The SELECT query to use for creating the new table (no trailing semicolon).

• target_table (str) – Name of the target table to create.

• target_schema (Optional[str]) – Target schema name. Can also be resolved from params for dynamic mapping.

• redshift_conn_id (str) – Airflow connection ID for Redshift.

• database (Optional[str]) – Redshift database to connect to. If None, uses the connection default.

• diststyle (Optional[str]) – Distribution style (EVEN, KEY, ALL, AUTO). If distkey is provided without diststyle, KEY is used automatically.

• distkey (Optional[str]) – Column name for DISTKEY. Only valid with diststyle KEY or unset.

• sortkey (Optional[List[str]]) – List of column names for compound SORTKEY.

• clean (bool) – Whether to DROP existing table before creation. Defaults to True.

• column_comments (Optional[Dict[str, str]]) – Optional dict mapping column names to their descriptions. If provided, runs COMMENT ON COLUMN statements after CTAS completes.

• table_description (Optional[str]) – Optional description for the table. If provided, runs COMMENT ON TABLE statement after CTAS completes.

• grant_select_to (Optional[List[str]]) –

  Optional list of Redshift users or groups to GRANT SELECT on the newly created table. Runs after CTAS completes so that DROP+CREATE cycles don’t revoke access from BI service accounts.

  Entries are plain usernames by default. To grant to a Redshift GROUP, prefix the entry with GROUP: (case-insensitive):

  ["mn_datamart", "GROUP:readydata"]
  

  This generates:

  GRANT SELECT ON TABLE "bi"."table" TO "mn_datamart";
  GRANT SELECT ON TABLE "bi"."table" TO GROUP "readydata";
  

execute(context)

Execute the CTAS operation within a transaction.

Return type:

dict

Returns:

dict with keys: table, schema, created (pushed to XCom)

template_fields: Sequence[str] = ('query', 'target_table', 'target_schema', 'redshift_conn_id', 'database', 'diststyle', 'distkey', 'sortkey', 'clean', 'grant_select_to')

template_fields_renderers: dict[str, str] = {'query': 'sql'}

ui_color: str = '#BADA55'

AthenaToS3Operator

This operator is used to export Athena query results to S3.

class operators.athena_to_s3.AthenaToS3Operator(sql, source_db, s3_bucket, s3_prefix, s3_filename, file_format, compression='None', delimiter='None', data_source='AwsDataCatalog', *args, **kwargs)

An operator that will execute a query against Athena and write the results to S3.

The operator wraps awswrangler’s athena.unload functionality to execute a query and then write the results to S3. It handles the configuration of the S3 bucket, prefix, and other parameters.

Parameters: :type source_db: str :param source_db: The source database to run the query against. :type source_db: str

Parameters:

• sql (str) – The SQL query to execute in Athena.

• s3_bucket (str) – The S3 bucket where the results will be stored.

• s3_prefix (str) – The S3 prefix where the results will be stored.

• file_format (str) – The file format for the output files (e.g., ‘parquet’, ‘csv’).

• compression (str) – The compression type for the output files (e.g., ‘gzip’, ‘snappy’).

• delimiter (str) – The delimiter for CSV files.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('sql', 's3_bucket', 's3_prefix', 's3_filename', 'file_format', 'compression', 'delimiter', 'data_source', 'source_db')

ui_color: str = '#71D1E5'

DeleteAthenaTableOperator

This operator is used to delete an Athena table from the Glue catalog.

class operators.datalake.catalog.DeleteAthenaTableOperator(*, database='shoreline', table, delete_s3_objects=False, dry_run=False, succeed_if_missing=False, region_name='us-east-1', **kwargs)

Drop a Glue/Athena table and (optionally) delete its S3 data prefix.

Behavior

1. Validates that the table exists in the Glue Data Catalog

   • If not found and succeed_if_missing=False: raises an error.

   • If not found and succeed_if_missing=True: logs and exits successfully

2. If delete_s3_objects=True: discovers the S3 location (StorageDescriptor.Location), drops the table, then recursively deletes all objects in that prefix.

3. If delete_s3_objects=False: drops the table only.

Parameters

type database:

str

param database:

Glue database (Athena schema).

type database:

str

type table:

str

param table:

Glue/Athena table name.

type table:

str

type delete_s3_objects:

bool

param delete_s3_objects:

If True, also delete the S3 location recursively.

type delete_s3_objects:

bool

type dry_run:

bool

param dry_run:

If True, do not modify anything; log intended actions.

type dry_run:

bool

type region_name:

Optional[str]

param region_name:

Optional AWS region (boto3/awslambda resolves from environment/instance profile if not set).

type region_name:

Optional[str]

type succeed_if_missing:

bool

param succeed_if_missing:

If True, do not fail if the table is missing; just log a warning.

type succeed_if_missing:

bool

Template fields

• database

• table

• delete_s3_objects

• dry_run

• succeed_if_missing

Returns

dict

{

“database”: “…”,

“table”: “…”, “existed”: True, “deleted_catalog”: True, “s3_prefix”: “s3://bucket/prefix/” | None, “deleted_objects”: <int> | 0, “dry_run”: <bool>, “succeed_if_missing”: <bool>

}

Guide for Use (DAG Mapping)

• Single table cleanup:

DeleteAthenaTableOperator( task_id=”drop_users_daily”, database=”shoreline”, table=”users_daily”, delete_s3_objects=True, )

• Map across many tables (TaskFlow ):

  @task def list_to_drop():

  return [ {“database”: “shoreline”, “table”: “t1”, “delete_s3_objects”: True}, {“database”: “shoreline”, “table”: “t2”, “delete_s3_objects”: False},

  ]

  DeleteAthenaTableOperator.partial(task_id=”drop_batch”).expand_kwargs(list_to_drop())

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('database', 'table', 'delete_s3_objects', 'dry_run', 'succeed_if_missing')

ui_color: str = '#B00B00'

S3ToAthenaParquetOperator

This operator is used to create an Athena table from Parquet files in S3.

class operators.datalake.catalog.S3ToAthenaParquetOperator(*, database='shoreline', bucket='463002504035-shoreline', output_table, state_abbr, output_path, input_path, drop_table_if_exists=True, region_name='us-east-1', column_mapping=None, **kwargs)

S3ToAthenaParquetOperator

Ingest raw data from S3 (or an external HTTP/HTTPS source) into the Ready data lake as partitioned Parquet files and register the resulting Glue/Athena tables.

This operator is the BaseOperator equivalent of the historical ingest_raw_file TaskFlow DAG. It provides a controlled, extensible, and map-friendly ingestion mechanism built entirely on the shared Athena/S3 helper functions in this module.

1. S3 Cleanup (safe) - Computes the destination folder from the provided output_path. - Uses the shared, panic-protected _delete_s3_prefix() helper to remove

existing objects under that prefix.

• Prevents accidental bucket-root deletes and respects Config.PANIC_PREFIXES.

2. File Ingestion - Supports three input modes:

• S3 folder (prefix ending with “/”):

  • Lists all files with extensions (.csv, .xls, .xlsx, .parquet).

  • Ingests each file into its own Athena table with an auto-generated

  suffix derived from the file’s stem.

• Single S3 file:

  • Ingests into a single Athena table.

• External URL (HTTP/HTTPS):

  • Reads via pandas (CSV or Excel).

  • Writes Parquet to S3 with awswrangler.

  • Recreates a Glue/Athena table for the output data.

3. Table Creation & Glue Catalog Management - Creates tables in the configured Athena/Glue database (Config.DATA_LAKE_DB). - Automatically builds canonical table names:

r_{state_abbr}up_{output_table}

or (for folder ingestion):

r_{state_abbr}up_{output_table}_{file_stem}

• Optionally drops the base table before ingestion (drop_table_if_exists=True).

4. Row Count Verification (non-S3 URL ingestion) - When ingesting via HTTP/HTTPS, performs an Athena query to confirm the

number of imported rows, logging results for transparency.

• All S3 deletion uses the shared _delete_s3_prefix() helper, ensuring:

  • guard-rails against bucket-root or panic prefixes,

  • consistent logging and dry-run capability.

• Uses lazy imports via _import_libs() for awswrangler/boto3 to remain DAG-parse safe.

• Inherits Airflow template fields for all user-provided paths, enabling

.expand_kwargs() and dynamic ingestion.

S3ToAthenaParquetOperator(

task_id=”ingest_raw”, database=Config.DATA_LAKE_DB, bucket=Config.DATA_LAKE_BUCKET, output_table=”proposal_uploads”, state_abbr=”va”, output_path=”s3://{DEST_BUCKET}/raw/uploads/Virginia/unzipped”, input_path=”s3://463002504035-shoreline/uploads/VA/proposals/2025-09-11/”,

)

• Ad-hoc ingestion of CSV/XLSX/Parquet into the data lake.

• Bulk-folder ingestion where each file becomes a separate table.

• Automated ingestion pipelines where inputs vary by tenant, state, or date.

• Eliminating legacy TaskFlow DAG logic while preserving full functionality.

dict containing:

{

“database”: “…”, “bucket”: “…”, “output_path”: “s3://bucket/path”, “input_path”: “…”, “tables_created”: [

{“table”: “…”, “source”: “…”, “prefix”: “…”}, …

],

}

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('database', 'bucket', 'output_table', 'state_abbr', 'output_path', 'input_path')

ui_color: str = '#006BB3'

TableToPGOperator

This operator is used to load a table from athena into Postgres.

class operators.datalake_to_pg.TableToPGOperator(dest_db, dest_table, copy_table_sql, upsert_origin_table=None, on_conflict_sql=None, skip_if_empty=False, partitions=[], copy_partitions=[], schema='public', create_table_sql=None, post_transfer_sql=None, create_partition_template="CREATE TABLE {{schema}}.{{table_name}}_{{parname}}_stage PARTITION OF {{schema}}.{{stage_table_name}} FOR VALUES in ('{{parname}}')", task_suffix='', drop_backup_before_start=True, drop_backup_at_end=True, template='', inlets=[], map_name='', table_description=None, column_descriptions={}, data_product_name=None, *args, **kwargs)

add_datahub_metadata(context)

add_table_metadata()

copy_to_pg(partition='')

create_table_with_partitions()

custom_operator_name = 'TableToPGOperator'

drop_stage_table()

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Any

get_lineage()

get_output_table_urn()

read_sql_template(sql, params={})

Return type:

str

resolve_self_x_com_attributes(variable, context)

resolve_x_com_arg(value, context)

sanitize_params(context)

read params and converts them to xcom results if they are passed from a task

Parameters:

context (Context) – operator context

Returns:

returns params object

Return type:

_type_

target_stage_table()

returns stage table name with schema

Returns:

stage table name with schema

Return type:

str

template_ext: Sequence[str] = '.sql'

template_fields_renderers: dict[str, str] = {'copy_table_sql': 'sql', 'create_table_sql': 'sql', 'post_transfer_sql': 'sql'}

transfer_data_per_partition()

ui_color: str = '#FFA500'

ui_fgcolor: str = '#000000'

GCSToS3RenameOperator

Transfer files from Google Cloud Storage to Amazon S3 with the ability to rename files.

class operators.gcs_to_s3_rename.GCSToS3RenameOperator(file_mappings, gcp_project=None, replace=False, *args, **kwargs)

Transfer files from Google Cloud Storage to Amazon S3 with the ability to rename files.

This operator supports copying a single GCS file to multiple S3 destinations.

Parameters:

• file_mappings (List[Tuple[str, str]]) –

  List of tuples mapping GCS object paths to S3 destination keys e.g., [

  (“gs://my-bucket/path/to/file.txt”, “s3://dest-bucket1/new/path1.txt”), (“gs://my-bucket/path/to/file.txt”, “s3://dest-bucket2/new/path2.txt”), (“gs://other-bucket/data.csv”, “s3://dest-bucket3/data.csv”)

  ] Note: GCS paths should include the bucket name with “gs://” prefix

• gcp_project (str) – The GCP project ID

• replace (bool) – Whether to replace existing files in S3 If True, will upload all files replacing existing ones in the destination bucket. If False, will upload only files that don’t exist in the destination bucket.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

template_fields: Sequence[str] = ['file_mappings']

ui_color: str = '#E0FFFF'

GeodatabaseToGeoparquetOperator

This operator is used to convert a geodatabase to geoparquet.

GoogleDriveToS3Operator

This operator is used to transfer files from Google Drive to S3.

class operators.gdrive_to_s3.GoogleDriveToS3Operator(google_drive_id, s3_key, s3_bucket_name='463002504035-shoreline', remove_spaces_from_filename=False, overwrite_file=True, skip_spreadsheets=True, recursive=True, *args, **kwargs)

Operator to transfer files from Google Drive to an S3 bucket.

Parameters:

• google_drive_id (str) – The Google Drive folder ID to transfer.

• s3_key (str) – The S3 key (path) where the file will be stored.

• s3_bucket_name (str) – The S3 bucket name where the file will be stored.

• remove_spaces_from_filename (bool) – If True, spaces in filenames will be replaced with underscores.

• overwrite_file (bool) – If True, existing files in S3 will be overwritten.

• recursive (bool) – If True, recursively process subfolders when a folder ID is provided.

download_file(service, s3, downloader_class, google_drive_file_id=None, s3_key=None)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('google_drive_file_id', 's3_bucket_name', 's3_key', 'overwrite_file', 'recursive')

ui_color: str = '#A6E6A6'

GDriveBaseOperator

Base operator for Google Drive operations.

class operators.gdrive.gdrive_base.GDriveBaseOperator(*, google_drive_id=None, gcp_conn_id='gdrive_api', scopes=None, **kwargs)

Parent class for every Google-Drive-related operator.

property drive

googleapiclient drive-v3 service, cached for this task-run.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('google_drive_id', 'gcp_conn_id')

ui_color: str = '#A6E6A6'

GDriveListFilesOperator

This operator is used to list files in a Google Drive folder.

class operators.gdrive.gdrive_list.GDriveListFilesOperator(*, drive_link=None, google_drive_id=None, include_trashed=False, page_size=1000, **kwargs)

Recursively list files (not sub-folders) under a Drive folder.

Returns → List[dict] (id, name, mimeType)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

List[Dict[str, str]]

template_fields: Sequence[str] = ('google_drive_id', 'gcp_conn_id', 'drive_link')

ui_color: str = '#FACADE'

GDriveGetFileIdOperator

This operator is used to get a file ID from Google Drive by name.

class operators.gdrive.gdrive_list.GDriveGetFileIdOperator(*, file_url=None, file_id=None, raise_if_missing=True, **kwargs)

Accept a public/share URL or a raw file-ID, verify the object exists, and return a small metadata dict via XCom.

Example

get_file_id = GDriveGetFileIdOperator(

task_id=”get_file_id”, file_url=”https://drive.google.com/file/d/123456789asdfce/view?usp=sharing”, # gcp_conn_id defaults to “google_cloud_default”

)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ('google_drive_id', 'gcp_conn_id', 'file_url', 'file_id')

ui_color: str = '#F1BBED'

GetAirflowDagsOperator

This operator is used to get the Airflow DAGs. .. autoclass:: operators.get_airflow_dags.GetAirflowDagsOperator

members:

GithubToS3Operator

This operator is used to transfer code and files from Github to S3.

class operators.github_to_s3.GithubToS3Operator(repo_url, s3_key, s3_bucket_name='463002504035-shoreline', *args, **kwargs)

this is a sin

execute(context)

Clones a git repository to a temporary directory, uploads it to S3, then cleans up the temporary files.

ui_color: str = '#4287f5'

BatchGooglePlaceFindPlaceOperator

This operator is used to geocoding places using Google Places API in batches

class operators.google_geocode_batch.BatchGooglePlaceFindPlaceOperator(input_df, api_key, language='en', fields='formatted_address,name,geometry', *args, **kwargs)

Airflow Operator to perform batch geocoding using the Google Place API’s findplacefromtext endpoint.

This operator expects a pandas DataFrame with the columns “place_name” and “address”. For each row, it will query the API by combining these two columns and append the following columns to the DataFrame:

• formatted_address

• latitude

• longitude

Parameters:

• input_df (pandas.DataFrame) – The input pandas DataFrame with columns “place_name” and “address”.

• api_key (str) – Your Google API key.

• language (str) – (optional) The language for the request. Default is ‘en’.

• fields (str) – (optional) A comma-separated list of fields to return (e.g., “formatted_address,name,geometry”).

Returns:

The input DataFrame with additional columns for formatted_address, latitude, and longitude.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

GooglePlaceFindPlaceOperator

This operator is used to geocoding a single place using Google Places API.

class operators.google_geocode.GooglePlaceFindPlaceOperator(input_text=None, api_key=None, language='en', fields='formatted_address,name,geometry', *args, **kwargs)

Airflow Operator to search for a place using the Google Place API’s findplacefromtext endpoint.

Parameters:

• input_text (str) – The text query for the place search.

• api_key (str) – Your Google API key.

• language (str) – (optional) The language for the request. Default is ‘en’.

• fields (str) – (optional) A comma-separated list of fields to return (e.g., “formatted_address,name,geometry”).

Returns:

A dictionary with the input text and the candidate place.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

PlacesTextSearchOperator

This operator is used to search for places using Google Places Text Search API.

class operators.google_place.text_search.PlacesTextSearchOperator(*, api_key, text_query, max_pages=5, page_pause_seconds=3.0, timeout=30, session=None, **kwargs)

Uses Places API v1 Text Search (new) to return a list of place ids.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

list[str]

template_fields: Sequence[str] = ('text_query', 'max_pages', 'page_pause_seconds')

PlaceDetailsBatchOperator

This operator is used to get place details in batches using Google Places API.

class operators.google_place.place_detail.PlaceDetailsBatchOperator(*, api_key, place_ids=None, batch_size=20, per_call_sleep=0.0, timeout=30, session=None, **kwargs)

Given a list of place ids, fetch selected details for each: name, lat, lon, address, url, phone.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

list[dict]

template_fields: Sequence[str] = ('place_ids', 'batch_size')

HttpToS3Operator

This operator is used to transfer files from an HTTP endpoint to S3.

class operators.http_to_s3.HttpToS3Operator(url, bucket, key, if_exists, headers=None, verify=None, *args, **kwargs)

Operator to upload a file to s3 from an http link :type url: :param url: path to HTTP file. :type url: str :type bucket: :param bucket: name of the s3 bucket :type bucket: str :type key: :param key: path to the s3 file within bucket :type key: str :type if_exists: :param if_exists: action to take if file already in s3. ‘noop’, ‘overwrite’ :type if_exists: str

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

ui_color: str = '#0380fc'

LakeMsckOperator

This operator is used to run an MSCK command on a data lake table.

class operators.lake_msck.LakeMsckOperator(table_name, database='shoreline', workgroup='airflow-dev', *args, **kwargs)

Run MSCK REPAIR TABLE on a table in the data lake. Running twice to ensure all partitions are picked up.

Parameters:

• table_name (str) – the name of the table to repair

• database (str) – the database the table is in, defaults to Config.DATA_LAKE_DB

:type database

Raises:

AirflowException – _description_

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

template_fields: Sequence[str] = ['table_name', 'database', 'workgroup']

ui_color: str = '#FF69B4'

LakePitTableOperator

This operator is used to create a PIT table in the data lake.

class operators.lake_pit_table.LakePitTableOperator(pit_table_name, source_table_name, version, database='shoreline', version_column_name='version_date', *args, **kwargs)

Create/insert data into a point in time table in the data lake.

Parameters:

• pit_table_name (str) – the name of the point in table that will be created/inserted into

• source_table_name (str) – the name of the base table that will be read from

• version (str) – the version of the data, gets inserted into the version column in the pit table

• database (str) – the database the table is in, defaults to Config.DATA_LAKE_DB

• version_column_name (str) – the name of the version column in the pit table, defaults to “version_date”

Raises:

AirflowException – _description_

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

template_fields: Sequence[str] = ('pit_table_name', 'source_table_name', 'version')

ui_color: str = '#9842f5'

PostgresToDatalake

This operator is used to load a table from Postgres into the data lake.

class operators.pg_to_datalake.PostgresToDatalake(source_sql, table_name, mode='overwrite', partition_cols=[], dtype={}, workgroup='airflow-dev', athena_target_db='shoreline', postgres_source_db='postgres_bossplatform_grapi_dev', chunksize=64000, cleanup_templog=True, *args, **kwargs)

A general operator to transfer data from pg to athena.

This is a revision from PostgresToDatalake which use pandas object, is sensitive to the memory limits, so cant handle very big dataset. This approach use df=read_sql(chunksize) and ingest each chunk to s3 with wr.s3.to_parquet(dataset=True)

CAUTION #1: Postgis geom col is not supported yet; so geom values might not be properly transferred!

CAUTION #2: ALL QUERIES that save to same athena table will be chunked (offset) using same chunk indexes (stored in airflow /tmp).

So e.g. SELECT * from cb20 limit 100 offset 100 <> SELECT * from cb20 limit 100 offset 200 will be both counted as chunk=0 and the subsequent transfer queries will be skipped!

Args:

source_sql (str): the SELECT query (CTAS) to run against pg. Here, you should cast, rename and manipulate incoming data into desired forms in Athena table_name (str): the name of the table to generate in athena. This operator will add pg_ to the table name automatically mode (Literal['append', 'overwrite', 'overwrite_partitions'], optional): This is a parameter to pass aws wrangler to replace or append the existing table. Defaults to ‘overwrite’. partition_cols (list[str], optional): The partitioning columns. make sure to set this for the performance. Defaults to None. dtype (dict, optional): A dict to force AWS Wrangler to use specified data types; also cast completely NULL cols to string, not specified workgroup (str, optional): athena workgroup. Defaults to Config.DEFAULT_WORKGROUP. athena_targt_db (str, optional): athena target db. Defaults to Config.DATA_LAKE_DB. postgres_source_db (str, optional): pg source db. Defaults to Config.BOSS_PLATFORM_GRAPI_DB. cleanup_templog (bool, optional): whether to remove all temp log (checkpoints) afterward!

execute(context)

main call to transfer data from pg to athena

Return type:

None

Returns:

None

template_fields: Sequence[str] = ('source_sql', 'table_name', 'mode')

ui_color: str = '#A6E6A6'

CreateRadIdNativeOperator

This operator is used to create a RAD ID for a list of addresses. This is a athena native operator, meaning its functions are selected from Athena functions.

class operators.rad_id_native.CreateRadIdNativeOperator(table_name=None, id_col_old=None, id_col_new='rad_id', city_col='city', state_col='state', address_col='address_primary', place_name_col='entity_name', location_id_col=None, *args, **kwargs)

Create unique RAD_ID for each record in a dataset. The dataset should contain: 1. place name (nullable) 2. address 3. city 4. location_id (nullable) 5. state 6. zip code 7. latitude (nullable) 8. longitude (nullable)

Parameters:

• table_name (Optional[str]) – The name of the table.

• id_col_old (Optional[str]) – The name of the original id column.

• id_col_new (str) – The name of the new rad_id column.

• city_col (str) – The name of the city column.

• state_col (str) – The name of the state column.

• address_col (str) – The name of the address column.

• place_name_col (str) – The name of the place name column.

• location_id_col (Optional[str]) – The name of the location id column. If None, the operator will not consider location_id in the duplicate detection.

Returns:

A dictionary of full records with new rad_id, and is_dup column

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('table_name', 'id_col_old', 'id_col_new', 'location_id_col', 'city_col', 'state_col', 'address_col', 'place_name_col')

BaseTileOperator

This operator is used as a base class for other tile operators.

class operators.tile.base_tile.BaseTileOperator(src_tiles=None, src_env='main', dest_s3=None, is_dry_run=True, verify_tile_exists=True, *args, **kwargs)

Base operator for tile operations that provides common utilities for working with tilesets on servers and S3.

add_to_datahub(context, tileset_name)

Add the tileset to datahub with tags, and ownership information

Return type:

None

check_file_exists_on_server(server, file_path)

Check if a file exists on a server.

Return type:

bool

create_directory_on_server(server, directory)

Create a directory on a server if it doesn’t exist.

execute_ssh_command(server, command, description, check_output=False)

Execute an SSH command on a remote server.

get_tileset_name(path=None)

Extract the tileset name from the path or dest_s3.

Parameters:

path (Optional[str]) – Optional path to extract name from, defaults to first src_tile/dest_s3

Return type:

str

Returns:

Tileset name without extension

notify_slack(message)

Send a notification to Slack

Return type:

None

on_kill()

Handle task kill event when Airflow kills the task instance

Return type:

None

remove_from_datahub(tileset_name)

Remove the tileset from datahub.

Return type:

None

restart_tileserver(server)

Stop, update inventory config (update_config.py), and restart the tileserver on a server.

run_command(command, description, check_output=False)

Run a command with handling for dry run mode.

s3_check_exists(s3_path)

Check if a file exists in S3.

Return type:

bool

s3_delete(s3_path)

Delete a file from S3.

s3_download(s3_path, local_path)

Download a file from S3.

s3_upload(local_path, s3_path)

Upload a file to S3.

scp_from_server(server, remote_path, local_path)

Copy a file from a server using SCP.

scp_to_server(local_path, server, remote_path)

Copy a file to a server using SCP.

tileserver_alias(env=None)

Get the appropriate tileserver alias based on the environment.

In production: - “prod”/”main” -> Config.TILESERVER_ALIAS (kanto) - “staging” -> Config.TILESERVER_ALIAS_STAGE (hoenn) - “dev” -> Config.TILESERVER_ALIAS_DEV (johto)

In development: - All environments default to Config.TILESERVER_ALIAS (johto)

Parameters:

env (Optional[str]) – Optional environment override (“prod”, “staging”, “dev”, “main”) If not provided, uses self.src_env

Return type:

str

Returns:

Server alias from Config

tileset_urn(tileset_name)

Get the URN for the tileset in datahub.

Return type:

Urn

verify_mbtiles_file(file_path, server_alias)

Verify that an mbtiles file exists on a server.

Return type:

bool

verify_s3_file(s3_path)

Verify that a file exists in S3.

Return type:

bool

ListTileOperator

This operator is used to list all available tiles in Tileserver TSGL or S3, by exact string or regex pattern.

class operators.tile.list_tile.ListTileOperator(input_select='titles', regex_pattern=None, search_s3=False, *args, **kwargs)

Operator for listing tiles based on input selection method (titles or regex). This operator can scan both the tileserver and S3 for available tiles.

Parameters:

• input_select (Literal['titles', 'regex']) – Method to select tiles - either “titles” or “regex”

• regex_pattern (Optional[str]) – Regex pattern to match tile names (used when input_select is “regex”)

• search_s3 (bool) – Whether to search in S3 instead of tileserver

custom_operator_name = 'ListTileOperator'

execute(context)

Execute the tile listing operation.

Parameters:

context (Context) – The Airflow task context

Return type:

List[str]

Returns:

A list of matched tile paths

render_template_fields(context, jinja_env=None)

Override to handle special template rendering cases

template_fields: Sequence[str] = ['src_tiles', 'src_env', 'input_select', 'regex_pattern', 'search_s3']

ui_color: str = '#5DADE2'

ui_fgcolor: str = '#FFFFFF'

CreateTileOperator

This operator is used to create a vector tile.

class operators.tile.create_tile.CreateTileOperator(source_sql, title, copy_main_to_stage=False, description='', tile_server_folder='', geometry_column='geometry', deploy_tiles=False, task_suffix='large', deploy_environment='main', column_mapping={}, workgroup='airflow-dev', tgt_db='shoreline', template='', label_columns=[], inlets=[], tippecanoe_cmd_template='tippecanoe -zg -Z0 --extend-zooms-if-still-dropping --drop-densest-as-needed --force -o {mbtiles_path} -l {title} -n {title_description} -N {description} {geojson_path}', map_name='', data_product_name=None, keep_tiles_on_mount_folder=False, *args, **kwargs)

An operator to generate tiles

Parameters:

• source_sql (str) – The sql query to run against database, if it ends with .sql we read the file relative to the template path

• title (str) – The title of the file,

• copy_main_to_stage (bool = False) – Copy the generated mbtiles to the staging tileserver. Only works on production

• description (str) – The description of the file,

• tile_server_folder (str) – Path of on the tileserver to store mbtiles. Disabled on purpose for future

• geometry_column (str = "geometry") – add your geometry column name here. It can be comma separated if you have two columns like lat, lon (with the same order).,

• deploy_tiles (bool = False) – Deploy tiles on the tileserver

• workgroup (str = Config.DEFAULT_WORKGROUP) – workgroup information to read sql query from it

• tgt_db (str = Config.DATA_LAKE_DB) – tgt_db information to read sql query from it

• template (str) – the path for the sql query template.

• task_suffix (Literal["large", "small", "xlarge"] = "small") – the scale of the ecs machine. Xlarge is only supported for the tippecaneo ecs and it will fall back to large for ecs_athena.

• deploy_environment (Literal["main", "staging"] = "main") – This will control the environment to deploy. on airflow dev main=dev and on prod main=prod and staging is always staging.

• tippecanoe_cmd_template (str = "tippecanoe -zg -Z0 --extend-zooms-if-still-dropping --drop-densest-as-needed --force -o {mbtiles_path} -l {title} -n {title_description} -N {description} {geojson_path}") – the command template to send to tippecaneo.Don’t change if you are not sure what are the params

• map_name (str) – the name of the map. if not set it will be set to {tile_envs}.{title}

• inlets (list[Urn]) – An array of inlets to this task. use Util.lineage_urn to create the urns they should be type of Urn

• label_columns (list[str]) – the columns to be used as labels in the map. it can be an array. mainly use this if your data are polygons and you want to show the name of the polygon on the map. NOTE: If you apply column mapping the new column name should be used here.

• data_product_name (str | None = None) – the name of the data product to add the tileset to. if not set it will not be added

• keep_tiles_on_mount_folder (bool = False) – if set to true, the mbtiles will not be removed from the mount folder. mainly used for debugging

Raises:

ex – if is_athena is set to false. pg is not yet supported

add_to_datahub(context)

Return type:

None

copy_mbtiles(environment_alias)

copy mbtile files to the tileserver

Parameters:

environment_alias (str) – the tile server’s ailas

Return type:

None

Returns:

None

Raises:

AirflowException – if the command fails

copy_to_stage()

Handles copying the mbtiles to the staging tileserver. only works on production

custom_operator_name = 'CreateTileOperator'

datahub_environment()

Return type:

str

execute(context)

main call to generate geojson -> tile

Parameters:

context (Context) – the context

Return type:

None

get_lineage()

Return type:

tuple[list[Urn], list[Urn]]

notify_slack(message)

Return type:

None

on_kill()

Override this method to clean up subprocesses when a task instance gets killed.

Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up, or it will leave ghost processes behind.

Return type:

None

remove_source_mbtiles_and_geojson()

removes files from efs-mount drive

Return type:

None

Returns:

None

Raises:

AirflowException – if the command fails

run_bash_command(command)

runs a command with subprocess hook

Parameters:

command (str) – the command

Raises:

AirflowException – if the command fails

Return type:

None

property subprocess_hook

Returns hook for running the bash command.

template_ext: Sequence[str] = ('.sql',)

template_fields: Sequence[str] = ('source_sql', 'title', 'task_suffix', 'deploy_tiles', 'description', 'deploy_environment')

template_fields_renderers: dict[str, str] = {'copy_main_to_stage': 'bool', 'deploy_tiles': 'bool', 'source_sql': 'sql'}

tileserver_alias()

Return type:

str

tileset_urn()

Return type:

Urn

ui_color: str = '#9933FF'

ui_fgcolor: str = '#FFFFFF'

RemoveTileOperator

This operator is used to remove a vector tile from Tileserver TSGL.

class operators.tile.remove_tile.RemoveTileOperator(src_tiles, src_env='main', is_dry_run=True, verify_tile_exists=True, *args, **kwargs)

Operator to remove tile(s) from the tileserver without archiving to S3.

Parameters:

• src_env (Literal["main", "staging"]) – source environment, defaults to “main”

• src_tiles (Union[str, List[str]]) – single path or list of paths to the tilesets (starting at home directory ~/)

• is_dry_run (bool, optional) – whether to perform a dry run, defaults to True

• verify_tile_exists (bool, optional) – whether to verify if the tile exists before performing the action, defaults to True

Raises:

AirflowException – if the removal fails

custom_operator_name = 'RemoveTileOperator'

execute(context)

Remove tile(s) from the tileserver without archiving to S3.

Return type:

None

template_fields: Sequence[str] = ['src_tiles', 'src_env', 'is_dry_run']

ui_color: str = '#ff9999'

ui_fgcolor: str = '#000000'

RemoveInS3Operator

This operator is used to remove a vector tile already archived in S3.

class operators.tile.remove_in_s3.RemoveInS3Operator(dest_s3, is_dry_run=True, verify_tile_exists=True, *args, **kwargs)

Operator to delete tile(s) from S3 without affecting the tileserver.

Parameters:

• dest_s3 (Union[str, List[str]]) – path(s) to the S3 files to delete

• is_dry_run (bool, optional) – whether to perform a dry run, defaults to True

• verify_tile_exists (bool, optional) – whether to verify if the tile exists in S3 before deleting, defaults to True

Raises:

AirflowException – if the deletion fails

custom_operator_name = 'RemoveInS3Operator'

execute(context)

Delete tile(s) from S3.

Return type:

None

template_fields: Sequence[str] = ['dest_s3', 'is_dry_run']

ui_color: str = '#FFA500'

ui_fgcolor: str = '#000000'

RecoverFromS3Operator

This operator is used to recover a vector tile from S3 to Tileserver TSGL.

class operators.tile.recover_from_s3.RecoverFromS3Operator(src_tiles, src_env='main', dest_s3=None, is_dry_run=True, verify_tile_exists=True, *args, **kwargs)

Operator to recover tile(s) from S3 to the tileserver (keeping the files in S3).

Parameters:

• src_env (Literal["main", "staging"]) – destination environment, defaults to “main”

• src_tiles (Union[str, List[str]]) – single path or list of paths where the tiles should be placed on the server

• dest_s3 (Union[str, List[str]]) – path(s) to the S3 source files

• is_dry_run (bool, optional) – whether to perform a dry run, defaults to True

• verify_tile_exists (bool, optional) – whether to verify if the tile exists in S3 before performing the action, defaults to True

Raises:

AirflowException – if the recovery fails

custom_operator_name = 'RecoverFromS3Operator'

execute(context)

Recover tile(s) from S3 to the tileserver.

Return type:

None

template_fields: Sequence[str] = ['dest_s3', 'src_tiles', 'src_env', 'is_dry_run']

ui_color: str = '#4CAF50'

ui_fgcolor: str = '#FFFFFF'

ArchiveToS3Operator

This operator is used to move (archive) a vector tile from Tileserver TSGL to S3.

class operators.tile.archive_to_s3.ArchiveToS3Operator(src_tiles, src_env='main', dest_s3=None, is_dry_run=True, verify_tile_exists=True, *args, **kwargs)

Operator to move tile(s) from the tileserver to S3 and remove them from the server.

Parameters:

• src_env (Literal["main", "staging"]) – source environment, defaults to “main”

• src_tiles (Union[str, List[str]]) – single path or list of paths to the tilesets (starting at home directory ~/)

• dest_s3 (Optional[str]) – Optional destination S3 path(s). If not provided, standardized paths will be used.

• is_dry_run (bool, optional) – whether to perform a dry run, defaults to True

• verify_tile_exists (bool, optional) – whether to verify if the tile exists before performing the action, defaults to True

Raises:

AirflowException – if the move operation fails

custom_operator_name = 'ArchiveToS3Operator'

execute(context)

Move tile(s) from the tileserver to S3 and remove them from the server.

Return type:

None

template_fields: Sequence[str] = ['dest_s3', 'src_tiles', 'src_env', 'is_dry_run']

ui_color: str = '#007acc'

ui_fgcolor: str = '#FFFFFF'

SyncTileOperator

This operator is used to sync tileset(s) between Tileserver TSGL environments.

class operators.tile.sync_tile.SyncTileOperator(src_tiles, src_env='prod', dest_envs=None, is_dry_run=True, verify_tile_exists=True, *args, **kwargs)

Operator to sync tileset(s) between tileserver environments.

This operator syncs tiles from the source environment to one or more target environments; in practice, only work in production mode, to sync tiles between prod and staging.

• prod (kanto, Config.TILESERVER_ALIAS)

• staging (hoenn, Config.TILESERVER_ALIAS_STAGE)

• dev (johto, Config.TILESERVER_ALIAS_DEV)

In non-production mode, is_dry_run is always set to True for logging.

Parameters:

• src_env (Literal["prod", "staging", "dev"]) – source environment, “prod”, “staging”, or “dev”

• src_tiles (Union[str, List[str]]) – single path or list of paths to the tilesets (starting at ~/tiles/)

• dest_envs (List[Literal["prod", "staging", "dev"]]) – target environments to sync to, any combination of “prod”, “staging”, and “dev”

• is_dry_run (bool, optional) – whether to perform a dry run, defaults to True

• verify_tile_exists (bool, optional) – whether to verify if the tile exists before performing the action, defaults to True

Raises:

• AirflowException – if the sync operation fails

• AirflowSkipException – if not in production mode and not in dry run mode

custom_operator_name = 'SyncTileOperator'

execute(context)

Sync tileset(s) from source environment to destination environments. In production mode, performs actual syncing. In non-production mode, only logs what would happen if in dry run mode.

Return type:

None

render_template_fields(context, jinja_env=None)

Override to handle special template rendering cases

template_fields: Sequence[str] = ['src_tiles', 'src_env', 'dest_envs', 'is_dry_run']

ui_color: str = '#7e57c2'

ui_fgcolor: str = '#FFFFFF'

S3GeoJsonToAthenaOperator

This operator is used to load a GeoJSON file from S3 into Athena.

class operators.s3_geojson_to_athena.S3GeoJsonToAthenaOperator(s3_geojson_path, output_table_name, output_path, geometry_column='geometry', custom_columns=None, *args, **kwargs)

Converts a GeoJSON file stored in S3 to an Athena table, stored as parquet The output fill will contain a ‘wkt’ column in addition to the WKB geometry column The expected input file must be in the geopandas-parseable .geojson format Please follow standard naming and file structure conventions for both the storage of the

geojson and the output table in Athena

This operation is performed ON THE AIRFLOW SERVER. Adapt as needed to scale.

S3_geojson_path:

the path to the GeoJSON in S3

Output_table_name:

the name to use for the output table

Output_path:

the path to the output table in S3

Geometry_column:

the name of the geometry column in the GeoJSON (default is “geometry”)

Custom_columns:

a dictionary of custom columns and their data types (default is None, string assigned for all columns)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

ui_color: str = '420a69'

S3ShapefileToAthenaOperator

This operator is used to load a shapefile from S3 into Athena.

class operators.s3_shp_to_athena.S3ShapefileToAthenaOperator(s3_shp_path, output_table_name, output_path, s3_shp_layer=None, *args, **kwargs)

To convert a shapefile in S3 to Athena table

Ensure that all the relevant files are in the same directory. (e.g. .shp, .shx, .dbf, .prj, .cpg)

Default EPSG is 4326

S3_shp_path:

the path to the shapefile in S3

Output_table_name:

the name to use for the output table

Output_path:

the path to the output table in S3

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

ui_color: str = '#A6E6A6'

SlackToS3Operator

This operator is used to transfer files from Slack to S3.

class operators.slack_to_s3.SlackToS3Operator(slack_channel_id, slack_users, s3_key, slack_conn='slack_default_api', s3_bucket_name='463002504035-shoreline', *args, **kwargs)

pull targetted slack history to s3 for channels that ReadyAirflow is added to

execute(context)

Gets all threads where specified users are mentioned, saves each thread as a separate JSON file to S3.

ui_color: str = '#eb4034'

NotifySlackOperator

Post a notification message to a Slack channel. Drop-in companion to EmailOperator for DAG failure/success callbacks; supports @mentions via static Slack user IDs or dynamic email lookup (users.lookupByEmail).

class operators.communications.notify_slack.NotifySlackOperator(channel, text, mention_slack_emails=None, mention_user_ids=None, slack_conn_id='slack_default_api', **kwargs)

Post a notification message to a Slack channel.

Args:

channel: Slack channel name or ID, e.g. “#airflow” text: Message body — supports Slack mrkdwn (bold, code, :emoji:) mention_slack_emails: Emails to @mention at the top of the message. Resolved to

Slack user IDs via local SSO lookup first, then Slack API users.lookupByEmail as fallback. Best-effort: unresolved emails are silently skipped.

mention_user_ids: Slack user IDs to @mention directly (no lookup needed). slack_conn_id: Airflow connection ID for the Slack API token

(defaults to Config.SLACK_DEFAULT_API = “slack_default_api”)

Usage in a callback:

NotifySlackOperator(
    task_id="slack_notify",
    channel="#airflow",
    text=":red_circle: *FAILED* | `my_dag`\nTriggered by: Nhat Nguyen | duration: 4s",
    mention_slack_emails=["nhat@ready.net"],
).execute(context)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('text',)

ui_color: str = '#4A154B'

UpdateKnowledgeBaseOperator

This operator is used to update the knowledge base (bedrock).

class operators.update_knowledge_base.UpdateKnowledgeBaseOperator(data_source_id, knowledge_base_id=None, kendra_index_id=None, *args, **kwargs)

Updates either a Bedrock Knowledge Base or Kendra Index with data from a specified source.

This operator handles: - Kendra index synchronization with status monitoring - Bedrock knowledge base data ingestion - Concurrent update protection - Automatic retry logic for Kendra sync (up to 10 attempts)

Args:

data_source_id: ID of the data source to sync/ingest knowledge_base_id: Optional Bedrock knowledge base ID kendra_index_id: Optional Kendra index ID

MAX_SYNC_ATTEMPTS = 10

SYNC_POLL_INTERVAL = 60

execute(context)

Executes the appropriate update operation based on configuration.

Return type:

Dict

logger = <Logger operators.update_knowledge_base (INFO)>

ui_color: str = '#fcba03'

update_kendra_index()

Syncs Kendra index with specified data source and monitors until completion. Retries up to 10 times with 60-second intervals between checks.

Return type:

Dict

update_knowledge_base(description)

Updates Bedrock knowledge base with specified data source.

Return type:

Dict

XLSXToPostgresOperator

This operator is used to load an XLSX file into Postgres.

class operators.xlsx_to_postgres.XLSXToPostgresOperator(csv_path, postgres_conn_id, schema_name, table_name, file_encoding='utf-8', if_table_exists='fail', header='infer', skip_row_count=0, compression='infer', column_rename=None, field_names=None, chunk_size=0, sheet=0, xlsb=False, normalize=False, force_html=False, clear_engine=False, stop_col=0, *args, **kwargs)

Operator to read a CSV file and save it in a Postgres database :type csv_path: :param csv_path: path to CSV file. For S3 use s3://<bucket>/<prefix>/<file> format. :type csv_path: str :type postgres_conn_id: :param postgres_conn_id: the Airflow connection ID to use to store data in Postgres. :type postgres_conn_id: str :type schema_name: :param schema_name: the Postgres schema name :type schema_name: str :type table_name: :param table_name: the Postgres table name within the schema :type table_name: str :type file_encoding: :param file_encoding: default to utf-8. Use ISO-8859-1 if CSV file contains chars outside of utf-8. :type file_encoding: str :type if_table_exists: :param if_table_exists: default to fail for data safety. Can also be replace or append. :type if_table_exists: str :type header: :param header: how to handle the first row of the CSV. Default infer. Can also be None. :type header: str :type skip_row_count: :param skip_row_count: number of rows directly after the header row (if any) to skip. Default 0. Useful when there are 2 or more header rows. :type skip_row_count: int :type compression: :param compression: the compression scheme of the CSV file. Default infer. Can also be gzip, bz2, zip, xz, None :type str :type column_rename: :param column_rename: map of {'oldname':'newname'}. Default None. :type dict :type stop_col: :param stop_col: column number to stop at, if theres a bunch of duplicate columns :type integer

check_table(dest_conn, dest_schema_table)

delete all contents of the given table before writing if necessary

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

handle_df_transform(df, conn, if_table_exists)

helper to write csv

ui_color: str = '#fcba03'

GrapiBaseOperator

Base operator for GRAPI (Grant Ready API) operations.

class operators.grapi.grapi_base.GrapiBaseOperator(tenant, stage='main', timeout=30, retry_count=3, retry_delay=5, *args, **kwargs)

Base operator for making GraphQL requests to the Grapi API. This operator handles authentication, request preparation, and response processing. Subclasses must implement the get_request_payload and post_process_response methods. :type tenant: str :param tenant: The tenant identifier for the request. :type stage: Literal['main', 'stage'] :param stage: The API stage to use (“main” or “stage”). :type timeout: int :param timeout: Request timeout in seconds. :type retry_count: int :param retry_count: Number of retry attempts for the request. :type retry_delay: int :param retry_delay: Delay between retries in seconds.

execute(context)

Execute the GraphQL mutation.

static get_grapi_api_connections()

Get grapi API connections

get_request_payload()

Prepare the GraphQL request payload.

Return type:

Dict[str, Any]

iv_length = 12

post_process_response(response, context=None)

Return type:

Dict[str, Any]

tag_length = 16

template_fields: Sequence[str] = ['tenant']

ui_color: str = '#e8f5e9'

UpdateApplicationAnswerOperator

This operator is used to update application answers in GRAPI.

class operators.grapi.update_application_answer.UpdateApplicationAnswerOperator(expense_line_item, application_id, data, *args, **kwargs)

GRAPHQL_QUERY = '\n    mutation UpdateApplicationAnswerProtected($data: ApplicationAnswerProtectedInput!, $where: ApplicationAnswerProtectedWhereInput!) {\n        updateApplicationAnswerProtected(data: $data, where: $where) {\n            id\n        }\n    }\n    '

application_id = None

data = None

execute(context)

Execute the GraphQL mutation.

expense_line_item = None

get_request_payload()

Prepare the GraphQL request payload.

Return type:

Dict[str, Any]

post_process_response(response, context=None)

Parse and validate GraphQL response.

Return type:

Dict[str, Any]

ui_color: str = '#b4ffb3'

UpdateApplicationResourceOperator

This operator is used to update application resources in GRAPI.

class operators.grapi.update_application_resource.UpdateApplicationResourceOperator(id, data, *args, **kwargs)

GRAPHQL_QUERY = '\n    mutation UpdateApplicationResourceProtected($data: ApplicationResourceProtectedInput!, $where: IdWhereInput!) {\n        updateApplicationResourceProtected(data: $data, where: $where) {\n            # Add the fields you want to return here\n            # For example:\n            id\n            # activity\n            # append\n            attributes\n            # createdAt\n            # updatedAt\n        }\n    }\n    '

data = None

execute(context)

Execute the GraphQL mutation.

get_request_payload()

Prepare the GraphQL request payload.

Return type:

Dict[str, Any]

id = None

post_process_response(response, context=None)

Parse and validate GraphQL response.

Return type:

Dict[str, Any]

ui_color: str = '#b4ffb3'

InvalidateCacheOperator

This operator is used to invalidate cache in GRAPI.

class operators.grapi.invalidate_cache.InvalidateCacheOperator(modules=['FabricLocationStatistics'], *args, **kwargs)

Operator to invalidate cache using the invalidateCache GraphQL mutation.

Parameters:

• tenant – The tenant identifier

• modules (List[str]) – List of modules to invalidate cache for. For the data team we can always set this to [“FabricLocationStatistics”].

• stage – API stage to use (“main” or “stage”)

• timeout – Request timeout in seconds

• retry_count – Number of retry attempts

• retry_delay – Delay between retries in seconds

GRAPHQL_QUERY = '\n    mutation InvalidateCache($data: InvalidateCacheInput!) {\n        invalidateCache(data: $data)\n    }\n    '

execute(context)

Execute the GraphQL mutation.

get_request_payload()

Prepare the GraphQL request payload.

Return type:

Dict[str, Any]

post_process_response(response, context=None)

Parse and validate GraphQL response.

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ['tenant', 'modules']

ui_color: str = '#ffecb3'

GrapiBedrockAutomationOperator

This operator is used for Bedrock automation tasks with GRAPI.

class operators.grapi.grapi_bedrock_invoice.GrapiBedrockAutomationOperator(input_pdf_key, output_uri, tenant_name, env_name, application_resource_id, application_id, **kwargs)

MODEL_ACTIVITIES = {'AiInvoiceProcessFailed': 'AiInvoiceProcessFailed', 'AiInvoiceProcessFinished': 'AiInvoiceProcessFinished', 'AiInvoiceProcessStarted': 'AiInvoiceProcessStarted'}

convert_numpy_types(data)

dest_db = None

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

generate_ai_message(similar_records, current_record)

generate_hash_id(invoice_data)

get_current_flag_info()

get_langfuse_agent()

get_llm()

identify_similar_invoices(invoice_hash)

is_there_any_running_processes()

is_total_exceeds_elis(total_amount, invoice_hash)

make_hashable(obj)

Convert lists to tuples to make them hashable

post_process(context, process_model_output, response)

safe_cast(val, to_type, default=None)

store_data(data)

BaseIcebergTableOperator

Base operator for Iceberg table operations in AWS Athena, providing common functionality for all Iceberg operators.

class operators.iceberg.base_iceberg_table.BaseIcebergTableOperator(table_name, database=None, verify_exists=True, *args, **kwargs)

Base operator for working with Iceberg tables in AWS Athena.

This abstract class provides common functionality for all Iceberg table operators including connection handling, validation, and utility methods.

Parameters:

• database (str) – The database containing the Iceberg table

• table_name (str) – The name of the Iceberg table to operate on

• verify_exists (bool) – Whether to verify the table exists before operations

pre_execute(context)

Hook called before execution. https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/operators/index.html#airflow.operators.BaseOperator.pre_execute Verifies table exists if verify_exists is True.

Return type:

None

template_fields: Sequence[str] = ['database', 'table_name']

ui_color: str = '#E8DAEF'

MaintainIcebergTableOperator

Performs maintenance operations on Iceberg tables, including OPTIMIZE to combine small files and VACUUM to clean up old snapshots.

class operators.iceberg.maintain_iceberg_table.MaintainIcebergTableOperator(table_name, database=None, optimize=True, vacuum=True, optimize_options='REWRITE DATA USING BIN_PACK', *args, **kwargs)

Performs maintenance operations on Iceberg tables in AWS Athena.

This operator combines small files, cleans up old snapshots, and provides before/after statistics on the table.

Parameters:

• table_name (str) – The name of the Iceberg table to maintain

• database (str) – The database containing the Iceberg table

• optimize (bool) – Whether to run OPTIMIZE to combine small files

• vacuum (bool) – Whether to run VACUUM to clean up old snapshots

• optimize_options (str) – Additional options for the OPTIMIZE command

execute(context)

Execute the maintenance operations on the Iceberg table.

Parameters:

context (Dict) – Airflow context

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ['database', 'table_name', 'optimize_options']

ui_color: str = '#D6EAF8'

DropIcebergPartitionOperator

Drops a specific partition from an Iceberg table and optionally runs maintenance afterward.

class operators.iceberg.drop_iceberg_partition.DropIcebergPartitionOperator(table_name, partition_col, partition_value, database=None, additional_filter=None, run_maintenance=True, maintenance_options=None, *args, **kwargs)

Drops a partition from an Iceberg table and optionally runs maintenance afterward.

This operator deletes data from a specific partition using a WHERE clause and can trigger maintenance operations after the deletion.

The DropIcebergPartitionOperator inherits from BaseIcebergTableOperator, so by default it will already verify the table exists during the pre_execute phase. Since your verify_exists parameter is set to True by default in the base operator, all derived operators will perform this check unless explicitly disabled.

Parameters:

• table_name (str) – The Iceberg table name

• partition_col (str) – The partition column name

• partition_value (str, int, or float) – The partition value to drop

• database (str) – The database containing the table

• additional_filter (str) – Additional WHERE conditions (without ‘AND’)

• run_maintenance (bool) – Whether to run maintenance after dropping the partition

• maintenance_options (dict) – Optional parameters for maintenance operation

execute(context)

Execute the partition drop operation. :type context: Dict :param context: Airflow context :type context: Dict

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ['database', 'table_name', 'partition_col', 'partition_value', 'additional_filter']

ui_color: str = '#FADBD8'

RemoveIcebergTableOperator

Completely removes an Iceberg table from the catalog and (optionally) its underlying data files.

class operators.iceberg.remove_iceberg_table.RemoveIcebergTableOperator(table_name, database=None, skip_if_not_exists=True, *args, **kwargs)

Completely remove an Iceberg table by deleting all data and then dropping the table.

This operator first deletes all data in the Iceberg table and then drops the table using AWS Athena.

Parameters:

• table_name (str) – The name of the Iceberg table to remove

• database (str) – The name of the database containing the Iceberg table

• skip_if_not_exists (bool) – Whether to skip without error if table doesn’t exist

execute(context)

Execute the table removal operation.

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ['database', 'table_name', 'skip_if_not_exists']

ui_color: str = '#E8DAEF'

MigrateHiveToIcebergOperator

Migrates a Hive table to Iceberg format with schema evolution capabilities, data validation, and optional optimization.

class operators.iceberg.migrate_hive_to_iceberg.MigrateHiveToIcebergOperator(hive_table, iceberg_table, location, partitioning=None, database=None, table_properties=None, column_mapping=None, include_columns=None, exclude_columns=None, where_clause=None, dry_run=False, optimize_if_successful=True, optimize_options='REWRITE DATA USING BIN_PACK', cleanup_on_failure=True, overwrite_if_exists=False, *args, **kwargs)

Migrates a Hive table to Iceberg format with data validation and enhanced controls.

This operator creates a new Iceberg table from a source Hive table with advanced options for schema customization, table properties, and validation controls.

Parameters:

• hive_table (str) – Source Hive table name

• iceberg_table (str) – Target Iceberg table name

• location (str) – S3 location for the new Iceberg table

• partitioning (Optional[Union[str, List[str]]]) – Optional partitioning configuration, can be: - String column name (e.g., “state”) - List of column names (e.g., [“state”, “year”]) - ARRAY expression (e.g., “ARRAY[‘bucket(id, 64)’, ‘state’]”)

• database (str) – The database containing the Hive and target Iceberg tables

• table_properties (Dict[str, str]) – Additional Iceberg table properties

• column_mapping (Dict[str, str]) – Column mapping from source to target {source_col: target_col}

• include_columns (Optional[List[str]]) – Columns to include (if None, includes all)

• exclude_columns (Optional[List[str]]) – Columns to exclude

• where_clause (Optional[str]) – WHERE clause to filter source data

• dry_run (bool) – If True, only validate and print migration steps without execution

• optimize_if_successful (bool) – Whether to optimize the table after successful creation

• optimize_options (str) – Options for OPTIMIZE command if performed

• cleanup_on_failure (bool) – Whether to remove the Iceberg table if validation fails

DEFAULT_TABLE_PROPERTIES = {'format': 'parquet', 'is_external': 'false', 'optimize_rewrite_delete_file_threshold': '10', 'write_compression': 'snappy'}

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

template_fields: Sequence[str] = ['database', 'table_name', 'hive_table', 'location', 'partitioning', 'optimize_options', 'table_properties', 'where_clause']

ui_color: str = '#D4EFDF'

RunCalixTestOperator

Run Calix speed and/or latency tests on multiple devices: Process test requests in chunks, make concurrent API requests (respecting rate limits), and write results to S3-backed Athena tables.

class operators.calix.run_calix_test.RunCalixTestOperator(test_configs, chunk_size=100, max_concurrent_requests=25, max_retries=2, retry_delay_seconds=5, redshift_conn_id='redshift_dw_dev', redshift_schema='data', redshift_db='dev', *args, **kwargs)

An operator to run Calix speed and/or latency tests on multiple devices. Processes test requests in chunks, makes concurrent API requests (respecting rate limits), and writes results to Redshift tables for efficient incremental inserts.

DEFAULT_TABLES = {'latency': 't_calix_api_latency_results', 'speed': 't_calix_api_speed_results'}

DELAY_START_SPEED_TEST = 10

REDSHIFT_CONN_ID = 'redshift_dw_dev'

REDSHIFT_DB = 'dev'

REDSHIFT_SCHEMA = 'data'

RUN_LATENCY_URL = 'https://api.calix.ai/v1/csc/operator/system/run-latency-test'

RUN_SPEED_URL = 'https://api.calix.ai/v1/csc/operator/system/run-speed-test'

TEST_TYPE_URLS = {'latency': 'https://api.calix.ai/v1/csc/operator/system/run-latency-test', 'speed': 'https://api.calix.ai/v1/csc/operator/system/run-speed-test'}

execute(context)

Execute the operator: run tests, process results, and store data directly to Redshift.

get_output_table(config)

Get output table name from config or default

Return type:

str

insert_to_redshift(results, test_type, output_table)

Insert test results directly into Redshift tables using parameterized SQL. Returns: Number of records successfully inserted

Return type:

int

async process_chunk(chunk_configs)

Process a chunk of test configurations with staggered execution for same-device tests.

async run_test(session, semaphore, config, retries=None, delay_start=0)

Submit a test request for a single device and process the response.

Return type:

Optional[Dict[str, Any]]

template_fields: Sequence[str] = ['test_configs']

ui_color: str = '#A6E6A6'

S3ToSFTPOperator

Operator to transfer a file from an S3 bucket to an SFTP server using paramiko.

class operators.s3_to_sftp.S3ToSFTPOperator(s3_file_uri, sftp_secret_name, sftp_file_path='/', secret_host_key='host', secret_username_key='username', secret_password_key='password', port=22, timeout=30, create_intermediate_dirs=True, create_remote_dirs=True, *args, **kwargs)

Operator to transfer a file from an S3 bucket to an SFTP server.

Parameters:

• s3_file_uri (str) – the S3 URI of the file to transfer (e.g., s3://bucket-name/path/to/file).

• sftp_secret_name (str) – the name of the secret containing SFTP credentials.

• sftp_file_path (str (default will be root of the SFTP server)) – the path on the SFTP server where the file will be uploaded.

• create_remote_dirs (bool (default True)) – whether to create directories on SFTP if they don’t exist

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('s3_file_uri', 'sftp_secret_name', 'sftp_file_path', 'secret_host_key', 'secret_username_key', 'secret_password_key')

ui_color: str = '#5ecede'

ValidateOrCreateSchemaOperator

Validates if a schema exists in Redshift and creates it if it does not, with optional ownership assignment. This operator is useful for ensuring that the necessary schema is present before running other tasks that depend on it, such as creating tables or views within that schema.

class operators.redshift.database_management.ValidateOrCreateSchemaOperator(redshift_conn_id, schema, schema_owner=None, create_if_not_exists=True, database='dev', *args, **kwargs)

Create a schema if it doesn’t exist. Optionally set the owner at creation. (Ownership is not changed if the schema already exists.)

Parameters:

• redshift_conn_id (str) – Airflow connection id for Redshift (cluster or serverless).

• schema (str) – Name of the schema to create.

• schema_owner (Optional[str]) – Optional user or role to own the schema (default is the current user).

• create_if_not_exists (bool) – If True, create the schema if it doesn’t exist.

• database (str) – Database to connect to (default ‘dev’).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('schema', 'schema_owner', 'redshift_conn_id', 'database')

ui_color: str = '#5AFE57'

SetAwsRolePermissiveGrantsOperator

This operator is used to set permissive grants for an AWS role in Redshift. It ensures that the specified role has the necessary permissions to access and manage resources within the Redshift cluster. This operator is for use in an administrative capacity to provision access to data accounts.

class operators.redshift.database_management.SetAwsRolePermissiveGrantsOperator(redshift_conn_id, schema, group_name, iamr_users=None, grant_default_privileges=True, default_privilege_owners=None, database='dev', grant_database_privileges=True, *args, **kwargs)

Ensure a Redshift DB group exists, (optionally) add specific users to it, and grant permissive privileges on a schema to that group.

Typical use: map Identity Center / Google SSO sessions into the group via DbGroups. If you also pass iamr_users, this operator will try to add those existing users to the group (skips any that don’t exist yet).

Parameters:

• redshift_conn_id (str) – Airflow connection id for Redshift (cluster or serverless).

• schema (str) – Target schema to grant.

• group_name (str) – DB group name to manage (e.g., ‘data_team_rw’).

• iamr_users (Optional[Iterable[str]]) – Optional iterable of DB usernames (e.g., ‘IAMR:AWSReservedSSO_…’). These must already exist in the database (user must have logged in once).

• grant_default_privileges (bool) – If True, set default privileges for future tables/sequences (applies to objects created by the user that runs this).

• default_privilege_owners (Optional[Iterable[str]]) – If grant_default_privileges is True, the list of users whose future objects will have default privileges granted to the group. If empty, no default privileges are set.

• database (str) – Database to connect to (default ‘dev’).

• grant_database_privileges (bool) – Whether to grant CREATE/TEMP on the database (default True).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('schema', 'group_name', 'iamr_users', 'redshift_conn_id', 'database', 'default_privilege_owners')

ui_color: str = '#DEFA75'

PostgresToRedshiftDDLOperator

This operator generates and executes a Redshift CREATE TABLE statement based on the schema of an existing Postgres table. It facilitates the migration of table structures from Postgres to Redshift by automatically translating data types and constraints. This operator is useful for data engineers looking to replicate Postgres tables in Redshift for analytics and reporting purposes.

class operators.redshift.pg_to_redshift.PostgresToRedshiftDDLOperator(*, source_table, target_table, pg_conn_id, rs_database, schema, region='us-east-1', workgroup_name='redshift-workgroup-dev', dry_run_mode=False, diststyle='AUTO', sortkey=None, distkey=None, include_pk_metadata=True, create_schema_if_needed=True, aws_conn_id='aws_default', case_sensitive_identifiers=True, identity_policy='strip', drop_if_exists=False, nullable_override=False, **kwargs)

Generate (and optionally execute) a Redshift Serverless CREATE TABLE equivalent to a source Postgres table.

• Pulls column metadata from Postgres (information_schema for PK).

• Maps types to Redshift equivalents (TIMESTAMPTZ, SUPER, VARBYTE, GEOMETRY/GEOGRAPHY, etc).

• Preserves NOT NULL, simple DEFAULTs; PK is added as informational metadata.

• Appends DISTSTYLE AUTO and AUTO SORTKEY (override via args).

• Please Note that Redshift does not enforce primary key constraints, but this operator will include the PK in the DDL for documentation purposes.

Parameters:

• source_table (str) – Name of the source Postgres table (CaSe SeNsItIvE).

• target_table (str) – Name of the target Redshift table (CaSe SeNsItIvE). Typically the same as source_table.

• pg_conn_id (str) – Airflow connection ID for the source Postgres database.

• rs_database (str) – Name of the Redshift database to connect to (typically “dev”).

• schema (str) – Name of the Redshift schema to create the table in (CaSe SeNsItIvE).

• region (str) – AWS region of the Redshift Serverless instance, defaults to “us-east-1”.

• workgroup_name (str) – Redshift Serverless workgroup name, defaults to Config.REDSHIFT_WORKGROUP.

• dry_run_mode (Union[str, bool]) – If True, do not execute the DDL, just return it. Defaults to False. Can also be the string “true”/”false” (case-insensitive).

• diststyle (str) – Redshift distribution style, one of “AUTO”, “EVEN”, “ALL”, “KEY”. Defaults to “AUTO”.

• sortkey (Union[str, List[str], None]) – Redshift sort key, either “AUTO” or a list of column names. Defaults to “AUTO”.

• distkey (Optional[str]) – Redshift distribution key column name, required if diststyle is “KEY”. Defaults to None.

• include_pk_metadata (Union[str, bool]) – If True, include PRIMARY KEY metadata in the DDL. Defaults to True. Can also be the string “true”/”false” (case-insensitive).

• create_schema_if_needed (bool) – If True, includes a “CREATE SCHEMA IF NOT EXISTS” statement before the CREATE TABLE. Defaults to True.

• aws_conn_id (str) – Airflow connection ID for AWS credentials. Defaults to “aws_default”.

• case_sensitive_identifiers (Union[str, bool]) – If True, sets Redshift to treat identifiers as case-sensitive for the session. Defaults to True. Can also be the string “true”/”false” (case-insensitive).

• identity_policy (Literal["preserve", "strip", "as_varchar"]) – How to handle identity/serial columns from Postgres. One of: - “strip” (default): create as normal integer columns (no identity). - “preserve”: create as IDENTITY columns (starts at 1, increments by 1). - “as_varchar”: create as VARCHAR(20) to hold textual representation of the value.

• drop_if_exists (Union[str, bool]) – If True, includes a “DROP TABLE IF EXISTS” statement before the CREATE TABLE. Defaults to False. Can also be the string “true”/”false” (case-insensitive).

• nullable_override (Union[str, bool]) – If True, drops all NOT NULL constraints from the generated DDL (except on PRIMARY KEY columns). Useful when the source Postgres schema has NOT NULL but the data contains NULLs. Defaults to False. Can also be the string “true”/”false” (case-insensitive).

execute(context)

Main execution method for the operator. Fetches Postgres table metadata, builds Redshift CREATE TABLE DDL, and either executes it or returns it in dry-run mode.

template_fields: Sequence[str] = ('source_table', 'target_table', 'rs_database', 'schema', 'diststyle', 'sortkey', 'distkey', 'include_pk_metadata', 'pg_conn_id', 'dry_run_mode', 'create_schema_if_needed', 'aws_conn_id', 'region', 'workgroup_name', 'case_sensitive_identifiers', 'identity_policy', 'drop_if_exists', 'nullable_override')

ui_color: str = '#c2acee'

PostgresToRedshiftUnloadLoadOperator

Unload table data from Postgres to S3 via aws_s3 extension, then COPY into Redshift staging and INSERT…SELECT into the final table with minimal, type-aware shims.

class operators.redshift.pg_to_redshift.PostgresToRedshiftUnloadLoadOperator(*, pg_conn_id, rs_workgroup, rs_database, target_schema, source_schema, table, s3_bucket, s3_prefix, rs_stage_schema=None, redshift_iam_role, region='us-east-1', header=True, gzip=True, where_clause=None, cleanup_after_load=True, aws_conn_id='aws_default', case_sensitive_identifiers=True, max_errors=0, empty_as_null=True, load_mode='APPEND', delete_where_clause=None, **kwargs)

Unload table data from Postgres to S3 via aws_s3 extension, then COPY into Redshift staging and INSERT…SELECT into the final table with minimal, type-aware shims.

• Use _pg_to_rs_type to determine the final RS column type.

• Adds transport-time shims for conversion only where needed:

• json/jsonb/arrays -> export as JSON text; stage VARCHAR; JSON_PARSE on load

• geometry/geography -> export hex EWKB; stage GEOMETRY/GEOGRAPHY; direct COPY load

• primitive types -> stage as VARCHAR; cast on load with NULLIF(‘’, NULL)

Parameters:

• pg_conn_id (str) – Airflow connection ID for the source Postgres database.

• rs_workgroup (str) – Redshift workgroup name.

• rs_database (str) – Redshift database name.

• target_schema (str) – Target schema in Redshift.

• source_schema (str) – Source schema in Postgres.

• table (str) – Table name to transfer.

• s3_bucket (str) – S3 bucket for staging data.

• s3_prefix (str) – S3 prefix for staging data.

• redshift_iam_role (str) – IAM role for Redshift to access S3.

• region (str) – AWS region for S3.

• header (bool) – Include header in CSV export.

• gzip (bool) – Gzip compress CSV files.

• where_clause (str) – Optional WHERE clause for data filtering.

• cleanup_after_load (Union[str, bool]) – Whether to clean up temporary S3 files after loading.

• max_errors (int) – Maximum number of rows that can fail during COPY before the load is aborted. Useful for tolerating a small number of rows with oversized SUPER string values (>65KB Redshift limit). Default 0 (fail on first error).

• empty_as_null (Union[str, bool]) – When True (default), the CSV COPY uses EMPTYASNULL BLANKSASNULL TRIMBLANKS so any empty / whitespace CSV field becomes NULL in staging. Set to False when the source PG distinguishes '' (empty string, exported as "") from NULL (exported as unquoted empty) and the target schema has NOT NULL text columns — leaving these options on conflates the two and breaks the downstream INSERT when an empty-string source row hits a NOT NULL target column. JSON mode is unaffected (these options are CSV-only).

• load_mode (str) – How the target is mutated. One of: APPEND (insert-only, no purge — default for backward compat), TRUNCATE_INSERT (atomic DELETE FROM target + INSERT FROM stage in one Redshift transaction — full refresh), DELETE_INSERT (atomic DELETE FROM target WHERE delete_where_clause + INSERT FROM stage in one Redshift transaction — predicate-based replacement). For refresh modes the operator owns the purge inside the transaction; the DAG’s optional_purge task must be check-only (no DELETE/TRUNCATE). TRUNCATE is intentionally avoided because Redshift TRUNCATE auto-commits and cannot participate in a transaction.

• delete_where_clause (Optional[str]) – SQL predicate (without leading WHERE) used as the DELETE filter in DELETE_INSERT mode. Required for DELETE_INSERT; ignored otherwise. Note: timestamp-based predicates (e.g. "updatedAt" >= cutoff) cannot evict RS rows whose stored timestamp is below the cutoff — use TRUNCATE_INSERT if PG rows are updatable in place and you need parity.

Note

When max_errors > 0 and load_mode is a refresh mode (TRUNCATE_INSERT / DELETE_INSERT), the operator raises if COPY skipped any rows — refusing to mutate the target from an incomplete stage. For APPEND, the prior warn-only behavior is preserved.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('source_schema', 'target_schema', 'table', 's3_bucket', 's3_prefix', 'pg_conn_id', 'rs_database', 'rs_workgroup', 'where_clause', 'aws_conn_id', 'region', 'redshift_iam_role', 'header', 'gzip', 'cleanup_after_load', 'case_sensitive_identifiers', 'max_errors', 'empty_as_null', 'load_mode', 'delete_where_clause')

ui_color: str = '#FFB03A'

GetTenantsOperator

This operator retrieves a list of tenant schema names from a Postgres database, excluding specified schemas. It is useful for multi-tenant applications where operations need to be performed on each tenant’s schema. The operator connects to the Postgres database using a provided connection ID and executes a query to fetch the schema names.

class operators.postgres.utils.get_tenants.GetTenantsOperator(postgres_conn_id, additional_exclude_schemas=None, *args, **kwargs)

Operator to retrieve a list of tenant schema names from a Postgres database.

Parameters:

• postgres_conn_id (str) – Airflow connection id for the Postgres database.

• additional_exclude_schemas (Optional[Any]) – Optional collection or string of schema names to exclude in addition to defaults. Accepts list/tuple/set, a JSON string (e.g. ‘[“public”,”demo”]’), a Python-literal string (e.g. “[‘public’]”), or a single plain string (e.g. “public”). This is CaSe SeNsItIvE.

execute(context)

Retrieve the list of tenant schema names from the Postgres database.

template_fields: Sequence[str] = ('postgres_conn_id', 'additional_exclude_schemas')

ui_color: str = '#4D6A42'

RedshiftGetTenantsOperator

This operator retrieves a list of tenant schema names from a Redshift database via the Redshift Data API. It queries svv_all_schemas filtered to a single source database (default ‘grapi_sync’) and is the Redshift analog of GetTenantsOperator. Useful for fanning out per-tenant work (e.g. ctasifier all_tenants mode).

class operators.redshift.get_tenants.RedshiftGetTenantsOperator(redshift_workgroup, database, tenant_source_database='grapi_sync', region='us-east-1', additional_exclude_schemas=None, *args, **kwargs)

Operator to retrieve a list of tenant schema names from a Redshift database.

Queries svv_all_schemas via the Redshift Data API, filtered to a single source database (default ‘grapi_sync’). Mirrors the API of GetTenantsOperator (Postgres version) so DAGs can swap discovery backends with minimal changes.

Parameters:

• redshift_workgroup (str) – Redshift workgroup name (e.g. Config.REDSHIFT_WORKGROUP).

• database (str) – Redshift database to issue the discovery query against (does not have to match tenant_source_database; this is just where the SELECT runs).

• tenant_source_database (str) – Value used in the svv_all_schemas.database_name = ... filter. Defaults to 'grapi_sync'.

• region (str) – AWS region for the Redshift Data API client.

• additional_exclude_schemas (Any) – Optional collection or string of schema names to exclude in addition to defaults. Accepts list/tuple/set, a JSON string (e.g. ‘[“casf”,”test_base”]’), a Python-literal string (e.g. “[‘casf’]”), or a single plain string (e.g. “casf”). This is CaSe SeNsItIvE.

execute(context)

Retrieve the list of tenant schema names from Redshift.

template_fields: Sequence[str] = ('redshift_workgroup', 'database', 'region', 'tenant_source_database', 'additional_exclude_schemas')

RedshiftGetDatabaseOperator

This operator retrieves a list of Redshift database names matching a SQL LIKE pattern via the Redshift Data API. It queries svv_redshift_databases (e.g. database_name LIKE '%_datamart'). Useful for fanning out CTAS work across many databases sharing a naming convention (e.g. ctasifier all_tenants + tenant_discovery='databases_matching').

class operators.redshift.get_database.RedshiftGetDatabaseOperator(redshift_workgroup, database, database_pattern, region='us-east-1', additional_exclude_databases=None, *args, **kwargs)

Operator to retrieve a list of Redshift database names matching a LIKE pattern.

Queries svv_redshift_databases via the Redshift Data API. Useful for fanning out a CTAS across all databases matching a naming convention (e.g. all ‘*_datamart’ databases for ctasifier all_tenants mode with tenant_discovery=’databases_matching’).

Unlike RedshiftGetTenantsOperator (which discovers schemas inside a single database), this operator discovers DATABASES across the cluster.

Parameters:

• redshift_workgroup (str) – Redshift workgroup name (e.g. Config.REDSHIFT_WORKGROUP).

• database (str) – Redshift database to issue the discovery query against (just a routing target for the Data API; the result set covers all databases visible to svv_redshift_databases).

• database_pattern (str) – SQL LIKE pattern for the database_name column, e.g. ‘%_datamart’. Required.

• region (str) – AWS region for the Redshift Data API client.

• additional_exclude_databases (Any) – Optional collection or string of database names to exclude from the result set. Accepts list/tuple/set, a JSON string (e.g. ‘[“dev”,”staging”]’), a Python-literal string (e.g. “[‘dev’]”), or a single plain string (e.g. “dev”). This is CaSe SeNsItIvE.

execute(context)

Retrieve the list of database names from Redshift.

template_fields: Sequence[str] = ('redshift_workgroup', 'database', 'region', 'database_pattern', 'additional_exclude_databases')

RedshiftDropTableOperator

Drops a single Redshift table with safety guardrails. Optionally authenticates as the cluster superuser (rsadmin) by fetching credentials from AWS Secrets Manager — useful when the target table is not owned by the airflow IAM role (Redshift’s DROP is owner-only and does not support GRANT DROP). Identifier comparisons and the DROP statement are case-sensitive.

class operators.redshift.redshift_drop_table.RedshiftDropTableOperator(*, redshift_conn_id, database, schema, table, dry_run=True, use_rsadmin_override=False, confirm_target=None, require_owner_check=True, forbid_uppercase_table=False, exclude_schemas=None, **kwargs)

Drop a single Redshift table with safety guardrails.

By default authenticates as the airflow IAM role (via the standard Airflow Postgres connection). When use_rsadmin_override=True, fetches the JSON secret named airflow/connections/{redshift_conn_id} from AWS Secrets Manager and authenticates as rsadmin (the cluster superuser) for that single task only — never persists the password and never logs it.

Identifier comparisons and the DROP statement are case-sensitive (enable_case_sensitive_identifier is set on the same physical session that performs the metadata lookup and the DROP).

Parameters:

• redshift_conn_id (str) – Airflow Postgres connection id for Redshift (e.g. Config.REDSHIFT_PG). Also used to derive the rsadmin secret name when use_rsadmin_override is true.

• database (str) – Redshift database within the cluster (e.g. "az_datamart", "dataops").

• schema (str) – Target schema (case-sensitive).

• table (str) – Target table (case-sensitive).

• dry_run (bool) – If true (default), log the DROP statement and table metadata without executing. Operator default is true so the reusable surface is non-destructive — set false explicitly.

• use_rsadmin_override (bool) – If true, authenticate as rsadmin via Secrets Manager (used when the table is not owned by the airflow IAM role).

• confirm_target (Optional[str]) – REQUIRED for real runs (dry_run=False). Must equal "{database}.{schema}.{table}" exactly. Operator refuses to execute any DROP unless the confirmation matches. Cheap defence against fat-finger mistakes.

• require_owner_check (bool) – If true (default), look up the table owner and refuse to issue DDL when the connecting user is not the owner (unless use_rsadmin_override is set, in which case mismatch is expected and only logged).

• forbid_uppercase_table (bool) – If true, refuse to drop CamelCase / mixed-case table names. Default false on the operator (general purpose). The companion redshift_drop_table DAG sets this to true to enforce its policy.

• exclude_schemas (Optional[List[str]]) – Optional list of additional schema names to refuse. The system-schema list (information_schema, pg_catalog, anything starting with pg_, aws_, svv_, stv_, stl_) is always applied on top of this.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

dict

template_fields: Sequence[str] = ('redshift_conn_id', 'database', 'schema', 'table', 'dry_run', 'use_rsadmin_override', 'confirm_target')

ui_color: str = '#E87C3E'

GetLatestFabricFixedAvailabilityVersionOperator

This operator retrieves the latest fabric fixed availability (FFA) version from the fabric_versions table. Supports filtering by status (e.g., ‘ingested’), fabric address version, or fixed availability version. Returns a FabricVersion dict with version, version_date, update_date, and process_uuid.

class operators.postgres.utils.get_latest_fabric_fixed_availability_version.GetLatestFabricFixedAvailabilityVersionOperator(postgres_conn_id='postgres_bossplatform_game_dev', schema='data', table='fabric_versions', status=None, fa_ver=None, fs_ver=None, *args, **kwargs)

Operator to retrieve the latest fabric fixed availability (FFA) version from the fabric_versions table.

Returns a FabricVersion dict with version, version_date, update_date, and process_uuid.

Parameters:

• postgres_conn_id (str) – Airflow connection id for the Postgres database.

• schema (str) – Schema containing the fabric_versions table.

• table (str) – Name of the fabric_versions table.

• status (Optional[str]) – Optional status filter (e.g., ‘ingested’).

• fa_ver (Optional[int]) – Optional fabric address version filter.

• fs_ver (Optional[int]) – Optional fabric fixed availability version filter (takes precedence over fa_ver).

VERSION_DATE_DICT = {1: '2023-06-30', 2: '2022-12-31', 3: '2023-06-30', 4: '2023-12-31', 5: '2024-06-30', 6: '2024-12-31', 7: '2025-06-30'}

execute(context)

Retrieve the latest fabric version from Postgres.

Return type:

FabricVersion

template_fields: Sequence[str] = ('postgres_conn_id', 'status', 'fa_ver', 'fs_ver')

ui_color: str = '#4D6A42'

PostgresGetTenantsOperator

This operator retrieves a list of tenant schema names from a Postgres database.

class operators.pg_get_tenants.PostgresGetTenantsOperator(use_stage_db=True, include_public=False, *args, **kwargs)

An operator to retrieve a list of tenants from a Postgres database.

Parameters:

• use_stage_db (bool) – Whether to use the stage database. Defaults to True.

• include_public (bool) – Whether to include the public schema in the list of tenants. Defaults to False.

execute(context)

Execute the operator.

Returns:

list of tenants

Return type:

list[str]

ui_color: str = '#A80085'

AthenaToPostgresOperator

This operator is used to load data from Athena into Postgres.

class operators.postgres.athena_to_postgres.AthenaToPostgresOperator(*, query, target_table, target_schema='public', postgres_conn_id='postgres_default', aws_conn_id='aws_default', s3_bucket, s3_prefix, if_exists='append', upsert_keys=None, column_transforms=None, create_table_if_not_exists=False, column_type_overrides=None, geometry_srid=None, geometry_srids=None, treat_empty_as_null=True, use_aws_s3_extension=True, cleanup_s3=True, region_name='us-east-1', workgroup='primary', dry_run='false', include_columns=None, skip_if_source_missing=False, **kwargs)

Execute an Athena query and load the results into a PostgreSQL table.

Uses the aws_s3 PostgreSQL extension for efficient loading. Falls back to streaming COPY if configured (TODO).

Workflow:

1. Check if target table exists; if not and create_table_if_not_exists=True, continue.

2. Run Athena Query (SELECT).

3. If table doesn’t exist, create it from Athena schema (with column_type_overrides).

4. Create staging table (with TEXT columns for special types).

5. Import S3 CSV into staging via aws_s3 extension.

6. INSERT from staging to target with type transformations.

7. Drop staging table and cleanup S3 output.

Array Handling:

Athena arrays are exported as [val1, val2] strings in CSV. This operator auto-detects PostgreSQL array columns and transforms them during INSERT. Use column_transforms to override auto-detection or handle complex types.

Note: The default array transform assumes comma-space (, ``) delimiter between elements. If your Athena data uses a different delimiter, provide a custom transform via ``column_transforms.

Geometry/Geography Handling:

PostGIS geometry and geography columns are auto-detected from the target table schema. Values from Athena (WKT or hex-encoded EWKB) are transformed using native ::geometry / ::geography casts with proper SRID handling:

• Use geometry_srid to set a default SRID for all geometry columns (e.g., 4326).

• Use geometry_srids dict for per-column SRID overrides.

• NULL/empty values are handled gracefully (no error on empty WKT).

• When using create_table_if_not_exists, geometry columns MUST be specified in column_type_overrides (e.g., {“geom”: “geometry(Point, 4326)”}) since Athena exports geometry as strings.

Parameters:

• query (str) – The Athena SQL query to execute.

• target_table (str) – Name of the PostgreSQL target table.

• target_schema (str) – PostgreSQL schema. Defaults to “public”.

• postgres_conn_id (str) – Airflow connection ID for PostgreSQL. Defaults to “postgres_default”.

• aws_conn_id (str) – Airflow connection ID for AWS. Defaults to “aws_default”.

• s3_bucket (str) – S3 bucket for Athena query output.

• s3_prefix (str) – S3 prefix for Athena query output.

• if_exists (str) – How to handle existing data - “append”, “replace”, or “upsert”. Defaults to “append”.

• upsert_keys (Optional[List[str]]) – Columns for upsert matching (required if if_exists=”upsert”).

• column_transforms (Optional[Dict[str, str]]) – Dict of {column_name: SQL_expression}. Use {col} placeholder for the column reference. Overrides auto-detected array transforms. Example: {“tags”: “string_to_array({col}, ‘,’)::text[]”}

• create_table_if_not_exists (bool) – If True, create target table from Athena query schema when it doesn’t exist. Defaults to False.

• column_type_overrides (Optional[Dict[str, str]]) – Dict of {column_name: postgresql_type}. Override inferred PostgreSQL types when auto-creating table. Example: {“id”: “uuid”, “geom”: “geometry”}. Note: Geometry/geography columns MUST be specified here since Athena stores them as strings; without override they become text and the geometry cast won’t be applied.

• geometry_srid (Optional[int]) – Default SRID for all geometry columns. Common values: 4326 (WGS84), 3857 (Web Mercator). If None, no SRID is specified in ST_GeomFromText(). Defaults to None.

• geometry_srids (Optional[Dict[str, int]]) – Dict of {column_name: srid} for per-column SRID overrides. Takes precedence over geometry_srid. Example: {“location”: 4326, “boundary”: 3857}.

• treat_empty_as_null (bool) – If True, treat empty CSV fields as NULL during import. Matches Athena’s NULL representation in CSV output. Defaults to True.

• use_aws_s3_extension (bool) – Use aws_s3 extension. Defaults to True.

• cleanup_s3 (bool) – Delete S3 output after load. Defaults to True.

• region_name (str) – AWS region. Defaults to “us-east-1”.

• workgroup (str) – Athena workgroup. Defaults to “primary”.

• dry_run (str) – If ‘true’, no changes will be made to the target database. Defaults to ‘false’. We are using str here to allow templating.

• include_columns (Optional[List[str]]) – List of column names to include in the target table insert. If specified, only these columns will be inserted, allowing Prisma/DB defaults to handle auto-generated columns (id, createdAt, etc.). The Athena query must return columns in the same order as this list. If None, all columns are included.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Dict[str, str]

template_fields: Sequence[str] = ('query', 'target_table', 'target_schema', 's3_bucket', 's3_prefix', 'postgres_conn_id', 'aws_conn_id', 'dry_run', 'include_columns')

ui_color: str = '#C0FFEE'

LoadFabricAddressFullTextSearchOperator

This operator is used to load fabric address data for full text search in Postgres.

class operators.postgres.load_fabric_address_full_text_search.operator.LoadFabricAddressFullTextSearchOperator(state_abbr, dest_db, ulfw_override=False, arc_override=False, dry_run=False, tenant_override='', program_id_override='', program_stage_id_override='', fabric_version_override='', source_table_override='', *args, **kwargs)

Load FCC Fabric Address data into a Postgres full text search table.

Extends TableToPGOperator with domain-specific logic for fabric address data. Automatically resolves tenant from state abbreviation, determines appropriate fabric version, and selects ECS instance size based on state data volume.

Parameters:

• state_abbr (str) – Two-letter state abbreviation (e.g., “CA”, “TX”).

• dest_db (str) – Destination Postgres connection ID.

• ulfw_override (bool) – If True, use ULFW fabric version (6) for applicable states.

• arc_override (bool) – If True, source fabric version from fabric_version_arc in bead profile.

• dry_run (bool) – If True, operator exits before doing any work.

• tenant_override (str) – If non-empty, use this tenant schema instead of deriving from state_abbr. Required for program-specific schemas (e.g., CASF’s california_casf where state=CA but tenant != “california”).

• program_id_override (str) – If non-empty, use this program_id in the FTS SQL filter instead of the hardcoded “Bead”. Required for non-BEAD programs (e.g., “BIG” for CASF).

• program_stage_id_override (str) – If non-empty, use this program_stage_id in the FTS SQL filter instead of the value derived from override flags (ARC / BOB / ACCEPTED). Required for non-BEAD stages (e.g., “BIG_2026”).

• fabric_version_override (str) – If non-empty, use this fabric version in the FTS SQL filter instead of looking it up via the bead profile. Required when the FTS run should match a fabric version that doesn’t correspond to the state’s bead profile entry (e.g., CASF on v6).

• source_table_override (str) – If non-empty, use this Athena table as the source for FTS curation instead of the default c_fabric_sbo_v{version}. Table must expose the columns curate_data*.sql selects (location_id, address_primary, city, state, zip, latitude, longitude, stateabbr). Needed for CASF, where FTS should read from the CASF merge tile so location_id carries the swapped casf_id value (DATASME-1908).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Any

read_sql_template(sql, params={})

Override to use SQL files from operator directory instead of DAG_ROOT.

Return type:

str

template_fields: Sequence[str] = ('state_abbr', 'dest_db', 'ulfw_override', 'arc_override', 'dry_run', 'tenant_override', 'program_id_override', 'program_stage_id_override', 'fabric_version_override', 'source_table_override')

BackupTableOperator

This operator is used to backup a table in Postgres.

class operators.postgres.utils.backup_table.BackupTableOperator(*, postgres_conn_id, tenant, source_table, backup_suffix='_backup', cast_enums_to_text=False, **kwargs)

Backup a Postgres table to a temporary table in the same schema.

Creates a copy of the source table with a suffix appended to the name. Drops any existing backup table with the same name first.

Parameters:

• postgres_conn_id (str) – Airflow connection ID for the Postgres database.

• tenant (str) – The tenant schema name.

• source_table (str) – The table name to backup.

• backup_suffix (str) – Suffix to append to backup table name (default: “_backup”).

• cast_enums_to_text (bool) – If True, cast ENUM columns to TEXT in the backup to avoid dependencies on ENUM types (default: False).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

str

template_fields: Sequence[str] = ('postgres_conn_id', 'tenant', 'source_table')

ui_color: str = '#BADA55'

MangleUsersOperator

This operator is used to mangle user data in Postgres for privacy purposes.

class operators.postgres.utils.mangle_users.MangleUsersOperator(*, postgres_conn_id, tenant, exclude_emails=None, prefix='synced', domain='ready.net', **kwargs)

Mangle user emails by prefixing and nulling passwords.

This disables login access while preserving user records for FK integrity. Emails are transformed: user@example.com -> synced+user_example.com@ready.net

Parameters:

• postgres_conn_id (str) – Airflow connection ID for the Postgres database.

• tenant (str) – The tenant schema name.

• exclude_emails (Optional[list[str]]) – List of email addresses to exclude from mangling.

• prefix (str) – Prefix to prepend to mangled emails (default: “synced”).

• domain (str) – Domain to append to mangled emails (default: “ready.net”).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

int

template_fields: Sequence[str] = ('postgres_conn_id', 'tenant', 'exclude_emails')

ui_color: str = '#DEFACE'

RestoreStagedUsersOperator

This operator restores staged users after a prod-to-stage Postgres sync. Access grants whose applicants did not survive the refresh are skipped because their foreign-key target no longer exists.

class operators.postgres.utils.restore_staged_users.RestoreStagedUsersOperator(*, postgres_conn_id, tenant, backup_suffix='_backup', mangle_prefix='synced', mangle_domain='ready.net', **kwargs)

Restore stage users after a prod-to-stage sync.

This operator: 1. Updates credentials for users that exist in both backup and synced db 2. Inserts stage-only users back into the User table 3. Rebuilds UserAccess from the backup, so stage keeps the access it granted

and users arriving from prod arrive with none

Parameters:

• postgres_conn_id (str) – Airflow connection ID for the Postgres database.

• tenant (str) – The tenant schema name.

• backup_suffix (str) – Suffix used for backup tables (default: “_backup”).

• mangle_prefix (str) – Prefix used for mangled emails to exclude (default: “synced”).

• mangle_domain (str) – Domain used for mangled emails to exclude (default: “ready.net”).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

dict

template_fields: Sequence[str] = ('postgres_conn_id', 'tenant')

ui_color: str = '#ACCEDE'

ValidateOrCreateDatabaseOperator

This operator validates if a database exists in Redshift and creates it if it does not.

class operators.redshift.database_management.ValidateOrCreateDatabaseOperator(redshift_conn_id, database, db_owner='rsadmin', create_if_not_exists=False, admin_database='dev', use_rsadmin_override=False, *args, **kwargs)

Ensure a Redshift database exists. If missing and create_if_not_exists=True, create it (optionally with a specific owner). Ownership is only set at creation time.

Parameters:

• redshift_conn_id (str) – Airflow connection id for Redshift.

• database (str) – Name of the database to ensure.

• db_owner (str) – Optional DB owner to set at creation (not changed if DB already exists).

• create_if_not_exists (bool) – Create the DB if it doesn’t exist.

• admin_database (str) – Database to connect to when issuing CREATE DATABASE. Defaults to ‘dev’.

• use_rsadmin_override (bool) – If True, authenticate as rsadmin (cluster superuser) via AWS Secrets Manager instead of the connection’s own identity (the Airflow EC2 IAM role). Required when the executing role is not a superuser and you need to CREATE DATABASE ... WITH OWNER <other_user> (e.g. owner rsadmin), which only a superuser may do. Defaults to False (backwards compatible).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('redshift_conn_id', 'database', 'db_owner', 'admin_database', 'create_if_not_exists', 'use_rsadmin_override')

ui_color: str = '#57D1FA'

SetAwsRoleAwsDataCatalogGrantsOperator

This operator is used to set AWS Data Catalog grants for an AWS role in Redshift.

class operators.redshift.database_management.SetAwsRoleAwsDataCatalogGrantsOperator(redshift_conn_id, iamr_users=None, group_name=None, glue_databases='ALL', lake_formation_mode=True, grant_default_select=True, database_name='awsdatacatalog', database='dev', include_current_user=True, *args, **kwargs)

Grant IAMR users/groups access to the auto-mounted Glue Data Catalog in Redshift (database: awsdatacatalog), plus selected Glue databases (as schemas).

Modes:

• lake_formation_mode=True -> Grant USAGE on database and schemas; LF governs table access.

• lake_formation_mode=False -> Also grant SELECT on existing tables (and optional defaults).

Parameters:

• redshift_conn_id (str) – Airflow connection id for Redshift.

• iamr_users (Optional[Iterable[str]]) – Optional iterable of DB usernames (e.g., ‘IAMR:…’).

• group_name (Optional[str]) – Optional DB group to grant.

• glue_databases (Optional[Iterable[str] | str]) – Iterable of Glue DB names (schemas) OR the string ‘ALL’ to auto-discover.

• lake_formation_mode (bool) – If True, rely on LF for table-level access (USAGE-only in Redshift).

• grant_default_select (bool) – If False, skip default privileges. (Default True when not LF mode.)

• database_name (str) – The auto-mounted external database name (default ‘awsdatacatalog’).

• database (Optional[str]) – Optional Redshift database to connect to for issuing statements.

• include_current_user (bool) – If True, include the current user in the grants.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('redshift_conn_id', 'iamr_users', 'group_name', 'glue_databases', 'lake_formation_mode', 'grant_default_select', 'database_name', 'database', 'include_current_user')

ui_color: str = '#CB80FD'

RefreshAwsDataCatalogGrantsOperator

This operator is used to refresh AWS Data Catalog grants in Redshift.

class operators.redshift.database_management.RefreshAwsDataCatalogGrantsOperator(redshift_conn_id, group_name=None, iamr_users=None, glue_databases='ALL', database_name='awsdatacatalog', grant_select=True, database='dev', include_current_user=True, *args, **kwargs)

Ensure Redshift principals can see and SELECT all current external tables in the auto-mounted Glue Data Catalog database (default: ‘awsdatacatalog’), and keep things fresh after new tables are added by Athena/awswrangler.

Actions (idempotent):

• GRANT USAGE ON DATABASE <catalog>

• For each Glue DB (schema) -> GRANT USAGE ON SCHEMA

• For each Glue DB (schema) -> GRANT SELECT ON ALL TABLES IN SCHEMA

Targets:

• A DB GROUP (recommended) via group_name

• And/or explicit users via iamr_users (only if they already exist in Redshift)

Parameters:

• redshift_conn_id (str) – Airflow connection to Redshift (works with Serverless).

• group_name (Optional[str]) – Optional DB group to grant (e.g., ‘data_team_rw’).

• iamr_users (Optional[Iterable[str]]) – Optional iterable of Redshift usernames (e.g., ‘IAMR:…’).

• glue_databases (Optional[Iterable[str] | str]) – List of Glue DB names (schemas), or “ALL” to auto-discover.

• database_name (str) – Catalog database name (default ‘awsdatacatalog’).

• grant_select (bool) – If False, only USAGE grants are applied (default True).

• database (Optional[str]) – Optional Redshift database to connect to for issuing statements.

• include_current_user (bool) – If True, include the current user in the grants.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('redshift_conn_id', 'group_name', 'iamr_users', 'glue_databases', 'database_name', 'grant_select', 'database', 'include_current_user')

ui_color: str = '#F859FB'

SetCaseSensitiveIdentifierOperator

This operator is used to set case-sensitive identifier settings in Redshift.

class operators.redshift.database_management.SetCaseSensitiveIdentifierOperator(redshift_conn_id, value=True, users='ALL', exclude_users=None, include_iam_users_only=False, include_superusers=False, additional_protected_users=None, dry_run=False, database='dev', *args, **kwargs)

Set (or clear) the Redshift user-level parameter enable_case_sensitive_identifier for many users at once. Useful for “baseline” runs so every new session for those users is case-sensitive by default without a per-session SET.

Actions (idempotent):

• ALTER USER “<user>” SET enable_case_sensitive_identifier TO true|false

Selection:

• users=”ALL” (default) -> act on every user from pg_user (minus exclusions)

• users=[…] -> act only on the provided usernames

• include_iam_users_only=True -> limit to IAM-authenticated principals (IAMR:/IAMA:)

Parameters:

• redshift_conn_id (str) – Airflow connection id for Redshift (cluster or serverless).

• value (bool) – Whether to set the parameter to TRUE (default) or FALSE.

• users (Union[str, Iterable[str]]) – “ALL” to target all users, or an iterable of specific usernames.

• exclude_users (Optional[Iterable[str]]) – Optional iterable of usernames to skip (applied after selection).

• include_iam_users_only (bool) – If True, restrict to users whose names start with IAMR:/IAMA:.

• include_superusers (bool) – If True, include superusers in the update (default False).

• additional_protected_users (Optional[Iterable[str]]) – Additional usernames to skip (beyond built-in service accounts).

• dry_run (bool) – If True, log planned changes without executing (default False).

• database (Optional[str]) – Optional database to connect to when issuing statements.

execute(context)

Apply enable_case_sensitive_identifier for the selected users.

Safety rails: - Skips superusers and common service accounts by default

(unless include_superusers=True).

• Can restrict to IAMR users only (include_iam_users_only=True).

• Supports dry_run to preview changes.

Returns a small summary dict for logging/metrics.

template_fields: Sequence[str] = ('redshift_conn_id', 'users', 'exclude_users', 'include_iam_users_only', 'database', 'value')

ui_color: str = '#ABA7E5'

CreateRedshiftUserOperator

This operator is used to create a user in Redshift.

class operators.redshift.database_management.CreateRedshiftUserOperator(redshift_conn_id, username, password=None, secret_arn=None, grant_connect=True, grant_temp=False, database='dev', createdb=False, connection_limit=None, *args, **kwargs)

Create a Redshift user with password if it doesn’t exist. Optionally retrieves password from AWS Secrets Manager. Grants CONNECT (and optionally TEMP) on the target database.

Password can be provided in two ways (one is required):

• secret_arn: Retrieve from AWS Secrets Manager (recommended for production). The secret must already exist and contain a JSON key named ‘password’.

• password: Pass directly as a string (less secure, suitable for dev/testing).

Parameters:

• redshift_conn_id (str) – Airflow connection id for Redshift.

• username (str) – Name of the user to create.

• password (Optional[str]) – Password for the user. Ignored if secret_arn is provided.

• secret_arn (Optional[str]) – AWS Secrets Manager ARN containing password. The secret must already exist and contain a JSON object with a ‘password’ key.

• grant_connect (bool) – Deprecated/ignored. Redshift does not support GRANT CONNECT. Users can connect by default once created. Kept for backwards compatibility.

• grant_temp (bool) – Grant TEMPORARY (create temp tables) on the database (default False).

• database (str) – Database to connect to and grant on.

• createdb (bool) – Allow the user to create databases (default False).

• connection_limit (Optional[int]) – Max concurrent connections for this user (default unlimited).

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('redshift_conn_id', 'username', 'password', 'secret_arn', 'database', 'grant_temp', 'createdb')

ui_color: str = '#5EEDED'

SetSchemaReadGrantsOperator

This operator is used to set read grants on a schema in Redshift.

class operators.redshift.database_management.SetSchemaReadGrantsOperator(redshift_conn_id, schema, grantee_users=None, grantee_group=None, grant_select=True, grant_execute=True, set_default_privileges=True, default_privilege_owners=None, database='dev', *args, **kwargs)

Grant read-only access on a schema: USAGE, SELECT on tables, EXECUTE on functions. Sets default privileges for future objects created by specified owners.

This operator follows least-privilege principles for service accounts that only need to read data or execute functions (e.g., Analytics Toolbox).

Parameters:

• redshift_conn_id (str) – Airflow connection id for Redshift.

• schema (str) – Target schema to grant read access on.

• grantee_users (Optional[Iterable[str]]) – List of usernames to grant access to.

• grantee_group (Optional[str]) – Group name to grant access to.

• grant_select (bool) – Grant SELECT on all tables (default True).

• grant_execute (bool) – Grant EXECUTE on all functions (default True).

• set_default_privileges (bool) – Set default privileges for future objects (default True).

• default_privilege_owners (Optional[Iterable[str]]) – Users whose future objects get default grants.

• database (str) – Database to connect to.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('redshift_conn_id', 'schema', 'grantee_users', 'grantee_group', 'default_privilege_owners', 'database')

ui_color: str = '#A55A55'

PitIngestBaseOperator

Base operator for Point-in-Time ingestion operations.

class operators.point_in_time.ingest.PitIngestBaseOperator(dest_db, schema, lake_table, program_id, program_stage, fabric_version, fixed_availability_version, reference_date, athena_alias=None, sub_program_id=None, sync_addresses=True, drop_existing_stage=False, cleanup_mismatched_versions=True, extract_sql_override=None, attributes_sql='', pre_ineligible_sql='False', task_suffix='', large_states=None, *args, **kwargs)

Base operator for Point-in-Time ingestion. Handles common ETL logic while delegating stage-specific SQL generation to subclasses.

Subclasses must implement get_extract_sql_path() to return the path to their stage-specific extraction SQL template.

Workflow:

1. Optionally drop existing program stage records

2. Sync addresses from lake to pit_fabric_addresses (optional)

3. Extract PIT data from lake using stage-specific SQL

4. Load to temp table, convert attributes column

5. Upsert into pit_address_metadata

6. Cleanup superseded FFA versions of this program/stage/fabric (optional)

7. Update table statistics

Parameters:

• dest_db (str) – Config attribute name for destination DB (e.g., “BOSS_PLATFORM_DB”)

• schema (str) – Tenant schema name (e.g., “virginia”, “arkansas”)

• lake_table (str) – Source merge tile table in Athena (e.g., “c_arc_merge_tile_va”)

• program_id (str) – Program identifier (e.g., “Bead”, “Baseline”)

• program_stage (str) – Program stage (e.g., “ARC”, “BOB”, “ACCEPTED”)

• fabric_version (int) – FCC fabric version used in source data

• fixed_availability_version (int) – Fixed availability version used in source data

• athena_alias (str) – Cross-DB alias for Athena queries. Defaults to Config.ATHENA_GRAPI_DB_ALIAS

• sub_program_id (str) – Sub-program identifier if applicable

• reference_date (str) – Program reference date (YYYY-MM-DD format). Required.

• sync_addresses (bool) – Whether to sync pit_fabric_addresses. Defaults to True

• drop_existing_stage (bool) – If True, delete existing stage records before insert. Defaults to False

• cleanup_mismatched_versions (bool) – If True, remove records of this program, stage and fabric version at a different FFA version. Other fabric versions are kept. Defaults to True

• extract_sql_override (str) – Path to custom extraction SQL template. Overrides stage default

• task_suffix (str) – ECS task size suffix (“”, “large”, “xlarge”). Auto-detected if not set

• large_states (List[str]) – List of state schemas that require “large” task suffix. Defaults to DEFAULT_LARGE_STATES

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

dict

abstract get_extract_sql_path()

Return the path to the stage-specific extraction SQL template.

Return type:

Path

get_extract_variables()

Return variables for the extraction SQL template. Override in subclass to add stage-specific variables.

Return type:

dict

get_task_suffix()

Determine ECS task size. Large states get ‘large’ suffix by default.

Return type:

str

template_fields: Sequence[str] = ('dest_db', 'schema', 'lake_table', 'program_id', 'sub_program_id', 'program_stage', 'reference_date', 'athena_alias', 'fabric_version', 'fixed_availability_version', 'drop_existing_stage', 'task_suffix', 'attributes_sql', 'pre_ineligible_sql')

ui_color: str = '#8B5CF6'

ui_fgcolor: str = '#FFFFFF'

PitIngestArcOperator

This operator is used for Point-in-Time ingestion for ARC data.

class operators.point_in_time.ingest.PitIngestArcOperator(dest_db, schema, lake_table, program_id, program_stage, fabric_version, fixed_availability_version, reference_date, athena_alias=None, sub_program_id=None, sync_addresses=True, drop_existing_stage=False, cleanup_mismatched_versions=True, extract_sql_override=None, attributes_sql='', pre_ineligible_sql='False', task_suffix='', large_states=None, *args, **kwargs)

Operator for ingesting ARC program stage data into Point-in-Time tables.

Uses extraction SQL at: SQL/ingest/stages/arc/extract_pit.sql

custom_operator_name = 'PitIngestArcOperator'

get_extract_sql_path()

Return the path to the stage-specific extraction SQL template.

Return type:

Path

BuildRedactedTileSqlOperator

This operator is used to build SQL for redacted tiles.

class operators.tile.build_redacted_tile_sql.BuildRedactedTileSqlOperator(lake_table, database=None, exclude_columns=None, column_renames=None, additional_exclude_columns=None, *args, **kwargs)

Builds a SELECT query from a lake table with PII/sensitive columns excluded (current implementation EXCLUDE PII columns rather than REDACT text).

Queries the Athena information_schema to get column names, filters out excluded columns, applies any column renames, and returns a SELECT statement suitable for use with CreateTileOperator.

Parameters:

• lake_table (str) – Source lake table name (e.g., “c_cpc_merge_tile_ca”)

• database (str) – Athena database name. Defaults to Config.DATA_LAKE_DB

• exclude_columns (List[str]) – List of column names to exclude from the SELECT. Defaults to DEFAULT_EXCLUDE_COLUMNS (PII columns).

• column_renames (Dict[str, str]) – Dict mapping original column names to new names. Defaults to DEFAULT_COLUMN_RENAMES ({“bsl_flag”: “is_cab”}).

• additional_exclude_columns (List[str]) – Additional columns to exclude beyond the defaults. Merged with exclude_columns.

Returns:

SELECT SQL string via XCom

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

str

template_fields: Sequence[str] = ('lake_table', 'database')

ui_color: str = '#ed17ed'

ui_fgcolor: str = '#FFFFFF'

BaseTileDeployOperator

Base operator for tile deployment operations.

class operators.tile.join_tile.BaseTileDeployOperator(title, copy_main_to_stage=False, description='', deploy_tiles=False, deploy_environment='main', inlets=[], data_product_name=None, keep_tiles_on_mount_folder=False, *args, **kwargs)

A Base operator to deploy a tile to tileserver

add_to_datahub(context)

Return type:

None

copy_mbtiles(environment_alias)

copy mbtile files to the tileserver

Parameters:

environment_alias (str) – the tile server’s ailas

Return type:

None

Returns:

None

Raises:

AirflowException – if the command fails

copy_to_stage()

Handles copying the mbtiles to the staging tileserver. only works on production

custom_operator_name = 'BaseTileDeployOperator'

datahub_environment()

Return type:

str

deploy_tilset()

deploys the tileset to the tileserver

Return type:

None

Returns:

None

Raises:

AirflowException – if the command fails

execute(context)

Parameters:

context (Context) – the context

Return type:

None

get_lineage()

Return type:

tuple[list[Urn], list[Urn]]

notify_slack(message)

Return type:

None

on_kill()

Override this method to clean up subprocesses when a task instance gets killed.

Any use of the threading, subprocess or multiprocessing module within an operator needs to be cleaned up, or it will leave ghost processes behind.

Return type:

None

post_deployment_actions(context)

Return type:

None

process_tile(context)

Return type:

None

remove_source_mbtiles_and_geojson()

removes files from efs-mount drive

Return type:

None

Returns:

None

Raises:

AirflowException – if the command fails

run_bash_command(command)

runs a command with subprocess hook

Parameters:

command (str) – the command

Raises:

AirflowException – if the command fails

Return type:

None

set_map_name(context)

Return type:

None

property subprocess_hook

Returns hook for running the bash command.

template_fields_renderers: dict[str, str] = {'copy_main_to_stage': 'bool', 'deploy_tiles': 'bool'}

tileserver_alias()

Return type:

str

tileset_urn()

Return type:

Urn

ui_color: str = '#9933FF'

ui_fgcolor: str = '#FFFFFF'

validate_deployment(context)

Return type:

None

JoinTileOperator

This operator is used to join tiles.

class operators.tile.join_tile.JoinTileOperator(title, input_mbtiles_stub, copy_main_to_stage=False, description='', deploy_tiles=False, task_suffix='large', deploy_environment='main', inlets=[], tippecanoe_cmd_template="find {mount_path} -name '{input_mbtiles_stub}_*.mbtiles' | xargs tile-join -z 15 -pk -f -o {mount_path}/{input_mbtiles_stub}.mbtiles -n '{name}' -N '{description}'", map_name='', data_product_name=None, keep_tiles_on_mount_folder=False, *args, **kwargs)

An operator to use join_tile tippecaneo command to join multiple tiles

custom_operator_name = 'JoinTileOperator'

process_tile(context)

process the tile using tippecanoe command

Parameters:

context (Context) – the context

Return type:

None

ui_color: str = '#9933FF'

ui_fgcolor: str = '#FFFFFF'

S3ToGoogleDriveOperator

This operator is used to transfer files from S3 to Google Drive.

class operators.s3_gdrive.S3ToGoogleDriveOperator(s3_paths, dest_folder_id, s3_bucket, overwrite, dest_file_name=None, *args, **kwargs)

Operator to transfer files from S3 to a Google Drive folder.

Parameters:

• s3_paths (list[str]) – List of S3 keys/paths for the source files.

• dest_folder_id (str) – The destination Google Drive folder ID.

• s3_bucket (str) – The S3 bucket name where the files are stored.

• overwrite (bool) – Flag to indicate whether to overwrite existing files in Google Drive.

• dest_file_name (str) – The destination file name in Google Drive. If not provided, the original file name will be used.

• google_service_account_file (str) – Path to the Google service account JSON file.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('s3_paths', 'dest_folder_id', 'dest_file_name')

S3GeoJsonToAthenaOperator

This operator is used to load GeoJSON from S3 into Athena.

class operators.s3_geojson_to_athena.S3GeoJsonToAthenaOperator(s3_geojson_path, output_table_name, output_path, geometry_column='geometry', custom_columns=None, *args, **kwargs)

Converts a GeoJSON file stored in S3 to an Athena table, stored as parquet The output fill will contain a ‘wkt’ column in addition to the WKB geometry column The expected input file must be in the geopandas-parseable .geojson format Please follow standard naming and file structure conventions for both the storage of the

geojson and the output table in Athena

This operation is performed ON THE AIRFLOW SERVER. Adapt as needed to scale.

S3_geojson_path:

the path to the GeoJSON in S3

Output_table_name:

the name to use for the output table

Output_path:

the path to the output table in S3

Geometry_column:

the name of the geometry column in the GeoJSON (default is “geometry”)

Custom_columns:

a dictionary of custom columns and their data types (default is None, string assigned for all columns)

allow_nested_operators: bool

depends_on_past: bool

do_xcom_push: bool

downstream_task_ids: set[str]

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

ignore_first_depends_on_past: bool

inlets: list

map_index_template: str | None

max_active_tis_per_dag: int | None

max_active_tis_per_dagrun: int | None

multiple_outputs: bool

outlets: list

owner: str

params: ParamsDict | dict

priority_weight: int

task_id: str

trigger_rule: TriggerRule

ui_color: str = '420a69'

upstream_task_ids: set[str]

wait_for_downstream: bool

wait_for_past_depends_before_skipping: bool

weight_rule: PriorityWeightStrategy

S3ShapefileToAthenaOperator

This operator is used to load Shapefiles from S3 into Athena.

class operators.s3_shp_to_athena.S3ShapefileToAthenaOperator(s3_shp_path, output_table_name, output_path, s3_shp_layer=None, *args, **kwargs)

To convert a shapefile in S3 to Athena table

Ensure that all the relevant files are in the same directory. (e.g. .shp, .shx, .dbf, .prj, .cpg)

Default EPSG is 4326

S3_shp_path:

the path to the shapefile in S3

Output_table_name:

the name to use for the output table

Output_path:

the path to the output table in S3

allow_nested_operators: bool

depends_on_past: bool

do_xcom_push: bool

downstream_task_ids: set[str]

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

ignore_first_depends_on_past: bool

inlets: list

map_index_template: str | None

max_active_tis_per_dag: int | None

max_active_tis_per_dagrun: int | None

multiple_outputs: bool

outlets: list

owner: str

params: ParamsDict | dict

priority_weight: int

task_id: str

trigger_rule: TriggerRule

ui_color: str = '#A6E6A6'

upstream_task_ids: set[str]

wait_for_downstream: bool

wait_for_past_depends_before_skipping: bool

weight_rule: PriorityWeightStrategy

S3ToFTPSOperator

This operator is used to transfer files from S3 to an FTPS server.

class operators.s3_to_ftps.S3ToFTPSOperator(s3_bucket, s3_key, secret_name, remote_path=None, use_mainframe_dataset=True, dataset_sequence=None, **kwargs)

Transfer file from S3 to FTPS server.

Supports: - FTPS (FTP over TLS) - required for MN SWIFT mainframe - SFTP (SSH FTP) - may be used by other states

Credentials retrieved via Airflow Connections (BaseHook.get_connection). FTPS-specific config is stored in the connection’s extra JSON field.

Airflow Connection fields used: - host: FTPS server hostname - login: Username - password: Password - port: Server port (default: 21) - extra (JSON): {

“protocol”: “ftps-explicit”, “remote_path”: “/base/path”, “tls”: {

“verify_server_cert”: false, “ca_bundle_pem”: “—–BEGIN CERTIFICATE—–…”

}, “ftp”: {

“post_login_commands”: [“SITE LRECL=644”, “SITE RECFM=FB”], “transfer_mode”: “binary”

}, “mainframe”: {

“remote_dataset”: “‘ADM.MB.UAT1.IBPV.B220701.IF0407.T001’”

}

}

Parameters:

• s3_bucket (str) – S3 bucket name

• s3_key (str) – S3 object key (path within bucket)

• secret_name (str) – Airflow connection ID for FTPS credentials

• remote_path (Optional[str]) – Remote path on FTPS server (overrides mainframe.remote_dataset if provided)

• use_mainframe_dataset (bool) – If True, use mainframe.remote_dataset from secret (default: True)

• dataset_sequence (Optional[str]) – Zero-padded transaction sequence (“001”, “002”, …) used to vary the FINAL qualifier of the mainframe dataset name, so a run that produces several files writes each to its own dataset instead of overwriting one. mainframe.remote_dataset in the connection supplies the prefix and the environment qualifier; only the trailing .Tnnn is substituted. Omit to keep the configured dataset name verbatim (previous behaviour).

XCom outputs: - transfer_status: SUCCESS/FAILURE - transfer_response: Response details from server - transfer_timestamp: ISO timestamp when transfer completed

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ['s3_bucket', 's3_key', 'remote_path', 'secret_name']

ui_color: str = '#f0ad4e'

UnzipOperator

This operator is used to unzip files in S3.

class operators.s3_unzip.UnzipOperator(src_bucket, src_key, dest_key, *args, **kwargs)

Unzip files from S3 to S3.

Parameters:

• src_bucket (str) – source S3 bucket name

• src_key (str) – source S3 object key of the zip file

• dest_key (str) – destination S3 prefix where unzipped files will be stored

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

ui_color: str = '#FF69B4'

ZipS3Operator

This operator is used to zip files in S3.

class operators.s3_zip.ZipS3Operator(*, zip_group, files, dest_bucket, dest_prefix, email_recipients, max_zip_gb=3, overwrite=False, zip_name=None, **kwargs)

Stream-zips files grouped by zip_group (from Athena metadata) into ≤ N GB archives and uploads the ZIPs back to S3 without calculating sizes or re-grouping.

Parameters:

• files (List[Dict[str, str]]) – list of dicts with keys: {“bucket”: str, “key”: str, “arcname”: str, “file_size_byte”: int, “zip_group”: int}

• dest_bucket (str) – bucket where ZIP(s) will be stored

• dest_prefix (str) – folder/prefix inside dest_bucket

• email_recipients (List[str]) – list of e-mail addresses (only passed through XCom so the e-mail task can reuse)

• max_zip_gb (int) – nominal size of each ZIP in GiB (for naming only)

• overwrite (bool) – if True, delete existing ZIPs under dest_prefix before uploading new ones

• zip_name (Optional[str]) – archive file name to use instead of the default zip_part_{group}.zip. When the operator holds more than one group, _part{group} is appended to keep the keys unique.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ('files', 'dest_bucket', 'dest_prefix', 'email_recipients', 'zip_name')

MlflowModelInvokeOperator

This operator is used to invoke an MLflow model.

class operators.mlflow_model_invoke.MlflowModelInvokeOperator(model_id, output_s3_prefix, output_athena_dataset_name, input_sql, datalake_db, output_fields=None, model_params={}, use_ecs=False, fallback_to_ecs_on_timeout=False, mode='overwrite', kserve_timeout=3600, ecs_size='', *args, **kwargs)

FIELD_MAPPINGS = {'BYTES': 'string', 'FP32': 'double', 'FP64': 'double', 'INT32': 'int', 'INT64': 'bigint', 'INT8': 'int'}

create_output_table(all_df_fields)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_model_base_url()

get_model_invocation_url()

get_model_metadata()

get_model_schema()

run_ecs_model(input)

run_using_kserve(input, url, all_df_fields, does_table_exist)

ui_color: str = '#A6E6A6'

GdriveZipToS3Operator

This operator is used to zip files from Google Drive and upload to S3.

class operators.multifile_zipper.GdriveZipToS3Operator(output_zip_file_name, source_gdrive_folder_url, s3_key='', s3_bucket_name='463002504035-boss-private-dev', remove_spaces_from_filename=False, little_zip_threshold_mb=500, generate_presigned_url=True, presigned_url_expiry=3600, *args, **kwargs)

This operator will zip files from a Google Drive folder and upload the zip file to an S3 bucket. The current implementation will raise an exception if the total size of the files exceeds a threshold. A future implementation will trigger an ECS execution to zip the files if the threshold is exceeded.

The default upload location is the BOSS private bucket.

Returns:

None

Return type:

None

Parameters:

• output_zip_file_name (str) – The name of the output zip file.

• source_gdrive_folder_url (str) – The URL of the Google Drive folder to zip files from.

• s3_key (str) – The S3 key where the zip file will be uploaded.

• s3_bucket_name (str) – The name of the S3 bucket where the zip file will be uploaded.

• remove_spaces_from_filename (bool) – Whether to remove spaces from the filenames in the zip file.

• little_zip_threshold_mb (int) – The threshold in MB for the total size of files to be zipped locally.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ['source_gdrive_folder_url', 'output_zip_file_name', 's3_key', 'generate_presigned_url', 'presigned_url_expiry', 's3_bucket_name', 'remove_spaces_from_filename']

ui_color: str = '#A69696'

PeeringDBToAthenaOperator

This operator is used to load PeeringDB data into Athena.

class operators.peeringdb_to_athena.PeeringDBToAthenaOperator(peeringdb_endpoint, output_table_name, output_path, where=None, limit=None, *args, **kwargs)

To convert a PeeringDB endpoint to Athena table

Parameters:

• peeringdb_endpoint – the endpoint of the PeeringDB API

• fields – list of fields to return, defaults to None which loads all fields

Type:

peeringdb_endpoint: str

add_transformed_fields_to_df(df_data)

Return type:

OutputDict

download_from_peeringdb(url, token, base_output_table_name)

Download data from PeeringDB endpoint

:param

# need to include depth= 2 to expand sets into objects # next / previous - pagination - while next is not None # count is total count? maybe

Return type:

List[OutputDict]

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

fetch_data(url, headers)

Fetch data without pagination when depth is not needed

Return type:

list

fetch_paginated_data(url, headers, total_count)

Fetch all data with pagination

Return type:

list

get_api_token()

Return type:

str

get_headers(token)

Set up API headers

Return type:

dict

get_total_count(url, headers)

Get total count of records for pagination upon depth expansion

Return type:

int

process_ix_data(ix_data)

Process IX data to extract both IXPs and facilities with US presence :type ix_data: list :param ix_data: Raw IX data from API :rtype: List[OutputDict] :return: List[OutputDict] with ‘ixps’ and ‘facilities’ DataFrames

write_to_athena(df, output_table_name, output_path, table_prefix)

Write a dataframe to Athena

Parameters:

• df (pd.DataFrame) – the DataFrame to write

• output_table_name (str) – name of the output table in Athena. “r_” prefix is added to the table name.

• output_path (str) – the path to the output table in S3

Return type:

None

StartReflectRunOperator

This operator is used to start a Reflect run test.

class operators.start_reflect_run_test.StartReflectRunOperator(test_id, base_url='https://api.reflect.run/v1/', *args, **kwargs)

Custom operator to start a reflect run in the specified environment. Uses requests.

Parameters:

test_id (str) – The ID of the test to run.

execute(test_id, context)

Execute the operator.

Parameters:

• test_id (str) – The ID of the test to run.

• context (Context) – The context of the task instance.

start_test(test_id)

Start a reflect test run.

Parameters:

test_id (str) – The ID of the test to run.

ui_color: str = '#FFFFFF'

validate_test_id(test_id)

Validate the test ID format.

Parameters:

test_id (str) – The ID of the test to run.

Returns:

True if valid, False otherwise.

Return type:

bool

TransformPostgresOperator

This operator is used to transform data in Postgres.

class operators.transform_postgres.TransformPostgresOperator(source_conn_id, dest_conn_id, dest_schema_name, dest_table_name, truncate=False, source_sql='', if_table_exists='replace', query_limit=0, *args, **kwargs)

Transform data from one Postgres connection to another Postgres Connection. :type source_conn_id: :param source_conn_id: the source Postgres connection ID :type source_conn_id: str :type source_sql: :param source_sql: the source SQL statement. Should return the data to be

saved to the destination. (templated)

Parameters:

• dest_conn_id (str) – the destination Postgres connection ID

• dest_schema_name (str) – the name of the destination schema

• dest_table_name (str) – the name of the destination table

• truncate (bool) – whether to truncate the destination table

• if_table_exists (str) – what to do if destination table exists. Can be fail, append, replace. Default fail.

• query_limit (int) – how many rows to pass in each chunk

ui_color: str = '#f542d7'

GenerateReadmeOperator

This operator is used to generate README files and export to Google Drive.

class operators.cya.generate_readme.export.GenerateReadmeOperator(*, google_drive_id=None, s3_path=None, client_file_name, overwrite=False, **kwargs)

Generate a README file from template and upload to Google Drive or S3.

The Jinja2 template placeholders {{ timestamp }} and {{ client_file }} are replaced with the logical execution date and the provided client file name.

Exactly one of google_drive_id or s3_path must be provided.

Parameters:

• google_drive_id (Optional[str]) – The Google Drive folder ID to upload to.

• s3_path (Optional[str]) – The S3 path to upload to (e.g., “s3://bucket/prefix/”).

• client_file_name (str) – The name of the client file this README accompanies.

• overwrite (bool) – If True, overwrite existing file with same name. If False, create new file.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

dict

template_fields: Sequence[str] = ('google_drive_id', 'gcp_conn_id', 'client_file_name', 's3_path')

ui_color: str = '#A6E6A6'

SafeSQLExecutorOperator

This operator executes SQL queries with safety validation.

class operators.bedrock.safe_sql_executor.SafeSQLExecutorOperator(sql, database, max_limit=1000, default_limit=100, dangerous_patterns=None, return_format='records', require_limit=True, *args, **kwargs)

Execute SQL query with safety validation.

This operator validates SQL queries before execution to prevent dangerous operations (DDL/DML) and enforces query limits.

Parameters:

• sql (str) – SQL query to execute (can be templated)

• database (str) – Athena database name

• max_limit (int) – Maximum allowed LIMIT value (default 1000)

• default_limit (int) – Default LIMIT if none specified (default 100)

• dangerous_patterns (Optional[List[Tuple[str, str]]]) – Custom SQL patterns to block (optional)

• return_format (str) – Output format - ‘dataframe’, ‘dict’, or ‘records’ (default)

• require_limit (bool) – Whether to require a LIMIT clause (default True)

execute(context)

Execute the SQL query with validation.

template_fields: Sequence[str] = ['sql', 'database']

ui_color: str = '#87CEEB'

CQACollectGeometryOperator

This operator invokes the CostQuest collect geometry API.

class operators.cost_quest.cqa_collect_geometry.CQACollectGeometryOperator(vintage, fields, output_athena_dataset_name, input_sql, skip_cqa_uuid_sql='', datalake_db='shoreline', *args, **kwargs)

collect_cqa(geojson)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_api_token()

get_bulk_locations(id, cqa_location_list)

get_error_df(id, len_chunk)

is_table_available()

split_into_chunks(array, chunk_size=1000)

ui_color: str = '#A6E6A6'

EsriToAthenaOperator

This operator converts an ESRI endpoint to an Athena table.

class operators.esri_to_athena.EsriToAthenaOperator(esri_endpoint, output_table_name, output_path, fields=None, where=None, limit=None, *args, **kwargs)

To convert an ESRI endpoint to Athena table

Parameters:

• sri_endpoint – the endpoint of the ESRI service

• fields (list, optional) – list of fields to return, defaults to None which loads all fields by passing * to the API

• where (str, optional) – filter criteria, defaults to None which means no filter and it sends 1=1 to the API

• limit (int, optional) – max number of records to get, defaults to all the values in the feature layer

• output_table_name (str) – the name to use for the output table

• output_path (str) – the path to the output table in S3

download_from_esri(url, fields=None, where=None, limit=None, **kwargs)

Download data from an ESRI endpoint

Parameters:

• url (str) – the endpoint of the ESRI service

• fields (list, optional) – list of fields to return, defaults to None which loads all fields by passing * to the API

• where (str, optional) – filter criteria, defaults to None which means no filter and it sends 1=1 to the API

• limit (int, optional) – max number of records to get, defaults to all the values in the feature layer

Returns:

A pandas DataFrame with the data

Return type:

pd.DataFrame

sample url: https://pcrdgis.agriculture.purdue.edu/arcgis/rest/services/DDI/Digital_Divide_Index_COUNTY_USA_2020/MapServer/0

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

None

ui_color: str = '#006400'

write_to_athena(gdf, output_table_name, output_path)

Write a GeoDataFrame to Athena

Parameters:

• gdf (pd.DataFrame) – the GeoDataFrame to write

• output_table_name (str) – name of the output table in Athena. “r_” prefix is added to the table name.

• output_path (str) – the path to the output table in S3

Return type:

None

ExportQueryOperator

This operator exports a query result to a flat file format and uploads it.

class operators.flat_file_exporter.export_query.ExportQueryOperator(sql_query, database, filename, file_format='csv', destination='default', gdrive_url='https://drive.google.com/drive/u/0/folders/1rNzWD6BOyUECJHWQLckoL14tmfvOPPrA', state='XY', extra_tenants=None, json_orient='records', geometry_config={}, parquet_compression='snappy', partition_column=None, partition_drop_column=True, *args, **kwargs)

Takes a query and exports the result to a specified flat file format (CSV, Parquet, etc.) and uploads it to a given destination (S3, GDrive, etc.).

Parameters:

sql_query (str) – The SQL query to execute in string form.

If you’re executing against Redshift or Postgres, ensure that the query has schema-qualified table names. Ex: ‘SELECT * FROM california.pit_address_metadata’ is valid, while ‘SELECT * FROM pit_address_metadata’ is not since the schema is not being passed as a parameter.

Parameters:

database (str) – Which database to execute the query in.

Valid options:

• shoreline

• readydatalake

• redshift_dw_(dev/prod)

• postgres_bossplatform_grapi_(dev/stage/prod)

• postgres_bossplatform_game_(dev/stage/prod)

Parameters:

• filename (str) – The name of the file to create (without extension).

• file_format (str) – The format of the file to export.

Valid types: csv, parquet, json, xlsx, geojson, shp, kml, kmz

Parameters:

destination (str) – Where to send the file once it has been created.

Valid options: default, gdrive, public_bucket_dev, public_bucket_stage, public_bucket_prod The default option looks up the preferred destination for the given state (requires that parameter) Choosing an unreachable environment bucket from the executing environment will result in an error.

Parameters:

gdrive_url (str) – The Google Drive folder URL to upload the file to (only necessary for GDrive).

By default, uses the flat file exporter folder in the File Transfer GDrive.

Parameters:

• state (str) – The 2 letter state abbreviation of where the file is going (only necessary for GDrive).

• extra_tenants (Optional[list[str]]) – Optional non-state tenant codes (e.g. [“PUBLIC”]) to accept as a valid state

in addition to Config.STATE_NAMES. Widens the allow-list only; use for tenants that intentionally live outside the shared state map.

Parameters:

json_orient (str) – If exporting to JSON, the orient parameter to pass to pandas.DataFrame.to_json().

Valid options: split, records, index, columns, values, table

Parameters:

geometry_config (dict) – Configuration dictionary for geometry export formats (geojson, shp, kml, kmz).

There are three types of geometry config supported: 1. Point geometry: {‘type’: ‘point’, ‘lon_col’: ‘lon_column_name’, ‘lat_col’: ‘lat_column_name’} 2. WKT geometry: {‘type’: ‘wkt’, ‘geom_col’: ‘wkt_column_name’} 3. Binary geometry: {‘type’: ‘binary’, ‘geom_col’: ‘geometry_column_name’} Also honored for parquet: when set, the file is written as GeoParquet.

Parameters:

parquet_compression (str) – Compression algorithm for parquet files.

Valid options: snappy, gzip, brotli, zstd, None Default is ‘snappy’. Use ‘zstd’ for better compression ratio.

convert_csv(df)

Converts the given DataFrame to CSV format and returns the temporary file object.

convert_geojson(df)

Converts the given DataFrame to GeoJSON format and returns the temporary file object.

convert_json(df, orient)

Converts the given DataFrame to JSON format and returns the temporary file object.

convert_kml(df)

Converts the given DataFrame to KML format and returns the temporary file object.

convert_kmz(df)

Converts the given DataFrame to KMZ format and returns the temporary file object.

convert_parquet(df)

Converts the given DataFrame to Parquet format and returns the temporary file object. Uses configurable compression (default: snappy, recommended: zstd for better ratio). When geometry_config is set, the frame is converted to a GeoDataFrame first and written as GeoParquet (geometry column + geo metadata, EPSG:4326).

convert_shp(df)

Converts the given DataFrame to Shapefile format and returns the temporary file object.

convert_xlsx(df, sheet_name)

Converts the given DataFrame to XLSX format and returns the temporary file object.

create_geo_dataframe(df)

Converts the given DataFrame to a GeoDataFrame based on the geometry configuration and returns it.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_via_athena(query, database)

Executes the given SQL query against the specified Athena database and returns the result as a DataFrame.

execute_via_postgres(query, database)

Executes the given SQL query against the specified Postgres database and returns the result as a DataFrame.

execute_via_redshift(query, database)

Executes the given SQL query against the specified Redshift database and returns the result as a DataFrame.

export_gdrive(local_file_path)

Exports the given local file to Google Drive and returns the file URL.

Return type:

str

export_s3(local_file_path)

Exports the given local file to S3 and returns a presigned URL.

Return type:

str

get_export_extension()

Returns the file extension for the export. Shapefiles are zipped, so return ‘zip’ for shp.

Return type:

str

get_valid_databases()

Returns a list of valid databases that can be used for executing against.

Return type:

list[str]

get_valid_destinations()

Returns a list of valid destinations the export can go based on the environment.

Return type:

list[str]

template_fields: Sequence[str] = ('sql_query', 'filename', 'file_format', 'destination', 'gdrive_url', 'state', 'json_orient', 'parquet_compression', 'partition_column', 'partition_drop_column')

ui_color: str = '#420a69'

validate_geometry_config()

Validates the geometry configuration dictionary. Throws an exception if invalid.

Return type:

None

QueryToGDriveCSVOperator

This operator exports a query result to CSV and uploads it to Google Drive.

class operators.flat_file_exporter.query_to_gdrive_csv.QueryToGDriveCSVOperator(source_postgres_db, source_datalake_db, source_sql, custom_columns, target_file_name, gdrive_parent_folder_id, pg_flag=False, gdrive_overwrite_existing=False, *args, **kwargs)

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

ui_color: str = '#420a69'

FTPSToS3Operator

This operator transfers files from FTPS/SFTP to S3.

class operators.ftps_to_s3.FTPSToS3Operator(secret_name, remote_path, file_pattern, s3_bucket, s3_prefix, delete_after_download=False, site_commands=None, use_mainframe_dataset=False, protocol=None, **kwargs)

Download files from FTPS/SFTP server to S3.

Supports: - FTPS (FTP over TLS) - required for MN SWIFT mainframe - SFTP (SSH FTP) - may be used by other states

Credentials retrieved from AWS Secrets Manager.

Parameters:

• secret_name (str) – AWS Secrets Manager secret name containing FTPS credentials

• remote_path (str) – Remote directory path on FTPS server

• file_pattern (str) – Glob pattern to match files (e.g., “.IF0411.”)

• s3_bucket (str) – Destination S3 bucket

• s3_prefix (str) – Destination S3 prefix (path within bucket)

• delete_after_download (bool) – If True, delete remote file after successful download

XCom outputs: - downloaded_files: List of dicts with file info (name, s3_key, size) - download_count: Number of files downloaded - download_timestamp: ISO timestamp when download completed

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

template_fields: Sequence[str] = ['remote_path', 'file_pattern', 's3_bucket', 's3_prefix', 'secret_name', 'site_commands', 'use_mainframe_dataset', 'protocol']

ui_color: str = '#5bc0de'

GDriveCreateFolderOperator

This operator creates a folder in Google Drive.

class operators.gdrive.gdrive_folder.GDriveCreateFolderOperator(*, folder_name, google_drive_id, **kwargs)

Create a folder in Google Drive (or return existing if it already exists).

Parameters:

• folder_name (str) – Name of the folder to create

• google_drive_id (str) – Parent folder ID (inherited from base class)

Returns dict with keys: id, name, webViewLink

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

dict

template_fields: Sequence[str] = ('google_drive_id', 'gcp_conn_id', 'folder_name')

ui_color: str = '#A6E6A6'

GDriveUploadFileOperator

This operator uploads a file to Google Drive.

class operators.gdrive.gdrive_upload.GDriveUploadFileOperator(*, google_drive_id, file_name, file_path=None, file_content=None, mimetype=None, overwrite=True, **kwargs)

Upload a file to a Google Drive folder.

Supports uploading from either a local file path or in-memory string content. Can optionally overwrite existing files with the same name.

Parameters:

• google_drive_id (str) – Target Google Drive folder ID

• file_path (Optional[str]) – Local file path to upload (mutually exclusive with file_content)

• file_name (str) – Name for the uploaded file in Google Drive

• file_content (Optional[str]) – String content to upload (mutually exclusive with file_path)

• mimetype (Optional[str]) – MIME type of the file (default: auto-detect or text/plain)

• overwrite (bool) – If True, replace existing file with same name (default: True)

Returns dict with keys: id, name, webViewLink

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

dict

template_fields: Sequence[str] = ('google_drive_id', 'gcp_conn_id', 'file_path', 'file_name', 'file_content')

ui_color: str = '#A6E6A6'

S3ToRedshiftOperator

This operator ingests Parquet files from S3 into Redshift.

class operators.redshift.s3_to_redshift.S3ToRedshiftOperator(*, s3_path, rs_workgroup, rs_database, target_schema, target_table, redshift_iam_role=None, column_mapping=None, geometry_type=None, geometry_config=None, reproject_srid=4326, h3_index_resolution=9, write_mode='append', region='us-east-1', aws_conn_id='aws_default', case_sensitive_identifiers=True, **kwargs)

Ingest Parquet files from S3 into Redshift.

Supports: - Column mapping (rename columns during ingestion) - Geometry creation from lat/lon or WKT columns - Append or overwrite write modes

Parameters:

• s3_path (str) – S3 path to parquet files (e.g., s3://bucket/prefix/)

• rs_workgroup (str) – Redshift Serverless workgroup name

• rs_database (str) – Redshift database name

• target_schema (str) – Target schema in Redshift

• target_table (str) – Target table name in Redshift

• redshift_iam_role (Optional[str]) – IAM role ARN for Redshift to access S3

• column_mapping (Optional[Dict[str, str]]) – Optional dict mapping source columns to target columns Example: {“src_col”: “tgt_col”, “old_name”: “new_name”}

• geometry_type (Optional[Literal['latlon', 'wkt']]) – Type of geometry to create: “latlon” or “wkt”

• geometry_config (Optional[Dict[str, str]]) – Config for geometry creation: - For latlon: {“lat_column”: “lat”, “lon_column”: “lon”, “geometry_column”: “geom”, “srid”: 4326} - For wkt: {“wkt_column”: “wkt”, “geometry_column”: “geom”, “srid”: 4326}

• write_mode (Literal['append', 'overwrite']) – “append” to add rows, “overwrite” to replace table

• region (str) – AWS region

• aws_conn_id (str) – Airflow AWS connection ID

• case_sensitive_identifiers (Union[str, bool]) – Enable case-sensitive identifiers in Redshift

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_spectrum_schema()

Return type:

str

ingest_to_athena()

template_fields: Sequence[str] = ('s3_path', 'rs_workgroup', 'rs_database', 'target_schema', 'target_table', 'redshift_iam_role', 'column_mapping', 'geometry_type', 'geometry_config', 'write_mode', 'region', 'aws_conn_id')

ui_color: str = '#E87C3E'

DisableSupersetUserOperator

This operator disables a Superset user (sets active=False) via the Superset REST API.

class operators.superset.disable_superset_user.DisableSupersetUserOperator(username, conn_id='superset_default', base_url=None, timeout=30, *args, **kwargs)

Disable a Superset user (sets active=False) via the Superset REST API.

Parameters:

• username (str) – The Superset username to disable.

• conn_id (str) – Airflow connection id pointing at the Superset instance. Defaults to superset_default.

• base_url (str) – Optional full URL override for the Superset API (e.g. "https://dap-tx.bead.example.com"). When provided, overrides the connection’s host/schema.

• timeout (int) – HTTP request timeout in seconds.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ('username', 'conn_id', 'base_url')

ui_color: str = '#fce4ec'

AddSupersetDbConnectionOperator

This operator creates a new database connection in Superset via the REST API.

class operators.superset.add_superset_db_connection.AddSupersetDbConnectionOperator(database_name, sqlalchemy_uri, expose_in_sqllab=True, allow_ctas=False, allow_cvas=False, allow_dml=False, extra=None, conn_id='superset_default', base_url=None, timeout=30, owner_email=None, *args, **kwargs)

Create a new database connection in Superset via the REST API.

Parameters:

• database_name (str) – Display label shown in the Superset UI. This is NOT the actual database name connected to — the real DB is determined entirely by the sqlalchemy_uri. Free-form; spaces allowed.

• sqlalchemy_uri (str) – SQLAlchemy URI for the database connection.

• expose_in_sqllab (bool) – Whether to expose the database in SQL Lab.

• allow_ctas (bool) – Whether to allow CTAS statements.

• allow_cvas (bool) – Whether to allow CVAS statements.

• allow_dml (bool) – Whether to allow DML statements.

• extra (dict) – Optional dict merged into the Superset payload (e.g. {"masked_encrypted_extra": "..."} or {"extra": "..."}).

• conn_id (str) – Airflow connection id for the Superset instance.

• base_url (str) – Optional full URL override for the Superset API (e.g. "https://dap-tx.bead.example.com"). When provided, overrides the connection’s host/schema.

• timeout (int) – HTTP request timeout in seconds.

• owner_email (str) – Optional canonical Ready email (e.g. majid@ready.net). When set, resolves to a single Superset user and adds them to the DB’s owner list alongside the API user. DAP stores emails with a per-tenant +<uuid> suffix; this resolver matches both shapes. Important when the API user is going to be disabled later in the run — the resolved end-user is what keeps DAP’s owner-based visibility filter from hiding the DB from listings post-disable.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ('database_name', 'sqlalchemy_uri', 'extra', 'conn_id', 'base_url', 'owner_email')

ui_color: str = '#fce4ec'

AddSupersetDbConnectionFromSecretOperator

This operator creates a new Superset database connection using an AWS Secrets Manager secret. The SQLAlchemy URI is built inside execute() so the password never appears in XCom.

class operators.superset.add_superset_db_connection_from_secret.AddSupersetDbConnectionFromSecretOperator(database_name, secret_name, dialect='redshift+psycopg2', default_port=5439, expose_in_sqllab=True, allow_ctas=False, allow_cvas=False, allow_dml=False, extra=None, conn_id='dap_superset_user', base_url=None, timeout=30, owner_email=None, *args, **kwargs)

Create a Superset DB connection using an AWS Secrets Manager secret.

The SQLAlchemy URI is built inside execute() so the password never appears in XCom or task return values.

Expected secret JSON fields:

username, password, host, database (or dbname). port - optional, defaults to 5439.

Parameters:

• database_name (str) – Display label shown in the Superset UI. This is NOT the actual database name connected to — the real DB name comes from the secret’s database (or dbname) field. Free-form; spaces allowed.

• secret_name (str) – AWS Secrets Manager secret id or ARN.

• dialect (str) – SQLAlchemy dialect prefix (default redshift+psycopg2).

• default_port (int) – Port to use when the secret omits one. Defaults to 5439.

• expose_in_sqllab (bool) – Whether to expose the database in SQL Lab.

• allow_ctas (bool) – Whether to allow CTAS statements.

• allow_cvas (bool) – Whether to allow CVAS statements.

• allow_dml (bool) – Whether to allow DML statements.

• extra (dict) – Optional dict merged into the Superset payload.

• conn_id (str) – Airflow connection id for the Superset instance.

• timeout (int) – HTTP request timeout in seconds.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ('database_name', 'secret_name', 'extra', 'conn_id', 'base_url', 'owner_email')

ui_color: str = '#fce4ec'

AddSupersetDashboardOperator

This operator creates a dataset and one or more dashboards in Superset via the REST API.

class operators.superset.add_superset_dashboard.AddSupersetDashboardOperator(dataset, dashboards, conn_id='superset_default', base_url=None, timeout=30, *args, **kwargs)

Create a dataset and one or more dashboards in Superset.

Order of operations:

1. Create the dataset described in dataset.

2. For each entry in dashboards, create a dashboard.

Returns a dict {"dataset_id": <int>, "dashboard_ids": [<int>, ...]}.

Parameters:

• dataset (dict) – Dict describing the dataset to create. Required keys: database_id and table_name. Optional: schema, owners, extra (merged into the Superset payload).

• dashboards (list[dict]) – List of dicts describing dashboards to create. Each dict requires dashboard_title. Optional: slug, owners, published (default False), extra (merged into the payload).

• conn_id (str) – Airflow connection id for the Superset instance.

• base_url (str) – Optional full URL override for the Superset API (e.g. "https://dap-tx.bead.example.com"). When provided, overrides the connection’s host/schema.

• timeout (int) – HTTP request timeout in seconds.

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Return type:

Dict[str, Any]

template_fields: Sequence[str] = ('dataset', 'dashboards', 'conn_id', 'base_url')

ui_color: str = '#fce4ec'

Utilities

The following are central utility classes that are used throughout the project. They are organized by the type of utility they provide. In general, Util classes should be created if they contain central logic that is used in multiple places. If the logic is only used in one place, it should likely be kept in the DAG itself.

BaseUtil

This class contains central utilities that may be used for any type of pipeine. They are mainly used for central logic around requests, text cleaning, and other ubiquitous logic.

class util.base.BaseUtil

The BaseUtil contains central airflow utilities. These can be used in any dag, operator, hook, or util. Many of these functions are wrappers around external libraries, such as requests, or centralized entrypoints into wrangler and pandas.

static external_get_request(url, headers=None, ultra_premium=False, api_conn='scraperapi_proxy_key_colello')

Make get request to external scraper via scraperapi

Parameters:

• url (str) – the url to call

• headers (Optional[bool]) – the headers to pass to the request

• ultra_premium (bool) – whether or not to use ultra premium proxy

static get_abbrev_from_fips(fips)

Returns a list of default states, either fips codes or capitalized abbreviations.

Parameters:

fips (str) – the fips code that we want to convert to the abbreviation

Return type:

str

Returns:

abbreviation

static get_aws_secret(secret_id)

Get the secret from the aws secret manager

Parameters:

secret_id (str) – the secret id

Return type:

str

Returns:

the secret in string format

Raises:

ClientError

static get_default_states(fips=True)

Returns a list of default states, either fips codes or capitalized abbreviations.

Parameters:

fips (bool) – whether or not to return fips codes

Return type:

list[str]

Returns:

the list of states

static get_json_safely(response)

Get the JSON from the response, safely

Parameters:

response (_type_) – the response to get the JSON from

Raises:

ValueError – if there is an error in the JSON

Returns:

the JSON

Return type:

dict

static get_num_unnamed(df, divisor=1)

Get the number of unnamed columns in a dataframe

Parameters:

df – the dataframe to count

Return num_unnamed:

the number of unnamed columns

Return type:

int

static get_request(url, headers=None, params=None, proxies=None, verify=None, log=True, timeout=3600, retry_count=1, stream=False)

Layer over requests.get to centralize access to external library

Parameters:

• url (str) – the url to call

• headers (Optional[dict]) – the headers to pass to the request

• params (Optional[dict]) – the params to pass to the request

• proxies (Optional[dict]) – the proxies to pass to the request

• verify (Optional[bool]) – whether or not to verify the request

• log (bool) – whether or not to log the request

• timeout (int) – the timeout for the request

• retry_count (int) – the number of times to retry the request

Returns:

The response.

Return type:

requests.Response

static lob_address(street_address, city='', state='', zip_code='')

Send address to the cached lob api

Parameters:

• street_address (str) – the street address, required

• city (str) – the city, must be provided with state

• state (str) – the state

• zip_code (str) – the zip code, optionally include instead of city and state

Return type:

dict

static lob_address_bulk(input_df, sleep=0.03, max_retries=5)

Send address to the cached lob api

Parameters:

input_df – a pandas dataframe with following columns “id”,”city”,”primary_line”,”state”,”zip_code”

Return type:

dict

returns an array with following columns query, id, response, created_at

static normalize_df(df)

Normalize a dataframe’s column names

Parameters:

df – the dataframe to prep

Returns:

df

static normalize_string(input_string)

Normalize a string to be used as a column name

Parameters:

input_string (str) – the string to normalize

Return type:

str

Returns:

the normalized string

static post_request(url, headers=None, data=None, params=None, verify=None, log=False, timeout=3600, retry_count=1, retry_sleep=5, form_data=None)

Make a post request to an endpoint

Parameters:

• url (str) – the url to call

• headers (dict, optional) – the headers to pass to the request

• data (dict, optional) – the data to serialize as JSON in request body

• params (dict, optional) – the params to pass to the request

• verify (bool, optional) – whether or not to verify the request

• log (bool, optional) – whether or not to log the request

• timeout (int, optional, defaults to 60 * 60 * Config.REQUEST_TIMEOUT_HOURS) – the timeout for the request in seconds

• retry_count (int, optional) – the number of times to retry the request

• retry_sleep (int, optional) – the time

• form_data (dict, optional) – the form data to pass to the request

Returns:

The response.

Return type:

requests.Response

static prep_wrangler_df(df)

Prep a wrangler dataframe for raw ingestion

Parameters:

df – the dataframe to prep

Returns:

df

Return type:

[pandas.Dataframe]

static proxy_get_request(url, headers=None, premium=False, stealth=False, api_conn='scraperbee_proxy_key_colello', retry_count=1)

Make get request to via paid proxy via scraperbee

Parameters:

• url (str) – the url to call

• headers (Optional[dict]) – the headers to pass to the request

• premium (bool) – whether or not to use premium proxy

• stealth (bool) – whether or not to use stealth proxy

• api_conn (str) – the connection to use for the proxy

Returns:

The response.

DagUtil

This class contains dag-level utilities that may be used for any type of DAG. They are mainly used for help with initialization, error handling, and central DAG behavior.

class util.dag.DagUtil

Central dag utilities that apply to initialization, behavior of dags

static create_default_dag(constant_object, params={}, file_name='', email='', template='', tags=[], notify_slack=True, on_success_callback=None, on_failure_callback=None, dag_id='', interval=<object object>, start_date=None)

This interface returns a dag object

Parameters:

• constant_object – the constant object

• params (dict) – the params to pass to the dag

• file_name (str) – the file name

• email (str) – the email to send to

• template (str) – the template to use

• tags (list[str]) – the tags to use

• notify_slack (bool) – if False, suppress Slack alerts from notify_failure (default True)

• on_success_callback – DAG-level callback on successful run completion

• on_failure_callback – DAG-level callback on failed run completion

• dag_id (str) – explicit dag_id (defaults to the id auto-built from file_name)

• interval – explicit schedule (defaults to constant_object.INTERVAL; pass None for an unscheduled, manual-trigger-only DAG)

• start_date – explicit start_date (defaults to constant_object.START_DATE)

Returns:

the dag

static default_dag(constant_object, params={}, file_name='', email='', template='', tags=[], notify_slack=True, on_success_callback=None, on_failure_callback=None, dag_id='', interval=<object object>, start_date=None)

This is an interface over _dag that returns a decorator

Parameters:

• constant_object – the constant object

• params (dict) – the params to pass to the dag

• file_name (str) – the file name

• email (str) – the email to send to

• template (str) – the template to use

• tags (list[str]) – the tags to use

• notify_slack (bool) – if False, suppress Slack alerts from notify_failure (default True)

• on_success_callback – DAG-level callback on successful run completion

• on_failure_callback – DAG-level callback on failed run completion

• dag_id (str) – explicit dag_id (defaults to the id auto-built from file_name)

• interval – explicit schedule (defaults to constant_object.INTERVAL; pass None for an unscheduled, manual-trigger-only DAG)

• start_date – explicit start_date (defaults to constant_object.START_DATE)

Returns:

the dag

static email_attachments(email='', cc=[], subject_dag_id='', subject='', content='', task_id='email_attachments', files=[])

Configured to send dataframe as csv attachement in an email, but easy to drop in another operator Calls operator.execute() because this is required to be a callback function – https://gist.github.com/mholtzscher/9648cfd27769d1df6a6ed855fdd7bd7a

Parameters:

• email (str) – the email to send to

• cc (list[str]) – the cc to send to

• subject_dag_id (str) – the subject dag id

• subject (str) – the subject to use

• content (str) – the content to use

• task_id (str) – the task id

• files (list[str]) – the files to attach

Returns:

the email operator

static get_dag_id(constant_object, path='')

Get dag id from constant object

Parameters:

• constant_object – the constant object

• path (str) – the path to the file

Return type:

str

Returns:

the dag id

static is_prod_instance()

A utility that will return a boolean value indicating if the current envrionment is prod or not

Return type:

bool

Returns:

True if prod, False otherwise

static notify_success(email='', subject='', content='', task_id='success_notification', trigger_rule='all_success', slack=False)

Success notification task: email + DAG Run Note + optional Slack @mention. Returns a PythonOperator — place in the DAG graph like any other task. Does not need to call .execute().

Parameters:

• email (str) – the email to send to

• subject (str) – the subject to use

• content (str) – the content to use

• task_id (str) – the task id

• trigger_rule (str) – the trigger rule

• slack (bool) – if True, also post to Config.SLACK_ALERTS_CHANNEL (default False)

Returns:

PythonOperator

DagTriggerCallbacks

Completion callbacks for target DAGs triggered by the DAG Trigger Scheduler. Reports success/failure back to data.dag_trigger_run in Redshift.

DAG Trigger Scheduler completion callbacks.

Provides on_success_callback / on_failure_callback functions that target DAGs use to report completion status back to airflow.dag_trigger_run in Redshift.

Usage in a target DAG:

from util.dag_trigger import scheduler_callbacks

@Util.default_dag(

Config, file_name=__file__, email=EMAIL, on_success_callback=scheduler_callbacks.on_success, on_failure_callback=scheduler_callbacks.on_failure,

) def my_dag():

…

Runs triggered manually (without _scheduler_run_id in conf) are ignored.

class util.dag_trigger.scheduler_callbacks

static on_failure()

Return type:

None

static on_success()

Return type:

None

DataLakeUtil

This class contains utilities for interacting with the data lake.

class util.data_lake.DataLakeUtil

Athena and glue utilities

static apply_column_mapping(df, column_mapping)

Apply column mapping to a DataFrame in-place. For each target column, finds the first matching alias in the DataFrame and renames it to the target name. Validates during application that no alias appears in multiple target lists.

Return type:

None

Args:

df: pandas DataFrame (optional, pass None to validate only) column_mapping: dict where keys are target column names and values are

alias(es) to look for. Example: {“md5_pkey_hash”: [“md5”, “md5_pkey_hash”, “md5hash”]}

Raises:

ValueError: if the same alias appears in multiple target lists

static create_and_start_crawler(crawler_name, tgt_path, table_prefix='', tgt_db='shoreline', tgt_bucket='463002504035-shoreline', wait=True, max_iterations=50000, sleep_time=5, append_dev=True)

Create and start a glue crawler

Parameters:

• crawler_name (str) – the crawler name

• tgt_path (str) – the target path

• table_prefix (str) – the table prefix for boto client.create_crawler to catalog tables that are created i.e. all external tables stored at tgt_path that have this names starting with table_prefix will be crawled (scanned)

• tgt_db (str) – the target db

• tgt_bucket (str) – the target bucket

• wait (bool) – whether or not to wait

• max_iterations (int) – the max iterations

• sleep_time (int) – the sleep time

• append_dev (bool) – whether or not to append dev to the crawler name

static ctas(prefix, query, tgt_table, partition_by=[], bucket='463002504035-shoreline', src_db='shoreline', tgt_db='shoreline', workgroup='airflow-dev', clean=True, data_source='AwsDataCatalog', bucketing_info=None)

[Deprecated] This method is deprecated and will be removed in a future release. Should use new AthenaCTASOperator instead! Create a table as select. Uses awswrangler.athena.create_ctas_table under the hood. See [docs](https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.athena.create_ctas_table.html#awswrangler.athena.create_ctas_table) for a more in depth explanation of the parameters. Uses sane defaults for data lake

Parameters:

• prefix (str) – the prefix

• query (str) – the query

• tgt_table (str) – the target table

• partition_by (list[str]) – the partitions

• bucket (str) – the bucket

• tgt_db (str) – the target db

• workgroup (str) – the workgroup

• clean (bool) – whether or not to clean

static drop_athena_partition(partition_key, partition_val, parent_tbname)

Drop partition (single partitioning scheme) if EXISTS - both metadata and data

static get_athena_columns_defs(table_name, lake_db='shoreline')

Returns columns_types (dictionary of non-partition cols and dtypes), partitions_types (separate partition cols and dtypes) :rtype tuple[dict[str, str], [dict[str, str]]]: a tuple of size 2: each item is a dictionary of columns and their athena dtypes e.g. {‘state’: ‘varchar(2)’, ‘volume’: ‘int’, ‘fa_ver’: ‘int’, ‘stage’: ‘int’} This is a working replacement that rfor unreliable read_parquet_metadata https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.s3.read_parquet_metadata.html. Cons of read_parquet_metadata are:

Unwieldy params: ignore_null, dtype, sampling, … It failed to recognize partition cols (partitions_types=None!). Examples:

table c_lsd: sampling=1.0 tries to read entire gigantic table: SLOW; table r_cai_sbo: due to NULL values in partition cols? “Field expected_size has incompatible types: int32 vs int64”

static get_athena_table_size(table_name, db_name='shoreline')

static get_table_path_info(tgt_table, tgt_db='shoreline')

Find the full S3 path, bucket, and prefix of a table in the Data Lake. If table does not exists, return None

static get_valid_fabric_table_from_version(version_number, tgt_db='shoreline')

Checks if the version_number is a valid number. It converts that to the table name and validates it

version_number: int a number. 0 means the latest version of fabric (v3).

Parameters:

• version_number (int) – int a number. 0 means the latest version of fabric (v3).

• tgt_db (str) – the target db

Raises:

AirflowFailException – if table name is invalid/does not exist

Return type:

Optional[str]

Returns:

fabric fix availabilty table name

static list_states_with_cpc(tgt_database='shoreline')

Check the datalake and returns the list of the states that their c_cpc_merge_tile_ is generated. This function ignores states which have sub city lie mi_detroit

Parameters:

tgt_database (str, optional) – the datalake table, defaults to Config.DATA_LAKE_DB

Returns:

an array of state_abbr

Return type:

list[str]

static msck_repair_table(table_name, database='shoreline')

Helper function to run MSCK REPAIR TABLE on an Athena table

Parameters:

• table_name (str) – the table name

• lake_db (str) – the database name

Raises:

Exception – if the table does not exist

static read_multiple_queries(queries, database='shoreline', max_concurrent=None)

Executes multiple queries simultaneously and returns the results as a list of dataframes

Parameters:

• queries (list[str]) – the queries to run

• database – the database to run the queries on

• max_concurrent (Optional[int]) – the maximum number of queries to run simultaneously.

Defaults to 50 if all queries are only DML and 10 otherwise

Return type:

list

Returns:

the results as a list of dataframes

static remove_datalake_table(tgt_table, tgt_prefix='', tgt_db='shoreline', tgt_bucket='463002504035-shoreline')

[This method is deprecated and will be removed in a future release.] Should use new DeleteAthenaTableOperator! Drop Athena table from Glue catalog (metadata) and also delete all S3 data, if table exists

Parameters:

• tgt_table (str) – the target table

• tgt_prefix (str) – the target prefix: OPTIONAL, if provided, will delete the specified prefix instead of inferring from table catalog (and even when the catalog table is missing — clears orphans from failed CTAS runs)

• tgt_db (str) – the target db

• tgt_bucket (str) – the target bucket: used only when the catalog table is missing and tgt_prefix is provided; otherwise inferred from the catalog

static replicate_athena_table(old_name, new_name, lake_db='shoreline', new_description=None, new_columns_comments=None, new_s3_path='')

Moves an athena table to new name, with new descriptions;

Perform COPY DDL if new_s3_path is specified: then a new table will be created with empty data at new location; old table is not affected! Perform MOVE TABLE if new_s3_path is not specified: new table will have exactly same data as old table (and old table will be dropped from Glue catalog!)

It is essentially a wrapper around awswrangler function wr create_parquet_table.

param: old_name: name of old existing table param: new_name: name of new table to create param: lake_db: aws database param: new_description: must include if want to include these comments in the new table param: new_columns_comments: must include if want to include these comments in the new table param: new_s3_path: EXTERNAL LOCATION for the NEW and EMPTY table; Source inspiration: https://stackoverflow.com/a/74394685/12572612

static s3_file_to_parquet(s3_file_path, raw_table_prefix, raw_table_name, chunk_size=1000000, tgt_bucket='463002504035-shoreline', tgt_db='shoreline', partition_cols=[], column_mapping=None, drop_empty_rows=False, dtypes=None)

Load the raw data from a s3 path and store it in the datalake as parquet file. Use this function when you do not know what is expected from the client shared data. All fields are forced to be string by default.

Parameters:

• s3_file_path (str) – the s3 path of the file

• raw_table_prefix (str) – the prefix of the table. Example is “raw/uploads/Ohio/unzipped/data_revision_requests/{raw_table_name}”

• raw_table_name (str) – the name of the table

• tgt_bucket (str, optional) – the target bucket, defaults to Config.DATA_LAKE_BUCKET

• tgt_db (str, optional) – the target db, defaults to Config.DATA_LAKE_DB

• column_mapping (dict, optional) – dict where keys are target column names and values are alias(es) to look for. Example: {“md5_pkey_hash”: [“md5”, “md5hash”]}

• dtypes (dict, optional) – opt-in map of column name -> Athena type (e.g. “double”, “bigint”, “date”, “timestamp”) for columns that should be typed instead of left as string. Keys must use the normalized (lowercased) column names. Defaults to None, which keeps every column a string (the historical behavior — all other callers are unaffected).

• mode – the mode of the write, defaults to “overwrite”, options are “overwrite”, “append”, “upsert”

Return type:

None

static update_glue_table_metadata(table_name, comments={}, table_description=None, database_name='shoreline')

adds comments to a glue database table. you can use this when creating tables with CTAS It breaks if the table_name does not exist. Make sure to use this for the tables you create

Parameters:

• database_name (str) – the name of the database

• table_name (str) – the table name

• comments (dict[str,str]) – a dict to include comments in format of {“column_name”:”column comment”}

DBUtil

This class contains utilities for interacting with the database.

class util.db.DBUtil

Central database utilities

static add_series_column(conn_id, table, column='id')

Adds a series column to a table

Parameters:

• conn_id (str) – the connection id

• table (str) – the table to add the column to

• column (str) – the column name

Returns:

the postgres operator that adds the column

static boss_db_param_values()

list BOSS DB parameters

Returns:

the dict of BOSS DB parameters

Return type:

dict[str, str]

static generate_case_from_dict(tgt_field, case_dict)

Generates a case statement from a dictionary

Parameters:

• tgt_field (str) – the target field

• case_dict (dict) – the dictionary to use

Return type:

str

Returns:

the case statement

static get_all_pg_conn_values()

Get all the pg connections to use in the dag params as values_display

static get_athena_cross_dbs()

Get all the Athena cross database list. Used in the dag params

static get_athena_grapi_pg_aliases()

Get all the Athena grapi PG aliases.

static get_boss_platform_connections()

Get boss_platform database connections

static get_game_db_connections()

Get GAME database connections

static get_pg_columns(table_name, schema='public', database_name='postgres_readydata_dev')

Returns an array of the columns of a table in pg

Parameters:

• table_name (str) – name of the table

• schema (str) – table schema

• database_name (str) – database name

Returns:

an array of strings or empty array if table does not exist

Return type:

_type_

static get_tenant_schemas(conn_id='postgres_bossplatform_grapi_dev', include_public=False)

!WARNING!: This function makes a database call. Do not use it outside of a single task context. If you call this when constructing your DAG structure or params, the schedule will run this query every minute.

Query grapi for all tenant schema names, excludes public by default as that maps to the Ready tenant

Parameters:

• conn_id (str) – the connection id

• include_public (bool) – whether to include the public schema

Return type:

list[tuple[str]]

Returns:

the list of tenant schema names

static get_tileserver_environments()

Get all the tileserver environments. Used in the dag params

Example:

```python env_list = DBUtil.get_tileserver_environments()

Param(

default=list(env_list)[0], type=”string”, title=”Destination Tileserver”, section=”Deployment params”, description=”Select Destination environment from a list”, enum=list(env_list.keys()), values_display=env_list,

)

```

static pg_swap_table(db_hook, stage_table, table, postfix='_old')

Within the public schema, swaps a stage table name for a production table, then drops the old table. Please note that this is a destructive operation. It is assumed that indexes and data have been transferred by the time this is called.

Parameters:

• db_hook (_type_) – the postgres hook

• stage_table (str) – the name of the stage table that will become production

• table (str) – the name of the production table

• postfix (str) – the postfix to add to the production table that was replaced

Return type:

bool

Returns:

True if both queries execute successfully

static pg_swap_with_partitions(db_hook, stage_table, table, postfix='_old', schema='public', drop_backup_at_end=False, drop_backup_before_start=False)

This is similar to the pg_swap_table but with experimental function to support swapping tables with partitions

Highly Experimental. better to use pg_swap_table TODO: After enough testing combine these two functions into one This should be an util. we already have one util similar to this but need to add partition name support to it for now I kept it here to test it more and make it work with tables without partitions too and combine it with the existing util

Parameters:

• db_hook – the postgres hook

• stage_table (str) – the name of the stage table that will become production. it Must have _stage at the end of its table name and partitions

• table (str) – the name of the production table

• postfix (str) – the postfix to add to the production table that was replaced

• drop_backup_at_end (bool) – it will drop the backup table at the end of the process

• drop_backup_before_start – it will drop {table}_{postfix} table before start the process

Returns:

True if both queries execute successfully

static pg_table_indexes(db_hook, schema, table_name)

gets a list of indexes on a table. it Includes all the indexes in the partitions too

Parameters:

• db_hook (_type_) – _description_

• schema (str) – _description_

• table_name (str) – _description_

• suffix (str, optional) – _description_, defaults to ‘_old’

static pg_table_partitions(db_hook, schema, table_name)

gets a list of the partitions of a table. returns empty list if table does not exist

Parameters:

• db_hook (_type_) – pg db hook

• schema (str) – table schema

• table_name (str) – table name

Returns:

an array of the table names

Return type:

_type_

static reset_sequence(conn_id, table, column='id', seq_name=None)

Resets the sequence for a table

static run_env_db(query, prod_db='postgres_bossplatform_grapi_dev', stage_db='postgres_bossplatform_grapi_dev')

Helper to run queries against stage + prod or just dev

Parameters:

• query (str) – the query to run

• prod_db (str) – the prod db

• stage_db (str) – the stage db (only executes when running on prod)

static schema_exists(db_hook, schema)

Checks if a postgres schema exists

Parameters:

• db_hook – the postgres hook

• schema (str) – the schema to check

Return type:

bool

Returns:

True if the schema exists

static table_exists(db_hook, schema, table)

Checks if a postgres table exists

Parameters:

• db_hook – the postgres hook

• schema (str) – the schema to check

• table (str) – the table to check

Return type:

bool

Returns:

True if the table exists

static tenant_to_statefips(conn_id='postgres_bossplatform_grapi_dev')

!WARNING!: This function makes a database call. Do not use it outside of a single task context. If you call this when constructing your DAG structure or params, the schedule will run this query every minute. returns a dictionary of tenant schema name to state fips

Return type:

dict[str, str]

Returns:

The tenant name and schema mapping dict

static transfer_datalake_db(temp_key, table_name, tgt_query, tgt_bucket='463002504035-shoreline', data_lake_db='shoreline', workgroup='airflow-dev', batch_size=3000000, dest_db='postgres_readydata_dev', create_suffix='', drop_table=False, use_pg_copy=False, task_suffix='', on_conflict_sql=None, upsert_origin_table=None)

Transfers from the data lake to a postgres db Sequentially loads parquet files from s3 to postgres This probably could be an operator

Parameters:

• temp_key (str) – the temp key to store data

• table_name (str) – the table name in the destination db

• tgt_query (str) – the target query in athena

• tgt_bucket (str) – the target bucket

• data_lake_db (str) – the data lake db

• workgroup (str) – the workgroup

• batch_size (int) – the batch size

• dest_db (str) – the destination db

• create_suffix (str) – if drop_table, something to add at the end of the create table query

• drop_table (bool) – whether to drop the table before query runs

• use_pg_copy (bool) – whether to use pg copy (usually faster and more efficient)

• task_suffix (str) – the task suffix, large or small

Returns:

task arn

Return type:

_type_

static update_pg_table_metadata(data_product_name, table_name, comments={}, table_description=None, database_name='postgres_bossplatform_game_dev', schema='data', add_to_data_product_metadata=True)

adds comments to a pg database table. you can use this when creating tables with CTAS It breaks if the table_name does not exist. Make sure to use this for the tables you create

Parameters:

• data_product_name (str) – the data product name

• table_name (str) – the table name

• comments (dict[str,str]) – a dict to include comments in format of {“column_name”:”column comment”}

• table_description (str) – a description of the table

• database_name (str) – The database connection name in airflow

• schema (str) – The schema name

Returns:

None

Return type:

_type_

static upsert_df(df, table_name, engine, schema_name='public')

Implements the equivalent of pd.DataFrame.to_sql(…, if_exists=’update’) (which does not exist). Creates or updates the db records based on the dataframe records. Conflicts to determine update are based on the dataframes index. This will set unique keys constraint on the table equal to the index names 1. Create a temp table from the dataframe 2. Insert/update from temp table into table_name Returns: True if successful

Parameters:

• df (pd.DataFrame) – the dataframe to upsert

• table_name (str) – the table name

• engine (sqlalchemy.engine.base.Connection) – the sqlalchemy engine

• schema_name (str) – the schema name

Return type:

bool

Returns:

True if successful

ECSUtil

This class contains utilities for interacting with ECS.

class util.ecs.ECSUtil

ECS utilities for running, monitoring, and managing tasks

static check_ecs_status(task_arn, task_definition, cluster='dev', log_success=False, log_failure=True, return_logs=False, strict_error=True)

Check the status of a task on ECS

Parameters:

• task_arn (str) – the task arn

• task_definition (str) – the task definition

• cluster (str) – the cluster

• log_success (bool) – whether or not to log success

• log_failure (bool) – whether or not to log failure

• return_logs (bool) – whether or not to return logs

• strict_error (bool) – whether or not to raise an error

Raises:

AirflowException – if the task fails

Returns:

the logs

static get_ecs_logs(task_arn, task_definition, cluster='dev', sleep_time=5, iterations=6)

Get the logs for a task from ECS using the cli

Parameters:

• task_arn (str) – the task arn

• task_definition (str) – the task definition

• cluster (str) – the cluster that the task is on

• sleep_time (int) – the sleep time between retries

• iterations (int) – the number of iterations to try

Raises:

Exception – if the logs cannot be retrieved

Return type:

str

Returns:

the logs

static run_athena_to_geojson(query, title, geometry_column='geometry', wr_database='shoreline', wr_workgroup='airflow-dev', source_epsg='EPSG:4326', export_path_template='{title}.geojson', wait=True, task_suffix='', column_mapping={}, label_columns=[], container_name='ecs_athena_to_geojson_container', task_definition='ecs_athena_to_geojson_dev')

Run a athena query and save the results as a geojson via ecs

Parameters:

• query (str) – the query string to run on athena. use AS to rename columns

• title (str) – the title of the job. we use this as output file name too

• wr_database (str) – the datatable in athena

• wr_workgroup (str) – the workgroup in athena

• geometry_column (str) – the column that includes wkt format of the data. if it is a comma seperated text we split it to lat and lon and read it as point

• source_epsg (str) – the epsg of the source data

• export_path_template (str) – the template for the export path

• wait (bool) – whether or not to wait for the task to finish

• task_suffix (str) – the task suffix, e.g. “large”

• column_mapping (dict[str, str]) – the column mapping

• container_name (str) – the container name

• task_definition (str) – the task definition

Returns:

the task arn

Return type:

_type_

static run_ecs_h3(output_tablename, aws_query, agg_dict, export_path, partition_cols=[], overwrite=False, resolutions=[5, 6, 7, 8, 9], geometry_column='lat,lon', s3_bucket='463002504035-shoreline', database='shoreline', workgroup='airflow-dev', wait=True, task_suffix='')

Run a athena query on ECS h3 aggregation machine, and save the results as table in s3

Parameters:

• output_tablename (str) – the destination table name in athena, it will have format: c_h3_{output_tablename}, with extra cols: cell_id (h3 index id), cell_geom (geometry), resolution, …

• aws_query (str) – the query string to run on athena. Use SELECT col_x AS col_y to rename columns as needed

• agg_dict (dict[str, str]) – the mapping of column-aggregation method; acceptable aggregation methods includes: count, median (AND LATER: sum, mean, std, var, min, max)

• partition_cols (list[str]) – the additional columns to partition the result table, in format: h3_res/additional-partition-cols. NOTE: these columns must be included in the query; they will be used for partitioning the result table output; they will not be used for aggregation!

• overwrite (bool) – To overwrite existing data (table or partition) in the destination table if already exists

• resolutions (list[int]) – h3 resolutions to use as aggregation unit

• geometry_column (str) – the column that includes wkt format of the data. if it is a comma seperated text we split it to lat and lon and read it as point

• export_path (str) – the S3 path to save result table, default is curated/h3agg/c_h3_{output_tablename}/resolution=<>/[…]/.parquet

• s3_bucket (str) – location to store temp UNLOAD parquet and result parquets

• database (str) – the datatable in athena

• wr_workgroup – the workgroup in athena

• wait (bool) – whether or not to wait for the task to finish

• task_suffix (Literal['', 'large', 'xlarge']) – the size of ECS machine to use for processing

Returns:

the task arn

static run_geopandas(query, title, conn_id='postgres_readydata_dev', wait=True, task_suffix='', container_name='geopandas_container', task_definition='geopandas_dev', local=False)

(More complete and accurate name: run_postgres_to_geojson) Run a Postgres query, and save response as geojson file via ecs In contrast to run_athena_to_geojson; Neither methods read the web response into a geopandas dataframe Under the hood: ready/terraform and ECSUtil.run_task

Parameters:

• query – the query to run

• title (str) – the title of the end file

• conn_id (str) – the connection id

• wait (bool) – whether or not to wait for the task to finish

• task_suffix (str) – the task suffix, e.g. “large”

• container_name (str) – the container name

• task_definition (str) – the task definition

• local (bool) – whether or not to run the task locally (deprecated)

Returns:

the task arn

Return type:

_type_

static run_graphify_model(project_name, github_path, wait=True, task_suffix='', container_name='ecs_graphify_container', task_definition='ecs_graphify_dev', timeout=50000)

static run_hld_model(statefp, storage_prefix, sql, bucket='463002504035-shoreline', datalake_db='shoreline', workgroup='airflow-dev', wait=True, task_suffix='', container_name='ecs_hld_process_container', task_definition='ecs_hld_process_dev')

Run HLD setup process

Parameters:

• statefp (str) – state fip to process

• storage_prefix (str) – the storage prefix

• sql (str) – the sql query to run

• bucket (str, optional) – the s3 bucket to store setup network, defaults to Config.DATA_LAKE_BUCKET

• datalake_db (int, optional) – the datalake db, defaults to Config.DATA_LAKE_DB

• workgroup (str, optional) – the workgroup, defaults to Config.DEFAULT_WORKGROUP

• wait (bool, optional) – whether or not to wait for the task to finish, defaults to True

• task_suffix (str, optional) – the task suffix, e.g. “large”, defaults to “”

• container_name (str, optional) – the container name, defaults to Config.ECS_HLD_CONTAINER_NAME

• task_definition (str, optional) – the task definition, defaults to Config.ECS_HLD_TASK_DEFINITION

Returns:

task ARN

Return type:

_type_

static run_hld_setup(statefp, bucket='463002504035-shoreline', datalake_db='shoreline', workgroup='airflow-dev', wait=True, task_suffix='', container_name='ecs_hld_process_container', task_definition='ecs_hld_process_dev')

Run HLD setup process

Parameters:

• statefp (str) – state fip to process

• bucket (str, optional) – the s3 bucket to store setup network, defaults to Config.DATA_LAKE_BUCKET

• datalake_db (int, optional) – the datalake db, defaults to Config.DATA_LAKE_DB

• workgroup (str, optional) – the workgroup, defaults to Config.DEFAULT_WORKGROUP

• wait (bool, optional) – whether or not to wait for the task to finish, defaults to True

• task_suffix (str, optional) – the task suffix, e.g. “large”, defaults to “”

• container_name (str, optional) – the container name, defaults to Config.ECS_HLD_CONTAINER_NAME

• task_definition (str, optional) – the task definition, defaults to Config.ECS_HLD_TASK_DEFINITION

Returns:

task arn

Return type:

_type_

static run_mlflow_model(model_uri, params, input, s3_root=None, output_athena_dataset_name=None, wr_database='shoreline', wait=True, task_suffix='', container_name='ecs_mlflow_model_container', task_definition='ecs_mlflow_model_dev', timeout=50000)

static run_pgdump_command(cmd, wait=True, container_name='ecs_transfer_with_pgdump_container', task_definition='ecs_transfer_with_pgdump_dev')

Run a pgdump command via bash

Parameters:

• cmd (str) – fully customized pgdump command to run in ecs

• src_conn_id – the connection id (NOT full conn str!) of the source db

Returns:

the task arn

Return type:

_type_

static run_pgdump_transform(input_src_table, src_conn_id, dest_conn_id, input_src_schema='public', temp_new_schema_name='', temp_new_table_name='', drop_table=False, cmd='', wait=True, container_name='ecs_transfer_with_pgdump_container', task_definition='ecs_transfer_with_pgdump_dev')

Run a pgdump command via bash

Parameters:

• cmd (str) – fully customized pgdump command to run in ecs

• input_src_table (str) – the input table name

• src_conn_id (str) – the connection id (NOT full conn str!) of the source db

• dest_conn_id (str) – the connection id (NOT full conn str!) of dest db

• temp_new_schema_name (str) – the new schema name for the dest data

• temp_new_table_name (str) – new table name for dest data

• drop_table (bool) – whether or not to drop the table before copying. This does work with custom commands, you can include the –clean flag in your custom command if desired

• wait (bool) – whether or not to wait for the task to finish

• container_name (str) – the container name; full details at ready/terraform

• task_definition (str) – the task definition

Returns:

the task arn

Return type:

_type_

static run_s3_parquet_to_pg(table, key, drop_table=False, suffix='', bucket='463002504035-shoreline', batch_size=1000000, conn_id='postgres_readydata_dev', wait=True, task_suffix='', container_name='s3_parquet_to_pg_container', task_definition='s3_parquet_to_pg', local=False, use_pg_copy=False, on_conflict_sql=None, upsert_origin_table=None)

Run an ECS task to copy a parquet file from s3 to postgres in a set of chunks

Parameters:

• table (str) – the table name for the end table

• key (str) – the key of the parquet file

• drop_table (bool) – whether or not to drop the table before copying

• suffix (str) – the suffix for the CREATE TABLE query that runs in ECS

• bucket (str) – the bucket of the parquet file

• batch_size (int) – the batch size for the transfer in pandas

• conn_id (str) – the connection id

• wait (bool) – whether or not to wait for the task to finish

• task_suffix (str) – the task suffix, e.g. “large”

• container_name (str) – the container name

• task_definition (str) – the task definition

• local (bool) – whether or not to run the task locally (deprecated)

• use_pg_copy (bool) – experimental use of pg_copy

Returns:

the task arn

Return type:

_type_

static run_task(container_name, task_definition, env_vars, cluster='dev', subnets=['subnet-05d37f00bd1245b61'], security_groups=['sg-0bcb7113fb9299e23'], timeout=50000, sleep_time=5, task_suffix='', wait=True, check_status=True, wait_timeout=50000)

Run a task on ECS

Parameters:

• container_name (str) – the container name

• task_definition (str) – the task definition

• env_vars (list[dict[str, str]]) – the env vars to pass to the task, parametrize new containers with env variables

• cluster (str) – the cluster to run the task on

• subnets (list[str]) – the subnets that one may run the task on, spreading across subnets allows us to allocate resources more efficiently, reduces start failures

• timeout (int) – the timeout, sometimes the task will fail to start and we need to retry

• sleep_time (int) – the sleep time between timeout retries

• task_suffix (str) – add addl options for differently sized default containers

• wait (bool) – whether or not to wait for the task to finish

• check_status (bool) – whether or not to check the status of the task

• wait_timeout (int) – the timeout that airflow will wait to announce task as timed-out

Raises:

Exception – if the task fails

Returns:

the task arn

Return type:

_type_

static run_tile_join(name, description, input_mbtiles_stub, wait=True, tippecanoe_cmd="find {mount_path} -name '{input_mbtiles_stub}_*.mbtiles' | xargs tile-join -z 15 -pk -f -o {mount_path}/{input_mbtiles_stub}.mbtiles -n '{name}' -N '{description}'", container_name='tippecanoe_container', task_definition='tippecanoe_dev', task_suffix='')

Run a bash command on a set of mbtiles to join them together [under dev] via a container with tippecanoe installed in ecs

Parameters:

• name (str) – the name of the output tileset

• description (str) – the description of the output tileset. We will automatically add latest_update to the tileset description for the records

• input_mbtiles_stub (str) – basename for input mbtiles files. Will be expanded to find all <basename>_*.mbtiles to join

• wait (bool) – whether or not to wait for the task to finish

• container_name (str) – the container name

• task_definition (str) – the task definition

• task_suffix (str) – the task suffix, e.g. “large”, defaults to small

Returns:

the task arn

Return type:

str

static run_tippecanoe(title, description='', cmd='tippecanoe --temporary-directory={mount_path} -zg -Z0 --extend-zooms-if-still-dropping --drop-densest-as-needed --force -o {mbtiles_path} -l {title} -n {title_description} -N {description} {geojson_path}', geojson_template='{mount_path}/{title}.geojson', mbtiles_template='{mount_path}/{title}.mbtiles', wait=True, container_name='tippecanoe_container', task_definition='tippecanoe_dev', task_suffix='', local=False)

Run a bash command on a geojson file and save the results as an mbtiles file via a container with tippecanoe installed in ecs

Parameters:

• title (str) – the title of the file

• description (str) – the description of the tileset. We will automatically add latest_update to the tileset description for the records

• cmd (str) – the command to run in ecs, runs in a container with tippecanoe dependencies installed

• geojson_template (str) – the template for the geojson file path

• mbtiles_template (str) – the template for the mbtiles file path

• wait (bool) – whether or not to wait for the task to finish

• container_name (str) – the container name

• task_definition (str) – the task definition

• task_suffix (str) – the task suffix, e.g. “large” or “xlarge”

• local (bool) – whether or not to run the task locally (deprecated)

Returns:

the task arn

Return type:

_type_

sql_to_tile(title, conn_id='postgres_readydata_dev', task_suffix='')

Run a SQL query and generate a tileset from the results. this should probably belong to another interface and have multiple methods of creating tiles e.g. create locally with TilerUtil.create_tile

Parameters:

• query (str) – the query to run

• title (str) – the title of the end file

• conn_id (str, optional) – the connection id, defaults to Config.READY_DATA_DB

• task_suffix (str, optional) – the task suffix, e.g. “large”, defaults to “”

Returns:

the arns of the geojson and tippecanoe tasks

Return type:

tuple[str, str]

static wait_for_task(task_arn, cluster='dev', timeout=50000, sleep_time=5)

Wait for a task to finish

Parameters:

• task_arn (str) – the task arn

• cluster (str) – the cluster

• timeout (int) – the timeout

• sleep_time (int) – the sleep time

Raises:

Exception – if the task times out

FCCUtil

This class contains utilities for interacting with FCC data.

class util.fcc.FCCUtil

FCC_API_KEYS = 'fcc_api_creds'

FCC_FM_FUNDED_LOCATIONS = 'funded_locations'

FCC_FM_PROGRAM_FUNDED_LOCATIONS = 'program_funded_locations'

FCC_FM_STATE_FILE_THEMES = ['unserved_unfunded', 'funded_locations', 'program_funded_locations']

FCC_FM_UNSERVED_UNFUNDED = 'unserved_unfunded'

static create_fcc_api_caller(max_per_minute=10, max_attempts=3)

static create_fcc_api_client(max_per_minute=10, max_attempts=3)

Return type:

Callable[[str, bool], Tuple[Any, List[str]]]

static fcc_fm_state_file_theme(file_name)

Map an FCC Funding Map state{fips}_* file name to its theme (DATASME-1997): - state{fips}_unserved_unfunded_* -> unserved_unfunded - state{fips}_funded_locations_* -> funded_locations - state{fips}_program{id}_funded_locations_* -> program_funded_locations

Raises AirflowException on anything else so a new FCC file category fails the run loudly instead of landing in the wrong theme.

Parameters:

file_name (str) – the file name (with or without extension)

Return type:

str

Returns:

the theme directory name

static gen_fcc_headers()

Return type:

Dict[str, str]

static get_current_process_uuid(external_request=False, proxy_request=False)

Get the current process uuid from fcc broadband map

Parameters:

• external_request (bool) – whether or not to use external request

• proxy_request (bool) – whether or not to use proxy request

Return type:

str

Returns:

the current process uuid

static get_fcc_funding_info(external_request=False, proxy_request=False)

Get both the current funding process uuid and last updated date from fcc funding map

Parameters:

• external_request (bool) – whether or not to use external request

• proxy_request (bool) – whether or not to use proxy request

Return type:

tuple[str, str]

Returns:

tuple of (process_uuid, last_updated_date)

Raises:

ConnectionError – when website is not available

static get_fcc_last_updated(process_uuid, external_request=False, proxy_request=False, as_datetime=True, override_key='last_updated_date')

Get the last updated date from fcc broadband map

Parameters:

• process_uuid (str) – the process uuid

• external_request (bool) – whether or not to use external request

• proxy_request (bool) – whether or not to use proxy request

• as_datetime (bool) – whether or not to return as datetime or string

• override_key (str) – the override key

Returns:

the last updated date, or data as of date

static get_latest_fabric_version(database_name='postgres_bossplatform_game_dev', fa_ver=None, fs_ver=None, schema='data', fabric_version_tbname='fabric_versions')

gets latest published fabric version from PG TABLE: fabric_versions

Parameters:

• database_name (str, optional) – _description_, defaults to Config.GAME_PG

• schema (str, optional) – _description_, defaults to “data”

• fa_ver (int, optional) – the fabric address version, defaults to None

• fs_ver (int, optional) – the fabric fixed availability version, defaults to None; will take precendence over fa_ver if both are provided

Returns:

an object with following structure version_date, update_date, version, process_uuid

Return type:

FabricVersion

static get_max_filing_subtype(data)

Get the max filing subtype from fcc broadband map

Parameters:

data (list[dict]) – the data to get the max filing subtype from

Return type:

dict[str, str]

Returns:

the max filing subtype

static ping_fcc_website(external_request=False, proxy_request=False, max_retries=10, retry_interval_minutes=1)

Check if the FCC website is available before making requests with retry capability. Need this because the FCC website is often appearing down but is up after a few minutes.

Parameters:

• external_request (bool) – whether or not to use external request

• proxy_request (bool) – whether or not to use proxy request

• timeout – timeout in seconds for the request

• max_retries (int) – maximum number of retry attempts

• retry_interval_minutes (int) – time to wait between retries in minutes

Return type:

bool

Returns:

True if website is available, False otherwise

RADUtil

This class looks like it’s under construction, and I’m not sure what it’s for. It’s not being used anywhere.

class util.rad.RADUtil

Utilities work working with RAD db.

static create_or_get_rad_source(source_name)

Create or get source_id from RAD database.

Parameters:

source_name (str) – name of the source

Return type:

str

Returns:

source_id

static get_naics_codes()

Get 2022 NAICS codes from US Census. Source: https://www.naics.com/search/ :rtype: dict :return: dict of NAICS codes

static get_null_string_uppercased()

Get list of common values for null strings. NOTE: all values are UPPERCASED!

static helper_clean_addr_gen_pakey(indf, STREET_COL, CITY_COL, ZIPFULL_COL, STATE_COL, fill_statecode=None, newpakeycol='clean_fakey')

GENERATE pakey as ‘clean_fakey’, and also validate address components RETURN new columns: clean_addr_street clean_addr_city clean_addr_zip5 clean_addr_zipplus4 clean_addr_state and newpakeycol/clean_fakey

static helper_gen_poi_id(indf, md5_cols, prepcols, res_idcol=None)

prepcols is a subset of md5_cols (should only prep cols that goes into making of poi_id) LATER, only need to prep place_name; as pakey had been prepped; and all other cols must be of enum type! e.g. helper_gen_poi_id(finaldf, “created_by datasource pakey place_name ptype_mid”.split(), “place_name”.split())

S3Util

This class contains utilities for interacting with S3.

class util.s3.S3Util

S3 utilities for airflow

static copy_between_buckets(source_path, target_path, pattern='')

Copies a set of s3 objects between two buckets. it will replace data in the destination if they already exist

Parameters:

• source_path (str) – the source path to copy data from. make sure to include s3://

• target_path (str) – the target path to copy data to. make sure to include s3://

• pattern (str | None) – file name regex pattern to match. Default: None (matches all files)

Return type:

List[str]

Returns:

List of new objects paths.

static delete_s3_folder(bucket, prefix, override_panic=False, max_delete=1000000)

Delete a folder from s3 Ensure prefix ends with a slash so it’s a folder due to key prefix matching behavior of s3 Panics if the prefix ends with any of the panic prefixes in the config

Parameters:

• bucket (str) – the bucket to delete from

• prefix (str) – the prefix match to delete, be careful with partial matches

• override_panic (bool) – whether or not to override the panic prefixes in the config

• max_delete (int) – the threshold/max authorized amount of items to delete,

Return type:

list[dict]

Returns:

the response from boto

static download_folder(s3_path, local_path, recursive=True, verbose=False)

Download all files in the directory from s3 to local airflow server. Recommend passing a local_path that is created from TemporaryDirectory() to ensure cleanup.

Parameters:

• s3_path (str) – the s3 path of the folder to download

• local_path (str) – the local directory to download the contents into

• recursive – whether to download recursively, or ignore subdirectories

• verbose – whether to print out the files being downloaded

Return type:

None

static folder_exists_and_not_empty(bucket, path)

Check if a folder exists and is not empty.

Parameters:

• bucket (str) – the bucket name

• path (str) – prefix

Returns:

True if folder exists and not empty

Return type:

bool

static get_existing_s3_files(bucket, prefix, file_type='csv', full_key=False, return_raw_objects=False)

Get files from s3 that match the given file type. Deals with pagination.

Parameters:

• bucket (str) – the bucket to get from

• prefix (str) – the prefix to match

• file_type (str, optional) – the file type to match, without leading dot. Default: csv

• full_key (bool, optional) – whether or not to return the full key. Default: False

Return type:

list[str]

Returns:

the list of files

static get_presigned_url(bucket, key, expiration=604800, response_type=None, *, region_name=None, signature_version='s3v4')

Get a presigned URL for an S3 object

Signed with the automation user credentials from Secrets Manager so the url actually lives for the full expiration, see get_signing_credentials

Parameters:

• bucket (str) – the bucket name

• key (str) – the key of the object

• expiration (int) – the expiration time in seconds (default: 604800 seconds = 7 days, Max)

• response_type – optional ResponseContentType to force on the download

Return type:

str

The two below are keyword only, so the first four positions stay exactly as callers already pass them (e.g. bucket, key, expiration, response_type=…)

Parameters:

• region_name (Optional[str]) – the bucket’s region. sigv4 urls must be signed for the region the bucket lives in, so pass it when the bucket is not in the session default region. None leaves it to the boto session

• signature_version (str) – s3v4 (default) or s3 for legacy sigv2 query auth. boto3 presigns s3 with sigv2 unless told otherwise, and buckets created after June 2020 reject sigv2 outright, so v4 is the safer default

static get_raw_csv_columns(bucket, prefix, header_row=0)

Get arbitrary file from s3, returns a dict of column names as strings

Parameters:

• bucket (str) – the bucket to get from

• prefix (str) – the prefix to match

• header_row (int) – row to use as header (0-based index). Default: 0 (first row)

Return type:

dict[str, str]

Returns:

the dict of column names

static get_signing_credentials(secret_id='airflow/connections/airflow_automation_user', refresh=False)

Read the automation user credentials from Secrets Manager as boto3 client kwargs

Parameters:

• secret_id (str) – the secrets manager secret id holding the credentials json

• refresh (bool) – skip the in process cache and re-read the secret

Return type:

dict

Returns:

boto3 credential kwargs, or an empty dict when the secret is missing or incomplete, in which case the caller falls back to the default credential chain (task role) rather than failing outright

static s3_url_to_table(url)

Convert an s3 url to a table name

Parameters:

url (str) – the url to convert

Return type:

str

Returns:

the table name

static store_s3_file(bucket, key, file_path, overwrite=True)

Store a file on s3

Parameters:

• bucket (str) – the bucket

• key (str) – the key in s3

• file_path (str) – the file path to store from

• overwrite – call the ifNoneMatch value (to prevent overwriting)

Return type:

dict

Returns:

the response from boto

CAIUtil

This class might contain utils for interacting with CAI data? It’s not being used anywhere.

class util.sbo_cai.CAIUtil

All sbos’ CAI files and their location in datalake

CAI_DEDUP_IDCOLS = ['state', 'volume', 'full_address_str', 'entity_name', 'cai_type']

CAI_RAW_REQUIRED_COLS = {'cms_number', 'download', 'erate_entity_number', 'explanation', 'frn', 'lat', 'lng', 'location_id', 'need1000', 'raw_cai_type', 'raw_city', 'raw_entity_name', 'raw_state', 'raw_street', 'raw_zipfull'}

CAI_UNIQUE_MD5_COLS = ['state', 'volume', 'full_address_str', 'entity_name', 'cai_type', 'datasource_idx']

DICT_PTYPE_MID_TO_RAW = {'GOVERNMENT/BUILDING': 'G', 'GOVERNMENT/SAFETY': 'F', 'HEALTH': 'H', 'HOUSING': 'P', 'LIBRARY': 'L', 'SCHOOL': 'S', 'SUPPORT': 'C'}

DICT_PTYPE_RAW_TO_MID = {'C': 'SUPPORT', 'F': 'GOVERNMENT/SAFETY', 'G': 'GOVERNMENT/BUILDING', 'H': 'HEALTH', 'L': 'LIBRARY', 'P': 'HOUSING', 'S': 'SCHOOL'}

EXCEL_FILE_EXTENSIONS = ('.xls', '.xlsx', '.xlsm', '.xlsb', '.odf', '.ods', '.odt')

static get_latlng_fractional_len(coord_df, coord_col)

static pprint(print_msg, marker='NHAT_LOGGING')

SBOUtil

This class contains utilities for interacting with SBO data. It’s only used in one DAG, which is perhaps not an ideal use case for a cental util.

class util.sbo_fabric_address.SBOUtil

All FABRIC ADDRESS version 2 and 3! inventory files and their location in datalake

static get_all_state(fabric_type, version)

list all states that provide data to Ready

Parameters:

• fabric_type (str) – bsl_primary, nonbsl, bsl_secondary

• version (int) – fabric version 1-8

Returns:

a list of states’ abbreviation that we have data for

Return type:

list

static get_fabric_s3_path(state, fabric_type, version)

Get the s3 path for the fabric inventory files :type state: str :param state: state abbreviation in all caps :type fabric_type: str :param fabric_type: bsl_primary, nonbsl, bsl_secondary :type version: int :param version: 1-8 :rtype: str :return: the s3 path for the fabric inventory files or empty string if fabric_type doesn’t exist

static get_s3_path_v1()

Get the s3 path in a dictionary format for the FABRIC ADDRESS version 1 inventory files

Return type:

dict

Returns:

the s3 path a dictionary format for the fabric inventory files

static get_s3_path_v2()

Get the s3 path in a dictionary format for the FABRIC ADDRESS version 2 inventory files

Return type:

dict

Returns:

the s3 path a dictionary format for the fabric inventory files

static get_s3_path_v3()

Get the s3 path for the fabric version 3 inventory files

Return type:

dict

Returns:

the s3 path in a dictionary format for the fabric inventory files

static get_s3_path_v4()

Return type:

dict

static get_s3_path_v5()

Return type:

dict

static get_s3_path_v6()

Return type:

dict

static get_s3_path_v7()

Return type:

dict

static get_s3_path_v8()

Return type:

dict

static get_s3_path_v9()

Return type:

dict

GDriveUtil

This class contains utilities for interacting with Google Drive.

class util.gdrive.GDriveUtil

Google Drive utilities for create folder, upload file, download file. The user must share folder or file with the service account email address

Returns:

_description_

Return type:

_type_

static append_row_to_google_sheets(spreadsheet_id_or_url, row_data, expected_column_order=None, sheet_names=None, style_sheet=True)

Append a single row to one or more sheets in a Google Sheets file.

Return type:

None

Args:

spreadsheet_id_or_url: Google Sheets ID or full URL row_data: Dictionary with column names as keys and values to append sheet_names: List of sheet names to append to. If None, appends to all sheets. credentials_file: Path to Google service account credentials JSON file.

If None, uses default credentials.

Example:

row_data = {

‘version_date’: ‘2024-01-08’, ‘environment’: ‘production’, ‘status’: ‘active’, ‘updated_at’: ‘2024-01-08 10:30:00’, ‘created_at’: ‘2024-01-08 10:30:00’, ‘dp-ro relationship’: ‘1DP-1RP’, ‘sa-relationship’: ‘linked’, ‘source_folder’: ‘s3://bucket/folder’

}

append_row_to_google_sheets(

‘1ABC123xyz…’, row_data, sheet_names=[‘AR’, ‘MN’], credentials_file=’credentials.json’

)

static construct_gdrive_service(service_account_conn_id='gdrive_api')

Construct a Google Drive service object using the sevice account. The service account email address must be added to the folder or file to be accessed. default service account is stored in aws secrets manager. default service account email address is:googledrive@inner-period-265200.iam.gserviceaccount.com

Parameters:

service_account_conn_id (str) – the connection id for the service account (stored in aws secrets manager)

Returns:

a Google Drive service object

Return type:

object

static create_new_gsheet(url, sheet_name, rows, cols)

Create a new sheet in the Google Sheet by URL and return the worksheet object.

Parameters:

• url (str) – the URL of the Google Sheet

• sheet_name (str) – the name of the new sheet to create

• rows (int) – the number of rows in the new sheet

• cols (int) – the number of columns in the new sheet

Return worksheet:

the Google Sheet worksheet object

Return type:

object

Returns:

the worksheet object

Return type:

object

static export_query_to_gdrive(query, export_file_name, gdrive_url='', sub_gdrive_id='', rename_dict={}, database_type='athena', database='shoreline', returning_df=False, overwrite_dest_file=True)

Export a query result to Google Drive as a CSV file.

Parameters:

• query (str) – the query to execute

• export_file_name (str) – the name of the file to export

• gdrive_url (str) – the Google Drive parent folder URL

• sub_gdrive_id (str) – the Google Drive parent folder ID

• rename_dict (dict) – the dictionary to rename the columns

• database_type (str) – the database type (athena or postgres)

• database (str) – the database name

• returning_df (bool) – whether to return the DataFrame

• overwrite_dest_file (bool) – whether to overwrite the destination file if it already exists

Returns:

the DataFrame if returning_df is True

Return type:

object

static gdrive_create_folder(folder_name, drive_service, parent_folder_id=None, pageSize=1000)

Create a folder in Google Drive and return its ID. If already created, return the ID of the existing folder.

Parameters:

• folder_name (str) – the name of the folder to create under the parent folder

• drive_service (object) – the Google Drive service object

• parent_folder_id (str) – the parent folder ID. This can be None if the folder is to be created at the root level

• pageSize (int) – the page size for the list request

Returns:

the ID of the created folder

Return type:

str

static gdrive_delete_files(drive_service, file_or_folder_id)

Delete a file or folder in Google Drive by ID.

Parameters:

• drive_service (object) – the Google Drive service object

• file_or_folder_id (str) – the ID of the file or folder to delete

Returns:

None

Return type:

None

static gdrive_download_file(drive_service, file_id, destination_path)

Download a file from Google Drive by its ID. WARINING: needs to add the service account to the drive as a user with access to the file first. Risk of out of storage error.

Parameters:

• drive_service (object) – the Google Drive service object

• file_id (str) – the ID of the file to download

• destination_path (str) – the destination path to save the file

Returns:

None

Return type:

None

static gdrive_list_folder(drive_service, parent_folder_id=None, pageSize=1000)

List sub folders and files in a Google Drive parent folder.

Parameters:

• drive_service (object) – the Google Drive service object

• parent_folder_id (str) – the ID of the parent folder. This can be None if the folder is to be created at the root level

• pageSize (int) – the page size for the list request. Default is 1000

Returns:

the list of folders and files

Return type:

list

static gdrive_share_with_emails(drive_service, file_or_folder_id, emails, role='reader', send_notification_email=False)

Grant a list of emails access to a Google Drive file or folder.

Best-effort: a failure to share with one email (e.g. an invalid address or a domain-restricted Shared Drive) is logged and skipped rather than failing the whole task.

Parameters:

• drive_service (object) – the Google Drive service object

• file_or_folder_id (str) – the ID of the file/folder to share

• emails (list) – the email addresses to grant access to

• role (str) – the Drive permission role (reader/commenter/writer). Default reader

• send_notification_email (bool) – whether Google should send its own share notification. Default False (the DAG sends its own email).

Returns:

None

Return type:

None

static gdrive_upload_file(drive_service, file_path, file_name, parent_folder_id=None, overwrite=False)

Upload a file to Google Drive.

Parameters:

• drive_service (object) – the Google Drive service object

• file_path (str) – the file path of the file to upload

• file_name (str) – the name of the file to upload

• parent_folder_id (str) – the ID of the parent folder. This can be None if the file is to be added to the root level.

• overwrite (bool) – whether to overwrite the file if it already exists. Default is False, meaning a new file with exact same name will be created in same folder!

Returns:

the ID of the created file

Return type:

str

static get_all_gdrive_files(service, folder_id, recursive=True)

Get all file IDs from a Google Drive folder.

Parameters:

• service (object) – Google Drive API service object

• folder_id (str) – the ID of the Google Drive folder

• recursive (bool) – whether to recursively traverse subfolders (default: True)

Returns:

the list of file IDs

Return type:

List[str]

static get_gcp_credential(service_account_conn_id='gdrive_api', scopes=['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/spreadsheets'])

Get the Google Cloud Platform credentials from the connection id. The service account email address must be added to the folder or file to be accessed. default service account is stored in AWS Secrets Manager. default service account email address is:googledrive@inner-period-265200.iam.gserviceaccount.com

Parameters:

service_account_conn_id (str) – the connection id for the service account (stored in AWS Secrets Manager)

Returns:

the Google Cloud Platform credentials

Return type:

dict

static get_gspread_service(service_account_conn_id='gdrive_api')

Construct a Google Spread Sheet service object using the sevice account. The service account email address must be added to the folder or file to be accessed. default service account is stored in aws secrets manager. default service account email address is:googledrive@inner-period-265200.iam.gserviceaccount.com

Parameters:

service_account_conn_id (str) – the connection id for the service account (stored in aws secrets manager)

Returns:

a Google Sheet service object

Return type:

object

static get_mimetype(service, file_id)

Get the mimetype of a Google Drive file.

Parameters:

• service (object) – Google Drive API service object

• file_id (str) – the ID of the Google Drive file

Returns:

the mimetype of the file

Return type:

str

static gsheet_to_dataframe(url, sheet_name, headerrow=1)

Read the Google Sheet by URL and return a pandas DataFrame. The Sheet must be shared with the service account email address.

Parameters:

• url (str) – the URL of the Google Sheet

• sheet_name (str) – the name of the sheet to read

• headerrow (int) – the row number of the header. Default is 1

Return df_spread:

the pandas DataFrame of the Google Sheet

Return type:

object

Return worksheet:

the Google Sheet worksheet object

Return type:

object

static is_drive_folder(service, google_drive_file_id)

Check if a Google Drive file is a folder.

Parameters:

• service (object) – Google Drive API service object

• google_drive_file_id (str) – the ID of the Google Drive file

Returns:

whether the file is a folder

Return type:

bool

static style_google_sheet(worksheet)

Apply professional styling to a Google Sheets worksheet.

Return type:

None

Args:

worksheet: gspread worksheet object

static write_df_to_gsheet(df, file_name, sheet_name='Sheet1', parent_folder_id='1kONOPVDuWMdDxR16ZME5xnyhOXUkMIeC', overwrite=True)

Return type:

str

DataHubUtil

This class contains utilities for interacting with Datahub.

class util.data_hub.DataHubUtil

A set of the utility functions to interact with DataHub.

static add_assets_to_data_product(data_product_id, assets=list[str], domain_urn=None)

It updates a data product and adds assets to it.

Parameters:

• data_product_id (_type_) – the id of the data product used to create it. does not need to be a urn

• assets (list[str], optional) – list of the assets. It should be generated using dataset_urn or be a valid urn, defaults to list[str]

• domain_urn (str, optional) – the domain urn. It should be a urn generated using the domain_urn method of the DataHubUtil class. or a valid urn, defaults to None

static add_documentation_to_dataset(dataset_urn, documentation_to_add, user_id='ingestion')

Adds a documentation to the institutional memory of a dataset.

Parameters:

• dataset_urn (str) – the item urn. should be a urn generated using the dataset_urn method of the DataHubUtil class. or a valid urn

• documentation_to_add (str) – the text. it can be md formatted text

• user_id (str, optional) – the user who created it. If you pass dag owner it should include the full name like majid@ready.net, defaults to “ingestion”

static add_domain_to_dataset(dataset_urn, domain_urn)

Adds a domain to a dataset.

Parameters:

• dataset_urn (str) – The dataset urn. It should be a valid urn or generated using the dataset_urn method of the DataHubUtil class.

• domain_urn (str) – Domain urn. It should be a valid urn or generated using the domain_urn method of the DataHubUtil class.

static add_link_to_dataset(dataset_urn, link_to_add, link_description, user_id='ingestion')

Adds a link to the institutional memory of a dataset.

Parameters:

• dataset_urn (str) – It should be a urn generated using the dataset_urn method of the DataHubUtil class. or a valid urn

• link_to_add (str) – A link url formatted as a string.

• link_description (str) – link description

• user_id (str, optional) – the user who generated the link. If you pass dag owner it should include the full name like majid@ready.net, defaults to “ingestion”

static add_new_platform(name, id, logo_url)

Adds a new platform to DataHub.

Parameters:

• name (str) – Name of the platform

• id (str) – Id of the platform

• logo_url (str) – Logo url of the platform

static add_owner_to_dataset(dataset_urn, owner_urn, ownership_type='TECHNICAL_OWNER')

Adds an owner to a dataset.

Parameters:

• dataset_urn (str) – dataset urn. It should be a urn generated using the dataset_urn method of the DataHubUtil class. or a valid urn

• owner_urn (str) – owner urn. It should be a urn generated using the user_urn method of the DataHubUtil class. or a valid urn

• user_id (str, optional) – the user who created it. If you pass dag owner it should include the full name like

• ownership_type (OwnershipTypeClass, optional) – the type of ownership. It should be a valid ownership type, defaults to OwnershipTypeClass.TECHNICAL_OWNER

static add_properties_to_dataset(dataset_urn, properties)

Add properties to a dataset

Parameters:

• properties (dict[str,str]) – the properties to add

• dataset_urn (str) – the dataset urn. It should be a urn generated using the dataset_urn method of the DataHubUtil class. or a valid urn

static add_query_to_urn(query_text, dataset_urn, title='Sample Query', description='', created_by='ingestion', check_duplicates=True)

Add a sample query to a dataset URN using GraphQL. Now includes duplicate checking by default.

Parameters:

• query_text (str) – The SQL query text to add

• dataset_urn (str) – The dataset URN to add the query to

• title (str) – Title for the query

• description (str) – Description of the query

• created_by (str) – User who created the query

• check_duplicates (bool) – Whether to check for duplicates before adding

Returns:

Success status

Return type:

bool

static add_tag_to_dataset(dataset_urn, tag_urn)

add a tag to a dataset

Parameters:

• dataset_urn (str) – the dataset urn. It should be a urn generated using the dataset_urn method of the DataHubUtil class. or a valid urn

• tag_urn (str) – the tag urn. It should be a urn generated using the tag_urn method of the DataHubUtil class. or a valid urn

static add_tile_set_to_datahub(base_url, tile_id, entity_urn)

Add a tileset to DataHub.

Parameters:

• base_url (str) – the base url of the tile server

• tile_id (str) – the tileset id

static add_tileset_assertions(entity_urn, assertion_urn, assertion_type='SUCCESS')

Add assertions to the tileset deployment.

Parameters:

• entity_urn (str) – the tileset dataset urn

• assertion_urn (str) – the assertion_urn id

• assertion_type (AssertionResultType, optional) – the assertion type, defaults to AssertionResultType.SUCCESS

static add_tileset_properties_to_datahub(tile_json, entity_urn, base_url)

Add tileset properties to DataHub.

Parameters:

• tile_json (dict) – the tileset json

• entity_urn (str) – the tileset dataset urn

• base_url (str) – the base url of the tile server

static add_tileset_schema_to_datahub(tile_json, entity_urn)

Add tileset schema to DataHub.

Parameters:

• tile_json (dict) – the tileset json

• entity_urn (str) – the tileset dataset urn

static create_data_product(id, display_name, description, assets=[], owners=[], terms=[], tags=[], properties={}, external_url=None, domain=None)

Create a data product in DataHub.

Parameters:

• id (str) – the id of the data product used to create it. does not need to be a urn and should be unique per data product

• display_name (str) – the display name of the data product

• domain (urn) – the domain of the data product. It should be a urn generated using the domain_urn method of the DataHubUtil class. or a valid urn. you can use DataHubUtil.domain_urn() to generate it.

• description (str) – the data product description

• assets (list, optional) – an array of assets urns in str format. , defaults to []

• owners (list, optional) – Urn of the owners, defaults to []

• terms (list, optional) – _description_, defaults to []

• tags (list, optional) – an array of tag urns, defaults to []

• properties (dict, optional) – a dict of properties, defaults to {}

• external_url (_type_, optional) – _description_, defaults to None

static create_domain(name, description, parent_domain_urn=None)

create a domain in datahub

Parameters:

• name (str) – name of the domain

• description (str) – description of the domain

• parent_domain_urn (str, optional) – the parent domain urn. It should be a urn, defaults to None

Returns:

the new domain urn

Return type:

str

static create_tag(name, description)

generates a new tag urn

Parameters:

• name (str) – name of the tag

• description (str) – description of the tag

Returns:

the newly created tag urn

Return type:

str

static dataset_exists(dataset_urn)

Check if a dataset exists in the datahub

Parameters:

dataset_urn (str) – The urn of the dataset

Returns:

True if the dataset exists, False otherwise

Return type:

bool

static dataset_urn(platform, name, is_production=False)

returns generated dataset urn considering Ready Data platform settings

Parameters:

• platform (Literal[&quot;athena&quot;,&quot;postgres&quot;,&quot;airflow&quot;, &quot;tileservergl&quot;, &quot;bossplatform&quot;, &quot;grapi&quot;]) – the platform where the dataset is located

• name (str) – name of the table or dataset or dag, task

• is_production (bool, optional) – the running environment., defaults to Config.PRODUCTION

Returns:

a dataset urn matching with datahub schema

Return type:

str

static delete_query(query_urn)

Delete a query by URN.

Parameters:

query_urn (str) – The URN of the query to delete

Returns:

Success status

Return type:

bool

static domain_urn(domain='Data Engineering')

Returns the domain URN for the given domain.

Parameters:

domain (str, optional) – The domain name., defaults to DATAHUB_DEFAULT_DOMAIN_NAME

Returns:

The domain URN.

Return type:

str

static emit_assertion_result(assertionResult)

Emit assertion result to DataHub.

Parameters:

assertionResult (AssertionRunEvent) – _description_

Return type:

None

static field_urn(field_name, dataset_urn)

Gets a field urn for a given field name and dataset urn.

Parameters:

• field_name (str) – field name

• dataset_urn (str) – dataset urn. It should be a urn generated using the dataset_urn method of the DataHubUtil class. or a valid urn

Returns:

a field urn

Return type:

str

static get_data_product(data_product_id, domain_urn)

Get a data product from DataHub.

Parameters:

• data_product_id (str) – a data product id

• domain_urn (urn) – the domain urn. It should be a urn generated using the domain_urn method of the DataHubUtil class. or a valid urn

Returns:

a data product entity

Return type:

_type_

static get_emitter()

Returns a DataHubRestEmitter object.

This function retrieves an access token using the _get_datahub_token method from the DataHubUtil class. It then creates and returns a DataHubRestEmitter object using the retrieved access token and the DATAHUB_GMS_ENDPOINT configuration value.

Returns:

A DataHubRestEmitter object.

Return type:

DataHubRestEmitter

static get_graph()

Returns a DataHubGraph object.

Returns:

DataHubGraph object.

static get_properties_from_tilejson(tile_json, base_url)

Get properties from a tile json.

Parameters:

• tile_json (dict) – the tile json

• base_url (str) – the base url of the tile server

Returns:

the properties

Return type:

dict

static get_queries_for_dataset(dataset_urn)

Get all queries associated with a dataset.

Parameters:

dataset_urn (str) – The dataset URN to get queries for

Returns:

List of queries with their details

Return type:

List[Dict[str, Any]]

static get_schema_from_tilejson(tile_json)

Get schema fields from a tile json.

Parameters:

tile_json (dict) – the tile json

Returns:

the schema fields

Return type:

list[SchemaFieldClass]

static get_tileset_assertion_urn(tile_id, env='DEV')

Get the tileset assertion urn.

Parameters:

• tile_id (str) – the tileset id

• env (Literal["DEV", "STAGE", "PROD"], optional) – the environment where the tileset is deployed, defaults to “DEV”

static get_time_stamp(user='ingestion')

Get a time stamp for the current time.

Parameters:

user (str, optional) – the user who generated the time stamp. If you pass dag owner it should include the full name like `

static get_user_urn_from_dag(context)

Get a user urn from a dag context.

Parameters:

context (Context) – the username

Return type:

str

static lineage_urn(platform, name, is_production=False)

_summary_

Parameters:

• platform (Literal[&quot;athena&quot;, &quot;postgres&quot;, &quot;airflow&quot;, &quot;tileservergl&quot;, &quot;bossplatform&quot;, &quot;grapi&quot;]) – the platform where the inlet or outlet is located

• name (str) – the name of the inlet or outlet

• is_production (bool, optional) – defaults to Config.PRODUCTION

Returns:

_description_

Return type:

_type_

static merge_data_product_assets(data_product_id, assets, domain_urn)

Merge assets of a data product with existing assets.

Parameters:

• data_product_id (str) – the id of the data product used to create it. does not need to be a urn

• assets (list[urn]) – list of the assets. It should be generated using dataset_urn or be a valid urn

• domain_urn (urn) – the domain urn. It should be a urn generated using the domain_urn method of the DataHubUtil class. or a valid urn

Returns:

a list of assets

Return type:

_type_

static query_exists_for_dataset(dataset_urn, query_text=None, title=None)

Check if a query already exists for a dataset. Can check by either query text (exact match) or title.

Parameters:

• dataset_urn (str) – The dataset URN to check

• query_text (str) – The SQL query text to check (optional)

• title (str) – The query title to check (optional)

Returns:

True if query exists, False otherwise

Return type:

bool

static remove_dataset(dataset_urn)

remove a dataset from DataHub.

Parameters:

dataset_urn (str) – a dataset urn. It should be a urn generated using the dataset_urn method of the DataHubUtil class. or a valid urn

static search_datasets(query='*', platform=None, is_production=False, count=100)

Search for datasets in DataHub.

Parameters:

• query (str) – Search query string (supports wildcards)

• platform (Optional[str]) – Filter by platform (e.g., “postgres”, “athena”)

• is_production (bool) – Whether to search PROD or DEV environment

• count (int) – Maximum number of results to return

Returns:

List of datasets with urn, name, and description

Return type:

List[Dict[str, Any]]

static send_graphql_request(query, variables=None)

Send a GraphQL request to DataHub.

Parameters:

• query (str) – GraphQL query string

• variables (Optional[Dict[str, Any]]) – Variables for the GraphQL query

Returns:

Response from the GraphQL endpoint

Return type:

Dict[str, Any]

static set_column_documentation(dataset_urn, columns)

Set documentation for multiple columns of a dataset.

Parameters:

• dataset_urn (str) – The dataset URN

• columns (List[Dict[str, Any]]) – List of column definitions, each with: - name: Column name (required) - description: Column description (required) - field_type: Data type (optional, included in description) - nullable: Whether nullable (optional) - is_primary_key: Whether PK (optional) - foreign_key_reference: FK reference like “Table.column” (optional) - relationship_cardinality: FK cardinality like “1:1”, “1:N”, “N:M” (optional, only if foreign_key_reference is provided)

Returns:

Results dict with counts of successes and failures

Return type:

Dict[str, Any]

static tag_urn(name)

generate tag urn

Parameters:

name (str) – name of the tag

Returns:

tag urn

Return type:

str

static user_urn(username)

generates and returns user urn

Parameters:

username (str) – name of the user. If you pass dag owner it should include the full name like majid@ready.net

Returns:

user urn

Return type:

str

CleanDataUtil

This class contains utilities for cleaning company name data and standardizing names

class util.clean_data.CleanDataUtil

A series of functions to clean the data before it is loaded into the data lake

static clean_company_names(option='all', special_char='!#$%^*".,()-')

This function is returns a sql string that you can add to the sql query to remove those special characters in a company name. Default removing all special characters.

Param:

option (str): the option to remove the special characters from the company name. The options are: - company_suffix: remove the company suffixes like LLP, LLC, LTD, LP, INC, CO, IL, LIMITED - common_words: remove the company suffixes and state abbreviations - all: remove all above mentioned character and the words like COMMUNICATIONS, COMMUNICATION, COOPERATIVE, CORPORATION, TELECOMMUNICATIONS, COMM, TELECOM, TELCOM, TELEPHONE, FIBER, INTERNET, COOP, COMPANY, NETWORK, NETWORKS, ELECTRIC, POWER, WIRELESS, TELEVISION, CABLE, ELECTRICITY, CONNECT, CONNECTION

Param:

special_char (str): the special characters to remove from the company name. Default: !#$%^*”.,()-

Return type:

str

Returns:

a function called clean_company_names that removes special characters from a company name

BeadUtil

This class contains state-specific BEAD and ARC Final Proposal configuration, including source versions and DP/RP/SGA relationship modes.

class util.bead.BeadUtil

Util class for Bead post-challenge configurations.

DEFAULT_1DP_1RP = '1DP-1RP'

DEFAULT_1DP_1SA = '1DP-1SA'

DEFAULT_1PROVIDER_1SA = '1Provider-1SA'

PROVIDER_DP_RP_RELATIONSHIP = 'provided-deployment_reporting.csv'

PROVIDER_SA_RELATIONSHIP = 'provided-subgrant_agreements.csv'

PROVIDER_SA_RP_RELATIONSHIP = 'provided-subgrant_agreements_rp.csv'

S3_ROOT = 's3://463002504035-shoreline/raw/uploads'

static bead_profile(state_abbr, check_pre=True)

table_pre_challenge: the table name before the challenge table_trueup: the table name after the true-up. Before the challenge results applied. This table is only for data processing. Not for actual frontend functionality org_id: the organization id for the state ffa_version: the post challenge version of the fabric fixed availability data fabric_version: the post challenge version of the fabric data true_up: whether the state has true-up or not dsl_mod: whether the state has DSL modification or not fw_mod: whether the state has Cellular fixed wireless modification or not tenant: the name of the tenant in the grapi database

Parameters:

• state_abbr (str) – the state abbreviation in upper case

• check_pre (str, optional) – whether to check the pre-challenge tables. Default is True.

Return type:

dict

Returns:

the post challenge configuration for the state

NOTE: this function does call the AWS Athena and AWS Glue APIs to check the existence of the tables. BEWARE: When use outside of a DAG/@task i.e. as top level import, must set check_pre to False.

fp_profiles = {'AR': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'gdrive_folder_id': '1Daxok5siXEMJasnlQdVIQDWj69B35lOy', 'sa_relationship': 'provided-subgrant_agreements.csv', 'sga_outside_platform': False, 'version_date': '20260827'}, 'AZ': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'excluded_applicant_int_ids': [22, 120], 'gdrive_folder_id': '1krHNEFqVBqeRAEYeOgEadyUtVrObrzfm', 'sa_relationship': 'provided-subgrant_agreements.csv', 'sga_outside_platform': True, 'version_date': '20260923'}, 'CA': {'dp_rp_relationship': '1DP-1RP', 'excluded_applicant_int_ids': [292, 59, 202, 189, 209, 188, 204, 186, 300, 301, 302], 'gdrive_folder_id': '1qhHWSYH_7Shn7n_UC4WgaNC-o1bcafL2', 'sa_relationship': '1DP-1SA', 'sga_outside_platform': False, 'version_date': '20260917'}, 'GU': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'excluded_project_ids': ['CM61-BEAD-GU-PA-001-A', 'CM61-BEAD-GU-PA-003-A', 'CM61-BEAD-GU-PA-005-A', 'CM61-BEAD-GU-PA-006-A'], 'gdrive_folder_id': '1MKc7Ut3VdJaVGlJAH0t3uA7sZsqTmnxR', 'sa_relationship': '1Provider-1SA', 'sga_outside_platform': True, 'version_date': '20260828'}, 'ID': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '1QBMCp4eBUJnWKRU_G3yZu38gft3UL6c7', 'sa_relationship': '1DP-1SA', 'sga_outside_platform': True, 'version_date': '20260825'}, 'IL': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '1nQYPZU81p82w1-C0X28l4GGSom797-o9', 'sa_relationship': '1Provider-1SA', 'sga_outside_platform': True, 'version_date': '20260731'}, 'IN': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '1zybjT4HQOxlv9zQAh9NNBr1d45VkLKK8', 'sa_relationship': 'provided-subgrant_agreements.csv', 'sga_outside_platform': True, 'version_date': '20260819'}, 'KS': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '1JWgHI0RtZqtpEjg-WKoMtOBgM_5Gz6ST', 'sa_relationship': '1DP-1SA', 'sga_outside_platform': True, 'version_date': '20260916'}, 'LA': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '122M5NR2maWwsT6nFsWboueHJwL9ztlCk', 'sa_relationship': '1Provider-1SA', 'sga_outside_platform': True, 'version_date': '20251118'}, 'MN': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '1vfd3ou0CpZ-QAi3JRsKzQo8LHrCTaD4H', 'sa_relationship': '1DP-1SA', 'sga_outside_platform': False, 'version_date': '20251222'}, 'MP': {'allow_empty_cai': True, 'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '188bp5cm_6rKhtcTyuRbwGwTPfvF3E-ZY', 'sa_relationship': '1Provider-1SA', 'sga_outside_platform': False, 'version_date': '20260928'}, 'MS': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'gdrive_folder_id': '1PcmC_co0b8fOla_OU0ZRZ3tMxOi814c-', 'sa_relationship': '1Provider-1SA', 'sga_outside_platform': False, 'version_date': '20260925'}, 'NE': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '1Z2B8wN1u2T5WdUE30365ZXxx6SaY_3ca', 'sa_relationship': '1DP-1SA', 'sga_outside_platform': True, 'version_date': '20260727'}, 'OK': {'dp_rp_relationship': '1DP-1RP', 'excluded_applicant_int_ids': [45, 48, 111, 258, 117, 177, 68], 'gdrive_folder_id': '1XVRfNztUdAy4vhTh2D4lzOZ1xm2sOnAT', 'sa_relationship': '1Provider-1SA', 'sga_outside_platform': False, 'version_date': '20260528'}, 'TX': {'dp_rp_relationship': '1DP-1RP', 'gdrive_folder_id': '1D2wSRC5JKfcLUUhUkmE135Vg0cjoou7I', 'sa_relationship': '1Provider-1SA', 'sga_outside_platform': False, 'version_date': '20261001'}, 'UT': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'gdrive_folder_id': '1MRCVT8ho_bRpO9vcoHD_cFyJMhkJuTiE', 'sa_relationship': '1DP-1SA', 'sga_outside_platform': True, 'version_date': '20260825'}, 'VA': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'excluded_project_ids': ['CM61-BEAD-VA-RSN002', 'CM61-BEAD-VA-RSN003', 'CM61-BEAD-VA-RSN004', 'CM61-BEAD-VA-RSN006'], 'excluded_reporting_project_names': ['RiverStreet MP', 'RiverStreet WP3', 'RiverStreet WP2', 'RiverStreet CRC', 'RiverStreet CR'], 'gdrive_folder_id': '17PDOKSo6kMdpaYQSTe_BgxswRQB48XWc', 'sa_relationship': 'provided-subgrant_agreements_rp.csv', 'sga_outside_platform': False, 'version_date': '20260709'}, 'XY': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'gdrive_folder_id': '1Dp1IKuqiN3XNRzBh4GL1CEMbFamAXu0N', 'sa_relationship': 'provided-subgrant_agreements.csv', 'sga_outside_platform': False, 'version_date': '20260507'}, 'ZZ': {'dp_rp_relationship': 'provided-deployment_reporting.csv', 'gdrive_folder_id': '1dvZlLv9Lly4GM1RTrF4FEOgkjoF5l0IW', 'sa_relationship': 'provided-subgrant_agreements.csv', 'sga_outside_platform': False, 'version_date': '20251204'}}

static get_all_pre_cha_lake_table()

Get all pre challenge lake table for all states

Return type:

dict

static get_all_ready_sbo()

Get all ready SBOs for the states. This is used to check if the state has a pre-challenge table.

Return type:

list

static get_arc_final_proposal_config(state_abbr)

Get the FP profile for the state.

Return type:

dict

static get_post_challenge_approved_files(state_abbr)

Get the post challenge approved files for the state.

Return type:

dict

static get_postchallenge_states()

Get the list of states that have post-challenge configurations recorded. This does not check the pre-challenge tables, so you would need to call bead_profile to check individual states.

Return type:

list

static get_pre_ffa_version(state_abbr)

Get the FFA version for the state.

Return type:

str

static get_state_abbr_by_tenant(tenant)

Get the state abbreviation by tenant name. :type tenant: str :param tenant: the tenant name :rtype: str :return: the state abbreviation

post_challenge_approved = {'AR': {'post_challenge_cai': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/post_challenge/post_challenge_cai.csv', 'post_challenge_location': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/post_challenge/post_challenge_locations.csv'}, 'AZ': {'post_challenge_cai': 's3://463002504035-shoreline/raw/uploads/Arizona/Post Challenge Locations + CAI/post_challenge_cais_11222024.csv', 'post_challenge_location': 's3://463002504035-shoreline/raw/uploads/Arizona/Post Challenge Locations + CAI/post_challenge_locations_01062025.csv'}, 'MN': {'post_challenge_location': 's3://463002504035-shoreline/raw/uploads/Minnesota/bead_prepare/MN Challenge curing 1-9-25/post_challenge_locations.csv'}, 'OR': {'post_challenge_cai': 's3://463002504035-shoreline/raw/uploads/Oregon/unzipped/post-challenge/post_challenge_cai.csv', 'post_challenge_location': 's3://463002504035-shoreline/raw/uploads/Oregon/unzipped/post-challenge/post_challenge_locations.csv'}}

state_profiles = {'AL': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 42, 'ffa_version_pre_challenge': 31, 'fw_mod': False, 'org_id': 4044, 'table_pre_challenge': 'c_cpc_merge_tile_al', 'table_trueup': 'c_cpc_merge_tile_al', 'tenant': 'alabama', 'true_up': False}, 'AR': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 3, 'ffa_version': 33, 'ffa_version_arc': 33, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'notes': 'state has missing CAI name issue', 'org_id': 2439, 'table_pre_challenge': 'c_cpc_merge_tile_ar', 'table_trueup': 'c_cpc_merge_tile_ar_fabric_v4', 'tenant': 'arkansas', 'true_up': True}, 'AZ': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 3, 'ffa_version': 38, 'ffa_version_arc': 59, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'notes': 'Did not do pre-challenge; just started on post-challenge', 'table_post_challenge': '', 'table_pre_challenge': '', 'table_trueup': 'c_bead_merge_tile_az', 'tenant': 'arizona', 'true_up': True}, 'CA': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 4, 'ffa_version': 40, 'ffa_version_arc': 68, 'ffa_version_pre_challenge': 30, 'fw_mod': True, 'notes': 'State has seperated the CAI and BSL into two complete datasets.', 'org_id': 3556, 'table_post_challenge': 'c_cpc_post_challenges_ca', 'table_pre_challenge': 'c_cpc_merge_tile_ca_fabric_v4', 'table_trueup': 'c_cpc_merge_tile_ca_fabric_v4', 'tenant': 'california', 'true_up': False}, 'CT': {'dsl_mod': True, 'fabric_version': 3, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'notes': 'State has seperated the CAI and BSL into two complete datasets.', 'org_id': 2853, 'table_pre_challenge': 'c_cpc_merge_tile_ct', 'table_trueup': 'c_cpc_merge_tile_ct', 'tenant': 'connecticut', 'true_up': False}, 'DC': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 30, 'ffa_version_pre_challenge': 30, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_dc', 'table_trueup': 'c_cpc_merge_tile_dc', 'tenant': 'district_of_columbia', 'true_up': False}, 'DE': {'dsl_mod': True, 'fabric_version': 3, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'org_id': 2532, 'table_pre_challenge': 'c_cpc_merge_tile_de', 'table_trueup': 'c_cpc_merge_tile_de', 'tenant': 'delaware', 'true_up': False}, 'GU': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 4, 'ffa_version': 38, 'ffa_version_arc': 59, 'ffa_version_pre_challenge': 38, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_gu', 'table_trueup': 'c_cpc_merge_tile_gu', 'table_trueup_v5': 'c_cpc_merge_tile_gu_fabric_v5', 'tenant': 'guam', 'true_up': True}, 'HI': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 42, 'ffa_version_pre_challenge': 31, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_hi_fabric_v4', 'table_trueup': 'c_cpc_merge_tile_hi_fabric_v4', 'tenant': 'hawaii', 'true_up': False}, 'ID': {'dsl_mod': False, 'fabric_version': 4, 'fabric_version_arc': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 42, 'ffa_version_arc': 42, 'ffa_version_pre_challenge': 42, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': '', 'table_trueup': '', 'tenant': 'idaho', 'true_up': False}, 'IL': {'dsl_mod': False, 'fabric_version': 3, 'fabric_version_arc': 3, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_arc': 29, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': '', 'table_trueup': '', 'tenant': 'illinois', 'true_up': False}, 'IN': {'dsl_mod': True, 'fabric_version': 3, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_arc': 59, 'ffa_version_pre_challenge': 17, 'fw_mod': False, 'org_id': 1944, 'table_pre_challenge': 'c_cpc_merge_tile_in', 'table_trueup': 'c_cpc_merge_tile_in', 'tenant': 'indiana', 'true_up': False}, 'KS': {'dsl_mod': True, 'fabric_version': 2, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 2, 'ffa_version': 56, 'ffa_version_arc': 56, 'ffa_version_pre_challenge': 17, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_ks', 'table_trueup': 'c_cpc_merge_tile_ks', 'tenant': 'kansas', 'true_up': False}, 'LA': {'dsl_mod': True, 'fabric_version': 3, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_arc': 66, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_la', 'table_trueup': 'c_cpc_merge_tile_la_fabric_v3', 'tenant': 'louisiana', 'true_up': True}, 'MN': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 4, 'ffa_version': 32, 'ffa_version_arc': 59, 'ffa_version_pre_challenge': 32, 'fw_mod': True, 'org_id': 0, 'table_pre_challenge': '', 'table_trueup': 'c_bead_merge_tile_mn', 'tenant': 'minnesota', 'true_up': False}, 'MP': {'dsl_mod': True, 'fabric_version': 6, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 6, 'ffa_version': 92, 'ffa_version_arc': 92, 'ffa_version_pre_challenge': 92, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': '', 'table_trueup': '', 'tenant': 'northern_mariana_islands', 'true_up': False}, 'MS': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 4, 'ffa_version': 39, 'ffa_version_arc': 91, 'ffa_version_pre_challenge': 31, 'fw_mod': True, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_ms_fabric_v4', 'table_trueup': 'c_cpc_merge_tile_ms_fabric_v4', 'tenant': 'mississippi', 'true_up': False}, 'MT': {'dsl_mod': True, 'fabric_version': 3, 'fabric_version_pre_challenge': 3, 'ffa_version': 19, 'ffa_version_pre_challenge': 19, 'fw_mod': False, 'org_id': 2515, 'table_pre_challenge': 'c_cpc_merge_tile_mt', 'table_trueup': 'c_cpc_merge_tile_mt', 'tenant': 'montana', 'true_up': False}, 'NC': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 33, 'ffa_version_pre_challenge': 33, 'fw_mod': True, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_nc', 'table_trueup': 'c_cpc_merge_tile_nc', 'tenant': 'north_carolina', 'true_up': False}, 'ND': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 3, 'ffa_version': 34, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'notes': 'Using customized sql to process, no true up table existing now', 'org_id': 2633, 'table_pre_challenge': 'c_cpc_merge_tile_nd', 'table_trueup': 'c_cpc_merge_tile_nd_fabric_v4', 'tenant': 'north_dakota', 'true_up': True}, 'NE': {'dsl_mod': False, 'fabric_version': 6, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 6, 'ffa_version': 68, 'ffa_version_arc': 68, 'ffa_version_pre_challenge': 68, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': '', 'table_trueup': '', 'tenant': 'nebraska', 'true_up': False}, 'NH': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 31, 'ffa_version_pre_challenge': 31, 'fw_mod': False, 'org_id': 3470, 'table_pre_challenge': 'c_cpc_merge_tile_nh', 'table_trueup': 'c_cpc_merge_tile_nh', 'tenant': 'new_hampshire', 'true_up': False}, 'NJ': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 42, 'ffa_version_pre_challenge': 30, 'fw_mod': True, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_nj', 'table_trueup': 'c_cpc_merge_tile_nj', 'tenant': 'new_jersey', 'true_up': False}, 'NM': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 3, 'ffa_version': 36, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'org_id': 2689, 'table_pre_challenge': 'c_cpc_merge_tile_nm', 'table_trueup': 'c_cpc_merge_tile_nm_fabric_v4', 'tenant': 'new_mexico', 'true_up': True}, 'NV': {'dsl_mod': True, 'fabric_version': 2, 'fabric_version_pre_challenge': 2, 'ffa_version': 16, 'ffa_version_pre_challenge': 16, 'fw_mod': False, 'org_id': 2315, 'table_pre_challenge': 'c_cpc_merge_tile_nv', 'table_trueup': 'c_cpc_merge_tile_nv', 'tenant': 'nevada', 'true_up': False}, 'NY': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 3, 'ffa_version': 34, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'org_id': 2563, 'table_pre_challenge': 'c_cpc_merge_tile_ny', 'table_trueup': 'c_cpc_merge_tile_ny_fabric_v4', 'tenant': 'new_york', 'true_up': True}, 'OH': {'dsl_mod': True, 'fabric_version': 3, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_pre_challenge': 29, 'fw_mod': True, 'notes': 'customized cellular modification method', 'org_id': 2853, 'table_pre_challenge': 'c_cpc_merge_tile_oh', 'table_trueup': 'c_cpc_merge_tile_oh', 'tenant': 'ohio', 'true_up': False}, 'OK': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 4, 'ffa_version': 31, 'ffa_version_pre_challenge': 33, 'fw_mod': False, 'notes': 'multiple programs at the same time', 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_ok', 'table_trueup': 'c_cpc_merge_tile_ok', 'tenant': 'oklahoma', 'true_up': False}, 'OR': {'dsl_mod': False, 'fabric_version': 3, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_pre_challenge': 28, 'fw_mod': False, 'org_id': 3336, 'table_pre_challenge': 'c_cpc_merge_tile_or', 'table_trueup': 'c_cpc_merge_tile_or', 'tenant': 'oregon', 'true_up': False}, 'PA': {'dsl_mod': False, 'fabric_version': 4, 'fabric_version_pre_challenge': 3, 'ffa_version': 33, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'notes': 'state requires to use mapping file to enhance the data quality after true-up', 'org_id': 3186, 'table_pre_challenge': 'c_cpc_merge_tile_pa', 'table_trueup': 'c_cpc_merge_tile_pa_fabric_v4', 'tenant': 'pennsylvania', 'true_up': True}, 'TX': {'dsl_mod': False, 'fabric_version': 5, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 5, 'ffa_version': 51, 'ffa_version_arc': 51, 'ffa_version_pre_challenge': 43, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_tx', 'table_trueup': 'c_cpc_merge_tile_tx', 'tenant': 'texas', 'true_up': False}, 'UT': {'dsl_mod': True, 'fabric_version': 7, 'fabric_version_arc': 7, 'fabric_version_pre_challenge': 7, 'ffa_version': 76, 'ffa_version_arc': 76, 'ffa_version_pre_challenge': 76, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': '', 'table_trueup': '', 'tenant': 'utah', 'true_up': False}, 'VA': {'fabric_version_arc': 3, 'fabric_version_pre_challenge': 2, 'ffa_version_arc': 75, 'ffa_version_pre_challenge': 16, 'notes': 'Did not do post-challenge with us', 'table_pre_challenge': 'c_cpc_merge_tile_va', 'tenant': 'virginia'}, 'WV': {'dsl_mod': True, 'fabric_version': 2, 'fabric_version_pre_challenge': 2, 'ffa_version': 16, 'ffa_version_pre_challenge': 16, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_wv', 'table_trueup': 'c_cpc_merge_tile_wv', 'tenant': 'west_virginia', 'true_up': False}, 'WY': {'dsl_mod': True, 'fabric_version': 3, 'fabric_version_pre_challenge': 3, 'ffa_version': 29, 'ffa_version_pre_challenge': 29, 'fw_mod': False, 'org_id': 3507, 'table_pre_challenge': 'c_cpc_merge_tile_wy', 'table_trueup': 'c_cpc_merge_tile_wy', 'tenant': 'wyoming', 'true_up': False}, 'XY': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 4, 'ffa_version': 33, 'ffa_version_pre_challenge': 33, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_xy', 'table_trueup': '', 'tenant': 'demo_base', 'true_up': False}, 'ZZ': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_arc': 6, 'fabric_version_pre_challenge': 4, 'ffa_version': 33, 'ffa_version_pre_challenge': 33, 'fw_mod': False, 'org_id': 0, 'table_pre_challenge': 'c_cpc_merge_tile_zz', 'table_trueup': '', 'tenant': 'test_base', 'true_up': False}}

FUNDUtil

This class contains all specific states’ bead funding information

class util.sbo_funding.FUNDUtil

Keep records of all transfered funding file from SBO.

static bdc_dedup_v2()

Deduplication of BDC data.

static bdc_dedup_v3()

Deduplication of BDC data.

static bdc_dedup_v4()

Deduplication of BDC data per fabric v4.

bead_data_post = {'AL': {'dedup': 's3://463002504035-shoreline/raw/uploads/Alabama/AL_Fed_Dedup_BDC_103024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Alabama/AL_CPF1_approved_Dec5_2024_EnforceableCommitments.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Alabama/ABAF_State_enforceable_commitment_AL_Dec2024.csv', 'version_date': '103024'}, 'AR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_Fed_Dedup_BDC_061124.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_StateFundedLocations_Fabric_v4_providerIDs.csv', 'version_date': '061124'}, 'AZ': {'dedup': '', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Arizona/bead_funding/fundingbylocation.csv', 'version_date': '022024'}, 'CA': {'dedup': 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/CA_Fed_Dedup_BDC_100124.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/California/unzipped/funding/PublicHousingAccountProjectLocations_geocoded.xlsx', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/Updated-2024-09-26_FFA_Awarded.geojson', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/Surfnet CASF Project Locations_11042024.csv', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/ca_tbcp_default.csv', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/CAFII_CASF_OverlapLocations_12192024.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/California/unzipped/funding/CASF_approved.geojson', 'version_date': '100124'}, 'GU': {'dedup': 's3://463002504035-shoreline/raw/uploads/Guam/pre-challenge/GU_Fed_Dedup_BDC_090624.csv', 'dedup_v5': 's3://463002504035-shoreline/raw/uploads/Guam/bead_post/GU_Fed_Dedup_BDC_121024.csv', 'version_date': '090624'}, 'HI': {'dedup': 's3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/Funding/Funding Data/HI_Fed_Dedup_BDC_103024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/Funding/Funding Data/Un_Underserved_in_DHHL_dedup_20231231.csv'], 'version_date': '103024'}, 'IN': {'dedup': 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/IN_State_and_Fed_Dedup_BDC_111423.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/funding_support/', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/ICP11_fcc_ids.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/495MissingNLCLocs.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/MercuryUniqueIDsFiberv1.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/MercuryFWALocIDs.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/MercuryRDOFDefault11_18_24.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/Mercury_Davies_Martin.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/bad_ec_in_daviess/bad_EC_IN_Daviess.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/NEW_NLC_ICP_ARPA_enforceable_commitments(in).csv', 'version_date': '111423'}, 'MN': {'dedup': 's3://463002504035-shoreline/raw/uploads/Minnesota/bead_prepare/MN_Fed_Dedup_20240711/MN_Fed_Dedup_BDC_061124.csv', 'other': [], 'state': '', 'version_date': '061124'}, 'MS': {'dedup': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/FED_DEDUP/MS_Fed_Dedup_BDC_091724.csv', 'dedup_v5': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/FED_DEDUP/MS_Fed_Dedup_BDC_112524.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/CPF Round 1 Provisional Awarded BSLs.csv', 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/cpf_r1_ancillary_bsls.csv', 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/MS_Programmatic_Review_011025_SBO_fed_ee.csv', 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/beam_bip_combined_060624_updated.csv'], 'state': '', 'version_date': '091724'}, 'NC': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_Fed_Dedup_BDC_062524_LC.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/LumosOrangeCounty_UPDATEDfundedBSLv4_250321.csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_state_funded_locations_v4_BEADprotectGREATRounds2and3AddOn_250324(in).csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NCBIO_state_funded_BSLv4_GREAT1000013247_withdrawn_250424 (1).csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/CharterTribalRDOFWithdrawals_FromLetterToFCC_250425.csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_state_funded_ATMC_v4_1000013247_withdrawal_250430.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_state_funded_locations_v4_BEADprotect_250324.csv', 'version_date': '062524post'}, 'ND': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthDakota/v3_v4_trueup/ND_Fed_Dedup_BDC_070924.csv', 'other': [], 'state': '', 'version_date': '070924'}, 'NH': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_Fed_Dedup_BDC_100124.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_Fed_Dedup_BMGI_101824.csv', 's3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_Fed_Dedup_Hub66Haverhill_101824.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_State_Dedup_Hub66Haverhill_101824.csv', 'version_date': '100124'}, 'NJ': {'dedup': '', 'other': ['s3://463002504035-shoreline/raw/uploads/NewJersey/unzipped/bead_post/simplified_dedupe_nj_v4_03052025bfm 2.csv'], 'state': '', 'version_date': '103024'}, 'NM': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/NM_Fed_Dedup_BDC_080624.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/USDA_Reconnect_5_Awards.xlsx', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/BEAD_ManualFlipDEFAULTS_BSLs_forReadyNet.csv', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/BEAD_ManualFlipsNEW_BSLs_forReadyNet.csv', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/Summary_of_Grant_ManualFlips.csv', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/NM_Programmatic_Review_020325_SBO.xlsx'], 'state': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/NM_StateDedup_BSLv4Updated20240822_CNM-PRC.geojson', 'version_date': '080624'}, 'NY': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/v4/funding/NY_Fed_Dedup_BDC_070924.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/post_challenge_commitments_to_add_10222024.csv', 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/post_challenge_commitments_to_remove_10222024.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/v4/state/post-challenge local commitments 7.24.24.csv', 'version_date': '070924'}, 'OH': {'dedup': 's3://463002504035-shoreline/raw/uploads/Ohio/unzipped/bead_post/2503_OH_Fed_Dedup_BDC_050524-1718297793438.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Ohio/unzipped/Funding Data/altafiber-CPF-post-challenge.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Ohio/unzipped/bead_post/OH-State-ORBEG-post-challenge-dedupe.csv', 'version_date': '050524'}, 'OK': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/OK_Fed_Dedup/OK_Fed_Dedup_BDC_120224.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/cpf_2025/Updated_CPF_Agreements.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/BEAD_prepare/OK_CUST-2856_dup_funding_upd.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/UpdatedCPF_Ready.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/SLFRF_V4_S2.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/TBCP2_choctaw.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Muscogee Creek-20250317T020546Z-001/Muscogee Creek/fundingdata_project14548_J24_05mar2025/areapolygons_program41_project14548_J24_05mar2025.gpkg', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Osage Nation/fundingdata_project14551_J24_05mar2025/locationlist_program27_project14551_J24_05mar2025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Osage Nation/fundingdata_project14550_J24_05mar2025/areapolygons_program27_project14550_J24_05mar2025.gpkg', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Muscogee Creek-20250317T020546Z-001/Muscogee Creek/fundingdata_project14549_J24_05mar2025/locationlist_program41_project14549_J24_05mar2025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Seminole Nation/Seminole Nation_proposed BSLs_03102025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Cherokee Nation/Cherokee Nation_proposed BSLs_above100_20_03182025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Cherokee Nation/Cherokee Nation_proposed BSLs_below100_20_03182025.csv'], 'version_date': '120224'}, 'OR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oregon/unzipped/Funding/OR_Fed_Dedup_BDC_050524.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Oregon/unzipped/Funding/post-challenge-revision/'], 'state': '', 'version_date': '050524'}, 'TX': {'dedup': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/simplified_dedupe_tx_v5_31925BFM.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Bastrop Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Burleson Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Caldwell Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Carson Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Coleman Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Colorado Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Edwards Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Fall Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Fannin Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Irion Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Karnes Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Kenedy Location Data.csv', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Liberty Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Marion Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Morris Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Navarro Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Newton Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Reagan Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Roberts Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Throckmorton Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Trinity Location Data.xlsx'], 'state': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/Bowie County Master List.csv', 'version_date': '030425'}, 'WY': {'dedup': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY_Fed_Dedup_BDC_050524_ISPResponse_Defaults_08262024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY_ISP_Provided_GrantLocations.xlsx', 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY BEAD - defaulted APRA CPF enforceable commitments.csv', 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY BEAD - new APRA CPF enforceable commitments (1).csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY_StateGrant_ALL_Locations_ver2_04162024.csv', 'version_date': '050524'}}

bead_data_pre = {'AL': {'dedup': 's3://463002504035-shoreline/raw/uploads/Alabama/AL_Fed_Dedup (1)/AL_Fed_Dedup_BDC_051424.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Alabama/Alabama_ABAF_grant_served_locations_revised_July2.xlsx', 'version_date': '051424'}, 'AR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_Fed_Dedup_NicksVersion_March18.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_StateFundedLocationsList_FabricV3_2_March18_2024.xlsx', 'version_date': '072123'}, 'AZ': {'dedup': 's3://463002504035-shoreline/raw/uploads/Arizona/NBAM_Deduplication_Data/AZ_State_and_Fed_Dedup_v7/AZ_State_and_Fed_Dedup_BDC_022024.csv', 'other': [], 'state': '', 'version_date': '022024'}, 'CA': {'dedup': 's3://463002504035-shoreline/raw/uploads/California/unzipped/CA_Fed_Dedup/CA_Fed_Dedup_BDC_051424.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/California/unzipped/funding/PublicHousingAccountProjectLocations_geocoded.xlsx'], 'state': 's3://463002504035-shoreline/raw/uploads/California/unzipped/funding/CASF_approved.geojson', 'version_date': '051424'}, 'GU': {'dedup': 's3://463002504035-shoreline/raw/uploads/Guam/pre-challenge/GU_Fed_Dedup_BDC_090624.csv', 'version_date': '090624'}, 'HI': {'dedup': 's3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/V4/HI_Fed_Dedup_BDC_051424.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/V4/Funding/'], 'version_date': '051424'}, 'IN': {'dedup': 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/IN_State_and_Fed_Dedup/IN_State_and_Fed_Dedup_BDC_020724.csv', 'other': [], 'state': '', 'version_date': '020724'}, 'MS': {'dedup': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/Funding/MS_Fed_Dedup_BDC_052824.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/cpf_round2_data_source/'], 'state': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/Funding/MS_CPF_Data_clean.csv', 'version_date': '052824'}, 'NC': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/Federal_Deduplication/NC_Fed_Dedup_BDC_062524.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/NC_Combined_State_And_Local_Funded_Locations_BSLv4_240801.csv', 'version_date': '062524'}, 'ND': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthDakota/ND_Fed_Dedup_BDC_121223.csv', 'other': [], 'state': '', 'version_date': '121223'}, 'NJ': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewJersey/unzipped/funding/NJ_Fed_Dedup_BDC_051424.csv', 'version_date': '051424'}, 'NM': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/NM_Fed_Dedup_BDC_020724_UpdatedFeb23.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/StateGrants_PRC_CNM_Project_Areas_NTIA_20240227_V2.geojson', 'version_date': '020724'}, 'NY': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/NY_Fed_Dedup_BDC_022024.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/commitment_list_03082024.csv', 'version_date': '022024'}, 'OH': {'dedup': 's3://463002504035-readydatalake/raw/uploads/boss_orgs/2503/OH_Fed_Dedup_BDC_050524-1718297793438.csv', 'other': [], 'state': '', 'version_date': '050524_post'}, 'OK': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/v4/OK_Fed_Dedup_BDC_052824.csv', 'fabric_version': '4', 'other': ['s3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/BEAD_prepare/SLFRFlocations.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/CPF_prepare/ARPA_SLFRF_Awarded.csv', 'version_date': '052824'}, 'OR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oregon/unzipped/Funding/OR_Fed_Dedup_BDC_030524.csv', 'other': [], 'state': '', 'version_date': '030524'}, 'TX': {'dedup': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/TX_Fed_Dedup_BDC_111324.csv', 'dedup_v4': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/TX_Fed_Dedup_BDC_103024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/BOOT II_combined.csv', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/cdbg.csv'], 'other_v4': ['s3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/BOOT II_combined.csv', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/CDBG_v4.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/BOOT I.csv', 'version_date': '111324'}, 'WY': {'dedup': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/Funding/WY_Fed_Dedup_BDC_030524.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/Funding/WY_FedGrant_ALL_Locations_ver3_04262024.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/Funding/WY_StateGrant_ALL_Locations_ver2_04162024.csv', 'version_date': '030524'}, 'XY': {'dedup': '', 'other': [], 'state': '', 'version_date': '052824'}}

cpf_data_post = {'MS': {}, 'OK': {}}

cpf_data_pre = {'MS': {'dedup': '', 'other': [], 'state': '', 'version_date': '052824'}, 'OK': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/v4/OK_Fed_Dedup_BDC_052824.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/CPF_prepare/ARPA_SLFRF_Awarded.csv', 'version_date': '052824'}, 'XY': {'dedup': '', 'other': [], 'state': '', 'version_date': '052824'}}

data_postcha = {'bead': {'AL': {'dedup': 's3://463002504035-shoreline/raw/uploads/Alabama/AL_Fed_Dedup_BDC_103024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Alabama/AL_CPF1_approved_Dec5_2024_EnforceableCommitments.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Alabama/ABAF_State_enforceable_commitment_AL_Dec2024.csv', 'version_date': '103024'}, 'AR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_Fed_Dedup_BDC_061124.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_StateFundedLocations_Fabric_v4_providerIDs.csv', 'version_date': '061124'}, 'AZ': {'dedup': '', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Arizona/bead_funding/fundingbylocation.csv', 'version_date': '022024'}, 'CA': {'dedup': 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/CA_Fed_Dedup_BDC_100124.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/California/unzipped/funding/PublicHousingAccountProjectLocations_geocoded.xlsx', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/Updated-2024-09-26_FFA_Awarded.geojson', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/Surfnet CASF Project Locations_11042024.csv', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/ca_tbcp_default.csv', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/CAFII_CASF_OverlapLocations_12192024.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/California/unzipped/funding/CASF_approved.geojson', 'version_date': '100124'}, 'GU': {'dedup': 's3://463002504035-shoreline/raw/uploads/Guam/pre-challenge/GU_Fed_Dedup_BDC_090624.csv', 'dedup_v5': 's3://463002504035-shoreline/raw/uploads/Guam/bead_post/GU_Fed_Dedup_BDC_121024.csv', 'version_date': '090624'}, 'HI': {'dedup': 's3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/Funding/Funding Data/HI_Fed_Dedup_BDC_103024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/Funding/Funding Data/Un_Underserved_in_DHHL_dedup_20231231.csv'], 'version_date': '103024'}, 'IN': {'dedup': 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/IN_State_and_Fed_Dedup_BDC_111423.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/funding_support/', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/ICP11_fcc_ids.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/495MissingNLCLocs.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/MercuryUniqueIDsFiberv1.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/MercuryFWALocIDs.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/MercuryRDOFDefault11_18_24.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/Mercury_Davies_Martin.csv', 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/bad_ec_in_daviess/bad_EC_IN_Daviess.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/funding_post_challenge/NEW_NLC_ICP_ARPA_enforceable_commitments(in).csv', 'version_date': '111423'}, 'MN': {'dedup': 's3://463002504035-shoreline/raw/uploads/Minnesota/bead_prepare/MN_Fed_Dedup_20240711/MN_Fed_Dedup_BDC_061124.csv', 'other': [], 'state': '', 'version_date': '061124'}, 'MS': {'dedup': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/FED_DEDUP/MS_Fed_Dedup_BDC_091724.csv', 'dedup_v5': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/FED_DEDUP/MS_Fed_Dedup_BDC_112524.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/CPF Round 1 Provisional Awarded BSLs.csv', 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/cpf_r1_ancillary_bsls.csv', 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/MS_Programmatic_Review_011025_SBO_fed_ee.csv', 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/bead_post_challenge/beam_bip_combined_060624_updated.csv'], 'state': '', 'version_date': '091724'}, 'NC': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_Fed_Dedup_BDC_062524_LC.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/LumosOrangeCounty_UPDATEDfundedBSLv4_250321.csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_state_funded_locations_v4_BEADprotectGREATRounds2and3AddOn_250324(in).csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NCBIO_state_funded_BSLv4_GREAT1000013247_withdrawn_250424 (1).csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/CharterTribalRDOFWithdrawals_FromLetterToFCC_250425.csv', 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_state_funded_ATMC_v4_1000013247_withdrawal_250430.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/funding/NC_state_funded_locations_v4_BEADprotect_250324.csv', 'version_date': '062524post'}, 'ND': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthDakota/v3_v4_trueup/ND_Fed_Dedup_BDC_070924.csv', 'other': [], 'state': '', 'version_date': '070924'}, 'NH': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_Fed_Dedup_BDC_100124.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_Fed_Dedup_BMGI_101824.csv', 's3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_Fed_Dedup_Hub66Haverhill_101824.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/NewHampshire/unzipped/NH_State_Dedup_Hub66Haverhill_101824.csv', 'version_date': '100124'}, 'NJ': {'dedup': '', 'other': ['s3://463002504035-shoreline/raw/uploads/NewJersey/unzipped/bead_post/simplified_dedupe_nj_v4_03052025bfm 2.csv'], 'state': '', 'version_date': '103024'}, 'NM': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/NM_Fed_Dedup_BDC_080624.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/USDA_Reconnect_5_Awards.xlsx', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/BEAD_ManualFlipDEFAULTS_BSLs_forReadyNet.csv', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/BEAD_ManualFlipsNEW_BSLs_forReadyNet.csv', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/Summary_of_Grant_ManualFlips.csv', 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/bead_post/NM_Programmatic_Review_020325_SBO.xlsx'], 'state': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/NM_StateDedup_BSLv4Updated20240822_CNM-PRC.geojson', 'version_date': '080624'}, 'NY': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/v4/funding/NY_Fed_Dedup_BDC_070924.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/post_challenge_commitments_to_add_10222024.csv', 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/post_challenge_commitments_to_remove_10222024.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/v4/state/post-challenge local commitments 7.24.24.csv', 'version_date': '070924'}, 'OH': {'dedup': 's3://463002504035-shoreline/raw/uploads/Ohio/unzipped/bead_post/2503_OH_Fed_Dedup_BDC_050524-1718297793438.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Ohio/unzipped/Funding Data/altafiber-CPF-post-challenge.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Ohio/unzipped/bead_post/OH-State-ORBEG-post-challenge-dedupe.csv', 'version_date': '050524'}, 'OK': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/OK_Fed_Dedup/OK_Fed_Dedup_BDC_120224.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/cpf_2025/Updated_CPF_Agreements.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/BEAD_prepare/OK_CUST-2856_dup_funding_upd.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/UpdatedCPF_Ready.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/SLFRF_V4_S2.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/TBCP2_choctaw.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Muscogee Creek-20250317T020546Z-001/Muscogee Creek/fundingdata_project14548_J24_05mar2025/areapolygons_program41_project14548_J24_05mar2025.gpkg', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Osage Nation/fundingdata_project14551_J24_05mar2025/locationlist_program27_project14551_J24_05mar2025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Osage Nation/fundingdata_project14550_J24_05mar2025/areapolygons_program27_project14550_J24_05mar2025.gpkg', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Muscogee Creek-20250317T020546Z-001/Muscogee Creek/fundingdata_project14549_J24_05mar2025/locationlist_program41_project14549_J24_05mar2025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Seminole Nation/Seminole Nation_proposed BSLs_03102025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Cherokee Nation/Cherokee Nation_proposed BSLs_above100_20_03182025.csv', 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/bead_post/Cherokee Nation/Cherokee Nation_proposed BSLs_below100_20_03182025.csv'], 'version_date': '120224'}, 'OR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oregon/unzipped/Funding/OR_Fed_Dedup_BDC_050524.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Oregon/unzipped/Funding/post-challenge-revision/'], 'state': '', 'version_date': '050524'}, 'TX': {'dedup': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/simplified_dedupe_tx_v5_31925BFM.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Bastrop Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Burleson Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Caldwell Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Carson Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Coleman Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Colorado Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Edwards Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Fall Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Fannin Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Irion Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Karnes Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Kenedy Location Data.csv', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Liberty Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Marion Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Morris Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Navarro Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Newton Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Reagan Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Roberts Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Throckmorton Location Data.xlsx', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/BOOT II/Trinity Location Data.xlsx'], 'state': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/funding/Bowie County Master List.csv', 'version_date': '030425'}, 'WY': {'dedup': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY_Fed_Dedup_BDC_050524_ISPResponse_Defaults_08262024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY_ISP_Provided_GrantLocations.xlsx', 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY BEAD - defaulted APRA CPF enforceable commitments.csv', 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY BEAD - new APRA CPF enforceable commitments (1).csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/post_challenge_funding_update/WY_StateGrant_ALL_Locations_ver2_04162024.csv', 'version_date': '050524'}}, 'cpf': {'MS': {}, 'OK': {}}, 'ulfw': {}}

data_precha = {'bead': {'AL': {'dedup': 's3://463002504035-shoreline/raw/uploads/Alabama/AL_Fed_Dedup (1)/AL_Fed_Dedup_BDC_051424.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Alabama/Alabama_ABAF_grant_served_locations_revised_July2.xlsx', 'version_date': '051424'}, 'AR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_Fed_Dedup_NicksVersion_March18.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/dedup/AR_StateFundedLocationsList_FabricV3_2_March18_2024.xlsx', 'version_date': '072123'}, 'AZ': {'dedup': 's3://463002504035-shoreline/raw/uploads/Arizona/NBAM_Deduplication_Data/AZ_State_and_Fed_Dedup_v7/AZ_State_and_Fed_Dedup_BDC_022024.csv', 'other': [], 'state': '', 'version_date': '022024'}, 'CA': {'dedup': 's3://463002504035-shoreline/raw/uploads/California/unzipped/CA_Fed_Dedup/CA_Fed_Dedup_BDC_051424.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/California/unzipped/funding/PublicHousingAccountProjectLocations_geocoded.xlsx'], 'state': 's3://463002504035-shoreline/raw/uploads/California/unzipped/funding/CASF_approved.geojson', 'version_date': '051424'}, 'GU': {'dedup': 's3://463002504035-shoreline/raw/uploads/Guam/pre-challenge/GU_Fed_Dedup_BDC_090624.csv', 'version_date': '090624'}, 'HI': {'dedup': 's3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/V4/HI_Fed_Dedup_BDC_051424.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Hawaii/unzipped/V4/Funding/'], 'version_date': '051424'}, 'IN': {'dedup': 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/IN_State_and_Fed_Dedup/IN_State_and_Fed_Dedup_BDC_020724.csv', 'other': [], 'state': '', 'version_date': '020724'}, 'MS': {'dedup': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/Funding/MS_Fed_Dedup_BDC_052824.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/cpf_round2_data_source/'], 'state': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/Funding/MS_CPF_Data_clean.csv', 'version_date': '052824'}, 'NC': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/Federal_Deduplication/NC_Fed_Dedup_BDC_062524.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/NorthCarolina/NC_Combined_State_And_Local_Funded_Locations_BSLv4_240801.csv', 'version_date': '062524'}, 'ND': {'dedup': 's3://463002504035-shoreline/raw/uploads/NorthDakota/ND_Fed_Dedup_BDC_121223.csv', 'other': [], 'state': '', 'version_date': '121223'}, 'NJ': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewJersey/unzipped/funding/NJ_Fed_Dedup_BDC_051424.csv', 'version_date': '051424'}, 'NM': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/NM_Fed_Dedup_BDC_020724_UpdatedFeb23.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/NewMexico/unzipped/StateGrants_PRC_CNM_Project_Areas_NTIA_20240227_V2.geojson', 'version_date': '020724'}, 'NY': {'dedup': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/NY_Fed_Dedup_BDC_022024.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/NewYork/unzipped/Funding/commitment_list_03082024.csv', 'version_date': '022024'}, 'OH': {'dedup': 's3://463002504035-readydatalake/raw/uploads/boss_orgs/2503/OH_Fed_Dedup_BDC_050524-1718297793438.csv', 'other': [], 'state': '', 'version_date': '050524_post'}, 'OK': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/v4/OK_Fed_Dedup_BDC_052824.csv', 'fabric_version': '4', 'other': ['s3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/BEAD_prepare/SLFRFlocations.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/CPF_prepare/ARPA_SLFRF_Awarded.csv', 'version_date': '052824'}, 'OR': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oregon/unzipped/Funding/OR_Fed_Dedup_BDC_030524.csv', 'other': [], 'state': '', 'version_date': '030524'}, 'TX': {'dedup': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/TX_Fed_Dedup_BDC_111324.csv', 'dedup_v4': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/TX_Fed_Dedup_BDC_103024.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/BOOT II_combined.csv', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/cdbg.csv'], 'other_v4': ['s3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/BOOT II_combined.csv', 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/CDBG_v4.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/bead-pre-challenge/state_funding_data/BOOT I.csv', 'version_date': '111324'}, 'WY': {'dedup': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/Funding/WY_Fed_Dedup_BDC_030524.csv', 'other': ['s3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/Funding/WY_FedGrant_ALL_Locations_ver3_04262024.csv'], 'state': 's3://463002504035-shoreline/raw/uploads/Wyoming/unzipped/Funding/WY_StateGrant_ALL_Locations_ver2_04162024.csv', 'version_date': '030524'}, 'XY': {'dedup': '', 'other': [], 'state': '', 'version_date': '052824'}}, 'cpf': {'MS': {'dedup': '', 'other': [], 'state': '', 'version_date': '052824'}, 'OK': {'dedup': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/v4/OK_Fed_Dedup_BDC_052824.csv', 'other': [], 'state': 's3://463002504035-shoreline/raw/uploads/Oklahoma/unzipped/CPF_prepare/ARPA_SLFRF_Awarded.csv', 'version_date': '052824'}, 'XY': {'dedup': '', 'other': [], 'state': '', 'version_date': '052824'}}, 'ulfw': {'CA': {'dedup': '', 'other': ['s3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/tbcp_indian_tribes_default.csv', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/tbcp_yurok_default.csv'], 'state': '', 'version_date': '06162025'}, 'MS': {'dedup': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/indicated_locations/dedupe_request_MS_2025-05-28.csv', 'other': [], 'state': '', 'version_date': '05282025'}}}

static get_funding_data(state, program, program_stage)

Initialize the class. state: the state abbreviation program: the grant program program_stage: the stage of the program, pre or post

Return type:

dict

static state_funding(version='3')

State funding.

static state_other_funding(version='3')

State funding. These are additional fundint files that may in different formats

ulfw_data_pre = {'CA': {'dedup': '', 'other': ['s3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/tbcp_indian_tribes_default.csv', 's3://463002504035-shoreline/raw/uploads/California/unzipped/bead_postchallenge/tbcp_yurok_default.csv'], 'state': '', 'version_date': '06162025'}, 'MS': {'dedup': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/indicated_locations/dedupe_request_MS_2025-05-28.csv', 'other': [], 'state': '', 'version_date': '05282025'}}

SpatialUtil

This class contains utilities to download data from an ESRI endpoints.

class util.spatial.SpatialUtil

static download_from_esri(url, fields=None, where=None, limit=None, **kwargs)

A util to download data from an ESRI endpoints. This is a wrapper around geopandas.read_file that handles the ESRI API’s max record count. It loads the data in chunks and concatenates them into a single GeoDataFrame. The final GeoDataFrame is returned and can get large if the limit is not set. One approach to do this is to set the limit to a small number and then filter the data in the where clause. or order them by a field and set the limit to the number of records you want to download.

Note: This function uses the pyogrio engine to read the data. This is a faster engine than the default fiona engine. To use this engine, you need to install the pyogrio package. You can install it using pip install pyogrio. Note: It uses get request. it might reject your request if you send large request body geometries. can use post for such cases but it is not implemented here. ask Majid for help if you need it.

Parameters:

• url (str) – Esri endpoint url for example https://services.arcgis.com/8DAUcrpQcpyLMznu/arcgis/rest/services/COVID19_Cases_by_Zip_Code/FeatureServer/0

• fields (_type_, optional) – the list of the fields to return, defaults to None which loads all fields by passing * to the API

• where (_type_, optional) – the filter criteria, defaults to None which means no filter and it sends 1=1 to the API

• limit (_type_, optional) – the max number of records to get, defaults to all the values in the feaure layer

Returns:

A pandas DataFrame with the data

Return type:

_type_

Example:

```python

url = “https://pcrdgis.agriculture.purdue.edu/arcgis/rest/services/DDI/Digital_Divide_Index_COUNTY_USA_2020/MapServer/0” gdf=SpatialUtil.download_from_esri(url=url)

```

RabbitMQUtil

This class contains utilities for interacting with RabbitMQ.

class util.rabbitmq.RabbitMQUtil

Utilities work working RabbitMQ.

static get_channel()

static get_connection()

Get connection to rabbitmq.

static publish_messages(exchange, queue_name, messages)

static publish_rad_messages(queue_name, messages)

MongoDbUtil

This class contains utilities for interacting with MongoDB.

class util.mongodb.MongoDbUtil

Utilities work working mongodb.

static get_discovery_db_connection()

Get connection to discovery database. :return: connection string

static mongodb_find(mongo_collection, query, find_one=False, mongo_db=None)

Run a mongo find query and returns the results. param mongo_collection: name of the collection param query: query to run param find_one: if True, return only the first result param mongo_db: name of the database

https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.find

Return type:

Optional[Any]

static mongodb_get_collection(mongo_collection, mongo_db=None)

Get a collection from a mongo database. param mongo_collection: name of the collection param mongo_db: name of the database return: collection object

static mongodb_insert_one(mongo_collection, doc, mongo_db=None, **kwargs)

Insert a single document into a mongo collection.

https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.insert_one

Return type:

Any

MetadataUtil

This class contains utilities for managing metadata in PostgreSQL for Airflow DAGs.

class util.metadata_pg.MetadataUtil

Manages metadata in PostgreSQL for Airflow DAGs.

static get_pgmeta(dag_id, metadata_key, mode='latest', month=None, limit=None, parname=None)

Retrieves metadata based on specified parameters and mode.

Modes: - ‘latest’: Latest entry - ‘max’: Maximum value - ‘all’: All entries - ‘fafs_month’: FAFS data for a specific month - ‘fafs_max’: Maximum FAFS versions - ‘fafs_latest’: Latest FAFS versions

Return type:

Union[str, None, Dict[str, Any], List[Dict[str, Any]]]

static get_pgmeta_all_versions(dag_id, version_key, limit=None)

Retrieves all versions for a DAG, with optional limit.

Return type:

List[Dict[str, Any]]

static get_pgmeta_fafs_latest_versions(dag_id, parname=None)

Retrieves latest FAFS versions from the most recent record, optionally for a specific parname.

Return type:

Union[Dict[str, Dict[str, Union[str, int]]], Dict[str, Union[str, int]]]

static get_pgmeta_fafs_max_versions(dag_id, parname=None)

Retrieves maximum FAFS versions, optionally for a specific parname.

Return type:

Union[Dict[str, Dict[str, Union[str, int]]], Dict[str, Union[str, int]]]

static get_pgmeta_fafs_versions(dag_id, month=None)

Retrieves FAFS versions for a specific month or the latest.

Return type:

Dict[str, List[Dict[str, Union[str, int]]]]

static get_pgmeta_latest_version(dag_id, version_key)

Retrieves the latest version for a DAG.

Return type:

Optional[str]

static get_pgmeta_max_version(dag_id, version_key)

Retrieves the maximum version for a DAG.

Return type:

Optional[str]

static store_pgmeta(dag_id, metadata_key, metadata_value, execution_date, json_data=None)

Return type:

None

static store_pgmeta_fafs(dag_id, month, version_info, execution_date)

Stores FAFS metadata for a given month.

static store_pgmeta_version(dag_id, version_key, version_value, execution_date)

Stores version metadata for DAGs.

GitHubUtil

This class contains utilities for interacting with GitHub API.

class util.github.GitHubUtil

GitHub utility functions to interact with GitHub API

static get_access_token(app_id, private_key, installation_id)

Get installation access token for GitHub App

Parameters:

• app_id (str) – GitHub App ID

• private_key (str) – GitHub App private key

• installation_id (str) – GitHub App installation ID

Returns:

Installation access token

Return type:

str

static get_github_creds()

Retrieve GitHub App credentials from AWS Secrets Manager Parse fields for custom attributes

Returns:

GitHub App credentials

Return type:

dict

FormatUtil

This class contains utilities for formatting and converting between Python dtypes such as datetime, string, etc.

class util.format.FormatUtil

Util to help with formatting/converting between python dtypes such as datetime, string, etc.

static to_date_string(date_obj)

Convert various date objects/strings to yyyy-mm-dd format string, for comparison

BeadPOCUtil

This class contains BEAD National Map view (Proof of Concept) post-challenge and pre-challenge configurations.

class util.bead_poc.BeadPOCUtil

DEPRECATED — see module banner above. No longer maintained; consumer DAGs (load_prechal, load_postchal) retired/archived.

Util class for Bead National Map view (Proof of Concept /POC) post-challenge, and pre-challenge configurations. For latest update on BEAD stage, refer to: https://www.ntia.gov/funding-programs/internet-all/broadband-equity-access-and-deployment-bead-program/progress-dashboard

• stage = pending: Challenge process has not concluded; as of 2025-03-28: remaining 8 entities are: FL, GU, MS, NJ, NC, OK, PR, TX, VI (6 of which are READY-partner SBOs)

• stage = adjusting: Challenge process has concluded, but the state is still adjusting the data, including: dedup, remove preliminary award project areas, …

• stage = final: the state has verbally confirmed that the eligiblity list (BSLs and CAIs) is NTIA-approved and finalized; the only change to this list would be: addition of no BEAD location list (at the end of our application process) for Final Proposal (FP) submission/for public comment

static bead_poc_profile(state_abbr)

DEPRECATED (2026-07-23): no longer maintained — consumer DAGs (load_prechal, load_postchal) have been retired/archived. Do not use for new work; values here are stale.

table_pre_challenge: the table name before the challenge table_trueup: the table name after the true-up. Before the challenge results applied. This table is only for data processing. Not for actual frontend functionality org_id: the organization id for the state ffa_version: the post challenge version of the fabric fixed availability data fabric_version: the post challenge version of the fabric data true_up: whether the state has true-up or not dsl_mod: whether the state has DSL modification or not fw_mod: whether the state has Cellular fixed wireless modification or not tenant: the name of the tenant in the grapi database

Parameters:

• state_abbr (str) – the state abbreviation in upper case

• check_pre (str, optional) – whether to check the pre-challenge tables. Default is True.

Return type:

dict

Returns:

the post challenge configuration for the state

NOTE: this function does call the AWS Athena and AWS Glue APIs to check the existence of the tables. BEWARE: When use outside of a DAG/@task i.e. as top level import, must set check_pre to False.

state_profiles = {'AK': {'dev_note': '\n                TODO: confirm pre+post x fa+ffa versions! (emailed jan.cox@alaska.gov and ced.abo.general@alaska.gov but no response yet)\n            ', 'post_fa': 4, 'post_ffa': 33, 'pre_fa': 3, 'pre_ffa': 27, 'sourced_at': '2025-04-11', 'stage': 'adjusting', 'update_note': '\n                The Alaska Broadband Grant Program application period is now open for 120 days between December 16, 2024, and April 14, 2025.\n            ', 'version_note': '\n                - pre-challenge: The Stage 1 review is to allow a preview of the locations to understand the categorization of a BSL or CAI.  The 30-day Stage 1 review runs from April 6, 2024 to May 5, 2024... Stage 2/3/4 Challenge Phase\n                - post-challenge:  Stage 4 -- the Alaska Broadband Office reviews rebuttals from Internet Service Providers (ISPs) -- This phase will begin on July 5, 2024 and will end on August 3, 2024.\n            '}, 'AS': {'dev_note': '\n                Best guesstimate on post-challenge versions!\n            ', 'post_fa': 4, 'post_ffa': 42, 'pre_fa': 2, 'pre_ffa': 15, 'sourced_at': '2025-04-11', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: American Samoa is utilizing the FCC’s Broadband DATA Map as of October 24, 2023\n                - post-challenge: \n            '}, 'CO': {'dev_note': '\n            ', 'post_fa': 3, 'post_ffa': 29, 'pre_fa': 3, 'pre_ffa': 17, 'sourced_at': '2025-04-10', 'stage': 'final', 'update_note': '\n                Final/This BEAD eligible location list was prepared and submitted on 2024-06-28 and was APPROVED by NTIA on 2024-08-05. \n            ', 'version_note': '\n                pre-challenge: from Volume I: using data current as of June 30, 2022 from the National Broadband Map (NOTE: this is fabric version 2), last updated on Nov. 14, 2023\n                post-challenge:  BEAD Eligibility data for Colorado, prepared using FCC Fabric Version 3.2 (07/21/2023) and FCC BDC Version 3 (05/05/2024).\n            '}, 'FL': {'dev_note': '\n                emailed FL SBO broadband@deo.myflorida.com but no response\n            ', 'post_fa': 4, 'post_ffa': 30, 'pre_fa': 3, 'pre_ffa': 19, 'sourced_at': '2025-04-10', 'stage': 'pending', 'update_note': '\n                FloridaCommerce has concluded its BEAD Challenge Process and is now in curing with the National Telecommunications Information Administration (NTIA)\n            ', 'version_note': "\n                pre-challenge (volume I): Florida will use the FCC's Broadband Availability Data from the National Broadband Map from September 30, 2023; in other place: The publication date of the National Broadband Map, December 12, 2023,\n                post-challenge: the FCC Fabric Version 4 dated 12.31.2023 and Broadband Data Collection (BDC) data from 5.14.2024. \n            "}, 'GA': {'dev_note': '\n            ', 'post_fa': 4, 'post_ffa': 35, 'pre_fa': 2, 'pre_ffa': 15, 'sourced_at': '2025-04-11', 'stage': 'final', 'update_note': "\n                The only changes to the location data will be removing locations per NTIA's location removal/non-service guidance. Locations approved for non-service may be found here: https://broadband.georgia.gov/gta-approved-bead-locations-non-service\n            ", 'version_note': '\n                https://broadband.georgia.gov/bead-challenges/supplemental-info The fabric used for the BEAD effort by both NTIA and GTA is the FCC December 2023 version (updated July 23, 2024)\n                Volume 1: https://drive.google.com/file/d/11yQLusmsI9MpLbx3VRvgZ1LQHoSh3JxG/view based on the October 24, 2023, publication date of the National Broadband Map.                \n                - pre-challenge: We used the December 2023 fabric\n                - post-challenge: for our challenge process. We trued up to the July 23, 2024, BDC.\n            '}, 'IA': {'dev_note': '\n                to rerun load_postchal DAG and reload BSL+CAI after the state publishes the update;\n                check back and update post_ffa after IA SBO finalized post-challenge\n            ', 'post_fa': 5, 'post_ffa': 53, 'pre_fa': 3, 'pre_ffa': 29, 'sourced_at': '2025-04-07', 'stage': 'adjusting', 'update_note': '\n                the state does not yet have approval to start the subgrantee selection process. \n                We are in the process of updating the post-challenge data.  Updates are planned.\n            ', 'version_note': '\n                Pre-challenged is June 2023 for both [Fabric and FFA]; post-challenge June 2024 for both\n            '}, 'ID': {'dev_note': '\n                all of the information you are looking for can be found on Link Up Idaho.  Specifically, see the BEAD Subgrantee Selection webpage.\n            ', 'post_fa': 4, 'post_ffa': 30, 'pre_fa': 3, 'pre_ffa': 18, 'sourced_at': '2025-04-10', 'stage': 'final', 'update_note': '\n                The posted CSV files are the final, NTIA approved eligible BSLs.\n            ', 'version_note': '\n                Idaho has only used FCC Fabric Version 4. There has been no formal true-up to other fabric versions.                \n            '}, 'IL': {'dev_note': '\n                - Did not get response from IL SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 3, 'post_ffa': 20, 'pre_fa': 3, 'pre_ffa': 18, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: The state is using data last updated November 28th, 2023, with FCC BSL fabric availability data as\nof June 30th 2023 (Req 1.2.3).\n                - post-challenge: The map is based on the FCC National Broadband Map last updated 01/23/24. Data on broadband-serviceable locations (BSLs) is based on availability reported in the FCC BDC as of June 30, 2023. \n            '}, 'KS': {'dev_note': '\n            ', 'post_fa': 4, 'post_ffa': 42, 'pre_fa': 2, 'pre_ffa': 17, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: from bead.py, 2 and 17\n                - post-challenge: bead.py says fab=2/17 but quite wrong\n            '}, 'KY': {'dev_note': '\n                - Did not get response from the SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 3, 'post_ffa': 25, 'pre_fa': 3, 'pre_ffa': 25, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': "\n                - pre-challenge: From Volume I -- `The data is sourced from the FCC's Broadband DATA Map as of March 19, 2024.`\n                - post-challenge: \n            "}, 'MA': {'dev_note': '\n            ', 'post_fa': 4, 'post_ffa': 42, 'pre_fa': 2, 'pre_ffa': 20, 'sourced_at': '2025-04-11', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': "\n                - pre-challenge: Volume 1) The unserved and underserved location data were identified from v2 of the BSL Fabric released on December 31, 2022, and Broadband Data Collection (BDC) filings, updated on January 9, 2024.\n                - post-challenge: The Massachusetts Challenge Portal was based on the Federal Communication Commission's (FCC) map, with version 4 of the CostQuest fabric data and availability data as of December 31, 2023... https://mapping.massbroadband.org/challengesummary About: Provider services listed for each BSL are based on the BDC data as of 12/31/2023 (published on 10/29/24)\n            "}, 'MD': {'dev_note': '\n                - Did not get response from the SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 5, 'post_ffa': 43, 'pre_fa': 3, 'pre_ffa': 18, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: The unserved and underserved locations identified in this document and its attachments are based on the November 28, 2023, publication date of the National Broadband Map.\n                - post-challenge: What version and date of the FCC Fabric data will be published for the MD BEAD challenge process? The data on the portal will be data from June 30, 2024.\n            '}, 'ME': {'dev_note': '\n                to rerun load_postchal DAG and reload BSL+CAI after the state removes non-locations\n            ', 'post_fa': 3, 'post_ffa': 29, 'pre_fa': 3, 'pre_ffa': 27, 'sourced_at': '2025-04-07', 'stage': 'final', 'update_note': '\n                We are using the Reason Codes from NTIA to remove locations where service has been built as of June 30, 2024 and where ISPs could provide evidence supporting these claims in the FCC BDC.\n                Currently published lists are expected to be final and what we will use to build our tables for the Final Proposal submission.\n            ', 'version_note': '\n                Fabric versions for both pre and post was version 3.2 released July 2023. \n                Corresponding availability data was from the April 3, 2024 update to the service as of June 30, 2023 BDC.\n            '}, 'MI': {'dev_note': '\n                - Did not get response from the SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 3, 'post_ffa': 22, 'pre_fa': 2, 'pre_ffa': 18, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: Michigan High‐Speed Internet Office utilized the Broadband Data Collection (BDC) data as of December 31st, 2022 last updated on November 28, 2023 from the National Broadband Map.\n                - post-challenge: Service data is as of 06/30/2024 published on 02/07/2024 and is the data used during the BEAD challenge process. \n            '}, 'MO': {'dev_note': "\n                Pre-challenge CAI file `CAI_List_DummyLocationIDs.csv` -- `CAI IDs - list of pseudo-location id's, generated for community anchor institutions without an FCC location id, to be able to map these locations to specific coordinates.`\n                The list of CAIs at the end of our challenge process are available as part of the download here: https://ded.mo.gov/sites/ded/files/media/file/2025/03/final_challenge_submission_3_25_25.zip\n            ", 'post_fa': 3, 'post_ffa': 29, 'pre_fa': 3, 'pre_ffa': 23, 'sourced_at': '2025-04-07', 'stage': 'final', 'update_note': '\n                Currently published data is final and approved by NTIA.\n            ', 'version_note': '\n                Fabric Version 3.2 (the only “as-of” date used was the one associated with the version 3.2 fabric – 06-30-2023)\n                    pre-challenge updated publication date: 02-20-2024 (as-of 06-30-2023 data as published on 02-20-2024)\n                    post-challenge updated publication date: October 10-01-2024 (as-of 06-30-2023 data as published on 10-01-2024)\n                All version 3.2 locations not appearing on this list had classification “2”, indicating they were served, that a provider planned to serve them by Dec. 31, 2024 if building with private funds, or that a provider is subject to an enforceable commitment to provide broadband service in the future through a public broadband subsidy program.\n            '}, 'MP': {'dev_note': '\n                - Did not get response from the SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 4, 'post_ffa': 42, 'pre_fa': 3, 'pre_ffa': 19, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: Volume I for public comment and review by the NTIA, BPD utilized the Broadband Data Collection (BDC)\ndata as of June 30, 2023, last updated on December 12, 2023, from the National Broadband Map.\n                - post-challenge: In May 2023, the Federal Communications Commission (FCC) released Version 2 of the National\nBroadband Map. \n            '}, 'NE': {'dev_note': '\n                - Did not get response from the SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 4, 'post_ffa': 42, 'pre_fa': 4, 'pre_ffa': 42, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: \n                - post-challenge: The NBO BEAD-eligible inventory was compiled using FCC Fabric Version 4 data... Locations and classifications were also trued up to Fabric version 4\n            '}, 'RI': {'dev_note': '\n            ', 'post_fa': 3, 'post_ffa': 21, 'pre_fa': 3, 'pre_ffa': 17, 'sourced_at': '2025-04-10', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                pre-challenge: VOLUME 1: V3 of the BSL Fabric and Broadband Data Collection (BDC) filings last updated on November 14, 2023.\n                post-challenge: https://ri.broadbandnavigator.com/map About (About The Map)\n                    based on the FCC Broadband Serviceable Location Fabric (version 3.2). \n                    Availability data is based on ISP submissions to the FCC with an effective date of June 30, 2023, and was downloaded with updates as of January 23, 2024.\n            '}, 'SC': {'dev_note': '\n                - Did not get response from the SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 3, 'post_ffa': 29, 'pre_fa': 3, 'pre_ffa': 25, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: Jun. 30, 2023 (as of Mar. 19, 2024), submissions that were audited through the SC BEAD Map Challenge process, which concluded on Jun. 30, 2024. \n                - post-challenge: The SC BEAD Map Challenge process relied upon FCC Broadband Serviceable Location (BSL) Fabric Jun. 30, 2023, Version 3.2 (pub. Jul. 21, 2023).\n            '}, 'SD': {'dev_note': '\n                - Did not get response from the SBO for my email inquiry (Subject: `Request for Verification: BEAD Program Data and Fabric Versions`)\n            ', 'post_fa': 5, 'post_ffa': 47, 'pre_fa': 3, 'pre_ffa': 18, 'sourced_at': '2025-05-01', 'stage': 'adjusting', 'update_note': '\n            ', 'version_note': '\n                - pre-challenge: The Broadband Team utilized the NTIA National Broadband Map from the Eligible Entity Planning Toolkit, current as of 11/28/23, to identify unserved and underserved locations. \n                - post-challenge: Version 3 of the Costquest Broadband Serviceable Location Fabric...Used with the BDC filing as of June 30, 2023; The Fixed Broadband Availability Data Data as of December 31, 2022 \n            '}, 'TN': {'dev_note': '\n            ', 'post_fa': 3, 'post_ffa': 25, 'pre_fa': 2, 'pre_ffa': 12, 'sourced_at': '2025-04-11', 'stage': 'final', 'update_note': '\n                The public data is final, and will not undergo any changes after applications close (Applications close April 18th and will then undergo public comment), and any actionable public comment submissions are rectified.\n            ', 'version_note': '\n                - pre-challenge: Volume 1) using the first publication of September dated 9/12/2023\n                - post-challenge: https://www.tn.gov/content/dam/tn/ecd/documents/broadband/infrastructure/bead/bead-challenge-process/5-30-24-update/Challenge_Process_FAQs-20240531.pdf We used CostQuest Fabric 3.2 and BDC from June 30, 2023, the March 19, 2024 update.\n            '}, 'UT': {'dev_note': '\n            ', 'post_fa': 3, 'post_ffa': 18, 'pre_fa': 3, 'pre_ffa': 17, 'sourced_at': '2025-04-10', 'stage': 'final', 'update_note': '\n                All the data on our website is final and the only thing that will change is when we add our no BEAD location list at the end of our application process for our Final Proposal submission.\n            ', 'version_note': '\n                - pre-challenge: All of our pre challenges were based on fabric version 3.2.1. \n                - post-challenge: We ran our challenge process based on what had been submitted as far as July, 21,2023 and provider data submitted as of Nov.28,2023.\n            '}, 'VA': {'dev_note': '\n                in bead.py but `Did not do post-challenge with us`\n                Where to access publicly available BSL lists (eligible and ineligible) and CAI lists\n                `These lists can be found on DHCD’s Virginia Open Data Portal page: Pre-Challenge Process...BEAD Recommended De-Obligation Locations`\n            ', 'post_fa': 3, 'post_ffa': 29, 'pre_fa': 2, 'pre_ffa': 6, 'sourced_at': '2025-04-10', 'stage': 'adjusting', 'update_note': '\n                Currently published BEAD data is not final, as we are still in the process of awarding subgrantees.\n            ', 'version_note': '\n                - pre-challenge: The data is sourced from the FCC’s Broadband DATA Map as of June 15, 2023, which is reflective of resolved challenges on the FCC’s Broadband DATA Map which were not resolved on the June 1, 2023 publication of the map\n                - post-challenge: `I don’t actually know what the FFA as_of_date for those Fabric vintages are`\n            '}, 'VT': {'dev_note': '\n            ', 'post_fa': 4, 'post_ffa': 37, 'pre_fa': 3, 'pre_ffa': 20, 'sourced_at': '2025-04-10', 'stage': 'final', 'update_note': '\n            ', 'version_note': '\n                - PRE-CHALLENGE: Note: The VCBB originally published the set of locations eligible for BEAD funding on November 20, 2023 using Version 2 of the CostQuest location fabric. This data was replaced on 2/9/2024 and now uses Version 3.2 of the CostQuest location fabric... is dated 1/9/2024.\n                - post-challenge: This data, approved by the NTIA on 9/23/24, includes federal funding deduplication and the true-up to the Version 4 CostQuest Fabric with availability data dated 12/31/2023...Data updated 8/21/24 and approved by the NTIA on 9/23/24.\n            '}, 'WA': {'dev_note': '\n                to rerun load_postchal DAG and reload BSL+CAI after the state removes non-locations and enforceable commitments\n            ', 'post_fa': 4, 'post_ffa': 38, 'pre_fa': 2, 'pre_ffa': 15, 'sourced_at': '2025-04-07', 'stage': 'adjusting', 'update_note': '\n                Bear in mind we have already closed our first round of applications and, consistent with the rules of our IPVII, all project areas that received a preliminary award have been removed from the map.\n                BEAD data is as final as possible. Updates for enforceable commitments, as required by NTIA, may alter the eligibility of BSLs, as can BSL removal for non-BSLs as outlined in NTIA guidance.\n            ', 'version_note': '\n                The post challenge fabric version and availability data used for our approved challenge process results is fabric v4 12/2023 Broadband Data Collection (BDC) published on September 9, 2024.\n            '}, 'WI': {'dev_note': '\n                List of all BSLs: https://apps.psc.wi.gov/pages/viewdoc.htm?docid=498680. We havent released separate lists that reflect ineligible locations, but this link includes all BSLs in Wisconsin for fabric version 3 (6/30/2023) so you can just compare the full list to the pre-post lists to create your own ineligible list.\n            ', 'post_fa': 4, 'post_ffa': 34, 'pre_fa': 3, 'pre_ffa': 26, 'sourced_at': '2025-04-10', 'stage': 'final', 'update_note': '\n            ', 'version_note': '\n                Pre-challenge Fabric v3 (6/30/2023). Availability date: March 26, 2024\n                Post-Challenge Fabric v4 (12/31/2023). Availability date: July 9, 2024            \n            '}}

BeadFpUtil

This class contains BEAD Final Proposal file path configurations for each state.

class util.bead_fp.BeadFpUtil

file_path = {'AR': [{'cai': 'fp_cai.csv', 'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_no_bead_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/final_proposal_20250930/', 'subgrantees': 'fp_subgrantees.csv', 'version_date': '20250930'}, {'cai': 'curing_fp_cai.csv', 'deployment_projects': 'curing_fp_deployment_projects.csv', 'locations': 'curing_fp_locations.csv', 'no_bead_locations': 'curing_fp_no_bead_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Arkansas/unzipped/final_proposal/20251112/', 'subgrantees': 'curing_fp_subgrantees.csv', 'version_date': '20251112'}], 'IN': [{'no_bead_locations': 'fp_no_bead_locations.xlsx', 's3_key': 's3://463002504035-shoreline/raw/uploads/Indiana/unzipped/final_proposal/20250923/', 'version_date': '20250923'}], 'LA': [{'cai': 'fp_cai_08_13_2025..csv', 'deployment_projects': 'fp_deployment_projects_08_13_2025..csv', 'locations': 'fp_locations_08_13_2025..csv', 'no_bead_locations': 'fp_no_bead_locations__08_13_2025..csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Louisiana/final_proposal_final/datasme_1113/', 'subgrantees': 'fp_subgrantees_08_13_2025.csv', 'version_date': '20250813'}, {'cai': 'cai/3_fp_cai_UPDATED_2024.12.30.csv', 'deployment_projects': 'deployment_projects/1_fp_deployment_projects_UPDATED_2024.12.30.csv', 'locations': 'locations/2_fp_locations_UPDATED_2024.12.30.csv', 'no_bead_locations': 'no_bead_locations/4_fp_no_BEAD_locations_2024.12.12.csv', 'non_deployment_projects': 'non_deployment_projects/1_fp_non_deployment_projects_2024.12.21.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Louisiana/final_proposal_final/', 'subgrantees': 'subgrantees/1_fp_subgrantees_2024.12.12.csv', 'version_date': '20241230'}, {'cai': 'fp_cais.csv', 'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_bead_no_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Louisiana/final_proposal_final_20250825/', 'subgrantees': 'fp_subgrantees.csv', 'version_date': '20250825'}, {'cai': 'fp_cais_11_18_2025.csv', 'deployment_projects': 'fp_deployment_projects_11_18_2025.csv', 'locations': 'fp_locations_11_18_2025.csv', 'no_bead_locations': 'fp_no_bead_11_18_2025.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Louisiana/unzipped/bead_final_proposal/20251118/', 'subgrantees': 'fp_subgrantees_11_18_2025.csv', 'version_date': '20251118'}], 'MN': [{'deployment_projects': 'deployment-projects_tcm1045-703453 09152025.csv', 'locations': 'final-proposal-locations_tcm1045-703455 09152025.csv', 'no_bead_locations': 'final-proposal-no-bead-locations_tcm1045-703457 09152025.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Minnesota/unzipped/final_proposal_curing/', 'subgrantees': 'final-proposal-subgrantees_tcm1045-703454 09152025.csv', 'version_date': '20250915'}, {'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_no_bead_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Minnesota/unzipped/final_proposal_20251016/', 'subgrantees': 'fp_subgrantees.csv', 'version_date': '20251016'}], 'MS': [{'cai': "Copy of Use of Funds to Serve CAI's.csv", 'deployment_projects': 'Copy of Last-Mile Deployment Projects.csv', 'locations': 'Copy of BEAD-funded Locations.csv', 'no_bead_locations': 'Copy of No BEAD Locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/final_proposal/', 'subgrantees': 'Copy of Last-Mile Deployment Subgrantees.csv', 'version_date': '20250905'}, {'cai': "Copy of Use of Funds to Serve CAI's.csv", 'deployment_projects': 'Copy of Last-Mile Deployment Projects.csv', 'locations': 'Copy of BEAD-funded Locations.csv', 'no_bead_locations': 'fp_no_bead_locations_09_18.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/final_proposal/', 'subgrantees': 'Copy of Last-Mile Deployment Subgrantees.csv', 'version_date': '20250918'}, {'cai': "Copy of Use of Funds to Serve CAI's.csv", 'deployment_projects': 'Copy of Last-Mile Deployment Projects.csv', 'locations': 'Copy of BEAD-funded Locations.csv', 'no_bead_locations': 'r_msup_fp_no_bead_locations_09_22.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/final_proposal/', 'subgrantees': 'Copy of Last-Mile Deployment Subgrantees.csv', 'version_date': '20250922'}, {'cai': "Use of Funds to Serve CAI's - 9.23.csv", 'deployment_projects': 'Last-Mile Deployment Projects - 10.21.csv', 'locations': 'BEAD-funded Locations - Updated -10-28-2025(in).csv', 'no_bead_locations': 'fp_no_bead_locations_10.14.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/final_proposal/20251030/', 'version_date': '20251030'}, {'cai': 'fp_cais.csv', 'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_no_bead_locations_10.14.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Mississippi/unzipped/final_proposal/20251114/', 'subgrantees': 'Copy of Last-Mile Deployment Subgrantees.csv', 'version_date': '20251114'}], 'OK': [{'cai': 'fp_cai(9-12-25).csv', 'deployment_projects': 'fp_deployment_projects(9-12-25).csv', 'locations': 'fp_locations(9-12-25).csv', 'no_bead_locations': 'fp_no_bead_locations(9-4-25).csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Oklahoma/final_proposal/20250912/', 'subgrantees': 'fp_subgrantees(9-12-25).csv', 'version_date': '20250912'}, {'cai': 'fp_cai.csv', 'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_no_bead_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Oklahoma/final_proposal/20250916/', 'subgrantees': 'fp_subgrantees.csv', 'version_date': '20250916'}, {'cai': 'fp_cai.csv', 'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_no_bead_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Oklahoma/final_proposal/20250924/', 'subgrantees': 'fp_subgrantees.csv', 'version_date': '20250924'}, {'cai': 'fp_cai.csv', 'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_no_bead_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Oklahoma/final_proposal/20251113/', 'subgrantees': 'fp_subgrantees.csv', 'version_date': '20251113'}], 'TX': [{'cai': 'FP_CAI.csv', 'deployment_projects': 'FP_DEPLOYMENT_PROJECTS.csv', 'locations': 'FP_LOCATIONS.csv', 'no_bead_locations': 'FP_NO_BEAD_LOCATIONS.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Texas/unzipped/final_proposal/20251118/', 'subgrantees': 'FP_SUBGRANTEES.csv', 'version_date': '20251118'}], 'VA': [{'cai': 'fp_cai.csv', 'deployment_projects': 'fp_deployment_projects.csv', 'locations': 'fp_locations.csv', 'no_bead_locations': 'fp_no_bead_locations.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Virginia/unzipped/final_proposal_09112025/', 'subgrantees': 'fp_subgrantees.csv', 'version_date': '20250911'}, {'cai': 'fp_cai_10_27_2025.csv', 'deployment_projects': 'fp_deployment_projects_10_27_2025.csv', 'locations': 'fp_locations_10_27_2025.csv', 'no_bead_locations': 'fp_no_bead_locations_10_27_2025.csv', 's3_key': 's3://463002504035-shoreline/raw/uploads/Virginia/unzipped/final_proposal_20251027/', 'subgrantees': 'fp_subgrantees_10_27_2025.csv', 'version_date': '20251027'}]}

static get_fp_file_path(state)

Return type:

list[dict]

static get_fp_round(state)

Return type:

int

static get_fp_states()

Return type:

list[str]

CPFUtil

This class contains CPF (Capital Projects Fund) pre/post-challenge configurations.

class util.cpf.CPFUtil

Util class for CPF pre/post-challenge configurations.

static cpf_profile(state_abbr, check_pre=True)

table_pre_challenge: the table name before the challenge table_trueup: the table name after the true-up. Before the challenge results applied. This table is only for data processing. Not for actual frontend functionality org_id: the organization id for the state ffa_version: the post challenge version of the fabric fixed availability data fabric_version: the post challenge version of the fabric data true_up: whether the state has true-up or not dsl_mod: whether the state has DSL modification or not fw_mod: whether the state has Cellular fixed wireless modification or not tenant: the name of the tenant in the grapi database

Parameters:

• state_abbr (str) – the state abbreviation in upper case

• check_pre (str, optional) – whether to check the pre-challenge tables. Default is True.

Return type:

dict

Returns:

the post challenge configuration for the state

NOTE: this function does call the AWS Athena and AWS Glue APIs to check the existence of the tables. Do not use outside of a DAG unless check_pre is set to False.

static get_cpf_postchallenge_states()

Get the list of states that have post-challenge configurations recorded. This does not check the pre-challenge tables, so you would need to call bead_profile to check individual states.

Return type:

list

state_profiles = {'MS': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 31, 'ffa_version_pre_challenge': 31, 'fw_mod': True, 'org_id': 0, 'table_pre_challenge': 'c_cpf_merge_tile_ms', 'table_trueup': 'c_cpf_merge_tile_ms', 'tenant': 'mississippi', 'true_up': False}, 'OK': {'dsl_mod': True, 'fabric_version': 4, 'fabric_version_pre_challenge': 4, 'ffa_version': 33, 'ffa_version_pre_challenge': 33, 'fw_mod': False, 'notes': 'multiple programs at the same time', 'org_id': 0, 'table_pre_challenge': 'c_cpf_merge_tile_ok', 'table_trueup': 'c_cpf_merge_tile_ok', 'tenant': 'oklahoma', 'true_up': True}}

TileUtil

This class contains tiling and TileserverGL utility functions for interacting with tippecanoe and tile servers.

class util.tile.TileUtil

Tiling and tileserver utility functions to interact with tippecanoe, and TileserverGL servers: https://johto.ready.net/ https://hoenn.ready.net/ https://kanto.ready.net/

static get_all_tiles_on_server(server_alias)

Get all the tiles on the server

Parameters:

server_alias (Literal['johto', 'hoenn', 'kanto']) – the alias to check

Return type:

list[str]

Returns:

the list of tiles

static get_tileset_info(env, title_filter=None, use_regex=False)

Get all the tileset information from the given environment’s tile server

Parameters:

• env (Literal['prod', 'staging', 'dev', 'johto', 'hoenn', 'kanto']) – Environment name (“prod”, “staging”, “dev”) or server alias (“kanto”, “hoenn”, “johto”)

• title_filter (Optional[str]) – The filter to apply to the tileset titles.

• use_regex (bool) – Whether to use regex for the title filter.

Return type:

Dict[str, Tuple[str, dict]]

Returns:

A dictionary of tileset information.

static is_mbtile_valid(path)

Check if the mbtile is valid

Parameters:

path (str) – the path to the mbtile

Returns:

True if the mbtile is valid, False otherwise

Return type:

bool

static tileserver_is_healthy(server_alias)

Check if a tileset exists in the server

Parameters:

• tile_id (str) – The id of the tileset

• server_alias (str) – The alias of the server

Returns:

True if the tileset exists, False otherwise

Return type:

bool

static tileset_exist_on_server(tile_id, server_alias)

Check if a tileset exists in the server

Parameters:

• tile_id (str) – The id of the tileset

• server_alias (str) – The alias of the server

Returns:

True if the tileset exists, False otherwise

Return type:

bool

Outbound Summary Workbook Builder

Builds state-specific cumulative outbound-payment workbooks from canonical templates.

exception util.financial_integration.outbound_summary.UnsupportedOutboundSummaryStateError

Raised when a state does not have an outbound summary builder.

util.financial_integration.outbound_summary.build_outbound_summary(state, rows, template_path, output_path)

Build a state-specific cumulative outbound-payment workbook.

Configs

The following are configuration objects that are used throughout the project. They are organized by the type of configuration they provide.

FipsCodes

This class contains FIPS codes for each state and territory, as well as various frequently used combinations.

class config.fips.FipsCodes

https://www.census.gov/library/reference/code-lists/ansi.html

ALL = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'FM': '64', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MH': '68', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'PW': '70', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UM': '74', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56', 'XY': '99', 'ZZ': '98'}

ALL_OTHER = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'FM': '64', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MH': '68', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'PW': '70', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UM': '74', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56', 'XY': '99', 'ZX': '00', 'ZZ': '98'}

DEFAULT = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56', 'XY': '99', 'ZX': '00', 'ZZ': '98'}

DEMO = {'XY': '99', 'ZZ': '98'}

OTHER = {'ZX': '00'}

STATES = {'AK': '02', 'AL': '01', 'AR': '05', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'GA': '13', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MI': '26', 'MN': '27', 'MO': '29', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UT': '49', 'VA': '51', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56'}

STATES_T1 = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56'}

TERRITORIES = {'AS': '60', 'FM': '64', 'GU': '66', 'MH': '68', 'MP': '69', 'PR': '72', 'PW': '70', 'UM': '74', 'VI': '78'}

TERRITORIES_T1 = {'AS': '60', 'GU': '66', 'MP': '69', 'PR': '72', 'VI': '78'}

TERRITORIES_T2 = {'UM': '74'}

TERRITORIES_T3 = {'FM': '64', 'MH': '68', 'PW': '70'}

ProductionConfig

This class contains configuration for production. This should not be overridden on production, only on development.

class config.production.ProductionConfig

Proposed naming convention: <service>_<type>(_<env>) Assume “default” connection for airflow env Thus, no env for prod

AIRFLOW_API = 'airflow_rest_api_dev'

AIRFLOW_URL = 'https://pidgeot.ready-internal.net'

ALDP_API_KEY = '5qJYV7XQAB0eeQUJ4t7o6u3ppvHzCLfBsPv7sZAZ'

ALL_FIPS_STATE = {'00': 'ZX', '01': 'AL', '02': 'AK', '04': 'AZ', '05': 'AR', '06': 'CA', '08': 'CO', '09': 'CT', '10': 'DE', '11': 'DC', '12': 'FL', '13': 'GA', '15': 'HI', '16': 'ID', '17': 'IL', '18': 'IN', '19': 'IA', '20': 'KS', '21': 'KY', '22': 'LA', '23': 'ME', '24': 'MD', '25': 'MA', '26': 'MI', '27': 'MN', '28': 'MS', '29': 'MO', '30': 'MT', '31': 'NE', '32': 'NV', '33': 'NH', '34': 'NJ', '35': 'NM', '36': 'NY', '37': 'NC', '38': 'ND', '39': 'OH', '40': 'OK', '41': 'OR', '42': 'PA', '44': 'RI', '45': 'SC', '46': 'SD', '47': 'TN', '48': 'TX', '49': 'UT', '50': 'VT', '51': 'VA', '53': 'WA', '54': 'WV', '55': 'WI', '56': 'WY', '60': 'AS', '64': 'FM', '66': 'GU', '68': 'MH', '69': 'MP', '70': 'PW', '72': 'PR', '74': 'UM', '78': 'VI', '98': 'ZZ', '99': 'XY'}

ALL_STATE_FIPS = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'FM': '64', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MH': '68', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'PW': '70', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UM': '74', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56', 'XY': '99', 'ZX': '00', 'ZZ': '98'}

ARCHIE_DEV_BUCKET = '463002504035-archie-dev'

ARCHIVE_TILES_S3_BUCKET = '463002504035-archive/tiles'

ATHENA_BOSS_DB_ALIAS = 'postgres_boss_prod'

ATHENA_BOSS_DB_ALIAS_STAGE = 'postgres_boss_stage'

ATHENA_GAME_AGENT_DB_ALIAS = 'postgres_rad_prod'

ATHENA_GAME_AGENT_DB_ALIAS_STAGE = 'postgres_rad_prod'

ATHENA_GRAPI_DB_ALIAS = 'postgres_grapi_prod'

ATHENA_GRAPI_DB_ALIAS_STAGE = 'postgres_grapi_stage'

ATHENA_RAD_DISCOVERY_DB_ALIAS = 'rad_discovery_dev.rad_discovery_dev_db'

Athena alias for RAD discovery database for the cross db connections. since it is under dev we use the dev connection string

ATHENA_READY_DATA_DB_ALIAS = 'postgres_readydata_prod'

ATHENA_READY_DATA_DB_ALIAS_DEV = 'postgres_readydata_dev'

AWS_AIRFLOW_EC2_FEDERATED_USER = 'IAMR:airflow_prod_ec2_role'

AWS_DATA_ADMIN_FEDERATED_USER = 'IAMR:AWSReservedSSO_AdministratorAccess_bd933a6f96357c52'

AWS_DATA_DEVELOPER_FEDERATED_USER = 'IAMR:AWSReservedSSO_Ready-Data-Developer_cf669028fab52727'

AWS_GITHUB_APP_TOKEN = 'ready_github_app_repo_scanner'

AWS_GOOGLE_PLACES_API_KEY = 'google-places-api-key'

Google Places API that returns formatted location data and imagery about establishments, geographic locations, or prominent points of interest.

BACKUP_BUCKET = '463002504035-archive'

BACKUP_EMAIL = ['yuan@ready.net']

BACKUP_PREFIX = 'backup'

BATCH_SIZE = 1000000

BATCH_SIZE_ECS = 3000000

BATCH_SIZE_LOCAL = 1000000

BEDROCK_BUCKET = '463002504035-compliancebot'

BOSS_API = 'boss_api'

BOSS_DATA_DB = 'postgres_bossdata_prod'

BOSS_DATA_DEV_DB = 'postgres_bossdata_dev'

BOSS_DATA_STAGE_DB = 'postgres_bossdata_stage'

BOSS_PLATFORM_DB = 'postgres_bossplatform_grapi_prod'

BOSS_PLATFORM_GRAPI_DB = 'postgres_bossplatform_grapi_prod'

BOSS_PLATFORM_GRAPI_DEV_DB = 'postgres_bossplatform_grapi_dev'

BOSS_PLATFORM_GRAPI_STAGE_DB = 'postgres_bossplatform_grapi_stage'

BOSS_PLATFORM_STAGE_DB = 'postgres_bossplatform_grapi_stage'

BOSS_PRIVATE_BUCKET = '463002504035-boss-private-prod'

BOSS_PRIVATE_BUCKET_STAGE = '463002504035-boss-private-stage'

BOSS_PUBLIC_BUCKET = '463002504035-boss-public'

BOSS_PUBLIC_BUCKET_STAGE = '463002504035-boss-public-stage'

BOSS_STAGE_DB = 'postgres_bossdata_stage'

BRANCH_TRIGGER = 'none_failed_min_one_success'

CALIX_CLOUD_API_CONN = 'calix_cloud_api_readynet'

CATCHUP = False

CENSUS_API_KEY = '3cbe5ce5311ca6cdddc6e7461bf7bf19857fea54'

CIRCUMFERENCE_EARTH = 40075017

CLAUDE_BEDROCK_SONNET = 'us.anthropic.claude-sonnet-4-5-20250929-v1:0'

CQA_API_KEY = 'cqa_token_prod'

CQA_BASE_URI = 'https://costquest.azure-api.net/'

CQA API base URI.

CRAWLER_ROLE = 'airflow_prod_ec2_role'

CRAWLER_SLEEP_TIME = 5

CRS_GPS = 'EPSG:4326'

CSV_SERDE = 'org.apache.hadoop.hive.serde2.OpenCSVSerde'

CTC_EGNYTE_SERVER = 'ctc_egnyte_server'

CURATED_PREFIX = 'curated'

DAG_ROOT = '/home/ubuntu/airflow/dags'

DAP_HOSTS = {'ar': 'https://dap.grantor.broadband.arkansas.gov/', 'az': 'https://dap.bead.azcommerce.com/', 'cpuc': 'https://dap.bead.grant.cpuc.ca.gov/', 'demo': 'https://dap.bossdemo.portal.ready.net/', 'in': 'https://dap.portal.connectingindiana.com/', 'mn': 'https://dap.bead.mn.gov/', 'ms': 'https://dap.apply.broadbandms.com/', 'tx': 'https://dap.applybead.broadband.texas.gov/'}

DATAHUB_BASE_URL = 'https://uxie.ready-internal.net'

DATAHUB_CONNECTION_ID = 'datahub_rest_default'

DATAHUB_GMS_ENDPOINT = 'https://uxie.ready-internal.net/api/gms'

DATAHUB_GRAPHQL_ENDPOINT = 'https://uxie.ready-internal.net/api/graphql'

DATAHUB_S3_BUCKET = '463002504035-datahub'

DATA_AGENT_BUCKET = '463002504035-data-agent-kb-dev'

DATA_LAKE_BUCKET = '463002504035-readydatalake'

DATA_LAKE_BUCKET_DEV = '463002504035-shoreline'

DATA_LAKE_DB = 'readydatalake'

DATA_LAKE_DB_DEV = 'shoreline'

DATA_PUBLIC_BUCKET = '463002504035-data-public'

DBT_ALIAS = 'grovyle'

The alias for the dbt server. This is used to connect to the dbt server. It will be used in the ssh command to connect to the server. Will be different for prod and dev

DBT_SERVER_FOLDER = '/home/ubuntu/airflow/'

The folder for the dbt server. This is used to connect to the dbt server.

DBT_SSH_KEY = '~/.ssh/id_rsa'

The ssh key for the dbt server. This is used to connect to the dbt server.

DBT_SSH_USER = 'ubuntu'

The ssh user for the dbt server. This is used to connect to the dbt server. It will be used in the ssh command to connect to the server. Will not be different for prod and dev

DBT_VENV_PATH = '/home/ubuntu/dbt-venv/bin/activate'

The venv path for the dbt server.

DEFAULT_ARGS = {'depends_on_past': False, 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': datetime.timedelta(seconds=60)}

DEFAULT_BUCKET = ''

DEFAULT_COMPRESSION = 'snappy'

DEFAULT_FIPS = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56', 'XY': '99', 'ZX': '00', 'ZZ': '98'}

DEFAULT_STORAGE = 'PARQUET'

DEFAULT_TS_FLAT_FOLDER = '~/tiles'

DEFAULT_WORKGROUP = 'batch-job'

DEMO_STATE_FIPS = {'XY': '99', 'ZZ': '98'}

DISCOVERY_POOL = 'rad_discovery_pool'

ECS_ATHENA_TO_GEOJSON_CONTAINER_NAME = 'ecs_athena_to_geojson_container'

ECS_ATHENA_TO_GEOJSON_TASK_DEFINITION = 'ecs_athena_to_geojson_prod'

ECS_CLUSTER = 'prod'

ECS_GEOPANDAS_CONTAINER_NAME = 'geopandas_container'

ECS_GEOPANDAS_TASK_DEFINITION = 'geopandas_prod'

ECS_GRAPHIFY_CONTAINER_NAME = 'ecs_graphify_container'

ECS_GRAPHIFY_TASK_DEFINITION = 'ecs_graphify_prod'

ECS_H3_CONTAINER_NAME = 'ecs_h3_container'

ECS_H3_TASK_DEFINITION = 'ecs_h3_prod'

ECS_HLD_CONTAINER_NAME = 'ecs_hld_process_container'

ECS_HLD_TASK_DEFINITION = 'ecs_hld_process_prod'

ECS_MLFLOW_CONTAINER_NAME = 'ecs_mlflow_model_container'

ECS_MLFLOW_TASK_DEFINITION = 'ecs_mlflow_model_prod'

ECS_MOUNT_PATH = '/mount'

ECS_PGDUMP_CONTAINER_NAME = 'ecs_transfer_with_pgdump_container'

ECS_PGDUMP_TASK_DEFINITION = 'ecs_transfer_with_pgdump_prod'

ECS_S3_PARQUET_PG_CONTAINER_NAME = 's3_parquet_to_pg_container'

ECS_S3_PARQUET_PG_TASK_DEFINITION = 's3_parquet_to_pg'

ECS_SECURITY_GROUPS = ['sg-0bcb7113fb9299e23']

ECS_SLEEP_TIME = 5

ECS_TIPPECANOE_CONTAINER_NAME = 'tippecanoe_container'

ECS_TIPPECANOE_TASK_DEFINITION = 'tippecanoe_prod'

EMAIL = 'yuan@ready.net'

EMAIL_CONTENT = '<h3>Data moved.</h3>'

EMAIL_SUBJECT = 'Airflow {env} DAG success: {dag_id}'

FABRIC_MECATOR_POOL = 'fabric_mercator'

FABRIC_SCRAPE_POOL = 'fabric_scrape'

FCC_DATE_FMT = '%Y-%m-%dT%H:%M:%S.%fZ'

FCC_DATE_FMT2 = '%Y-%m-%d'

FCC_FFA_API_KEY = 'fcc_ffa_api'

FCC_FUNDING_MAP_API_KEY = 'fcc_funding_map_api'

FCC_FUNDING_PUBLISHED_RUN_URL = 'https://fundingmap.fcc.gov/bfm/api/map/getPublishedRun'

FCC_HEADERS = {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9', 'accept-language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7', 'authority': 'broadbandmap.fcc.gov', 'cache-control': 'max-age=0', 'referer': 'https://broadbandmap.fcc.gov/location-summary/fixed?version=jun2023&zoom=16.70&vlon=-95.992263&vlat=36.156955&br=r&speed=25_3&tech=1_2_3_4_5_6_7_8', 'sec-ch-ua': '"Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"', 'sec-ch-ua-mobile': '?0', 'sec-ch-ua-platform': '"macOS"', 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'none', 'sec-fetch-user': '?1', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'}

FCC_PROCESS_UUID_URL = 'https://broadbandmap.fcc.gov/nbm/map/api/published/filing'

FCC_TECH_CODES = {0: 'Other', 10: 'Copper', 40: 'Cable', 50: 'Fiber to the Premises', 60: 'GSO Satellite', 61: 'NGSO Satellite', 70: 'Unlicensed Fixed Wireless', 71: 'Licensed Fixed Wireless', 72: 'Licensed-by-Rule Fixed Wireless'}

FCC_UPDATED_URL = 'https://broadbandmap.fcc.gov/api/reference/map_processing_updates/{process_uuid}'

FTP_MN_IF407_CONN = 'mn_ftp_if407_prod'

FTP_MN_IF411_CONN = 'mn_ftp_if411_prod'

GAME_AGENT_BUCKET = '463002504035-game-agent-prod'

GAME_AGENT_CONN = 'postgres_game_agnet_dev'

GAME_ATHENA = 'postgres_game_prod'

GAME_ATHENA_DEV = 'postgres_game_dev'

GAME_ATHENA_STAGE = 'postgres_game_stage'

GAME_PG = 'postgres_bossplatform_game_prod'

GAME_PG_DEV = 'postgres_bossplatform_game_dev'

GAME_PG_STAGE = 'postgres_bossplatform_game_staging'

GCP_BOSS_API_PROJECT = 'boss-api-289815'

GCP_BOSS_API_PROJECT_STAGE = 'boss-api-stage'

GCP_SERVICE_ACCOUNT = 'boss_api'

GDRIVE_FOLDER_BASE_URI = 'https://drive.google.com/drive/u/0/folders/'

GDRIVE_FOLDER_BBM = 'BBM_EXPORT'

GDRIVE_FOLDER_BBM_ID = '0AOvn0ywn4UcEUk9PVA'

GDRIVE_FOLDER_BBM_ID_PRIVATE = '1a5Q02qNeMVwPfY70J5DkaI8xV-1lPkXC'

GDRIVE_FOLDER_BEAD_TRANSFER = '1UgpVBKs8KQQ6-G77qPAwZYjqBar1L3Aa'

GDRIVE_FOLDER_CPF_TRANSFER = '1JiODwfPhpRTNSJRAf7nrA7ulkOWDwaI8'

GDRIVE_FOLDER_DATA_TEAM_ASSET = '1ScrRXXCInjH8Cs1nriqSWIah7LNWr-dp'

GDRIVE_FOLDER_FLAT_EXPORTER = '1rNzWD6BOyUECJHWQLckoL14tmfvOPPrA'

GDRIVE_FOLDER_GSHEET = '1kONOPVDuWMdDxR16ZME5xnyhOXUkMIeC'

GDRIVE_SERVICE_ACCOUNT = 'gdrive_api'

GENERAL_PREFIX = 'general_'

GLUE_PMTILES_JOB_NAME = 'PMTiles_Builder_Job'

GLUE_PMTILES_POLL_SECONDS = 30

GLUE_PMTILES_REGION = 'us-east-1'

GLUE_PMTILES_SCRIPT_S3 = 's3://aws-glue-assets-463002504035-us-east-1/scripts/pmtiles_builder.py'

GRAPHQL_API_URL = 'https://grapi.ready.net/graphql'

GRAPHQL_API_URL_DEV = 'https://dev.grapi.ready.net/graphql'

GRAPHQL_API_URL_STAGE = 'https://staging.grapi.ready.net/graphql'

GRAPI_API_SECRET_NAME = 'jenkins/boss-platform-auth-security-key-prod'

GRAPI_API_SECRET_NAME_DEV = 'jenkins/boss-platform-auth-security-key-dev'

GRAPI_API_SECRET_NAME_STAGE = 'jenkins/boss-platform-auth-security-key-stage'

GRAPI_ATHENA = 'postgres_grapi_prod'

GRAPI_ATHENA_DEV = 'postgres_grapi_dev'

GRAPI_ATHENA_STAGE = 'postgres_grapi_stage'

GRAPI_PG = 'postgres_bossplatform_grapi_prod'

GRAPI_PG_DEV = 'postgres_bossplatform_grapi_dev'

GRAPI_PG_STAGE = 'postgres_bossplatform_grapi_stage'

HLD_BUCKET = '463002504035-hld-prod'

IF_EXIST = 'replace'

INITIAL_GDRIVE_PAGE_TOKEN = 'ready-initial-dummy-token!'

INTERVAL = None

IN_SFTP_SERVER = 'in_ftp'

KMZ_KML_DEFAULT = 'doc.kml'

LINEAR_API_KEY = 'linear_api_key'

LINEAR_API_KEY_SUSHEEL = 'susheel_linear_api_'

MAX_ACTIVE_RUNS = 1

MAX_ACTIVE_TASKS = 16

MAX_ITERATIONS = 50000

MLAB_SECRET = 'mlab_bigquery_json'

MOUNT_PATH = '/home/ubuntu/efs-mount'

NUM_ECS_TILING_MACHINES = 20

Prefered number of maximum ECS machines to launch for tiling for each DAG (not a hard limit)

OOKLA_BROADBAND_PR = 'ookla_broadband_pr'

OOKLA_BROADBAND_US = 'ookla_broadband_us'

OOKLA_MOBILE_US = 'ookla_mobile_us'

OVERTURE_BUCKET = '463002504035-overture'

OVERTURE_TRANSFORM_POOL = 'overture_transform_pool'

PANIC_PREFIXES = ['backup', 'curated', 'postgres', 'raw', 'temp', 'transformed', 'summaryraw']

PEERINGDB_API_KEY = 'peeringdb_api'

PG_NUM_CONN_LARGE = 'pg_num_conn_large'

One of Airflow pools that are defined e.g. dev env at: https://pidgey.ready-internal.net/pool/list/

PG_NUM_CONN_NORMAL = 'pg_num_conn_normal'

PG_SCHEMA = 'data'

The SQL schema to use for the postgres database in PG GAME PROD STAGE and DEV

PMTILES_CDN_BASE_URL = ''

PMTILES_S3_PREFIX = 'tiles/pmtiles_public'

PMTILES_STAGING_PREFIX = 'tiles/pmtiles_staging'

PMTILES_TIPPECANOE_MLT_S3 = 's3://aws-glue-assets-463002504035-us-east-1/bin/tippecanoe-mlt/9aef0512'

POSTGRES_PREFIX = 'postgres'

PRODUCTION = True

RAD_ADDRESS_PRIMARY = 'address_primary'

RAD_DB_CONN = 'postgres_rad_prod'

RAD_DISCOVERY_SCHEMA = 'discovery'

RAD_DOCDB = 'documentdb_rad'

The connection string for the RAD database. It is a document db connection string

RAD_DOC_DB_CONN = 'documentdb_rad'

RAD_PG = 'postgres_rad_prod'

RAD_PREFIXES

List of the prefixes that are used in RAD

alias of RADPrefixes

RAD_RMQ = 'rabbitmq_dev'

The connection string for the RAD rabbitmq. since it is under dev we use the dev connection string

RAD_RMQ_CONN = 'rabbitmq_dev'

RAW_PREFIX = 'raw'

READY_DATA_DB = 'postgres_readydata_prod'

READY_DATA_DB_DEV = 'postgres_readydata_dev'

REDSHIFT = 'redshift_dw_prod'

REDSHIFT_DATA_TEAM_GROUP = 'readydata'

REDSHIFT_DEFAULT_PRIVILEGE_OWNERS = ['IAMR:AWSReservedSSO_AdministratorAccess_bd933a6f96357c52', 'IAMR:AWSReservedSSO_Ready-Data-Developer_cf669028fab52727', 'IAMR:airflow_prod_ec2_role', 'rdsadmin']

REDSHIFT_DEV = 'redshift_dw_dev'

REDSHIFT_DEV_PG = 'redshift_dw_dev_pghook'

REDSHIFT_DEV_WORKGROUP = 'redshift-workgroup-dev'

REDSHIFT_PG = 'redshift_dw_prod_pghook'

REDSHIFT_PROD_PG = 'redshift_dw_prod_pghook'

REDSHIFT_PROD_WORKGROUP = 'redshift-workgroup-prod'

REDSHIFT_S3_ROLE_ARN = 'arn:aws:iam::463002504035:role/service-role/AmazonRedshift-CommandsAccessRole-20250731T224539'

REDSHIFT_WORKGROUP = 'redshift-workgroup-prod'

REFLECT_RUN_API_KEY = 'reflect_run_api'

REFLECT_RUN_BASE_URL = 'https://api.reflect.run/v1/'

REGRID_INGEST_BUCKET = '463002504035-regrid'

RENDER_TEMPLATE_AS_NATIVE_OBJ = True

REQUEST_TIMEOUT_HOURS = 1

RESEARCH_NODE_FUNDING_POOL = 'research_node_funding'

ROB_API_LAMBDA_FUNCTION_URL = 'https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/lambda-rob-api-RobAPIFunction-PQw1kczf7WFM/invocations'

This is the rob-api lambda function url. it does not have 30 second timeout

ROB_AUTH_KEY = 'rob-auth-key'

A simple api security token for the ROB api

ROB_URL = 'https://meowth.ready-data.net'

ROB api call with a 30 second timeout

SCRAPERAPI_CONN = 'scraperapi_proxy_key_colello'

SCRAPERBEE_CONN = 'scraperbee_proxy_key_colello'

SCRAPER_TIMEOUT = 30

SCRAPER_URL = 'http://api.scraperapi.com/'

SENDGRID_CONN = 'sendgrid_default'

SFTP_TX_USAS_CONN = 'tx_cpa_prod'

SFTP_TX_USAS_PGP_KEY = 'tx_cpa_pgp'

SLACK_ALERTS_CHANNEL = '#airflow'

SLACK_DEFAULT_API = 'slack_default_api'

SLACK_DEFAULT_HOOK = 'slack_default_hook'

SLEEP_TIME = 5

SPLIT_PATH = '/dags/'

START_DATE = datetime.datetime(2023, 1, 1, 0, 0)

STATES_T1 = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56'}

STATE_FIPS = {'AK': '02', 'AL': '01', 'AR': '05', 'AS': '60', 'AZ': '04', 'CA': '06', 'CO': '08', 'CT': '09', 'DC': '11', 'DE': '10', 'FL': '12', 'GA': '13', 'GU': '66', 'HI': '15', 'IA': '19', 'ID': '16', 'IL': '17', 'IN': '18', 'KS': '20', 'KY': '21', 'LA': '22', 'MA': '25', 'MD': '24', 'ME': '23', 'MI': '26', 'MN': '27', 'MO': '29', 'MP': '69', 'MS': '28', 'MT': '30', 'NC': '37', 'ND': '38', 'NE': '31', 'NH': '33', 'NJ': '34', 'NM': '35', 'NV': '32', 'NY': '36', 'OH': '39', 'OK': '40', 'OR': '41', 'PA': '42', 'PR': '72', 'RI': '44', 'SC': '45', 'SD': '46', 'TN': '47', 'TX': '48', 'UT': '49', 'VA': '51', 'VI': '78', 'VT': '50', 'WA': '53', 'WI': '55', 'WV': '54', 'WY': '56', 'XY': '99', 'ZX': '00', 'ZZ': '98'}

STATE_NAMES = {'AK': 'Alaska', 'AL': 'Alabama', 'AR': 'Arkansas', 'AS': 'American Samoa', 'AZ': 'Arizona', 'CA': 'California', 'CO': 'Colorado', 'CT': 'Connecticut', 'DC': 'District of Columbia', 'DE': 'Delaware', 'FL': 'Florida', 'GA': 'Georgia', 'GU': 'Guam', 'HI': 'Hawaii', 'IA': 'Iowa', 'ID': 'Idaho', 'IL': 'Illinois', 'IN': 'Indiana', 'KS': 'Kansas', 'KY': 'Kentucky', 'LA': 'Louisiana', 'MA': 'Massachusetts', 'MD': 'Maryland', 'ME': 'Maine', 'MI': 'Michigan', 'MN': 'Minnesota', 'MO': 'Missouri', 'MP': 'Northern Mariana Islands', 'MS': 'Mississippi', 'MT': 'Montana', 'NC': 'North Carolina', 'ND': 'North Dakota', 'NE': 'Nebraska', 'NH': 'New Hampshire', 'NJ': 'New Jersey', 'NM': 'New Mexico', 'NV': 'Nevada', 'NY': 'New York', 'OH': 'Ohio', 'OK': 'Oklahoma', 'OR': 'Oregon', 'PA': 'Pennsylvania', 'PR': 'Puerto Rico', 'RI': 'Rhode Island', 'SC': 'South Carolina', 'SD': 'South Dakota', 'TN': 'Tennessee', 'TX': 'Texas', 'UM': 'U.S. Minor Outlying Islands', 'UT': 'Utah', 'VA': 'Virginia', 'VI': 'U.S. Virgin Islands', 'VT': 'Vermont', 'WA': 'Washington', 'WI': 'Wisconsin', 'WV': 'West Virginia', 'WY': 'Wyoming', 'XY': 'Demo Base', 'ZZ': 'Test Base'}

STATE_ONLY_NAMES = {'AK': 'Alaska', 'AL': 'Alabama', 'AR': 'Arkansas', 'AZ': 'Arizona', 'CA': 'California', 'CO': 'Colorado', 'CT': 'Connecticut', 'DE': 'Delaware', 'FL': 'Florida', 'GA': 'Georgia', 'HI': 'Hawaii', 'IA': 'Iowa', 'ID': 'Idaho', 'IL': 'Illinois', 'IN': 'Indiana', 'KS': 'Kansas', 'KY': 'Kentucky', 'LA': 'Louisiana', 'MA': 'Massachusetts', 'MD': 'Maryland', 'ME': 'Maine', 'MI': 'Michigan', 'MN': 'Minnesota', 'MO': 'Missouri', 'MS': 'Mississippi', 'MT': 'Montana', 'NC': 'North Carolina', 'ND': 'North Dakota', 'NE': 'Nebraska', 'NH': 'New Hampshire', 'NJ': 'New Jersey', 'NM': 'New Mexico', 'NV': 'Nevada', 'NY': 'New York', 'OH': 'Ohio', 'OK': 'Oklahoma', 'OR': 'Oregon', 'PA': 'Pennsylvania', 'RI': 'Rhode Island', 'SC': 'South Carolina', 'SD': 'South Dakota', 'TN': 'Tennessee', 'TX': 'Texas', 'UT': 'Utah', 'VA': 'Virginia', 'VT': 'Vermont', 'WA': 'Washington', 'WI': 'Wisconsin', 'WV': 'West Virginia', 'WY': 'Wyoming'}

SUBNETS = ['subnet-05d37f00bd1245b61']

SUMMARY_RAW_PREFIX = 'summaryraw'

SUPERSET_DAP_CONN_ID = 'dap_superset_user'

The connection id for the Superset instance. It will be used in the Superset operators to connect to the Superset instance. It is not different for prod and dev because we will disable user after each dag run. If you get login issues you need to check the Superset instance and enable the user for the next run. For the first run also need to create the user in the Superset instance and give it the appropriate permissions.

SWAMP_BUCKET = '463002504035-swamp'

TEMPLATE_SEARCHPATH = ['/home/ubuntu/airflow/templates/sql']

TEMP_PREFIX = 'temp'

TILESERVER_ALIAS = 'kanto'

TILESERVER_ALIAS_DEV = 'johto'

TILESERVER_ALIAS_STAGE = 'hoenn'

TILE_ENVS_TS_AND_S3 = {'hoenn': {'s3': 'staging', 'ts': 'staging'}, 'kanto': {'s3': 'prod', 'ts': 'main'}}

TILE_OPERATOR_MAP = {'archive_to_s3': 'operators.tile.archive_to_s3.ArchiveToS3Operator', 'recover_from_s3': 'operators.tile.recover_from_s3.RecoverFromS3Operator', 'remove_in_s3': 'operators.tile.remove_in_s3.RemoveInS3Operator', 'remove_tile': 'operators.tile.remove_tile.RemoveTileOperator'}

TRANSFER_POOL = 'transfer_small'

TRANSFORMED_PREFIX = 'transformed'

class datetime(year, month, day[, hour[, minute[, second[, microsecond[, tzinfo]]]]])

The year, month and day arguments are required. tzinfo may be None, or an instance of a tzinfo subclass. The remaining arguments may be ints.

astimezone()

tz -> convert to local time in new timezone tz

combine()

date, time -> datetime with same date and time fields

ctime()

Return ctime() style string.

date()

Return date object with same year, month and day.

dst()

Return self.tzinfo.dst(self).

fold

fromisoformat()

string -> datetime from datetime.isoformat() output

fromtimestamp()

timestamp[, tz] -> tz’s local time from POSIX timestamp.

hour

isoformat()

[sep] -> string in ISO 8601 format, YYYY-MM-DDT[HH[:MM[:SS[.mmm[uuu]]]]][+HH:MM]. sep is used to separate the year from the time, and defaults to ‘T’. The optional argument timespec specifies the number of additional terms of the time to include. Valid options are ‘auto’, ‘hours’, ‘minutes’, ‘seconds’, ‘milliseconds’ and ‘microseconds’.

max = datetime.datetime(9999, 12, 31, 23, 59, 59, 999999)

microsecond

min = datetime.datetime(1, 1, 1, 0, 0)

minute

now()

Returns new datetime object representing current time local to tz.

tz

Timezone object.

If no tz is specified, uses local timezone.

replace()

Return datetime with new specified fields.

resolution = datetime.timedelta(microseconds=1)

second

strptime()

string, format -> new datetime parsed from a string (like time.strptime()).

time()

Return time object with same time but with tzinfo=None.

timestamp()

Return POSIX timestamp as float.

timetuple()

Return time tuple, compatible with time.localtime().

timetz()

Return time object with same time and tzinfo.

tzinfo

tzname()

Return self.tzinfo.tzname(self).

utcfromtimestamp()

Construct a naive UTC datetime from a POSIX timestamp.

utcnow()

Return a new datetime representing UTC day and time.

utcoffset()

Return self.tzinfo.utcoffset(self).

utctimetuple()

Return UTC time tuple, compatible with time.localtime().

pendulum = <module 'pendulum' from '/home/ubuntu/airflow-venv/lib/python3.10/site-packages/pendulum/__init__.py'>

tags

alias of Tags

class timedelta

Difference between two datetime values.

timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0)

All arguments are optional and default to 0. Arguments may be integers or floats, and may be positive or negative.

days

Number of days.

max = datetime.timedelta(days=999999999, seconds=86399, microseconds=999999)

microseconds

Number of microseconds (>= 0 and less than 1 second).

min = datetime.timedelta(days=-999999999)

resolution = datetime.timedelta(microseconds=1)

seconds

Number of seconds (>= 0 and less than 1 day).

total_seconds()

Total seconds in the duration.

LocalConfig

This class contains configuration for local development. This should not be overridden on production, only on development. .. autoclass:: config.development.LocalConfig

members:

Tags

This class contains tags that are used throughout the project. Don’t use magic strings in your tags - use these.

class config.tags.Tags

ADDRESS = 'address'

ADMIN = 'administrative'

ALERT = 'alert'

ARC = 'arc'

ATHENA = 'athena'

AUDIT = 'audit'

AWS = 'aws'

BBM = 'bbm'

BEAD = 'bead'

BEDROCK = 'bedrock'

BOSS = 'boss'

BREAKFIX = 'breakfix'

CAFIIA = 'cafiia'

CAI = 'cai'

CALIX = 'calix'

CASF = 'casf'

CDC = 'cdc'

CENSUS = 'census'

COMPLIANCE = 'compliance'

CPC = 'cpc'

CURATE = 'curate'

DATAHUB = 'datahub'

DBT = 'dbt'

DEPRECATED = 'deprecated'

DIGITAL_EQUITY = 'digital_equity'

EACAM = 'eacam'

ECS = 'ecs'

EXAMPLE = 'example'

FABRIC = 'fabric'

FAS = 'fas'

FCC = 'fcc'

FINISHED = 'FINISHED'

FUNDING = 'funding'

GAME = 'game'

GCP = 'gcp'

GENERAL = 'general'

GEOMETRY_LIBRARY = 'geometry_library'

GEOTEL = 'geotel'

GLOBAL = 'global'

GOOGLE = 'google'

GOOGLE_MS_BSL = 'GOOGLE_MS_BUILDINGS'

GRAPI_API = 'grapi'

GROW = 'grow'

GUMBO = 'gumbo'

HLD = 'hld'

INDONESIA = 'indonesia'

INGEST = 'ingest'

IN_PROGRESS = 'in_progress'

LOB = 'lob'

LSD = 'lsd'

MANAGE = 'manage'

MBDR = 'minority_business_development'

MLFLWOW = 'mlflow'

MVR = 'mvr'

OUTDATED = 'OUTDATED'

OVERTURE = 'overture'

POI = 'poi'

POILOC = 'poiloc'

PPA = 'proposed_project_areas'

PUBLISH = 'publish'

RAD = 'RAD'

READY = 'READY'

RECONNECT = 'reconnect'

REDSHIFT = 'redshift'

REGRID = 'regrid'

SAFEGRAPH = 'safegraph'

SALES = 'sales'

SBO = 'sbo'

SCHEDULED = 'SCHEDULED'

SPEEDMAIN = 'speedmain'

SUPERSET = 'superset'

TCC = 'tcc'

TEST = 'test'

TILE = 'tile'

TRAINING = 'training'

TRANSFER = 'transfer'

TRANSFORM = 'transform'

TRIBE = 'tribe'

WIP = 'WIP'

WORLD = 'world'

S3Prefixes

This class contains S3 path prefixes used throughout the project (curated, raw, transformed, etc.).

class config.prefixes.S3Prefixes

BACKUP_PREFIX = 'backup'

CURATED_PREFIX = 'curated'

POSTGRES_PREFIX = 'postgres'

RAW_PREFIX = 'raw'

RAW_UPLOADS_PREFIX = 'uploads'

SUMMARY_RAW_PREFIX = 'summaryraw'

TEMP_PREFIX = 'temp'

TRANSFORMED_PREFIX = 'transformed'

RADPrefixes

This class contains RAD (Ready Address Discovery) table and queue prefixes.

class config.prefixes.RADPrefixes

DISCOVERY_FABRIC_EXTENSION_TABLE = 'fabric_extension'

DISCOVERY_INGEST_FABRIC_QUEUE = 'address.ingest.fabric'

DISCOVERY_INGEST_REGRID_QUEUE = 'address.ingest.regrid'

DISCOVERY_INGEST_SAFEGAPH_QUEUE = 'address.ingest.safegraph'

DISCOVERY_PRIMARY_ADDRESSES_TABLE = 'primary_addresses'

DISCOVERY_QUEUE = 'rad'

DISCOVERY_RIGRID_EXTENSION_TABLE = 'regrid_extension'

DISCOVERY_RIGRID_SECONDARY_TABLE = 'regrid_secondary_addresses'

DISCOVERY_SAFEGRAPH_EXTENSION_TABLE = 'safegraph_extension'

DISCOVERY_SECONDARY_ADDRESSES_TABLE = 'secondary_addresses'

DISCOVERY_SOURCES_TABLE = 'sources'

SOURCE_NAME_FABRIC = 'fabric'

SOURCE_NAME_RIGRID = 'regrid'

StateName

This class contains state and territory name mappings (state code to full name).

class config.statename.StateName

DEMO_STATE = {'XY': 'Demo Base', 'ZZ': 'Test Base'}

STATE = {'AK': 'Alaska', 'AL': 'Alabama', 'AR': 'Arkansas', 'AZ': 'Arizona', 'CA': 'California', 'CO': 'Colorado', 'CT': 'Connecticut', 'DE': 'Delaware', 'FL': 'Florida', 'GA': 'Georgia', 'HI': 'Hawaii', 'IA': 'Iowa', 'ID': 'Idaho', 'IL': 'Illinois', 'IN': 'Indiana', 'KS': 'Kansas', 'KY': 'Kentucky', 'LA': 'Louisiana', 'MA': 'Massachusetts', 'MD': 'Maryland', 'ME': 'Maine', 'MI': 'Michigan', 'MN': 'Minnesota', 'MO': 'Missouri', 'MS': 'Mississippi', 'MT': 'Montana', 'NC': 'North Carolina', 'ND': 'North Dakota', 'NE': 'Nebraska', 'NH': 'New Hampshire', 'NJ': 'New Jersey', 'NM': 'New Mexico', 'NV': 'Nevada', 'NY': 'New York', 'OH': 'Ohio', 'OK': 'Oklahoma', 'OR': 'Oregon', 'PA': 'Pennsylvania', 'RI': 'Rhode Island', 'SC': 'South Carolina', 'SD': 'South Dakota', 'TN': 'Tennessee', 'TX': 'Texas', 'UT': 'Utah', 'VA': 'Virginia', 'VT': 'Vermont', 'WA': 'Washington', 'WI': 'Wisconsin', 'WV': 'West Virginia', 'WY': 'Wyoming'}

STATE_ABBR_NAME = {'AK': 'Alaska', 'AL': 'Alabama', 'AR': 'Arkansas', 'AS': 'American Samoa', 'AZ': 'Arizona', 'CA': 'California', 'CO': 'Colorado', 'CT': 'Connecticut', 'DC': 'District of Columbia', 'DE': 'Delaware', 'FL': 'Florida', 'GA': 'Georgia', 'GU': 'Guam', 'HI': 'Hawaii', 'IA': 'Iowa', 'ID': 'Idaho', 'IL': 'Illinois', 'IN': 'Indiana', 'KS': 'Kansas', 'KY': 'Kentucky', 'LA': 'Louisiana', 'MA': 'Massachusetts', 'MD': 'Maryland', 'ME': 'Maine', 'MI': 'Michigan', 'MN': 'Minnesota', 'MO': 'Missouri', 'MP': 'Northern Mariana Islands', 'MS': 'Mississippi', 'MT': 'Montana', 'NC': 'North Carolina', 'ND': 'North Dakota', 'NE': 'Nebraska', 'NH': 'New Hampshire', 'NJ': 'New Jersey', 'NM': 'New Mexico', 'NV': 'Nevada', 'NY': 'New York', 'OH': 'Ohio', 'OK': 'Oklahoma', 'OR': 'Oregon', 'PA': 'Pennsylvania', 'PR': 'Puerto Rico', 'RI': 'Rhode Island', 'SC': 'South Carolina', 'SD': 'South Dakota', 'TN': 'Tennessee', 'TX': 'Texas', 'UM': 'U.S. Minor Outlying Islands', 'UT': 'Utah', 'VA': 'Virginia', 'VI': 'U.S. Virgin Islands', 'VT': 'Vermont', 'WA': 'Washington', 'WI': 'Wisconsin', 'WV': 'West Virginia', 'WY': 'Wyoming', 'XY': 'Demo Base', 'ZZ': 'Test Base'}

TERRITORY = {'AS': 'American Samoa', 'DC': 'District of Columbia', 'GU': 'Guam', 'MP': 'Northern Mariana Islands', 'PR': 'Puerto Rico', 'UM': 'U.S. Minor Outlying Islands', 'VI': 'U.S. Virgin Islands'}

ISO3166

This class contains ISO 3166 country codes and names.

class config.countries.ISO3166

countries = [{'alpha-3': 'AFG', 'country-code': '004', 'name': 'Afghanistan'}, {'alpha-3': 'ALA', 'country-code': '248', 'name': 'Åland Islands'}, {'alpha-3': 'ALB', 'country-code': '008', 'name': 'Albania'}, {'alpha-3': 'DZA', 'country-code': '012', 'name': 'Algeria'}, {'alpha-3': 'ASM', 'country-code': '016', 'name': 'American Samoa'}, {'alpha-3': 'AND', 'country-code': '020', 'name': 'Andorra'}, {'alpha-3': 'AGO', 'country-code': '024', 'name': 'Angola'}, {'alpha-3': 'AIA', 'country-code': '660', 'name': 'Anguilla'}, {'alpha-3': 'ATA', 'country-code': '010', 'name': 'Antarctica'}, {'alpha-3': 'ATG', 'country-code': '028', 'name': 'Antigua and Barbuda'}, {'alpha-3': 'ARG', 'country-code': '032', 'name': 'Argentina'}, {'alpha-3': 'ARM', 'country-code': '051', 'name': 'Armenia'}, {'alpha-3': 'ABW', 'country-code': '533', 'name': 'Aruba'}, {'alpha-3': 'AUS', 'country-code': '036', 'name': 'Australia'}, {'alpha-3': 'AUT', 'country-code': '040', 'name': 'Austria'}, {'alpha-3': 'AZE', 'country-code': '031', 'name': 'Azerbaijan'}, {'alpha-3': 'BHS', 'country-code': '044', 'name': 'Bahamas'}, {'alpha-3': 'BHR', 'country-code': '048', 'name': 'Bahrain'}, {'alpha-3': 'BGD', 'country-code': '050', 'name': 'Bangladesh'}, {'alpha-3': 'BRB', 'country-code': '052', 'name': 'Barbados'}, {'alpha-3': 'BLR', 'country-code': '112', 'name': 'Belarus'}, {'alpha-3': 'BEL', 'country-code': '056', 'name': 'Belgium'}, {'alpha-3': 'BLZ', 'country-code': '084', 'name': 'Belize'}, {'alpha-3': 'BEN', 'country-code': '204', 'name': 'Benin'}, {'alpha-3': 'BMU', 'country-code': '060', 'name': 'Bermuda'}, {'alpha-3': 'BTN', 'country-code': '064', 'name': 'Bhutan'}, {'alpha-3': 'BOL', 'country-code': '068', 'name': 'Bolivia, Plurinational State of'}, {'alpha-3': 'BES', 'country-code': '535', 'name': 'Bonaire, Sint Eustatius and Saba'}, {'alpha-3': 'BIH', 'country-code': '070', 'name': 'Bosnia and Herzegovina'}, {'alpha-3': 'BWA', 'country-code': '072', 'name': 'Botswana'}, {'alpha-3': 'BVT', 'country-code': '074', 'name': 'Bouvet Island'}, {'alpha-3': 'BRA', 'country-code': '076', 'name': 'Brazil'}, {'alpha-3': 'IOT', 'country-code': '086', 'name': 'British Indian Ocean Territory'}, {'alpha-3': 'BRN', 'country-code': '096', 'name': 'Brunei Darussalam'}, {'alpha-3': 'BGR', 'country-code': '100', 'name': 'Bulgaria'}, {'alpha-3': 'BFA', 'country-code': '854', 'name': 'Burkina Faso'}, {'alpha-3': 'BDI', 'country-code': '108', 'name': 'Burundi'}, {'alpha-3': 'CPV', 'country-code': '132', 'name': 'Cabo Verde'}, {'alpha-3': 'KHM', 'country-code': '116', 'name': 'Cambodia'}, {'alpha-3': 'CMR', 'country-code': '120', 'name': 'Cameroon'}, {'alpha-3': 'CAN', 'country-code': '124', 'name': 'Canada'}, {'alpha-3': 'CYM', 'country-code': '136', 'name': 'Cayman Islands'}, {'alpha-3': 'CAF', 'country-code': '140', 'name': 'Central African Republic'}, {'alpha-3': 'TCD', 'country-code': '148', 'name': 'Chad'}, {'alpha-3': 'CHL', 'country-code': '152', 'name': 'Chile'}, {'alpha-3': 'CHN', 'country-code': '156', 'name': 'China'}, {'alpha-3': 'CXR', 'country-code': '162', 'name': 'Christmas Island'}, {'alpha-3': 'CCK', 'country-code': '166', 'name': 'Cocos (Keeling) Islands'}, {'alpha-3': 'COL', 'country-code': '170', 'name': 'Colombia'}, {'alpha-3': 'COM', 'country-code': '174', 'name': 'Comoros'}, {'alpha-3': 'COG', 'country-code': '178', 'name': 'Congo'}, {'alpha-3': 'COD', 'country-code': '180', 'name': 'Congo, Democratic Republic of the'}, {'alpha-3': 'COK', 'country-code': '184', 'name': 'Cook Islands'}, {'alpha-3': 'CRI', 'country-code': '188', 'name': 'Costa Rica'}, {'alpha-3': 'CIV', 'country-code': '384', 'name': "Côte d'Ivoire"}, {'alpha-3': 'HRV', 'country-code': '191', 'name': 'Croatia'}, {'alpha-3': 'CUB', 'country-code': '192', 'name': 'Cuba'}, {'alpha-3': 'CUW', 'country-code': '531', 'name': 'Curaçao'}, {'alpha-3': 'CYP', 'country-code': '196', 'name': 'Cyprus'}, {'alpha-3': 'CZE', 'country-code': '203', 'name': 'Czechia'}, {'alpha-3': 'DNK', 'country-code': '208', 'name': 'Denmark'}, {'alpha-3': 'DJI', 'country-code': '262', 'name': 'Djibouti'}, {'alpha-3': 'DMA', 'country-code': '212', 'name': 'Dominica'}, {'alpha-3': 'DOM', 'country-code': '214', 'name': 'Dominican Republic'}, {'alpha-3': 'ECU', 'country-code': '218', 'name': 'Ecuador'}, {'alpha-3': 'EGY', 'country-code': '818', 'name': 'Egypt'}, {'alpha-3': 'SLV', 'country-code': '222', 'name': 'El Salvador'}, {'alpha-3': 'GNQ', 'country-code': '226', 'name': 'Equatorial Guinea'}, {'alpha-3': 'ERI', 'country-code': '232', 'name': 'Eritrea'}, {'alpha-3': 'EST', 'country-code': '233', 'name': 'Estonia'}, {'alpha-3': 'SWZ', 'country-code': '748', 'name': 'Eswatini'}, {'alpha-3': 'ETH', 'country-code': '231', 'name': 'Ethiopia'}, {'alpha-3': 'FLK', 'country-code': '238', 'name': 'Falkland Islands (Malvinas)'}, {'alpha-3': 'FRO', 'country-code': '234', 'name': 'Faroe Islands'}, {'alpha-3': 'FJI', 'country-code': '242', 'name': 'Fiji'}, {'alpha-3': 'FIN', 'country-code': '246', 'name': 'Finland'}, {'alpha-3': 'FRA', 'country-code': '250', 'name': 'France'}, {'alpha-3': 'GUF', 'country-code': '254', 'name': 'French Guiana'}, {'alpha-3': 'PYF', 'country-code': '258', 'name': 'French Polynesia'}, {'alpha-3': 'ATF', 'country-code': '260', 'name': 'French Southern Territories'}, {'alpha-3': 'GAB', 'country-code': '266', 'name': 'Gabon'}, {'alpha-3': 'GMB', 'country-code': '270', 'name': 'Gambia'}, {'alpha-3': 'GEO', 'country-code': '268', 'name': 'Georgia'}, {'alpha-3': 'DEU', 'country-code': '276', 'name': 'Germany'}, {'alpha-3': 'GHA', 'country-code': '288', 'name': 'Ghana'}, {'alpha-3': 'GIB', 'country-code': '292', 'name': 'Gibraltar'}, {'alpha-3': 'GRC', 'country-code': '300', 'name': 'Greece'}, {'alpha-3': 'GRL', 'country-code': '304', 'name': 'Greenland'}, {'alpha-3': 'GRD', 'country-code': '308', 'name': 'Grenada'}, {'alpha-3': 'GLP', 'country-code': '312', 'name': 'Guadeloupe'}, {'alpha-3': 'GUM', 'country-code': '316', 'name': 'Guam'}, {'alpha-3': 'GTM', 'country-code': '320', 'name': 'Guatemala'}, {'alpha-3': 'GGY', 'country-code': '831', 'name': 'Guernsey'}, {'alpha-3': 'GIN', 'country-code': '324', 'name': 'Guinea'}, {'alpha-3': 'GNB', 'country-code': '624', 'name': 'Guinea-Bissau'}, {'alpha-3': 'GUY', 'country-code': '328', 'name': 'Guyana'}, {'alpha-3': 'HTI', 'country-code': '332', 'name': 'Haiti'}, {'alpha-3': 'HMD', 'country-code': '334', 'name': 'Heard Island and McDonald Islands'}, {'alpha-3': 'VAT', 'country-code': '336', 'name': 'Holy See'}, {'alpha-3': 'HND', 'country-code': '340', 'name': 'Honduras'}, {'alpha-3': 'HKG', 'country-code': '344', 'name': 'Hong Kong'}, {'alpha-3': 'HUN', 'country-code': '348', 'name': 'Hungary'}, {'alpha-3': 'ISL', 'country-code': '352', 'name': 'Iceland'}, {'alpha-3': 'IND', 'country-code': '356', 'name': 'India'}, {'alpha-3': 'IDN', 'country-code': '360', 'name': 'Indonesia'}, {'alpha-3': 'IRN', 'country-code': '364', 'name': 'Iran, Islamic Republic of'}, {'alpha-3': 'IRQ', 'country-code': '368', 'name': 'Iraq'}, {'alpha-3': 'IRL', 'country-code': '372', 'name': 'Ireland'}, {'alpha-3': 'IMN', 'country-code': '833', 'name': 'Isle of Man'}, {'alpha-3': 'ISR', 'country-code': '376', 'name': 'Israel'}, {'alpha-3': 'ITA', 'country-code': '380', 'name': 'Italy'}, {'alpha-3': 'JAM', 'country-code': '388', 'name': 'Jamaica'}, {'alpha-3': 'JPN', 'country-code': '392', 'name': 'Japan'}, {'alpha-3': 'JEY', 'country-code': '832', 'name': 'Jersey'}, {'alpha-3': 'JOR', 'country-code': '400', 'name': 'Jordan'}, {'alpha-3': 'KAZ', 'country-code': '398', 'name': 'Kazakhstan'}, {'alpha-3': 'KEN', 'country-code': '404', 'name': 'Kenya'}, {'alpha-3': 'KIR', 'country-code': '296', 'name': 'Kiribati'}, {'alpha-3': 'PRK', 'country-code': '408', 'name': "Korea, Democratic People's Republic of"}, {'alpha-3': 'KOR', 'country-code': '410', 'name': 'Korea, Republic of'}, {'alpha-3': 'KWT', 'country-code': '414', 'name': 'Kuwait'}, {'alpha-3': 'KGZ', 'country-code': '417', 'name': 'Kyrgyzstan'}, {'alpha-3': 'LAO', 'country-code': '418', 'name': "Lao People's Democratic Republic"}, {'alpha-3': 'LVA', 'country-code': '428', 'name': 'Latvia'}, {'alpha-3': 'LBN', 'country-code': '422', 'name': 'Lebanon'}, {'alpha-3': 'LSO', 'country-code': '426', 'name': 'Lesotho'}, {'alpha-3': 'LBR', 'country-code': '430', 'name': 'Liberia'}, {'alpha-3': 'LBY', 'country-code': '434', 'name': 'Libya'}, {'alpha-3': 'LIE', 'country-code': '438', 'name': 'Liechtenstein'}, {'alpha-3': 'LTU', 'country-code': '440', 'name': 'Lithuania'}, {'alpha-3': 'LUX', 'country-code': '442', 'name': 'Luxembourg'}, {'alpha-3': 'MAC', 'country-code': '446', 'name': 'Macao'}, {'alpha-3': 'MDG', 'country-code': '450', 'name': 'Madagascar'}, {'alpha-3': 'MWI', 'country-code': '454', 'name': 'Malawi'}, {'alpha-3': 'MYS', 'country-code': '458', 'name': 'Malaysia'}, {'alpha-3': 'MDV', 'country-code': '462', 'name': 'Maldives'}, {'alpha-3': 'MLI', 'country-code': '466', 'name': 'Mali'}, {'alpha-3': 'MLT', 'country-code': '470', 'name': 'Malta'}, {'alpha-3': 'MHL', 'country-code': '584', 'name': 'Marshall Islands'}, {'alpha-3': 'MTQ', 'country-code': '474', 'name': 'Martinique'}, {'alpha-3': 'MRT', 'country-code': '478', 'name': 'Mauritania'}, {'alpha-3': 'MUS', 'country-code': '480', 'name': 'Mauritius'}, {'alpha-3': 'MYT', 'country-code': '175', 'name': 'Mayotte'}, {'alpha-3': 'MEX', 'country-code': '484', 'name': 'Mexico'}, {'alpha-3': 'FSM', 'country-code': '583', 'name': 'Micronesia, Federated States of'}, {'alpha-3': 'MDA', 'country-code': '498', 'name': 'Moldova, Republic of'}, {'alpha-3': 'MCO', 'country-code': '492', 'name': 'Monaco'}, {'alpha-3': 'MNG', 'country-code': '496', 'name': 'Mongolia'}, {'alpha-3': 'MNE', 'country-code': '499', 'name': 'Montenegro'}, {'alpha-3': 'MSR', 'country-code': '500', 'name': 'Montserrat'}, {'alpha-3': 'MAR', 'country-code': '504', 'name': 'Morocco'}, {'alpha-3': 'MOZ', 'country-code': '508', 'name': 'Mozambique'}, {'alpha-3': 'MMR', 'country-code': '104', 'name': 'Myanmar'}, {'alpha-3': 'NAM', 'country-code': '516', 'name': 'Namibia'}, {'alpha-3': 'NRU', 'country-code': '520', 'name': 'Nauru'}, {'alpha-3': 'NPL', 'country-code': '524', 'name': 'Nepal'}, {'alpha-3': 'NLD', 'country-code': '528', 'name': 'Netherlands, Kingdom of the'}, {'alpha-3': 'NCL', 'country-code': '540', 'name': 'New Caledonia'}, {'alpha-3': 'NZL', 'country-code': '554', 'name': 'New Zealand'}, {'alpha-3': 'NIC', 'country-code': '558', 'name': 'Nicaragua'}, {'alpha-3': 'NER', 'country-code': '562', 'name': 'Niger'}, {'alpha-3': 'NGA', 'country-code': '566', 'name': 'Nigeria'}, {'alpha-3': 'NIU', 'country-code': '570', 'name': 'Niue'}, {'alpha-3': 'NFK', 'country-code': '574', 'name': 'Norfolk Island'}, {'alpha-3': 'MKD', 'country-code': '807', 'name': 'North Macedonia'}, {'alpha-3': 'MNP', 'country-code': '580', 'name': 'Northern Mariana Islands'}, {'alpha-3': 'NOR', 'country-code': '578', 'name': 'Norway'}, {'alpha-3': 'OMN', 'country-code': '512', 'name': 'Oman'}, {'alpha-3': 'PAK', 'country-code': '586', 'name': 'Pakistan'}, {'alpha-3': 'PLW', 'country-code': '585', 'name': 'Palau'}, {'alpha-3': 'PSE', 'country-code': '275', 'name': 'Palestine, State of'}, {'alpha-3': 'PAN', 'country-code': '591', 'name': 'Panama'}, {'alpha-3': 'PNG', 'country-code': '598', 'name': 'Papua New Guinea'}, {'alpha-3': 'PRY', 'country-code': '600', 'name': 'Paraguay'}, {'alpha-3': 'PER', 'country-code': '604', 'name': 'Peru'}, {'alpha-3': 'PHL', 'country-code': '608', 'name': 'Philippines'}, {'alpha-3': 'PCN', 'country-code': '612', 'name': 'Pitcairn'}, {'alpha-3': 'POL', 'country-code': '616', 'name': 'Poland'}, {'alpha-3': 'PRT', 'country-code': '620', 'name': 'Portugal'}, {'alpha-3': 'PRI', 'country-code': '630', 'name': 'Puerto Rico'}, {'alpha-3': 'QAT', 'country-code': '634', 'name': 'Qatar'}, {'alpha-3': 'REU', 'country-code': '638', 'name': 'Réunion'}, {'alpha-3': 'ROU', 'country-code': '642', 'name': 'Romania'}, {'alpha-3': 'RUS', 'country-code': '643', 'name': 'Russian Federation'}, {'alpha-3': 'RWA', 'country-code': '646', 'name': 'Rwanda'}, {'alpha-3': 'BLM', 'country-code': '652', 'name': 'Saint Barthélemy'}, {'alpha-3': 'SHN', 'country-code': '654', 'name': 'Saint Helena, Ascension and Tristan da Cunha'}, {'alpha-3': 'KNA', 'country-code': '659', 'name': 'Saint Kitts and Nevis'}, {'alpha-3': 'LCA', 'country-code': '662', 'name': 'Saint Lucia'}, {'alpha-3': 'MAF', 'country-code': '663', 'name': 'Saint Martin (French part)'}, {'alpha-3': 'SPM', 'country-code': '666', 'name': 'Saint Pierre and Miquelon'}, {'alpha-3': 'VCT', 'country-code': '670', 'name': 'Saint Vincent and the Grenadines'}, {'alpha-3': 'WSM', 'country-code': '882', 'name': 'Samoa'}, {'alpha-3': 'SMR', 'country-code': '674', 'name': 'San Marino'}, {'alpha-3': 'STP', 'country-code': '678', 'name': 'Sao Tome and Principe'}, {'alpha-3': 'SAU', 'country-code': '682', 'name': 'Saudi Arabia'}, {'alpha-3': 'SEN', 'country-code': '686', 'name': 'Senegal'}, {'alpha-3': 'SRB', 'country-code': '688', 'name': 'Serbia'}, {'alpha-3': 'SYC', 'country-code': '690', 'name': 'Seychelles'}, {'alpha-3': 'SLE', 'country-code': '694', 'name': 'Sierra Leone'}, {'alpha-3': 'SGP', 'country-code': '702', 'name': 'Singapore'}, {'alpha-3': 'SXM', 'country-code': '534', 'name': 'Sint Maarten (Dutch part)'}, {'alpha-3': 'SVK', 'country-code': '703', 'name': 'Slovakia'}, {'alpha-3': 'SVN', 'country-code': '705', 'name': 'Slovenia'}, {'alpha-3': 'SLB', 'country-code': '090', 'name': 'Solomon Islands'}, {'alpha-3': 'SOM', 'country-code': '706', 'name': 'Somalia'}, {'alpha-3': 'ZAF', 'country-code': '710', 'name': 'South Africa'}, {'alpha-3': 'SGS', 'country-code': '239', 'name': 'South Georgia and the South Sandwich Islands'}, {'alpha-3': 'SSD', 'country-code': '728', 'name': 'South Sudan'}, {'alpha-3': 'ESP', 'country-code': '724', 'name': 'Spain'}, {'alpha-3': 'LKA', 'country-code': '144', 'name': 'Sri Lanka'}, {'alpha-3': 'SDN', 'country-code': '729', 'name': 'Sudan'}, {'alpha-3': 'SUR', 'country-code': '740', 'name': 'Suriname'}, {'alpha-3': 'SJM', 'country-code': '744', 'name': 'Svalbard and Jan Mayen'}, {'alpha-3': 'SWE', 'country-code': '752', 'name': 'Sweden'}, {'alpha-3': 'CHE', 'country-code': '756', 'name': 'Switzerland'}, {'alpha-3': 'SYR', 'country-code': '760', 'name': 'Syrian Arab Republic'}, {'alpha-3': 'TWN', 'country-code': '158', 'name': 'Taiwan, Province of China'}, {'alpha-3': 'TJK', 'country-code': '762', 'name': 'Tajikistan'}, {'alpha-3': 'TZA', 'country-code': '834', 'name': 'Tanzania, United Republic of'}, {'alpha-3': 'THA', 'country-code': '764', 'name': 'Thailand'}, {'alpha-3': 'TLS', 'country-code': '626', 'name': 'Timor-Leste'}, {'alpha-3': 'TGO', 'country-code': '768', 'name': 'Togo'}, {'alpha-3': 'TKL', 'country-code': '772', 'name': 'Tokelau'}, {'alpha-3': 'TON', 'country-code': '776', 'name': 'Tonga'}, {'alpha-3': 'TTO', 'country-code': '780', 'name': 'Trinidad and Tobago'}, {'alpha-3': 'TUN', 'country-code': '788', 'name': 'Tunisia'}, {'alpha-3': 'TUR', 'country-code': '792', 'name': 'Türkiye'}, {'alpha-3': 'TKM', 'country-code': '795', 'name': 'Turkmenistan'}, {'alpha-3': 'TCA', 'country-code': '796', 'name': 'Turks and Caicos Islands'}, {'alpha-3': 'TUV', 'country-code': '798', 'name': 'Tuvalu'}, {'alpha-3': 'UGA', 'country-code': '800', 'name': 'Uganda'}, {'alpha-3': 'UKR', 'country-code': '804', 'name': 'Ukraine'}, {'alpha-3': 'ARE', 'country-code': '784', 'name': 'United Arab Emirates'}, {'alpha-3': 'GBR', 'country-code': '826', 'name': 'United Kingdom of Great Britain and Northern Ireland'}, {'alpha-3': 'USA', 'country-code': '840', 'name': 'United States of America'}, {'alpha-3': 'UMI', 'country-code': '581', 'name': 'United States Minor Outlying Islands'}, {'alpha-3': 'URY', 'country-code': '858', 'name': 'Uruguay'}, {'alpha-3': 'UZB', 'country-code': '860', 'name': 'Uzbekistan'}, {'alpha-3': 'VUT', 'country-code': '548', 'name': 'Vanuatu'}, {'alpha-3': 'VEN', 'country-code': '862', 'name': 'Venezuela, Bolivarian Republic of'}, {'alpha-3': 'VNM', 'country-code': '704', 'name': 'Viet Nam'}, {'alpha-3': 'VGB', 'country-code': '092', 'name': 'Virgin Islands (British)'}, {'alpha-3': 'VIR', 'country-code': '850', 'name': 'Virgin Islands (U.S.)'}, {'alpha-3': 'WLF', 'country-code': '876', 'name': 'Wallis and Futuna'}, {'alpha-3': 'ESH', 'country-code': '732', 'name': 'Western Sahara'}, {'alpha-3': 'YEM', 'country-code': '887', 'name': 'Yemen'}, {'alpha-3': 'ZMB', 'country-code': '894', 'name': 'Zambia'}, {'alpha-3': 'ZWE', 'country-code': '716', 'name': 'Zimbabwe'}]

SQLReference

This class contains SQL file references for DAGs. Generated by dev_tools/generate_sql_file_class.sh.

class config.sqlreference.SQLReference

SQL_FILES = {'/home/ubuntu/airflow/dags/arc/demo_tenant/seed_customer_tenant/SQL': ['enum_labels.sql', 'user_fk_columns.sql', 'owned_sequences.sql', 'seed_sets.sql', 'column_compatibility.sql', 'divergence_check.sql'], '/home/ubuntu/airflow/dags/arc/demo_tenant/stage0_mock_data/dbt_project/models': ['c_xy_fpmock_tile.sql', 'c_xy_fpmock_subgrantees.sql', 'c_xy_fpmock_supp_grantor_users.sql', 'c_xy_fpmock_supp_subgrant_agreements.sql', 'c_xy_fpmock_supp_reporting_projects.sql', 'c_xy_fpmock_supp_users.sql', 'c_xy_fpmock_no_bead_locations.sql', 'c_xy_fpmock_supp_deployment_reporting.sql', 'c_xy_fpmock_supp_reporting_project_cai.sql', 'c_xy_fpmock_locations.sql', 'c_xy_fpmock_deployment_projects_internal.sql', 'c_xy_fpmock_supp_reporting_project_locations.sql', 'c_xy_fpmock_cai.sql', 'c_xy_fpmock_deployment_projects.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/archive_projects/SQL': ['export_location_summary.sql', 'export_subgrant_agreements.sql', 'close_requests.sql', 'archive_deployment_projects.sql', 'cancel_reminder_jobs.sql', 'resolve.sql', 'export_application_locations.sql', '_targets.sql', 'preview.sql', 'archive_locations.sql', 'archive_project_rows.sql', 'export_report_chains.sql', 'archive_reports.sql', 'preimage.sql', 'export_projects.sql', 'archive_subgrant_agreements.sql', 'export_requests.sql', 'export_deployment_projects.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/breakfix/fix_missing_app_answer/SQL': ['reporting_project_answers.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/breakfix/fix_version_date/SQL': ['update_version_date.sql', 'census.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/breakfix/in_joink_ocremc_removal/SQL': ['preview_applications.sql', 'count.sql', 'delete.sql', 'preview_applicants.sql', 'preview_locations.sql', 'preview_children.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/breakfix/in_joink_ocremc_restore/SQL': ['update_derived_columns.sql', 'update_status.sql', 'source_files.sql', 'source_locations.sql', 'verify.sql', 'source_applications.sql', 'repair_array_columns.sql', 'source_answers.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/breakfix/va_baba_report_attachment_backup/SQL': ['baba_attachments.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/ee_validate_files/SQL/version_comparison': ['sample_removed.sql', 'sample_added.sql', 'count_rows.sql', 'count_removed.sql', 'project_breakdown_modified.sql', 'count_added.sql', 'project_breakdown_full.sql', 'project_breakdown_removed.sql', 'sample_modified.sql', 'project_breakdown_added.sql', 'count_modified.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/ee_validate_files/dbt_project/models': ['schema.yml'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/fp_grapi_cleanup/SQL/count': ['applicant.sql', 'application_location_answer.sql', 'application_request.sql', 'application_relation.sql', 'permit_application.sql', 'application_request_revision.sql', 'application.sql', 'child_application.sql', 'application_location.sql', 'application_answer.sql', 'application_activity.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/fp_grapi_cleanup/SQL/delete': ['application_location_answer.sql', 'application_notes.sql', 'application_request.sql', 'application_relation.sql', 'permit_application.sql', 'child_applications.sql', 'application_request_revision.sql', 'application_location.sql', 'target_applications.sql', 'application_answer.sql', 'application_activity.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/fp_grapi_cleanup/SQL/lookups': ['get_child_apps.sql', 'get_target_apps.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/fp_sync_lake_from_grapi/dbt_project/models': ['schema.yml', 't_fp_application_rp.sql', 't_fp_application_sa.sql', 'x_fp_deployment_reporting.sql', 't_fp_reporting_projects.sql', 'x_fp_reporting_project_cai.sql', 't_fp_application_rp_wip.sql', 't_fp_cai.sql', 'x_fp_locations.sql', 't_fp_deployment_projects_answers.sql', 't_fp_subgrant_agreements.sql', 'x_fp_subgrant_agreements.sql', 't_fp_subgrantees.sql', 't_fp_reporting_project_cai.sql', 't_fp_deployment_projects.sql', 'x_fp_deployment_projects.sql', 'x_fp_reporting_project_locations.sql', 'x_fp_subgrantees.sql', 't_fp_locations.sql', 't_fp_application_sa_wip.sql', 'x_fp_reporting_projects.sql', 't_fp_subgrant_agreement_projects_answers.sql', 't_fp_subgrant_agreement_dp_map.sql', 't_fp_reporting_projects_answers.sql', 't_fp_application_dp_wip.sql', 't_fp_deployment_reporting.sql', 't_fp_reporting_project_locations.sql', 'x_fp_cai.sql', 't_fp_no_bead_locations.sql', 'x_fp_no_bead_locations.sql', 't_fp_application_dp.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/sga_dp_migration/SQL': ['verification_after.sql', 'verification_before.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/sga_rp_migration/SQL': ['00_delete_old_sga_answers_count.sql', 'verification_after.sql', '00_delete_old_sga_answers_update.sql', '02_delete_old_sga_apps_count.sql', '01_delete_old_relations_update.sql', 'verification_before.sql', '02_delete_old_sga_apps_update.sql', '01_delete_old_relations_count.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage00_controller/dbt_project/models': ['schema.yml', 't_fp_delta_relations.sql', 't_fp_delta_answer_orphans.sql', 't_fp_delta_renames.sql', 't_fp_delta_projects.sql', 't_fp_delta_locations.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage0_prep/dbt_project/models': ['schema.yml', 't_fp_reporting_projects.sql', 't_fp_cai.sql', 't_fp_deployment_projects_answers.sql', 't_fp_subgrant_agreements.sql', 't_fp_users.sql', 't_fp_subgrantees.sql', 't_fp_grantor_users.sql', 't_fp_reporting_project_cai.sql', 't_fp_deployment_projects.sql', 't_fp_locations.sql', 't_fp_subgrant_agreement_projects_answers.sql', 't_fp_subgrant_agreement_dp_map.sql', 't_fp_reporting_projects_answers.sql', 't_fp_deployment_reporting.sql', 't_fp_reporting_project_locations.sql', 't_fp_no_bead_locations.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage1_subgrantees/SQL': ['TX.sql', 'XY.sql', 'MN.sql', 'CA.sql', 'VA.sql', 'LA.sql', 'AR.sql', 'OK.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects/SQL/DeploymentProject': ['pg_insert.sql', 'load_lake.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects/SQL/ReportingProject': ['pg_insert.sql', 'load_lake.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects/SQL/SubgrantAgreement': ['pg_insert.sql', 'load_lake.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects/dbt_project/models': ['t_fp_application_rp.sql', 't_fp_application_sa.sql', 't_fp_application_dp.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects_wip/SQL': ['delete_project.sql', 'guard_unhandled_children.sql', 'rename_project.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects_wip/SQL/DeploymentProject': ['pg_insert.sql', 'pg_update.sql', 'load_lake.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects_wip/SQL/ReportingProject': ['pg_insert.sql', 'pg_update.sql', 'load_lake.sql', 'sync_child_application_owner.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects_wip/SQL/SubgrantAgreement': ['pg_insert.sql', 'pg_update.sql', 'load_lake.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage2_projects_wip/dbt_project/models': ['t_fp_application_rp.sql', 't_fp_application_sa.sql', 't_fp_application_sa_compat.sql', 't_fp_application_dp_compat.sql', 't_fp_application_rp_compat.sql', 't_fp_application_dp.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations/SQL/lookups': ['pg_select_app_ids_by_status.sql', 'pg_select_app_locations.sql', 'pg_select_app_location_answers.sql', 'ath_select_matched_applicants.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations/SQL/n_to_m': ['cai_ReportingProject.sql', 'bsl_ReportingProject.sql', 'cai_ReportingProject_CA.sql', 'cai_ReportingProject_VA.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations/SQL/no_bead': ['extract_no_bead_attributes.sql', 'update_pit_no_bead_attributes.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations/SQL/one_to_one': ['cai_ReportingProject.sql', 'bsl_DeploymentProject.sql', 'bsl_ReportingProject.sql', 'cai_DeploymentProject_VA.sql', 'cai_DeploymentProject.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations/dbt_project/models': ['schema.yml', 'x_export_nepa_locations.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations_wip/SQL/lookups': ['pg_select_app_ids_by_status.sql', 'pg_select_app_locations.sql', 'pg_select_app_location_answers.sql', 'ath_select_matched_applicants.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations_wip/SQL/n_to_m': ['cai_ReportingProject.sql', 'bsl_ReportingProject.sql', 'cai_ReportingProject_CA.sql', 'cai_ReportingProject_VA.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations_wip/SQL/no_bead': ['extract_no_bead_attributes.sql', 'update_pit_no_bead_attributes.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations_wip/SQL/one_to_one': ['cai_ReportingProject.sql', 'bsl_DeploymentProject.sql', 'bsl_ReportingProject.sql', 'cai_DeploymentProject_VA.sql', 'cai_DeploymentProject.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage3_locations_wip/dbt_project/models': ['schema.yml', 'x_export_nepa_locations.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage4_application_relation/SQL': ['r_xyup_supp_awarded_application_projects.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage4_application_relation_wip/SQL': ['r_xyup_supp_awarded_application_projects.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage4b_llu_reports/SQL': ['pg_insert_proof_of_passings.sql', 'pg_preview.sql', 'pg_clone_application_locations.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage5_application_answer_wip/dbt_project/models': ['schema.yml', 't_fp_location_place_input.sql', 't_fp_answer_targets.sql', 't_fp_location_place_resolution.sql', 't_fp_answer_project.sql', 't_fp_answer_location_counts.sql', 't_fp_answer_location_places.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage5_extra_application_answers/SQL': ['all_update_location_counts.sql', 'all_active_location_parity.sql', 'all_places.sql', 'all_location_version_diagnostics.sql', 'all_project_answers.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage6_users/dbt_project/models': ['t_fp_stage_user_access.sql', 't_fp_stage_users.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage8_bead_phub/dbt_project/models': ['schema.yml', 'c_cpc_merge_tile_final_proposal_grapi.sql', 'phub_merge.sql', 'phub_speedrank_qaqc.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/stage9_export/dbt_project/models': ['schema.yml', 'x_fp_deployment_reporting.sql', 'x_fp_reporting_project_cai.sql', 'x_fp_locations.sql', 'x_fp_subgrant_agreements.sql', 'x_fp_deployment_projects.sql', 'x_fp_reporting_project_locations.sql', 'x_fp_subgrantees.sql', 'x_fp_reporting_projects.sql', 'x_fp_cai.sql', 'x_fp_no_bead_locations.sql'], '/home/ubuntu/airflow/dags/arc/final_proposal_ingest/version_compare/dbt_project/models': ['t_fp_compare_versions_rp.sql', 't_fp_compare_versions_dp.sql', 'sources.yml'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage1_ingest_schema/dbt_project/models': ['schema.yml', 'sources.yml'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage1_ingest_schema/dbt_project/models/az': ['c_outbound_spec_az.sql', 'c_inbound_spec_az.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage1_ingest_schema/dbt_project/models/mn': ['c_outbound_spec_mn.sql', 'c_inbound_spec_mn.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage1_ingest_schema/dbt_project/models/tx': ['c_outbound_spec_tx.sql', 'c_inbound_spec_tx.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage2_ingest_vendor_details/SQL': ['texas_pca_codes.sql', 'minnesota.sql', 'texas_vendor_onboarding.sql', 'arizona.sql', 'texas_tmap_pca_codes.sql', 'virginia.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage3_outbound_generator/dbt_project/models': ['schema.yml'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage3_outbound_generator/dbt_project/models/intermediate': ['int_latest_reimbursements.sql', 'int_eligible_requests.sql', 'int_reimbursement_versions.sql', 'int_request_details.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage3_outbound_generator/dbt_project/models/marts': ['c_mart_payment_queue.sql', 'c_payment_batch.sql', 'c_payment_relationship.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage3_outbound_generator/dbt_project/models/staging': ['stg_application_relations.sql', 'stg_application_po.sql', 'stg_invoice_data_tx.sql', 'stg_application_answers.sql', 'stg_applicant_profiles.sql', 'stg_applications.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage3b_outbound_summary/dbt_project/models': ['schema.yml', 'sources.yml'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage3b_outbound_summary/dbt_project/models/tx': ['c_outbound_summary_tx.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage4_inbound_processor/dbt_project/models': ['schema.yml', 'sources.yml'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage4_inbound_processor/dbt_project/models/intermediate': ['int_payment_matches.sql', 'int_unmatched_records.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage4_inbound_processor/dbt_project/models/marts': ['c_payment_status_updates.sql'], '/home/ubuntu/airflow/dags/arc/financial_integration/stage4_inbound_processor/dbt_project/models/staging': ['stg_payment_confirmations.sql', 'stg_payment_relationships.sql'], '/home/ubuntu/airflow/dags/arc/ne_project_plan_bulk_load/SQL': ['apply.sql'], '/home/ubuntu/airflow/dags/arc/nepa/ingest_locations/dbt_project/models': ['schema.yml', 'c_nepa_locations_in.sql', 'c_nepa_locations_ar.sql', 'c_nepa_locations.sql', 'c_nepa_locations_va.sql', 'c_nepa_locations_mn.sql', 'c_nepa_locations_az.sql', 'c_nepa_locations_tx.sql', 'sources.yml'], '/home/ubuntu/airflow/dags/arc/one_offs/SQL': ['arc_828.sql', 'arc_494_revision_request_evidence.sql', 'arc_747_2_update_lastreimbursementnumber.sql', 'arc_441_4_applicationanswer_reportingperiod.sql', 'arc_441_1_application_submittedat.sql', 'arc_747_1_update_currentreimbursementnumber.sql', 'va_1440_fix_location_count.sql', 'arc_672_2_update_grant_remaining.sql', 'arc_672_3_update_lastreimbursementnumber.sql', 'arc_406_ar_remap_application_applicant.sql', 'xy_reset_rp_dp.sql', 'il_delete_dup_bsl_applicationlocation_20260731.sql', 'arc_672_fix_1_update_grant_requests.sql', 'arc_819_2.sql', 'ar_4_drop_old_tables.sql', 'arc_441_2_application_dueat.sql', 'arc_819_1.sql', 'arc_672_1_update_grant_requests.sql', 'arc_484_update_boone.sql', 'arc_759_1.sql', 'arc_759_3.sql', 'arc_441_3_application_name.sql', 'arc_759_2.sql', 'ar_2_restore_backup_tables.sql', 'arc_799.sql', 'arc_818_delete_version20250813.sql', 'arc_441_dueat_submittedat_name.sql', 'ar_delete_all.sql', 'ms_update_report_status.sql', 'arc_759_2a.sql', 'arc_818_dev.sql', 'arc_871_1.sql', 'la_delete_all_by_applicant.sql', 'ar_3_drop_backup_tables.sql', 'arc_827.sql', 'arc_672_revert_1_update_grant_requests.sql', 'ar_1_backup_tables.sql', 'arc_871_2.sql', 'arc_746_update_reimbursementrequestnumber.sql'], '/home/ubuntu/airflow/dags/arc/phub_agg_data/SQL': ['town_agg.sql', 'congressional_district_agg.sql', 'county_agg.sql'], '/home/ubuntu/airflow/dags/arc/phub_agg_div_progress/SQL': ['pg_c_fabric_location_divisions.sql', 'town_agg.sql', 'congressional_district_agg.sql', 'state_house_district_agg.sql', 'tribal_agg.sql', 'location_divisions.sql', 'state_senate_district_agg.sql', 'county_agg.sql'], '/home/ubuntu/airflow/dags/arc/phub_project_view/SQL': ['AR_projects.sql', 'XY_locations.sql', 'AR_locations.sql', 'VA_projects.sql', 'XY_projects.sql'], '/home/ubuntu/airflow/dags/arc/tenant/mock_subgrantee/SQL': ['locations_by_id.sql', 'remove.sql', 'pit_permutations.sql', 'verify.sql', 'sample_cais.sql', 'remove_scope.sql', 'availability.sql', 'sample_locations.sql', 'cais_by_id.sql', 'collisions.sql'], '/home/ubuntu/airflow/dags/arc/upload_onboard_templates/dbt_project/models': ['.gitkeep', '_sources.yml'], '/home/ubuntu/airflow/dags/arc/upload_onboard_templates/dbt_project/models/standard': ['users.sql', 'companies.sql', 'projects.sql'], '/home/ubuntu/airflow/dags/architecture/boss_prd2stg_keep_users/SQL': ['empty_signature_documents.sql', 'reset_sga_applications.sql'], '/home/ubuntu/airflow/dags/bbm/audit/SQL': ['create_table_c_bbm_audit_in_pg.sql', 't_bbm_audit_ffa_stats_tribal.sql', 't_bbm_audit_location_census_v3.sql', 'tile_cai_poi.sql', 't_bbm_audit_ffa_stats_tract.sql', 't_bbm_audit_ffa_stats_county.sql', 't_bbm_audit_location_census_v4.sql', 't_bbm_audit_demographics_county_place_aiannh_consity_tract.sql', 't_bbm_audit_ffa_stats_state.sql', 't_bbm_audit_fa_stats_v3.sql', 't_bbm_audit_location_tract_v3.sql', 't_bbm_audit_location_cb10_v3.sql', 't_bbm_audit_fa_stats_v4.sql', 'c_bbm_audit_city.sql', 't_bbm_audit_demographics_state_bg_cb.sql', 't_bbm_audit_location_cb_v4.sql', 'c_bbm_audit_county.sql', 'c_bbm_audit_tract.sql', 'c_bbm_audit_tribal.sql', 't_bbm_audit_cpc.sql', 't_bbm_audit_location_cb_v3.sql', 'c_bbm_audit_state.sql', 't_bbm_audit_ffa_stats_cb.sql', 'c_bbm_audit_cb.sql', 't_bbm_audit_ffa_stats_city.sql', 't_bbm_audit_cai.sql'], '/home/ubuntu/airflow/dags/bbm/audit/SQL/archive': ['t_bbm_audit_location_census_v3.sql', 't_bbm_audit_fa_stats_v3.sql', 't_bbm_audit_fa_stats.sql', 't_bbm_audit_location_cb_v3.sql'], '/home/ubuntu/airflow/dags/bbm/audit/stage_0_fabric_address_stats/SQL': ['t_bbm_audit_location_census_v3.sql', 't_bbm_audit_location_census_v4.sql', 't_bbm_audit_fa_stats_v3.sql', 't_bbm_audit_fa_stats_v4.sql', 't_bbm_audit_location_cb_v4.sql', 'c_fcc_fabric_address_v4.sql', 't_bbm_audit_location_cb_v3.sql'], '/home/ubuntu/airflow/dags/bead/bead12_map/dbt_project/models': ['schema.yml', 'c_bead12_merge.sql'], '/home/ubuntu/airflow/dags/bead/bead_award_map/SQL': ['mississippi.sql'], '/home/ubuntu/airflow/dags/bead/curate_c_national_bead/SQL': ['default.sql'], '/home/ubuntu/airflow/dags/bead/export_bead_app_meta/SQL': ['pg_question_mapping.sql', 'pg_question_no_mapping.sql'], '/home/ubuntu/airflow/dags/bead/export_bead_app_wip/SQL': ['pg_attachments.sql'], '/home/ubuntu/airflow/dags/bead/export_complete_application/dbt_project/models/export': ['c_complete_export_application_spacex.sql', 'schema.yml', 'c_complete_export_application_amazon.sql', 'c_complete_export_application_all.sql', 'c_complete_export_application_submitted_nepa.sql'], '/home/ubuntu/airflow/dags/bead/export_complete_application/dbt_project/models/standard': ['schema.yml', 'c_complete_export_application_answer.sql', 'c_complete_export_applicant_profile.sql', 'c_complete_export_application_note.sql', 'c_complete_export_application_relation.sql', 'c_complete_export_application_request_revision_fcc_fabric_location.sql', 'c_complete_export_application_location_answer.sql', 'c_complete_export_application_review_stage.sql', 'c_complete_export_application_inventory.sql', 'c_complete_export_application_request.sql', 'c_complete_export_file_metadata.sql', 'c_complete_export_applicant_profile_contact.sql', 'c_complete_export_application_node.sql', 'c_complete_export_application_resource.sql', 'c_complete_export_application_location_resource.sql', 'c_complete_export_applicant.sql', 'c_complete_export_application_review_stage_answer.sql', 'c_complete_export_application_activity.sql', 'c_complete_export_application_request_revision.sql', 'c_complete_export_application_location.sql'], '/home/ubuntu/airflow/dags/bead/export_pre_bead_bob/SQL': ['application_review_qa.sql', 'application_locations.sql', 'get_applicants.sql', 'org_contact.sql', 'application_qa.sql', 'get_applications_for_applicant.sql', 'attachment_metadata.sql'], '/home/ubuntu/airflow/dags/bead/national_bead_aggregation/SQL': ['upsert_db_table.sql', 'create_data_lake_table_county.sql', 'create_data_lake_table_state.sql', 'create_data_lake_table_congressional_district.sql', 'create_db_table.sql'], '/home/ubuntu/airflow/dags/bead/post_challenge_summary/dbt_project/models': ['.gitkeep', 'challenge_type_provider.sql'], '/home/ubuntu/airflow/dags/bead/report_export_ca/dbt_project/models': ['schema.yml', 't_bob_ca_day1_technology.sql', 't_bob_ca_day1_project_summary.sql', 't_bob_ca_day1_top_num.sql', 't_bob_ca_day1_project_summary_part_4.sql', 't_bob_ca_day1_project_summary_part_1.sql', 't_bob_ca_day1_project_summary_part_3.sql', 't_bob_ca_day1_county.sql', 't_bob_ca_day1_project_summary_part_2.sql', '.gitkeep', 't_bob_ca_day1_finance.sql', 't_bob_ca_day1_top_funding.sql', 't_bob_ca_day1_top_bsls.sql', 't_bob_ca_day1_application_answers.sql', 't_bob_ca_day1_provider_type.sql'], '/home/ubuntu/airflow/dags/bead/sar_export/dbt_project/models': ['schema.yml', 'sar_no_bead_locations_tx.sql', 'sar_served_cais_tx.sql', 'sar_deployment_projects_ar.sql', 'qa_served_locations_guard_tx.sql', 'sar_subgrantees_tx.sql', 'sar_deployment_projects_tx.sql', 'sar_subgrantees_ar.sql', 'sar_baba_tracker_tx.sql', 'sar_served_bsls_tx.sql', 'sar_subgrantees.sql'], '/home/ubuntu/airflow/dags/bead/sdpn_challenge/dbt_project/models': ['schema.yml', 'c_sdpn_merge.sql'], '/home/ubuntu/airflow/dags/bead/tx_permit_bulk_load/SQL': ['apply.sql', 'manifest.sql'], '/home/ubuntu/airflow/dags/bedrock/data_comparison_agent/cleanup/SQL': ['find_expired_tables.sql'], '/home/ubuntu/airflow/dags/bedrock/data_comparison_agent/comparison/SQL': ['create_temp_table.sql', 'drop_temp_table.sql'], '/home/ubuntu/airflow/dags/bedrock/data_comparison_agent/comparison/dbt_project/models': ['schema.yml'], '/home/ubuntu/airflow/dags/bedrock/data_comparison_agent/comparison/dbt_project/models/comparison': ['comp_row_level.sql', 'comp_duplicates.sql', 'comp_row_counts.sql', 'comp_schema_diff.sql'], '/home/ubuntu/airflow/dags/bedrock/data_comparison_agent/comparison/dbt_project/models/staging': ['stg_source.sql', 'stg_target.sql'], '/home/ubuntu/airflow/dags/bedrock/ingest_db/sql': ['AR_DATASME-1075_archie_bsls.sql', 'AR_DATASME-1075_archie_cais.sql'], '/home/ubuntu/airflow/dags/census/adjacency_matrix/curate/SQL': ['cbg_adjacency.sql', 'cb_adjacency.sql', 'ct_adjacency.sql'], '/home/ubuntu/airflow/dags/census/demographics_data_ingest/SQL': ['cbg_to_cb_interpolate.sql', 'cbg_to_cb_lookup.sql'], '/home/ubuntu/airflow/dags/cpc/point_in_time_refactor/stage1_create_schema/SQL': ['1_load_fcc_ffa_tech_map.sql', '0_create_tables.sql', '1_load_fcc_fabric_addresses.sql'], '/home/ubuntu/airflow/dags/cpc/point_in_time_refactor/stage2_load_point_in_time/SQL': ['0_ath_extract_point_in_time_bob_ok.sql', '0_ath_extract_point_in_time_demo.sql', '0_ath_extract_point_in_time.sql', '0_ath_extract_point_in_time_legacy.sql', '0_ath_extract_point_in_time_post_challenge_ca.sql', '0_ath_extract_point_in_time_post_challenge_legacy.sql', '0_ath_extract_point_in_time_arc.sql', '0_ath_extract_point_in_time_objection.sql', '0_ath_extract_point_in_time_post_challenge_tx.sql', '0_ath_extract_point_in_time_post_challenge_ok.sql', '0_ath_extract_point_in_time_bob.sql', '2_pg_load_point_in_time.sql', '1_pg_upsert_address.sql', '1_ath_extract_address_updates.sql', '0_ath_extract_point_in_time_ulfw.sql', '0_ath_extract_point_in_time_bob_ca.sql', '0_ath_extract_point_in_time_az.sql', '0_ath_extract_point_in_time_post_challenge.sql'], '/home/ubuntu/airflow/dags/cpc/point_in_time_refactor/stage2_load_providers/SQL': ['2_upsert_fcc_providers.sql', '1_extract_fcc_providers.sql', '2_upsert_fcc_tech_map.sql', '1_extract_fcc_tech_map.sql'], '/home/ubuntu/airflow/dags/cpc/point_in_time_refactor/stage2_load_providers_wip/SQL': ['1_extract_fcc_providers.sql', '1_extract_fcc_tech_map.sql'], '/home/ubuntu/airflow/dags/cpc/raw_to_transformed_state_dedup/SQL': ['state_fed_insert_into.sql', 'create_t_combined_state_and_fed_dedup_bdc.sql', 'federal_insert_into.sql'], '/home/ubuntu/airflow/dags/cpc/stage2_load_provider_tech_map/SQL': ['MS_load_ffa.sql', 'CA_load_ffa.sql', 'TX_load_ffa.sql'], '/home/ubuntu/airflow/dags/cpc/stage3_load_addresses/SQL': ['MS_load_BOSS_cpf.sql', 'DE_load_BOSS.sql', 'OH_load_BOSS.sql', 'CT_load_BOSS.sql', 'AZ_load_BOSS.sql', 'LA_load_BOSS.sql', 'CA_load_BOSS.sql', 'MS_load_BOSS.sql', 'TX_load_BOSS.sql', 'NY_load_BOSS.sql', 'AL_load_BOSS.sql', 'WV_load_BOSS.sql', 'MT_load_BOSS.sql', 'NH_load_BOSS.sql', 'OK_load_BOSS.sql', 'KS_load_BOSS.sql', 'NM_load_BOSS.sql', 'VA_load_BOSS.sql', 'DC_load_BOSS.sql', 'NJ_load_BOSS.sql', 'PA_load_BOSS.sql', 'IN_load_BOSS.sql', 'HI_load_BOSS.sql', 'OR_load_BOSS.sql', 'ND_load_BOSS.sql', 'NC_load_BOSS.sql', 'AR_load_BOSS.sql', 'MN_load_BOSS.sql', 'OK_load_BOSS_cpf.sql', 'MI_Detroit_load_BOSS.sql', 'WY_load_BOSS.sql', 'GU_load_BOSS.sql', 'NV_load_BOSS.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/challenge_cai': ['CT_challenge_cai.sql', 'DE_challenge_cai.sql', 'XZ_challenge_cai.sql', 'XZ_challenge_cai_no_trueup.sql', 'DC_challenge_cai.sql', 'MS_challenge_cai.sql', 'LA_challenge_cai.sql', 'OR_challenge_cai.sql', 'no_selection.sql', 'AR_challenge_cai.sql', 'GU_challenge_cai_no_latency.sql', 'ND_challenge_cai.sql', 'CA_challenge_cai.sql', 'OK_challenge_cai.sql', 'TX_challenge_cai.sql', 'NC_challenge_cai.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/challenge_rebuttal_summary': ['CA_challenge_rebuttal.sql', 'XZ_challenge_rebuttal.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/challenger': ['PA_challenger.sql', 'TX_challenger.sql', 'DC_challenger.sql', 'DE_challenger.sql', 'NV_challenger.sql', 'OK_challenger.sql', 'no_selection.sql', 'XZ_challenger.sql', 'NM_challenger.sql', 'MT_challenger.sql', 'ND_challenger.sql', 'MS_challenger.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/challenges': ['TX_challenges.sql', 'OR_challenges.sql', 'PA_challenges.sql', 'DC_challenges.sql', 'NY_challenges.sql', 'XZ_challenges_dsl_fw.sql', 'NV_challenges.sql', 'DE_challenges.sql', 'LA_challenges.sql', 'CA_challenges.sql', 'AR_challenges.sql', 'XZ_challenges_no_mod.sql', 'CT_challenges.sql', 'WY_challenges.sql', 'OK_challenges.sql', 'GU_challenges_no_latency.sql', 'no_selection.sql', 'IN_challenges.sql', 'XZ_challenges.sql', 'NJ_challenges.sql', 'OH_challenges.sql', 'NM_challenges.sql', 'HI_challenges.sql', 'MT_challenges.sql', 'AL_challenges.sql', 'MS_challenges.sql', 'NC_challenges.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/post_challenge_cai': ['TX_post_challenge_cai.sql', 'OK_post_challenge_cai.sql', 'XZ_post_challenge_cai.sql', 'MS_post_challenge_cai.sql', 'DC_post_challenge_cai.sql', 'LA_post_challenge_cai.sql', 'CA_post_challenge_cai.sql', 'GU_post_challenge_cai_no_latency.sql', 'PA_post_challenge_cai.sql', 'no_selection.sql', 'NC_post_challenge_cai.sql', 'NM_post_challenge_cai.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/post_challenge_locations': ['NM_post_challenges.sql', 'DE_post_challenge.sql', 'AR_post_challenges.sql', 'CT_post_challenge.sql', 'GU_post_challenge_no_latency.sql', 'XZ_post_challenges.sql', 'no_selection.sql', 'AL_post_challenge.sql', 'MS_post_challenges.sql', 'CA_post_challenge.sql', 'NV_post_challenge.sql', 'LA_post_challenge.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/post_challenge_summary': ['post_challenge_summary.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_ntia_post_challenge/SQL/post_challenge_table': ['no_selection.sql', 'XZ_post_challenge_table.sql'], '/home/ubuntu/airflow/dags/cpc/stage5_sbo_validation/SQL': ['ZX_dedup_dsl_cfw.sql', 'NH_comp.sql', 'ZX_dedup_dsl.sql', 'MS_comp.sql', 'ZX_comp_dsl.sql', 'NH_dedup.sql', 'MS_dedup.sql', 'ZX_comp_dsl_cfw.sql', 'ZX_comp.sql', 'ZX_dedup.sql'], '/home/ubuntu/airflow/dags/cpc/stage6_post_challenge_summary/SQL': ['athena_challenges_by_tenant.sql'], '/home/ubuntu/airflow/dags/cpc/stage7_post_challenge_adjustment/SQL/cai': ['XZ_post_challenge_cai.sql'], '/home/ubuntu/airflow/dags/cpc/stage8_final_proposal_export/SQL': ['project_id.sql'], '/home/ubuntu/airflow/dags/cpc/stage8_final_proposal_export/SQL/fp_cai': ['test.sql', 'OK_customized_20260413.sql', 'OK_customized_20260416.sql', 'OK_customized_20260210.sql', 'OK_customized_20260317.sql', 'TX.sql', 'template.sql', 'OK_customized_20260506.sql', 'AZ.sql', 'MN.sql', 'OK_customized_20260515.sql', 'OK_customized.sql', 'MS.sql', 'AR.sql', 'OK.sql', 'IN_customized_20260409.sql', 'OK_customized_20260326.sql', 'IN.sql'], '/home/ubuntu/airflow/dags/cpc/stage8_final_proposal_export/SQL/fp_deployment_projects': ['OK_customized_20260413.sql', 'TX_applicant.sql', 'OK_customized_20260416.sql', 'OK_customized_20260210.sql', 'OK_customized_20260317.sql', 'TX.sql', 'OK_customized_20260506.sql', 'AZ.sql', 'MN.sql', 'OK_customized_20260515.sql', 'OK_customized.sql', 'MS.sql', 'AR.sql', 'OK.sql', 'IN_customized_20260409.sql', 'OK_customized_20260326.sql', 'IN.sql'], '/home/ubuntu/airflow/dags/cpc/stage8_final_proposal_export/SQL/fp_locations': ['OK_customized_20260413.sql', 'OK_customized_20260416.sql', 'OK_customized_20260210.sql', 'OK_customized_20260317.sql', 'TX.sql', 'template.sql', 'OK_customized_20260506.sql', 'AZ.sql', 'MN.sql', 'OK_customized_20260515.sql', 'OK_customized.sql', 'MS.sql', 'AR.sql', 'OK.sql', 'IN_customized_20260409.sql', 'OK_customized_20260326.sql', 'IN.sql'], '/home/ubuntu/airflow/dags/cpc/stage8_final_proposal_export/SQL/fp_no_bead_locations': ['OK_customized_20260413.sql', 'OK_customized_20260416.sql', 'OK_customized_20260210.sql', 'OK_customized_20260317.sql', 'TX.sql', 'template.sql', 'OK_customized_20260506.sql', 'AZ.sql', 'MN.sql', 'OK_customized_20260515.sql', 'OK_customized.sql', 'MS.sql', 'AR.sql', 'IN_customized_20260409.sql', 'OK_customized_20260326.sql', 'IN.sql'], '/home/ubuntu/airflow/dags/cpc/stage8_final_proposal_export/SQL/fp_qc': ['template.sql'], '/home/ubuntu/airflow/dags/cpc/stage8_final_proposal_export/SQL/fp_subgrantees': ['OK_customized_20260413.sql', 'OK_customized_20260416.sql', 'OK_customized_20260210.sql', 'OK_customized_20260317.sql', 'TX.sql', 'template.sql', 'OK_customized_20260506.sql', 'AZ.sql', 'MN.sql', 'OK_customized_20260515.sql', 'OK_customized.sql', 'MS.sql', 'AR.sql', 'OK.sql', 'IN_customized_20260409.sql', 'OK_customized_20260326.sql', 'IN.sql'], '/home/ubuntu/airflow/dags/cpc/stage9_backfill_final_proposal/SQL/Applicant': ['pg_update_ms.sql', 'pg_update_ok.sql', 'pg_insert_ok.sql', 'load_lake_mn.sql', 'load_lake_az.sql', 'pg_insert_ms.sql', 'load_lake_ms.sql', 'pg_update.sql', 'load_lake_ok.sql', 'pg_insert_mn.sql', 'pg_insert_az.sql', 'pg_update_az.sql'], '/home/ubuntu/airflow/dags/cpc/stage9_backfill_final_proposal/SQL/ApplicantProfile': ['pg_upsert_ms.sql', 'load_lake_mn.sql', 'load_lake_az.sql', 'pg_upsert.sql', 'load_lake_ms.sql', 'load_lake.sql', 'pg_upsert_mn.sql'], '/home/ubuntu/airflow/dags/cpc/stage9_backfill_final_proposal/SQL/Application': ['test.sql', 'pg_update_ok.sql', 'load_lake_mn.sql', 'load_lake_az.sql', 'check_delta_mn.sql', 'pg_insert.sql', 'load_lake_ms.sql', 'pg_update.sql', 'load_lake_ok.sql', 'pg_insert_mn.sql', 'pg_update_ok_int.sql', 'load_lake.sql', 'pg_update_az.sql', 'pg_update_mn.sql', 'load_lake_ok_int.sql'], '/home/ubuntu/airflow/dags/cpc/stage9_backfill_final_proposal/SQL/ApplicationAnswer': ['temp.sql', 'load_lake_mn.sql', 'load_lake_az.sql', 'pg_insert.sql', 'load_lake_ms.sql', 'pg_update.sql', 'load_lake_ok.sql', 'load_lake_ok_int.sql'], '/home/ubuntu/airflow/dags/cpc/stage9_backfill_final_proposal/SQL/ApplicationLocation': ['pg_delete_ms.sql', 'pg_delete_ok.sql', 'pg_delete.sql', 'pg_update_ok.sql', 'pg_insert_ok.sql', 'load_lake_az.sql', 'check_delta.sql', 'pg_upsert.sql', 'load_lake_ms.sql', 'load_lake_ok.sql', 'load_lake.sql', 'pg_delete_az.sql', 'load_lake_ok_test.sql', 'load_lake_ok_int.sql'], '/home/ubuntu/airflow/dags/datalake/load_location_lake_table/SQL': ['DE_post_challenge_tile.sql', 'XZ_ulfw_tile.sql', 'TX_final_proposal_tile.sql', 'PA_post_challenge_tile.sql', 'NM_fabric_v4_tile.sql', 'XZ_bead_tile.sql', 'XZ_tile_dsl_mod_pre.sql', 'CA_tile.sql', 'MP_final_proposal_tile.sql', 'HI_post_challenge_adj_tile.sql', 'NJ_tile.sql', 'OR_tile.sql', 'NV_post_challenge_tile.sql', 'UT_final_proposal_tile.sql', 'MN_final_proposal_tile.sql', 'WY_tile.sql', 'IN_final_proposal_tile.sql', 'AR_fabric_v4_tile.sql', 'CT_tile.sql', 'NC_post_challenge_tile.sql', 'PA_tile.sql', 'AZ_bead_r1_tile.sql', 'OK_post_challenge_adj_tile.sql', 'XY_bead_tile.sql', 'MS_post_challenge_tile.sql', 'GU_bob_tile.sql', 'NY_fabric_v4_tile.sql', 'LA_post_challenge_tile.sql', 'AR_bead_tile.sql', 'MN_bead_tile.sql', 'NY_tile.sql', 'OK_tile.sql', 'GU_final_proposal_tile.sql', 'IN_post_challenge_adj_tile.sql', 'CA_bob_r3_tile.sql', 'MN_arc_mission_control_tile.sql', 'AR_post_challenge_adj_tile.sql', 'OR_post_challenge_adj_tile.sql', 'MS_cpf_awardedlocationspoints_tile.sql', 'OK_final_proposal_tile.sql', 'OH_post_challenge_tile.sql', 'IN_bead_tile.sql', 'MS_bob_tile.sql', 'MS_cpf_tile.sql', 'MS_cpf_awardedlocations_tile.sql', 'NM_tile.sql', 'MT_post_challenge_tile.sql', 'MS_objection_tile.sql', 'OK_bob_breakfix_tile.sql', 'LA_final_proposal_tile.sql', 'MS_final_proposal_tile.sql', 'VA_vati_tile.sql', 'CA_bob_tile.sql', 'AZ_bob_tile.sql', 'MS_ulfw_tile.sql', 'TX_bob_tile.sql', 'NE_final_proposal_tile.sql', 'CA_bead_tile.sql', 'GU_fabric_v5_tile.sql', 'ID_final_proposal_tile.sql', 'XY_tile.sql', 'GU_tile.sql', 'MS_bead_tile.sql', 'GU_post_challenge_adj_tile.sql', 'AL_scabcd_tile.sql', 'CA_ulfw_tile.sql', 'NV_tile.sql', 'AZ_final_proposal_tile.sql', 'IN_final_proposal_datasme1907_tile.sql', 'XZ_post_challenge_adj_tile.sql', 'TX_bead_tile.sql', 'AL_post_challenge_tile.sql', 'LA_tile.sql', 'CA_post_challenge_tile.sql', 'CA_fabric_v4_tile.sql', 'HI_tile.sql', 'MN_bob_tile.sql', 'HI_fabric_v4_tile.sql', 'NM_post_challenge_tile.sql', 'MS_tile.sql', 'LA_fabric_v3_tile.sql', 'TX_fabric_v4_tile.sql', 'XZ_post_challenge_tile.sql', 'AZ_bead_tile.sql', 'VA_final_proposal_tile.sql', 'IN_bob_tile.sql', 'AZ_bead_r2_tile.sql', 'ND_fabric_v4_tile.sql', 'DE_accepted_tile.sql', 'OK_bead_tile.sql', 'NC_tile.sql', 'CT_post_challenge_tile.sql', 'OK_ulfw_tile.sql', 'OK_bob_tile.sql', 'KS_final_proposal_tile.sql', 'OK_cpf_tile.sql', 'IN_tile.sql', 'KS_tile.sql', 'VA_arc_mission_control_tile.sql', 'DE_tile.sql', 'MS_fabric_v5_tile.sql', 'MS_fabric_v4_tile.sql', 'ZZ_tile.sql', 'NH_tile.sql', 'NJ_post_challenge_tile.sql', 'ND_tile.sql', 'AR_tile.sql', 'XZ_tile_no_mod_pre.sql', 'TX_tile.sql', 'XZ_final_proposal_tile.sql', 'IL_final_proposal_tile.sql', 'DC_tile.sql', 'MS_post_challenge_adj_tile.sql', 'WV_tile.sql', 'CA_post_challenge_adj_tile.sql', 'CA_final_proposal_tile.sql', 'AR_final_proposal_tile.sql', 'XZ_post_challenge_tile_no_mod.sql', 'XY_post_challenge_tile.sql', 'MI_Detroit_tile.sql', 'TX_post_challenge_adj_tile.sql', 'AL_tile.sql', 'MS_post_challenge_v5_tile.sql', 'ND_post_challenge_adj_tile.sql', 'OH_tile.sql', 'MT_tile.sql'], '/home/ubuntu/airflow/dags/digital_equity/community_resilience/ingest_data/SQL': ['t_acs_digital_equity.sql', 'c_digital_equity.sql'], '/home/ubuntu/airflow/dags/digital_equity/maternal_child_health/SQL': ['curate.sql'], '/home/ubuntu/airflow/dags/digital_equity/places_health/curate/SQL': ['county_curate.sql', 'census_tract_curate.sql'], '/home/ubuntu/airflow/dags/digital_equity/places_health/publish/SQL': ['pg_c_places_health.sql'], '/home/ubuntu/airflow/dags/funding/SQL': ['cet_parquet_r_fcc_fm_unserved_unfunded.sql', 't_funding_arpa_coords.sql', 'cet_parquet_r_fcc_fm_areaattributes.sql', 'cet_parquet_r_fcc_fm_program_funded_locations.sql', 'cet_r_funding_reconnect.sql', 't_funding_mtts_other_admin_div.sql', 't_funding_type_ref.sql', 'cet_funding_fcc_fm_versions.sql', 'c_funding_fcc_coords.sql', 't_fcc_fm_areaattributes_v.sql', 'c_fabric_location_funding.sql', 't_funding_rmlei_other_admin_div.sql', 't_fcc_fm_areapolygons_v.sql', 't_fcc_fm_programdata_v.sql', 't_fcc_fm_areabuildout_v.sql', 't_funding_cafiia_cb10.sql', 'cet_parquet_r_fcc_fm_locationattributes.sql', 't_funding_rep_other_admin_div.sql', 'cet_parquet_r_fcc_fm_programdata.sql', 't_fcc_fm_locationlist_v.sql', 't_funding_cnm_other_admin_div.sql', 't_funding_ctr_other_admin_div.sql', 'cet_parquet_r_fcc_fm_areapolygons.sql', 't_funding_slfrf_other_admin_div.sql', 'cet_t_fabric_location_funding.sql', 't_location_id_funding_str_long_cb10.sql', 't_location_id_funding_str_long_oad.sql', 't_funding_usac_coords.sql', 't_funding_jci_other_admin_div.sql', 'c_funding_to_tiles.sql', 'c_funding_fcc_other_admin_div.sql', 'c_fcc_fabric_address_from_sbo.sql', 't_fcc_fm_locationattributes_v.sql', 'cet_parquet_r_fcc_fm_locationlist.sql', 'cet_r_fcc_fm_areapolygons.sql', 't_funding_arc_other_admin_div.sql', 'cet_parquet_r_fcc_fm_areabuildout.sql', 'cet_parquet_r_fcc_fm_locationbuildout.sql', 't_fcc_fm_unserved_unfunded_v.sql', 't_funding_abaf_other_admin_div.sql', 't_funding_cm_other_admin_div.sql', 'cet_parquet_r_fcc_fm_funded_locations.sql', 't_funding_begp_other_admin_div.sql', 't_fcc_fm_program_funded_locations_v.sql', 'r_funding_type_ref.sql', 't_fcc_fm_locationbuildout_v.sql', 't_location_id_funding_str_long_coords.sql'], '/home/ubuntu/airflow/dags/funding/ingest_ffm_state/SQL': ['transform.sql'], '/home/ubuntu/airflow/dags/funding_cpc/default/SQL': ['create_default_or.sql', 'create_default_nc.sql', 'create_default_al.sql', 'create_default_ca.sql', 'create_default_in.sql', 'create_default_template.sql', 'create_default_hi.sql', 'create_default_la.sql', 'create_default_nh.sql', 'create_default_ms.sql', 'create_default_ar.sql', 'create_default_ny.sql', 'create_default_oh.sql', 'create_default_wy.sql', 'create_default_nm.sql'], '/home/ubuntu/airflow/dags/funding_cpc/refactor/SQL': ['NM_bead_pre.sql', 'MS_bead_pre.sql', 'OH_bead_pre.sql', 'WY_bead_post.sql', 'NC_bead_post.sql', 'CA_bead_pre.sql', 'AR_bead_post.sql', 'HI_bead_pre.sql', 'OR_bead_post.sql', 'NY_bead_pre.sql', 'HI_bead_post.sql', 'IN_bead_post.sql', 'TEMPLATE_fed_sql_header_awardee_only.sql', 'NJ_bead_post.sql', 'TEMPLATE_fed_sql_header.sql', 'NM_bead_post.sql', 'OK_cpf_pre.sql', 'MS_cpf_pre.sql', 'MN_arc_pre.sql', 'NH_bead_post.sql', 'AR_bead_pre.sql', 'OK_bead_post.sql', 'AL_bead_post.sql', 'CA_arc_pre.sql', 'NY_bead_post.sql', 'TEMPLATE_funding_awardee.sql', 'MS_bead_post.sql', 'CA_bead_post.sql', 'TX_bead_pre_v4.sql', 'OK_arc_pre.sql', 'TX_bead_post.sql', 'ND_bead_pre.sql', 'NC_bead_pre.sql', 'AL_bead_pre.sql', 'MS_arc_pre.sql', 'MS_bead_post_v5.sql', 'AZ_bead_post.sql', 'OH_bead_post.sql', 'OK_bead_pre.sql', 'NH_bead_pre.sql', 'CA_ulfw_pre.sql', 'TEMPLATE_funding.sql', 'TX_bead_pre.sql', 'IN_bead_pre.sql', 'WY_bead_pre.sql', 'VA_arc_pre.sql', 'MS_ulfw_pre.sql'], '/home/ubuntu/airflow/dags/funding_cpc/update/general_datalake/SQL': ['merge_ready_dedup.sql', 'create_state_and_fed_dedup_nm_v3.sql', 'create_state_and_fed_dedup_al_v4.sql', 'create_state_and_fed_dedup_ny_v3.sql', 'create_state_and_fed_dedup_nc_v4.sql', 'delete_staging.sql', 'create_state_and_fed_dedup_ny_v4.sql', 'create_state_and_fed_dedup_ms_v4.sql', 'create_state_and_fed_dedup_nm_v4.sql', 'find_provider_id.sql', 'create_state_and_fed_dedup_de_v3.sql', 'insert_final.sql', 'create_state_and_fed_dedup_in_v3.sql', 'update_bidder_ny.sql', 'create_state_and_fed_dedup_std.sql', 'create_state_and_fed_dedup_ok_v4.sql', 'create_state_and_fed_dedup_std_v4.sql', 'create_state_and_fed_dedup_ct_v3.sql', 'create_state_and_fed_dedup_pa_v4.sql', 'create_ready_funding.sql', 'create_state_and_fed_dedup_ar_v4.sql', 'template_fed_sql_header.sql', 'create_state_and_fed_dedup_la_v3.sql', 'create_state_and_fed_dedup_std_awardee.sql', 'create_state_dedup_de.sql', 'create_state_and_fed_dedup_ok_v3.sql', 'create_state_and_fed_dedup_ar_v3.sql', 'create_fed_dedup_v3.sql', 'create_state_and_fed_dedup_nv_v3.sql', 'map_provider_id.sql', 'create_state_and_fed_dedup_nv_v2.sql', 'create_state_and_fed_dedup_oh_v3.sql', 'create_state_and_fed_dedup_hi_v3.sql', 'create_state_and_fed_dedup_mt_v3.sql', 'create_state_dedup.sql', 'create_state_and_fed_dedup_nh_v4.sql', 'create_state_and_fed_dedup_wy_v3.sql', 'create_state_and_fed_dedup_hi_v4.sql', 'create_state_and_fed_dedup_ca_v4.sql'], '/home/ubuntu/airflow/dags/funding_cpc/update/la/gumbo_ingest_v3/SQL': ['merge_gumbo.sql'], '/home/ubuntu/airflow/dags/funding_cpc/update/ms/SQL': ['update_location_id.sql', 'bip_update_geoid.sql', 'melt_cpf_table.sql', 'bip_union_three.sql', 'create_stage_table.sql', 'bip_update_franklin_loc_id.sql'], '/home/ubuntu/airflow/dags/funding_cpc/update/nh/SQL': ['merge_state.sql'], '/home/ubuntu/airflow/dags/game/ingest_data/SQL/game_mississippi': ['geotel_lata.sql', 'geotel_rate_centers.sql', 'geotel_data_centers.sql', 'geotel_bdc_fiber_to_premise_fcc.sql', 'geotel_bdc_wireless_5g_fcc.sql', 'public_safety_answering_point_broadband_services.sql', 'geotel_central_offices.sql', 'bsl_locations.sql', 'geotel_cable_operating_areas.sql', 'geotel_fiber_lit_buildings.sql', 'geotel_cell_towers.sql', 'counties.sql', 'cdc_health.sql', 'geotel_fiber_routes.sql', 'geotel_area_codes.sql', 'geotel_wire_centers.sql'], '/home/ubuntu/airflow/dags/game/ingest_data/SQL/shared': ['dams.sql', 'electric_utility_territories.sql', 'waterways.sql', 'submarine_cables_coastal.sql', 'fcc_towers.sql', 'roads.sql', 'schools.sql', 'indian_lands.sql', 'health_facilities.sql', 'electric_power_plants.sql', 'railways.sql', 'census_water_areas.sql', 'natural_gas_facilities.sql', 'public_libraries.sql', 'county_demographics.sql', 'electric_grid_equipment.sql', 'submarine_cables_river.sql', 'oil_gas_pipelines.sql', 'counties.sql', 'data_centers.sql', 'oil_gas_wells.sql', 'electric_grid_regions.sql', 'electric_power_lines.sql'], '/home/ubuntu/airflow/dags/general/airflow_config_update/table_reference/SQL': ['DL_CONTROL.sql', 'DL_POST_BEAD.sql', 'PG_CONTROL.sql', 'PG_TENANTS.sql', 'DL_PRE_BEAD.sql'], '/home/ubuntu/airflow/dags/general/aws_breakfixer/SQL': ['ALL_SAMPLE_show_create_table.sql'], '/home/ubuntu/airflow/dags/general/bbm_export/SQL': ['bsl_dl.sql', 'cai_dl.sql', 'towers.sql', 'fabric_service_v3.sql', 'speed_test.sql', 'cai.sql', 'bsl.sql', 'fabric_service.sql', 'lsd_cb20.sql', 'poi.sql', 'lsd.sql', 'parcel.sql'], '/home/ubuntu/airflow/dags/general/canopy_cover_analysis/gap_analysis/SQL': ['gap_analysis_export.sql'], '/home/ubuntu/airflow/dags/general/ctasifier/json': ['bi_bead_project_plan_detail.json', 'ca_bi_application_answer.json', 'ca_bi_application_answer_wide.json', 'bi_bead_location_detail.json', 'ar_bi_application_cai.json', 'ca_bi_application_cai.json', 'VA_DATASME-1943_cai_corrected_tile_projection.json', 'bi_application_answer_wide.json', 'ar_bi_application_location.json', 'ar_bi_application_answer.json', 'bi_application_question_answer_mapping.json', 'ar_bi_application_question_answer_mapping.json', 'demo_bi_deployment_projects.json', 'ca_bi_applicant_contact.json', 'bi_bead_provider_sga.json', 'az_bi_application.json', 'az_bi_application_cai.json', 'bi_bead_award_overview.json', 'ca_bi_application_location.json', 'mn_bi_application_question_answer_mapping.json', 'demo_bi_application_location.json', 'bi_application_cai.json', 'VA_DATASME-1895_rp_financial_subgrant_agreements_backup.json', 'demo_bi_application_answer_wide.json', 'ApplicationLocation_backup_082626.json', 'mn_bi_application_answer_wide.json', 'ar_bi_application.json', 'VA_DATASME-1895_rp_financial_reporting_projects_backup.json', 'bi_bead_project_plan.json', 'az_bi_application_location.json', 'bi_bead_project_review_stage.json', 'bi_user_applicant_access.json', 'bi_applicant_contact.json', 'ca_cpc_funding_arc_pre_20260217.json', 'bi_bead_compliant_report_detail.json', 'bi_bead_project_permit.json', 'bi_bead_compliant_report.json', 'bi_bead_project_overview.json', 'VA_DATASME-1895_rp_financial_reporting_project_answers_backup.json', 'bi_bead_project_milestone_1.json', 'demo_bi_application_answer.json', 'demo_bi_application_cai.json', 'az_bi_application_answer.json', 'bi_application_location.json', 'bi_bead_project_activity.json', 'az_bi_application_answer_wide.json', 'VA_DATASME-1943_non_cai_hash_control.json', 'az_bi_application_question_answer_mapping.json', 'mn_bi_application_answer.json', 'demo_bi_application.json', 'demo_bi_application_question_answer_mapping.json', 'mn_bi_applicant_contact.json', 'bi_application_answer.json', 'ca_bi_application.json', 'mn_bi_application_cai.json', 'tx_economic_regions.json', 'demo_bi_applicant_contact.json', 'mn_bi_application_location.json', 'ar_bi_application_answer_wide.json', 'c_bob_merge_tile_tx_backup_082626.json', 'ca_bi_application_question_answer_mapping.json', 'bi_bead_project_budget.json', 'ar_bi_applicant_contact.json', 'mn_bi_application.json', 'bi_application.json', 'VA_DATASME-1895_rp_financial_subgrant_agreement_answers_backup.json'], '/home/ubuntu/airflow/dags/general/ctasifier/sql': ['VA_DATASME-1895_rp_financial_reporting_project_answers_backup.sql', 'ar_bi_applicant_contact.sql', 'bi_application_answer_wide.sql', 'VA_DATASME-1895_rp_financial_subgrant_agreements_backup.sql', 'c_bob_merge_tile_tx_backup_082626.sql', 'ca_bi_application_answer.sql', 'mn_bi_application_cai.sql', 'bi_bead_project_activity.sql', 'VA_DATASME-1943_non_cai_hash_control.sql', 'bi_bead_compliant_report_detail.sql', 'bi_bead_project_review_stage.sql', 'ar_bi_application_answer_wide.sql', 'VA_DATASME-1895_rp_financial_reporting_projects_backup.sql', 'demo_bi_application_answer_wide.sql', 'bi_application_question_answer_mapping.sql', 'demo_bi_applicant_contact.sql', 'bi_bead_project_plan_detail.sql', 'VA_DATASME-1895_rp_financial_subgrant_agreement_answers_backup.sql', 'bi_application_answer.sql', 'ca_bi_deployment_projects.sql', 'ApplicationLocation_backup_082626.sql', 'ca_bi_application_answer_wide.sql', 'bi_application_location.sql', 'ar_bi_application_location.sql', 'ar_bi_deployment_projects.sql', 'demo_bi_application_question_answer_mapping.sql', 'bi_application_cai.sql', 'bi_deployment_projects.sql', 'demo_bi_application_cai.sql', 'bi_bead_project_plan.sql', 'mn_bi_deployment_projects.sql', 'az_bi_application_question_answer_mapping.sql', 'ar_bi_application_question_answer_mapping.sql', 'demo_bi_application_location.sql', 'ca_bi_application_question_answer_mapping.sql', 'bi_bead_project_permit.sql', 'az_bi_application_answer_wide.sql', 'ca_cpc_funding_arc_pre_20260217.sql', 'ar_bi_application_cai.sql', 'VA_DATASME-1943_cai_corrected_tile_projection.sql', 'bi_applicant_contact.sql', 'mn_bi_application_question_answer_mapping.sql', 'tx_economic_regions.sql', 'mn_bi_applicant_contact.sql', 'bi_bead_provider_sga.sql', 'mn_bi_application_answer_wide.sql', 'az_bi_application_answer.sql', 'mn_bi_application_location.sql', 'ca_bi_application_cai.sql', 'bi_bead_project_milestone_1.sql', 'ar_bi_application_answer.sql', 'ca_bi_applicant_contact.sql', 'bi_bead_project_overview.sql', 'demo_bi_application_answer.sql', 'mn_bi_application_answer.sql', 'bi_bead_project_budget.sql', 'bi_user_applicant_access.sql', 'az_bi_application_cai.sql', 'bi_bead_location_detail.sql', 'az_bi_deployment_projects.sql', 'bi_bead_award_overview.sql', 'demo_bi_deployment_projects.sql', 'ca_bi_application_location.sql', 'bi_bead_compliant_report.sql', 'az_bi_application_location.sql'], '/home/ubuntu/airflow/dags/general/data_summary/SQL': ['fabric_change_state.sql', 'fabric_change_nation.sql'], '/home/ubuntu/airflow/dags/general/data_summary/bead_allocation/SQL': ['select_unserved_loc.sql', 'merge_staging.sql', 'summarize_unserved_archive.sql', 'unserved_unfunded_pg_state.sql', 'unserved_unfunded.sql', 'unserved_unfunded_pg.sql'], '/home/ubuntu/airflow/dags/general/data_summary/bead_vote/SQL': ['bead_eligible_percentage.sql', 'bead_eligible.sql'], '/home/ubuntu/airflow/dags/general/data_summary/isp_cheat/SQL': ['unserved_change-nationwide.sql', 'unserved_change-state.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter/SQL': ['AR_CUST-4133-cai-outliers.sql', 'AR_DATASME-1258.sql', 'VA_DATASME_1408.sql', 'CA_DATASME-849_application_summary.sql', 'MS_CUST-2399_CPF_ffa_04122023.sql', 'CA_DATASME-849_project_summary_part1.sql', 'NC_DATASME-738_public_bsl_challenges.sql', 'PA_CUST-847.sql', 'TX_CUST-2990_Unserved.sql', 'MS_DATASME-1023.sql', 'LA_DATASME_1155_22085.sql', 'LA_DATASME_1155_22091.sql', 'AR_DATASME-1099_bsl_paf_mapping.sql', 'IN_services_status_v4.sql', 'LA_CUST-2439_gumbo2_bsls_spa_id.sql', 'CA_CUST_2255_E.sql', 'MS_CUST-2399_CPF_application_eligible_location_fabric_v3_ms.sql', 'AZ_DATASME-990-exactmap.sql', 'CA_DATASEME_1279.sql', 'LA_CUST-3438.sql', 'MS_DATASME-1234.sql', 'MS_DATASME-1125-p2.sql', 'LA_DATASME_1155_22053.sql', 'LA_DATASME_1155_22047.sql', 'OK_DATASME-1138.sql', 'MS-CUST-110.sql', 'LA_CUST-2110_fp_locations_archive.sql', 'NY_post_challenge_summary.sql', 'LA_DATASME_1155_22127.sql', 'OK_DATASME-1064.sql', 'OK_DATASME-595-latlon-add-google.sql', 'LA_CUST-2447_pending_alt_tech.sql', 'OK_CUST-3597_changelog.sql', 'NY_CUST-2127.sql', 'LA_ARC_83.sql', 'IN_CPC-430.sql', 'XZ_BEAD_VERIFICATION.sql', 'OK_CUST-1214.sql', 'OH-CUST-165-Fiber.sql', 'OK_DATASME-595.sql', 'NC_CUST-817_served_funded.sql', 'AR_CUST-2984.sql', 'CA_DATASME-849_project_summary_part2.sql', 'LA_DATASME_1155_22119.sql', 'TX_DATASME-900.sql', 'TX_DATASME-971-delta.sql', 'LA_DATASME_1155_22125.sql', 'TX_DATASME_fp_location_temp.sql', 'TX_DATASME-2.sql', 'MN_DATASME-1053_overlaps_round_2.sql', 'TX_DATASME-739-update-pre-challenge-cai.sql', 'IN_DATASME-1060_pg_application_count_by_provider.sql', 'LA_DATASME_1155_22079.sql', 'LA_DATASME_1155_22051.sql', 'PA_CUST-918.sql', 'TX_DATASME-739-no-challenges.sql', 'LA_DATASME_1155_22045.sql', 'MS_DATASME-1006_fabric_fixed_availability_v59.sql', 'MS_CUST_1454.sql', 'PA_county-summary.sql', 'TX_DATASME_842_missing_bsl.sql', 'OK_CUST-3563.sql', 'CA_DATASME-999_not_applied_STAGE.sql', 'NC_DATASME-738_public_cai_challenges.sql', 'LA_CUST-1280_building_type.sql', 'OK_DATASME-991_eligible_locations.sql', 'TX_DATASME-1328_location.sql', 'MS_DATASME-1125-p1.sql', 'LA_CUST-2533.sql', 'LA_DATASME_1155_22087.sql', 'LA_DATASME_1155_22093.sql', 'MS_CUST-1859_CPFProject_pg.sql', 'AR_CUST_707.sql', 'OK_CUST-106.sql', 'LA_CUST-2110_fp_deployment_projects.sql', 'OK_DATASME-802-unserved.sql', 'OK_DATASME-999.sql', 'CA_DATASME-849_project_summary_part3.sql', 'CA_DATASME-1186_pg.sql', 'AZ_CUST-3479b.sql', 'MS_CUST-2399_cpf_prelim_awarded_locations.sql', 'TX_DATASME_842_remaining_cais.sql', 'CA-update_cbg.sql', 'AZ_DATASME-990.sql', 'OR_post_challenge_classification_change.sql', 'NC_DATASME-738_public_challengers.sql', 'LA_DATASME-1113.sql', 'IN_DATASME-814_cais.sql', 'CA_DATASME-1209_pg.sql', 'CA-DSL-Cellular.sql', 'OH_CUST-1228.sql', 'CA-CUST-267-underserved.sql', 'AZ_DATASME-1135.sql', 'OK_DATASME-915-ulfw-v6.sql', 'LA_CUST-1280_land_use_code.sql', 'CA_DATASME-849_top_applicants_by_funding.sql', 'LA_DATASME_1155_22097.sql', 'LA_DATASME_1155_22083.sql', 'TX_CUST-2656.sql', 'GU_DATASME-736_Case1.sql', 'MN_DATASME-817_overlaps.sql', 'IN_services_status_v3.sql', 'LA_DATASME_1155_22069.sql', 'ND-CUST-1913.sql', 'AR_DATASME-973-replicate-bcg.sql', 'IN_DATASME-1062.sql', 'TX_DATASME-739-cai-location_id.sql', 'LA_DATASME_1155_22055.sql', 'MS_DATASME-957_indicated_rate.sql', 'XZ-post-challenge_summary.sql', 'OK_DATASME_1094_cai.sql', 'TX_APPSME-1958.sql', 'AR_CUST_1628.sql', 'LA_DATASME_1155_22109.sql', 'AR_DATASME-820.sql', 'MS_ARC_799_all.sql', 'CA_DATASME-999_excluded_STAGE.sql', 'AZ_DATASME-1134.sql', 'LA_DATASME_1155_22121.sql', 'MS_CUST-4021_indicated.sql', 'CA_DATASME-1018.sql', 'OK_DATASME_1094_bsl.sql', 'LA-CUST-2137-NOT-GUMBO.sql', 'OK_DATASME-915-ulfw.sql', 'CA-DSL-Cellular-CAI-dedup-casf.sql', 'MS_fp_no_bead_locations_09_2_export.sql', 'XZ_fabric_change.sql', 'GU_DATASME-943_v6removed.sql', 'CA_DATASME-1141_pg.sql', 'NY-CPC-1311.sql', 'OH-CUST-165-Cable.sql', 'PA_post_challenge_summary_challenge_types.sql', 'MS_DATASME-914_indicated.sql', 'LA_DATASME_1155_22123.sql', 'IN_DATASME-929.sql', 'OK_CUST-2519.sql', 'CA_DATASME-999_excluded.sql', 'OR_dsl_mod_classification.sql', 'CA_CUST_2477_full_cai.sql', 'OH-CUST-173-ORBEG.sql', 'TX_DATASME-899.sql', 'AR_DATASME-1103.sql', 'ND_CPC-532.sql', 'LA_DATASME_1155_22043.sql', 'TX_CUST-2641.sql', 'LA_DATASME_1155_22057.sql', 'TX_DATASME-668-latency-check.sql', 'CA-CUST-267-cai.sql', 'OK_DATASME_FP_1455_cai_evaluation.sql', 'OK_DATASME-802-underserved.sql', 'TX_DATASME-918_with_speedranks.sql', 'OR_dsl_mod_impact.sql', 'CA_DATASME-860_application_locations.sql', 'IN_CPC-470_fce0.sql', 'GU_DATASME-736_Case2.sql', 'IN_DATASME-814_locations.sql', 'MS_CUST-1374.sql', 'AR_DATASME-1116.sql', 'LA_DATASME_1155_22095.sql', 'NY-CUST-1995.sql', 'LA_DATASME_1155_22081.sql', 'MS_DATASME-956_cai.sql', 'LA_CUST-2502_location_type_breakdown.sql', 'OH-CUST-171-DSL-Cellular-Busi-CAI.sql', 'ZX_TEST_001.sql', 'OK_CUST-2915.sql', 'AR_DATASME_1050.sql', 'MN_DATASME-1218_recent_updates.sql', 'CA_DATASME-1054.sql', 'MS_DATASME-1108.sql', 'TX_DATASME-867-bsl.sql', 'OH_CUST-1122-c5.sql', 'AR_post_challenge_validation.sql', 'IN_DATASME-1172.sql', 'LA_DATASME_1155_22019.sql', 'LA_DATASME_1155_22025.sql', 'LA_DATASME_1155_22031.sql', 'CA_DATASME-999_not_applied.sql', 'TX_DATASME-867-cai.sql', 'TX_DATASME-1156.sql', 'IN_CUST-3299-coverage.sql', 'TX_DATASME-739-no-deduplication.sql', 'MS_CUST-1845.sql', 'LA_DATASME_888_CD6.sql', 'AZ_DATASME-880.sql', 'VA_DATASME_1254_r_vaup_fp_reporting_projects.sql', 'TX_CUST-2632.sql', 'MS_DATASME-996_all.sql', 'CA_DATASME-849_applicant_type_report.sql', 'OK_CUST-4130-loc-cai-overlap.sql', 'OH_CUST-1122-c4.sql', 'NH_post_challenge_summary.sql', 'NM_bead_post_challenge_verification.sql', 'MS-CUST-1293-survey.sql', 'MS_DATASME-996_part_of_applicaton_true.sql', 'ND_post_challenge_summary.sql', 'OH_CUST-2221.sql', 'OH_CUST_2706.sql', 'CA_DATASME-849_county_report.sql', 'CA_CUST_2477_full_challenges.sql', 'TX_CUST-4096_1_general.sql', 'PA_post_challenge_summary_funding.sql', 'IN_DATASME-1060_pg_full_impacted_locations.sql', 'MS_CUST-2399_CPF_application_eligible_location_fabric_v5_ms.sql', 'LA_DATASME_1155_franklin.sql', 'GU_CUST-2648.sql', 'MS_DATASME-813.sql', 'CA_DATASME_1162.sql', 'LA_DATASME_888_CD4.sql', 'CA_DATASME-849_provider_type.sql', 'CA_DATASME-818_pg.sql', 'LA_DATASME_888_CD5.sql', 'LA_DATASME_1155_22027.sql', 'LA_DATASME_1155_22033.sql', 'LA_DATASME_1155_tammany.sql', 'MN_DATASME-1058_organization_details_report.sql', 'TX_CUST-4096_2_time_applicant.sql', 'MS_CUST-2036.sql', 'TX_DATASME-1236_pg.sql', 'IN_CUST-3299-provider.sql', 'CA_DATASME-999_not_applied_no_prelim.sql', 'IN-CUST-2997.sql', 'CA-CUST-267-served.sql', 'AR_DATASME-1211.sql', 'CA_DATASME-1095.sql', 'TX_DATASME-1024_application_locations_bsl.sql', 'MS_DATASME-1324.sql', 'ND_CUST-1197.sql', 'AR_DATA_1654.sql', 'OR_post_challenge_summary.sql', 'OK_CUST-1624_indian_electric_coop.sql', 'CA_DATASME-849_top_applicants_by_num.sql', 'CA_CUST-3105_served.sql', 'CA_CUST_2477.sql', 'TX_CUST-2990_Underserved.sql', 'LA_DATASME_1155_tensas.sql', 'AR_DATASME-933-export2.sql', 'LA_DATASME_1155_22037.sql', 'MS_CUST-1859_CPF1BroadbandinfrasturctureLocationData_pg.sql', 'LA_DATASME_1155_22023.sql', 'TX_DATASME-1024_application_locations_cai.sql', 'LA_DATASME_888_CD1.sql', 'AR_PLAT-65.sql', 'TX_CUST-4096_3_time_dist_applicant.sql', 'AZ_DATASME-893.sql', 'OK_bead_post_challenge_verification.sql', 'OH-CUST_postchallenge-qa.sql', 'TX_DATASME-1150.sql', 'IN_CUST_2319_defaul_mercury_fiber.sql', 'MN_DATASME-1058_deconfliction_report.sql', 'AR_DATASME_1130_pg.sql', 'NM_CPC-544.sql', 'MS_CUST-3004-removed-bip-from-dedup.sql', 'MS_unserved_underserved_cpf.sql', 'CA_CUST-1404_pg_10a.sql', 'CA_DATASME-999_applied_STAGE.sql', 'OH_CPC-540-unserved.sql', 'IN_APPSME-1900.sql', 'TX_DATASME-971-ppa-summary.sql', 'CA_CUST-3105_underserved.sql', 'MS_DATASME-351_TreasuryTemplate_pg.sql', 'OK_DATASME-1015.sql', 'MS_DATASME-1127.sql', 'TX_CUST-4096_4_application_submission.sql', 'CA-DSL-Cellular-CAI.sql', 'TX_DATASME-668-post_challenge_validation.sql', 'AR_DATASME-789_applicantXlocation_pg.sql', 'LA_CUST-3501.sql', 'AR_DATA-1628.sql', 'NY-CPC-541.sql', 'OK_DATASME_FP_1472_application_to_add.sql', 'AZ_DATASME-990b.sql', 'MS_DATASME-1131.sql', 'TX_CUST-2641-b.sql', 'TX_DATASME-971.sql', 'TX_DATASME-1231.sql', 'AR_DATASME-933-export1.sql', 'CA_CUST_2354_casf.sql', 'LA_DATASME_888_CD2.sql', 'MS_DATASME-1291.sql', 'TX_DATASME-1152.sql', 'OK_DATASME-595-latlon.sql', 'TX_DATASME-1177_application_locations_bsl_split.sql', 'MS_DATASME-1006_rate_review.sql', 'LA_DATASME_888_CD3.sql', 'LA_DATASME_1155_22009.sql', 'CA_DATASME_1075_application_attachment_question_mapping.sql', 'CA_CUST-1404_pg_10b.sql', 'LA_DATASME_1155_22021.sql', 'MS_DATASME-854-cai.sql', 'TX_CUST-3664_awardee_name_match.sql', 'OH_CPC-540-served.sql', 'LA_CPC-406.sql', 'MN_DATASME-817_scores.sql', 'PA_post_challenge_summary.sql', 'TX_DATASME-757.sql', 'TX_DATASME-899b.sql', 'WY_post_challenge_summary.sql', 'CA_DATASME-999_excluded_no_prelim.sql', 'AR_DATASME-1203.sql', 'CA-CUST-267-funded.sql', 'CA_DATASME-849_equity_report.sql', 'MS-CUST-3441.sql', 'ZX_CUST-3484b.sql', 'PA_post_challenge_cai.sql', 'MN_DATASME-1058_final_proposal_report.sql', 'IN_DATASME-1147.sql', 'OK_DATASME-1153.sql', 'LA_CUST-2110_fp_subgrantees.sql', 'CA-CUST-267-unserved.sql', 'LA_ARC-818_applicants.sql', 'MS_DATASME-956_underserved.sql', 'AR_DATASME-1122_pg.sql', 'CA_DATASME-869_application_pau.sql', 'AR_DATASME-931_adjusted_v6ffa_served.sql', 'OK_DATASME-746-challenges-ffa31.sql', 'MN_DATASME_1055_application_attachment_question_mapping.sql', 'CA_DATASME-849_finance_report.sql', 'TX_DATASME-1177_application_locations_cai.sql', 'MS_CUST-2399_BCI_objection.sql', 'MS_CUST-1708-challenge-locations.sql', 'LA_DATASME_1155_22011.sql', 'LA_DATASME_1155_22005.sql', 'LA_DATASME_1155_22039.sql', 'OK_DATASME-991_cais_with_net_id.sql', 'NC_CUST-2872.sql', 'AR_APP_1817_2.sql', 'OK_DATA-1573-application_ranking.sql', 'TX_DATASME-1177_application_locations_bsl.sql', 'TX_DATASME-767.sql', 'OH_CUST-2770.sql', 'OH-CUST-167-DSL-mod.sql', 'TX_CUST-2595_pre_challenge_comparison.sql', 'LA_ARC_127.sql', 'CA_DATASME-1269_pg.sql', 'WY_CUST-2084.sql', 'LA_CUST-2110_fp_locations.sql', 'MS_locations_for_david.sql', 'OR_pre_challenge_post_challenge_update.sql', 'IN_DATASME-1144.sql', 'MS_DATASME-854-funded-served.sql', 'CA_CUST-1404_pg_7.sql', 'LA_DATASME_1155_22013.sql', 'IN_CPC-470_bc0d.sql', 'TX_CUST-2611.sql', 'LA_DATASME_1155_22007.sql', 'MS_DATASME-1274.sql', 'GU_DATASME-1067.sql', 'MS_CUST-4021_served.sql', 'OK_DATASME-746-challenges-ffa33.sql', 'OK_CUST-1624_dobson_fiber_rogers.sql', 'MN_DATASME-817_locations.sql', 'CA_CUST-3105_unserved.sql', 'MS_CUST-1708-applicant.sql', 'OK_DATASME-991_eligible_bsl_with_net_id.sql', 'AR_DATASME-931_v6removed.sql', 'OH-CUST-169-DSL-Cellular-mod.sql', 'AZ_CUST-3479.sql', 'AR_DATASME_1339.sql', 'CA_CUST-1404_pg_6.sql', 'AR_APP_1817_1.sql', 'XZ_CUST_3295.sql', 'AR_CUST-2494-cai-challenge.sql', 'OH_CPC-540-cai.sql', 'GU_DATASME-774.sql', 'MS_DATASME-956_served.sql', 'LA_DATASME-1155.sql', 'CA_DATASME-922-v6.sql', 'MS_DATASME-1315.sql', 'CA_CUST-1404_1.sql', 'AZ_CUST-3079.sql', 'CA_post_cai_validation.sql', 'IN_post_challenge_summary.sql', 'TX_CUST-2861_p3.sql', 'GU_CUST-2985_part3.sql', 'OK_CUST-2996.sql', 'OK_DATASME_1438_cost_per_net.sql', 'CA_CUST_1797_postchallenge_E.sql', 'OR_dsl_mod_typeA_ffa.sql', 'AZ_CUST-3509.sql', 'CA_DATASME-999_applied.sql', 'TX_DATASME-739-update-pre-challenge-cai-formatted.sql', 'IN_DATASME-954.sql', 'AR_DATASME-931_aristotle.sql', 'MS_DATASME-837.sql', 'CA_CUST-1404_pg_2.sql', 'AR_DATA_1701.sql', 'MS_DATASME-1136_unserved.sql', 'OR_CUST-1878_pre_challenge_dsl_mod.sql', 'AR_CUST_1694.sql', 'IN_CUST-3276.sql', 'MS_fp_no_bead_locations_09_22.sql', 'TX_DATASME-833.sql', 'AR_DATASME-1181.sql', 'CA_DATASME-849_top_applicants_by_bsls.sql', 'MS-CUST-1330.sql', 'MS_DATASME-854-unserved.sql', 'OK_DATASME_1438_other_locs_add.sql', 'TX_DATASME-971-bsl.sql', 'AZ_DATASME-1027_application_export.sql', 'AR_CUST_783.sql', 'LA_DATASME_1155_22003.sql', 'LA_DATASME_1155_22017.sql', 'TX_DATASME-1084_pg.sql', 'LA_CUST_3373.sql', 'LA_CUST-2439_gumbo2_bsls_spa_id_post_pit_check.sql', 'CA_CUST-1404_pg_3.sql', 'TX_DATASME-971-cai.sql', 'LA_DATASME-1352.sql', 'OH-CUST-173-DSL-Cellular-Busi-CAI-dedup-orbeg.sql', 'CA-DSL-Cellular-CAI-dedup.sql', 'CA_DATASME-922.sql', 'MS_CUST-3161-bip-funded-kmz.sql', 'MN_DATASME-825.sql', 'AR_DATASME-1109_comp.sql', 'GU_CUST-2985_part2.sql', 'TX_CUST-2861_p2.sql', 'AR_CUST-2494-location-ls.sql', 'MS_DATASME-956_indicated.sql', 'VA_DATASME_1254_r_vaup_fp_rp_locations.sql', 'CA-Explain-Delta.sql', 'OH_CUST-2606-cai.sql', 'OK_DATASME-1196_fp_cai.sql', 'OK_CUST-1734.sql', 'LA_CUST-2110_fp_cai.sql', 'IN_DATASME-822-no-application.sql', 'LA_DATASME_1155_22001.sql', 'LA_DATASME_1155_22015.sql', 'AR_DATASME-1182.sql', 'ZX_CUST-3484.sql', 'LA_DATASME_1155_22029.sql', 'CA_APPSME-1571.sql', 'AZ_DATASME-1057_pg.sql', 'TX_DATASME-1199.sql', 'MS_CUST-4021_funded.sql', 'TX_DATASME-971-pau.sql', 'AR_DATASME-1394.sql', 'VA_DATASME_1254_r_vaup_fp_rp_cai.sql', 'MN_DATASME-1058_application_export.sql', 'TX_DATASME-781_idtofile.sql', 'LA_CUST-2110_fp_cai_all_location_ids.sql', 'IN_CPC-501.sql', 'AR_DATASME-1099_v6_removed.sql', 'IN_DATASME-822.sql', 'AR_CUST_1696.sql', 'DATA_CHALLENGE_50.sql', 'TX_DATASME-1106_pg.sql', 'CA_DATASME-999_applied_no_prelim.sql', 'TX_APPSME-1931.sql', 'TX_DATASME-775-CAI.sql', 'CA-CAI-verification.sql', 'MS_DATASME-914_removed.sql', 'MS_DATASME_1017_application_attachment_question_mapping.sql', 'OH_CUST-2946.sql', 'CA-DSL.sql', 'LA-CUST-2137-GUMBO-only.sql', 'MS_DATASME-956_unserved.sql', 'MS_DATASME-1105.sql', 'AZ_CUST-3523.sql', 'AR_DATASME-1103-mismatches.sql', 'MS_DATASME-956_funded.sql', 'AR_DATASME-1008.sql', 'TX_DATASME-1024_application_locations_bsl_split.sql', 'TX_DATASME-1328_stats.sql', 'GU_CUST-2985_part1.sql', 'TX_DATASME-775-BSL.sql', 'AZ_CUST-3047.sql', 'OK_CUST-1624_dobson_fiber_osage.sql', 'TX_CUST-2861_p1.sql', 'TX_DATASME-1011_application_calculated.sql', 'CA_CUST-2477-CWF_MOD_VERIZON.sql', 'OK_DATASME-746-post-challenge-location-ffa33.sql', 'AR_DATA-1692.sql', 'OH-CUST-171-DSL-Cellular-Busi.sql', 'TX_DATASME-739.sql', 'MS_CUST-3690-supervisor-district-counts.sql', 'VA_DATASME_1254_locations.sql', 'CA_DATASME-819_pg.sql', 'OK_POLSME-25.sql', 'LA_DATASME_1155_22113.sql', 'CA_CUST_2399_QA.sql', 'OH-CUST-165-FW.sql', 'LA_DATASME_1155_22067.sql', 'LA_DATASME_1155_22073.sql', 'OH_CPC-540-underserved.sql', 'MS_CUST-4021_cai.sql', 'GU_DATASME-943_adjusted_v6ffa_eligible.sql', 'NC_DATASME-921.sql', 'MS_CUST-1859_CPFSubrecipientTemplate_pg.sql', 'OK_CUST-3597.sql', 'AZ_CUST-2934.sql', 'MS_CUST-1708-public-comment.sql', 'IN_DATASME-850.sql', 'LA_CUST-1280_cais.sql', 'OK_DATASME-740_changelog.sql', 'AR_DATASME_933_2.sql', 'LA_DATASME_1155_22099.sql', 'CA_DATASME_1056.sql', 'OK_DATASME-1079.sql', 'CA_DATASME-849_project_report.sql', 'CA_CUST_2354.sql', 'CA_CUST-2856.sql', 'CA_CUST-1404_pg_5b.sql', 'VA_ARC_436_projects.sql', 'OH-CUST-165-Copper.sql', 'LA_CUST-3223.sql', 'GU_CUST-3138.sql', 'LA_CUST-2502_awards_made.sql', 'MS_CUST_2317_county.sql', 'CA_DATASME-1263_pg.sql', 'TX-CUST-1948.sql', 'NY_CUST-986.sql', 'CA_CUST-3545.sql', 'TX_DATASME-1249.sql', 'MS_DATASME_1256.sql', 'TX_DATASME-882.sql', 'VA_ARC_436_companies.sql', 'CA_DATASME-1312_pg.sql', 'CA_DATASME-849_technology_report.sql', 'OH-data-summary.sql', 'CA-CUST-267-allbsl.sql', 'OK_CUST-1744_pg.sql', 'TX_CUST-2990_challenge_portal_export.sql', 'LA_CUST-2110_fp_locations_all_field.sql', 'LA_CUST-1219.sql', 'LA_DATASME_1155_22065.sql', 'LA_DATASME_1155_22071.sql', 'LA_DATASME_1155_22059.sql', 'OH-CUST-165-LBRFW.sql', 'AR_DATASME_933_1.sql', 'TX_CUST-2990_Served.sql', 'MT_CPC-462-bsl.sql', 'AZ_DATASME-795_pg.sql', 'VA_ARC_436_users.sql', 'LA_DATASME_1155_22111.sql', 'LA_DATASME_1155_22105.sql', 'LA_DATASME-1320.sql', 'OK_CUST-2915_num.sql', 'MS_DATASME-1136_underserved.sql', 'TX_CUST-3425.sql', 'VA_DATASME_1254_cai.sql', 'CA_DATASME-849A_overlapping_paus.sql', 'AZ_DATASME_1390.sql', 'OH-ORGEB_mapping.sql', 'AR_DATASME-789_cost_per_loc_pg.sql', 'CA_CUST-1404_pg_5a.sql', 'CA_DATASME-1210_eligible_ppas.sql', 'OK_DATASME-902_pg.sql', 'MT_CPC-462-cai.sql', 'CA_CUST-3604-missing.sql', 'OR_CUST-1878.sql', 'MS_DATASME-1404.sql', 'MS_DATASME-854-underserved.sql', 'OK_DATASME-915-v6.sql', 'CA_CUST-1404_pg_4a.sql', 'CA_post_challenge_validation.sql', 'IN_CPC-399.sql', 'MN_DATASME-1053_locations_round_2.sql', 'AZ_DATASME-966.sql', 'OH-CUST-173-DSL-Cellular-Busi-CAI-dedup.sql', 'TX_DATASME_842_PAU_05102025_cai.sql', 'LA_DATASME_1155_22101.sql', 'LA_DATASME_1155_22115.sql', 'CA_CUST-3554.sql', 'TX_DATASME-918.sql', 'CA_DATASME-886_all_bsl_pau.sql', 'IN_DATASME-1081.sql', 'AR_DATASME-1109.sql', 'LA_DATASME_1155_east_caroll.sql', 'MS_CUST-1708-challenge-evidence.sql', 'AR_CUST-2494-challenge.sql', 'LA_DATASME_1155_22075.sql', 'LA_DATASME_1155_22061.sql', 'CA_DATASME-1054_2.sql', 'LA_DATASME_1155_22049.sql', 'OK_CUST-2856.sql', 'TX_DATASME_842_PAU_05102025_bsl.sql', 'AR_CUST_1628_bcg_followup.sql', 'LA_CUST-2502_funding_amount.sql', 'AR_CUST_2843.sql', 'MN_DATASME_1205.sql', 'CA_CUST-3545a.sql', 'AR_CUST_1628_2nd_round.sql', 'LA_CUST-2439_gumbo2_bsls.sql', 'OR-CUST-1110.sql', 'MN_DATASME-1053_scores_round_2_initial.sql', 'CA_DATASME-1091_pg.sql', 'MS-CUST-1293-data.sql', 'MS_CUST-4021_unserved.sql', 'AZ_DATASME-990-verify.sql', 'CA_CUST-1404_pg_4b.sql', 'MS_CUST-4021_underserved.sql', 'OK_CPF_application.sql', 'CA-CAI-BSL.sql', 'LA_DATASME-1333.sql', 'NC_CUST-2020.sql', 'LA_DATASME-993.sql', 'OR_pre_challenge_non_cai_status.sql', 'MS-CUST-109.sql', 'MS_CUST-3161-bip-dedup-kmz.sql', 'NY_post_challenge_version_integrity_check.sql', 'LA_DATASME_1155_22089.sql', 'AR_DATASME-793.sql', 'OK_CUST-4133-cai-additional.sql', 'OR_raw_fabric_fixed_availability_4-16-24.sql', 'OR-CUST-1110_validation_by_location_id.sql', 'MS_DATASME-854-served.sql', 'OK_DATASME-740_202503_revert_funding.sql', 'TX_DATASME_842_PAU_05102025_combined.sql', 'OK_CUST-2883.sql', 'CA_CUST-2435-CASF_additional.sql', 'OK_CUST-1178.sql', 'MS-CUST-2033-MBI.sql', 'MN_DATASME-1053_scores_round_2_modified.sql', 'LA_DATASME_1155_22077.sql', 'LA_DATASME_1155_22063.sql', 'AZ_CUST-3434.sql', 'IN_DATASME-1083.sql', 'TX_CUST-2990_Funded.sql', 'XZ_DATASME-856_all_providers.sql', 'AR_CUST-4133-cai-qa.sql', 'LA_DATASME_1155_22117.sql', 'PA-CUST-544-challenges.sql', 'TX_APPSME-1931-applications.sql', 'OK_DATASME-1256.sql', 'AZ_DATASME-890-delta.sql', 'MS_DATASME-1364.sql', 'IN_CPC-470_10f6.sql', 'OK_DATASME-746-post-challenge-location_ffa31.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/al': ['x_al_datasme_1468_scabcd_report_locations.sql', 'x_al_datasme_1468_scabcd_report_cai.sql', 'x_al_datasme_1468_scabcd_report_locations_class12.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/ar': ['x_ar_datasme1567.sql', 'x_ar_datasme1937_subgrantee.sql', 'x_ar_datasme1937_dp_export.sql', 'x_ar_datasme1992.sql', 'x_ar_datasme1934.sql', 'x_ar_datasme1667.sql', 'x_ar_datasme1338.sql', 'x_ar_datasme1924_locations.sql', 'x_ar_datasme1850.sql', 'x_ar_datasme1677.sql', 'x_ar_datasme2020.sql', 'x_ar_datasme2009.sql', 'x_ar_datasme1337.sql', 'x_ar_datasme1937_dp_export_real_value.sql', 'x_ar_datasme1861.sql', 'x_ar_datasme1491.sql', 'x_ar_datasme1648_locations.sql', 'x_ar_datasme1782.sql', 'x_ar_datasme1924_cais.sql', 'x_ar_datasme1648_cais.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/az': ['x_az_datasme1719_spacex_consolidation_locations.sql', 'x_az_appsme2215_rp.sql', 'x_az_datasme1956_bead_locations_missing_v9.sql', 'x_az_datasme2146_funding_fabric_version.sql', 'x_az_appsme2215_dp.sql', 'x_az_datasme2109_fp_locations_bsl_v8_v9.sql', 'x_az_datasme2048_nepa_location_md5_mapping.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/ca': ['x_ca_datasme1432.sql', 'x_ca_datasme1961.sql', 'x_ca_datasme1312.sql', 'x_ca_datasme1945.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/gu': ['x_gu_datasme2033_supp_reporting_projects.sql', 'x_gu_datasme2033_supp_deployment_reporting.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/in': ['x_in_datasme1726.sql', 'x_in_datasme1456.sql', 'x_in_datasme1457.sql', 'x_in_datasme1440.sql', 'x_in_datasme1454.sql', 'x_in_datasme2117_joink_ocremc_cai.sql', 'x_in_datasme2117_joink_ocremc_bsl.sql', 'x_in_datasme2117_joink_ocremc_bsl_lake.sql', 'x_in_datasme2117_joink_ocremc_cai_lake.sql', 'x_in_datasme1821.sql', 'x_in_datasme1950.sql', 'x_in_datasme1417.sql', 'x_in_datasme1459.sql', 'x_in_datasme1458.sql', 'x_in_datasme1461.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/la': ['x_la_datasme_1533_funding_awardees.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/mn': ['x_mn_datasme2104_budget.sql', 'x_mn_datasme2104_public_awareness.sql', 'x_mn_datasme2104_overview.sql', 'x_mn_datasme1917_locations.sql', 'x_mn_datasme1930_budget.sql', 'x_mn_datasme1930_overview.sql', 'x_mn_datasme1754.sql', 'x_mn_datasme1917_schedule.sql', 'x_mn_datasme1930_locations.sql', 'x_mn_datasme1917_awareness_low_cost.sql', 'x_mn_datasme1917_overview.sql', 'x_mn_datasme1930_awareness_low_cost.sql', 'x_mn_datasme1930_schedule.sql', 'x_mn_datasme1702.sql', 'x_mn_datasme2104_locations.sql', 'x_mn_datasme2104_schedule.sql', 'x_mn_datasme2104_low_cost_offer.sql', 'x_mn_datasme1917_budget.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/ms': ['x_ms_datasme2171.sql', 'x_ms_dataseme1479.sql', 'x_ms_datasme1482.sql', 'x_ms_datasme1906_awarded_v9_status.sql', 'x_ms_datasme1906_address_changed.sql', 'x_ms_data2401_mississippi_river_crossing_geometry.sql', 'x_ms_datasme1482_cat4.sql', 'x_ms_datasme1482_cat5.sql', 'x_ms_datasme1886_att_national_forest.sql', 'x_ms_datasme1482_cat2.sql', 'x_ms_datasme1378.sql', 'x_ms_datasme1906_reason_code_3_missing.sql', 'x_ms_datasme1482_cat3.sql', 'x_ms_datasme1482_cat1.sql', 'x_ms_data2401_mississippi_river_crossings.sql', 'x_ms_datasme2150_red_blue.sql', 'x_ms_datasme1940.sql', 'x_ms_datasme1886_fp_locations_national_forest.sql', 'x_ms_datasme1377.sql', 'x_ms_datasme2073_tds_eacam.sql', 'x_ms_datasme2072_rdof_awarded.sql', 'x_ms_datasme1749.sql', 'x_ms_datasme1629.sql', 'x_ms_datasme1818.sql', 'x_ms_datasme1906_latlong_changed.sql', 'x_ms_datasme1274.sql', 'x_ms_datasme1671.sql', 'x_ms_datasme2134_bead12_loc.sql', 'x_ms_datasme1971_monroe_rdof.sql', 'x_ms_datasme1869.sql', 'x_ms_datasme1662.sql', 'x_ms_datasme1846.sql', 'x_ms_datasme2153_mpulse_rdof_okolona.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/ok': ['x_ok_datasme_1438_project_cost_per_adding_net_id.sql', 'x_ok_datasme_1462_applications_per_net.sql', 'x_ok_datasme1433_missing_ppa_cai.sql', 'x_ok_datasme_1438_fp_cai.sql', 'x_ok_datasme2034_nepa_template.sql', 'x_ok_datasme1970.sql', 'x_ok_datasme_1524_fp_cai.sql', 'x_ok_datasme_1438_project_cost_per_ppa.sql', 'x_ok_datasme1422_funded_cai.sql', 'x_ok_datasme_1732_check_project_id.sql', 'x_ok_datasme_1438_deployment_projects.sql', 'x_ok_datasme_1513_check_project_id.sql', 'x_ok_datasme_1558_check_project_id.sql', 'x_ok_datasme_1521_check_project_id.sql', 'x_ok_datasme_1712_check_project_id.sql', 'x_ok_datasme_1524_fp_no_bead_locations.sql', 'x_ok_datasme_1524_fp_locations.sql', 'x_ok_datasme_1438_fp_locations.sql', 'x_ok_datasme_1524_fp_deployment_projects.sql', 'x_ok_datasme_2105_bead12.sql', 'x_ok_datasme1433_datavalidation.sql_', 'x_ok_datasme_1438_dup_location.sql', 'x_ok_datasme1422_eligible_cai.sql', 'x_ok_datasme_1531_check_project_integrity.sql', 'x_ok_datasme1422_eligible_locations.sql', 'x_ok_datasame_1438_identify_other_loc.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/public': ['x_public_datasme1915.sql', 'x_public_test.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/rad': ['x_nc_rad_negotiator_buildings_buncombe.sql', 'x_nc_rad_negotiator_parcels_buncombe.sql', 'x_nc_rad_tiger_fallback_neighbor_dist.sql', 'x_nc_rad_negotiator_addresses_buncombe.sql', 'x_nc_rad_missing_addr.sql', 'x_nc_rad_openaddress_buncombe.sql', 'x_nc_rad_e911_buncombe.sql', 'x_nc_rad_e911_37119.sql', 'x_nc_rad_negotiator_structures_buncombe.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/tx': ['x_tx_datasme1894_eupa_debug.sql', 'x_tx_datasme1539_budget.sql', 'x_tx_datasme2002_application_log.sql', 'x_tx_datasme1539_activities_non_leo.sql', 'x_tx_datasme1539_additional.sql', 'x_tx_datasme1596.sql', 'x_tx_datasme1392.sql', 'x_tx_datasme2084_remaining_research_list.sql', 'x_tx_datasme2084_county_summary.sql', 'x_tx_datasme1674_appdata.sql', 'x_tx_datasme1674_budget.sql', 'x_tx_datasme1370.sql', 'x_tx_datasme1991.sql', 'x_tx_datasme1674_nonleo.sql', 'x_tx_datasme1674_leo.sql', 'x_tx_datasme2084.sql', 'x_tx_datasme1674_permitting.sql', 'x_tx_datasme1410.sql', 'x_tx_datasme1925_bead_locations_missing_v9.sql', 'x_tx_datasme1539_activities_leo.sql', 'x_tx_datasme1539_permits.sql', 'x_tx_datasme1674_apploc.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/ut': ['x_ut_datasme1755.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/va': ['x_va_datasme2095_llu_nodes.sql', 'x_va_datasme2095_llu_applications.sql', 'x_va_datasme2095_llu_file_metadata.sql', 'x_va_datasme1847_unfunded_addresses.sql', 'x_va_datasme2136_baba_report_attachments.sql', 'x_va_datasme2095_llu_location_activities.sql', 'x_va_datasme2095_llu_form_answers.sql', 'x_va_datasme2095_llu_locations.sql', 'x_va_datasme1419_reporting_project_cai.sql', 'x_va_datasme2136_baba_report_history.sql', 'x_va_datasme2136_baba_report_answers.sql', 'x_va_datasme1419_reporting_project_locations.sql', 'x_va_datasme1419_reporting_projects.sql', 'x_va_datasme2136_baba_reports.sql', 'x_va_datasme2095_llu_reviews_revisions.sql', 'x_va_datasme1419_deployment_reporting.sql'], '/home/ubuntu/airflow/dags/general/flat_file_exporter_dbt/dbt_project/models/xz': ['x_xz_datasme1500_nepa_project_locations.sql', 'x_xz_datasme2027_fabric_churn_summary.sql', 'x_xz_datasme2058_bead15_loc_funded.sql', 'x_xz_datasme2058_bead15_loc.sql', 'x_xz_all_potential_dup_applicants.sql', 'x_xz_datasme1568.sql', 'x_xz_datasme2058_bead15.sql', 'x_xz_datasme2027_fabric_churn_locations.sql', 'x_xz_default_query.sql', 'x_xz_datasme2027_fabric_churn_all_locations.sql', 'x_xz_datasme2027_fabric_coverage.sql'], '/home/ubuntu/airflow/dags/general/ingest_from_s3/json': ['ar_cust_4133_no_bead.json', 'ms_proposed_project_areas.json', 'ms_cpf_datasme813_ackerman.json', 'oh_cpc_post_challenge_ntia_classification_cust_2706.json', 'tx_cpc_curing_20250415.json', 'nc_cpc_post_challenge_cai_name_revision.json', 'ok_update_cai_list_07082025_flagged_for_removal.json', 'ar_cust_bead_results_stage.json', 'ms_ulfw_20250626_latest_fabric.json', 'in_final_proposal_combine_20260409.json', 'ar_cust_4133_ar_bb_round_20250819.json', 'or_app_2927.json', 'mn_datasme_1314_fp_subgrant_agreements.json', 'in_cpc_post_remediated.json', 'in_address_data_no_code_3_4_update.json', 'la_final_proposal_final_no_bead_locations.json', 'tx_ulfw_gvtc_challenge_loc.json', 'tx_datasme_2084.json', 'ms_bip_polarville_locations.json', 'in_r2_ppa_master_project_area_join.json', 'ar_datasme_973_cai_verification.json', 'ms_cpf_indicated_locations.json', 'ok_cust_4130_award_locs.json', 'ar_cpc_post_challenge_cai_revision.json', 'ok_datasme-1153_final_proposal_cai.json', 'data_1601.json', 'ok_datasme_1513_cai_removal.json', 'ms_cpc_post_challenger.json', 'tx_fp_no_bead_usda.json', 'ms_cpc_challenger.json', 'ar_datasme_1211.json', 'la_rdof_removal_dup.json', 'in_bad_ec_in_daviess.json', 'ca_cpuc_high_need_locations_20250522.json', 'la_final_proposal_final_deployment_projects.json', 'oh_cpc_post_challenge_ntia_classification_cust_2770.json', 'ms_cpc_curing_update_20250131.json', 'TX_curing_results_20251015.json', 'tx_cpc_conceded_planned_locations_spectrum_actual.json', 'ms_rate_no_bead_20250827.json', 'ms_cpc_post_challenges.json', 'in_r2_ppa_raw_project_areas.json', 'ms_ulfw_code5.json', 'ms_cpc_curing_update_20250326.json', 'ar_cust_4133_org194.json', 'va_datasme_1409_fp_supp_users.json', 'ar_datasme_1861.json', 'in_cust_4136_provider_mapping.json', 'tx_ulfw_spectrum_actual.json', 'tx_ppa_ingest.json', 'ms_cpf_round_2_20250303.json', 'ok_slfrf_v4_s2.json', 'nc_cai_beadchallenge_240804.json', 'nj_cpc_dedup_adjustments.json', 'la_final_proposal_final_locations.json', 'in_r2_ppa_master_project_name_join.json', 'az_cpc_bead_r2_ppa.json', 'va_datasme_1356_fp_reporting_projects.json', 'nm_cpc_post_review_020625.json', 'oh_cpc_pre_challenge_orbeg.json', 'ok_datasme_1513_project_to_add.json', 'az_bob_locations.json', 'tx_final_proposal_leo_20251020.json', 'ar_datasme_1850.json', 'in_eligible_addresses_null.json', 'ms_bip_kmz_dedup_dropped.json', 'mn_restructured_bead_eligible_locations.json', 'al_datasme_1468_scabcd_report_cai.json', 'az_cpc_bead_r1_covered_cai.json', 'ar_datasme_973_paf_map.json', 'ca_cpc_post_cai_0924.json', 'tx_bob_potential_preliminary_awards.json', 'ok_bead_highcost_cbg.json', 'md_rate_datasme_1125_p1.json', 'az_cpc_post_challenge_locations.json', 'nm_cpc_post_review.json', 'ms_ulfw_20250626_cai_eligible.json', 'ca_cpc_post_cai_qa.json', 'oh_cpc_challenge_provider_id.json', 'az_cpc_post_challenge_locations_1122.json', 'ms_ rc5_not_served_latest_bdc_src.json', 'ms_ulfw_20250701_cai_master.json', 'ar_cust_cumulative_bead_results_t3.json', 'ms_bob_priority_pau.json', 'naics_index_codes.json', 'tx_ppa_20250606_donut.json', 'tx_fp_location_reference_20251013.json', 'ms_cpc_post_challenges_review.json', 'tx_ulfw_locations_20250702.json', 'tx_ppa_20250707_adjacency.json', 'tx_fp_projects_compiled_20251007.json', 'gu_bob_pre_challenge_loc.json', 'ms_cpc_curing_20250321_removed_challenges.json', 'xz_bead_highcost_cbg.json', 'ms_bead_cost_v5.json', 'la_final_proposal_final_non_deployment_projects.json', 'ms_cpc_copy_fabric_v6.json', 'ms_bead_ppa_20250512.json', 'tx_fp_20251125_projects.json', 'tx_final_proposal_leo_20251020_project.json', 'ar_datasme_1934.json', 'ca_cpuc_high_need_locations.json', 'va_datasme_1356_fp_subgrant_agreements.json', 'la_rdof_removal_pre.json', 'in_cpc_post_default_mercury_fwa.json', 'tx_final_proposal_curing_20251024.json', 'tx_cpc_conceded_planned_locations_spectrum_planned.json', 'ok_datasme_1523_fp_archive.json', 'in_omitted_code_3_4_locations.json', 'tx_fp_no_bead_cai_removed.json', 'va_bead_post_challenge_locations.json', 'ms_datasme_1886_national_forest_boundaries.json', 'la_rdof_removal_post.json', 'ms_psap_locations_id.json', 'ok_datasme-1153_final_proposal_locations.json', 'ms_no_bead_initial_submission_datasme_1171.json', 'ms_rate_no_bead.json', 'tx_fp_location_qc_20151024.json', 'in_cpc_post_speed_adjustment.json', 'ar_bb_round_sgs_results_approved_20250808.json', 'tx_bob_remove_locations_rc6.json', 'ms_cpc_curing_update_20250224.json', 'tx_cpc_conceded_planned_locations_easttex.json', 'ms_ulfw_20250626_master_indicated.json', 'ar_cust_bead_results_t3.json', 'ok_update_cai_list_07082025_new_updated_list.json', 'ms_34_ATT_Marshall_1617_provisional_awarded_bsls.json', 'tx_fp_20251125_locations.json', 'la_additional_fedfunded_012326.json', 'tx_fp_locations_compiled_20251007.json', 'la_final_proposal_final_cai.json', 'tx_fp_no_bead_v7_removed.json', 'ca_bob_round_3_ppa_awarded_apps.json', 'ar_cpc_post_challenge_cai_sbo_cust_1628.json', 'tx_ulfw_spectrum_planned.json', 'tx_bob_remove_locations_rc5.json', 'tx_found_cai_names.json', 'nm_cpc_post_validation.json', 'ms_datasme_1886_att_national_forest_list.json', 'mn_cpc_post_remediated.json', 'in_cpc_post_default_mercury_fiber.json', 'ar_cust_cbg_recommended_results.json', 'ms_bip_kmz_funded.json', 'ms_rdof_funded_d25_20260722.json', 'or_cpc_post_challenge.json', 'ms_cpc_curing_20250321_updated_locations.json', 'datasme_1341.json', 'mn_irrrb_pbc_no_bead.json', 'ca_bob_round_3_ppa_awarded_ppas.json', 'ca_cpuc_fire_threat_map_v3r.json', 'in_madison_co_name_update.json', 'ca_bead_urban_rural_counties.json', 'tx_fp_no_bead_state_challenge.json', 'tx_cpc_post_challenge.json', 'tx_fp_deployment_project_reference_20251013.json', 'ok_proposed_project_areas_20250305.json', 'va_datasme_1254_fp_subgrant_agreements.json', 'ok_datasme_1523_match_revision.json', 'tx_fp_potential_special_process_202151007.json', 'tx_ppa_20250514.json', 'ms_bead_ppa_20250602.json', 'az_project_areas_attributes.json', 'ms_tlp_location_speedcheck.json', 'xz_deployment_projs_datasme_1330.json', 'ca_cpc_post_cai_add.json', 'ar_cpc_post_location_sb_cust_1628.json', 'tx_bead_cai_outside.json', 'mn_bob_all_bead_locations.json', 'ca_bob_bsl_pau_mapping.json', 'mn_application_area_polygons.json', 'md_rate_datasme_1125_p2.json', 'tx_ulfw_comcast_concedes.json', 'cust_1699.json', 'ca_gms_locations_20250304.json', 'ms_ulfw_20250626_latest_rdof.json', 'ar_cust_cbg_recommended_results_stage.json', 'national_bead_progress.json', 'nc_cpc_datasme738_post_challenge_cai.json', 'az_cpc_bob_ppa_price.json', 'la_final_proposal_final_subgrantees.json', 'az_cpc_bead_r2_ppa_price.json', 'va_datasme_1356_fp_reporting_projects_locations.json', 'tx_cpc_conceded_planned_locations_gvtc.json', 'tx_fp_cai_reference_20251013.json', 'ms_app_1297_bsl_primary.json', 'tx_datasme_2052_bead_permit_bulk_load_consolidated.json', 'tx_ppa_20250606.json', 'ms_cpc_curing_update_20250210.json', 'ms_lookback_locations_datasme_1940.json', 'ca_bob_files_20250711.json', 'ok_bead_fp_cai_2.json', 'in_fp_no_bead_warren.json', 'ar_cpc_post_location_sbo.json', 'in_fp_no_bead_rossville.json', 'ny_default_funding_locations.json', 'ar_cust_bead_result_t1.json', 'tx_fp_no_bead_state_details.json', 'or_cpc_post_challenge_disposition.json', 'ok_oklahoma_locations_npv.json', 'tx_bob_remove_locations_rc4.json', 'ar_cust_cumulative_bead_results2.json', 'ok_datasme_1497.json', 'ms_cpc_project_areas_redrawn_20250313.json', 'va_datasme_1848_vati_funding.json', 'ar_cbg_post_challenge.json', 'tx_fp_20251125_cai.json', 'ms_cpf_datasme813_noackerman.json', 'tx_fp_20251125_no_bead_locations.json', 'in_bob_eligible_locations.json', 'gu_bob_pre_challenge_loc_20250722.json', 'ar_cust_cumulative_bead_results.json', 'xz_ppa_datasme_1330.json', 'ok_datasme_1513_fp_archive.json', 'tx_datasme_2052_permitting_excel_template_frontier_southwest.json', 'ar_datasme_1301_fp_subgrant_agreements.json', 'gu_cpc_post_challenge_ntia_20250404.json', 'gu_bob_pre_challenge_loc_project.json', 'az_cpc_bead_r2_locs.json', 'ar_cust_bead_results.json', 'al_datasme_1468_scabcd_report_locations.json', 'oh_cpc_post_challenge_0102.json', 'ca_cpuc_high_cost.json', 'ms_cpf_round_2_preliminary_determination_locations.json', 'ms_cpc_post_challenges_011524.json', 'ok_slfrf_v4.json', 'tx_bob_potential_preliminary_awards_20251007.json', 'ar_paf_datasme_949.json', 'ok_bead_fp_cai.json', 'nc_cpc_post_challenge_cai_zipcode.json', 'tx_challenge_portal_export_01222025.json', 'in_fp_no_bead_9095.json', 'tx_ppa_20250707_donut.json', 'va_datasme_1356_fp_reporting_projects_cais.json', 'ms_exhibit_c_bead_locations_csv_file_082826.json', 'ms_datasme_2134_cpf_beam_10122026.json', 'ms_datasme_2153_psc_current_cas.json', 'la_ready_data_compare012026V1.json', 'ms_cpc_curing_20250321_updated_challenges.json', 'az_cpc_bead_r1_covered_bsl.json', 'oh_arc_mbdr_1076.json', 'ok_datasme-1559_final_proposal_choctaw_nation.json', 'ms_ulfw_20250626_cai_ineligible.json', 'tx_bead_pau_summary_20250508.json', 'tx_bob_remove_locations_rc2.json', 'in_appsme_1900.json', 'ok_bead_net_20250414.json', 'ok_fp_tribal_update_20260515.json', 'nd_cpc_post_location_sbo.json', 'ca_bob_cai_pau_mapping.json', 'ms_exhibit_c_bead_locations_csv_file_091026.json', 'ar_datasme_1992.json', 'gu_bob_pre_challenge_loc_20250724.json', 'in_r2_ppa_dissolved_project_areas.json', 'ar_cpc_post_challenge_cai_sbo.json', 'ok_datasme-1559_final_proposal_choctaw_nation_v2.json', 'ca_cpc_post_cai_20250321.json', 'tx_ppa_clean_geometry_20250609.json', 'ok_datasme_1433.json', 'ca_gms_locations_20250314.json', 'ok_datasme_1472.json', 'ms_bip_curing_update_20250210.json', 'ok_proposed_project_areas.json', 'ca_datasme_1638_tribal_locations.json', 'az_project_areas.json', 'oh_cpc_post_challenge_ntia_underserved.json', 'ar_datasme_1809_spacex.json', 'ca_casf_initial.json', 'xz_boundary_datasme_1330.json', 'ca_bob_round_3_ppa.json', 'tx_fp_deployment_project_reference_20251014.json', 'in_tcc_data.json', 'ms_psap_locations.json', 'tx_fp_20251125_subgrantees.json', 'ca_cafii_casf_overlap_locations.json'], '/home/ubuntu/airflow/dags/general/map_image_generator/json': ['az_datasme1719_spacex_consolidation.json'], '/home/ubuntu/airflow/dags/general/map_image_generator/sql': ['az_datasme1719_spacex_consolidation.sql'], '/home/ubuntu/airflow/dags/general/pg_sql_breakfixer/SQL': ['AR_DATASME-1892p_archive_superseded_records_stage1.sql', 'UT_DATASME-2029_preset_nn21_rp_to_ter.sql', 'AZ_DATASME-1870_fix_dp_sga_financial_fanout.sql', 'MS_DATASME-1887_fix_sa_legacy_id.sql', 'AR_DATASME-2030_remove_superseded_comcast_sga.sql', 'test.sql', 'VA_DATASME-1943_prod_rollback_stale_location_generations.sql', 'VA_DATASME-1895_remove_stale_apps_and_orphan_status.sql', 'VA_DATASME-1519_restore_stage_users_with_preferences.sql', 'UT_DATASME-1647_fix_marina_user_record.sql', 'VA_DATASME-1895_reset_changed_20260709_cai_apploc.sql', 'AR_DATASME-1892_archive_crossprovider_spacex.sql', 'TX_DATASME-1126_update_cai_address.sql', 'AR_DATASME-1892p_archive_superseded_records_stage2.sql', 'XY_DATASME-2159_backup_mode_smoke.sql', 'AR_DATASME-1892_backup_stale_null_versiondate_apploc.sql', 'GU_DATASME-2033_dev_realign_bead_app_locations.sql', 'UT_DATASME-1647_delete_null_address_apploc.sql', 'AR_DATASME-1892_fix_natc_sa_dp_relation.sql', 'VA_DATASME-1943_stage_rollback_cai_repair.sql', 'MS_DATASME-1887_archive_stale_reporting_projects_stage2.sql', 'IN_DATASME-2018_repoint_nested_project_plans.sql', 'VA_DATASME-1895_drop_amazon_sga_financial_backups.sql', 'AR_DATASME-1873_fix_rp_subgrantee_match_answer.sql', 'MN_DATASME-1964_dev_fix_demo_project_nepa_answers.sql', 'VA_DATASME-1943_dev_delete_stale_location_generations_stage2.sql', 'MS_DATASME-1887_drop_stale_reporting_project_backups_stage3.sql', 'TX_DATASME-1709_fix_dp_sga_answer_fanout.sql', 'VA_DATASME-1895_drop_historical_only_apploc_backups.sql', 'VA_DATASME-1895_remove_historical_only_apploc.sql', 'AR_DATASME-1892p_backup_affected_records.sql', 'VA_DATASME-1895_drop_retained_dp_financial_backups.sql', 'VA_DATASME-1546_populate_sga_financial_fields.sql', 'MN_DATASME-1748_create_demo_applicant.sql', 'IN_DATASME-1874_fix_dp_sga_answer_fanout.sql', 'VA_DATASME-1943_prod_delete_stale_location_generations_stage2.sql', 'VA_DATASME-1943_stage_snapshot_cai_repair_stage1.sql', 'VA_DATASME-1696_revert_unintended_rp_sgas.sql', 'VA_DATASME-1895_drop_comcast_sga_financial_backups.sql', 'GU_DATASME-1878_fix_dp_sga_answer_fanout.sql', 'VA_DATASME-1895_remove_misassigned_amazon_rp_locations.sql', 'MN_DATASME-1877_fix_dp_sga_financial_fanout.sql', 'CA_DATASME-1945_backfill_bob_round_3_kuiper_metadata.sql', 'VA_DATASME-2095_reset_llus.sql', 'VA_DATASME-1696_normalize_dp_sga_legacy_ids.sql', 'AZ_DATASME-1730_spacex_consolidation_prod_cleanup.sql', 'CA_DATASME-2152_dev_delete_fp_test_artifacts.sql', 'VA_DATASME-1860_fix_dp_sga_financial_fanout.sql', 'VA_DATASME-2014_fix_invalid_pit_speedrank.sql', 'IN_DATASME-2018_unlink_mainstream_parent_sga_dp_relations.sql', 'OK_DATASME-1872_fix_dp_sga_financial_fanout.sql', 'MS_DATASME-1887_backup_stale_reporting_projects_stage1.sql', 'AZ_DATASME-1870_fix_rp_subgrantee_match_answer.sql', 'VA_DATASME-1895_reconcile_comcast_dp_based_sga_answer.sql', 'VA_DATASME_1346_clean_stage.sql', 'GU_DATASME-1811_clean_pit_unicode_whitespace.sql', 'ID_DATASME-2028_align_sga_legacy_ids_to_project_num.sql', 'AR_DATASME-1873_fix_dp_sga_financial_fanout.sql', 'VA_DATASME-1943_stage_rollback_stale_location_generations.sql', 'AR_DATASME-1892_archive_att_consolidation.sql', 'IN_DATASME-2018_remove_duplicate_sgas.sql', 'MS_DATASME-1887_fix_rp_legacy_ids.sql', 'VA_DATASME-1943_prod_snapshot_stale_location_generations_stage1.sql', 'AR_DATASME-2030_align_sga_legacy_ids.sql', 'VA_DATASME-1843_backfill_pop_cai_technology.sql', 'GU_DATASME-2033_stage_realign_gta_bsl_locations.sql', 'VA_DATASME-1943_stage_snapshot_stale_location_generations_stage1.sql', 'VA_DATASME-2136_prod_delete_baba_reports.sql', 'MN_DATASME-1748_create_demo_projects.sql', 'VA_DATASME-1943_dev_rollback_stale_location_generations.sql', 'VA_DATASME-1943_dev_snapshot_cai_repair.sql', 'VA_DATASME-1943_drop_cai_repair_backups.sql', 'VA_DATASME-1943_stage_delete_stale_location_generations_stage2.sql', 'ID_DATASME-2035_backfill_bsl_addresses_from_pit.sql', 'VA_DATASME-1829_delete_null_technology_cai_apploc.sql', 'OK_DATASME-1872_fix_dp_subgrantee_match_answer.sql', 'VA_DATASME-1943_dev_apply_cai_repair_stage2.sql', 'AZ_DATASME-1870_fix_sga_subgrantee_match_answer.sql', 'XY_DATA-2306_seed_cpf_applications.sql', 'CA_DATASME-2102_reset_sga_submission_status.sql', 'VA_DATASME-1895_reconcile_retained_dp_financials.sql', 'VA_DATASME-1895_reconcile_rp_financial_rebuild.sql', 'AZ_DATASME-1866_fix_navajo_rp_reallocation.sql', 'TX_DATASME-1979_dev_reset_llu_test_artifacts.sql', 'VA_DATASME-1895_remove_source_absent_20260709_cai_apploc.sql', 'UT_DATASME-2029_fix_rp_display_names.sql', 'TX_DATASME-1126_post_deploy_fix.sql', 'VA_DATASME-1895_drop_archive_delta_cai_backups.sql', 'IN_DATASME-2008_remove_surf_berrycomm_arc.sql', 'VA_DATASME-1895_reconcile_amazon_dp_based_sga_financials.sql', 'GU_DATASME-1878_fix_dp_sga_financial_fanout.sql', 'AR_DATASME-1873_fix_dp_subgrantee_match_answer.sql', 'VA_DATASME-1895_drop_cai_apploc_backups.sql', 'AR_DATASME-1892_recalc_dp_rp_location_counts.sql', 'IN_DATASME-1897_backup_south_central_remc_arc.sql', 'AR_DATASME-1892_archive_leo_superseded_dps.sql', 'AR_DATASME-1873_fix_dp_sga_answer_fanout.sql', 'DATA-1944_rename_fcc_address_search.sql', 'IN_DATASME-1897_drop_south_central_remc_backup.sql', 'KS_DATASME-2159_remove_test_ehp_milestone_submission.sql', 'IN_DATASME-1897_remove_south_central_remc_arc.sql', 'CA_DATASME-1875_fix_dp_sga_answer_fanout.sql', 'CA_DATASME-1875_fix_dp_sga_financial_fanout.sql', 'AR_DATASME-1892_archive_superseded_records_stage2.sql', 'VA_DATASME-1943_stage_apply_cai_repair_stage2.sql', 'VA_DATASME-1546_swap_sga_dp_relation_direction.sql', 'UT_DATASME-1876_fix_dp_sga_answer_fanout.sql', 'VA_DATASME-1546_fix_sga_legacy_ids.sql', 'AR_DATASME-1892_backup_crossprovider_spacex.sql', 'VA_DATASME-2136_dev_delete_baba_reports.sql', 'AR_DATASME-1892_backup_leo_dp_collapse.sql', 'ID_DATASME-1891_remove_stale_sga_applications.sql', 'VA_DATASME-1943_dev_rollback_cai_repair.sql', 'VA_DATASME-1829_backfill_dp_rp_cai_technology.sql', 'AR_DATASME-1892_archive_superseded_records_stage1.sql', 'OK_DATASME-1872_fix_sga_miles_jobs_answer_fanout.sql', 'MS_DATASME-2123_move_conexon_llu_locations.sql', 'OK_DATASME-1872_fix_rp_subgrantee_match_answer.sql', 'AR_DATASME-1892_backup_affected_records.sql', 'TX_DATASME-2145_drop_orphan_fts_stage_table.sql', 'AZ_DATASME-1870_fix_sga_miles_jobs_answer_fanout.sql', 'AZ_DATASME-1972_remove_excluded_trico_cai_locations.sql', 'OK_DATASME-1872_fix_sga_subgrantee_match_answer.sql', 'IN_DATASME-1874_fix_dp_sga_financial_fanout.sql', 'AR_DATASME-1892_drop_backup_temp_tables.sql', 'VA_DATASME-1895_drop_rp_financial_rebuild_backups.sql', 'ALL_DATASME-1654_fix_rp_application_name.sql', 'AZ_DATASME-1730_spacex_consolidation.sql', 'VA_DATASME-1895_drop_stale_app_backups.sql', 'AR_DATASME-1892_delete_stale_null_versiondate_apploc.sql', 'AR_DATASME-1892_backup_att_consolidation.sql', 'AR_DATASME-1892_drop_staleal_backup_temp_tables.sql', 'MN_DATASME-1877_fix_dp_sga_answer_fanout.sql', 'UT_DATASME-1876_fix_dp_sga_financial_fanout.sql', 'MS_DATASME-2116_repoint_public_gcp_files_to_s3_private.sql', 'CA_DATASME-2102_prod_wipe_bead_projects.sql', 'UT_DATASME-2028_align_sga_legacy_ids_to_project_num.sql', 'VA_DATASME-1943_dev_snapshot_stale_location_generations_stage1.sql', 'AR_DATASME-1892_repoint_leo_aggregates.sql', 'MS_DATASME-1684_remove_stale_fp_projects.sql', 'AR_DATASME-1892_backup_leo_repoint.sql', 'ID_DATASME-1891_drop_stale_sga_backup_tables.sql', 'VA_DATASME-1943_drop_stale_location_generations_backups.sql', 'TX_DATASME-1709_fix_dp_sga_financial_fanout.sql', 'AR_DATASME-1892_remove_natc_duplicate_sga.sql', 'IN_DATASME-2018_repoint_project_ids_to_affiliate_tokens.sql', 'VA_DATASME-2136_stage_delete_baba_reports.sql', 'AZ_DATASME-1870_fix_dp_subgrantee_match_answer.sql', 'VA_DATASME-1546_missing_sga_dp_relations.sql', 'VA_DATASME-1895_drop_amazon_reassignment_backups.sql', 'TX_DATASME-1979_delete_pristine_llus.sql'], '/home/ubuntu/airflow/dags/general/s3_geofile_to_athena/json': ['ca_rail_road_network_highway_rail_crossings.json', 'ca_rail_road_network_california_rail_network.json', 'tx_congressional_district_house.json', 'tx_congressional_district_house_20251201.json', 'ca_state_highway_network_lines.json', 'ca_senate_boundaries_2022.json', 'ca_assembly_boundaries_2022.json', 'ca_cpuc_high_cost.json', 'tx_congressional_district_senate.json', 'tx_congressional_district_senate_20251201.json'], '/home/ubuntu/airflow/dags/general/test/dbt_intro/majid/dbt_project/models/staging/athena': ['test_dbt_cross_db_query.sql', 'test_dbt_athena__permutations.sql'], '/home/ubuntu/airflow/dags/general/test/dbt_intro/travis/dbt_project/models/staging': ['dbt_schema.yml'], '/home/ubuntu/airflow/dags/general/test/dbt_intro/travis/dbt_project/models/staging/athena': ['stg_athena__permutations.sql'], '/home/ubuntu/airflow/dags/general/test/dbt_intro/travis/dbt_project/models/staging/grapi': ['stg_grapi__permutations.sql'], '/home/ubuntu/airflow/dags/general/test/dbt_test2/dbt_project/models': ['t_dbt_test003_reference_select.sql', 't_dbt_test002_source_select.sql', 't_dbt_test004_loop_select.sql', 't_dbt_test002_source_select.yml', 't_dbt_test001_simple_select.sql', 'dbt_schema.yml', 't_dbt_test005_marco_select.sql'], '/home/ubuntu/airflow/dags/general/test/mock_ingest_export/dbt_project/models': ['x_ca_rail_network.sql', '_sources.yml'], '/home/ubuntu/airflow/dags/general/test/mock_ingest_export_2/dbt_project/models': ['x_ca_rail_network.sql', '_sources.yml'], '/home/ubuntu/airflow/dags/general/test/shaoying_dbt_project/dbt_project/models': ['select_cross_db.sql', 'test_run_cross_db.sql'], '/home/ubuntu/airflow/dags/geometry_library/SQL': ['single_dataset_insert.sql', 'standard_geography.sql', 'create_table_c_geometry_library_in_pg.sql'], '/home/ubuntu/airflow/dags/grow/rec/site_selection/SQL': ['adjacent_matrix.sql', 'town_isp.sql', 'business_overall.sql', 'dist_rec.sql', 'bsl_coverage.sql'], '/home/ubuntu/airflow/dags/lsd/both/SQL': ['athena_cet_c_lsd.sql', 'athena_cet_t_lsd.sql', 'pg_ddl_c_lsd_individual.sql', 'athena_insert_c_lsd.sql', 'pg_create_c_lsd.sql'], '/home/ubuntu/airflow/dags/lsd/cb20_level/SQL': ['cb20_lsd.sql', 'create_table_c_lsd_cb20_in_pg_boss_platform.sql', 'create_table_c_lsd_cb20_in_pg.sql'], '/home/ubuntu/airflow/dags/lsd/other/SQL': ['ctas_stage1_combine.sql', 'ctas_stage0_fa.sql'], '/home/ubuntu/airflow/dags/lsd/sbo/SQL': ['t_lsd_sbo.sql'], '/home/ubuntu/airflow/dags/mbdr/curate_mbdr/dbt_project/models': ['c_mbdr_businesses.sql', 't_ohio_google_locations.sql', 't_ohio_mbdr_locations.sql', 'c_mbdr_businesses.yml', 't_ohio_utility_commission.sql', 't_ohio_frn_providers.sql', 'sources.yml'], '/home/ubuntu/airflow/dags/mbdr/get_raw_data/SQL': ['mbdr_offline_dedup_input.sql', 'mbdr_curated.sql', 'merge_locations.sql', 'mbdr_offline_dedup_llm_input.sql', 'mbdr_ohio_normalization.sql'], '/home/ubuntu/airflow/dags/mbdr/overture_gap_fill/SQL': ['05_dedup.sql', '01_categories.sql', '02_businesses.sql', '06_gap_fill.sql', '04_pairs.sql', '03_candidates.sql'], '/home/ubuntu/airflow/dags/mbdr/publish_to_pg/SQL': ['pg_create_c_mbdr_businesses.sql'], '/home/ubuntu/airflow/dags/network_resilience/SQL': ['fiber_edges.sql', 'co_nodes.sql'], '/home/ubuntu/airflow/dags/point_in_time/create_schema/SQL': ['1_load_fcc_ffa_tech_map.sql', '0_create_tables.sql', '1_load_fcc_fabric_addresses.sql'], '/home/ubuntu/airflow/dags/proposed_project_areas/curate_ppa/SQL': ['indiana.sql', 'arkansas.sql', 'mississippi.sql', 'demo_base.sql', 'minnesota.sql', 'texas.sql', 'arizona.sql', 'california.sql', 'pg_c_proposed_project_areas.sql', 'lake_pg_copy.sql', 'oklahoma.sql'], '/home/ubuntu/airflow/dags/proposed_project_areas/curate_ppa_locations/SQL': ['indiana.sql', 'arkansas.sql', 'mississippi.sql', 'demo_base.sql', 'template.sql', 'texas.sql', 'arizona.sql', 'california.sql', 'pg_c_proposed_project_areas_locations.sql', 'oklahoma.sql'], '/home/ubuntu/airflow/dags/proposed_project_areas/tile/SQL': ['indiana.sql', 'arkansas.sql', 'mississippi.sql', 'demo_base.sql', 'texas.sql', 'california.sql', 'oklahoma.sql'], '/home/ubuntu/airflow/dags/rad/address/e911/ingest/dbt_project/models': ['t_e911_nc.sql', 't_e911_or.sql', 'sources.yml'], '/home/ubuntu/airflow/dags/rad/address/e911/ingest_fl/dbt_project/models': ['t_e911_fl.sql', 'sources.yml'], '/home/ubuntu/airflow/dags/rad/classifier/stage1_features/SQL': ['audit_unmapped_usecodes.sql', 'label_distribution.sql'], '/home/ubuntu/airflow/dags/rad/classifier/stage1_features/dbt_project/models': ['schema.yml', 't_rad_classifier_features.sql'], '/home/ubuntu/airflow/dags/rad/classifier/stage2_building_type/dbt_project/models': ['schema.yml', 't_rad_classifier_building_type.sql'], '/home/ubuntu/airflow/dags/rad/classifier/stage3_verification/SQL': ['step4_non_oth_audit.sql', 'step4b_non_oth_usedesc.sql', 'step6_bsl_delta_sample.sql', 'step8_location_id_yield.sql', 'step7_geometry_outliers.sql', 'step5_assessor_gap.sql', 'step2_bus_drivers.sql', 'step3b_non_comm_audit.sql', 'step1_county_bsl.sql', 'step3_non_gov_size.sql'], '/home/ubuntu/airflow/dags/rad/google_alpha_earth/SQL': ['alphaearth_embeddings.sql'], '/home/ubuntu/airflow/dags/rad/google_alpha_earth/dbt_project/models': ['schema.yml'], '/home/ubuntu/airflow/dags/rad/negotiator/stage0_clean/SQL': ['address_units.sql', 'flag_subparcels.sql', 'clean_parcels.sql', 'clean_places.sql', 'clean_buildings.sql', 'clean_addresses.sql', 'excluded_addresses.sql', 'excluded_parcels.sql'], '/home/ubuntu/airflow/dags/rad/negotiator/stage0_clean/dbt_project/models': ['schema.yml', 't_rad_excluded_addresses.sql', 't_rad_address_units.sql', 'r_rad_parcels_negotiator.sql', 'r_rad_places_negotiator.sql', 't_rad_excluded_parcels.sql', 'c_rad_addresses_negotiator.sql', 'r_rad_addresses_negotiator.sql', 'r_rad_buildings_negotiator.sql'], '/home/ubuntu/airflow/dags/rad/negotiator/stage1_relationships/SQL': ['rel_address_parcel.sql', 'rel_place_building.sql', 'rel_address_building.sql', 'rel_building_parcel.sql'], '/home/ubuntu/airflow/dags/rad/negotiator/stage1_relationships/dbt_project/models': ['schema.yml', 't_rad_rel_place_building.sql'], '/home/ubuntu/airflow/dags/rad/negotiator/stage2_negotiate/SQL': ['01_classify.sql', '02_structures.sql'], '/home/ubuntu/airflow/dags/rad/negotiator/stage2_negotiate/dbt_project/models': ['schema.yml', 't_rad_negotiate_classify.sql', 't_rad_structures.sql'], '/home/ubuntu/airflow/dags/rad/negotiator/stage3_address/dbt_project/models': ['schema.yml', 't_rad_structures_addressed.sql', 't_census_addrfeat_25_v1.sql', 't_rad_addr_candidates_tiger.sql'], '/home/ubuntu/airflow/dags/rad/negotiator_edge_score/stage1_features/SQL': ['training_rows.sql', 'addr_competing_bldgs_30m.sql', 'bldg_competing_addrs_30m.sql', 'addr_density_50m.sql', 'addr_alphaearth_2025.sql'], '/home/ubuntu/airflow/dags/rad/negotiator_edge_score/stage1_features/dbt_project/models': ['schema.yml', 't_rad_rel_address_building_candidates.sql', 't_rad_rel_address_building_features.sql'], '/home/ubuntu/airflow/dags/rad/negotiator_edge_score/stage2_score/dbt_project/models': ['schema.yml', 't_rad_rel_address_building_scored.sql'], '/home/ubuntu/airflow/dags/rad/openaddresses/transform/dbt_project/models': ['t_openaddress_county.sql', 'schema.yml', 't_openaddress_usa.sql', '_t_openaddress_usa_county.sql', 't_openaddress_summary.sql'], '/home/ubuntu/airflow/dags/redshift/cdc/register_spectrum_table/SQL': ['create_external_table.sql'], '/home/ubuntu/airflow/dags/redshift/dag_trigger_scheduler/SQL': ['seed_pg_to_redshift_tables.sql', 'create_tables.sql', 'mark_triggered.sql', 'claim_items_batch.sql', 'mark_completed.sql', 'find_due_items.sql'], '/home/ubuntu/airflow/dags/regrid/sql': ['buildings_footprints_unload.sql', 'buildings_tile_partition.sql'], '/home/ubuntu/airflow/dags/safegraph/normalize_addresses/SQL': ['ctas_normalize_address.sql'], '/home/ubuntu/airflow/dags/sbo/cai/SQL': ['create_table_r_cai_sbo.sql', 'create_table_c_cai_sbo.sql'], '/home/ubuntu/airflow/dags/sbo/fabric_address/load_db/SQL': ['pg_create_table.sql', 'curate_data_first_time.sql', 'curate_data.sql'], '/home/ubuntu/airflow/dags/sbo/ms/SQL': ['bip-cust-445.sql', 'find_location_id_unique.sql', 'find_location_id.sql', 'cpf_objection_fabric.sql', 'find_service_status.sql', 'bip-cust-412_add_missing.sql', 'cpf_add_missing.sql'], '/home/ubuntu/airflow/dags/sbo/ms/bead/SQL': ['MS_fabric_post_challenge.sql', 'MS_version_comp.sql', 'MS_v5_summary.sql', 'MS_version_comp_district.sql', 'MS_post_challenge_summary_bsl.sql', 'MS_post_challenge_summary_cai.sql'], '/home/ubuntu/airflow/dags/sbo/ms/cpf_s3_to_boss/SQL': ['t_ms_cpf_awarded_locations.sql'], '/home/ubuntu/airflow/dags/sbo/oh/bead_mod/SQL': ['bizres_comp.sql', 'compare_mod.sql', 'bizres_comp_curate.sql'], '/home/ubuntu/airflow/dags/speedtest/update_ookla_mlab/SQL': ['mlab_curate_daily.sql', 'ookla_create_curate_daily.sql', 'speed_main.sql', 'ookla_m_create_transform.sql', 'ookla_m_curate.sql', 'mlab_curate_3month.sql', 'ookla_b_create_transform.sql', 'ookla_daily_create_base.sql', 'ookla_daily_unload.sql', 'mlab_create_curate.sql', 'mlab_ingest_downloads.sql', 'mlab_create_curate_daily.sql', 'ookla_m_transform.sql', 'ookla_m_partition.sql', 'mlab_daily_unload.sql', 'ookla_b_transform_pr.sql', 'ookla_b_create_curate.sql', 'speedmain_create_base.sql', 'ookla_b_curate_3month.sql', 'speedmain_unload.sql', 'ookla_b_transform.sql', 'ookla_curate_daily.sql', 'mlab_daily_create_base.sql', 'mlab_ingest_uploads.sql'], '/home/ubuntu/airflow/dags/subsidy/transfer_to_pg/SQL': ['pg_create_table.sql'], '/home/ubuntu/airflow/dags/suitability/ms_hypercube/01_ingest/cdc_places/SQL': ['county_curate.sql'], '/home/ubuntu/airflow/dags/suitability/ms_hypercube/01_ingest/hrsa/SQL': ['county_curate.sql'], '/home/ubuntu/airflow/dags/suitability/ms_hypercube/03_composite/dbt_project/models/ms_hyper/curated': ['ms_hyper_suitability_scores.sql', 'ms_hyper_county_suitability.sql'], '/home/ubuntu/airflow/dags/suitability/ms_hypercube/03_composite/dbt_project/models/ms_hyper/transformed': ['ms_fcc_towers.sql', 'ms_client_preferred.sql', 'ms_hospitals_by_county.sql', 'ms_hrsa_health_centers.sql'], '/home/ubuntu/airflow/dags/suitability/ok_portal/02_h3_need_surface/SQL': ['t_ok_portal_tract_need.sql'], '/home/ubuntu/airflow/dags/superset/SQL': ['superset_v3_v4_trueup_comparison.sql', 'superset_ffa_version_data_comparison.sql', 'superset_ffa_version_numbers.sql'], '/home/ubuntu/airflow/dags/tileserver/generate_public_map_tiles/SQL': ['MS_latest_ffa_tile.sql', 'LA_latest_ffa_tile.sql'], '/home/ubuntu/airflow/dags/tileserver/generate_tileset/SQL': ['c_mbdr_businesses.sql', 'c_assembly_boundaries_2025_tx.sql', 'c_national_bead_final_proposal_state.sql', 'c_senate_boundaries_2025_tx.sql', 'c_bead_area_attributes_ar.sql', 'c_national_bead_progress.sql', 'c_census_city_2023.sql', 'c_census_county_2023.sql', 'c_national_bead.sql', 'locations_npv_ok.sql', 'c_village_boundaries_gu.sql', 'c_national_bead_state.sql', 'c_town_boundaries.sql', 'c_cpuc_fire_threat_ca.sql', 'cpc_merge_p_tx.sql', 'c_cpuc_high_cost_ca.sql', 'c_national_bead_final_proposal.sql'], '/home/ubuntu/airflow/dags/tileserver/generate_tileset/json': ['cpc_merge_p_tx.json', 'c_assembly_boundaries_2025_tx.json', 'locations_npv_ok.json', 'c_census_county_2023.json', 'c_cpuc_fire_threat_ca.json', 'c_national_bead.json', 'c_national_bead_progress.json', 'c_senate_boundaries_2025_tx.json', 'c_village_boundaries_gu.json', 'c_national_bead_final_proposal.json', 'c_mbdr_businesses.json', 'c_national_bead_final_proposal_state.json', 'c_cpuc_high_cost_ca.json', 'c_census_city_2023.json', 'c_town_boundaries.json', 'c_national_bead_state.json', 'c_bead_area_attributes_ar.json'], '/home/ubuntu/airflow/dags/training/08_colocated_sql_jinja/SQL': ['XY.sql', 'LA.sql'], '/home/ubuntu/airflow/dags/training/11_dbt_run/dbt_project/models': ['training_demo.sql'], '/home/ubuntu/airflow/templates/sql/CET': ['r_inup_unserved.sql', 'r_mtup_fed_dedup_bdc_121223.sql', 'r_mtup_funding_all_grant_served_locations.sql', 'r_msup_provider_common_names.sql', 'r_ksup_12_08_23_BSL_CAI_Eligibility.sql', 'r_mtup_post_challenge_cai_summit_lightbox.sql', 'r_nyup_underserved.sql', 'r_dcup_raw_cai_correction.sql', 'r_ksup_BSL_List_Eligibility_12_06_23.sql', 'r_ndup_underserved.sql', 'r_ksup_bsl_list_eligible.sql', 'r_ohup_underserved.sql', 'r_cpc_area_challenge_filtered_va.sql', 'r_inup_underserved.sql', 'r_ohup_unserved.sql', 'r_cpc_challenge_addresses_va.sql', 'r_caup_cpuc_cai_manual_removal.sql', 'r_wyup_ntia_schema_revised_april_2024_v4.sql', 'r_laup_gumbo2_data_final.sql', 'r_deup_unserved.sql', 'r_ksup_BSL_List_Eligibility_12_01_23.sql', 'r_nyup_unserved.sql', 'r_ms_cpf_awarded_locations.sql', 'r_ksup_cai_list_provider_final.sql', 'r_ndup_unservfed.sql', 'r_ksup_state_and_fed_dedup_bdc.sql', 'r_wvup_fed_dedup_bdc.sql', 'r_ksup_providers.sql', 'r_deup_state_and_fed_dedup_bdc.sql', 'r_nyup_state_and_fed_dedup_bdc.sql', 'r_deup_underserved.sql', 'r_mtup_funding_local_grants_served_locations.sql', 'r_okup_supplemenrtal_providers.sql', 'r_dcup_raw_cai_challenges_override_CUST2761.sql', 'r_ksup_cai_11_21_23.sql'], '/home/ubuntu/airflow/templates/sql/CTAS': ['c_cpc_merge_tile_ks_new_funding.sql', 'c_cpc_ks_program_name_mapping.sql', 't_arup_state_and_fed_dedup_bdc.sql', 't_okup_post_challenge_locations.sql', 'r_orup_bead_post_disposition.sql', 't_cpc_challenge_addresses_va.sql', 't_nvup_cqa_v6_fcc_many_cats.sql', 'r_vaup_underserved.sql', 'r_vaup_unserved_cai.sql', 't_okup_arpa.sql', 't_wvup_fed_dedup_bdc.sql', 't_fabric_sbo_v2_primary.sql', 'pg_cai_v5.sql', 't_nvup_cqa_v6_fcc_many_cats_oct2023_dedupe.sql', 'r_vaup_unserved.sql']}

get_dag_sql_list(path, prefix=False)

Get the list of SQL files for a given DAG path. This is used for choosing an SQL query in the parameters. :type path: str :param path: The DAG path to look up. :type prefix: bool :param prefix: If this is True, it will concatenate all paths that begin with the path instead of checking an exact match.

Indices and tables

• Index

• Search Page