=================================== Ready Airflow =================================== Welcome to Ready Airflow's documentation! Pattern Registry ================ Before creating new operators, utilities, DAG scaffolds, or SQL patterns, check: - docs/patterns/registry.md Operators ========= The following are custom operators that are used throughout the project. They are organized by the type of operator they provide. In general, operators should be created if they contain central logic that is used in multiple places. If the logic is only used in one place, it should likely be kept in the DAG itself. BaseTransformOperator --------------------- This operator is used to transform data. It is used in the `transform` task of the `BaseDAG`. .. autoclass:: operators.base_transform.BaseTransformOperator :members: LinearIssuesToS3Operator --------------------- This operator is used to ingest Linear issues from its API into S3. .. autoclass:: operators.linear_issues_to_s3.LinearIssuesToS3Operator :members: CQACollectGeometryOperator --------------------- This operator is used to invoke a CQA collect api call. .. autoclass:: operators.cost_quest.cqa_collect_geometry.BedrockInvokeOperator :members: CQAGetLocationId --------------------- An operator to get the location id from CostQuest API from longitude and latitude .. autoclass:: operators.cost_quest.cqa_get_location_id.CQAGetLocationId :members: CensusDownloaderOperator --------------------- An operator to download census shape files from the Census FTP server and store them in S3 and Athena. .. autoclass:: operators.census.census_downloader.CensusDownloaderOperator :members: BaseDBTOperator --------------------- Execute `dbt run` on the remote server. .. autoclass:: operators.dbt.dbt_base.BaseDBTOperator :members: DBTRunOperator --------------------- Execute `dbt run` on the remote server. .. autoclass:: operators.dbt.dbt_run.DBTRunOperator :members: DBTTestOperator --------------------- Execute `dbt test` on the remote server. .. autoclass:: operators.dbt.dbt_test.DBTTestOperator :members: DBTDocsOperator --------------- Execute `dbt docs generate` on the remote server. .. autoclass:: operators.dbt.dbt_docs.DBTDocsOperator :members: BedrockInvokeOperator --------------------- This operator is used to invoke a Bedrock model. .. autoclass:: operators.bedrock_agent.BedrockInvokeOperator :members: BedrockPDFExtractorOperator --------------------------- This operator is used to extract text from a PDF using a Bedrock model. .. autoclass:: operators.bedrock_pdf_extractor.BedrockPDFExtractorOperator :members: BedrockAgentOperator -------------------- This operator extends BedrockInvokeOperator to invoke a Bedrock agent. .. autoclass:: operators.bedrock_agent.BedrockAgentOperator :members: BedrockFileSummaryOperator -------------------------- This operator is used to summarize files using a Bedrock model. .. autoclass:: operators.bedrock_agent.BedrockFileSummaryOperator :members: BedrockModelOperator -------------------- This operator extends BedrockInvokeOperator to invoke a Bedrock model directly. .. autoclass:: operators.bedrock_agent.BedrockModelOperator :members: BedrockAutomationBaseOperator ----------------------------- Base operator for Bedrock automation tasks. .. autoclass:: operators.bedrock_automation_base_operator.BedrockAutomationBaseOperator :members: BedrockDataComparisonOperator ----------------------------- This operator is used to compare data using a Bedrock model. .. autoclass:: operators.bedrock_data_comp.BedrockDataComparisonOperator :members: BedrockDataCompAnalyzer ----------------------- AI analysis helper module for data comparison with structured output types. .. automodule:: operators.bedrock_data_comp_analyzer :members: WebsiteScraperOperator ---------------------- This operator is used to scrape websites and update a knowledge base. .. autoclass:: operators.bedrock_scraper.website_scraper_operator.WebsiteScraperOperator :members: CreateSchemaPostgresOperator ---------------------------- This operator is used to create a schema in Postgres. .. autoclass:: operators.create_schema_postgres.CreateSchemaPostgresOperator :members: CSVToPostgresOperator --------------------- This operator is used to load a CSV into Postgres. .. autoclass:: operators.csv_to_postgres.CSVToPostgresOperator :members: AthenaCTASOperator ------------------ This operator is used to create a table in Athena. .. autoclass:: operators.ctas.AthenaCTASOperator :members: PostgresCTASOperator -------------------- This operator is used to create a table in Postgres using CREATE TABLE AS SELECT. .. autoclass:: operators.ctas.PostgresCTASOperator :members: RedshiftCTASOperator -------------------- This operator is used to create a table in Redshift using CREATE TABLE AS SELECT with transaction support and Redshift-specific distribution/sort key options. .. autoclass:: operators.redshift.redshift_ctas.RedshiftCTASOperator :members: AthenaToS3Operator ------------------ This operator is used to export Athena query results to S3. .. autoclass:: operators.athena_to_s3.AthenaToS3Operator :members: DeleteAthenaTableOperator ------------------------- This operator is used to delete an Athena table from the Glue catalog. .. autoclass:: operators.datalake.catalog.DeleteAthenaTableOperator :members: S3ToAthenaParquetOperator ------------------------- This operator is used to create an Athena table from Parquet files in S3. .. autoclass:: operators.datalake.catalog.S3ToAthenaParquetOperator :members: TableToPGOperator ----------------- This operator is used to load a table from athena into Postgres. .. autoclass:: operators.datalake_to_pg.TableToPGOperator :members: GCSToS3RenameOperator --------------------- Transfer files from Google Cloud Storage to Amazon S3 with the ability to rename files. .. autoclass:: operators.gcs_to_s3_rename.GCSToS3RenameOperator :members: GeodatabaseToGeoparquetOperator ------------------------------- This operator is used to convert a geodatabase to geoparquet. .. autoclass:: operators.gdb_to_gpq.GeodatabaseToGeoparquetOperator :members: GoogleDriveToS3Operator ----------------------- This operator is used to transfer files from Google Drive to S3. .. autoclass:: operators.gdrive_to_s3.GoogleDriveToS3Operator :members: GDriveBaseOperator ------------------ Base operator for Google Drive operations. .. autoclass:: operators.gdrive.gdrive_base.GDriveBaseOperator :members: GDriveListFilesOperator ----------------------- This operator is used to list files in a Google Drive folder. .. autoclass:: operators.gdrive.gdrive_list.GDriveListFilesOperator :members: GDriveGetFileIdOperator ----------------------- This operator is used to get a file ID from Google Drive by name. .. autoclass:: operators.gdrive.gdrive_list.GDriveGetFileIdOperator :members: GetAirflowDagsOperator ---------------------- This operator is used to get the Airflow DAGs. .. autoclass:: operators.get_airflow_dags.GetAirflowDagsOperator :members: GithubToS3Operator ------------------ This operator is used to transfer code and files from Github to S3. .. autoclass:: operators.github_to_s3.GithubToS3Operator :members: BatchGooglePlaceFindPlaceOperator --------------------------------- This operator is used to geocoding places using Google Places API in batches .. autoclass:: operators.google_geocode_batch.BatchGooglePlaceFindPlaceOperator :members: GooglePlaceFindPlaceOperator ---------------------------- This operator is used to geocoding a single place using Google Places API. .. autoclass:: operators.google_geocode.GooglePlaceFindPlaceOperator :members: PlacesTextSearchOperator ------------------------ This operator is used to search for places using Google Places Text Search API. .. autoclass:: operators.google_place.text_search.PlacesTextSearchOperator :members: PlaceDetailsBatchOperator ------------------------- This operator is used to get place details in batches using Google Places API. .. autoclass:: operators.google_place.place_detail.PlaceDetailsBatchOperator :members: HttpToS3Operator ---------------- This operator is used to transfer files from an HTTP endpoint to S3. .. autoclass:: operators.http_to_s3.HttpToS3Operator :members: LakeMsckOperator ---------------- This operator is used to run an MSCK command on a data lake table. .. autoclass:: operators.lake_msck.LakeMsckOperator :members: LakePitTableOperator --------------------- This operator is used to create a PIT table in the data lake. .. autoclass:: operators.lake_pit_table.LakePitTableOperator :members: PostgresToDatalake ------------------ This operator is used to load a table from Postgres into the data lake. .. autoclass:: operators.pg_to_datalake.PostgresToDatalake :members: CreateRadIdNativeOperator ------------------------- This operator is used to create a RAD ID for a list of addresses. This is a athena native operator, meaning its functions are selected from Athena functions. .. autoclass:: operators.rad_id_native.CreateRadIdNativeOperator :members: BaseTileOperator ------------------ This operator is used as a base class for other tile operators. .. autoclass:: operators.tile.base_tile.BaseTileOperator :members: ListTileOperator ------------------ This operator is used to list all available tiles in Tileserver TSGL or S3, by exact string or regex pattern. .. autoclass:: operators.tile.list_tile.ListTileOperator :members: CreateTileOperator ------------------ This operator is used to create a vector tile. .. autoclass:: operators.tile.create_tile.CreateTileOperator :members: RemoveTileOperator ------------------ This operator is used to remove a vector tile from Tileserver TSGL. .. autoclass:: operators.tile.remove_tile.RemoveTileOperator :members: RemoveInS3Operator ------------------ This operator is used to remove a vector tile already archived in S3. .. autoclass:: operators.tile.remove_in_s3.RemoveInS3Operator :members: RecoverFromS3Operator ------------------ This operator is used to recover a vector tile from S3 to Tileserver TSGL. .. autoclass:: operators.tile.recover_from_s3.RecoverFromS3Operator :members: ArchiveToS3Operator ------------------ This operator is used to move (archive) a vector tile from Tileserver TSGL to S3. .. autoclass:: operators.tile.archive_to_s3.ArchiveToS3Operator :members: SyncTileOperator ------------------ This operator is used to sync tileset(s) between Tileserver TSGL environments. .. autoclass:: operators.tile.sync_tile.SyncTileOperator :members: S3GeoJsonToAthenaOperator ------------------------- This operator is used to load a GeoJSON file from S3 into Athena. .. autoclass:: operators.s3_geojson_to_athena.S3GeoJsonToAthenaOperator :members: S3ShapefileToAthenaOperator --------------------------- This operator is used to load a shapefile from S3 into Athena. .. autoclass:: operators.s3_shp_to_athena.S3ShapefileToAthenaOperator :members: SlackToS3Operator ----------------- This operator is used to transfer files from Slack to S3. .. autoclass:: operators.slack_to_s3.SlackToS3Operator :members: NotifySlackOperator ------------------- Post a notification message to a Slack channel. Drop-in companion to EmailOperator for DAG failure/success callbacks; supports @mentions via static Slack user IDs or dynamic email lookup (``users.lookupByEmail``). .. autoclass:: operators.communications.notify_slack.NotifySlackOperator :members: UpdateKnowledgeBaseOperator --------------------------- This operator is used to update the knowledge base (bedrock). .. autoclass:: operators.update_knowledge_base.UpdateKnowledgeBaseOperator :members: XLSXToPostgresOperator ---------------------- This operator is used to load an XLSX file into Postgres. .. autoclass:: operators.xlsx_to_postgres.XLSXToPostgresOperator :members: GrapiBaseOperator ----------------- Base operator for GRAPI (Grant Ready API) operations. .. autoclass:: operators.grapi.grapi_base.GrapiBaseOperator :members: UpdateApplicationAnswerOperator ------------------------------- This operator is used to update application answers in GRAPI. .. autoclass:: operators.grapi.update_application_answer.UpdateApplicationAnswerOperator :members: UpdateApplicationResourceOperator --------------------------------- This operator is used to update application resources in GRAPI. .. autoclass:: operators.grapi.update_application_resource.UpdateApplicationResourceOperator :members: InvalidateCacheOperator ----------------------- This operator is used to invalidate cache in GRAPI. .. autoclass:: operators.grapi.invalidate_cache.InvalidateCacheOperator :members: GrapiBedrockAutomationOperator ------------------------------ This operator is used for Bedrock automation tasks with GRAPI. .. autoclass:: operators.grapi.grapi_bedrock_invoice.GrapiBedrockAutomationOperator :members: BaseIcebergTableOperator ------------------------ Base operator for Iceberg table operations in AWS Athena, providing common functionality for all Iceberg operators. .. autoclass:: operators.iceberg.base_iceberg_table.BaseIcebergTableOperator :members: MaintainIcebergTableOperator ---------------------------- Performs maintenance operations on Iceberg tables, including OPTIMIZE to combine small files and VACUUM to clean up old snapshots. .. autoclass:: operators.iceberg.maintain_iceberg_table.MaintainIcebergTableOperator :members: DropIcebergPartitionOperator ---------------------------- Drops a specific partition from an Iceberg table and optionally runs maintenance afterward. .. autoclass:: operators.iceberg.drop_iceberg_partition.DropIcebergPartitionOperator :members: RemoveIcebergTableOperator -------------------------- Completely removes an Iceberg table from the catalog and (optionally) its underlying data files. .. autoclass:: operators.iceberg.remove_iceberg_table.RemoveIcebergTableOperator :members: MigrateHiveToIcebergOperator ---------------------------- Migrates a Hive table to Iceberg format with schema evolution capabilities, data validation, and optional optimization. .. autoclass:: operators.iceberg.migrate_hive_to_iceberg.MigrateHiveToIcebergOperator :members: RunCalixTestOperator -------------------------- Run Calix speed and/or latency tests on multiple devices: Process test requests in chunks, make concurrent API requests (respecting rate limits), and write results to S3-backed Athena tables. .. autoclass:: operators.calix.run_calix_test.RunCalixTestOperator :members: S3ToSFTPOperator -------------------------- Operator to transfer a file from an S3 bucket to an SFTP server using paramiko. .. autoclass:: operators.s3_to_sftp.S3ToSFTPOperator :members: ValidateOrCreateSchemaOperator -------------------------- Validates if a schema exists in Redshift and creates it if it does not, with optional ownership assignment. This operator is useful for ensuring that the necessary schema is present before running other tasks that depend on it, such as creating tables or views within that schema. .. autoclass:: operators.redshift.database_management.ValidateOrCreateSchemaOperator :members: SetAwsRolePermissiveGrantsOperator -------------------------- This operator is used to set permissive grants for an AWS role in Redshift. It ensures that the specified role has the necessary permissions to access and manage resources within the Redshift cluster. This operator is for use in an administrative capacity to provision access to data accounts. .. autoclass:: operators.redshift.database_management.SetAwsRolePermissiveGrantsOperator :members: PostgresToRedshiftDDLOperator -------------------------- This operator generates and executes a Redshift CREATE TABLE statement based on the schema of an existing Postgres table. It facilitates the migration of table structures from Postgres to Redshift by automatically translating data types and constraints. This operator is useful for data engineers looking to replicate Postgres tables in Redshift for analytics and reporting purposes. .. autoclass:: operators.redshift.pg_to_redshift.PostgresToRedshiftDDLOperator :members: PostgresToRedshiftUnloadLoadOperator ------------------------------------- Unload table data from Postgres to S3 via aws_s3 extension, then COPY into Redshift staging and INSERT…SELECT into the final table with minimal, type-aware shims. .. autoclass:: operators.redshift.pg_to_redshift.PostgresToRedshiftUnloadLoadOperator :members: GetTenantsOperator -------------------------- This operator retrieves a list of tenant schema names from a Postgres database, excluding specified schemas. It is useful for multi-tenant applications where operations need to be performed on each tenant's schema. The operator connects to the Postgres database using a provided connection ID and executes a query to fetch the schema names. .. autoclass:: operators.postgres.utils.get_tenants.GetTenantsOperator :members: RedshiftGetTenantsOperator -------------------------- This operator retrieves a list of tenant schema names from a Redshift database via the Redshift Data API. It queries svv_all_schemas filtered to a single source database (default 'grapi_sync') and is the Redshift analog of GetTenantsOperator. Useful for fanning out per-tenant work (e.g. ctasifier all_tenants mode). .. autoclass:: operators.redshift.get_tenants.RedshiftGetTenantsOperator :members: RedshiftGetDatabaseOperator --------------------------- This operator retrieves a list of Redshift database names matching a SQL LIKE pattern via the Redshift Data API. It queries svv_redshift_databases (e.g. ``database_name LIKE '%_datamart'``). Useful for fanning out CTAS work across many databases sharing a naming convention (e.g. ctasifier all_tenants + ``tenant_discovery='databases_matching'``). .. autoclass:: operators.redshift.get_database.RedshiftGetDatabaseOperator :members: RedshiftDropTableOperator ------------------------- Drops a single Redshift table with safety guardrails. Optionally authenticates as the cluster superuser (``rsadmin``) by fetching credentials from AWS Secrets Manager — useful when the target table is not owned by the airflow IAM role (Redshift's ``DROP`` is owner-only and does not support ``GRANT DROP``). Identifier comparisons and the ``DROP`` statement are case-sensitive. .. autoclass:: operators.redshift.redshift_drop_table.RedshiftDropTableOperator :members: GetLatestFabricFixedAvailabilityVersionOperator ----------------------------------------------- This operator retrieves the latest fabric fixed availability (FFA) version from the fabric_versions table. Supports filtering by status (e.g., 'ingested'), fabric address version, or fixed availability version. Returns a FabricVersion dict with version, version_date, update_date, and process_uuid. .. autoclass:: operators.postgres.utils.get_latest_fabric_fixed_availability_version.GetLatestFabricFixedAvailabilityVersionOperator :members: PostgresGetTenantsOperator -------------------------- This operator retrieves a list of tenant schema names from a Postgres database. .. autoclass:: operators.pg_get_tenants.PostgresGetTenantsOperator :members: AthenaToPostgresOperator ------------------------ This operator is used to load data from Athena into Postgres. .. autoclass:: operators.postgres.athena_to_postgres.AthenaToPostgresOperator :members: LoadFabricAddressFullTextSearchOperator --------------------------------------- This operator is used to load fabric address data for full text search in Postgres. .. autoclass:: operators.postgres.load_fabric_address_full_text_search.operator.LoadFabricAddressFullTextSearchOperator :members: BackupTableOperator ------------------- This operator is used to backup a table in Postgres. .. autoclass:: operators.postgres.utils.backup_table.BackupTableOperator :members: MangleUsersOperator ------------------- This operator is used to mangle user data in Postgres for privacy purposes. .. autoclass:: operators.postgres.utils.mangle_users.MangleUsersOperator :members: RestoreStagedUsersOperator -------------------------- This operator restores staged users after a prod-to-stage Postgres sync. Access grants whose applicants did not survive the refresh are skipped because their foreign-key target no longer exists. .. autoclass:: operators.postgres.utils.restore_staged_users.RestoreStagedUsersOperator :members: ValidateOrCreateDatabaseOperator -------------------------------- This operator validates if a database exists in Redshift and creates it if it does not. .. autoclass:: operators.redshift.database_management.ValidateOrCreateDatabaseOperator :members: SetAwsRoleAwsDataCatalogGrantsOperator -------------------------------------- This operator is used to set AWS Data Catalog grants for an AWS role in Redshift. .. autoclass:: operators.redshift.database_management.SetAwsRoleAwsDataCatalogGrantsOperator :members: RefreshAwsDataCatalogGrantsOperator ----------------------------------- This operator is used to refresh AWS Data Catalog grants in Redshift. .. autoclass:: operators.redshift.database_management.RefreshAwsDataCatalogGrantsOperator :members: SetCaseSensitiveIdentifierOperator ---------------------------------- This operator is used to set case-sensitive identifier settings in Redshift. .. autoclass:: operators.redshift.database_management.SetCaseSensitiveIdentifierOperator :members: CreateRedshiftUserOperator -------------------------- This operator is used to create a user in Redshift. .. autoclass:: operators.redshift.database_management.CreateRedshiftUserOperator :members: SetSchemaReadGrantsOperator --------------------------- This operator is used to set read grants on a schema in Redshift. .. autoclass:: operators.redshift.database_management.SetSchemaReadGrantsOperator :members: PitIngestBaseOperator --------------------- Base operator for Point-in-Time ingestion operations. .. autoclass:: operators.point_in_time.ingest.PitIngestBaseOperator :members: PitIngestArcOperator -------------------- This operator is used for Point-in-Time ingestion for ARC data. .. autoclass:: operators.point_in_time.ingest.PitIngestArcOperator :members: BuildRedactedTileSqlOperator ---------------------------- This operator is used to build SQL for redacted tiles. .. autoclass:: operators.tile.build_redacted_tile_sql.BuildRedactedTileSqlOperator :members: BaseTileDeployOperator ---------------------- Base operator for tile deployment operations. .. autoclass:: operators.tile.join_tile.BaseTileDeployOperator :members: JoinTileOperator ---------------- This operator is used to join tiles. .. autoclass:: operators.tile.join_tile.JoinTileOperator :members: S3ToGoogleDriveOperator ----------------------- This operator is used to transfer files from S3 to Google Drive. .. autoclass:: operators.s3_gdrive.S3ToGoogleDriveOperator :members: S3GeoJsonToAthenaOperator ------------------------- This operator is used to load GeoJSON from S3 into Athena. .. autoclass:: operators.s3_geojson_to_athena.S3GeoJsonToAthenaOperator :members: S3ShapefileToAthenaOperator --------------------------- This operator is used to load Shapefiles from S3 into Athena. .. autoclass:: operators.s3_shp_to_athena.S3ShapefileToAthenaOperator :members: S3ToFTPSOperator ---------------- This operator is used to transfer files from S3 to an FTPS server. .. autoclass:: operators.s3_to_ftps.S3ToFTPSOperator :members: UnzipOperator ------------- This operator is used to unzip files in S3. .. autoclass:: operators.s3_unzip.UnzipOperator :members: ZipS3Operator ------------- This operator is used to zip files in S3. .. autoclass:: operators.s3_zip.ZipS3Operator :members: MlflowModelInvokeOperator ------------------------- This operator is used to invoke an MLflow model. .. autoclass:: operators.mlflow_model_invoke.MlflowModelInvokeOperator :members: GdriveZipToS3Operator --------------------- This operator is used to zip files from Google Drive and upload to S3. .. autoclass:: operators.multifile_zipper.GdriveZipToS3Operator :members: PeeringDBToAthenaOperator ------------------------- This operator is used to load PeeringDB data into Athena. .. autoclass:: operators.peeringdb_to_athena.PeeringDBToAthenaOperator :members: StartReflectRunOperator ----------------------- This operator is used to start a Reflect run test. .. autoclass:: operators.start_reflect_run_test.StartReflectRunOperator :members: TransformPostgresOperator ------------------------- This operator is used to transform data in Postgres. .. autoclass:: operators.transform_postgres.TransformPostgresOperator :members: GenerateReadmeOperator ---------------------- This operator is used to generate README files and export to Google Drive. .. autoclass:: operators.cya.generate_readme.export.GenerateReadmeOperator :members: SafeSQLExecutorOperator ----------------------- This operator executes SQL queries with safety validation. .. autoclass:: operators.bedrock.safe_sql_executor.SafeSQLExecutorOperator :members: CQACollectGeometryOperator -------------------------- This operator invokes the CostQuest collect geometry API. .. autoclass:: operators.cost_quest.cqa_collect_geometry.CQACollectGeometryOperator :members: EsriToAthenaOperator -------------------- This operator converts an ESRI endpoint to an Athena table. .. autoclass:: operators.esri_to_athena.EsriToAthenaOperator :members: ExportQueryOperator ------------------- This operator exports a query result to a flat file format and uploads it. .. autoclass:: operators.flat_file_exporter.export_query.ExportQueryOperator :members: QueryToGDriveCSVOperator ------------------------ This operator exports a query result to CSV and uploads it to Google Drive. .. autoclass:: operators.flat_file_exporter.query_to_gdrive_csv.QueryToGDriveCSVOperator :members: FTPSToS3Operator ---------------- This operator transfers files from FTPS/SFTP to S3. .. autoclass:: operators.ftps_to_s3.FTPSToS3Operator :members: GDriveCreateFolderOperator -------------------------- This operator creates a folder in Google Drive. .. autoclass:: operators.gdrive.gdrive_folder.GDriveCreateFolderOperator :members: GDriveUploadFileOperator ------------------------ This operator uploads a file to Google Drive. .. autoclass:: operators.gdrive.gdrive_upload.GDriveUploadFileOperator :members: S3ToRedshiftOperator -------------------- This operator ingests Parquet files from S3 into Redshift. .. autoclass:: operators.redshift.s3_to_redshift.S3ToRedshiftOperator :members: DisableSupersetUserOperator --------------------------- This operator disables a Superset user (sets ``active=False``) via the Superset REST API. .. autoclass:: operators.superset.disable_superset_user.DisableSupersetUserOperator :members: AddSupersetDbConnectionOperator ------------------------------- This operator creates a new database connection in Superset via the REST API. .. autoclass:: operators.superset.add_superset_db_connection.AddSupersetDbConnectionOperator :members: AddSupersetDbConnectionFromSecretOperator ----------------------------------------- This operator creates a new Superset database connection using an AWS Secrets Manager secret. The SQLAlchemy URI is built inside ``execute()`` so the password never appears in XCom. .. autoclass:: operators.superset.add_superset_db_connection_from_secret.AddSupersetDbConnectionFromSecretOperator :members: AddSupersetDashboardOperator ---------------------------- This operator creates a dataset and one or more dashboards in Superset via the REST API. .. autoclass:: operators.superset.add_superset_dashboard.AddSupersetDashboardOperator :members: Utilities ========= The following are central utility classes that are used throughout the project. They are organized by the type of utility they provide. In general, Util classes should be created if they contain central logic that is used in multiple places. If the logic is only used in one place, it should likely be kept in the DAG itself. BaseUtil -------- This class contains central utilities that may be used for any type of pipeine. They are mainly used for central logic around requests, text cleaning, and other ubiquitous logic. .. autoclass:: util.base.BaseUtil :members: DagUtil ------- This class contains dag-level utilities that may be used for any type of DAG. They are mainly used for help with initialization, error handling, and central DAG behavior. .. autoclass:: util.dag.DagUtil :members: DagTriggerCallbacks ------------------- Completion callbacks for target DAGs triggered by the DAG Trigger Scheduler. Reports success/failure back to data.dag_trigger_run in Redshift. .. automodule:: util.dag_trigger :members: DataLakeUtil ------------ This class contains utilities for interacting with the data lake. .. autoclass:: util.data_lake.DataLakeUtil :members: DBUtil ------ This class contains utilities for interacting with the database. .. autoclass:: util.db.DBUtil :members: ECSUtil ------- This class contains utilities for interacting with ECS. .. autoclass:: util.ecs.ECSUtil :members: FCCUtil ------- This class contains utilities for interacting with FCC data. .. autoclass:: util.fcc.FCCUtil :members: RADUtil ------- This class looks like it's under construction, and I'm not sure what it's for. It's not being used anywhere. .. autoclass:: util.rad.RADUtil :members: S3Util ------- This class contains utilities for interacting with S3. .. autoclass:: util.s3.S3Util :members: CAIUtil ------- This class might contain utils for interacting with CAI data? It's not being used anywhere. .. autoclass:: util.sbo_cai.CAIUtil :members: SBOUtil ------- This class contains utilities for interacting with SBO data. It's only used in one DAG, which is perhaps not an ideal use case for a cental util. .. autoclass:: util.sbo_fabric_address.SBOUtil :members: GDriveUtil ---------- This class contains utilities for interacting with Google Drive. .. autoclass:: util.gdrive.GDriveUtil :members: DataHubUtil ----------- This class contains utilities for interacting with Datahub. .. autoclass:: util.data_hub.DataHubUtil :members: CleanDataUtil ------------- This class contains utilities for cleaning company name data and standardizing names .. autoclass:: util.clean_data.CleanDataUtil :members: BeadUtil -------- This class contains state-specific BEAD and ARC Final Proposal configuration, including source versions and DP/RP/SGA relationship modes. .. autoclass:: util.bead.BeadUtil :members: FUNDUtil -------- This class contains all specific states' bead funding information .. autoclass:: util.sbo_funding.FUNDUtil :members: SpatialUtil ----------- This class contains utilities to download data from an ESRI endpoints. .. autoclass:: util.spatial.SpatialUtil :members: RabbitMQUtil ------------ This class contains utilities for interacting with RabbitMQ. .. autoclass:: util.rabbitmq.RabbitMQUtil :members: MongoDbUtil ----------- This class contains utilities for interacting with MongoDB. .. autoclass:: util.mongodb.MongoDbUtil :members: MetadataUtil ------------ This class contains utilities for managing metadata in PostgreSQL for Airflow DAGs. .. autoclass:: util.metadata_pg.MetadataUtil :members: GitHubUtil ---------- This class contains utilities for interacting with GitHub API. .. autoclass:: util.github.GitHubUtil :members: FormatUtil ---------- This class contains utilities for formatting and converting between Python dtypes such as datetime, string, etc. .. autoclass:: util.format.FormatUtil :members: BeadPOCUtil ----------- This class contains BEAD National Map view (Proof of Concept) post-challenge and pre-challenge configurations. .. autoclass:: util.bead_poc.BeadPOCUtil :members: BeadFpUtil ---------- This class contains BEAD Final Proposal file path configurations for each state. .. autoclass:: util.bead_fp.BeadFpUtil :members: CPFUtil ------- This class contains CPF (Capital Projects Fund) pre/post-challenge configurations. .. autoclass:: util.cpf.CPFUtil :members: TileUtil -------- This class contains tiling and TileserverGL utility functions for interacting with tippecanoe and tile servers. .. autoclass:: util.tile.TileUtil :members: Outbound Summary Workbook Builder --------------------------------- Builds state-specific cumulative outbound-payment workbooks from canonical templates. .. automodule:: util.financial_integration.outbound_summary :members: Configs ========= The following are configuration objects that are used throughout the project. They are organized by the type of configuration they provide. FipsCodes --------- This class contains FIPS codes for each state and territory, as well as various frequently used combinations. .. autoclass:: config.fips.FipsCodes :members: ProductionConfig ---------------- This class contains configuration for production. This should not be overridden on production, only on development. .. autoclass:: config.production.ProductionConfig :members: LocalConfig ----------- This class contains configuration for local development. This should not be overridden on production, only on development. .. autoclass:: config.development.LocalConfig :members: Tags ---- This class contains tags that are used throughout the project. Don't use magic strings in your tags - use these. .. autoclass:: config.tags.Tags :members: S3Prefixes ---------- This class contains S3 path prefixes used throughout the project (curated, raw, transformed, etc.). .. autoclass:: config.prefixes.S3Prefixes :members: RADPrefixes ----------- This class contains RAD (Ready Address Discovery) table and queue prefixes. .. autoclass:: config.prefixes.RADPrefixes :members: StateName --------- This class contains state and territory name mappings (state code to full name). .. autoclass:: config.statename.StateName :members: ISO3166 ------- This class contains ISO 3166 country codes and names. .. autoclass:: config.countries.ISO3166 :members: SQLReference ------------ This class contains SQL file references for DAGs. Generated by `dev_tools/generate_sql_file_class.sh`. .. autoclass:: config.sqlreference.SQLReference :members: .. toctree:: :maxdepth: 2 :caption: Contents: Indices and tables ================== * :ref:`genindex` * :ref:`search`