AWS.Glue reference

BatchCreatePartition

Source: src/AWS/Glue/BatchCreatePartition.ts

Runtime binding for glue:BatchCreatePartition.

Registers up to 100 partitions on the bound Table in one call — the bulk variant of CreatePartition for backfills. Per-partition failures come back in the response’s Errors list. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.BatchCreatePartitionHttp).

BatchCreatePartition: Managing Partitions

// init
const batchCreatePartition = yield* AWS.Glue.BatchCreatePartition(table);

// runtime
const { Errors } = yield* batchCreatePartition({
  PartitionInputList: days.map((dt) => ({ Values: [dt] })),
});

BatchDeletePartition

Source: src/AWS/Glue/BatchDeletePartition.ts

Runtime binding for glue:BatchDeletePartition.

Deletes up to 25 partitions of the bound Table in one call — the bulk variant of DeletePartition for retention sweeps. Per-partition failures come back in the response’s Errors list. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.BatchDeletePartitionHttp).

BatchDeletePartition: Managing Partitions

// init
const batchDeletePartition = yield* AWS.Glue.BatchDeletePartition(table);

// runtime
const { Errors } = yield* batchDeletePartition({
  PartitionsToDelete: expired.map((dt) => ({ Values: [dt] })),
});

BatchGetPartition

Source: src/AWS/Glue/BatchGetPartition.ts

Runtime binding for glue:BatchGetPartition.

Reads up to 1000 partitions of the bound Table by their partition values in one call — the bulk variant of GetPartition. Values that don’t resolve come back in UnprocessedKeys. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.BatchGetPartitionHttp).

BatchGetPartition: Managing Partitions

// init
const batchGetPartition = yield* AWS.Glue.BatchGetPartition(table);

// runtime
const { Partitions } = yield* batchGetPartition({
  PartitionsToGet: [{ Values: ["2026-01-01"] }, { Values: ["2026-01-02"] }],
});

BatchStopJobRun

Source: src/AWS/Glue/BatchStopJobRun.ts

Runtime binding for glue:BatchStopJobRun.

Stops one or more in-flight runs of the bound Job. Per-run failures come back in the response’s Errors list (the call itself succeeds), so inspect SuccessfulSubmissions/Errors rather than the error channel. The job name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.BatchStopJobRunHttp).

BatchStopJobRun: Running Jobs

// init
const batchStopJobRun = yield* AWS.Glue.BatchStopJobRun(job);

// runtime
const { SuccessfulSubmissions, Errors } = yield* batchStopJobRun({
  JobRunIds: [runId],
});

BatchUpdatePartition

Source: src/AWS/Glue/BatchUpdatePartition.ts

Runtime binding for glue:BatchUpdatePartition.

Rewrites up to 100 partitions of the bound Table in one call — the bulk variant of UpdatePartition. Per-partition failures come back in the response’s Errors list. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.BatchUpdatePartitionHttp).

BatchUpdatePartition: Managing Partitions

// init
const batchUpdatePartition = yield* AWS.Glue.BatchUpdatePartition(table);

// runtime
const { Errors } = yield* batchUpdatePartition({
  Entries: [
    {
      PartitionValueList: ["2026-01-01"],
      PartitionInput: {
        Values: ["2026-01-01"],
        Parameters: { compacted: "true" },
      },
    },
  ],
});

Connection

Source: src/AWS/Glue/Connection.ts

An AWS Glue connection — stores the connection details (JDBC URL, VPC networking, credentials) that crawlers and jobs use to reach a data store.

Connection: Creating Connections

import * as AWS from "alchemy/AWS";
import * as Redacted from "effect/Redacted";

const connection = yield* AWS.Glue.Connection("Warehouse", {
  connectionType: "JDBC",
  connectionProperties: {
    JDBC_CONNECTION_URL: "jdbc:postgresql://db.example.com:5432/warehouse",
    USERNAME: "glue",
    PASSWORD: Redacted.make("secret"),
  },
  physicalConnectionRequirements: {
    subnetId: subnet.subnetId,
    securityGroupIdList: [securityGroup.groupId],
    availabilityZone: "us-west-2a",
  },
});

Crawler

Source: src/AWS/Glue/Crawler.ts

An AWS Glue crawler — connects to an S3 (or JDBC/DynamoDB/catalog) data store, infers schemas, and populates the Glue Data Catalog with tables. Runs are asynchronous: create the crawler, then invoke startCrawler (or attach a schedule).

Crawler: Creating Crawlers

S3 Crawler

import * as AWS from "alchemy/AWS";

const database = yield* AWS.Glue.Database("Analytics", {
  databaseName: "analytics",
});

const crawler = yield* AWS.Glue.Crawler("EventsCrawler", {
  role: crawlerRole.roleArn,
  databaseName: database.databaseName,
  targets: {
    s3Targets: [{ path: "s3://my-data-lake/events/" }],
  },
});

Scheduled Crawler with Schema Policy

const crawler = yield* AWS.Glue.Crawler("EventsCrawler", {
  role: crawlerRole.roleArn,
  databaseName: database.databaseName,
  targets: { s3Targets: [{ path: "s3://my-data-lake/events/" }] },
  schedule: "cron(0 12 * * ? *)",
  tablePrefix: "raw_",
  schemaChangePolicy: {
    updateBehavior: "UPDATE_IN_DATABASE",
    deleteBehavior: "DEPRECATE_IN_DATABASE",
  },
});

CreatePartition

Source: src/AWS/Glue/CreatePartition.ts

Runtime binding for glue:CreatePartition.

Registers a new partition on the bound Table — the write half of the classic “data landed in S3, register the partition so Athena sees it” pipeline. Fails with the typed AlreadyExistsException if the partition is already registered. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.CreatePartitionHttp).

CreatePartition: Managing Partitions

// init
const createPartition = yield* AWS.Glue.CreatePartition(table);

// runtime — idempotent registration
yield* createPartition({
  PartitionInput: {
    Values: ["2026-01-01"],
    StorageDescriptor: { Location: "s3://my-data-lake/events/dt=2026-01-01/" },
  },
}).pipe(Effect.catchTag("AlreadyExistsException", () => Effect.void));

Database

Source: src/AWS/Glue/Database.ts

An AWS Glue Data Catalog database — the top-level container for Glue tables that Athena, EMR, Redshift Spectrum, and Glue jobs query. Databases are free and instant to create.

Database: Creating Databases

Basic Database

import * as AWS from "alchemy/AWS";

const database = yield* AWS.Glue.Database("Analytics", {
  databaseName: "analytics",
});

Database with a Default S3 Location

const database = yield* AWS.Glue.Database("Analytics", {
  databaseName: "analytics",
  description: "Curated analytics tables",
  locationUri: "s3://my-data-lake/analytics/",
  parameters: { classification: "parquet" },
});

DeletePartition

Source: src/AWS/Glue/DeletePartition.ts

Runtime binding for glue:DeletePartition.

Deregisters one partition of the bound Table by its partition values (the underlying data is untouched). Fails with the typed EntityNotFoundException when the partition is already gone. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.DeletePartitionHttp).

DeletePartition: Managing Partitions

// init
const deletePartition = yield* AWS.Glue.DeletePartition(table);

// runtime — idempotent removal
yield* deletePartition({ PartitionValues: ["2026-01-01"] }).pipe(
  Effect.catchTag("EntityNotFoundException", () => Effect.void),
);

GetCrawler

Source: src/AWS/Glue/GetCrawler.ts

Runtime binding for glue:GetCrawler.

Reads the bound Crawler’s metadata — most usefully its State (READY, RUNNING, STOPPING) and LastCrawl outcome — so runtime code can poll a crawl started with StartCrawler to completion. The crawler name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetCrawlerHttp).

GetCrawler: Running Crawlers

// init
const startCrawler = yield* AWS.Glue.StartCrawler(crawler);
const getCrawler = yield* AWS.Glue.GetCrawler(crawler);

// runtime
yield* startCrawler();
const done = yield* getCrawler().pipe(
  Effect.repeat({
    schedule: Schedule.spaced("10 seconds"),
    until: (r) => r.Crawler?.State === "READY",
    times: 8,
  }),
);

GetJobBookmark

Source: src/AWS/Glue/GetJobBookmark.ts

Runtime binding for glue:GetJobBookmark.

Reads the bound Job’s bookmark entry — the incremental-processing checkpoint Glue keeps when a job runs with --job-bookmark-option job-bookmark-enable. Fails with the typed EntityNotFoundException when the job has never recorded a bookmark. The job name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetJobBookmarkHttp).

GetJobBookmark: Job Bookmarks

// init
const getJobBookmark = yield* AWS.Glue.GetJobBookmark(job);

// runtime
const entry = yield* getJobBookmark().pipe(
  Effect.map((r) => r.JobBookmarkEntry),
  Effect.catchTag("EntityNotFoundException", () =>
    Effect.succeed(undefined),
  ),
);

GetJobRun

Source: src/AWS/Glue/GetJobRun.ts

Runtime binding for glue:GetJobRun.

Reads the metadata of a single run of the bound Job — its JobRunState (RUNNING, SUCCEEDED, FAILED, TIMEOUT, …), timings, and error message — so a function can poll a run it started to a terminal state. The job name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetJobRunHttp).

GetJobRun: Running Jobs

// init
const getJobRun = yield* AWS.Glue.GetJobRun(job);

// runtime
const { JobRun } = yield* getJobRun({ RunId: runId });
if (JobRun?.JobRunState === "FAILED") {
  yield* Effect.logError(JobRun.ErrorMessage ?? "run failed");
}

GetJobRuns

Source: src/AWS/Glue/GetJobRuns.ts

Runtime binding for glue:GetJobRuns.

Lists the runs of the bound Job (newest first, paginated via NextToken), so a function can report run history or find in-flight runs. The job name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetJobRunsHttp).

GetJobRuns: Running Jobs

// init
const getJobRuns = yield* AWS.Glue.GetJobRuns(job);

// runtime
const { JobRuns } = yield* getJobRuns({ MaxResults: 10 });
const running = (JobRuns ?? []).filter(
  (run) => run.JobRunState === "RUNNING",
);

GetPartition

Source: src/AWS/Glue/GetPartition.ts

Runtime binding for glue:GetPartition.

Reads a single partition of the bound Table by its partition values. Fails with the typed EntityNotFoundException when the partition does not exist. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetPartitionHttp).

GetPartition: Managing Partitions

// init
const getPartition = yield* AWS.Glue.GetPartition(table);

// runtime
const { Partition } = yield* getPartition({
  PartitionValues: ["2026-01-01"],
});

GetPartitions

Source: src/AWS/Glue/GetPartitions.ts

Runtime binding for glue:GetPartitions.

Lists the bound Table’s partitions, optionally filtered with a partition-predicate Expression (e.g. dt >= '2026-01-01') and paginated via NextToken. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetPartitionsHttp).

GetPartitions: Managing Partitions

// init
const getPartitions = yield* AWS.Glue.GetPartitions(table);

// runtime
const { Partitions } = yield* getPartitions({
  Expression: "dt >= '2026-01-01'",
});

GetTable

Source: src/AWS/Glue/GetTable.ts

Runtime binding for glue:GetTable.

Reads the bound Table’s full catalog definition — columns, storage descriptor, partition keys, and parameters — so a function can introspect the schema it is writing against. The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetTableHttp).

GetTable: Reading the Data Catalog

// init
const getTable = yield* AWS.Glue.GetTable(table);

// runtime
const { Table } = yield* getTable();
const columns = Table?.StorageDescriptor?.Columns ?? [];

GetTables

Source: src/AWS/Glue/GetTables.ts

Runtime binding for glue:GetTables.

Lists the table definitions of the bound Database (optionally filtered by an Expression pattern, paginated via NextToken) — runtime schema discovery over the Data Catalog. The database name and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.GetTablesHttp).

GetTables: Reading the Data Catalog

// init
const getTables = yield* AWS.Glue.GetTables(database);

// runtime
const { TableList } = yield* getTables({ Expression: "events_*" });
const names = (TableList ?? []).map((t) => t.Name);

Job

Source: src/AWS/Glue/Job.ts

An AWS Glue job — a Spark (glueetl), Python shell (pythonshell), or streaming ETL job definition (script in S3 + IAM role + arguments). The definition lifecycle is instant and free; job runs are billed and are started via startJobRun.

Job: Creating Jobs

Python Shell Job

import * as AWS from "alchemy/AWS";

const job = yield* AWS.Glue.Job("Etl", {
  role: jobRole.roleArn,
  command: {
    name: "pythonshell",
    pythonVersion: "3.9",
    scriptLocation: "s3://my-bucket/scripts/etl.py",
  },
  maxCapacity: 0.0625,
  glueVersion: "3.0",
  defaultArguments: { "--job-language": "python" },
});

Spark ETL Job

const job = yield* AWS.Glue.Job("SparkEtl", {
  role: jobRole.roleArn,
  command: {
    name: "glueetl",
    scriptLocation: "s3://my-bucket/scripts/spark.py",
  },
  glueVersion: "4.0",
  workerType: "G.1X",
  numberOfWorkers: 2,
  timeout: "1 hour",
});

Job: Running Jobs

// init
const startJobRun = yield* AWS.Glue.StartJobRun(job);

// runtime
const { JobRunId } = yield* startJobRun({});

ResetJobBookmark

Source: src/AWS/Glue/ResetJobBookmark.ts

Runtime binding for glue:ResetJobBookmark.

Resets the bound Job’s bookmark entry so the next bookmarked run reprocesses the source data from the beginning — the standard remediation after a bad deploy consumed data incorrectly. The job name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.ResetJobBookmarkHttp).

ResetJobBookmark: Job Bookmarks

// init
const resetJobBookmark = yield* AWS.Glue.ResetJobBookmark(job);

// runtime
yield* resetJobBookmark();

StartCrawler

Source: src/AWS/Glue/StartCrawler.ts

Runtime binding for glue:StartCrawler.

Starts a crawl of the bound Crawler on demand — the standard way to refresh the Data Catalog right after new data lands. Fails with the typed CrawlerRunningException if a crawl is already in progress. The crawler name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.StartCrawlerHttp).

StartCrawler: Running Crawlers

// init
const startCrawler = yield* AWS.Glue.StartCrawler(crawler);

// runtime — tolerate an already-running crawl
yield* startCrawler().pipe(
  Effect.catchTag("CrawlerRunningException", () => Effect.void),
);

StartJobRun

Source: src/AWS/Glue/StartJobRun.ts

Starts a run of a Glue job definition.

Grants glue:StartJobRun and glue:GetJobRun on the bound job. Returns the JobRunId; poll getJobRun for the run’s terminal state (SUCCEEDED, FAILED, TIMEOUT, …).

StartJobRun: Running Jobs

Provide the StartJobRunHttp implementation layer on the Function effect (.pipe(Effect.provide(AWS.Glue.StartJobRunHttp))), bind the job in the init phase, then start runs at runtime.

// init
const startJobRun = yield* AWS.Glue.StartJobRun(job);

return {
  fetch: Effect.gen(function* () {
    // runtime
    const { JobRunId } = yield* startJobRun({
      Arguments: { "--input": "s3://my-bucket/input/" },
    });
    return HttpServerResponse.json({ JobRunId });
  }),
};

StopCrawler

Source: src/AWS/Glue/StopCrawler.ts

Runtime binding for glue:StopCrawler.

Stops an in-progress crawl of the bound Crawler. Fails with the typed CrawlerNotRunningException when idle and CrawlerStoppingException when a stop is already underway. The crawler name is injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.StopCrawlerHttp).

StopCrawler: Running Crawlers

// init
const stopCrawler = yield* AWS.Glue.StopCrawler(crawler);

// runtime — tolerate an idle or already-stopping crawler
yield* stopCrawler().pipe(
  Effect.catchTag(
    ["CrawlerNotRunningException", "CrawlerStoppingException"],
    () => Effect.void,
  ),
);

Table

Source: src/AWS/Glue/Table.ts

An AWS Glue Data Catalog table — a schema (columns), storage location, and SerDe over data in S3 (or another store). This is the unit Athena, Redshift Spectrum, and EMR query; it is the analytics foundation of a Glue database.

Table: Creating Tables

import * as AWS from "alchemy/AWS";

const database = yield* AWS.Glue.Database("Analytics", {
  databaseName: "analytics",
});

const events = yield* AWS.Glue.Table("Events", {
  databaseName: database.databaseName,
  tableName: "events",
  tableType: "EXTERNAL_TABLE",
  storageDescriptor: {
    location: "s3://my-data-lake/events/",
    inputFormat:
      "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
    outputFormat:
      "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
    serdeInfo: {
      serializationLibrary:
        "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
    },
    columns: [
      { name: "id", type: "string" },
      { name: "amount", type: "double" },
    ],
  },
  partitionKeys: [{ name: "dt", type: "string" }],
  parameters: { classification: "parquet" },
});

UpdatePartition

Source: src/AWS/Glue/UpdatePartition.ts

Runtime binding for glue:UpdatePartition.

Replaces the definition of one partition of the bound Table — PartitionValueList addresses the existing partition and PartitionInput is its new definition (location, parameters, schema). The database/table names and catalog id are injected from the binding. Provide the implementation with Effect.provide(AWS.Glue.UpdatePartitionHttp).

UpdatePartition: Managing Partitions

// init
const updatePartition = yield* AWS.Glue.UpdatePartition(table);

// runtime
yield* updatePartition({
  PartitionValueList: ["2026-01-01"],
  PartitionInput: {
    Values: ["2026-01-01"],
    StorageDescriptor: { Location: "s3://my-data-lake/v2/dt=2026-01-01/" },
  },
});