AWS.DataBrew reference

Dataset

Source: src/AWS/DataBrew/Dataset.ts

An AWS Glue DataBrew dataset — a pointer to source data (S3 file/prefix, Glue Data Catalog table, or JDBC query) plus parsing options. The dataset definition itself stores no data and is free; it is consumed by DataBrew projects and jobs.

Dataset: Creating Datasets

CSV Dataset from S3

import * as AWS from "alchemy/AWS";

const dataset = yield* AWS.DataBrew.Dataset("Sales", {
  format: "CSV",
  formatOptions: { csv: { delimiter: ",", headerRow: true } },
  input: {
    s3InputDefinition: {
      bucket: bucket.bucketName,
      key: "raw/sales.csv",
    },
  },
});

JSON Dataset

const dataset = yield* AWS.DataBrew.Dataset("Events", {
  format: "JSON",
  formatOptions: { json: { multiLine: false } },
  input: {
    s3InputDefinition: { bucket: bucket.bucketName, key: "events/" },
  },
});

Dataset: Glue Data Catalog

const dataset = yield* AWS.DataBrew.Dataset("Curated", {
  input: {
    dataCatalogInputDefinition: {
      databaseName: glueDatabase.databaseName,
      tableName: "curated_sales",
    },
  },
});

DescribeJobRun

Source: src/AWS/DataBrew/DescribeJobRun.ts

Runtime binding for databrew:DescribeJobRun — reads the state, timings, and outputs of one run of the bound DataBrew job.

DescribeJobRun: Observing Job Runs

const describeJobRun = yield* AWS.DataBrew.DescribeJobRun(job);

const run = yield* describeJobRun({ RunId: runId });
// run.State: "STARTING" | "RUNNING" | "SUCCEEDED" | ...

Job

Source: src/AWS/DataBrew/Job.ts

An AWS Glue DataBrew job definition — either a PROFILE job that analyzes a dataset and writes a data-quality profile to S3, or a RECIPE job that applies a published recipe’s transformations and writes the result to S3. The definition is free and instant; job runs are billed per node-hour and are started with StartJobRun.

Job: Profile Jobs

import * as AWS from "alchemy/AWS";

const profile = yield* AWS.DataBrew.Job("Profile", {
  type: "PROFILE",
  datasetName: dataset.datasetName,
  role: role.roleArn,
  outputLocation: { bucket: bucket.bucketName, key: "profiles/" },
  jobSample: { mode: "CUSTOM_ROWS", size: 1000 },
});

Job: Recipe Jobs

const transform = yield* AWS.DataBrew.Job("Transform", {
  type: "RECIPE",
  datasetName: dataset.datasetName,
  recipeReference: { name: recipe.recipeName },
  role: role.roleArn,
  outputs: [
    {
      location: { bucket: bucket.bucketName, key: "curated/" },
      format: "CSV",
      overwrite: true,
    },
  ],
});

ListJobRuns

Source: src/AWS/DataBrew/ListJobRuns.ts

Runtime binding for databrew:ListJobRuns — lists the previous runs of the bound DataBrew job (newest first, paginated via NextToken).

ListJobRuns: Observing Job Runs

const listJobRuns = yield* AWS.DataBrew.ListJobRuns(job);

const { JobRuns } = yield* listJobRuns();

Project

Source: src/AWS/DataBrew/Project.ts

An AWS Glue DataBrew project — the interactive workspace binding a dataset to a recipe’s working version. The project definition is free; costs only accrue when an interactive session is started in the console.

Project: Creating Projects

Dataset + Recipe Project

import * as AWS from "alchemy/AWS";

const project = yield* AWS.DataBrew.Project("Explore", {
  datasetName: dataset.datasetName,
  recipeName: recipe.recipeName,
  role: role.roleArn,
});

Custom Sample

const project = yield* AWS.DataBrew.Project("Explore", {
  datasetName: dataset.datasetName,
  recipeName: recipe.recipeName,
  sample: { type: "RANDOM", size: 250 },
  role: role.roleArn,
});

PublishRecipe

Source: src/AWS/DataBrew/PublishRecipe.ts

Runtime binding for databrew:PublishRecipe — snapshots the bound recipe’s LATEST_WORKING steps as the next numbered published version (recipe jobs consume the latest published version by default).

PublishRecipe: Publishing Recipes

const publishRecipe = yield* AWS.DataBrew.PublishRecipe(recipe);

const { Name } = yield* publishRecipe({ Description: "nightly cut" });

Recipe

Source: src/AWS/DataBrew/Recipe.ts

An AWS Glue DataBrew recipe — an ordered list of data-transformation steps (rename, filter, case conversion, etc.). Edits modify the LATEST_WORKING version; setting publish: true snapshots numbered versions that recipe jobs consume.

Recipe: Creating Recipes

Simple Transform Recipe

import * as AWS from "alchemy/AWS";

const recipe = yield* AWS.DataBrew.Recipe("Clean", {
  description: "normalize customer names",
  steps: [
    {
      action: {
        operation: "UPPER_CASE",
        parameters: { sourceColumn: "name" },
      },
    },
  ],
});

Published Recipe for Jobs

// publish: true snapshots a numbered version (1.0, 2.0, ...) whenever the
// steps change — recipe jobs run the latest published version by default.
const recipe = yield* AWS.DataBrew.Recipe("Clean", {
  publish: true,
  steps: [
    {
      action: {
        operation: "REMOVE_VALUES",
        parameters: { sourceColumn: "email" },
      },
      conditionExpressions: [
        { condition: "IS_MISSING", targetColumn: "email" },
      ],
    },
  ],
});

Ruleset

Source: src/AWS/DataBrew/Ruleset.ts

An AWS Glue DataBrew ruleset — a set of data-quality rules bound to a dataset. Attach it to a profile job via validationConfigurations to produce pass/fail validation results alongside the data profile.

Ruleset: Creating Rulesets

Data-Quality Rules for a Dataset

import * as AWS from "alchemy/AWS";

const ruleset = yield* AWS.DataBrew.Ruleset("Quality", {
  targetArn: dataset.datasetArn,
  rules: [
    {
      name: "no-missing-ids",
      checkExpression: "AGG(MISSING_VALUES_PERCENTAGE) == :val1",
      substitutionMap: { ":val1": "0" },
      columnSelectors: [{ name: "id" }],
    },
  ],
});

Validate in a Profile Job

const profile = yield* AWS.DataBrew.Job("Profile", {
  type: "PROFILE",
  datasetName: dataset.datasetName,
  role: role.roleArn,
  outputLocation: { bucket: bucket.bucketName, key: "profiles/" },
  validationConfigurations: [{ rulesetArn: ruleset.rulesetArn }],
});

Schedule

Source: src/AWS/DataBrew/Schedule.ts

An AWS Glue DataBrew schedule — a cron expression that starts one or more DataBrew jobs at recurring times. The schedule definition is free; only the job runs it triggers are billed.

Schedule: Creating Schedules

Nightly Job Schedule

import * as AWS from "alchemy/AWS";

const schedule = yield* AWS.DataBrew.Schedule("Nightly", {
  cronExpression: "cron(0 3 * * ? *)",
  jobNames: [job.jobName],
});

Schedule Without Jobs (attach later)

const schedule = yield* AWS.DataBrew.Schedule("Standing", {
  cronExpression: "cron(0 12 ? * MON-FRI *)",
});

SendProjectSessionAction

Source: src/AWS/DataBrew/SendProjectSessionAction.ts

Runtime binding for databrew:SendProjectSessionAction — performs a recipe step (optionally as a preview) inside an open interactive session on the bound DataBrew project. Authenticate with the Redacted ClientSessionId returned by StartProjectSession.

SendProjectSessionAction: Interactive Sessions

const sendProjectSessionAction =
  yield* AWS.DataBrew.SendProjectSessionAction(project);

const { ActionId } = yield* sendProjectSessionAction({
  Preview: true,
  ClientSessionId: clientSessionId,
  ViewFrame: { StartColumnIndex: 0 },
});

StartJobRun

Source: src/AWS/DataBrew/StartJobRun.ts

Runtime binding for databrew:StartJobRun — lets a workload kick off a run of a DataBrew job (a PROFILE analysis or a RECIPE transformation).

The response carries the RunId, which can be observed with the DescribeJobRun binding and cancelled with StopJobRun.

StartJobRun: Starting Job Runs

const startJobRun = yield* AWS.DataBrew.StartJobRun(job);

const { RunId } = yield* startJobRun();

StartProjectSession

Source: src/AWS/DataBrew/StartProjectSession.ts

Runtime binding for databrew:StartProjectSession — opens an interactive session on the bound DataBrew project. The response’s ClientSessionId (a Redacted session token) authenticates subsequent SendProjectSessionAction calls.

Interactive sessions are billed per 30-minute session.

StartProjectSession: Interactive Sessions

const startProjectSession = yield* AWS.DataBrew.StartProjectSession(project);

const { ClientSessionId } = yield* startProjectSession({
  AssumeControl: true,
});

StopJobRun

Source: src/AWS/DataBrew/StopJobRun.ts

Runtime binding for databrew:StopJobRun — cancels a run of the bound DataBrew job that is still starting or running.

StopJobRun: Stopping Job Runs

const stopJobRun = yield* AWS.DataBrew.StopJobRun(job);

yield* stopJobRun({ RunId: runId });