AWS.OSIS reference

GetPipeline

Source: src/AWS/OSIS/GetPipeline.ts

Runtime binding for osis:GetPipeline.

Reads the bound Pipeline’s live detail — status, capacity, ingest endpoint URLs, and destinations — so an ops function can health-check the pipeline or discover its ingest endpoints at runtime. The pipeline name is injected from the binding. Provide the implementation with Effect.provide(AWS.OSIS.GetPipelineHttp).

GetPipeline: Monitoring a Pipeline

// init — bind the operation to the pipeline
const getPipeline = yield* AWS.OSIS.GetPipeline(pipeline);

// runtime
const { Pipeline } = yield* getPipeline();
if (Pipeline?.Status !== "ACTIVE") {
  yield* Effect.logWarning(`pipeline is ${Pipeline?.Status}`);
}

GetPipelineBlueprint

Source: src/AWS/OSIS/GetPipelineBlueprint.ts

Runtime binding for osis:GetPipelineBlueprint.

Retrieves one Data Prepper blueprint’s full configuration template so config-authoring tooling can render or specialize it (pair with ListPipelineBlueprints to discover blueprint names). Account-level: no resource argument. Provide the implementation with Effect.provide(AWS.OSIS.GetPipelineBlueprintHttp).

GetPipelineBlueprint: Authoring Pipeline Configurations

// init — account-level binding, no resource argument
const getPipelineBlueprint = yield* AWS.OSIS.GetPipelineBlueprint();

// runtime
const { Blueprint } = yield* getPipelineBlueprint({
  BlueprintName: "AWS-ApacheLogPipeline",
});
// Blueprint?.PipelineConfigurationBody — the YAML template

GetPipelineChangeProgress

Source: src/AWS/OSIS/GetPipelineChangeProgress.ts

Runtime binding for osis:GetPipelineChangeProgress.

Reads progress information for the current change happening on the bound Pipeline (stage-by-stage status while the pipeline is being created) — useful for surfacing provisioning progress in an operational dashboard. The pipeline name is injected from the binding. Provide the implementation with Effect.provide(AWS.OSIS.GetPipelineChangeProgressHttp).

GetPipelineChangeProgress: Monitoring a Pipeline

// init — bind the operation to the pipeline
const getChangeProgress = yield* AWS.OSIS.GetPipelineChangeProgress(pipeline);

// runtime
const { ChangeProgressStatuses } = yield* getChangeProgress();
for (const status of ChangeProgressStatuses ?? []) {
  yield* Effect.log(`${status.Status}: ${status.ChangeProgressStages?.length} stages`);
}

Ingest

Source: src/AWS/OSIS/Ingest.ts

Runtime binding for the osis:Ingest data plane.

Sends events into the bound Pipeline’s ingest endpoint — the pipeline’s actual data plane. There is no SDK operation for ingestion; each batch is a SigV4-signed HTTP POST (service "osis") to the pipeline’s ingest endpoint URL, made with the host Function’s own credentials. The endpoint and pipeline name are injected from the binding; pass the source path configured on the pipeline’s http source and the batch of events. Provide the implementation with Effect.provide(AWS.OSIS.IngestHttp).

Ingest: Ingesting Data

// init — bind the data plane to the pipeline
const ingest = yield* AWS.OSIS.Ingest(pipeline);

// runtime — path must match the pipeline config's http source path
yield* ingest({
  path: "/logs/ingest",
  events: [{ message: "hello", level: "info" }],
});

ListPipelineBlueprints

Source: src/AWS/OSIS/ListPipelineBlueprints.ts

Runtime binding for osis:ListPipelineBlueprints.

Lists the available Data Prepper blueprints — AWS-provided configuration templates for common source/sink topologies — so config-authoring tooling can offer them as starting points. Account-level: no resource argument. Provide the implementation with Effect.provide(AWS.OSIS.ListPipelineBlueprintsHttp).

ListPipelineBlueprints: Authoring Pipeline Configurations

// init — account-level binding, no resource argument
const listPipelineBlueprints = yield* AWS.OSIS.ListPipelineBlueprints();

// runtime
const { Blueprints } = yield* listPipelineBlueprints();
// [{ BlueprintName: "AWS-CloudTrailLogsToOpenSearch", ... }, ...]

ListPipelineEndpointConnections

Source: src/AWS/OSIS/ListPipelineEndpointConnections.ts

Runtime binding for osis:ListPipelineEndpointConnections.

Lists the VPC endpoint connections attached to pipelines in the account — including endpoints owned by other accounts — so an ops function can audit who can ingest into your pipelines (pair with RevokePipelineEndpointConnections to cut off access). Account-level: no resource argument. Provide the implementation with Effect.provide(AWS.OSIS.ListPipelineEndpointConnectionsHttp).

ListPipelineEndpointConnections: Managing Endpoint Connections

// init — account-level binding, no resource argument
const listConnections = yield* AWS.OSIS.ListPipelineEndpointConnections();

// runtime
const { PipelineEndpointConnections } = yield* listConnections();
for (const connection of PipelineEndpointConnections ?? []) {
  yield* Effect.log(
    `${connection.EndpointId} (owner ${connection.VpcEndpointOwner})`,
  );
}

Pipeline

Source: src/AWS/OSIS/Pipeline.ts

An Amazon OpenSearch Ingestion (OSIS) pipeline — a managed Data Prepper pipeline that ingests, transforms, and delivers data to OpenSearch domains, serverless collections, or S3.

Pipelines take roughly 5-10 minutes to provision and are billed per Ingestion-OCU-hour while they exist (minimum 1 OCU). Destroy pipelines you are not using.

Pipeline: Creating a Pipeline

HTTP Source to S3 Sink

const pipeline = yield* Pipeline("Logs", {
  minUnits: 1,
  maxUnits: 1,
  pipelineConfigurationBody: Output.interpolate`version: "2"
log-pipeline:
  source:
    http:
      path: "/logs"
  sink:
    - s3:
        aws:
          sts_role_arn: "${role.roleArn}"
          region: "us-west-2"
        bucket: "${bucket.bucketName}"
        threshold:
          event_collect_timeout: "60s"
        codec:
          ndjson:
`,
});

Pipeline with CloudWatch Logging

const pipeline = yield* Pipeline("Logs", {
  minUnits: 1,
  maxUnits: 2,
  pipelineConfigurationBody: configYaml,
  logPublishingOptions: {
    isLoggingEnabled: true,
    cloudWatchLogDestination: {
      logGroup: "/aws/vendedlogs/OpenSearchIngestion/logs",
    },
  },
});

PipelineEndpoint

Source: src/AWS/OSIS/PipelineEndpoint.ts

A VPC endpoint for an Amazon OpenSearch Ingestion (OSIS) pipeline — lets clients inside a VPC ingest data into a pipeline privately, without traversing the public ingest endpoint.

All properties are create-only; any change replaces the endpoint. The endpoint id is assigned by OSIS on create.

PipelineEndpoint: Creating a Pipeline Endpoint

const endpoint = yield* OSIS.PipelineEndpoint("Private", {
  pipelineArn: pipeline.pipelineArn,
  vpcOptions: {
    subnetIds: [subnet.subnetId],
    securityGroupIds: [securityGroup.securityGroupId],
  },
});
// endpoint.ingestEndpointUrl — the VPC-private ingest URL

ResourcePolicy

Source: src/AWS/OSIS/ResourcePolicy.ts

The resource-based policy of an Amazon OpenSearch Ingestion (OSIS) pipeline — grants cross-account principals access to the pipeline, e.g. osis:Ingest for cross-account ingestion or osis:CreatePipelineEndpoint so another account can attach a VPC endpoint. A resource has at most one.

ResourcePolicy: Creating a Resource Policy

const policy = yield* OSIS.ResourcePolicy("CrossAccountIngest", {
  resourceArn: pipeline.pipelineArn,
  policy: Output.interpolate`{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": { "AWS": "arn:aws:iam::123456789012:root" },
        "Action": ["osis:Ingest"],
        "Resource": "${pipeline.pipelineArn}"
      }
    ]
  }`,
});

RevokePipelineEndpointConnections

Source: src/AWS/OSIS/RevokePipelineEndpointConnections.ts

Runtime binding for osis:RevokePipelineEndpointConnections.

Revokes VPC endpoint connections attached to the bound Pipeline — a security control that cuts off ingest access from specific endpoints (e.g. after an account is offboarded). The pipeline ARN is injected from the binding; pass the endpoint IDs to revoke. Provide the implementation with Effect.provide(AWS.OSIS.RevokePipelineEndpointConnectionsHttp).

RevokePipelineEndpointConnections: Managing Endpoint Connections

// init — bind the operation to the pipeline
const revokeConnections =
  yield* AWS.OSIS.RevokePipelineEndpointConnections(pipeline);

// runtime
yield* revokeConnections({ EndpointIds: ["pe-1234567890abcdef0"] });

StartPipeline

Source: src/AWS/OSIS/StartPipeline.ts

Runtime binding for osis:StartPipeline.

Starts the bound Pipeline after it was stopped — e.g. resuming ingestion on a schedule or in response to an operational signal. Starting takes several minutes; the response reports the transitional STARTING status. The pipeline name is injected from the binding. Provide the implementation with Effect.provide(AWS.OSIS.StartPipelineHttp).

StartPipeline: Controlling a Pipeline

// init — bind the operation to the pipeline
const startPipeline = yield* AWS.OSIS.StartPipeline(pipeline);

// runtime
const { Pipeline } = yield* startPipeline();
// Pipeline?.Status === "STARTING"

StopPipeline

Source: src/AWS/OSIS/StopPipeline.ts

Runtime binding for osis:StopPipeline.

Stops the bound Pipeline — ingestion halts and OCU billing stops (persistent-buffer storage is retained) — e.g. pausing a non-production pipeline outside working hours. The pipeline name is injected from the binding. Provide the implementation with Effect.provide(AWS.OSIS.StopPipelineHttp).

StopPipeline: Controlling a Pipeline

// init — bind the operation to the pipeline
const stopPipeline = yield* AWS.OSIS.StopPipeline(pipeline);

// runtime
const { Pipeline } = yield* stopPipeline();
// Pipeline?.Status === "STOPPING"

ValidatePipeline

Source: src/AWS/OSIS/ValidatePipeline.ts

Runtime binding for osis:ValidatePipeline.

Checks whether a Data Prepper pipeline configuration body is valid prior to creation — the building block of config-authoring tooling (validate a generated configuration before calling CreatePipeline, or lint user-submitted configs in a self-service portal). Account-level: no resource argument. Provide the implementation with Effect.provide(AWS.OSIS.ValidatePipelineHttp).

ValidatePipeline: Authoring Pipeline Configurations

// init — account-level binding, no resource argument
const validatePipeline = yield* AWS.OSIS.ValidatePipeline();

// runtime
const { isValid, Errors } = yield* validatePipeline({
  PipelineConfigurationBody: configYaml,
});
if (!isValid) {
  yield* Effect.logError(Errors?.map((e) => e.Message).join("\n") ?? "");
}