tap-postgres - Meltano Hub

PostgreSQL

tap-postgres (transferwise variant)🥈

PostgreSQL database extractor

The tap-postgres extractor pulls data from PostgreSQL that can then be sent to a destination using a loader.

Alternate Implementations

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the tap-postgres extractor to your project using
```
meltano add
```
:

meltano add extractor tap-postgres --variant transferwise

Configure the tap-postgres settings using

meltano config

meltano config tap-postgres set --interactive

Test that extractor settings are valid using

meltano config

meltano config tap-postgres test

Next steps

Follow the remaining steps of the Getting Started guide:

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for tap-postgres may have been automatically set when originally added to the Hub. Please review the capabilities when using this extractor. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this extractor.

This plugin has the following capabilities:

catalog
discover
state

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

The tap-postgres settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

break_at_end_lsn
dbname
default_replication_method
filter_schemas
host
logical_poll_total_seconds
max_run_seconds
password
port
ssl
user

You can also list these settings using

meltano config

with the list subcommand:

meltano config tap-postgres list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Break At End LSN (break_at_end_lsn)

Environment variable: TAP_POSTGRES_BREAK_AT_END_LSN
Default Value: true

Stop running the tap if the newly received lsn is after the max lsn that was detected when the tap started

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set break_at_end_lsn [value]

Database Name (dbname)

Environment variable: TAP_POSTGRES_DBNAME

PostgreSQL database name

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set dbname [value]

Default Replication Method (default_replication_method)

Environment variable: TAP_POSTGRES_DEFAULT_REPLICATION_METHOD

The prefered replication method.

Note:

Key-based incremental (INCREMENTAL) requires configuring a replication_key column within the catalog's stream definitions for each supported stream.
Requirements and setup instructions for log-based incremental (LOG_BASED) are documented here.

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set default_replication_method [value]

Filter Schemas (filter_schemas)

Environment variable: TAP_POSTGRES_FILTER_SCHEMAS

Scan only the specified comma-separated schemas to improve the performance of data extraction

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set filter_schemas [value]

Host (host)

Environment variable: TAP_POSTGRES_HOST
Default Value: localhost

PostgreSQL host

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set host [value]

Logical Poll Total Seconds (logical_poll_total_seconds)

Environment variable: TAP_POSTGRES_LOGICAL_POLL_TOTAL_SECONDS
Default Value: 10800

Stop running the tap when no data received from wal after certain number of seconds

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set logical_poll_total_seconds [value]

Max Run Seconds (max_run_seconds)

Environment variable: TAP_POSTGRES_MAX_RUN_SECONDS
Default Value: 43200

Stop running the tap after certain number of seconds

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set max_run_seconds [value]

Password (password)

Environment variable: TAP_POSTGRES_PASSWORD

PostgreSQL password

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set password [value]

Port (port)

Environment variable: TAP_POSTGRES_PORT
Default Value: 5432

PostgreSQL port

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set port [value]

SSL (ssl)

Environment variable: TAP_POSTGRES_SSL
Default Value: false

Using SSL via postgres sslmode='require' option. If the server does not accept SSL connections or the client certificate is not recognized the connection will fail

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set ssl [value]

User (user)

Environment variable: TAP_POSTGRES_USER

PostgreSQL user

Configure this setting directly using the following Meltano command:

meltano config tap-postgres set user [value]

Troubleshooting

Error: `pg_config executable not found` or `libpq-fe.h: No such file or directory`

This error message indicates that the libpq dependency is missing.

To resolve this, refer to the "Dependencies" section above.

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the tap-postgres extractor to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the