Skip to main content
Version: 5.0.x

Manage packages

tip

This page only shows some frequently used operations.

  • For the latest and complete information about Pulsar admin, including commands, flags, descriptions, and more, see Pulsar admin doc.

  • For the latest and complete information about REST API, including parameters, responses, samples, and more, see REST API doc.

  • For the latest and complete information about Java admin API, including classes, methods, descriptions, and more, see Java admin API doc.

Package managers or package-management systems automatically manage packages in a consistent manner. These tools simplify the installation tasks, upgrade process, and deletion operations for users. A package is a minimal unit that a package manager deals with. In Pulsar, packages are organized at the tenant-level and namespace-level to manage Pulsar Functions and Pulsar IO connectors (i.e., source and sink).

What is a package?​

A package is a set of elements that the user would like to reuse in later operations. In Pulsar, a package can be a group of functions, sources, and sinks. You can define a package according to your needs.

The package management system in Pulsar stores the data and metadata of each package (as shown in the table below) and tracks the package versions.

MetadataDescription
descriptionThe description of the package.
contactThe contact information of a package. For example, an email address of the developer team.
create_timeThe time when the package is created.
modification_timeThe time when the package is lastly modified.
propertiesA user-defined key/value map to store other information.

How to use a package​

Packages can efficiently use the same set of functions and IO connectors. For example, you can use the same function, source, and sink in multiple namespaces. The main steps are:

  1. Create a package in the package manager by providing the following information: type, tenant, namespace, package name, and version.

    ComponentDescription
    typeSpecify one of the supported package types: function, sink and source.
    tenantSpecify the tenant where you want to create the package.
    namespaceSpecify the namespace where you want to create the package.
    nameSpecify the complete name of the package, using the format <tenant>/<namespace>/<package name>.
    versionSpecify a version string, such as 1.0 or v0.1. Numeric major/minor formatting is not required.

    The information you provide creates a URL for a package, in the format <type>://<tenant>/<namespace>/<package name>@<version>.

    Pulsar validates each tenant, namespace, package name, and version as a single storage-path component. Components cannot be . or .., contain /, \, or a NUL character, or be empty. An omitted or empty version resolves to latest before validation. Check existing package names and automation for these restrictions before upgrading.

  2. Upload the elements to the package, i.e., the functions, sources, and sinks that you want to use across namespaces.

  3. Apply permissions to this package from various namespaces.

Now, you can use the elements you defined in the package by calling this package from within the package manager. The package manager locates it by the URL. For example,

sink://public/default/mysql-sink@1.0
function://my-tenant/my-ns/my-function@0.1
source://my-tenant/my-ns/mysql-cdc-source@2.3

Metadata format and upgrades​

The package-management service stores metadata for versioned Pulsar Functions and IO connector packages (function://, source://, and sink:// URLs). These records contain the package description, contact, timestamps, and custom properties. Package management is disabled by default (enablePackagesManagement=false); the following guidance applies when you enable it.

Pulsar stores this package metadata as JSON by default (packagesManagementJsonSerializationEnabled=true), replacing the previous default of Java-based serialization. With packagesManagementAllowLegacyJavaSerialization=true (the default), it can read Java-serialized metadata from Pulsar 4.0.x and 4.2.x through a restricted deserialization filter. These settings concern the package's metadata record, not the uploaded JAR, NAR, or other package payload.

Uploading a package or updating its metadata writes the configured format. Changing the write-format setting does not convert existing records. If you need rollback to 4.x, set packagesManagementJsonSerializationEnabled=false before upgrading and retain legacy reads. Keep Java writes throughout mixed-version operation and the rollback window; 4.x cannot read JSON metadata. See package-management rollback guidance.

The default broker.conf does not list these serialization settings. For Pulsar Helm chart deployments, set the environment variables in broker.configData using PULSAR_PREFIX_packagesManagementJsonSerializationEnabled=false and PULSAR_PREFIX_packagesManagementAllowLegacyJavaSerialization=true so the configuration helper adds the missing keys. See the Helm example in the upgrade guide.

After the old package-metadata readers have been retired and the rollback window has closed, you can migrate retained metadata to JSON:

  1. Ensure every package-management broker uses JSON writes and still allows legacy reads.
  2. List the packages and versions that must remain usable. Read each Java record and submit its complete metadata to the REST metadata-update endpoint or Java admin updateMetadata method through a broker configured for JSON writes. The update replaces the stored record without deserializing it first and leaves the package payload in place. Supply the complete description, contact, properties, and createTime that must be retained; the server sets modificationTime to the update time. An omitted field is not preserved. The CLI update-metadata command supplies description, contact, and properties, but does not provide an option to retain createTime.
  3. Verify that all retained versions can be inspected and downloaded, including versions still referenced by Functions or connectors.
  4. Once no required legacy records remain and the rollback window is closed, set packagesManagementAllowLegacyJavaSerialization=false on every package-management broker and restart them. Do not disable legacy reads while writes are configured to use the legacy format.

The write-format and legacy-read flags are startup settings. Turning off legacy reads rejects remaining Java-serialized records; it does not migrate them. Keep a record of which package versions were rewritten so a rollback plan accounts for metadata compatibility as well as broker binaries.

Package management in Pulsar​

You can use the command line tools, REST API, or the Java client to manage your package resources in Pulsar. More specifically, you can use these tools to upload, download, and delete a package, get the metadata and update the metadata of a package, get the versions of a package, and get all packages of a specific type under a namespace.

To use package management service, ensure that the package management service has been enabled in your cluster by setting the following properties in broker.conf.

note

Package management service is not enabled by default.

enablePackagesManagement=true
packagesManagementStorageProvider=org.apache.pulsar.packages.management.storage.bookkeeper.BookKeeperPackagesStorageProvider
packagesReplicas=1
packagesManagementLedgerRootPath=/ledgers

Upload a package​

You can use the following commands to upload a package.

bin/pulsar-admin packages upload function://public/default/example@v0.1 --path package-file --description package-description

Download a package​

You can use the following commands to download a package.

bin/pulsar-admin packages download function://public/default/example@v0.1 --path package-file

Delete a package​

You can use the following commands to delete a package.

The following command deletes a package of version 0.1.

bin/pulsar-admin packages delete function://public/default/example@v0.1

Get the metadata of a package​

You can use the following commands to get the metadata of a package.

bin/pulsar-admin packages get-metadata function://public/default/test@v1

Update the metadata of a package​

You can use the following commands to update the metadata of a package.

bin/pulsar-admin packages update-metadata function://public/default/example@v0.1 --description update-description

List all versions of a package​

You can use the following commands to list all versions of a package.

bin/pulsar-admin packages list-versions type://tenant/namespace/packageName

List all packages of a specific type under a namespace​

You can use the following commands to list all packages of a specific type under a namespace.

bin/pulsar-admin packages list --type function public/default
Was this helpful?
💡 Suggest changes🛟 Get support