Debugging

The Debugging section of the Temporal TypeScript SDK developer's guide covers tools for debugging and how to troubleshoot common issues.

How to debug in a development environment

In addition to the normal development tools of logging and a debugger, you can also see what's happening in your Workflow by using the Web UI or Temporal CLI.

How to debug in a production environment

You can debug production Workflows using:

You can debug and tune Worker performance with metrics and the Worker performance guide. For information on setting up SDK metrics, see Metrics in the Observability section of the TypeScript SDK developer's guide.

Debug Server performance with Cloud metrics or self-hosted Server metrics.

How to troubleshoot common issues in the TypeScript SDK

Two locations to watch

Workflow Errors are reflected in Temporal Web.
Worker errors and logs are reflected in the terminal.

If something isn't behaving the way you expect, make sure to check both locations for helpful error messages.

Stale Workflows

If you are developing Workflows and finding that code isn't executing as expected, the first place to look is whether old Workflows are still running.

If those old Workflows have the same name and are on the same task queue, Temporal will try to continue executing them on your new code by design. You may get errors that make no sense to you because

Temporal is trying to execute old Workflow code that no longer exists in your codebase, or
your new Client code is expecting Temporal to execute old Workflow/Activity code it doesn't yet know about.

The biggest sign that this is happening is if you notice Temporal is acting non-deterministically: running the same Workflow twice gets different results.

Stale workflows are usually a non-issue because the errors generated are just noise from code you no longer want to run. If you need to terminate old stale Workflows, you can do so with Temporal Web or the Temporal CLI.

Workflow/Activity registration errors

If your Workflows or Activities are not imported or spelled correctly, here are some errors we've seen:

ApplicationFailure: 'MyFunction' is not a function
Workflow did not register a handler for MyQuery

Double check that your Workers are registering the right Workflow and Activity Definitions (function names) on the right Task Queues.

If you are running Temporal in a monorepo, then your node_modules may be in a different location than where Temporal expects to find it by default, which results in errors like:

[ERROR] Module not found: Error: Can't resolve '@temporalio/workflow/lib/worker-interface.js' in '/src'

Our Next.js tutorial is written for people setting up Temporal within an existing monorepo, which may be of use here.

When you pass a workflowsPath, our Webpack config expects to find node_modules in the same or a parent/ancestor directory.

If you are custom bundling your own Workflows you may get errors like these:

[ERROR] Failed to activate workflow {
  runId: 'aaf84a83-51ce-462a-9ab7-6a641a703bff',
  error: ReferenceError: exports is not defined,
  workflowExists: false
}

Temporal Workflow Bundles need to export a set of methods that fit the compiled worker-interface.ts from @temporalio/workflow as an entry point. We do offer a bundleWorkflowCode method to assist you with this, though it uses our Webpack settings. For more information, see the Register types section.

Webpack errors

The TypeScript SDK's Worker bundles Workflows based on workflowsPath with Webpack and run them inside v8 isolates.

If Webpack fails to create the bundle, the SDK will throw an error and emit webpack logs using the SDK's logger.

If you do not see Webpack output in your terminal make sure that you have not disabled SDK logging (see reference to Runtime.install() in the link above).

A common mistake for newcomers to the TypeScript SDK is trying to use Node.js built-ins and modules in their Workflow code. Usually, the best thing to do is move that code to an Activity.

Some common examples that will not work in the Workflow isolate:

Importing node built-in modules

Antipattern

import fs from 'fs';

const config = fs.readFileSync('config.json', 'utf8');

This is invalid because reading from the filesystem is a non-deterministic operation: the file may change from the time of the original Workflow Execution to when the Workflow is replayed.

You'll typically see an error in this form in the Webpack output:

2021-10-14T19:22:00.606Z [INFO] Module not found: Error: Can't resolve 'fs' in '/Users/you/your-project/src'
2021-10-14T19:22:00.606Z [INFO] resolve 'fs' in '/Users/you/your-project/src'
2021-10-14T19:22:00.606Z [INFO]   Parsed request is a module
2021-10-14T19:22:00.606Z [INFO]   using description file: /Users/you/your-project/package.json (relative path: ./src)
2021-10-14T19:22:00.606Z [INFO]     Field 'browser' doesn't contain a valid alias configuration

Importing and calling Activities directly from Workflow code

Antipattern

import { makeHTTPRequest } from './activities';

export async function yourWorkflow(): Promise<string> {
  return await makeHTTPRequest('https://temporal.io');
}

This is invalid because activity implementations should not be directly referenced by Workflow code. Activities are used by Workflows in order make network calls and reading from the filesystem, operations which are non-deterministic by nature because they rely on external state. Temporal records Activity results in the Workflow history and in case your Workflow is replayed, completed Activities will not be rerun, instead their recorded result will be delivered to the Workflow.

You'll typically see an error in this form in the Webpack output:

2021-10-14T19:46:52.731Z [INFO] ERROR in ./src/activities.ts 8:31-46
2021-10-14T19:46:52.731Z [INFO] Module not found: Error: Can't resolve 'http' in '/Users/you/your-project/src'
2021-10-14T19:46:52.731Z [INFO]
2021-10-14T19:46:52.731Z [INFO] BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
2021-10-14T19:46:52.731Z [INFO] This is no longer the case. Verify if you need this module and configure a polyfill for it.
2021-10-14T19:46:52.731Z [INFO]
2021-10-14T19:46:52.731Z [INFO] If you want to include a polyfill, you need to:
2021-10-14T19:46:52.731Z [INFO]         - add a fallback 'resolve.fallback: { "http": require.resolve("stream-http") }'
2021-10-14T19:46:52.731Z [INFO]         - install 'stream-http'
2021-10-14T19:46:52.731Z [INFO] If you don't want to include a polyfill, you can use an empty module like this:
2021-10-14T19:46:52.731Z [INFO]         resolve.fallback: { "http": false }

To properly call your Activities from Workflow code use proxyActivities and make sure to only import the Activity types.

import { proxyActivities } from '@temporalio/workflow';

import type * as activities from './activities';
const { makeHTTPRequest } = proxyActivities<typeof activities>();

export async function yourWorkflow(): Promise<string> {
  return await makeHTTPRequest('https://temporal.io');
}

Works in Dev but not in Prod

The two main sources of dev-prod discrepancies are in bundling and connecting.

Production bundling

You may experience your Client sending stripped names as the Workflow "Type" when scheduling a Workflow. Webpack can change the Workflow's function name to something shorter. Temporal won't know how to handle the mismatch between the shorter name and the expect Workflow type.

You may experience errors like this:

Error: 3 INVALID_ARGUMENT: WorkflowType is not set on request.

Or you may see shorter names in the Temporal Service's Web UI when Webpack changed the Workflow's function name to something shorter, in this case the single letter 's':

Temporal Web UI showing stripped "Workflow Type" entries, in this case the single letter 's'

This issue can happen when your bundler strips out Workflow function names. Temporal relies on those names to set the "Workflow Type" in the Service Web UI. To prevent the build process from shortening Workflow function names, modify the webpack configuration file ( webpack.config.js) to set the Boolean that retains the original names in the TerserPlugin configuration section. Setting the option (keep_fnames) to true prevents name stripping.

Webpack with Terser
ESbuild

// webpack.config.js
module.exports = {
  optimization: {
    minimize: true,
    minimizer: [
      new TerserPlugin({
        terserOptions: {
          keep_fnames: true, // don't strip function names in production
        },
      }),
    ],
  },
};

require('esbuild').buildSync({
  entryPoints: ['app.js'],
  minify: true,
  keepNames: true,
  outfile: 'out.js',
});

See the esbuild docs for more information.

Connecting to Temporal Server

If you are trying to connect in production and getting this:

[TransportError: transport error]

It is a sign that something is wrong with your Cert/Key pair. Log it out and make sure it is an exact match with what is expected (often, the issue can be whitespace when injecting from your production secrets management environment).

Resetting Workflows to deal with logical bugs

You can "rewind time" using the Temporal CLI, resetting Workflow History to some previous point in time. You can read the Temporal CLI docs on:

If you need to reset programmatically, the TS SDK does not have any high level APIs for this, but you can make raw gRPC calls to resetWorkflowExecution.

Resetting should only be used to deal with serious logical bugs in your code: it's not for handling transient failures, like a downstream service being unreachable. It should not be used in the course of normal application flows.

gRPC call timeouts (context deadline exceeded)

The opaque context deadline exceeded error comes from gRPC:

Error: 4 DEADLINE_EXCEEDED: context deadline exceeded
    at Object.callErrorFromStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client.js:179:52)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /Users/swyx/Work/Temporal/samples-typescript/nextjs-oneclick/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  code: 4,
  details: 'context deadline exceeded',
  metadata: Metadata {
    internalRepr: Map(1) { 'content-type' => [Array] },
    options: {}
  },
  page: '/api/getBuyState'
}

Several conditions can cause this error, including network hiccups, timeouts that are too short, and an overloaded server. Querying a Workflow Execution whose query handler causes an error can result in the query call timing out.

Some troubleshooting actions you can take:

Verify the connection from your Worker to the Temporal Server is working and doesn't have unusually high latency.
If you are running Temporal Server yourself, check your server metrics to ensure it's not overloaded.
If what's timing out is a query, check the logs of your Workers to see if they are having issues handling the query.

If none of the preceding actions help you discover why timeouts are occurring, please try to produce a minimal repro and we'll be glad to help.

How to debug in a development environment​

How to debug in a production environment​

How to troubleshoot common issues in the TypeScript SDK​

Two locations to watch​

Stale Workflows​

Workflow/Activity registration errors​

Webpack errors​

Works in Dev but not in Prod​

Production bundling​

Connecting to Temporal Server​

Resetting Workflows to deal with logical bugs​

gRPC call timeouts (context deadline exceeded)​