DEV Community

Nick Lucas
Nick Lucas

Posted on • Updated on

Typescript Runtime Validators and DX, a type-checking performance analysis of zod/superstruct/yup/typebox

Preface

In 2023, Typescript is rarely questioned as an important tool for modern JavaScript developers, but one of its biggest limitations is the lack of added runtime type safety, particularly when dealing with IO at the boundaries of your application.

To solve this problem a number of popular runtime validation and type-safety tools have popped up, usually competing on runtime parsing speed, API expressiveness, and a TypeScript vs JSON-schema based core.

Compile-time and Editor Performance Pain

What seems to be degrading is performance of the developer experience. TypeScript drives the code intelligence of many popular editors, and as you add type complexity and size to an application or monorepo, code-completion and type-checking starts to draaaaag. This is because driving features like autocomplete means compiling your code on the fly repeatedly, and the more types & files needed to populate autocomplete for a line of code, the longer it takes to get a response.

This is compounded by the growing movement of fully and deeply typed libraries like tRPC, and Tanstack's Query & upcoming Router, which utilise new layers of generics and mapped types over your DTOs, deepening your types, and surfacing compile-time performance issues which weren't as noticeable before.

What's prompted me to look at this is my tRPC+Zod editor experience at work has become unusable. 2-3 seconds to just get autocompletion options on a tRPC path, and then another 2-3 for the next path, repeat. When investigated using TypeScript's tracing tools the data entirely points back to my team's Zod DTOs. What I learned is that Zod's performance is okay at the start, but when you start using methods like .extend/.pick/.omit (and so on) the performance regresses in the order of a magnitude. Rather than making this into a "Zod considered bad" post, I wanted to investigate how the alternatives which can be integrated with tRPC fare, and see whether I can do better.

Analysis Setup

For these benchmarks I'm using a Macbook Pro with M2 Pro and 32GB of RAM.

I've worked on 2 codebases in an Nx monorepo to set up this experiment.

  1. Light Type - an experimental runtime type-checker with feature parity in a number of key areas (and an inspired API) to Zod
  2. Benchmarks - a tRPC API and React SPA which uses a number of popular and tRPC-compatible type-checkers

The type-checkers under test are:

There may be more libraries which can be integrated with tRPC, but the first 3 are recommended by the project, and TypeBox has had some effort made to integrate it. Light-Type is designed to fit the necessary interfaces of Zod so it's also compatible natively.

The Benchmarks are each made up of a tRPC router with types co-located, and a React Component using tRPC's @tanstack/query abstraction which consumes the router.

TypeScript is pretty good at optimising its compiles. It caches where it can and where there's a single path back to a type it doesn't spread itself wider. So when performance tracing this setup I've seen that compile-times for each router/component don't bleed onto the others, making benchmarking the compiler much easier.

The Actual Types

Each library has two benchmarks, a 'simple' one, and a 'complex' one which adds usage of the equivalent to extend/omit in each library. Except Yup which doesn't have the necessary features to join the complex round.

Simple Router Example
// simpleZodRouter.ts

import { z } from 'zod'
import { publicProcedure, router } from '../trpc'

const PersonDto = z.object({
  id: z.number(),
  firstName: z.string(),
  lastName: z.string(),
  tel: z.string().optional(),
})

const CarDto = z.object({
  id: z.number(),
  name: z.string(),
  age: z.number(),
  brand: z
    .union([
      z.literal('Volvo'),
      z.literal('Mercedes'),
      z.literal('BMW'),
      z.literal('Ferrari'),
      z.literal('Bazmus'),
    ])
    .optional(),
  previousOwners: z.array(PersonDto).default([]),
})

type DbCar = typeof CarDto['_input']
type Brand = typeof CarDto['_output']['brand']

export const simpleZodRouter = router({
  list: publicProcedure
    .input(z.object({ count: z.number().default(100) }))
    .output(z.array(CarDto))
    .query((opts) => {
      return new Array(opts.input.count).fill(null).map<DbCar>((_, idx) => {
        return {
          id: idx,
          name: 'Foobarmo',
          age: 3,
          brand: 'Bazmus' as Brand,
          previousOwners: [
            {
              id: idx * 100,
              firstName: 'Bob',
              lastName: 'Owneverythingman',
              tel: undefined,
            },
          ],
        }
      })
    }),
  get: publicProcedure
    .input(z.object({ id: z.number() }))
    .output(CarDto)
    .query((opts) => {
      return {
        id: opts.input.id,
        name: 'Foobarmo',
        age: 3,
        brand: 'Bazmus' as Brand,
        previousOwners: [
          {
            id: opts.input.id * 100,
            firstName: 'Bob',
            lastName: 'Owneverythingman',
            tel: undefined,
          },
        ],
      }
    }),
  create: publicProcedure
    .input(CarDto)
    .output(CarDto)
    .mutation((opts) => {
      return opts.input as DbCar
    }),
})

Enter fullscreen mode Exit fullscreen mode

Complex Router Example
// complexZodRouter.ts

import { z } from 'zod'
import { publicProcedure, router } from '../trpc'

const EntityDto = z.object({
  id: z.number(),
})

const PersonDto = EntityDto.extend({
  firstName: z.string(),
  lastName: z.string(),
  tel: z.string().optional(),
})

const CarDto = EntityDto.extend({
  id: z.number(),
  name: z.string(),
  age: z.number(),
  brand: z
    .union([
      z.literal('Volvo'),
      z.literal('Mercedes'),
      z.literal('BMW'),
      z.literal('Ferrari'),
      z.literal('Bazmus'),
    ])
    .optional(),
  previousOwners: z.array(PersonDto).default([]),
})

type DbCar = typeof CarDto['_input']
type Brand = typeof CarDto['_output']['brand']

export const complexZodRouter = router({
  list: publicProcedure
    .input(z.object({ count: z.number().default(100) }))
    .output(z.array(CarDto))
    .query((opts) => {
      return new Array(opts.input.count).fill(null).map<DbCar>((_, idx) => {
        return {
          id: idx,
          name: 'Foobarmo',
          age: 3,
          brand: 'Bazmus' as Brand,
          previousOwners: [
            {
              id: idx * 100,
              firstName: 'Bob',
              lastName: 'Owneverythingman',
              tel: undefined,
            },
          ],
        }
      })
    }),
  get: publicProcedure
    .input(z.object({ id: z.number() }))
    .output(CarDto)
    .query((opts) => {
      return {
        id: opts.input.id,
        name: 'Foobarmo',
        age: 3,
        brand: 'Bazmus' as Brand,
        previousOwners: [
          {
            id: opts.input.id * 100,
            firstName: 'Bob',
            lastName: 'Owneverythingman',
            tel: undefined,
          },
        ],
      }
    }),
  create: publicProcedure
    .input(CarDto.omit({ id: true }))
    .output(CarDto)
    .mutation((opts) => {
      return opts.input as DbCar
    }),
})

Enter fullscreen mode Exit fullscreen mode

Running the benchmark

# Set up the monorepo
yarn install

# The app/api can served as a test of the integration
yarn light-type-benchmark:start

# Run a traced compile of the repo
yarn tsc:trace
> tsc -p tsconfig.base.json --noEmit --incremental false --jsx react-jsx --allowSyntheticDefaultImports --generateTrace ./.trace
Enter fullscreen mode Exit fullscreen mode

tsc:trace produces ./.trace/trace.json which can be analysed in your browser's dev tools or Perfetto.

This does produce a lot of tracing output which we don't need, but since the benchmarks are isolated in their own projects TypeScript produces reliable results for them. I've edited down the trace file just to the files we're concerned with.

Results

And here are the results:

Benchmarks Flamegraph

You can download and explore the trace yourself here: https://gist.github.com/Nick-Lucas/ee964bf004f40ea6601bd2543869efee

While compile-times do vary slightly from run to run, the scale of difference between each library remains the same.

Library Test Router ms Component ms Note
Zod Simple 24ms 10ms
Zod Complex 281ms 17ms 281ms is not a typo
Yup Simple 22ms 9ms
Yup Complex Could not join the fun
Superstruct Simple 10ms 11ms
Superstruct Complex 42ms 18ms
Typebox Simple 28ms 11ms
Typebox Complex 38ms 9ms
Light Type Simple 8ms 25ms
Light Type Complex 22ms 11ms

Conclusions

The big outlier here is Zod, which seems to have some structural issues with its types, ballooning in cost by 10+ times after just 2 .extend and 1 .omit call. All the other libraries perform pretty well in the same situation, though Superstruct seems to balloon by about 4x, and TypeBox is slower to begin with but then degrades less poorly. Yup doesn't compete on complex features but performs very well at what it's designed for.

  • It's clear that all the type processing for the libraries is happening in the processing of router files where those types are located, and those are quite varied in cost. These results are reproducible over and over again too, it's not a die-roll for the big differences. The react components themselves seem to be isolated and very fast to compile.

  • Light Type is consistently the fastest across multiple benchmarks. This is going to be down to a couple factors:

    1. It has fewer features than the other libraries. I would predict it will slow slightly over time as I continue developing it.
    2. Light Type has a lot of focus put into producing the simplest possible types at every stage. Which for example Zod doesn't seem to prioritise.

Notes specifically on Zod: That big block in the flamegraph is Zod's Complex test. Basically don't try to use any features outside of primitives, object, array, or you may immediately gain 10x šŸ¤’ to that type.

  • The sum total of all type compilations is your editor experience, and 281ms is already about as high as you probably want your entire library of types to take for autocomplete.
  • Those brown tags in the top of the flamegraph are labelled recursiveTypeRelatedTo_DepthLimit, I'm really not an expert on TypeScript's insides, but that sounds like it's trying to tell us something important, and Zod has been guilty of causing "Type instantiation is excessively deep and possibly infinite" errors.
  • Zod right now, looks to be a big footgun and in need of rethinking its types, at least when combined with other modern tools like tRPC. If you're starting a new project it might one to avoid for now, though there's really not much alternative if you love Zod's API

Top comments (11)

Collapse
 
michaelangrivera profile image
Michael Angelo • Edited

Hey there @nicklucas ! I'm on the core team at ts-rest.com. We've actually done a very similar analysis here: github.com/ts-rest/ts-rest/issues/162. We use Zod for a lot of our implementation. Curious to see if you've enabled strict (more specifically, strictNullChecks) in all of your tsconfig.json/compilerOptions?
If not, I implore you to follow the steps here ts-rest.com/docs/troubleshoot#why-... (you don't have to use ts-rest) and see if it improves your TS performance.
strictNullChecks is actually required by Zod github.com/colinhacks/zod#requirem..., and the reasoning is here: github.com/colinhacks/zod/issues/1750

I suggest rethinking posting this article if you haven't enabled that compiler option for every tsconfig.json in your monorepo, as that would lead to an unfair comparison against the other libaries.

Thanks for reading!

Collapse
 
nicklucas profile image
Nick Lucas • Edited

Hey Michael, yes these benchmarks were all done using strict=true and it's the same at work. Strict mode just makes TypeScript that much better. Good question though as I didn't think to bring it up :)

Collapse
 
crobinson42 profile image
Cory Robinson

Have you tried SureType?!
npmjs.com/package/suretype

Collapse
 
m0ltzart profile image
Roman

We maintain a whole repo of runtime benchmark tests here:

github.com/moltar/typescript-runti...

Didn't know about Light Type, added an issue here:

github.com/moltar/typescript-runti...

Collapse
 
nicklucas profile image
Nick Lucas

Looks fantastic! Any plans to add comparative benchmarks for type-checking in the spirit of this article?

Light-Type is very much still an experiment, but I'd love to see it incorporated, especially if it goes to a release šŸ˜„

Collapse
 
m0ltzart profile image
Roman • Edited

in the spirit of this article?

Do you mean this:

Each library has two benchmarks, a 'simple' one, and a 'complex' one

If so, there are a few issues and discussions surrounding that. No clear consensus yet on how to achieve parity across all packages. Basically there is a debate whether we should have a common denominator or if we should open up to different test cases. Opening up to different (per package) test cases can open a can of worms.

You can find an awesome, and very thoughtful response by @sinclairzx81 (the author of typebox) here:

github.com/moltar/typescript-runti...

Thread Thread
 
nicklucas profile image
Nick Lucas

Ah I mean, my understanding from looking through your results is it's about run-time performance of the parsing of data. This piece here is about language server / compile-time performance of typescript itself. I believe they're both very important in their own ways.

Zod has put a lot of effort into being pretty fast at parsing and that shows in your suite, but it's demonstrably bad at producing performant typescript inference. I've worked on a benchmark suite lately which demonstrates this and could easily be extended to more libraries

Thread Thread
 
m0ltzart profile image
Roman

Ok, I understood your point. Yes, I thought about TS parser performance. I think it would be a great addition to the suite. Please create an issue with any thoughts or suggestions you may have to foster a public discussion. We would appreciate your existing knowledge transfer so we can make the benchmark suite more robust.

Collapse
 
m0ltzart profile image
Roman • Edited

I'd love to see it incorporated

Feel free to submit a PR. Use a previous PR for submission as guidance. Even if it's not fully ready, it's ok. As long as it's a published package, it's fair game. At least that will give you some feedback how it stacks against the rest of the ecosystem!

Issue: github.com/moltar/typescript-runti...

Collapse
 
crobinson42 profile image
Cory Robinson

I highly encourage folks to checkout the little talked about lib SureType... it is amazingly fast and a pleasure to use.

npmjs.com/package/suretype

Collapse
 
dutziworks profile image
Eldad Bercovici

How does this affect time-to-autosuggest (considering TS caches a lot of stuff)?