Doing performance testing for components

[allison-c]

Goals

Testing paprika component performance.

What we want:

Prevent unnecessary changes making a component's slow down component rendering time
We want an automated process that can continuously identify performance issues in paprika. We need to integrate the tool with our building pipeline. Reducing package bundle size, render time, rerender times after user interaction, expansive calculation is the things we want to improve.

Log unnecessary re-renders
It would be nice if the tool can catch unnecessary renderings and report them, like what printWasted doing in react-addon-perf https://reactjs.org/docs/perf.html

Nice to have:

Ability to test user interactions
Although initial render time is important, UI components need to be high performance to react to user interactions like select an item in a listbox, or typing in an input.

Report to show how did we improve the performance in paprika
Generating reports for each component and use the report in jest to integrate with the build pipeline.

We don't need to:

Figure out how to improve the entire application performance
Application performance and component performance has quite a lot of different requirement. For example, application performance is related not only to the react code but also the infrastructure in that app. Where do you publish the app? How do you compile the assets? Since paprika is a component library, so we don't need to solve the application performance issues.

Method

Lighthouse

Demo: Lighthouse demo in paprika

The good news is lighthouse is a very handy tool that I only spent 1 day to integrate with paprika. I've accomplished these:

• Run lighthouse with production build storybook iframe.html page and get a consistent result
• Showcase the report lighthouse generated
• Bonus: generated accessibility score as well

The sad news is as you can see the performance score from lighthouse is pretty low. It's not even reasonable since I'm just testing with <Button /> component. So I spent more time taking a deeper look, and got 3 different results from different environments.

Running lighthouse within storybook(production build) iframe.html locally, performance score is 2x
Running lighthouse within storybook(production build) iframe.html on https://paprika.highbond.com/ , performance score is 0
Running lighthouse with a clean react app using create-react-app, also production build, the score is 99

As you can see, the lighthouse score is either too low to be a benchmark or too high. The reason is in paprika storybook, we pack everything together, which definitely slows down the first meaningful paint, plus there might be some storybook code even in the iframe.html. If we use the score as the benchmark value in jest, I'm not sure how much value it can bring. Other than that, it doesn't look nice if we use the 2x number in our test. For the third method I commented above, I didn't find a good way to automatically build a separate react app for each component.

Some people have the same issue: https://stackoverflow.com/questions/59088598/bad-react-performance-in-lighthouse-audit

Summary for lighthouse:

+Easy to use
+Nice report
-Not that useful in our case

React <Profiler> api

reactjs/rfcs#51

There's an experimental profiler API in react 16, but only works in the dev build. I haven't get time to try it. The chrome plugin React Profiler is using it. I was concerned about how useful the data is if we can only run the test in the dev build version, but if we don't have better options, we might give it a try.

Benchmarking tools

I was kind of don't know what to do with lighthouse, then I moved on to search for benchmarking tools. Comparing to the 0~100 score in lighthouse, a timing cost of rendering seems like more useful for component level testing. I found 2 libraries, and both of them are not popular, but I still tried.

The first one is react-component-benchmark. To start benchmarking, I have to call the start() method on the <BenchMark /> component instance. The author is using enzyme, so he calls component.instance().start(), but with rtl there's no way to get the instance and call the method. So I failed.

The second one is react-benchmark, I was simply failed when run the script in paprika root. I think it's related to the conflict between lerna package and the build process in that tool.

Although the result is sad that I didn't find an existing tool to help us, I was thinking about did I go the wrong way from the beginning?

rtl and cypress?

No. Both of them are not for performance testing purposes.

Next step

Try <Profiler /> with dev build?
Build our own benchmarking tool

Resources

• For React 15 https://engineering.musefind.com/how-to-benchmark-react-components-the-quick-and-dirty-guide-f595baf1014c
• How I got a 100 lighthouse score in react app https://medium.com/@april9288/how-i-got-a-100-lighthouse-score-with-my-react-app-2b676ef62113
• Cypress trade offs https://docs.cypress.io/guides/references/trade-offs.html#Permanent-trade-offs-1
• Accurate benchmarking is hard https://medium.com/p/ff105586036/responses/show

[nahumzs]

Thank you allison for the research, bummer we can't integrate lighthouse with our current stack, but worth the learning process and the research 🙏