Chapter 6: Environment

Your code doesn’t run alone, like some ethereal idea in a perfectly uneventful void.

Instead, it runs in an onion.

Fancy performance engineers generally call this “the environment”. I call it an onion.

{ DIAGRAM of the onion }

The onion has layers, and each of them affects your code’s performance. Let’s start from the inside-most layer.

Flutter

Much of your Flutter code is programmed in a declarative style, and it’s on the framework to figure out how to make things work. This in itself is a bit of a layered onion — from C++ code of the Flutter engine all the way up to common Flutter widgets such as ListView — but you get it as an atomic package. When you upgrade Flutter, the new version contains a specific version of Dart, a specific Flutter engine, and so on. It’s rare for people to be running, say, Flutter 3.29.0 with anything else than framework revision 35c388afb5, engine revision f73bfc4522, and Dart 3.7.0.

The version of the Flutter SDK can obviously have a large effect on the performance of your code. It’s common for Flutter to get performance improvements even in minor version bumps. It’s also very much possible to encounter a performance regression from version to version.

When evaluating performance, always be aware which version of the Flutter SDK your app is compiled in. You can find the version by running flutter --version. You can also run flutter --version --machine to get machine-readable JSON output like this:

{
  “frameworkVersion”: “3.29.0”,
  “channel”: “stable”,
  “repositoryUrl”: “https://github.com/flutter/flutter.git”,
  “frameworkRevision”: “35c388afb57ef061d06a39b537336c87e0e3d1b1”,
  “frameworkCommitDate”: “2025-02-10 12:48:41 -0800”,
  “engineRevision”: “f73bfc4522dd0bc87bbcdb4bb3088082755c5e87”,
  “dartSdkVersion”: “3.7.0”,
  “devToolsVersion”: “2.42.2”,
  “flutterVersion”: “3.29.0”,
  “flutterRoot”: "/path/to/flutter"
}

If you need to access this information from within a running app, use FlutterVersion.

import 'package:flutter/services.dart' show FlutterVersion;
var version = FlutterVersion.version;

Dart runtime

When you run dart compile exe — either directly or through flutter build — your Dart code gets compiled into machine instructions.

So instead of

var x = z + 1;

you get something like

mov rdi, 0x1
add rax, rdi

except in binary, so it’s something like

01001000100000111100011100000001010010001000100111111000

in the actual .exe file.

But that’s not all that goes into that file. The compiler must also include the Dart runtime. That’s all the things that your program does that you don’t need to worry about. For example:

Memory management. When you create a new List in Dart, you don’t need to worry about allocating memory for it, nor do you have to remember to free the memory once the list is no longer needed. But your code wouldn’t work without this, so it’s included in the .exe as part of the runtime. This includes the garbage collector, which will be a recurring character in this book.
Exception handling. When you decide to throw an exception in your Dart code, you don’t need to worry about things like stopping execution mid-function, creating the StackTrace object, or finding whoever’s responsible of catching it. But again, your code wouldn’t work without this, so it’s included in the .exe as part of the Dart runtime.
I/O. When you read from the file system with something like file.readAsString(), you don’t need to worry about interfacing with the underlying system. Same goes for any other input or output. This, too, goes into your binary as part of the Dart runtime.

The binary code you ship is deeply intertwined with the Dart runtime. Some of the most basic operations — things that you never ever think about while writing Dart, such as passing objects to functions — are performed by the runtime.

So, as you can imagine, the Dart runtime has a major effect on the performance of your code.

You can find your installed Dart’s version by running dart --version. The version is also printed with flutter --version.

The runtime will be different for different operating systems and architectures. So, the runtime for linux_arm64 will be different from the one for windows_x64 even if they are the same version number. But we’ll get back to this further along our path through the onion.

If you need to find out the version of the runtime from within a running Dart program, you can use the following:

import 'dart:io' show Platform;
var version = Platform.version;

This assumes you’re not running the program in the browser. Then again, if you’re running Dart in the browser, chances are you’re running a Flutter app, and you can use the aforementioned FlutterVersion class and its FlutterVersion.dartVersion member.

The browser is a whole separate layer, of course. When compiling for the Web, your Dart code and the Dart runtime is either compiled into JavaScript or into Web Assembly. The browser then executes that JavaScript or Web Assembly. Suffice to say, the browser and its version matters. Plus, unlike the runtime that is compiled with your code, the browser environment can (and will) change “under your feet.” Browser vendors (Google, Microsoft, Mozilla, Apple) constantly work on their technology. This means that the same build of your app (the same compiled blob of JavaScript or Web Assembly) can run at different speeds today than they will do tomorrow.

Operating System

Let’s get back to your compiled .exe with all its machine instructions. Not even this can run by itself. It needs an operating system to talk to. The OS provides access to files, the network, and important services such as the time. It also allocates memory, provides processing threads, starts processes, and more. Any of these things can be more efficient, or less efficient, depending on the OS and its version.

So, once again, you should remember to capture this context when analyzing performance. To get the information about the exact version of the operating system, use the Platform class:

import 'dart:io' show Platform;
var osName = Platform.operatingSystem;
var osVersion = Platform.operatingSystemVersion;

But the real issue with operating systems isn’t that their performance changes slightly between versions. It’s that they constantly and actively affect performance in real time.

We talked about governors in the previous chapter about Performance lottery. In short, operating systems change processing speed depending on various heuristics. If the OS governor thinks that the current load isn’t urgent, it lowers the clock speed of a processor or even puts some CPUs to sleep. This can happen really quickly: one millisecond, the device is firing on all cylinders, and the next one, it’s in battery saving mode. Then a fraction of a second later, the user touches the screen, and the governor switches everything back on.

Governors are a pain when it comes to benchmarking but they’re obviously a good thing from the more general perspective of app performance. Yes, we want our apps to be snappy, but not at the cost of draining all of the user’s battery, or keeping their cooling fans on at all times. Once again, it’s a tradeoff.

We’ll talk about how to disable governors in a later chapter. For now, let’s just acknowledge that these heuristics, which are included in all our operating systems (from desktop to laptops to mobile phones), have an effect on the performance of our programs.

Hardware

We live in a world of thousands of devices from hundreds of manufacturers, all of them with slightly different performance characteristics. A 13-inch 2020 MacBook Pro M1 is different from a custom-build Intel x64 PC is different from Samsung A25 6GB is different from Samsung A25 8GB is different from iPhone 16e is different from iPhone 16 Pro Max 256GB is different from Raspberry Pi 4B.

The great thing is that Flutter can target each of these devices and a thousand more.

The absolutely terrifying thing is that Flutter can target each of these devices and a thousand more.

Well, it’s only terrifying if you expect your app’s performance to be precisely the same on your user’s devices as it is on yours. Once you let go of this silly notion, you’re good.

In practical terms, it’s important to test your app on a wide variety of devices. Don’t do the Silicon Valley thing where people only ever see their app running on the latest flagship phone connected to spotless 5G. Ideally, you should have your app on your daily-driver phone — so that the app needs to co-exist with a myriad other apps and services — and that phone should be on the lower end of your target audience’s device spectrum.

If you’re really into shiny new iPhones but you’re shipping apps that will be used by hundreds of millions of users in emerging markets (or just ordinary users in the US), I have bad news for you. You have two options:

Downgrade the phone you carry around in your pocket
Lose touch with what your app trully feels like to use by the vast majority of your users

If you’re benchmarking, select a single device and use that. Comparing two benchmark results from two different devices is useless.

Room

When I was employed at Google on the Flutter team, my family and I lived in a two-bedroom apartment in Mountain View, California, not far from the Google offices. In 2020, the pandemic came, and we had to work from home. I took the bedroom.

I had an Intel MacBook Pro back then, and I remember one Summer day I started having issues with my video conferencing — very choppy playback, to the point of being unusable. I just assumed it was an internet bandwidth problem, switched from video to audio, and that fixed it.

But then one of the online calls ended, and suddenly it wasn’t just the video conferencing that was choppy. The mouse cursor, which on macOS is basically always driven by the system, was choppy. Opening basic system windows took significant fractions of a second. The whole system had terribly low FPS. Definitely not an internet bandwidth issue.

I opened System Monitor (Mac’s version of Task Manager) and realized there’s a service running at something like 720% CPU. If you wonder how something can be running at more than 100% capacity, know that for macOS, 100% CPU usage literally means 100% of a single processor core. So an 8-core processor can go all the way up to 800% CPU usage. Anyway, what System Monitor was seemingly saying was that there was something very resource-intensive running at the time. But what?

Nothing, it turned out. This was just the macOS way of thermal throttling. By creating a fake process that supposedly eats most of the CPU processing power, other processes need to slow down.

The reason my computer was slow wasn’t lazy programming, hard drive thrashing or outdated firmware. It was Summer.

You see, in Mountain View, we rented an apartment under the roof and so, during the Summer, it got really hot. (We didn't have air conditioning. We’re one of those weird types who avoid air conditioning unless it’s unbearably hot. And in Silicon Valley, it’s almost never unbearably hot.) The temperature in the room was above 30 °C, and that made it hard for the MacBook to cool itself. macOS wanted to avoid melting the CPU, so it throttled the system performance. Everything got super slow and choppy until I put a cooling pad under the poor, overheated device.

The moral of the story is that the onion doesn’t stop at the device boundary. The device can be in a hot car — suddenly your app slogs. Your Dart app server can be running on a server with hard drives, and the rack can be experiencing extra vibration — suddenly disk access is significantly slower, and so is your server response.

As a performance engineer, you should be aware of these effects. When performance issues are reported, you may want to ask about the environment, and when you’re building a performance testing rig, you’ll want to keep it in mind, too.

Controlling the onion

When you try to address performance problems, the onion is your enemy.

The environment introduces noise and biases, sometimes at a magnitude much larger than the effect you’re trying to address. If you’re not careful, you could easily get the idea that your improvements make a difference when they actually don’t, or else you might think they don’t make a difference when they actually do.

I can’t stress how important this is. You might be thinking that “every measurement is better than no measurement” but that’s wrong. A misleading measurement is not better than no measurement, and in some ways it’s worse. It can come with a false sense of confidence — you might not only be wrong, you might be confidently wrong.

Plus, unlike with no measurement, misleading measurement still takes time.

There’s a sliding scale of how much control you exert on the onion. On one extreme, you could spend significant amount of time and money on an air-conditioned performance testing lab with a bunch of devices and a complex testing infrastructure. On the other extreme, you could just close other apps while running the benchmark on your laptop.

Later on in this book, I want to take us all the way to the lab. But for now, let’s just do the basics. When running any kind of performance measurement (assuming you actually care about the result):

Take note of the framework / engine / programming language / OS version numbers. If you can pin them so that they are constant between measurements even if those measurement are months apart, do so.
Take note of the hardware. If at all possible, don’t compare numbers measured on different devices, even if they are the same type. Pick one device, use it exclusively. Remember the case from [[050 Performance lottery]] of a different username having an effect on measurement.
Keep the device plugged in.
Close other apps (especially resource hogs like browsers or games).
Avoid thermal throttling. More on that later, but for now, at least run sudo powermetrics -s thermal (Mac), sudo turbostat --interval 1 --show Core,CPU,CoreTmp,CoreThr (Linux), HWiNFO64 app (Windows), adb shell dumpsys thermalservice (Android) or log collect --device (iOS). If you detect thermal throttling, you aren’t measuring your benchmark anymore — you’re measuring the operating system’s throttling policy.

With this, we can cautiously step to the next chapter, [[070 Old-school Benchmarking]]. But first I’d like to make sure I haven’t set up an analytical trap.

I just spent a whole chapter talking about all the ways measurements can go wrong and all the ways external factors play role in your app’s performance. After that kind of talk, it can be tempting to say that it’s all futile. Does making a method a little bit faster actually matter when the user will have their phone on the car’s dashboard in direct sunlight, and thus massively throttled? Do performance optimizations actually matter?

They do. Our job as performance engineers is to isolate as much of the external factors as possible, understanding that we will never be 100% successful. And then act on what we learn, understanding that there could be users out there who will experience bad performance even after our best shot at the problem — but there will be fewer of them, or the performance problems will become less severe. We are only making performance better by degrees. You can never be 100% sure that your app will never jank. You can just make it less probable.

(back to index)