• Support a more flexible coordination model
• Support multi-channel segmentation bitmasks
• Support per-obsType segmentation bitmasks
• Use semantically-meaningful data types for points and spots
• Unify the visualization properties for polygon and bitmask segmentations

These views are available in JS package v3.2.0 as spatialBeta and layerControllerBeta.

Note: automatic initialization of the coordination space is not yet supported. We suggest using the object-oriented VitessceConfig API (and the new linkViewsByObject method) to manually initialize the values.

Flexible coordination model​

Previously, the spatial view used two coordination types spatialSegmentationLayer and spatialImageLayer which required an array of nested objects as values. Usage of nested objects as coordination values prevents coordination of a subset of those values. For example, this prevented coordinating channel colors while not coordinating other properties such as channel intensity windows.

In contrast, spatialBeta and layerControllerBeta are more flexible because they use multi-level- and meta- coordination to achieve the configuration of hierarchical properties including image layers and channels.

In the base case of the single-level coordination model, each view can be mapped to a set of coordination scopes, one scope per coordination type. Multi-level coordination extends this model by allowing each coordination scope to itself have its own set of coordination scope mapping, generating a new "level" of coordination.

For example, it allows each image layer coordination scope to be mapped to its own coordinated properties such as opacity and visibility. Meanwhile, the opacity and visibility properties remain primitive numeric and boolean values, respectively, which are agnostic to the existence of any image layer.

Other benefits:

• Ability to configure global Z/T/C indices for coordination of spatial/temporal/channel selections across data types, datasets, or views
• Ability to configure 3D vs. 2D mode globally (i.e., per-view, as opposed to per-image layer).
• Ability to fall back to coordination values in the previous level (if not initialized in the subsequent level).

A disadvantage of this more flexible coordination model is that it requires a more complex JSON representation that is tedious to write and reason about. To address this, we have extended our object-oriented configuration APIs:

• VitessceConfig.linkViewsByObject
• VitessceConfig.addCoordinationByObject
• VitessceConfig.addMetaCoordination
• VitessceConfigView.useCoordinationByObject
• VitessceConfigView.useMetaCoordination
• VitessceConfigMetaCoordinationScope
• CoordinationLevel

The resulting VitessceConfig instance can be converted to JSON.

Multi-channel, per-obsType segmentation bitmasks​

The flexible coordination model described above enables data-related properties such as obsType ("observation type") to be mapped per-image-channel. In other words, for segmentation bitmask images, each bitmask channel can represent a different type of observation. For example, channel zero can contain nucleus segmentations, while channel one contains cell segmentations, and channel two contains glomerulus (or other functional tissue unit) segmentations.

Thus, obsType can be used to join multiple data types together, as described in our previous blog post (but now on a per-channel basis). For example, the obsType: 'cell' channel might have its own obsSets for cell type annotations, while the obsType: 'glomerulus' channel has its own obsSets for annotation of which glomeruli are globally sclerotic vs. non-globally sclerotic.

For more details about data storage, see data-conventions/multi-channel-label-images.

obsPoints and obsSpots data types​

The new data types obsPoints and obsSpots enable more straightforward configuration of visualizations containing point-based measurements (e.g., MERFISH transcript points) and spot-based measurements (e.g., Visium spots). While the data representations for points and spots are currently identical (XY coordinates), the more semantically-meaningful nature of the data type names helps to infer (from the Vitessce configuration) how each should be visualized. For example, points have a fixed size in pixels as opposed to a physical size (i.e., "dimensionless"), while spots have a configurable physical size which is reflected by changes in pixel size during zoom interactions.

These data types also more closely reflect SpatialData's point and (circle) shape Elements, which will make SpatialData support more straightforward to implement.

Unified visualization properties for segmentations​

Vitessce supports both polygon- and bitmask-based segmentations. Previously, these different types of segmentations had to be configured differently, with polygon segmentations supporting a custom object like

{ opacity, radius, visible, stroked }

while bitmask segmentations supported same nested array of image properties as the image data type. In addition to exposing details about the internal implementations of the visualizations, these different configurations added complexity to the controls for segmentations, making them dependent on knowledge of the segmentation type. In part, these differences were related to the fact that the previous spatial view did not support a stroked: true/false property for bitmask-based segmentations. This limitation led to the need to pre-compute and redundantly store both filled and stroked segmentations:

The new spatial view implementation supports on-the-fly stroked: true/false for bitmask-based segmentations, with configurable stroke width. Using multi-level coordination, the same visual properties can be configured for both polygons and bitmasks. The boundaries of bitmask observations are computed on the GPU using a custom DeckGL layer.

Note: the GPU-based boundary visualization is based on a heuristic and is not perfect. Parts of bitmask boundaries that are close to image tile boundaries have the potential to be under-drawn (because the shader only has access to the data for one tile at a time). Boundaries that are concave may also be under-drawn. However, the boundary will never be over-drawn (i.e., pixels will not be rendered that would lie outside of the boundary if it had been rendered perfectly). The imperfections are more noticible at higher zoom levels. In the future, strategies such as multi-pass rendering of staggered tiles could resolve these limitations.

Demos​

We have configured several examples which make use of the spatialBeta and layerControllerBeta views:

• RGB image
• Multiplex + 3D image
• Comparison of volumetric rendering algorithms
• Visium with obsSpots data type
• osmFISH with obsPoints data type
• Bitmask-based obsSegmentations for cells and nuclei
• Bitmask-based obsSegmentations for functional tissue units
• Bitmask-based obsSegmentations for functional tissue units with coordinated opacity

Usage from Python​

We have implemented object-oriented APIs for the multi-level coordination model in both JavaScript and Python. The equivalent of the linkViewsByObject function from JS is the link_views_by_dict function in Python. This enables initialization of the coordination space using the same patterns in both languages. Jupyter notebooks demonstrating usage of the Python API are available in the documentation website for the vitessce Python package.

Next steps​

There are some remaining features to implement, including a mechanism for automatic initialization of the multi-level coordination values based on the data configuration. If testing goes smoothly, we plan to replace the current spatial and layerController views with their Beta counterparts. We would do this in a new major version due to the UI differences, but this would be implemented as a non-breaking change with automatic upgrade functions to internally convert previously-configured visualizations (in the same way we have implemented upgrade functions for all prior introductions of new config schema versions).

However, in Vitessce, many other things complicate both the development and production setups.

Luckily, there are new ways to address these complications due to improvements in language standards and build tooling. Thanks to discussions with Trevor Manz, we have been able to greatly improve our build setup.

In the following sections I describe each requirement and how we approached it in the new setup, in the hopes that it may be useful to others.

Previous build setup​

We previously used ejected create-react-app (CRA) scripts. The base CRA scripts work well for publishing web apps but are not meant for bundling a library. We had to eject the base scripts to add support for the library build. The resulting scripts consisted of two Webpack configurations which used many plugins that were difficult to maintain but supported many of our requirements.

Failed attempts​

While I previously made attempts to simplify our bundle scripts by using Rollup or Vite in place of CRA, these efforts were never successful. They usually required hours of debugging to find a perfect (ordered) set of plugins and parameter settings to support all of the requirements of Vitessce and its dependencies. It often felt like a game of whack-a-mole, fixing each issue just for another to emerge. After days of debugging, I achieved fully working setups on a few occasions. Unfortunately, even on these successful occasions (i.e., bundles, tests, and development server with no errors), I would still encounter issues preventing full adoption: the bundle size would be larger, or hot reloading in the development server would be unacceptably slow.

Incentives for this work​

As a graduate student writing academic software, there is little external incentive to spend time to improve these build processes (or address tech debt in general), as they take time away from the implementation of new features. It is difficult to communicate to others the current problems and the benefits that a smaller, faster, modular, or more modern JavaScript bundle would enable. It is not possible to quantify how many people didn't use your package because the bundle size was too large or its format was not compatible with direct inclusion in an HTML page via CDN.

Further, it is difficult to estimate how long this work will take. It often requires major code refactoring, during which any unrelated code changes by other team members will need to be re-implemented in the refactored code base. For external consumers of the NPM package, bundle structure or format changes often constitute breaking changes. All of these factors mean that this work is often perfomed outside of normal work hours so that it does not prevent the development of new features or conflict with the work of other team members.

The problems​

Our previous bundle was very large at 127 MB. The Unpkg CDN refuses to serve files from packages this large, preventing direct usage in HTML files and sites like ObservableHQ. Further, this prevents dynamically loading Vitessce from CDN for our Python or R widgets, for example. For library consumers, our unminified UMD build was 32.7 MB, slowing down their build processes and bloating their bundles. For end users, such a large bundle of JavaScript increases network usage and slows down page load times. For potential library adopters, seeing these numbers on NPM adds a sense of sticker-shock that could prevent adoption.

New build setup, by requirement​

Bundle size and individual component bundles​

One factor contributing to the overall bundle size is that we also included self-contained builds for individual React components. With such a large main bundle file, library consumers who only want to use individual components can achieve better development performance in their own environments by directly importing from the individual component bundles. However, this requirement adds redundant code to the bundle and contributes to the overall bundle size on NPM, as each individual component also requires a matrix of builds (minified/unminified, development/production, etc).

We addressed this by adopting a "monorepo" repository structure. This way, we can implement individual components in their own sub-packages which can be published to NPM as standalone bundles. Practically, this meant using PNPM for dependency management (as opposed to NPM), which implements many features for monorepo ("workspaces") management, including recursive installation and parallel script running.

Minimizing bundle steps in sub-packages​

As browser, NodeJS, and JavaScript module standards have matured, there is becoming less of a need for a complex bundle step in between source code development and published code deployment, as described by others. In short, when the consumer of your library is another JavaScript library, it makes more sense for them to import from the original source code. On the other hand, when the consumer of your library is an HTML page, it currently makes more sense for them to import a single-file, minified bundle. As browser module features like importmaps improve, perhaps these considerations will evolve.

In the meantime, bundlers can be useful for generating the single-file, minified bundles used in the browser. Historically, another role of a bundler has been to resolve non-JS and non-standard file imports. For instance, it has become common to want to import CSS, SCSS, SVG, JSON, GLSL, and Web Worker files via JS-like syntax (e.g., import obj from './file.json'). But such non-JS import statements are not supported in the JavaScript standard and therefore will result in an error when run in a browser. A bundler provides the ability to transform non-standard source code into browser-compatible code.

On the other hand, in order to publish re-usable source code to NPM and avoid bundle steps, we need to avoid these non-standard imports. We have achieved this by using only tsc as a simple build step in our sub-packages, with only minor exceptions. Where there are exceptions, the separation of concerns enabled by the monorepo structure allows more complex bundle steps to be scoped to the sub-repo code and its requirements, reducing complexity, improving performance, and allowing different bundler tools to be used in each context. For example, we make exceptions in the following places:

• The main vitessce package: For both backwards compatibility and to enable inclusion in HTML pages, we generate a matrix of single-file bundles using Vite and its library mode.
• The @vitessce/icons sub-package: To enable SVGs to be imported and transformed into React components automatically, we use Vite and vite-plugin-svgr to produce a plain JS bundle that can be consumed by other monorepo packages. (Note: This setup does not allow SVGs to be hot-reloaded during development, but we can probably assume they will not change frequently.)
• The @vitessce/workers sub-package: To enable Web Workers to be in-lined, we use Rollup + rollup-plugin-web-worker-loader. (This comes with the same hot reloading caveat as the @vitessce/icons case above.)

Website development with hot reloading​

We have two website requirements in this project: a minimal development website (an index.html page with a list of demos), and a documentation website. These websites both need to consume the library code, ideally with hot reloading support during development.

In the same way that the monorepo's separation of concerns facilitates different bundling tools scoped to each sub-repo, it also enables different development servers in each "sub-website". This allows us to use Vite development server mode and its React plugin for the minimal development website. Meanwhile, we can continue to use Docusaurus and its internal Webpack configuration for the documentation website. To enable hot reloading during development of both websites, rather than tsc, we run tsc --watch so that the dist directory of each sub-package is updated on every source code change. To further improve performance, we enable TypeScript's "incremental" mode with "incremental": true.

React 18​

To leverage the benefits of React 18, we use the TypeScript option "jsx": "react-jsx", which also comes with the minor benefit that JSX code can be written without explicitly importing React at the top of each file. To use React 18 in the context of Docusaurus and Material UI dependencies, which do not yet specify compatibility above React 17 in their peerDependencies, we use PNPM's peerDependencyRules in the root of the repository to override them and ensure that we are only depending on React 18 (as two versions of React cannot be included in the same web page).

Externalizing React​

Because the Vitessce React component is intended to be consumed by a React application (or other React component library), we can assume the consumer is using their own copy of React. In fact, including two copies of React in the same web application typically causes a website to crash. To make React an external dependency in the main development and production bundles, we can use the rollupOptions part of the Vite configuration:

...,
rollupOptions: {
  external: ['react', 'react-dom'],
  output: {
    globals: {
      react: 'React',
      'react-dom': 'ReactDOM',
    },
  }
},
...

Then, we can use the React-less Vitessce bundle in different contexts: UMD, ESM with importmap, or d3.require.alias.

Managing and publishing sub-packages​

To streamline the management of many sub-packages, we use PNPM and its meta-updater plugin. This allows syncing changes to every sub-package's package.json file, for instance to update the main "version" or particular dependency version values.

GLSL shaders and glslify shader modules​

To implement custom DeckGL shaders and extensions, we need to write GLSL shader code. We use the glslify shader module system to leverage published GLSL functions such as colormaps. Writing GLSL code in .glsl files enables IDE support but introduces the need for a build step when they are imported into JS files. The glslify module system also requires a build step to substitute strings like #pragma glslify: plasma = require("glsl-colormap/plasma") with the actual colormap function. As a workaround to prevent a build step, we can use the glslify CLI and simply include the shaders as strings in our JS files. This is not ideal, so we will need to look into ways to address the IDE support and modularity that were lost.

Style definitions​

To style our React components, we previously used a combination of SCSS and CSS-in-JS (via Material UI v4 + JSS). SCSS requires a build step to convert the source code to plain CSS code. Further, CSS imports in general also require a build step as they must be transformed from the JS-like import syntax to JS-compatible syntax. To address this, we have manually converted all SCSS to JSS using Material UI's makeStyles function and theming utilities. (Note: we have not upgraded to Material UI v5 which no longer uses JSS so this solution may not be advised in a new project.) Using JSS everywhere comes with the benefits of standardizing our styling code and facilitating direct style injection for consumers (i.e., no need for library users to include or think about a separate CSS file). While JSS auto-generates class names by default (which prevents naming conflicts), the jss-global-plugin enables global class names to be used to refer to elements defined by dependencies such as react-grid-layout.

Testing​

To modernize our testing infrastructure, we have adopted Jest + Vitest + React Testing Library and upgraded to the latest version of Cypress. However, there were a few hiccups, namely using code that depends on Web Workers in unit tests and serving test fixtures as static files during unit tests. Jest + Vitest use jsdom rather than a real web browser during unit tests, for efficiency. However, jsdom lacks support for web workers, which can be addressed by including the jsdom-worker library in the test setup script. To serve static files during unit testing, we can hook into Vite's development server and add static file routes via serve-static.

Bundle analysis​

It can be difficult to know what matters when it comes to the minified bundle size, and where to focus efforts when trying to reduce the size. Visualizing the bundle composition is a great way to understand which dependencies or source files are contributing. Luckily, rollup-plugin-visualizer is compatible with this build setup because Vite uses the rollup plugin API. From this visual analysis, I found that three different versions of the Zarr package were being included in the bundle, which I resolved using PNPM's overrides in the root of the monorepo.

Concluding thoughts​

Hopefully, most JavaScript developers only have a small subset of these requirements, but maintainers of more complex projects might find these notes useful. If you are interested in learning more, check out the Vitessce repository or a minimal example.

However, this problem has a broad scope and we've had to prioritize certain features over others: we have 142 open feature requests. This broad scope, coupled with the initial set of technologies used to generate data in the first phase of the HuBMAP Consortium (the first user of Vitessce), meant that we initially developed many features for scRNA-seq and bioimaging datasets. Early on, we also discovered a gap in web-based tools for dynamic rendering of bioimaging data on the client side, so members of our team took a detour to develop the Viv library. Meanwhile, the HuBMAP Consortium moved into its second phase, adding new labs and contributors, and generating a wide range of multi-modal datasets using technologies like SNARE-seq.

In parallel, the single-cell community is generalizing the cell-by-gene matrix from transcriptomics to represent other types of experimental data using "observation-by-feature" matrices (abbreviated FOM in the Feature and Observation Matrix Schema Working Group terminology). This is enabling an ecosystem of tools to be developed around the data structure, such as AnnData and Scanpy. Emerging formats such as MuData extend this paradigm yet further to support multi-modal use cases via container data structures that hold multiple observation-by-feature matrices. Multiple observation-by-feature matrices may share the feature axis (multi-batch), the observation axis (multi-modal), neither axis (diagonal), or some combination (mosaic). Our tools should support visualization of datasets that conform to any of these four cases.

Adapted from Figure 1, Argelaguet et al., Nature Biotechnology 2021

Updates in Vitessce​

Vitessce was initally coupled to the more specific cell-by-gene ideas, preventing straightforward use with non-transcriptomics or multi-modal datasets. We needed a major update to realize the FOM paradigm in Vitessce. In our latest update, we have changed how we represent data and how we configure mappings between data and views in order to support the visualization of datasets containing multiple observation-by-feature matrices.

Data types​

Data types no longer assume observation types like cell.

Instead, to facilitate re-use across observation types, we have split up each of the former data types into multiple more minimal data types. For example, obsLocations stores (x, y) coordinates per observation, obsEmbedding stores (dim1, dim2) coordinates per observation for a given embedding type, and obsSegmentations stores polygon vertices per observation. Simplified data types should also make it easier to implement new file types (including via plugins).

The former raster data type has been split into two new data types, image and obsSegmentations, to better reflect semantics rather than exposing the implementation differences between bitmask and polygon segmentations. (Internally, the fact that an obsSegmentations file type may return either polygons or bitmasks is handled by returning an obsSegmentationsType from the data loader class.)

For the full list of data types, please visit the updated Data Types and File Types page.

Dataset definitions​

Each of the datasets in the view configuration may now contain multiple file definitions corresponding to the same data type. Previously, dataset definitions were limited to one file per data type.

New coordination types​

The new coordination types obsType, featureType, and featureValueType enable "entity types" to be specified.

For example, if we have a cell-by-peak matrix from an scATAC-seq experiment, then the entities along the observation axis are cells and the entities along the feature axis are peaks:

dataset: 'my_atac_experiment',
obsType: 'cell',
featureType: 'peak',
featureValueType: 'count'

With these coordination types:

• more accurate strings like 100 cells x 200 peaks can be rendered in the interface (as opposed to the more abstract 100 observations x 200 features).
• specific values like featureType: 'gene' can be used to offer specific analysis functionality (e.g., differential expression tests that are valid only for transcriptomics experiments).
• the mapping between files and views can leverage both data types and entity types, as described in the next section.

Mapping between files and views​

File lookup from within a view now takes into account
(dataset, dataType, viewCoordinationValues).

The viewCoordinationValues refers to the coordination values that a particular view takes on at a given point in time. This is required to disambiguate between multiple files within a dataset which have the same dataType (now that each dataset may contain multiple files per data type).

The diagram below shows how the mapping between views and files would be performed for a configuration containing one dataset and three views.

• Heatmap
• Scatterplot
• Spatial
• Hide matches

Joint file types​

Joint file types can now be defined via functions which expand one file definition into an array of file definitions.

Name changes​

View type names have been updated to use obs and feature terminology (e.g., cellSets is now obsSets and genes is now featureList).

Coordination type names have been updated to use obs and feature terminology (e.g., cellColorEncoding is now obsColorEncoding and geneExpressionColormap is now featureValueColormap).

Exported constants​

Exported constants have been updated to reflect the changes to view types, file types, data types, and coordination types as described here.

File definitions​

File definitions no longer require the type (data type) property. Instead, we now map file types to data types internally.

View config schema​

The changes described in this blog post are reflected in the view config schema versions 1.0.15 and above. You can diff new and old versions to see the changes.

VitessceConfig constructor​

The VitessceConfig constructor now requires schemaVersion as a named parameter.

This will help us to understand which schema version the config was constructed against, enabling better warning messages and error handling. For instance, the API may show warnings or throw errors when passed invalid constants based on the specified view config schema version. New coordination types like obsType would be invalid when using an older schema version such as 1.0.7. We intend to update the constructors in R and Python to enforce this requirement as well.

For new code, we recommend specifying the latest view config schema version:

const vc = new VitessceConfig({ schemaVersion: "1.0.15", name: "My config" });

For a full list of valid schemaVersion values, see the table of schema versions.

Next steps​

We hope that these changes make it easier to configure, prepare data, and develop plugins for visualizing one or more related single-cell experiments with Vitessce. While we want our tools to be general enough to facilitate many use cases, we recognize the danger of over abstraction (e.g., preventing users from easily configuring Vitessce), and hope that our implementation strikes a useful balance. We welcome your feedback via issues and discussions.

Some additional use cases that we are excited about:

• using obsType: 'nucleus' to accurately refer to observations from single-nucleus experiments (similarly obsType: 'nucleolus', obsType: 'mitochondria', etc. as technologies for isolating subcellular components mature)
• using featureType: 'isoform' to accurately refer to features from isoform-resolved sequencing- and FISH-based experiments
• using featureType: 'topic' and featureValueType: 'probability' to enable visualization of topic model-based analysis results

Demos​

The following examples showcase how the updates described here can be used to perform integrative visualization of single-cell data.

Migration​

The Vitessce JavaScript, Python, R packages will continue to be backwards compatible with previous view config schema versions (i.e., using version: '1.0.1' in a JSON view config would result in the previously documented behavior despite upgrading the Vitessce package version).

For developers looking to upgrade to the latest version of the vitessce JavaScript package, please visit the upgrade guide.