Introduction

vitessce-image-viewer ("Viv")

A viewer for high bit depth, high resolution, multi-channel images using DeckGL over the hood and WebGL under the hood. To learn more about the "theory" behind this, look at this.

Build

To build the component alone via webpack use npm run-script build-component. To build the demo used for visual testing (seen on npm start), run npm run-script build-site.

Publish

To bump the version number, clean up/update the CHANGELOG.md, and push the tag to Github, please run npm version [major | minor | patch] depending on which you want. Then run ./publish.sh to publish the package/demo.

Development

Please install the Prettier plug-in for your preferred editor. (Badly formatted code will fail on Travis.)

For the demo, run npm start and you will be able to update the component and use the demo/src/App.js to visually test.

HTTP is acceptable but potentially slower than HTTP2. Our demo uses Google Cloud Storage, which is HTTP2 by default.

Due to difficulties around compiling shaders on Travis, unit tests and layer lifecycle tests are run locally as a pre-push hook. Travis runs a test build, linting, and prettier.

Browser Support

We support both WebGL1 and WebGL2 contexts, which should give near universal coverage. Please file an issue if you find a browser in which we don't work.

Documentation

Please navigate to viv.vitessce.io/docs to see full documenation.

Getting Started

Here are two snippets to help get you started with our higher-level components. For a more complete example of using these higher level components, look at the source code of the demo here, or look at the source code of the library here for building your own components with custom VivViews or custom deck.gl layers.

This snippet is the most basic view: a simple view of the data. With overviewOn=false, this will just be a single view of the data. Turn overviewOn=true for a picture-in-picture.

We also export DTYPE_VALUES and MAX_CHANNELS_AND_SLIDERS so you can get some information (array type, max) for each dtype of a loader (such as uint16/<u2) and the number of channels the current release of Viv supports, respectively.

import { createZarrLoader, PictureInPictureViewer, createOMETiffLoader } from '@hubmap/vitessce-image-viewer';

/* Zarr Loader */
const zarrInfo = {
  url: `https://vitessce-data.storage.googleapis.com/0.0.25/master_release/spraggins/spraggins.mxif.zarr`,
  dimensions: [
    { field: 'channel', type: 'nominal', values: [
        'DAPI - Hoechst (nuclei)',
        'FITC - Laminin (basement membrane)',
        'Cy3 - Synaptopodin (glomerular)',
        'Cy5 - THP (thick limb)'
      ]
    },
    { field: 'y', type: 'quantitative', values: null },
    { field: 'x', type: 'quantitative', values: null }
  ],
  isPublic: true,
  isPyramid: true,
};
const loader = await createZarrLoader(zarrInfo);
/* Zarr loader */
// OR
/* Tiff Loader */
const url =
  'https://vitessce-demo-data.storage.googleapis.com/test-data/deflate_no_legacy/spraggins.bioformats.raw2ometiff.ome.tif';
const loader = await createOMETiffLoader({ url, offsets: [], headers: {} });
/* Tiff Loader */

const sliders = [[0,2000], [0,2000]];
const colors = [[255, 0, 0], [0, 255, 0]];
const isOn = [true, false];
const selections = [{ channel: 1 }, { channel: 2 }];
const initialViewState = {
  height: 1000,
  width: 500,
  zoom: -5,
  target: [10000, 10000, 0].
};
const colormap = '';
const overview = {
  boundingBoxColor: [0, 0, 255]
};
const overviewOn = false;
const PictureInPictureViewer = (
  <PictureInPictureViewer
    loader={loader}
    sliderValues={sliders}
    colorValues={colors}
    channelIsOn={isOn}
    loaderSelection={selections}
    initialViewState={initialViewState}
    colormap={colormap.length > 0 && colormap}
    overview={overview}
    overviewOn={overviewOn}
  />
);

If you wish to use the SideBySideViewer, simply replace PictureInPictureViewer with SideBySideViewer and add props for zoomLock and panLock while removing overview and overviewOn.

More in Depth

API Structure

If you are looking to go deeper than just instantiating a loader + PictureInPictureViewer/SideBySideViewer component, this information may be useful - otherwise, hopefully the APIs + docs for those are sufficient (if not, please feel free to open an issue just to ask a question - we will reply!).

Viewer

There are 3 "Viewers", although this is something of a naming-overload: PictureInPictureViewer, SideBySideViewer, and VivViewer (React only for now - non-React coming soon). The VivViewer serves as the workhorse. It takes as input and handles managing multiple views (for example, the multiple images in the PictureInPictureViewer). It also handles resizing and passing props down to the DeckGL component. The PictureInPictureViewer and SideBySideViewer are then higher level functional components that return a VivViewer configured with the proper Views (see below) of the image(s) to be rendered.

View

To manage where in the cooridnate space the image is rendered, we have developed our own wrappers around deck.gl's Views to provide a clean API for using the VivViewer component and giving some out-of-the-box support for certain things. A View must inherit from a VivView and implement/override its methods, including a filter for updating ViewState, instantiating a View , and rendering Layers into that View. Our Views that we have implemented exist to support our components - for example, the OverviewView is used in the PictureInPictureViewer to provide a constant overview of the high resolution image, including a bounding box for where you are viewing and a border (both implemented as layers). The SideBySideView has built-in-support for locked/unlocked zoom/pan in the SideBySideViewer.

Layer

This is the lowest level of our rendering API. These are deck.gl Layers that we use in our Views, but can also be used in any way with a DeckGL component/Deck object. The workhorse is the XRLayer (eXtended Range Layer) which handles the actual GPU rendering for the VivViewerLayer and StaticImageLayer.

Loader

Finally, we also export our loaders, OMETiffLoader, and ZarrLoader, as well as utility functions for creating them, OMEZarrReader, createZarrLoader, and createOMETiffLoader, which expose a simple API to get a loader as opposed to the more complicated API/data structures of the loaders themselves. These loaders are tightly integrated with the rest of the API, providing metadata about the data such as size and resolution when needed.

OME-TIFF Loading

OME-TIFF Loading

Viv has the ability to load OME-TIFF files directly through a simple API:

import { createOMETiffLoader } from '@hubmap/vitessce-image-viewer';

const url =
  'https://vitessce-demo-data.storage.googleapis.com/test-data/deflate_no_legacy/spraggins.bioformats.raw2ometiff.ome.tif';
const loader = await createOMETiffLoader({ url, offsets: [], headers: {} });

A bit is going on under the hood here, though. Here are some of those things:

1. First and foremost, if the tiff has many channels, then you will need to provide the offsets to each IFD.
   The createOMETiffLoader takes in a list of offsets via offsets so that they can be stored anyhere i.e https://vitessce-demo-data.storage.googleapis.com/test-data/deflate_no_legacy/spraggins.bioformats.raw2ometiff.offsets.json. To get this information, you may use this docker container and push the output to the url.

2. Related to the above, if your tiff is a pyramid from bioformats, then there are two potential routes:

• bioformats recently came out with a new specification that uses the SubIFD to store most of the offset information. You should be good with this, but potentially may need to still use the docker contianer if there are a lot of image planes (i.e z, t, and channel stack) in the original data.
• Older bioformats pyramids need the above docker container to run properly.

3. We are not experts on the OMEXML format, but we make a reasonable attempt to parse the OMEXML metadata for channel names, z stack size, time stack size etc. Please open a PR against the OMEXML class if it fails for your use case.

4. If you are interested in generating your own image pyramids, we use the new bioformats image pyramid from here and here - we have this docker container for that purpose. Both viv and this new bioformats software are under development, so there will likely be tweaks and changes as time goes on, but the current implementation-pairing should be stable (it currently backs the public OME-TIFF demo as well as one of the not-public ones). Additionally, the intermediary n5 format can be quickly ported to zarr for analysis locally. Please use zlib as LZW is not great on the browser, it seems. If you need LZW please open an issue. Here is a snippet to help get you started if you have a folder /my/path/test-input/ containing OME-TIFF files:

# Pull docker images
docker pull portal-contianer-ome-tiff-offsets:0.0.2
docker pull portal-contianer-ome-tiff-tiler:0.0.2

# Run docker images
# For images that have large z/t/channel stack combinations.
docker run \
    --name offsets \
    --mount type=bind,source=/my/path/test-input/,target=/input \
    --mount type=bind,source=/my/path/test-output/,target=/output \
    portal-contianer-ome-tiff-offsets:0.0.2
# For large resolution images, to be downsampled and tiled.
docker run \
    --name tiler \
    --mount type=bind,source=/my/path/test-input/,target=/input \
    --mount type=bind,source=/my/path/test-output/,target=/output \
    portal-contianer-ome-tiff-tiler:0.0.2

# Push output to the cloud
gsutil -m cp -r /my/path/test-output/ gs://my/path/test-output/

Note that if your tiff file is large in neither channel count nor resolution, you can simply load it in viv directly without passing in offsets or running this pipeline.

Finally we are in the experimental stages of supporting RGB images. Right now we only support de-interleaved images.
One of the biggest issues we have is lack of (what we know to be) properly formatted sample OME-TIFF data. Please open an issue if this is important to your use-case.

Viewers (React Components)

Views

Utility Methods

Layers

Loaders

Layers (Internal)

Overview Of Viv Image Rendering

Author's (Ilan's) note: This is my understanding of the problem and our solutions. If I am mistaken, please feel free to make a pull request. I had literally no background in imaging or rendering before taking this on, but continue to be exhilarated as problems and solutions present themselves.

Images and Computers

On some level, images can be thought of simply as height x width x channels matrices containing numbers as entries. Computers have three little lights per pixel on the monitor, each of which can take on 256 different values (0-255). For this and definitely other reasons, most images have three (or four) channels, RGB (or RGBA where A controls the alpha blending), each with 256 possible values. However, this is somewhat arbitrary - our microscopy data, for example, can have many channels, each corresponding to a different antibody staining that is imaged, and each taking on a far greater range than the standard 256 options. Beyond microscopy, many normal DSLR cameras can take photographs that go beyond the 256 options as well. This data (in a standard 256 bit depth format) is then often compressed (in a lossless, like .png files via DEFLATE, or in a lossy, like .jpeg files with the Discrete Cosine Transform, manner) and ready to be transported over the internet or opened on a computer (in both cases, the data must be first decompressed before it can be viewed).

We have thus established that standard image files are 8-bit (256 value-options) height x width x 3 or height x width x 4 matrices, often compressed. Our data, as noted, is different, often being 16-bit or 32-bit and taking on more than the standard 3 or 4 channels (or possibly less).

Viewing High Resolution Images

A challenge of working in a browser is that you don't have access to data unless it is fed over the internet to you. Thus if you have a very large image in height and width, storing the entire image in memory is tough. For this reason, people have developed various image tiling/pyramimd schemes. Tiling is the manner in which the underlying image is broken up into smaller images and the pyramid represents the image at increasingly downsampled resolutions. This allows you to efficiently view the image in different locations at different zoom levels, given the current viewport in your browser, the most famous example of this being OpenSeaDragon.

For example, if you are very zoomed in on an image, say the top-left corner of 0:512 in both height and width on the original image, you only need that portion of the image served to your browser for you to view it. You don't need the rest of the image at this moment and should you pan to it, you can always fetch it from the server which contains the output of the tiling/pyramid generation.

Our Data: Problems and solutions

We thus have two main problems: our data is often in a non-standard format, and it is very large. Therefore, we need efficient methods of rendering and serving.

1. Rendering >256 bits of data per channel, with more than 3 channels

To tackle this we look to WebGL, a Javascript API for using OpenGL in the browser, as the core technology and also DeckGL as the high-level API for our application to communicate "business logic" to WebGL. WebGL is very flexible and will allow us to pass down data (i.e a flattened image matrix) with 16 and 32 bit precision to shaders, which can do graphics rendering and basic arithmetic using the GPU very efficiently (this is why deep learning works so well as it relies almost entirely on the GPU). This extended range of 16 or 32 bits can then be mapped by the user to a subset of down to a "0 to 1" range that WebGL can render. Additionally, we can pass down a flag to tell WebGL which color each channel should be mapped to, thus allowing us to store each channel separately.

2. Serving Data Efficiently

Most high resolution tiling image viewing systems rely on jpeg files, which can obtain 10:1 compression ratios with imperceptible loss of quality, to serve the image tiles. The jpeg files are very small and therefore there is minimal bottleneck with network requests. However, this file type/compression scheme is not an easy option at this moment for more than 8 bits of data per channel in current browsers as they do not ship with the decompression algorithm for more than 8 bit data. However libraries like zarr.js and geotiff.js could eventually use an implementation to decode said data. It also it clear that we wish to lose any precision in what is essentially a scientific measurement. The other popular image file type, png, achieves considerably worse compression (around 2:1) but is lossless.

Our approach to solving this is twofold.

First, we store the image in a compressed, chunked (i.e tiled), "raw" format that can be decoded in the browser. For this client-side work, we currently use zarr with zarr.js and geotiff.js.

zarr provides an interface for reading and writing chunked, compressed, N-dimensional arrays. In other words, zarr handles chunking large N-dimensional arrays into compressed file blobs, each flattened in row major order by default. Zarr uses a key-value store (often a file system, where filenames are they keys and the values are the blobs), which makes assembling/indexing the original array efficient, since each chunk is just a small subset of the original array. Taking a step back to see why this is helpful, let us note that images are really (flattened) pixel arrays with a notion of width and height. Thus, JavaScript APIs like WebGL and 2D Canvas rely on flattened array buffers to render images. Since zarr can be used to chunk and store a very large image, we can lazily load the zarr-based image data in the browser. This leads to large performance benefits (since pixels can be fetched on a by-chunk basis), and is largely responsible for why zarr is becoming a popular image format. It is important to note that zarr is originally a python library, but there are an increasing number of implementations in other languages since zarr at its core is a pipeline for storing and accessing blobs of data. We heavily rely on (and have contributed to) zarr.js, which is a TypeScript port. By chunking and downsampling large images into zarr stores, we create tiled pyramids for lazily loading "raw" image data in the browser. Therefore, given a particular level of the pyramid (i.e. zoom level), we can use the zarr.js API directly to request and decode individual compressed chunks (tiles) on the fly. In addition, by chunking additional axes of N-dimensional images (i.e. time, channel), we can lazily load specific image panes from the source (directly!). This provides much finer control over which blobs to access, in contrast to other image formats. TIFF files are stored somewhat similarly but rather using the tiff file format so all the data lives in one file and then geotiff.js makes range requests to get a tile's worth of flattened image data. All this is to say, if you have a method that can store image data in a way that the client can efficiently request a tile's worth of data in one request, you can use that as the "backend" for this project (for example, if you had a javascript hdf5 reader, you could probably use that!).

Second, we believe that HTTP/2 can help speed up the requests. Even though the requests are not particularly large (normally far less than 2MB per tile with a 512x512 tile size, 16bits of data), there are many of them and in normal HTTP/1, they block one another after a certain point. HTTP/2 more or less circumvents this blocking problem. A possible extension of this is to use gRPC (which uses HTTP/2 under the hood together with Protobuf) but for now, HTTP/2 alone works well enough. We have not tested this too much officially, but we use GCS and s3 for our applications, and they perform comparably, with a slight edge to GCS potentially. Below you will find a very rough side-by-side comparison of s3 vs GCS. THe first image is the first request for to the zarr store for image data, and the second image is the last.

s3

GCS

Pitfalls of other routes

• One possible solution is to use an already existing tiling service, like mapbox for creating/hosting our data or OpenSeaDragon with DeepZoom (or some combination of the two). However, these services does not support high bit-depth-per-channel images. To use these then, we could pack 32 bits per channel (for high bit-depth-per-channel microscopy images) across the total 32 bits of the RGBA of a standard image file and then serve that and decode it, doing some sort of mapping on the shaders. Besides the kludge-y nature of this, we don't save anything really in terms of time as far as transporting the data is concerned, assuming we use png - we still need to serve the same amount of data. And, if we attempted to do this with jpeg format, it's not clear (to me, a novice) how the color-space transformation, DCT, and entropy encoding will react to what is no longer color values, but rather a bit packing scheme.
• Vector tiles from mapbox are another option, relying on Protobuf, but our data is not vector data. This raises questions of how we can get the data into that format, and, once it is in that format, why we would not just serve it ourselves over gRPC which appears to be faster.
• OMERO relies on a client-server model in which the server renders a JPEG/PNG 8-bit RGB image from a source TIFF file. This does not fit the model within which we are trying to develop, in which the server serves chunks of n-bit data which are then rendered on the client.

• There is a high resolution image format called OpenEXR that supports our data type somewhat natively; that is, it supports high bit-depth-per-channel natively and also supports arbitrary numbers of channels. There is a ported C++ library as well as a native Javascript library for handling these files. There are potentially others. However, unless we use a lossy compression with them, we again don't really gain anything (of note is that three.js' implementation, the aforementioned javascript library, does not support lossy compression). A nice thing about the format, though, is that it appears to support image pyramids/tiling natively, although it is not a guarantee that any library we might use does as well (or at least easily). There are two main potential pitfalls with EXR:

  1. The OpenEXR format appears to only support three different data types (16 and 32 bit floats, and 32 bit unsigned integers), which means we probably have to kludge our data a bit and will need two different image-serving systems if we get 8-bit data; granted, making an image pyramid requires some loss of precision anyway (it is essentially a statistical process), so getting from native integers in the OME-TIFF to floating point is not the end of the world, but will likely require we lose precision.
  2. One of the lossy compression schemes it supports is lossy in so far as it just lops of the least significant 8 bits of data (scroll down to the data compression sections here). The others, while less hacky in this sense, do not work for 32 bit data

Helpful Libraries for Data Processing

• Bioformats provides pyramid-writing capabilities for OME-TIFF files. With these, you should configure Viv to use geotiff.js.
• Libvips is another, perhaps lower level, library that can be of service for similar needs.
• vitessce-data also contains similar code for writing zarr pyramids.