Integration Guide

Integration Overview

The Zebrafish Atlas is designed to integrate with existing research workflows, tools, and databases. This guide outlines various integration options and provides recommendations for different use cases.

Integration Methods

• RESTful API: Programmatic access to retrieve and contribute data
• Data Download: Bulk export of datasets for local processing
• Embeddable Widgets: Interactive components for websites and applications
• Automated Pipelines: Scheduled data exchange and synchronization
• Ontology Mappings: Standardized terminology for cross-platform data integration

API Integration

The Zebrafish Atlas provides a comprehensive RESTful API for integrating with your applications, scripts, and analysis pipelines.

API Integration Steps

1. Obtain API access: Request API permissions through your account settings
2. Generate an API token: Create a secure token for authentication
3. Explore available endpoints: Review the API documentation
4. Develop your integration: Use standard HTTP libraries in your language of choice
5. Test and implement: Start with development endpoints before moving to production

API Integration Examples

Common API integration scenarios include:

Data Retrieval Script (Python)

import requests
import json
import os

# Configuration
API_BASE_URL = "https://zebrafishatlas.example.org/api/v1"
API_TOKEN = os.environ.get("ZEBRAFISH_API_TOKEN")  # Store token in environment variable

# Set up headers
headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}

# Function to download images matching criteria
def download_zebrafish_images(age_type, cut_type, output_dir):
    # Create output directory if it doesn't exist
    os.makedirs(output_dir, exist_ok=True)
    
    # Get list of images matching criteria
    params = {
        "age_type": age_type,
        "cut_type": cut_type,
        "limit": 100
    }
    
    response = requests.get(f"{API_BASE_URL}/images/", headers=headers, params=params)
    
    if response.status_code == 200:
        data = response.json()
        print(f"Found {data['count']} images, downloading...")
        
        # Download each image
        for image in data["results"]:
            image_id = image["id"]
            filename = os.path.basename(image["file_path"])
            output_path = os.path.join(output_dir, filename)
            
            # Get the high-resolution image
            download_url = f"{API_BASE_URL}/images/{image_id}/download/"
            download_response = requests.get(download_url, headers=headers, stream=True)
            
            if download_response.status_code == 200:
                with open(output_path, 'wb') as f:
                    for chunk in download_response.iter_content(chunk_size=8192):
                        f.write(chunk)
                print(f"Downloaded: {filename}")
            else:
                print(f"Failed to download {filename}: {download_response.status_code}")
        
        print(f"Download complete. {len(data['results'])} files saved to {output_dir}")
        return True
    else:
        print(f"Error: {response.status_code}")
        print(response.json())
        return False

# Example usage
download_zebrafish_images("5dpf", "Sagittal", "./downloads/5dpf_sagittal")

Automated Data Submission (JavaScript)

// Example Node.js script for automated data submission
const axios = require('axios');
const fs = require('fs');
const path = require('path');
const FormData = require('form-data');

// Configuration
const API_BASE_URL = 'https://zebrafishatlas.example.org/api/v1';
const API_TOKEN = process.env.ZEBRAFISH_API_TOKEN;

// Function to upload a batch of images
async function uploadBatch(imageDir, metadata) {
    try {
        // Read all image files in directory
        const files = fs.readdirSync(imageDir)
            .filter(file => /\.(tif|tiff|jpg|jpeg|png)$/i.test(file))
            .map(file => path.join(imageDir, file));
        
        if (files.length === 0) {
            console.log('No image files found in directory');
            return;
        }
        
        console.log(`Found ${files.length} files, preparing for upload...`);
        
        // Create form data with multiple files
        const formData = new FormData();
        
        // Add each file to the form
        files.forEach((file, index) => {
            formData.append('files[]', fs.createReadStream(file));
        });
        
        // Add metadata
        formData.append('age_type', metadata.ageType);
        formData.append('cut_type', metadata.cutType);
        formData.append('species_category', metadata.speciesCategory);
        formData.append('segmentation_progress', metadata.segmentationProgress || 0);
        formData.append('magnification', metadata.magnification || '');
        formData.append('treatment', metadata.treatment || '');
        
        // Send request
        const response = await axios.post(`${API_BASE_URL}/bulk-upload/`, formData, {
            headers: {
                ...formData.getHeaders(),
                'Authorization': `Bearer ${API_TOKEN}`
            },
            maxContentLength: Infinity,
            maxBodyLength: Infinity
        });
        
        console.log(`Upload completed with ${response.data.results.success} successes and ${response.data.results.failed} failures`);
        
        if (response.data.results.errors.length > 0) {
            console.log('Errors:');
            response.data.results.errors.forEach(error => {
                console.log(`- ${error.file}: ${error.error}`);
            });
        }
        
        return response.data;
    } catch (error) {
        console.error('Upload failed:', error.response ? error.response.data : error.message);
        return null;
    }
}

// Example usage
const metadata = {
    ageType: '3dpf',
    cutType: 'Coronal',
    speciesCategory: 'Transgenic',
    magnification: '40x',
    treatment: 'GFP',
    segmentationProgress: 0
};

uploadBatch('./new_images/3dpf_coronal', metadata);

Data Pipelines

For ongoing research projects, you can set up automated data pipelines between your systems and the Zebrafish Atlas.

Pipeline Components

• Data Extraction: Retrieving data from the Zebrafish Atlas or your local system
• Data Transformation: Converting between formats and applying processing steps
• Data Loading: Importing data into the destination system
• Scheduling: Automating pipeline execution at regular intervals
• Monitoring: Tracking pipeline performance and addressing issues

Sample Pipeline Architecture

Your Microscopy System → Data Preprocessing → Metadata Enrichment → Zebrafish Atlas API
      ↓                                                                   ↓
Local Storage ← Quality Check ← Confirmation & Validation ← Upload Response
      ↓
Analysis Tools

Pipeline Best Practices

1. Idempotent Operations: Design pipelines to safely handle retries without duplicating data
2. Error Handling: Implement robust error detection and recovery mechanisms
3. Logging: Maintain detailed logs for troubleshooting and auditing
4. Throttling: Respect API rate limits with appropriate delays and batching
5. Validation: Verify data integrity before and after transfers

Recommended Pipeline Tools

Embedding and Widgets

The Zebrafish Atlas provides embeddable components that you can integrate into your websites, applications, or presentations.

Available Widgets

• Image Viewer: Interactive viewer for individual slides
• Data Table: Browsable table of available images
• Anatomical Browser: Interactive anatomical structure browser
• Search Widget: Search interface for the Zebrafish Atlas

Image Viewer Embedding

To embed the image viewer in your web page:

<!-- Add the Zebrafish Atlas viewer script to your page -->
<script src="https://zebrafishatlas.example.org/static/js/embed.js"></script>

<!-- Create a container for the viewer -->
<div id="zebrafish-viewer" style="width: 100%; height: 600px;"></div>

<!-- Initialize the viewer with your desired configuration -->
<script>
  const viewer = new ZebrafishViewer({
    container: 'zebrafish-viewer',
    imageId: 'bdc5a24e-9f3e-42f1-a53c-6c822a0f7abe', // Specific image ID
    showControls: true,
    showInfoPanel: true,
    allowFullscreen: true,
    apiKey: 'YOUR_PUBLIC_API_KEY' // Optional, for private images
  });
  
  // You can also listen for events
  viewer.on('load', () => {
    console.log('Viewer loaded successfully');
  });
  
  viewer.on('error', (error) => {
    console.error('Viewer error:', error);
  });
</script>

IFrame Embedding

For simpler integration, you can use an iframe:

<iframe 
  src="https://zebrafishatlas.example.org/embed/viewer/bdc5a24e-9f3e-42f1-a53c-6c822a0f7abe?controls=1&info=1" 
  width="100%" 
  height="600" 
  frameborder="0" 
  allowfullscreen
></iframe>

Data Table Embedding

<!-- Add the Zebrafish Atlas table script to your page -->
<script src="https://zebrafishatlas.example.org/static/js/embed-table.js"></script>
<link rel="stylesheet" href="https://zebrafishatlas.example.org/static/css/embed-table.css">

<!-- Create a container for the table -->
<div id="zebrafish-table" style="width: 100%;"></div>

<!-- Initialize the table with your desired configuration -->
<script>
  const table = new ZebrafishTable({
    container: 'zebrafish-table',
    ageType: '5dpf', // Optional: filter by age
    cutType: 'Sagittal', // Optional: filter by orientation
    speciesCategory: 'Wildtype', // Optional: filter by species
    compact: true, // Use compact view
    apiKey: 'YOUR_PUBLIC_API_KEY' // Optional, for private images
  });
</script>

Integration with Analysis Tools

The Zebrafish Atlas can be integrated with popular scientific image analysis and bioinformatics tools.

ImageJ/Fiji Integration

For integration with ImageJ/Fiji:

1. Download our ImageJ plugin
2. Place the JAR file in your ImageJ/Fiji plugins directory
3. Restart ImageJ/Fiji
4. Configure the plugin with your API credentials
5. Use the "Zebrafish Atlas" menu to access images directly in ImageJ

R Integration

For R users, our R package provides convenient access:

# Install the package
install.packages("zebrafishatlasR")

# Load the library
library(zebrafishatlasR)

# Configure API access
zfa_configure(api_token = "YOUR_API_TOKEN")

# Query available images
images <- zfa_query_images(age_type = "24hpf", cut_type = "Coronal")

# Download an image
img_data <- zfa_download_image(images$id[1])

# Load into an R image object
library(imager)
img <- load.image(img_data$path)

# Display the image
plot(img)

Python Integration

For Python-based analysis, use our Python package:

# Install the package
pip install zebrafishatlas

# Import the library
import zebrafishatlas as zfa

# Configure API access
zfa.configure(api_token="YOUR_API_TOKEN")

# Query available images
images = zfa.query_images(age_type="24hpf", cut_type="Coronal")

# Download an image
img_path = zfa.download_image(images[0]['id'])

# Load with popular libraries
import numpy as np
from skimage import io
img = io.imread(img_path)

# Display the image
import matplotlib.pyplot as plt
plt.imshow(img)
plt.show()

MATLAB Integration

For MATLAB users:

% Add the toolbox to your path
addpath('/path/to/zebrafishatlas-matlab');

% Initialize the client
zfa = ZebrafishAtlas('YOUR_API_TOKEN');

% Query images
images = zfa.queryImages('ageType', '24hpf', 'cutType', 'Coronal');

% Download an image
[imagePath, metadata] = zfa.downloadImage(images(1).id);

% Load and display the image
img = imread(imagePath);
imshow(img);

Batch Processing

For large-scale analysis or data contribution, batch processing capabilities are available.

Batch Download

To download multiple images in a single operation:

import zebrafishatlas as zfa
import os

# Configure API access
zfa.configure(api_token="YOUR_API_TOKEN")

# Define batch parameters
batch_params = {
    "age_types": ["24hpf", "48hpf", "5dpf"],
    "cut_type": "Sagittal",
    "species_category": "Wildtype",
    "output_dir": "./zebrafish_data"
}

# Create output directory
os.makedirs(batch_params["output_dir"], exist_ok=True)

# Download batch
result = zfa.batch_download(
    age_types=batch_params["age_types"],
    cut_type=batch_params["cut_type"],
    species_category=batch_params["species_category"],
    output_dir=batch_params["output_dir"],
    include_metadata=True,
    max_images_per_category=10
)

print(f"Downloaded {result['total_files']} files")
print(f"Total data size: {result['total_size_mb']:.2f} MB")

Batch Upload

To upload multiple images with consistent metadata:

import zebrafishatlas as zfa
import os
import glob

# Configure API access
zfa.configure(api_token="YOUR_API_TOKEN")

# Define common metadata
common_metadata = {
    "age_type": "5dpf",
    "cut_type": "Coronal",
    "species_category": "Transgenic",
    "magnification": "40x",
    "treatment": "GFP",
    "tags": ["transgenic", "neural", "5dpf"]
}

# Find all image files in a directory
image_dir = "./new_zebrafish_images"
image_files = glob.glob(os.path.join(image_dir, "*.tif"))

print(f"Found {len(image_files)} images to upload")

# Perform batch upload
result = zfa.batch_upload(
    file_paths=image_files,
    metadata=common_metadata,
    parallel=True,
    max_concurrent=5
)

print(f"Successfully uploaded {result['success']} of {result['total']} images")
if result['failed'] > 0:
    print(f"Failed uploads: {result['failed']}")
    for error in result['errors']:
        print(f"- {error['file']}: {error['message']}")

Batch Processing Best Practices

• Chunking: Process data in manageable chunks (e.g., 10-100 files per batch)
• Parallelization: Use parallel processing for faster execution, but respect API rate limits
• Checkpointing: Implement checkpoints to resume interrupted operations
• Validation: Verify data integrity before and after batch operations
• Error Handling: Implement robust error handling and reporting mechanisms

Database Integration

The Zebrafish Atlas can be integrated with other biological databases and research repositories.

Compatible Databases

Integration via Ontology Mapping

Zebrafish Atlas data can be linked to external databases through standard ontologies:

• ZFA: Zebrafish Anatomy Ontology
• ZFS: Zebrafish Developmental Stages
• GO: Gene Ontology
• UBERON: Cross-species Anatomy Ontology

Example of accessing data with ontology mappings:

import zebrafishatlas as zfa

# Configure API access
zfa.configure(api_token="YOUR_API_TOKEN")

# Query by ZFA term
zfa_term = "ZFA:0000119"  # Olfactory bulb
results = zfa.query_by_ontology(ontology_term=zfa_term)

print(f"Found {len(results)} images with {results[0]['structure_name']} ({zfa_term})")

# Get cross-referenced information
for image in results[:5]:
    print(f"Image: {image['title']}")
    print(f"  - ZFA term: {image['zfa_term']}")
    print(f"  - UBERON equivalent: {image['uberon_term']}")
    print(f"  - Associated genes: {', '.join(image['associated_genes'])}")
    print()

Institutional Repository Integration

For research institutions, we offer custom integration solutions to connect the Zebrafish Atlas with institutional repositories:

• OAI-PMH Endpoint: Standard protocol for metadata harvesting
• Custom Data Connectors: Tailored solutions for specific repository systems
• Metadata Mapping: Translation between institutional schemas and Zebrafish Atlas metadata