Note
Go to the end to download the full example code. or to run this example in your browser via Binder
Full pipeline on real data: derotation and center of rotation estimation#
This example shows how to run the full derotation pipeline on real data, including: a TIFF movie, analog signals (.npy), and a CSV file with stimulus randomization. The pipeline performs automatic estimation of the center of rotation, derotation, and saves intermediate results and visualizations in an output folder.
- Steps performed:
Load configuration from a YAML file and update file paths
- Initialize the FullPipeline class with:
A TIFF movie to derotate
A .npy file with analog signals from ScanImage and a step motor (ordered as: frame clock, line clock, rotation on, rotation ticks)
A CSV file with stimulus randomization
- Run the full pipeline, which:
Interpolates rotation angles per acquired line from analog signals
Estimates the center of rotation using a Bayesian optimization approach
Derotates the movie based on estimated parameters
Saves the derotated movie and rotation angles
At the end, we visualize key plots generated during processing and explore the results saved in the output folder.
Imports#
from pathlib import Path
import matplotlib.pyplot as plt
from derotation.analysis.full_derotation_pipeline import FullPipeline
from derotation.config.load_config import load_config, update_config_paths
from derotation.sample_data import fetch_data
Load and update configuration#
We define paths relative to the current working directory
current_module_path = Path.cwd()
config = load_config()
config = update_config_paths(
config=config,
tif_path=str(fetch_data("rotation_sample.tif")),
aux_path=str(fetch_data("analog_signals.npy")),
stim_randperm_path=str(fetch_data("stimulus_randperm.csv")),
output_folder=str(current_module_path),
)
Initialize pipeline#
pipeline = FullPipeline(config)
INFO 2025-07-24 15:36:25 PM - INFO - fancylog.py:451
MainProcess fancylog.py:451 -
Starting logging
2025-07-24 15:36:25 PM INFO 2025-07-24 15:36:25 PM - INFO - fancylog.py:451
MainProcess fancylog.py:451 -
Starting logging
INFO 2025-07-24 15:36:25 PM - INFO - fancylog.py:452
MainProcess fancylog.py:452 -
Not logging multiple processes
INFO 2025-07-24 15:36:25 PM - INFO - fancylog.py:452
MainProcess fancylog.py:452 -
Not logging multiple processes
INFO 2025-07-24 full_derotation_pipeline.py:164
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:164
- Loading
data...
INFO 2025-07-24 full_derotation_pipeline.py:164
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:164
- Loading
data...
INFO 2025-07-24 full_derotation_pipeline.py:165
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:165
- Loading
/home/runner/.de
rotation/data/ro
tation_sample.ti
f
INFO 2025-07-24 full_derotation_pipeline.py:165
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:165
- Loading
/home/runner/.de
rotation/data/ro
tation_sample.ti
f
INFO 2025-07-24 full_derotation_pipeline.py:186
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:186
- Number of
rotations: 2
INFO 2025-07-24 full_derotation_pipeline.py:186
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:186
- Number of
rotations: 2
INFO 2025-07-24 full_derotation_pipeline.py:194
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:194
- Rotation
direction:
speed 200
direction
-1 1
1 1
INFO 2025-07-24 full_derotation_pipeline.py:194
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:194
- Rotation
direction:
speed 200
direction
-1 1
1 1
INFO 2025-07-24 full_derotation_pipeline.py:253
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:253
- Deleting
previous debug
plots...
INFO 2025-07-24 full_derotation_pipeline.py:253
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:253
- Deleting
previous debug
plots...
INFO 2025-07-24 full_derotation_pipeline.py:263
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:263
- Dataset
rotation_sample
loaded
INFO 2025-07-24 full_derotation_pipeline.py:263
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:263
- Dataset
rotation_sample
loaded
INFO 2025-07-24 full_derotation_pipeline.py:264
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:264
- Filename:
derotated
INFO 2025-07-24 full_derotation_pipeline.py:264
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:264
- Filename:
derotated
Peek into the loaded data#
Loaded movie shape: (270, 256, 256)
Useful attributes before running the pipeline
print(f"Number of frames: {pipeline.num_frames}")
print(f"Lines per frame: {pipeline.num_lines_per_frame}")
print(f"Rotation speeds: {pipeline.speed} deg/s")
print(f"Rotation direction: {pipeline.direction} (−1 = CCW, 1 = CW)")
print(f"Estimated number of full rotations: {pipeline.number_of_rotations}")
Number of frames: 270
Lines per frame: 256
Rotation speeds: [200 200] deg/s
Rotation direction: [-1 1] (−1 = CCW, 1 = CW)
Estimated number of full rotations: 2
Run the full pipeline#
pipeline()
INFO 2025-07-24 full_derotation_pipeline.py:361
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:361
- Finding
rotation ticks
peaks...
INFO 2025-07-24 full_derotation_pipeline.py:361
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:361
- Finding
rotation ticks
peaks...
INFO 2025-07-24 full_derotation_pipeline.py:436
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:436
- Cleaning start
and end rotation
signal...
INFO 2025-07-24 full_derotation_pipeline.py:436
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:436
- Cleaning start
and end rotation
signal...
INFO 2025-07-24 full_derotation_pipeline.py:461
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:461
- Creating
signed rotation
array...
INFO 2025-07-24 full_derotation_pipeline.py:461
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:461
- Creating
signed rotation
array...
INFO 2025-07-24 full_derotation_pipeline.py:483
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:483
- Dropping ticks
outside of the
rotation
period...
INFO 2025-07-24 full_derotation_pipeline.py:483
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:483
- Dropping ticks
outside of the
rotation
period...
INFO 2025-07-24 full_derotation_pipeline.py:519
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:519
- Ticks dropped:
1.
Ticks remaining:
3451
INFO 2025-07-24 full_derotation_pipeline.py:519
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:519
- Ticks dropped:
1.
Ticks remaining:
3451
INFO 2025-07-24 full_derotation_pipeline.py:550
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:550
- Number of
rotations is as
expected
INFO 2025-07-24 full_derotation_pipeline.py:550
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:550
- Number of
rotations is as
expected
WARNING 2025-07-24 full_derotation_pipeline.py:597
15:36:25 PM -
WARNING -
MainProcess
full_derotation_
pipeline.py:597
- Number of
ticks is not as
expected: 3451.
Expected ticks:
3600.0
Delta: -149.0
WARNING 2025-07-24 full_derotation_pipeline.py:597
15:36:25 PM -
WARNING -
MainProcess
full_derotation_
pipeline.py:597
- Number of
ticks is not as
expected: 3451.
Expected ticks:
3600.0
Delta: -149.0
INFO 2025-07-24 full_derotation_pipeline.py:649
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:649
- New increment
example: 0.209
INFO 2025-07-24 full_derotation_pipeline.py:649
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:649
- New increment
example: 0.209
INFO 2025-07-24 full_derotation_pipeline.py:662
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:662
- Interpolating
angles...
INFO 2025-07-24 full_derotation_pipeline.py:662
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:662
- Interpolating
angles...
INFO 2025-07-24 full_derotation_pipeline.py:702
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:702
- Cleaning
interpolated
angles...
INFO 2025-07-24 full_derotation_pipeline.py:702
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:702
- Cleaning
interpolated
angles...
INFO 2025-07-24 full_derotation_pipeline.py:761
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:761
- Calculating
angles by line
and frame...
INFO 2025-07-24 full_derotation_pipeline.py:761
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:761
- Calculating
angles by line
and frame...
INFO 2025-07-24 full_derotation_pipeline.py:856
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:856
- Plotting
rotation ticks
and rotation on
signal...
INFO 2025-07-24 full_derotation_pipeline.py:856
15:36:25 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:856
- Plotting
rotation ticks
and rotation on
signal...
2025-07-24 15:36:26 PM INFO 2025-07-24 full_derotation_pipeline.py:898
15:36:26 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:898
- Plotting
rotation
angles...
2025-07-24 15:36:26 PM INFO 2025-07-24 full_derotation_pipeline.py:898
15:36:26 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:898
- Plotting
rotation
angles...
INFO 2025-07-24 full_derotation_pipeline.py:901
15:36:26 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:901
- Speeds:
{np.int64(200)}
INFO 2025-07-24 full_derotation_pipeline.py:901
15:36:26 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:901
- Speeds:
{np.int64(200)}
INFO 2025-07-24 full_derotation_pipeline.py:902
15:36:26 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:902
- len (speeds):
1
INFO 2025-07-24 full_derotation_pipeline.py:902
15:36:26 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:902
- len (speeds):
1
2025-07-24 15:36:27 PM INFO 2025-07-24 full_derotation_pipeline.py:348
15:36:27 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:348
- ✨ Analog
signals
processed ✨
2025-07-24 15:36:27 PM INFO 2025-07-24 full_derotation_pipeline.py:348
15:36:27 PM -
INFO -
MainProcess
full_derotation_
pipeline.py:348
- ✨ Analog
signals
processed ✨
INFO 2025-07-24 full_derotation_pipeline.py:1121
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
21 - Optimal
center of
rotation read
from file.
INFO 2025-07-24 full_derotation_pipeline.py:1121
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
21 - Optimal
center of
rotation read
from file.
INFO 2025-07-24 full_derotation_pipeline.py:1167
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
67 - Starting
derotation by
line...
INFO 2025-07-24 full_derotation_pipeline.py:1167
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
67 - Starting
derotation by
line...
INFO 2025-07-24 full_derotation_pipeline.py:1135
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
35 - Plotting
max projection
with center...
INFO 2025-07-24 full_derotation_pipeline.py:1135
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
35 - Plotting
max projection
with center...
0%| | 0/69120 [00:00<?, ?it/s]
48%|████▊ | 33028/69120 [00:00<00:00, 330266.14it/s]
96%|█████████▌| 66055/69120 [00:00<00:00, 108430.14it/s]
100%|██████████| 69120/69120 [00:00<00:00, 124018.58it/s]
INFO 2025-07-24 full_derotation_pipeline.py:1135
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
35 - Plotting
max projection
with center...
INFO 2025-07-24 full_derotation_pipeline.py:1135
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
35 - Plotting
max projection
with center...
INFO 2025-07-24 full_derotation_pipeline.py:1198
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
98 - ✨ Image
stack derotated
✨
INFO 2025-07-24 full_derotation_pipeline.py:1198
15:36:27 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:11
98 - ✨ Image
stack derotated
✨
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:3860: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/numpy/_core/_methods.py:136: RuntimeWarning: invalid value encountered in divide
ret = um.true_divide(
2025-07-24 15:36:31 PM INFO 2025-07-24 full_derotation_pipeline.py:1338
15:36:31 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:13
38 - Saving
/home/runner/wo
rk/derotation/d
erotation/examp
les/derotated.t
if
2025-07-24 15:36:31 PM INFO 2025-07-24 full_derotation_pipeline.py:1338
15:36:31 PM -
INFO -
MainProcess
full_derotation
_pipeline.py:13
38 - Saving
/home/runner/wo
rk/derotation/d
erotation/examp
les/derotated.t
if
WARNING 2025-07-24 full_derotation_pipeline.py:1357
15:36:31 PM -
WARNING -
MainProcess
full_derotation
_pipeline.py:13
57 - Number of
rotation angles
by frame is
higher than the
number of
frames
WARNING 2025-07-24 full_derotation_pipeline.py:1357
15:36:31 PM -
WARNING -
MainProcess
full_derotation
_pipeline.py:13
57 - Number of
rotation angles
by frame is
higher than the
number of
frames
Inspecting the output#
Convenience handles for later use
debug_folder = pipeline.debug_plots_folder
debug_images = sorted(debug_folder.glob("*.png"))
mean_images_folder = debug_folder / "mean_images"
def get_image_path(name):
for img_path in debug_images:
if name == img_path.name.split(".")[0]:
return img_path
def show_image(path):
img = plt.imread(path)
plt.imshow(img)
plt.axis("off")
plt.tight_layout()
plt.show()
Rotation detection based on analog signals
Expected number of ticks: 3600.0
Detected ticks: 3451
Adjusted rotation increment: 0.200 degrees
Interpolated rotation angles per line Green = frame-level angles, Yellow = interpolated per-line angles
show_image(get_image_path("rotation_angles"))
Calculated baseline (offset) of the image in arbitrary units
print(f"Estimated image offset: {pipeline.offset}")
Estimated image offset: -3623.787105605227
Original max projection with estimated center
show_image(get_image_path("max_projection_with_center"))
Position of the most detected cell after finding the optimal center of rotation. As you can see it is pretty stable.
show_image(get_image_path("most_detected_blob_centers"))
Derotated max projection with center overlaid Now the cells are aligned, although registration might still be needed
show_image(get_image_path("derotated_max_projection_with_center"))
Rotation angles and derotation metadata are accessible as a pandas DataFrame
print(pipeline.derotation_output_table.iloc[125:153])
frame rotation_angle clock direction speed rotation_count
125 125 -0.000000 1211775 NaN NaN NaN
126 126 -0.000000 1219197 NaN NaN NaN
127 127 -0.000000 1226620 NaN NaN NaN
128 128 -0.000000 1234043 NaN NaN NaN
129 129 -0.000000 1241466 NaN NaN NaN
130 130 3.457208 1248889 -1.0 200.0 0.0
131 131 10.372414 1256312 -1.0 200.0 0.0
132 132 20.128228 1263735 -1.0 200.0 0.0
133 133 33.321739 1271157 -1.0 200.0 0.0
134 134 49.754037 1278580 -1.0 200.0 0.0
135 135 69.574618 1286003 -1.0 200.0 0.0
136 136 92.609551 1293426 -1.0 200.0 0.0
137 137 118.937888 1300849 -1.0 200.0 0.0
138 138 148.173913 1308272 -1.0 200.0 0.0
139 139 178.005351 1315695 -1.0 200.0 0.0
140 140 207.573421 1323118 -1.0 200.0 0.0
141 141 237.029156 1330540 -1.0 200.0 0.0
142 142 263.770777 1337963 -1.0 200.0 0.0
143 143 287.242733 1345386 -1.0 200.0 0.0
144 144 307.447826 1352809 -1.0 200.0 0.0
145 145 324.397365 1360232 -1.0 200.0 0.0
146 146 338.060243 1367655 -1.0 200.0 0.0
147 147 348.422481 1375078 -1.0 200.0 0.0
148 148 355.495714 1382500 -1.0 200.0 0.0
149 149 359.403773 1389923 -1.0 200.0 0.0
150 150 -0.000000 1397346 NaN NaN NaN
151 151 -0.000000 1404769 NaN NaN NaN
152 152 -0.000000 1412192 NaN NaN NaN
Total running time of the script: (0 minutes 6.393 seconds)
