Pose Embedding: Improving Object Classification using Pose Information

This repository contains the code used for the experiments in the research report:

Improving Object Classification using Pose Information Hugo Penedones, Ronan Collobert, François Fleuret, David Grangier Idiap Research Report Idiap-RR-30-2012, November 2012 https://hpenedones.me/publications/Penedones_Idiap-RR-30-2012.pdf

We investigate whether pose information can be used as a regularizer to improve object classification. The key idea is to train neural networks in a multi-task learning setup, where the network simultaneously predicts:

1. The object class (classification task)
2. The camera azimuth (pose estimation task)

The hypothesis is that forcing the network to learn a pose-aware representation will produce smoother, more generalizable features — and ultimately reduce classification error.

Experiments are conducted on the small-NORB dataset (LeCun et al., 2004), a collection of 24,300 stereo images of 50 toy objects across 5 categories, photographed at varying azimuths, elevations, and lighting conditions. NORB is well-suited for this study because it provides ground-truth pose labels and the texture-less objects require shape-based reasoning.

The multi-task approach consistently outperforms the single-task baseline:

Adding pose supervision reduced the convolutional network error from 6.6% to ~5.4%, and improved the MLP from ~13% (SVM level) down to ~12% — without any changes to the network architecture or hand-crafted features.

The network's pose output for class y is a 2D vector g_y(x). Predicting a 2D vector (rather than a scalar angle) avoids boundary issues and naturally handles the 180° symmetry of the NORB data. Two loss functions are explored:

• Cosine loss: maximizes the cosine similarity between the predicted vector and r(θ) = (cos θ, sin θ).
• L1/L2 regression loss: forces the predicted vector to have unit norm, which is more robust to numerical issues.

pose_embedding/
├── data_utils/          # Dataset preparation (NORB, synthetic, faces, letters)
├── embedder/            # Core neural network trainer (multi-task learning)
├── includes/            # Shared Lua modules (utilities, dataset loading, visualization)
├── norb-baseline/       # NORB-specific baselines (ranking, angle regression, push-based)
├── pose_corrector/      # Pose-corrected classifier variants
├── sge/                 # SGE grid engine job submission scripts
└── sketchup_scripts/    # Ruby scripts to render synthetic 3D training data via SketchUp

The main training code. embedder.lua trains a neural network with a combined classification + pose regression loss using Torch7 and stochastic gradient descent (backpropagation).

Several baseline approaches for the NORB dataset:

• embedd.lua — standard embedding
• embedd-angle.lua — angle regression
• embedd-rank.lua — ranking-based embedding
• embedd-with-push.lua — push-based contrastive embedding
• Custom criteria: AngleCriterion.lua, Atan2Criterion.lua, DistanceCriterion.lua

Experiments with lookup-table layers (LookupSpatialConvolution.lua, LookupLinear.lua) to learn pose-conditioned classifiers, along with modulo-aware rotation criteria (ModuloRotationCriterion.lua).

Scripts to load and preprocess NORB (including a MATLAB extractor), generate synthetic datasets from SketchUp 3D models, and create angle-predictor training sets.

Ruby scripts (get_view.rb, norb_view.rb) that automate SketchUp to render 3D objects at controlled azimuths and elevations, enabling synthetic data augmentation.

• Torch7 — deep learning framework (Lua)
• SketchUp — 3D modeling (for synthetic data generation)
• MATLAB — for NORB data extraction

• Hugo Penedones (Idiap Research Institute)
• Ronan Collobert (Idiap Research Institute)
• François Fleuret (Idiap Research Institute)
• David Grangier (NEC Laboratories, Princeton)