Seminario Ruggero Pintus, 4-10-2012

transcript

www.crs4.it/vic/

TecnologieTecnologie didi Visual ComputingVisual Computingper per ii BeniBeni CulturaliCulturaliper per ii BeniBeni CulturaliCulturali

R. PintusCRS4 Visual Computing

R. Pintus – CRS4/ViC, October 2012

TecnologieTecnologie per per ii benibeni culturaliculturali

• Focus: digitalizzazione accurata (forma e colore) di siti e manufatti + …– Partire dai dati: Acquisizione -> Trattamento !

– Modelli misurabili

• Molti usi oltre la visualizzazione• Molti usi oltre la visualizzazione

– Riproduzione materica

– Studio di opere d’arte

– Documentazione in-situ di scavi archeologici

– Supporto al restauro e sua documentazione

– Valorizzazione

TecnologieTecnologie per per ii benibeni culturaliculturali

• Le quantità di dati prodotte dai moderni sensori sono però difficili da trattare, archiviare, distribuire, visualizzare– Scalabilità!

• Tecniche attuali sub-ottimali• Tecniche attuali sub-ottimali– Costi, tempi, qualità

• Bisogno di ricerca in tecnologie abilitanti scalabili– Acquisizione

– Processamento geometrico

– Visualizzazione

– …

Tecnologie per i beni culturaliTecnologie per i beni culturali

• Come acquisire e processare efficacemente forma e colore di siti e manufatti?siti e manufatti?– Tecniche di fusione multi-sensore, stream-processing, multiresolution, external

memory algorithms, parallel programming, GPGPUs

• Come archiviare e distribuire efficacemente i modelli?– Multiresolution, adaptive streaming, compression

• Come visualizzarli efficacemente?– Multiresolution, adaptive rendering, out-of-core methods, GPU programming,

parallelization, rasterization, ray-casting

• Come esplorarli?– Novel 3D displays, specific interaction techniques

– Portable devices

AlcuniAlcuni esempiesempi

• Allineamento geometria/colore

• Colorazione di modelli 3D

• Fusione di dati e ricostruzione geometrica

• Visualizzazione scalabile ed interattiva

• Distribuzione di dati in rete• Distribuzione di dati in rete

• Esplorazione su display innovativi

(… e molto altro)

Our GoalOur Goal

Modelling vs AcquisitionModelling vs Acquisition

ModellingSubjective Reality

AcquisitionObjective reality

3D Reconstruction3D Reconstruction

• Acquire geometry and color• A lot of techniques

– Structured light, laser scanning (triangulation or time-of-flight), photometric stereo, shape-from-X, …

• Which technique?

• Which technique?– Object type (big/small, material….) – Cost– Accuracy/Resolution– Time– Complexity

OutlineOutline

• 3D Reconstruction Techniques

• 3D Reconstruction Pipeline – Photo mapping/blending

– Printing

• Case study

Taxonomy (nonTaxonomy (non--destructive)destructive)

• Contact– Direct Measurements

• rulers, calipers, pantographs, coordinate measuring machines (CMM), AFM

• Non-Contact

– Passive• Shape-from-X

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Active• Transmissive

– Computed Tomography (CT)

– Transmissive Ultrasound

• Reflective– Non-Optical Methods

» reflective ultrasound, radar, sonar, MRI

– Time-of-Flight

– Triangulation

» laser striping

» structured lighting

– Photometric Stereo

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– StereoStereo

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– MultiviewMultiview

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– SilhouettesSilhouettes

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– Depth from Depth from focus/defocusfocus/defocus

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– TransmissiveTransmissive

Computed Tomography

Density Function

Trasmissive Ultrasound

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– NonNon--OpticalOptical

Ultrasound Radar MRI

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– TimeTime--ofof--FlightFlight

0.528 ≈

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– Laser StripingLaser Striping

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– Structured LightingStructured Lighting

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Taxonomy Taxonomy –– Photometric StereoPhotometric Stereo

Photometric Stereo Photometric Stereo –– SEMSEM

Taxonomy Taxonomy –– Photometric StereoPhotometric Stereo

Taxonomy SEMTaxonomy SEM

• Non-Contact

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Time-of-Flight

– Triangulation

» laser striping

Cultural HeritageCultural Heritage

• Techniques– Triangulation (laser scanner)

– Time of Flight

– Texture Mapping

– Multi-view reconstruction

• Deal with multiple acquisitions

• Manage a huge amount of data for visualization purposes

3D Reconstruction Pipeline3D Reconstruction PipelineReal Object Acquisition Devices

Photos

3D Digital Model

=== Processing ===-Cleaning- Merging

- Photo Alignment- Color Projection

Geometry

3D Reconstruction Pipeline3D Reconstruction Pipeline

• Real Model Inspection (onsite)

• Scans design (offsite/onsite)

• Acquisition (onsite)

• Alignment (offsite)

• Editing (offsite)

• Merge (offsite)

• Texture (offsite)

• Final Model (offsite)

• 3D Printing (offsite)

3D Reconstruction Pipeline3D Reconstruction Pipeline

• Real Model Inspection (onsite)

• Scans design (offsite/onsite)

• Acquisition (onsite)

• Alignment (offsite)

• Editing (offsite)

• Merge (offsite)

• Texture (offsite)

• Final Model (offsite)

• 3D Printing (offsite)

GoalGoal

• Fast and low-cost technique for creating accurate colored models

• Acquisition – 3D – laser scanners

– Color – digital cameras– Color – digital cameras

• Mapping photo-to-geometry– Fast and Robust Semi-Automatic Registration of Photographs

to 3D Geometry

• Photo blending– A Streaming Framework for Seamless Detailed Photo

Blending on Massive Point Clouds

www.crs4.it/vic/

Photo MappingPhoto MappingPhoto MappingPhoto Mapping

Ruggero Pintus, Enrico Gobbetti, and Roberto Combet. “Fast and Robust Semi-Automatic Registration of Photographs to 3D Geometry”. In The 12th International Symposium on Virtual Reality, Archaeology and Cultural Heritage, October 2011.

Problem StatementProblem Statement

3D Geometry Unordered SetOf n Uncalibrated

Photos

n Camera Poses(2D/3D Registration)

Related workRelated work

• Manual selection of 2D-3D matches– Massive user intervention – Tiring and time-consuming

• Automatic feature matching– Not robust enough for a generic dataset

• Semi-automatic statistical correlation• Semi-automatic statistical correlation– Point cloud attributes not always provided

• Geometric multi-view reconstruction– 2D-3D problem � 3D-3D registration task

– dense and ordered frame sequence

• Our contribution– Minimize user intervention / Large datasets / Semi-

automatic / Multi-view based approach / No Attributes

Input DataInput Data

• Dense Geometry– Point cloud, triangle

mesh, etc.

– No attributes

– No particular features

SfM Reconstruction

Dense 3D n Photos

– No particular features

• n photos– Naïve constraints:

• Blur, Noise, Under- or over-exposured

– Sufficient overlap

Coarse Registration

Refinement

Output Data

MultiMulti--viewview

• Bundler [Snavely et al. 2006]– SfM system for unordered

image collections

– http://phototour.cs.washington.edu/bundler/

SfM Reconstruction

Dense 3D n Photos

n.edu/bundler/

• Output– A sparse point cloud

– n camera poses

– SIFT keypoints (projections of sparse 3D points)

Coarse Registration

Refinement

Output Data

Coarse registrationCoarse registration

• Register two point clouds with different:

– scales

– reference frames

– resolutions

• Automatic methods are not

SfM Reconstruction

Dense 3D n Photos

• Automatic methods are not robust and efficient enough

• User aligns few images (one or more) to the dense geometry

• Affine transformation is applied to all cameras and sparse points

Coarse Registration

Refinement

Output Data

RefinementRefinement

SfM Reconstruction

Dense 3D n Photos1C

jpjs ,1

( )jpCQ ,2

( )jF pNN

Coarse Registration

Refinement

Output Data

( )( )jF pNNCQ ,2

( ) ( )( )∑∑= =

−=P CN

ijijFiij spNNCQvPCE

RefinementRefinement

• Sparse Bundle Adjustment (SBA)– Constants – SIFT keypoints,

dense 3D points

– Variables – Camera poses, sparse 3D points

SfM Reconstruction

Dense 3D n Photos

sparse 3D points

– SBA

• A Generic SBA C/C++ Package Based on the Levenberg-Marquardt Algorithm

• http://www.ics.forth.gr/~lourakis/sba/

Coarse Registration

Refinement

Output Data

Output dataOutput data

• n camera poses

• Input of photo blending

SfM Reconstruction

Dense 3D n Photos

blending– n photos

– n camera poses

– Dense 3D geometry

Coarse Registration

Refinement

Output Data

Results Results –– Photo mappingPhoto mapping

www.crs4.it/vic/

Photo BlendingPhoto BlendingPhoto BlendingPhoto Blending

Ruggero Pintus, Enrico Gobbetti, and Marco Callieri. A Streaming Framework for Seamless Detailed Photo Blending on Massive Point Clouds. In Proc. Eurographics Area Papers. Pages 25- 32, 2011.

Point Cloud CalibratedPhotos

ColoredPoint Cloud

• Problem �� Unlimited size of 3D model (Gpoints) and unlimited number of images

Related workRelated work

• State-of-the-art techniques

– Image quality estimation

– Stitching or blending

• Data representation

– Triangle meshes – exploit connectivity

– Meshless approaches– Meshless approaches

• Both triangle meshes and point clouds

• Memory settings

– All in-core – no massive geometry/images

– 3D in-core and images out-of-core – no massive geometry

– All out-of-core – Low performances

• Our contribution– Blending function / Streaming framework / Massive point cloud /

Adaptive geometry refinement

PipelinePipeline

Photo StencilPer-pixelWeight

MaskedPer-pixelWeight

Simple blendingSimple blending

Edge extraction and Distance Edge extraction and Distance TransformTransform

Smooth weightSmooth weight

Single band blendingSingle band blending

Multi band blendingMulti band blending

Adaptive point refinementAdaptive point refinement

ResultsResults

• Callieri et. al 2008 – David 28M

– Disk space occupancy –6.2GB

– Computation time – 15.5 hours

David470Mpoints

– Computation time – 15.5 hours

Results Results –– Church’s ApseChurch’s Apse

14 Mpoint Geometry 40 photos

Results Results –– Church’s ApseChurch’s Apse

Results Results –– Grave Grave

8 Mpoint Geometry21 photos

Results Results –– Grave Grave

ResultsResults

David470Mpoints470Mpoints

Image size – 19456x532481Gpixel

ResultsResults

ConclusionConclusion

• Image-to-geometry registration approach

• Minimum user intervention

• No constraints on geometry, attributes and features

• Specific robust cost function and SBA

• Out-of-core photo blending approach (Point clouds of unlimited size)

• Incremental color accumulation (Unlimited number of images)• Incremental color accumulation (Unlimited number of images)

• Smooth weight function (Seamless color blending)

• Streaming framework (Performance improvement)

• Adaptive point refinement

• Future work

– Automatic sparse-to-dense geometry registration

– Interactive blending - adding and removing images in an interactive tool

– Fast visual check of previous alignment step

ConclusionConclusion

• Low cost

– Personal computer

– Digital camera

– Decreased manual intervention

• Open Source / Free Software

– Bundler – SfM reconstruction –http://phototour.cs.washington.edu/bundler/http://phototour.cs.washington.edu/bundler/

– Sparse Bundle Adjustment – SBA – Minimization –http://www.ics.forth.gr/~lourakis/sba/

– Opengl / GLSL shaders – Rendering – http://www.opengl.org/

– Qt – Interface – http://qt.nokia.com/

– Opencv – Manual registration – http://opencv.willowgarage.com/wiki/

– Spaceland Library – Geometric computation –http://spacelib.sourceforge.net/

– IIPImage – Web-based Viewer – http://iipimage.sourceforge.net/

3D Printing3D Printing

Printing ProcessPrinting Process

• Original model

• Slice representation

representation

• Layer by layer deposition

• Cleaning

• Printed model

• Original model

representation

• Cleaning

• Printed model

• Original model

representation

• Cleaning

• Printed model

• Original model

representation

• Cleaning

• Printed model

• Original model

representation

• Cleaning

• Printed model

Geometry processingGeometry processing

SubSub--surface scatteringsurface scattering

Color Color enhancementenhancement

ConclusioniConclusioni

• Lavorare su dati misurati è un pre-requisito di molti lavori (tutti?) nel contesto dei beni culturali– Applicazioni specialistiche o per grande pubblico

• Le moderne tecnologie di acquisizione • Le moderne tecnologie di acquisizione consentono di acquisire una grande quantità di informazioni (forma e colore)– Laser scanning, camere digitali, ecc.

• Uso potenziale vasto!– Valorizzazione, restauro, studio, ecc.

• Queste quantità di dati sono però difficili da trattare, archiviare, distribuire, visualizzare– Scalabilità!

• Tecniche attuali sub-ottimali• Tecniche attuali sub-ottimali– Costi, tempi, qualità

• Il CRS4 è impegnato in attività di ricerca per migliorare le tecnologie…– Stato dell’arte internazionale

– Collaborazioni e ricadute locali

• PMI, Contro Restauro SS, Soprintendenze, CNR, UniCA• PMI, Contro Restauro SS, Soprintendenze, CNR, UniCA

• … e per applicarle a casi concreti– Collaborazioni multidisciplinari!

Questions & ContactsQuestions & Contacts

• CRS4 – VIC www.crs4.it/vic/

• Ruggero Pintus • Ruggero Pintus ruggero@crs4.it

Seminario Ruggero Pintus, 4-10-2012

Technology