Seminario Ruggero Pintus, 4-10-2012

www.crs4.it/vic/

TecnologieTecnologie didi Visual ComputingVisual Computingper per ii BeniBeni CulturaliCulturaliper per ii BeniBeni CulturaliCulturali

R. PintusCRS4 Visual Computing

R. Pintus – CRS4/ViC, October 2012

TecnologieTecnologie per per ii benibeni culturaliculturali

• Focus: digitalizzazione accurata (forma e colore) di siti e manufatti + …– Partire dai dati: Acquisizione -> Trattamento !

– Modelli misurabili

• Molti usi oltre la visualizzazione• Molti usi oltre la visualizzazione

– Riproduzione materica

– Studio di opere d’arte

– Documentazione in-situ di scavi archeologici

– Supporto al restauro e sua documentazione

– Valorizzazione


TecnologieTecnologie per per ii benibeni culturaliculturali

• Le quantità di dati prodotte dai moderni sensori sono però difficili da trattare, archiviare, distribuire, visualizzare– Scalabilità!

• Tecniche attuali sub-ottimali• Tecniche attuali sub-ottimali– Costi, tempi, qualità

• Bisogno di ricerca in tecnologie abilitanti scalabili– Acquisizione

– Processamento geometrico

– Visualizzazione

– …


Tecnologie per i beni culturaliTecnologie per i beni culturali

• Come acquisire e processare efficacemente forma e colore di siti e manufatti?siti e manufatti?– Tecniche di fusione multi-sensore, stream-processing, multiresolution, external

memory algorithms, parallel programming, GPGPUs

• Come archiviare e distribuire efficacemente i modelli?– Multiresolution, adaptive streaming, compression

• Come visualizzarli efficacemente?– Multiresolution, adaptive rendering, out-of-core methods, GPU programming,

parallelization, rasterization, ray-casting

• Come esplorarli?– Novel 3D displays, specific interaction techniques

– Portable devices


AlcuniAlcuni esempiesempi

• Allineamento geometria/colore

• Colorazione di modelli 3D

• Fusione di dati e ricostruzione geometrica

• Visualizzazione scalabile ed interattiva

• Distribuzione di dati in rete• Distribuzione di dati in rete

• Esplorazione su display innovativi

(… e molto altro)


Our GoalOur Goal

6


Modelling vs AcquisitionModelling vs Acquisition

ModellingSubjective Reality

7

AcquisitionObjective reality


3D Reconstruction3D Reconstruction

• Acquire geometry and color• A lot of techniques

– Structured light, laser scanning (triangulation or time-of-flight), photometric stereo, shape-from-X, …

• Which technique?

8

• Which technique?– Object type (big/small, material….) – Cost– Accuracy/Resolution– Time– Complexity


OutlineOutline

• 3D Reconstruction Techniques

• 3D Reconstruction Pipeline – Photo mapping/blending

– Printing

9

– Printing

• Case study


Taxonomy (nonTaxonomy (non--destructive)destructive)

• Contact– Direct Measurements

• rulers, calipers, pantographs, coordinate measuring machines (CMM), AFM

• Non-Contact

10

– Passive• Shape-from-X

– Stereo

– Multiview

– Silhouettes

– Focus/Defocus

– Active• Transmissive

– Computed Tomography (CT)

– Transmissive Ultrasound

• Reflective– Non-Optical Methods

» reflective ultrasound, radar, sonar, MRI

– Time-of-Flight

– Triangulation

» laser striping

» structured lighting

– Photometric Stereo





• Non-Contact

11


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping







• Non-Contact

12


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– StereoStereo

13


Taxonomy Taxonomy –– StereoStereo

14





• Non-Contact

15


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– MultiviewMultiview

16


Taxonomy Taxonomy –– MultiviewMultiview

17





• Non-Contact

18


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– SilhouettesSilhouettes

19





• Non-Contact

20


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– Depth from Depth from focus/defocusfocus/defocus

21





• Non-Contact

22


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– TransmissiveTransmissive

Computed Tomography

23

Density Function

Trasmissive Ultrasound





• Non-Contact

24


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– NonNon--OpticalOptical

25

Ultrasound Radar MRI





• Non-Contact

26


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– TimeTime--ofof--FlightFlight

27

nssm

m

c

dt 17

103

0.528 ≈

×==





• Non-Contact

28


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– Laser StripingLaser Striping

29





• Non-Contact

30


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– Structured LightingStructured Lighting

31





• Non-Contact

32


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Taxonomy Taxonomy –– Photometric StereoPhotometric Stereo

33


Photometric Stereo Photometric Stereo –– SEMSEM

34


Taxonomy Taxonomy –– Photometric StereoPhotometric Stereo

35


Taxonomy SEMTaxonomy SEM



• Non-Contact

36


– Stereo

– Multiview

– Silhouettes

– Focus/Defocus






– Time-of-Flight

– Triangulation

» laser striping




Cultural HeritageCultural Heritage

• Techniques– Triangulation (laser scanner)

– Time of Flight

– Texture Mapping

– Multi-view reconstruction


37


• Deal with multiple acquisitions

• Manage a huge amount of data for visualization purposes


3D Reconstruction Pipeline3D Reconstruction PipelineReal Object Acquisition Devices

Photos

38

3D Digital Model

=== Processing ===-Cleaning- Merging

- Photo Alignment- Color Projection

- …

Geometry


3D Reconstruction Pipeline3D Reconstruction Pipeline

• Real Model Inspection (onsite)

• Scans design (offsite/onsite)

• Acquisition (onsite)

• Alignment (offsite)

39

• Editing (offsite)

• Merge (offsite)

• Texture (offsite)

• Final Model (offsite)

• 3D Printing (offsite)


3D Reconstruction Pipeline3D Reconstruction Pipeline

• Real Model Inspection (onsite)

• Scans design (offsite/onsite)

• Acquisition (onsite)

• Alignment (offsite)

40

• Editing (offsite)

• Merge (offsite)

• Texture (offsite)

• Final Model (offsite)

• 3D Printing (offsite)


GoalGoal

• Fast and low-cost technique for creating accurate colored models

• Acquisition – 3D – laser scanners

– Color – digital cameras– Color – digital cameras

• Mapping photo-to-geometry– Fast and Robust Semi-Automatic Registration of Photographs

to 3D Geometry

• Photo blending– A Streaming Framework for Seamless Detailed Photo

Blending on Massive Point Clouds

www.crs4.it/vic/

Photo MappingPhoto MappingPhoto MappingPhoto Mapping

Ruggero Pintus, Enrico Gobbetti, and Roberto Combet. “Fast and Robust Semi-Automatic Registration of Photographs to 3D Geometry”. In The 12th International Symposium on Virtual Reality, Archaeology and Cultural Heritage, October 2011.


Problem StatementProblem Statement

3D Geometry Unordered SetOf n Uncalibrated

Photos

n Camera Poses(2D/3D Registration)


Related workRelated work

• Manual selection of 2D-3D matches– Massive user intervention – Tiring and time-consuming

• Automatic feature matching– Not robust enough for a generic dataset

• Semi-automatic statistical correlation• Semi-automatic statistical correlation– Point cloud attributes not always provided

• Geometric multi-view reconstruction– 2D-3D problem � 3D-3D registration task

– dense and ordered frame sequence

• Our contribution– Minimize user intervention / Large datasets / Semi-

automatic / Multi-view based approach / No Attributes


Input DataInput Data

• Dense Geometry– Point cloud, triangle

mesh, etc.

– No attributes

– No particular features

User

SfM Reconstruction

Dense 3D n Photos

– No particular features

• n photos– Naïve constraints:

• Blur, Noise, Under- or over-exposured

– Sufficient overlap

Coarse Registration

Refinement

Output Data


MultiMulti--viewview

• Bundler [Snavely et al. 2006]– SfM system for unordered

image collections

– http://phototour.cs.washington.edu/bundler/

User

SfM Reconstruction

Dense 3D n Photos

n.edu/bundler/

• Output– A sparse point cloud

– n camera poses

– SIFT keypoints (projections of sparse 3D points)

Coarse Registration

Refinement

Output Data


Coarse registrationCoarse registration

• Register two point clouds with different:

– scales

– reference frames

– resolutions

• Automatic methods are not

User

SfM Reconstruction

Dense 3D n Photos

• Automatic methods are not robust and efficient enough

• User aligns few images (one or more) to the dense geometry

• Affine transformation is applied to all cameras and sparse points

Coarse Registration

Refinement

Output Data


RefinementRefinement

User

SfM Reconstruction

Dense 3D n Photos1C

jpjs ,1

js ,2

( )jpCQ ,2

jP

( )jF pNN

Coarse Registration

Refinement

Output Data

2C

js ,2

( )( )jF pNNCQ ,2

( ) ( )( )∑∑= =

−=P CN

j

N

ijijFiij spNNCQvPCE

1 1

2

,,,


RefinementRefinement

• Sparse Bundle Adjustment (SBA)– Constants – SIFT keypoints,

dense 3D points

– Variables – Camera poses, sparse 3D points

User

SfM Reconstruction

Dense 3D n Photos

sparse 3D points

– SBA

• A Generic SBA C/C++ Package Based on the Levenberg-Marquardt Algorithm

• http://www.ics.forth.gr/~lourakis/sba/

Coarse Registration

Refinement

Output Data


Output dataOutput data

• n camera poses

• Input of photo blending

User

SfM Reconstruction

Dense 3D n Photos

blending– n photos

– n camera poses

– Dense 3D geometry

Coarse Registration

Refinement

Output Data


Results Results –– Photo mappingPhoto mapping

www.crs4.it/vic/

Photo BlendingPhoto BlendingPhoto BlendingPhoto Blending

Ruggero Pintus, Enrico Gobbetti, and Marco Callieri. A Streaming Framework for Seamless Detailed Photo Blending on Massive Point Clouds. In Proc. Eurographics Area Papers. Pages 25- 32, 2011.



Point Cloud CalibratedPhotos




P




P

ColoredPoint Cloud




P

ColoredPoint Cloud

• Problem �� Unlimited size of 3D model (Gpoints) and unlimited number of images


Related workRelated work

• State-of-the-art techniques

– Image quality estimation

– Stitching or blending

• Data representation

– Triangle meshes – exploit connectivity

– Meshless approaches– Meshless approaches

• Both triangle meshes and point clouds

• Memory settings

– All in-core – no massive geometry/images

– 3D in-core and images out-of-core – no massive geometry

– All out-of-core – Low performances

• Our contribution– Blending function / Streaming framework / Massive point cloud /

Adaptive geometry refinement


PipelinePipeline

Photo StencilPer-pixelWeight

MaskedPer-pixelWeight


Simple blendingSimple blending


Edge extraction and Distance Edge extraction and Distance TransformTransform


Smooth weightSmooth weight


Smooth weightSmooth weight


Single band blendingSingle band blending


Multi band blendingMulti band blending


Adaptive point refinementAdaptive point refinement








ResultsResults

• Callieri et. al 2008 – David 28M

– Disk space occupancy –6.2GB

– Computation time – 15.5 hours

David470Mpoints

– Computation time – 15.5 hours


Results Results –– Church’s ApseChurch’s Apse

14 Mpoint Geometry 40 photos


Results Results –– Church’s ApseChurch’s Apse


Results Results –– Grave Grave

8 Mpoint Geometry21 photos


Results Results –– Grave Grave


ResultsResults


ResultsResults


ResultsResults

David470Mpoints470Mpoints

Image size – 19456x532481Gpixel


ResultsResults


ConclusionConclusion

• Image-to-geometry registration approach

• Minimum user intervention

• No constraints on geometry, attributes and features

• Specific robust cost function and SBA

• Out-of-core photo blending approach (Point clouds of unlimited size)

• Incremental color accumulation (Unlimited number of images)• Incremental color accumulation (Unlimited number of images)

• Smooth weight function (Seamless color blending)

• Streaming framework (Performance improvement)

• Adaptive point refinement

• Future work

– Automatic sparse-to-dense geometry registration

– Interactive blending - adding and removing images in an interactive tool

– Fast visual check of previous alignment step


ConclusionConclusion

• Low cost

– Personal computer

– Digital camera

– Decreased manual intervention

• Open Source / Free Software

– Bundler – SfM reconstruction –http://phototour.cs.washington.edu/bundler/http://phototour.cs.washington.edu/bundler/

– Sparse Bundle Adjustment – SBA – Minimization –http://www.ics.forth.gr/~lourakis/sba/

– Opengl / GLSL shaders – Rendering – http://www.opengl.org/

– Qt – Interface – http://qt.nokia.com/

– Opencv – Manual registration – http://opencv.willowgarage.com/wiki/

– Spaceland Library – Geometric computation –http://spacelib.sourceforge.net/

– IIPImage – Web-based Viewer – http://iipimage.sourceforge.net/


3D Printing3D Printing

80


Printing ProcessPrinting Process

• Original model

• Slice representation

81

representation

• Layer by layer deposition

• Cleaning

• Printed model



• Original model


82

representation


• Cleaning

• Printed model



• Original model


83

representation


• Cleaning

• Printed model



• Original model


84

representation


• Cleaning

• Printed model



• Original model


85

representation


• Cleaning

• Printed model


Geometry processingGeometry processing

86



87



88


SubSub--surface scatteringsurface scattering

89


Color Color enhancementenhancement

90



91



92


ConclusioniConclusioni

• Lavorare su dati misurati è un pre-requisito di molti lavori (tutti?) nel contesto dei beni culturali– Applicazioni specialistiche o per grande pubblico

• Le moderne tecnologie di acquisizione • Le moderne tecnologie di acquisizione consentono di acquisire una grande quantità di informazioni (forma e colore)– Laser scanning, camere digitali, ecc.

• Uso potenziale vasto!– Valorizzazione, restauro, studio, ecc.



• Queste quantità di dati sono però difficili da trattare, archiviare, distribuire, visualizzare– Scalabilità!

• Tecniche attuali sub-ottimali• Tecniche attuali sub-ottimali– Costi, tempi, qualità



• Il CRS4 è impegnato in attività di ricerca per migliorare le tecnologie…– Stato dell’arte internazionale

– Collaborazioni e ricadute locali

• PMI, Contro Restauro SS, Soprintendenze, CNR, UniCA• PMI, Contro Restauro SS, Soprintendenze, CNR, UniCA

• … e per applicarle a casi concreti– Collaborazioni multidisciplinari!



96


Questions & ContactsQuestions & Contacts

• CRS4 – VIC www.crs4.it/vic/

• Ruggero Pintus • Ruggero Pintus [email protected]

Date post:	10-May-2015
Category:	Technology
Upload:	crs4-research-center-in-sardinia
View:	531 times
Download:	3 times

Seminario Ruggero Pintus, 4-10-2012

Technology