(参考资料)Scanning 3D Full Human Bodies Using Kinects

IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 18, NO. 4, APRIL 2012643

Scanning3DFullHumanBodiesUsingKinects

JingTong,JinZhou,LigangLiu,ZhigengPan,andHaoYan

Abstract—DepthcamerasuchasMicrosoftKinect,ismuchcheaperthanconventional3Dscanningdevices,andthusitcanbeacquiredforeverydayuserseasily.However,thedepthdatacapturedbyKinectoveracertaindistanceisofextremelowquality.Inthispaper,wepresentanovelscanningsystemforcapturing3DfullhumanbodymodelsbyusingmultipleKinects.Toavoidtheinterferencephenomena,weusetwoKinectstocapturetheupperpartandlowerpartofahumanbodyrespectivelywithoutoverlappingregion.AthirdKinectisusedtocapturethemiddlepartofthehumanbodyfromtheoppositedirection.Weproposeapracticalapproachforregisteringthevariousbodypartsofdifferentviewsundernon-rigiddeformation.First,aroughmeshtemplateisconstructedandusedtodeformsuccessiveframespairwisely.Second,globalalignmentisperformedtodistributeerrorsinthedeformationspace,whichcansolvetheloopclosureproblemef?ciently.Misalignmentcausedbycomplexocclusioncanalsobehandledreasonablybyourglobalalignmentalgorithm.Theexperimentalresultshaveshowntheef?ciencyandapplicabilityofoursystem.Oursystemobtainsimpressiveresultsinafewminuteswithlowpricedevices,thusispracticallyusefulforgeneratingpersonalizedavatarsforeverydayusers.Oursystemhasbeenusedfor3Dhumananimationandvirtualtryon,andcanfurtherfacilitatearangeofhome-orientedvirtualreality(VR)applications.

IndexTerms—3DBodyScanning,globalnon-rigidregistration,MicrosoftKinect.

1INTRODUCTION

ThereareacoupleofworksthatusedKinecttocapturehumanfaces.[8]presentedanalgorithmforcomputingapersonalizedavatarfromasinglecolorimageanditscorrespondingdepthmap.[9]fur-thercapturedandtrackedthefacialexpressiondynamicsoftheusersinreal-timeandmapthemtoadigitalcharacter.

Toscanafullhumanshape,Kinectshouldbeputaround3metersawayfromthebody,andtheresolutionisverylow,asshowninFig-ure1(a).Littlegeometryinformationiscapturedinthedepthmap.Thoughusingtheinformationofmultipleframestoenhancethe?nalresolution[6],theresultisstillnotacceptable.

Recently,[10]estimatedthebodyshapeusingSCAPEmodel[11]byimagesilhouettesanddepthdatafromonesingleKinect.However,duetothelimitedsubspaceofparametricmodel,personalizeddetailedshapes,suchasfaces,hairstyles,anddressescannotbereconstructedbyusingthismethod.Furthermore,ittakesapproximately65minutestooptimize,whichseemstooslowforsomepracticalapplicationsinVR.

Inthispaper,wepresentasystemtoscan3DfullhumanbodyshapesusingmultipleKinects.EachKinectscandifferentpartofthehumanbodysothattheycanbeputclosetothebodytoobtainhigherqualityofdata.However,therearetwochallengesindesigningthiskindofscanningsystem.

First,thedataqualityoftheoverlappingregionbetweentwoKinectsisreducedduetotheinterferencebetweenthem.ShutterscanbeusedtoallowdifferentKinectstocapturedataatdifferenttime.However,itreducestheframeratesandexposuretime,whichalsore-ducesdataquality.Toavoidtheinterferenceissue,weusetwoKinectstocapturetheupperpartandthelowerpartofahumanbodyfromonedirection,respectively.Thereisnooverlappingregionbetweenthesetwoparts.WeuseathirdKinecttocapturethemiddlepartofthehumanbodyfromtheoppositedirection.Apersonisstandingin-betweentheKinectsandturnsaroundwiththehelpofaturntable.Theotherchallengeisthatthebodycanhardlybekeptstillduringthecapturingprocess.Thusnon-rigidregistrationofthecaptureddataisrequired.However,itisadif?cultjobduetothelowqualityofthedataandthecomplexocclusion.Weproposeatwo-phasemethodtodealwiththischallenge.Inthe?rstphase,pairwisenon-rigiddefor-mationisperformedonthegeometry?eldbasedonaroughtemplate.Inthesecondphase,aglobalalignmentwithloopclosureconstraintisusedonthedeformation?eld.

Ourscanningsystemiseasytobebuilt,asshowninFigure2aswellastheaccompanyingvideo.Theexperimentalresultshaveshownthatoursystemcapturesimpressive3Dhumanshapeswithpersonalizeddetailedshapessuchashairstylesanddresswrinkles.Withthetotalcostofabout$600,oursystemismuchcheaperthantheconventional

Manycomputergraphicsapplications,suchasanimation,computergames,humancomputerinteractionandvirtualreality,requirerealis-tic3Dmodelsofhumanbodies.Using3Dscanningtechnologies,suchasstructuredlightorlaserscan,detailedhumanmodelscouldbecre-ated[1].However,thesedevicesareveryexpensiveandoftenrequireexpertknowledgefortheoperation.Moreover,itisdif?cultforpeo-pletostayrigidduringthecapturingprocess.Image-basedmethodisanothersolutionforhumanmodeling.Thestate-of-the-artmulti-viewmethodcangetveryimpressiveresults[2].Butthiskindofmethodsiscomputationallyexpensive,andtheyhaveproblemswhentherearesparsetexturesorcomplexocclusionsamongdifferentviews[3].Asanewkindofdevices,depthcamerassuchasMicrosoftKinect[4]haveattractedmuchattentioninthecommunityrecently.Com-paredwithconventional3Dscanners,theyareabletocapturedepthandimagedataatvideorateandhavelittleconsiderationofthelightandtexturecondition.Kinectiscompact,low-price,andaseasytouseasavideocamera,whichcanbeacquiredbygeneralusers.

SomeresearchershavetriedtouseKinectsas3Dscanners.Forexample,[5]usedRGBimagesalongwithper-pixeldepthinforma-tiontobuilddense3Dmapsofindoorenvironments.However,themainproblemisthatKinecthasacomparablylowX/Yresolutionanddepthaccuracyfor3Dscanning.Toaddressthisissue,[6]describedamethodtoimprovethedata’squalitybydepthsuperresolution.UsingaKinectandcommoditygraphicshardware,[7]presentedasystemforaccuratereal-timemappingofcomplexandarbitraryindoorscenes.However,mostoftheseworkonlyusedKinecttoscanrigidobjects.

?JingTongiswithStateKeyLaboratoryofCAD&CGatZhejiang

University;CollegeofComputer&InformationEngineeringatHoHaiUniversity,E-mail:tongjing.cn@gmail.com.

?JinZhouiswithInstituteofAppliedMathematicsandEngineeringComputationsatHangzhouDianziUniversity,E-mail:zhoujin10@gmail.com.

?LigangLiuiswithDepartmentofMathematicsatZhejiangUniversity,E-mail:ligangliu@zju.edu.cn.

?ZhigengPaniswithDigitalMediaandHCIResearchCenteratHangzhouNormalUniversity,E-mail:zhigengpan@gmail.com.

?HaoYaniswithStateKeyLaboratoryofCAD&CGatZhejiangUniversity,E-mail:yhao880514@gmail.com.Manuscriptreceived15September2011;accepted3January2012;postedonline4March2012;mailedon27February2012.

Forinformationonobtainingreprintsofthisarticle,pleasesendemailto:tvcg@computer.org.

1077-2626/12/$31.00 ? 2012 IEEE Published by the IEEE Computer Society

644IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 18, NO. 4, APRIL 2012

3Dscanners,andcanbeusedforlotsofhome-orientedVRapplica-tions.

Thecontributionsofthispaperaresummarizedasfollows:?Abuilt;fullbodyscanningsystemusingmultipleMicrosoftKinectsis?Aposed;non-rigidregistrationmethodwitharoughtemplateispro-?Asionsglobalandnon-rigidmeetsthealignmentloopclosuremethodconstraintwhichisproposed.

dealswithocclu-Fig.1.(a)TherawdataofcapturingafullhumanbodywithonesingleKinecthasmuchlowqualityastheKinecthastobeputfarfromthebody.(b)TherawdatacapturedusingoursystemhashigherqualityasmultipleKinectsareusedtocapturedifferentpartsofthebodyatacloserdistance.(c)Thereconstructedhumanmodelcreatedusingourmethod.

2RELATEDWORK

2.1ScanningSystems

Acquiring3Dgeometriccontentfromrealworldisanessentialtaskformanyapplicationsincomputergraphics.Unfortunately,evenforstaticscenes,thereisnolow-pricedoff-the-shelfsystem,whichcanprovidegoodquality,highresolutiondistanceinformationinrealtime[12].Scanningdevicesbasedonstructuredlightorlaserscancancapturehumanbodywithmuchhighquality.However,thesedevicesareexpensiveandoftenrequirespecialknowledgetooperate.Forexample,thepriceofCyberwareWholeBodyColor3DScannerisabout$240,000[13].

Beinganewlydevelopeddistancemeasuringhardware,thedepthcameratechnologyopensanewepochfor3Dcontentacquisition.Therearetwomainapproachesemployedindepthcameratechnolo-giescurrently.The?rstoneisbasedonthetime-of-?ight(ToF)princi-ple[12],measuringtimedelaybetweentransmissionsofalightpulse.Becauseofthespeci?cchipneededforactivelightingpulse,thepriceofSwissRanger4000isabout$8,000.Thesecondapproachisbasedonlightcoding,projectingaknowninfraredpatternontothesceneanddeterminingdepthbasedonthepattern’sdeformation.Suchde-viceonlyneedsstandardinfraredCMOSimager,sothecostismuchlowerthantheToFdevice.AmostpopularoneistheMicrosoftKinect

sensor[4],whichisatapriceofonly$150.Inthispaper,wehavede-signedascanningsystemforautomaticallycapturing3Dfullhumanbodiesbyusing3Kinects,whichcanbepurchasedbyeverydayusersforhome-orientedVRapplications.2.2Non-rigidRegistration

Ashumanbodies’non-rigiddeformationinthescanningprocess,weneedtoregisterthescanneddata.Therehavebeenthreemaintypesofmethodsfornon-rigidregistrationintheliterature.

Registrationwithoutatemplate.Thiskindofmethodsoftenre-quireshighqualityscandata,andoftenneedssmallchangesintempo-ralcoherence.Forexample,[14]putsallscansintoa4Dspace-timesurfaceanduseskinematicpropertiestotrackpointsandregistermul-tipleframes.[15]registerstwopointcloudsthatundergoapproximateisometricdeformations.

Registrationwithatemplate.Suchkindofmethodsoftenneedstoacquirearelativeaccuratetemplate,andthenusesthetemplateto?teachscan.Someworksusemarkersto?tthetemplate[1],andtheothersutilizeglobaloptimization[16].In[16],asmoothtemplateusingstaticscannerisgeneratedandisthenregisteredtoeachoftheinputscanusinganon-linear,adaptivedeformationmodel[17].

Registrationwithasemi-template.Accuratetemplatecanhardlybeobtainedinmanycases.However,roughtemplate,suchastheskeletonmodelofarticulatedobject,canbegenerallyutilized.[18]manuallysegmentsthe?rstframe,thenidenti?esandtrackstherigidcompo-nentsofeachframe,whileaccumulatingthegeometricinformation.[19]presentsanarticulatedglobalregistrationalgorithmastheop-timizationofboththealignmentofrangescansandthearticulatedstructureofthemodel.

The?rsttypeofmethodsoftenrequireshighqualityinputdata,andiscomputationallyexpensive;thesecondoneneedsanaccuratetemplate,whichishardtoful?lformanyapplications,especiallyfordeformingobjects.Ourmethodusesaroughtemplatewhichiscon-structedbythe?rstdataframe.Basedonthefeaturecorrespondencesreturnedinthecorrespondingcolormaps,thetemplateisdeformed[17]andthusitcandrivethepointsaccordingly.Thankstothedefor-mationmodel[17],thepointsindifferentframescanthenbeapproxi-matelyaligned.

2.3GlobalAlignment

Globalalignment,especiallytheloopclosureproblem,iswell-knowninrigidscanning[20,21].Abrute-forcesolutionistobringallscan-sintotheIteratedClosestPoint(ICP)[22]iterationloop.However,thisoftenrequiressolvingawfullylargesystemsofequations.An-othergreedysolutionistoaligneachnewscantoallpreviousscans[23].However,itcannotspreadouterrorsofpreviousscans,andisnotguaranteedtoachieveconsistentcycle.

Weagreewiththeideaofcreatingagraphofpairwisealignmentsbetweenscans[24,25].First,pairwiserigidalignmentiscomputedinthegeometrylevel.Globalerrordistributionthenoperatesonanupperlevel,whereerrorsaremeasuredintermsoftherelativerotationsandtranslationsofpairwisealignments.Thegraphmethodscansimulta-neouslyminimizetheerrorsofallviewsrapidly,anddonotneedallscaninmemory.Thismakesitespeciallypracticalformodelswithlotsofscans.

Globalalignmenthasbeenlessstudiedinnon-rigidregistration.[26]warpstwoconsecutiveframesbyfeaturepointcorrespondencesandLaplaciancoordinatesconstraints[27].Toalignallframessimul-taneously,aglobalmatrixsystemlistsallconstraintsasDiagonalsub-matricesissolved,whichissimilartothebrute-forcesolution.[18]maintainstheaccumulated3Dpointcloudsofpreviousframes,andregistersthepointsofanewinputframetotheaccumulatedpoints,whichsuffersthesameproblemofthegreedysolution.

Anotherproblemofnon-rigidregistrationisocclusion.Inmostpre-viouspapers,occludedpartsareoftenpredictedbytheirtemporalorspatialneighbors[19,26].Theinterpolationbasedmethodsarehardtogetcorrectregistrationduetocomplexocclusions.Inourmethod,misalignmentcausedbycomplexocclusioncanalsobehandledrea-sonablyintheglobalalignmentprocedure.

TONG ET AL: SCANNING 3D FULL HUMAN BODIES USING KINECTS645

Fig.3.Overviewofourreconstructionalgorithm.

WiththreeKinects($450),twopillarswhich?xtheKinects($30)andoneturntable($120),thetotalcostofourscanningsystemisabout$600,whichismuchcheaperthantheconventional3Dscanningde-vices.Thusthesystemcanbeutilizedformanyhome-orientedappli-cations.4

RECONSTRUCTIONAPPROACH

DenoteDi={Mi,Ii},i=1,...,nasthecaptureddata,wherenisthenumberofthecapturedframes,MiisthemergedmeshandIiisthecorrespondingimageofthei-thframerespectively.4.1Overview

Generallythebodycanhardlybekeptrigidwhenthepersonstandsontheturntableduringthescanningprocess.Forexample,thearmsandheadmaybemovingandthechestisdeformedduetobreathing.

Weproposeapracticalapproachforregisteringmultipleframesofnoisypartialdataofhumanbodyundernon-rigiddeformation.Figure3showsanoverviewofoursystem.Aroughtemplateisconstruct-edbythedepthdataofthe?rstframe.Thenthetemplateisusedtodeformthegeometryofsuccessiveframespairwisely.Globalregis-trationisthenperformedtodistributeerrorsinthedeformationspace,whereproblemsofcycleconsistencyandocclusionarehandled.Thepairwiseregistrationandglobalregistrationiteratesuntilconvergence.Then,everyframeisdeformedtothe?rstframedrivenbythetem-plates.Finally,reconstructedmodelisgeneratedusingPoissonrecon-structionmethod[30].4.2TemplateGeneration

Unliketheworkof[16],anaccuratetemplateisunavailableinoursystem.Weusethemethodproposedin[31]toconstructanesti-matedbodyshapeasthetemplatemeshT1fromthe?rstframe.Duetothedatanoiseandin?uenceofdresses,theresultingtemplatecanonlyapproximatethegeometryofthebodyshape,asshowninFig-ure4.Itisimpossibletousethistemplatetoregistereachframebygeometry?tting.Wewillshowthatitisenoughtousethistem-platetotrackthepairwisedeformationofsuccessiveframes.DenoteT1={vk1},k=1,...,KwhereKisthenumberofthenodesofT1.Toreducethecomputationcost,wesimplifyT1sothatabout50-60nodesareusedinoursystem.

4.3PairwiseGeometryRegistration

SupposeMi,i=1,...,nformingacycle.fi,jdenotestheregistrationthatcandeformmeshMitoregisterwithmeshMj.Inthissection,weintroducehowto?ndthepairwiseregistrationf1,2,f2,3,...,fn?1,n,fn,1.

Fig.2.Thesetupofoursystem.ThreeKinectsareusedtocapturedifferentpartsofahumanbodyataclosedistancewithoutinterference.Thecaptureddataofasingleframefromthreesensorsisalsoillustrat-ed.

3SYSTEMSETUP

ThesetupofourscanningsystemisillustratedinFigure2.Toavoidtheinterferenceproblem,twoKinectsareusedtocapturetheupperpartandthelowerpartofahumanbodyrespectively,withoutover-lappingregion,fromonedirection.AthirdKinectisusedtocapturethemiddlepartofthehumanbodyfromtheoppositedirection.ThedistancebetweentwosetsofKinectsareabout2meters.Aturntableisputinbetweenthem.

Whilescanning,thepersonstandsontheturntableandrotates360degreesinabout30seconds.Pleaseseetheaccompanyingvideo.EachKinectacquires1280×1024colorimagesand640×480depthimagesat15framespersecondindividually.3Dcoordinatescanbeautomat-icallygeneratedusingOpenNI[28].Using[28],thedepthandcolorimagearealsowellcalibrated.Thecaptureddatafromthreesensorsaresynchronizedandcalibratedautomatically[29].

Adepthandcolorthresholdmethodisusedtoroughlysegmenttheforegroundhumandatafromthebackground.Then,sliverfacesfollowedbyverticesnotreferencedbyanyfacearedeleted.Asthebodyisonly1meterawayfromtheKinects,thequalityofdepthval-ues,comparedwiththatinFigure1(a),hasbeenimprovedgreatly,asshowninFigure2.Laplaciansmooth[27]isperformedtoreducethenoise.Toreducethecomputationcost,simpli?edmeshwith1/10verticesisusedinourexperiment,asshowninFigure3.

646IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 18, NO. 4, APRIL 2012

Fig.4.Anestimatedbodyshapeisconstructedfromthe?rstframeasthetemplatemesh(a).Toreducethecomputationcost,thetemplatemeshissimpli?edto50-60nodes(b).

Deformationmodel.Ourdeformationmodelisbasedon[17].Sup-posewehavetwomeshesMi??andMj,andthetemplatemeshatframe

iisTi.Denotefi,j={(Rki,tki)??vki∈Ti},wherevkiisanodeofTi,3×3

matrixRkiand3×1vectortkispecifythelocaldeformationinduced

byvki.Speci?cally,forapointpofMinearvki,itsdestinyp

?canbecalculatedasp?=Rki(p?vkkkregistration.Fori)+vtwoi+tsuccessivei.

PairwiseframesMrespondingfeaturepointscanbeobtainedbyopticaliandM?ow[32]i+1,cor-inthecorrespondingimages(seeFigure5).Particularly,weuse[33]to?ndthefeaturecorrespondencesbetweenMnandM1.Following[17],we

compute(Rki,tki)bysolving

(Rminkk(Ereg+wrotErot+wconEcon)

i,ti)

Suppose??vj

i,vkiaretwoneighboring∑??kj∈∑??nodesofMi,Ereg=??Rki(vji?vki)+vki+tk

i?(vjji+ti)????ensuresthesmoothness

N(k)

oftheneighboringdeformation.Letc3matrixR,Rot(R)=(c1,c2,c3bethe3×1columnvectorsof3×21·c2)2+(c1·c3)2+(c2·c3)2+(c1·c1?1)2+(c2·c2?1)+(c3·c3?1)2.Erot=∑Rot(Rkk

i)isusedtospecifytheaf?netransformationtoberotation.Supposevlindex(l)

kandvk+1

arecorrespondingfeaturepointsoftwosuccessiveisthedeformedpointofvl.E??con=∑??frames,v?ll

??v?l)kklindex(??

k?vk+1????isusedtomatchthefeaturepointsconstraints.Theenergyisminimizedusingstandard

Gauss-Newtonalgorithmasdescribedin[17].

Projectiontothe?rstframe.Afterthepairwiseregistration,wecan?ndthatitisoverdetermined.Sinceonlyn?1pairwisedefor-mationisneededtorecoveralltherelativepositionofallframes.

Here,todeformMjbacktothe?rstframe,wesimplyletM

?fj=2,1(...(fj?1,j?2(fj,j?1(Mj)))),wherefj,j?1istheinversetransfor-mationoffj?1,j.TheresultisshowninFigure6(a).Intheheadarea,the?rstandlastframesdonotmatchduetoerroraccumulationofpairwiseregistration.Thearmareahasmoreseriousproblemofmisalignmentwhichiscausedbycomplexocclusions.Soweneedaglobaldeformationregistrationtosolvetheseissues.4.4GlobalDeformationRegistration

Similartotheproblemariseninrigidregistration[25],thedesiredpairwisedeformationf?1,2,f?2,3,...,f?n?1,n,f?n,1shouldmeetthefollow-ingtwoconditions:

Fig.5.Correspondingfeaturepointsoftwosuccessiveframesandthedeformedtemplates.

Fig.6.(a)Afterpairwiseregistration,the?rstandlastframedoes

notwellmatchduetoerroraccumulation,asshowninthenosepart.Moreseriousproblemoccursbymisalignmentofcomplexocclusions,asshowninthehandpart.(b)Globaldeformationregistrationisusedtodealwiththeseproblems.

1.Itiscycleconsistent,thatis,thecompositionofanydeformationaroundacycleshouldbeidentity:

?i,f?i,i+1f?i+1,i+2...f?n?1,nf?n,1f?1,2...f?i?1,i=I

(1)

2.Theoriginalpairwisedeformationisrelativelycorrect,soweshouldminimizetheweightedsquarederrorofthenewandolddefor-mation:min∑w2??i,j??f???i,j?fi,j??2

(2)wheretheweightwi,jisthecon?denceofeachpairwisede-formationfi,j.Here,wesetwaveragedistancei,j=1/Dist(f(fofcorrespondingi,j(Mi),Mpointj),where

Disti,j(Mi),Mj)isthepairsoffi,j(Mi)andMTosatisfythesej.

conditions,let’scheckanodevioftemplateTneighborofvcanbeapproximatedasai.Inthei,thelocaldeformationro-tationri,i+1andtranslationtto?ndingr?i,i+1,?t

i,i+1.Findingoptimalf?i,i+1isequivalent

Toreducethei,complexityi+1foreachnode.oftheproblem,following[25],we?rstconsidertranslationtobeindependentofrotation,andfocusoncalcu-latingrotationr?i,i+1.LetthematrixEi,i+1betherotationsuchthat

r1,2r2,3...ri,i+1Ei,i+1ri+1,i+2...rn?1,n,rn,1=I.

/

Letαj,j+1=

11

w2,Ej,j+1

∑j

w2i<,iα+>=exp{αlnEj,j+1

1i,i+1}.Ithas

TONG ET AL: SCANNING 3D FULL HUMAN BODIES USING KINECTS647

Fig.7.Acycleoflocallyrigidpairwiseregistration.v1isregisteredtov2,v2isregisteredtov3,...,vnisregisteredtov1.

beenproventhatr?i,i+1=ri,i+1E<αi,i+1

>

constraint(1),andminimizesthei,satis?esthecycleconsistentenergyi+1

(2)inthemeaningofsquaredangularerror[25].?Afterr?i,i+1isset,wedistributetheaccumulatedtranslationerror.t

i,i+1shouldalsosatisfythecycleconsistentconstraint:r?i,i+1...r?i?2,i?1?ti?1,i+...+r?i,i+1?t

i+1,i+2+t?i,i+1=0,andminimizetheenergy:

∑w2????

???i,jt

i,j?ti,j??2.

Thisisaquadraticminimizationproblemwithlinearconstraints,anditcanbesolvedusingLagrangemultipliers.

Theweightederrordistributionstrategytakesadvantageofboththepairwiseregistrationandcycleconsistency.Toillustrateit,let’stakealookattwoextremecases.Supposeri,i+1,ti,i+1deformtheneighborofvitotheneighborofvi+1exactly,thenwehave

r?i,i+1=ri,i+1E<αi,i+1>

i,i+1

≈ri,i+1Ei<,i0+>

1=ri,i+1.

Inthiscase,thecorrectpairwisedeformationisreservedintheresult-ingdeformation.

Anotherextremecaseoftenhappenswhenthereiscomplexocclu-sion,asshowninFigure8.Duetolargenumberofframesofocclu-sion,theleftarminframei,jmayhavelittleintersectionarea,sotheresultingrsincewei,j,thave

i,jcantotallymisalignthecorrespondingareas.Howev-er,r?i,j=ri,jE<αi,j>

i,j

≈ri,jEi<,j

1>

=ri,i?1...r2,1r1,nrn,n?1...rj+1,j,reasonableregistrationcanstillbeobtainedusingthecycleconsistent

constraint.

Fig.8.Complexocclusionhappensduringthescanningprocess.Inthisexample,leftarmisoccludedfromframei+1toframej?1.

5RESULTS

Figure9(a)showstheresultofglobalrigidalignment.Itcanbeseentheresultisnotsogood,especiallyinthearmandheadareas.Our

Fig.9.Theresultofglobalrigidalignment(a)andourglobalnon-rigid

alignmentalgorithm(b);(c)thereconstructedmodelusingourmethod.Notethatthedresswrinklesarewellcapturedinthemodel.

globalnon-rigidregistrationcangetmuchbetterresult(Figure9(b)),andcanfurthergetveryimpressivemodel(Figure9(c)).

Toaligntwosuccessiveframes,Harriscornersarefoundinthe?rstimage,andthecorrespondingpixelsofthesecondimagearefoundastheICPclosestpointonmeshes.Usingtheseinitialestimations,Lucas-Kanadeoptical?ow[32]isusedto?ndmoreaccuratecorre-spondingfeaturepoints.Forareaswithlittletextureinformation,clos-estpointpairsareuseddirectly.Foreachpairwiseregistration,about2000pairsofcorrespondingfeaturepointsareused.

Occlusionsareinevitableinourexperiment,especiallyinthearmarea.Inourexperiment,onearmwillbeoccludedforabout1/6ofallcapturedframes.Theinterpolationbasedmethodsarehardtogetcorrectregistration,whilereasonableresultscanstillbeobtainedusingtheloopclosureconstraintattheglobalregistrationprocedure.

Afteronestepofglobalalignment,allmeshesaredeformed,andpairwisedeformationcanbefurthercalculated.Thepairwiseandglobalregistrationiterateswhentheaveragedistanceofallneighbor-ingmeshesisabove50%oftheaverageedgelength.Inourexperi-ment,thealgorithmconvergesforabout1to2iterations.Finally,1/6uniformlysampledframesareusedtogeneratereconstructedmodelbyPossionsurfacereconstruction[30].Sincethecolorimageanddepthimagecanbesimultaneouslycapturedandcalibrated[28],thecolorinformationofdeformedmeshisgeneratedautomatically.Us-ingthemethodof[34],texturedinformationcanbegeneratedforthereconstructedmodel.

Figure10showsmoreresultsofouralgorithm.Theaveragecom-putingtimeofeachstepwithanIntelCorei3processorat3.1GHzisshowninTable1.Sixbiometricmeasurementsarecalculatedontheconstructedhumanmodelsandarecomparedwiththosemeasuredonthecorrespondingrealpersons.TheaverageerrorincentimeterisshowninTable1.Itisseenthatourreconstructedmodelsapproximatetherealpersonswell.Largererrorsinthegirthmeasuresarecausedbythedresses.

Theaccuracyofourbiometricmeasurementsissimilartotheresultshownin[10].Unlike[10]whichusesthesubspaceofSCAPEpresen-