• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    The present invention relates to a system and a method for synchronizing images by visual interaction at a distance between at least one user and an expert. The system includes a user camera set up to capture the user's field of view presented to the expert as a user image (505) and an expert camera set up to image an object (502) as an expert object image (500). The system also includes a user computer configured to define a fixed point (507) in the user image (505), an expert computer configured to define a fixed point (501) in the expert object image (500) and a synchronization unit (508) arranged to synchronize the user image (505) and the expert image ( 500) by placing the user fix point (507) and the expert fix point (501) on top of each other (509) in the mixed image. (Figure 12)
    公开号:SE0950609A1
    申请号:SE0950609
    申请日:2009-08-25
    公开日:2011-02-26
    发明作者:Torbjoern Gustafsson；Per Carleberg
    申请人:Xmreality Res Ab；
    IPC主号:
    专利说明:

    15 20 25 30 35 The expert, on the other hand, should, in some form, be centrally located in a company or organization so that as many people as possible can quickly get help from him / her. It can be both expensive and impractical to send out an expert on various assignments in order to help a user.
    Through modern technology, however, it is possible to let the expert be virtually in place with the user and guide him visually and verbally.
    There are several different variants of systems and methods, with different advantages and disadvantages, which allow a visual and verbal interaction and communication between a user and an expert.
    An example of such a known system and method, which has been shown at an exhibition and described in report form, consists of two main-stage micro-display equipment (presentation equipment) with integrated video camera. The user carries one piece of equipment and the expert the other. The user's and the expert's equipment is connected by means of computers and transmission channel. The expert's hand, arm or any other object that the expert has placed on a work surface can be extracted from the image from the expert's video camera and superimposed in the image the user sees and which describes what the user has in front of them. The principle of the known system is shown in Figure 1 where: 0 100 shows a "user" 0 101 shows a user's main microdisplay equipment 0 102 shows a video camera integrated in a user's presentation device 0 103 shows a computer used by a user 0 200 shows an "expert" 0 201 shows an expert's main microdisplay equipment 0 202 shows a video camera integrated in an expert's presentation device 0 203 shows a computer used by an expert 0 204 shows a work surface 0 205 shows an expert's hand and arm 0 300 shows a transmission channel between a user and an expert Users and experts also each have their own verbal communication device that includes a microphone and speaker device. This device is not shown in the figure. 10 15 20 25 30 However, the above-mentioned system and solution have certain disadvantages: 1. Since both the user's and the expert's equipment is head-mounted, and thus moves in step with the head's movements, it can be difficult to spatially (spatially) synchronize the user's and expert's image. in such a way that a visual interaction is practically possible. It can thus be difficult for the expert to be able to point to a specific point on the equipment in question, whether the expert or the user or both are touching the head. 2. The delay of the transmission channel may further impede the visual interaction in a manner similar to main movements. 3. The expert can possibly guide several users at the same time, ie switch between different users and will then have to spend longer periods with the main equipment. The expert may then find his equipment uncomfortable and tiring.
    SUMMARY OF THE INVENTION Accordingly, an object of the invention is to provide a system and method for synchronizing images in visual interaction at a distance between an expert and at least one user in an interaction system comprising at least one workstation comprising a presentation device and a camera for use by the user. said workstation is operatively connected to a support station comprising a presentation device and a camera for use by the expert, the user camera is arranged to capture the user's field of view presented to the expert via the expert presentation device as a user image and the expert camera is arranged to image an object at the support station as an expert object image and mix combine the expert object image with the user image presented to the user via the user presentation device as a mixed image.
    According to one aspect of the invention, this object is achieved by a system according to the characterizing part of claim 1, which states that images in visual interaction at a distance between an expert and at least one user are synchronized by the system comprising a user computer arranged to define a fixed point in the user image. (505), an expert computer arranged to define a fixed point in the expert object image and a synchronization unit arranged to synchronize the user image and the expert image by placing the user fix point and the expert fix point in a predetermined manner over each other in the mixed image.
    According to another aspect of the invention, this object is achieved by a method according to the characterizing part of claim 9, which states that images in visual interaction at a distance between an expert and at least one user are synchronized by the method comprising the steps of defining a fixed point in the user image. to define a fixed point in the expert object image and to synchronize the user image and the expert object image by placing the user fix point and the expert fix point in a predetermined manner on top of each other in the mixed image.
    Advantageous embodiments of the invention appear from the subclaims.
    BRIEF DESCRIPTION OF THE DRAWINGS In the drawings, the same reference numeral denotes the same elements throughout the various figures and: Fig. 1 shows an example of a prior art system for remote interaction; Figs. 2a - 2e show relevant parts of the user's equipment; Figs. 3 - 9 show relevant parts of the expert's equipment in different embodiments; Fig. 10 shows relevant parts of the method for visual interaction; Figs. 11 - 12 show relevant parts of the method for spatial synchronization; Fig. 13 shows examples of image compression; Fig. 14 is a flow chart showing the method steps according to the invention.
    DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS The invention can be roughly divided into the following parts: 1. The user's equipment 10 15 20 25 30 35. The following parts are described in the text: 1. The user's equipment The user uses: a head-worn (helmet-worn) or hand-held equipment consisting of a visual presentation device, a camera unit and a device for attaching a head or helmet, or a hand grip. Note, however, that the user could remove or put away this equipment, but attach it in some way in front of him so that the user can still obtain a visual experience while the user performs some work step. a computer, or any equipment with computer-like computing power a transmission channel of some kind an interaction tool a device for verbal communication The presentation device can be for example: A microdisplay system with magnifying optics, to be used near the eyes A projector A display (monitor) of a computer type, mobile phones, etc.
    The camera unit can be, for example: 1. 2.
    A camcorder sensitive in the visual area A camcorder sensitive in other areas, such as near-IR (Infra Red), thermal IR or UV (Ultra Violet) Consisting of several different types of camcorder interaction tools can be, for example: 1. 2. 3.
    A keypad (mechanical control using body part) Voice control unit Eye control unit 10 15 20 25 Gesture control unit Consisting of a combination of different control variants A physical unit, such as a block or plate, with a pattern that a camera can detect. Different types of virtual objects can be connected to the patterns, for example 3D objects, images, texts, etc. These virtual objects can be mixed into the user's and / or the expert's image.
    The unit for verbal communication can be, for example: 1.
    Traditional telephone system 2. Mobile telephone system 3. Based on computer / internet, for example “Voice-IP” Relevant parts of the user's equipment, in different typical cases, are described in figure 2 where: 100 shows a “user” 101 shows a user's main display device (microdisplay system ) 102 shows a video camera integrated in a user's display device 104 shows a device for attaching a display device to head or helmet 105 a user's handheld presentation device (micro display system) 106 shows a device for a hand grip 107 shows a user's handheld presentation device in the form of a monitor of the type available for computers, mobile telephones, etc. 108 shows a user's presentation device in the form of a head-mounted video projector 109 shows a user's presentation device in the form of a handheld video projector The user also has a verbal communication device which includes a microphone and speaker device. This unit is not shown in Figure 2. 2. The expert's equipment and method for extracting objects The expert is sitting or standing at a device that can be compared to a fixed installation, but could also be mobile. What distinguishes this equipment from the user's equipment is that the expert's equipment, in general, is not worn by the expert in such a clear way as the user does.
    The expert's device consists of: a work surface on which an image is projected and on the (or over) surface of which the expert's hand / arm and / or objects can be placed or move a computer, or any equipment with computer-like computing capacity a transmission channel of some kind a interaction tools which may for example be: I. A keypad (mechanical control using body part) ll. Voice control unit lll. Eye control unit IV. Gesture control unit V. Consisting of a combination of different control variants VI. A physical device, such as a block or plate, with a pattern that a camera can detect. Different types of virtual objects can be connected to the patterns, for example 3D objects, images, texts, etc. These virtual objects can be inserted into the image the expert sees, and thus the expert can, for example, interact and point to these virtual objects, as they had been real objects. These virtual objects could be said to look like "mask camera" and "object camera" (see below). a unit for verbal communication a video camera, "object camera", located above the work surface and which registers (films) the objects that are on / over the work surface, for example a hand another video camera, "mask camera", whose task is to mask out the existing objects on the work surface from the image projected on the work surface. Note that the subject camera can also act as a mask camera. The mask camera can be placed above or below the work surface. an image-generating device that "projects", real or virtual, an image onto the work surface. This unit can, for example, be designed according to the following proposals: l. A horizontal monitor (such as a flat screen TV or flat computer monitor of a larger type) on which a transparent and durable glass or plastic disc is placed. An optical filter can be placed between the monitor and the disc. The filter can be transparent (largely) for visual radiation (lets the screen image 10 15 20 25 30 so that the eye perceives it), but opaque (largely) for radiation other than the visual, for example near-IR (Infra Red) radiation or UV radiation. The filter could also be of a type that blocks certain ranges within the visible wavelength band. ll. A "back-projection" surface (screen + disc (glass / plastic)) on which a projector projects an image from below. In front of the projector lens, a filter can be placed that only allows visual radiation (largely) to pass. The filter could also be of a type that blocks certain ranges within the visible wavelength band. lll. A disc (white or with a pattern) over which a projector is placed "front-projection". In front of the projector lens, a filter can be placed that only allows visual radiation (largely) to pass. The filter could also be of a type that blocks certain ranges within the visible wavelength band.
    IV. A flat screen that is placed between the expert's head and the expert's work surface.
    V. A microdisplay system located between the expert's head and the expert's work surface.
    Relevant parts of the expert's equipment, in different typical cases, are described in Figures 3 - 9 where: 0 200 shows an “expert” 0 204 shows a work surface 0 206 shows a horizontal screen (type flat screen TV or flat screen computer of a larger type) 0 207 shows a video camera, "object camera" 0 208 shows another video camera, "mask camera" 0 209 shows an optical beam splitter, such as a "hot mirror" or a "cold mirror" (beam splitter with different reflecting and transmitting properties depending on the distance range) 0 210 displays an optical filter, in front of the mask camera, which blocks certain wavelengths, for example the visual wavelengths 0 211 shows a video projector 10 15 20 25 30 35 0 212 shows an optical filter, in front of the video projector, which blocks certain wavelengths, for example when-IR 0 213 shows a beam splitter for visual wavelengths, (beam splitter) 0 214 shows a screen / disc for “back” or “front-projection” 0 215 shows a screen / disc that acts as an optical filter that blocks certain wavelengths, e for example when-IR 0 216 shows a video screen, (computer screen), with flat or other design 0 217 shows a microdisplay with magnifying optics 0 218 shows a mechanical fastening device, for example a mechanical arm Note that alternative solutions are shown in the dashed circles in figure 3 - The expert also has a verbal communication device that includes a microphone and speaker device. This unit is not shown in Figure 3 - 9. Also note that in Figure 7-9 the expert sees a similar image as the user, ie his own objects mixed in the image the expert sees.
    Each figure 3-9 is described separately: Description of figure 3: The expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, for example a hand or a tool. Below the work surface is a flat screen, 206, which shows i.a. user view. Above the monitor is an optical filter, 215, which blocks near-IR radiation. At a distance of 50-100 cm above the work surface, two cameras are placed, the object camera 207 and the mask camera 208. In front of the mask camera is an optical filter, 210, placed which blocks visual radiation but transmits near-IR radiation. In the left dashed ring in the figure, the object camera and mask camera are arranged in such a way that they share the optical axis via a beam splitter, a "hot mirror" which is transparent to visual radiation and reflective to near-IR radiation. The right dashed ring in the figure shows an alternative solution with object and mask camera, where these are placed next to each other. When the expert places an object, object, on the work surface, the mask camera extracts this object against the, in this case, known background on the work surface. The flash camera is not disturbed by the changing image from the monitor because that image does not reach the mask camera via the optical filters. On the surfaces in the image which the mask camera has masked, the corresponding surfaces from the image are then applied from the object camera, and in this way a new image is obtained with a known and well-defined background and the object 10 15 20 25 30 10 in question. This image is sent to the user, whose computer can extract the object in question, and then mix this object into the user's image, view.
    Description of Figure 4: The expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool. Below the work surface is a "film screen" or "film disc", 214, with "back-projection" properties. A projector, 211, projects an image onto the "screen". An optical filter, 212, located in front of the projector's lens blocks near-IR radiation from the projector, if any. At a distance of 50-100 cm above the work surface, two cameras are placed, the object camera 207 and the mask camera 208. In front of the mask camera is an optical filter, 210, placed which blocks visual radiation but transmits near-IR radiation. In the left dashed ring in the figure, the object camera and mask camera are arranged in such a way that they share the optical axis via a beam splitter, a "hot mirror" which is transparent to visual radiation and reflective to near-IR radiation. The right dashed ring in the figure shows an alternative solution with object and mask camera, where these are placed next to each other. When the expert places an object, object, on the work surface, the mask camera extracts this object against the, in this case, known background on the work surface. The mask camera is not disturbed by the changing image from the monitor because that image does not reach the mask camera via the optical filters. On the surfaces in the image that the mask camera has masked, the corresponding surfaces from the image are then applied from the object camera, and in this way a new image is obtained with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract the object in question, and then mix this object into the user's image, view.
    Description of Figure 5: The expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool. Integrated in the work surface is a "film screen" or "film disc", 214, with "front-projection" properties. A projector, 211, which is placed at a distance of 50-100cm, projects an image on the "film screen". An optical filter, 212, located in front of the projector's lens blocks near-IR radiation from the projector, if any. The subject camera, 207, is located close to the projector. In the left dashed ring in the figure, the object camera and projector are arranged in such a way that they divide the optical axis via a beam splitter, 213. The right dashed ring in the figure shows an alternative solution with object camera and projector where these are placed next to each other. 10 15 20 25 30 35 11 Under the work surface at a distance of 25-100 cm, the mask camera is placed. In front of the mask camera, an optical filter, 210, is placed which blocks visual radiation but transmits near-IR radiation. When the expert places an object, object, on the work surface, the mask camera extracts this object against the, in this case, known background on the work surface. The mask camera is not disturbed by the changing image from the monitor because that image does not reach the mask camera via the optical filters. The corresponding surfaces from the image from the object camera are then applied to the surface images that the mask camera has masked out, and in this way a new image is obtained with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract the object in question, and then mix this object into the user's image, view.
    Description of Figure 6: The expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool. Integrated in the work surface is a "film screen" or "film disc", 214, with "front-projection" properties. A projector, 211, which is placed at a distance of 50-100cm, projects an image on the "film screen". An optical filter, 212, located in front of the projector's lens blocks near-IR radiation from the projector, if any. Close to the projector, the subject camera, 207, and the mask camera, 208, are located. In the left dashed ring in the figure, the object camera and mask camera are arranged in such a way that they share the optical axis via a beam splitter, a "hot mirror" which is transparent to visual radiation and reflective to near-IR radiation. The right dashed ring in the figure shows an alternative solution with object and mask camera, where these are placed next to each other. In front of the mask camera, an optical filter, 210, is placed which blocks visual radiation but transmits near-IR radiation. When the expert places an object, object, on the work surface, the mask camera extracts this object against the, in this case, known background on the work surface. The mask camera is not disturbed by the changing image from the monitor because that image does not reach the mask camera via the optical filters.
    The corresponding surfaces from the image from the object camera are then applied to the surface images that the mask camera has masked out, and in this way a new image is obtained with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract the object in question, and then mix this object into the user's image, view.
    Description of Figure 7: The expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool. A monitor, 216, is located between the expert's head and the work surface. Near, or on, this monitor is a camera located that acts as both an object and mask camera, 207/208. Car screen and camera are held in place by a mechanical device, 218. The work surface can have such a color or pattern that allows objects to be masked out of the work surface. The work surface can be, for example, blue or green, and then the masking can be done by "chroma-key" technology, thus obtaining a new image with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract the object in question, and then mix this object into the user's image, view.
    Description of Figure 8: The expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool. A microdisplay system with magnifying optics, 217, is located between the expert's head and the work surface. Near, or on, this monitor is a camera located that acts as both an object and mask camera, 207/208. Car screen and camera are held in place by a mechanical device, 218. The work surface can have such a color or pattern that allows objects to be masked out of the work surface. The work surface can, for example, be blue or green, and then the masking can be done using "chroma-key" technology. In this way, a new image is obtained with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract the object in question, and then mix this object into the user's image, view.
    Description of Figure 9: The expert, 200, stands or sits in front of a work surface, 204, on which the expert can place objects, such as a hand or a tool. A display type monitor, 206/216 or projector projected surface, 214, is placed on the side next to the expert. Above the work surface at a distance of 50-100 cm, a camera is placed that functions as both an object and a mask camera, 207/208. The work surface can have such a color or pattern that allows objects to be masked out of the work surface. The work surface can, for example, be blue or green, and then the masking can be done using "chroma-key" technology. In this way, a new image is obtained with a known and well-defined background and the object in question. This image is sent to the user, whose computer can extract the object in question, and then mix this object into the user's image, view. 10 15 20 25 30 35 13 3. Method for visual interaction In order for users and experts to experience that they are in the same “virtual room”, a method for visual interaction must be used. The method is based on moving images being transferred (and mixed) between user and expert and vice versa. Such a method can work as follows: 1. An image (image sequence) describing the user's field of view, the 'user image', is transmitted partly to the user's computer (calculation unit) and partly via a transmission channel to the expert's computer (calculation unit). 2. The expert's presentation device displays the "user image" for the expert. 3. The expert places an object, such as his hand, on his work surface and this object is extracted using the mask camera and the object camera, and this object is thus described as a subset of the “expert object image.” Note that this gives the expert a mixed visual experience of a real object is placed on a presentation device that displays the "user image". 4. The "expert object image" is transmitted via a transmission channel to the user's computer where the "expert object image" is superimposed on, (mixed with), the "user image" (the instantaneous). mixbi | den. ”5. The user's presentation screen displays the“ mix image ”for the user.
    Note that the "expert object image" (sequence) is slightly delayed relative to the "user image" (sequence).
    Relevant parts of the visual interaction method are described in Figure 10 where: 0 300 shows a transmission channel between a user and an expert 0 301 shows a one-way transmission channel between a user and an expert 0 302 shows a one-way transmission channel between an expert and a user 0 120 shows a user's computer (computing device) 0 220 shows an expert's computer (computing device) 0 110 shows a user's presentation device 0 121 shows a user's equipment 0 221 shows an expert's equipment 0 222 shows an expert's presentation device / workspace 0 223 shows an expert's mask camera / object camera 0 224 shows an object provided by the expert 10 15 20 25 30 14 0 122 shows a user's view / object 4. Method for spatial synchronization (spatial) synchronization As previously mentioned, there may be a problem with the spatial synchronization between the user and the expert. The synchronization is disturbed partly by the user's movements and partly by the delay caused by the transmission channel. One way to solve this synchronization problem is described below (see also more detailed explanation of spatial synchronization below after the points): 1. The expert's presentation device is, in principle, independent of how the expert moves his head. Thus, in the image presented in this presentation device, a fix point, "expert fix point", can be obtained, for example in the middle of the image. This can be predefined in terms of size (z-position), position (x, y) and rotation. 2. The user touches, for the most part, his head and thus changes the view that the user's camcorder registers. In the user's view, however, there may be image properties such as an edge, a corner or a bright dot, which a “feature detection” system can detect. If the conditions are not such that certain image properties are too weak, the user can place his own "marker" describing a pattern that can be detected. This marker can be attached, for example, with a magnet or tape. 3. The user and / or the expert can by a command "freeze" an image from the user from which a number of "features" are detected. The user's computer (system) and / or the expert's computer (system) can also, according to any timetable, automatically "freeze" and detect "features". These "features" may form the basis for defining a fix point, "user fix point", for example centrally in the image. The user fix point is defined by its size (z-position), position (x, y) and rotation. the expert object image ”and the“ user image ”must be spatially synchronized, the images are superimposed so that the“ expert fix point ”and the“ user fix point ”coincide, of course given that these fixed points are present in the instantaneous images. This could possibly be compensated for by the expert's computer system "remembering" the surroundings of the instantaneous image from the user, 10 15 20 25 30 15 and thus filling in the parts that have fallen away with parts from the "memory", which of course are not updated in time with reality.
    Spatial synchronization here means that when the images are mixed together, the system takes into account both how the expert's and the user's view is synchronized in the (x, y) direction, in the z direction and in terms of rotation. Regarding synchronization in z-direction, this means that an enlargement or reduction of any image must take place in order for complete spatial synchronization to be performed. Spatial synchronization also means that when the images are mixed together, the system also takes into account the rotation views and rotates the images correctly, ie synchronizes them. The spatial synchronization can be determined using the "features" or "markers" that the system detects. These change size and interrelationship as the user approaches or moves away from them, or keeps their head tilted.
    Relevant parts of the method for spatial synchronization are described in Figures 11 - 12 where: 500 shows the image registered by the expert's object camera, the "expert object image" 0 501 shows a centrally located fix point, the "expert fix point", in the "expert object image" 0 502 shows an "object" in the "expert object image" 0 505 shows the image the user sees, the "user image" 0 506 shows detectable "features" in the "user image" 0 507 shows a fixed point, "user fix point", whose position in the image is calculated with respect to relevant "features" 0 508 shows a device that symbolizes spatial synchronization of the "user image" and the "expert image" 0 509 shows results of, in the point above, synchronization 0 510 shows the image the expert sees after the synchronization 0 511 shows a field in the expert image that does not have continuous updating from However, the field can be filled with image information from the “memory” where image information of the field may have been stored since previously in image 0 512 shows the image user one looks at the synchronization 0 513 shows a field in the user's image which does not have continuous update from the expert 10 15 20 25 30 35 16 Description of figure 11.
    This figure shows how a synchronization takes place when the user is not moving.
    The Expert Fix Point, 501, and the User Fix Point, 507, do not change position, size, and rotation. Therefore, the synchronization, 508, becomes simple and the two fix points are placed correctly on top of each other, 509.
    Description of Figure 12.
    This figure shows how a synchronization takes place when the user moves.
    The expert fix point, 501, does not change position, size and rotation. In contrast, the user fix point, 507, as defined in Figure 11, changes position, size, and rotation. Therefore, the synchronization, 508, becomes complicated because the user image and the expert object image must be adapted in such a way that the two fix points are placed correctly on top of each other, 509. The synchronization has certain consequences regarding the images the user and the expert have in their field of view. In the image the user sees, 512, some surfaces from the expert, 513, are not continuously updated. In the image the expert sees, 510, some surfaces from the expert, 511, are not continuously updated. 5. Method for compressing transmitted image data There are many methods for compressing image and audio data. The area is well developed with methods such as MPEG4 and JPEG2000.
    Depending on the transmission capacity (bit / time unit) and delay (time to transmit a bit) on the transmission channel used, picture and sound quality may need to be adjusted.
    The image quality depends on the compression method used and the degree of compression. The relationship between transmission capacity and image quality should in principle be kept constant and at such a level that stable interactivity is achieved, both in terms of image and also sound. If the delay on the transmission channel is shorter in time than the time it takes to transmit an image, you can allow the image quality to deteriorate in order to maintain interactivity.
    Compression methods for high image quality normally provide a greater delay than other methods, as it takes longer for compression and decompression. However, by using special hardware solutions, time for compression and decompression can be reduced. 10 15 20 25 30 35 17 Instead of transferring the expert's 'object' as a whole image, it would be possible to have only relevant parts of the image transferred. For example, each row (or column) in the image would contain a number of start and stop values for relevant image data, ie contain data from some "object".
    Figure 13 shows a schematic diagram of an example of a compression procedure where: 0 600 shows a color camera that generates a “Bayern Pattern” signal, (compressed color signal). In another embodiment, the camera could, for example, generate an RGB or YUV image, whereby moments in the figure could be omitted. 0 601 shows a module that converts the “Bayern Pattern” image into three images, an R (red), a G (green) and a B (blue) image. 0 602 shows three compression modules where each image is compressed, for example with JPEG2000. 0 603 shows a transmission channel where the compressed images are transmitted, for example one after the other, i.e. sequentially. 0 604 shows three decompression modules where the images are decompressed according to JPEG2000. 0 605 shows a module that combines the decompressed images into one color image. 0 606 shows a presentation device for the user / expert.
    Note that the procedure described above can be used by expert and to users, and vice versa.
    According to a general embodiment of the invention shown in Figure 14, the invention provides a method for synchronizing images in visual interaction at a distance between an expert and at least one user in an interaction system comprising at least one workstation comprising a presentation device and a camera for use by the user, said workstation is operatively connected to a support station comprising a presentation device and a camera for use by the expert, the user camera is arranged to capture the user's field of view presented to the expert via the expert presentation device as a user image and the expert camera is arranged to image an object at the support station as an expert object image and mix together the expert object image with the user image presented to the user via the user presentation device as a mixed image. The method includes the steps of: taking the user image with the user camera (step 700); define a fixed point in the user image (step 701); present the user image to the expert via the expert presentation device (step 702); take the expert object image with the expert camera and mask the object (step 703); define a fixed point in the expert object image (step 704); synchronize the user image and the expert object image by placing the user fix point and the expert fix point in a predetermined manner regarding position, size and rotation on top of each other into a mixed image (step 705); present the mixed image to the user (step 706).
    权利要求:
    Claims (1)
    [1]
    A system for synchronizing images in the case of visual interaction at a distance between an expert (200) and at least one user (100) in an interaction system comprising at least one workstation comprising a presentation device (101) and a camera (102) for use by the user (100), said workstation is operatively connected to a support station comprising a presentation device (201) and a camera (202) for use by the expert (200), the user camera (102) is arranged to capture the user's (100) 100) field of view presented to the expert (200) via the expert presentation device (201) as a user image (505) and the expert camera (202) is arranged to image an object (502) at the support station as an expert object image (500) and mix together the expert object image (500) with the user image (505) presented to the user (100) via the user presentation device (101) as a mixed image, characterized in that the system comprises: computer (120) arranged to define a fixed point (507) in the user image (505); an expert computer (220) arranged to define a fixed point (501) in the expert object image (500); a synchronizing unit (508) arranged to synchronize the user image (505) and the expert image (500) by placing the user fix point (507) and the expert fix point (501) in a predetermined manner one above the other (509) in the mixed image. System according to claim 1, characterized in that the equipment of the workstation is either head-mounted or hand-held. System according to claim 1, characterized in that the equipment of the support station is either permanently installed or mobile. System according to claim 1, characterized in that the support station comprises a work surface on which the user image is presented and which is arranged to receive said object. System according to claim 4, characterized in that, the expert presentation device is an image generating unit which provides an image of the work surface and is one of the following: a monitor; a projector arranged to project the user image onto the work surface from below; or, a projector set up to project the user image onto the workspace from above. System according to claim 1, characterized in that the support station comprises a microphone and a loudspeaker enabling verbal communication for the expert with the user. A system according to claim 1, characterized in that the support station comprises an interaction tool which is at least one of the following: a keypad; a voice control unit; an eye control unit; a gesture control unit; or, a physical object which the expert camera is set up to detect. System according to claim 1, characterized in that the system further comprises at least one compression module and at least one decompression module located at the workstation and / or the support station for compressing and decompressing the user image and / or the expert object image, respectively. Method for synchronizing images in visual interaction at a distance between an expert and at least one user in an interaction system comprising at least one workstation comprising a presentation device and a camera for use by the user, said workstation being operatively connected to a support station comprising a presentation device and a camera for using the expert, the user camera is arranged to capture the user's field of view presented to the expert via the expert presentation device as a user image and the expert camera is arranged to image an object at the support station as an expert object image and mix the expert object image with the user image presented to the user via a user image device. characterized in that the method comprises the steps of: - defining a fixed point in the user image; - define a fixed point in the expert object image; synchronize the user image and the expert object image by placing the user fix point and the expert fix point in a predetermined manner on top of each other in the mixed image. Method according to claim 9, characterized in that the method further comprises the step of compressing the user image and / or the expert object image before it is transferred to the workstation and the support station, respectively.
    类似技术:
    公开号 | 公开日 | 专利标题
    US10579320B2|2020-03-03|Display system, display device, information display method, and program
    US10642564B2|2020-05-05|Display system, display device, information display method, and program
    CN108957761B|2021-08-13|Display device and control method thereof, head-mounted display device and control method thereof
    US7817104B2|2010-10-19|Augmented reality apparatus and method
    US10303435B2|2019-05-28|Head-mounted display device, method of controlling head-mounted display device, and computer program
    KR20160021284A|2016-02-24|Virtual object orientation and visualization
    Krum et al.2012|Augmented reality using personal projection and retroreflection
    KR20140125183A|2014-10-28|Eye-glasses which attaches projector and method of controlling thereof
    WO2013166362A2|2013-11-07|Collaboration environment using see through displays
    KR20160091316A|2016-08-02|Video interaction between physical locations
    WO2013056187A1|2013-04-18|User controlled real object disappearance in a mixed reality display
    CN103731659A|2014-04-16|Head-mounted display device
    KR20130034125A|2013-04-05|Augmented reality function glass type monitor
    US20160070101A1|2016-03-10|Head mounted display device, control method for head mounted display device, information system, and computer program
    CN203858414U|2014-10-01|Head-worn type voice recognition projection device and system
    CN107003520A|2017-08-01|See-through display optical texture
    WO2008036931A3|2008-07-17|3-d displays and telepresence systems and methods therefore
    CN108431738A|2018-08-21|Cursor based on fluctuation ties
    JP6540426B2|2019-07-10|Display system, display device, information display method, and program
    KR102200491B1|2021-01-08|Augmented reality device and method
    CN105635713B|2019-07-26|A kind of display methods and electronic equipment
    JP6492673B2|2019-04-03|Head-mounted display device, method for controlling head-mounted display device, computer program
    CN108427194A|2018-08-21|A kind of display methods and equipment based on augmented reality
    SE0950609A1|2011-02-26|Visual interaction system and method
    EP3465631B1|2020-07-08|Capturing and rendering information involving a virtual environment
    同族专利:
    公开号 | 公开日
    SE537143C2|2015-02-17|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    法律状态:
    优先权:
    申请号 | 申请日 | 专利标题
    SE0950609A|SE537143C2|2009-08-25|2009-08-25|Visual interaction system and method|SE0950609A| SE537143C2|2009-08-25|2009-08-25|Visual interaction system and method|
    PCT/SE2010/050916| WO2011025450A1|2009-08-25|2010-08-25|Methods and systems for visual interaction|
    [返回顶部]