Explain Like I'm Five: Optical Camera Communications



Notes from the author

The purpose of this series of entries "Explain Like I'm Five" is to greatly simplify concepts in a way that is accessible for anyone without a background on the topic. The idea comes from a subreddit with the same name and purpose.

The posts deliberately follow a lie-to-children model, where ideas are first presented in a technically inaccurate but useful form, to be refined later on.

Feel free to make any contribution, revision, or feedback in the comments section. They are more than welcome, and they will be useful to deliver quality content!.


What is Optical Camera Communications?

Optical Camera Communications is a technology that enables data transmission between conventional lamps (such as stand lamps, floodlights, or even displays, screens, tv monitors...) and cameras (smartphone, surveillance, laptop's cameras...).

The data being transmitted could be of any kind, could be the user location in a building, marketing advertisements, area access codes, user emotional state, websites...

From a user perspective, what he has to do is to point the camera to the lamp, and... let's start the communication.

Examples Basic scenario where the user points the camera to a circle lamp.

Notes If you feel comfortable with theory or you are just looking for examples, go directly to the last section of this article, where I have uploaded some images and videos showing the technology.

...So... wait, how cameras and lamps could talk to each other?

Well... Said like that sounds a little bit confusing. Taking apart the communication aspect, it is well-known to us that cameras receive light coming from any direction to output an image. Therefore when the camera is pointing to a lamp, it captures its light during a short period and generates a picture where it can be seen. No need to mention that if the light is off while the camera is recording, we would be capturing just the turned off lamp's surface.

Therefore within an image, a light source can have a particular identifiable state, whether it is on, off, even partially on, or has a different color (violet, blue, yellow). Now let's go back to the communication aspect. If we assign a specific block of data to a particular lamp' state, we have set a symbol. Now, we could rely on detecting those symbols within consecutive images and convert them into useful data.

For example, we can assign a blue-colored lamp to the "Hello" word, so any time that we recognize a blue surface in an image, we know that the lamp is welcoming us.

Examples * Fundamental approach to camera communications.*

...Interesting enough... but what about speed, how fast could a lamp change its state?

That's a great question! To answer that, let's choose a generic camera: the conventional camera that we could have in a medium quality smartphone. This type of camera usually records videos with excellent resolution at 30 frames per second. That means that the camera records 30 images every second ... and... this is a limiting factor in communications. If the lamp changes its states faster than we can record, we will start losing data. In this setup, the light could not vary more quickly than 15 changes per second.

...Wow! Then, is it the communication fast or slow?

Well... it depends. It depends highly on the number of lamp' states, symbols, we could set at the start. If we could differentiate a bunch of lamps' states (red, green, blue, off, on, partially red, and so on...), we could have a relatively fast communication. Nevertheless, there is always a limit in the number of different discriminable states.

However, we could increase the speed if we use, for example, another lamp or transmitter. Or maybe three of them. And split the data between them. Think about that, three transmitters could provide a communication link that is three times faster.

Examples Multiple lamps communications. Refers to MIMO systems.

In conclusion, the frame rate (the number of images per second) of the camera limits the maximum switching speed of the lamp inherently, and thus the communication speed.


Global-shutter vs. Rolling-shutter cameras

Now that you have a light sense of how we can receive data using a camera and a lamp, let's rebuild the concept but using adequate terminology.

The camera I was mention uses a specific image acquisition mode called Global-shutter. The acquisition mode will tell us how the camera acquires the image. In this mode, while taking a photograph, the whole pixels of the image sensor is exposed to light at the same time.

Let's breakdown this a little bit (if you know how cameras work, step this paragraph). Cameras contains an image sensor that is in charge of converting light into images. To do this, the image sensor is composed of a matrix of n-by-m photodiodes or photoreceptors (grouped in pixels). Each photodiode converts light coming from one specific direction into an electric current. Intense light generates intense output currents. The output current is then measured and assigned a value, in a way that higher values refer to higher currents, thus higher luminous intensity. In the end, the camera hardware assembles the image using the overall pixel values obtained.

However, there is another image acquisition mode called rolling-shutter. In this mode, the image is scanned row by row of pixels sequentially in an overlapped manner. First, the top row of the camera starts the scanning, then the next one up to the final bottom row.

This could lead to artifacts that we could see when recording moving plane's propellers. This distortion effect is caused by the propellers' motion while the camera is taking a photograph. This video provides more information about this effect.

Examples Rolling shutter effect after taking a photo of moving plane's propellers.


...Okey... but why is important the camera acquisition mode?

In terms of communications, the essential difference of rolling-shutter cameras versus global-shutter cameras is that they allow capturing light variations within one image. Thanks to the row-by-row scanning, the lamp could switch at higher speed, generating an image with a compound of horizontal bands that could be used for communication. To illustrate this effect, I've created the following animation.

Rolling-shutter Rolling-shutter scanning simulation of a switching lamp.

On the right side, we can see the output image generated by a rolling-shutter camera that scans a switching circle lamp (left side). To fully understand the difference between rolling-shutter vs. global-shutter, I've made another animation showing the behavior of global-shutter cameras.

Global-shutter Global-shutter scanning simulation of a switching lamp.

In this case, despite the lamp is switching at high speed, the camera could only capture one state and loses the rest of them. In other words, the camera is blind to some of the lamp's symbols.

So we arrive at the following conclusion. Rolling-shutter cameras allow us to increase the switching speed of the lamp and, hence the communication speed.

Still, this is a very ideal scenario. Actually, we must tweak some camera parameters to achieve these results: the exposure time of the camera.


The exposure time of rolling shutter cameras

Conventionally the exposure time is defined as the duration while one row of pixels starts scanning until it finishes. Notice that a row of pixels does not capture light instantaneously but during a short period.

Setting the exposure time has an essential significance during communication. Long exposure times generate brighter images as photodiodes accumulate light during a prolonged time. Short exposure times, on the other hand, produce darker images. Generally, long exposure times are used while recording a starry night, while short exposure times, for racing competitions when there is enough background light.

In terms of communications, long exposure times could generate a critical effect. The following picture shows how severely the exposure time affects the "bands" received. From left to right, the exposure time is increasing. As you might notice, increasing the exposure time produces an increasing mix between connected bands, up to a point where no stripes are distinguished, and the lamps look uniformly illuminated.


So for this to work, it' s necessary to reduce to a minimum the exposure time achievable by the camera.


Time for examples!

Now it's time to explore some visual examples of real experiments. Highlight that these images are not synthetically generated but obtained from working experiments using a custom-designed lamp and a Logitech camera (model C920), as shown in the following picture.

Scenario Example scenario where a Logitech C920 is pointing to the custom lamp

With that setup, I have created a composition of real image captures of actual data transmission. Also, I have uploaded a video of my master's thesis defense, where I show how the system performs. The video is in Spanish, and I'm working on uploading English subtitles :D.

Examples Samples of the data sent by the custom lamp


Another example that I would like to show you is a ceil lamp installation that we, our team at IDeTIC's, deployed in our labs. The following image shows a picture of myself pointing a smartphone camera to the flat panel before installing it.

Laboratory Flat-panel modified to transmit data. The smarphone camera that I hold shows the received data bands.


...But, wait... why the lamp looks uniform in the picture?

Ohhh, yes, I forgot to mention... This picture was taken using a smartphone camera that has a relatively high exposure time. For that reason, despite the camera I'm holding shows the data bands, the lamp itself within the general picture does not.

This led me to introduce the last topic.

...How a person sees the lamp?

Optical Camera Communications systems share a common denominator with its parent technology, the Visible Light Communications: the use of conventional lamps as sources of illumination and communication at the same time. In this sense, the communication performance mustn't significantly affect the illumination provided by the lamp. In another way, the communication must remain hidden to our naked eyes.

Therefore the lamp should switch rapidly with little variances between light' states. Otherwise, we will perceive flickering in the illumination that can be painfully annoying.

In conclusion, the flat panel shown in the last picture is switching at high speed. To the naked eye, it looks uniform without variations, as it is shown in the image. However, the smartphone camera with a short exposure time can detect the fast changes that the lamp is undergoing.

Laboratory Flat-panel modified to transmit data. The smarphone camera that I hold shows the received data bands.

Here, there is a video showing in real-time the operation of the transmitter lamp.

For more video examples, check the "VLC for Industry 4.0" project in the projects category.