Strobed IR Illumination for
Image Quality Improvement in Surveillance Cameras
STEVE DARMADI
K T H R O Y A L I N S T I T U T E O F T E C H N O L O G Y
I N F O R M A T I O N A N D C O M M U N I C A T I O N T E C H N O L O G Y
DEGREE PROJECT IN EMBEDDED SYSTEMS SECOND LEVEL, 30 CREDITS
STROBED IR ILLUMINATION FOR IMAGE QUALITY IMPROVEMENT IN
SURVEILLANCE CAMERAS
THESIS REPORT by
STEVE DARMADI
SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE KTH ROYAL INSTITUTE OF TECHNOLOGY
-ii-
ABSTRACT
Infrared (IR) illumination is commonly found in a surveillance camera to improve night-time recording quality. However, the limited available power from Power over Ethernet (PoE) connection in network- enabled cameras restricts the possibilities of increasing image quality by allocating more power to the illumination system.
The thesis explored an alternative way to improve the image quality by using strobed IR illumination.
Different strobing methods will be discussed in relation to the rolling shutter timing commonly used in CMOS sensors. The method that benefits the evaluation scenario the most was implemented in a prototype which is based on a commercialized fixed-box camera from Axis. The prototype demonstrated how the synchronization of the sensor and the strobing illumination system can be achieved.
License plate recognition (LPR) in a dark highway was chosen as the evaluation scenario and an analysis on the car movements was done in a pursue of creating an indoor test. The indoor test provided a controlled environment while the outdoor test exposed the prototype to real-life conditions. The test results show that with strobed IR, the output image experienced brightness improvement and reduction in rolling shutter artifact, compared to constant IR. The theoretical calculation also proved that these improvement does not compromise the average power consumption and eye-safety level of the illumination system.
Keyword: surveillance camera, infrared, strobing, rolling shutter, license plate recognition
-iii-
SAMMANFATTNING
Infraröd (IR) belysning påträffas ofta i övervakningskameror för att förbättra bildkvalitén vid videoinspelning på natten. Den begränsade tillgängliga effekten från Power over Ethernet-anslutningen (PoE) i nätverksaktiverade kameror sätter dock en övre gräns för hur mycket effekt som kameran tillåts använda till belysningssystemet, och därmed hur pass mycket bildkvalitén kan ökas.
I detta examensarbete undersöktes ett alternativt sätt att förbättra bildkvalitén genom att använda blixtrande (eng: ”strobed”) IR-belysning. Olika strobe-metoder undersöktes i relation till rullande slutare, vilket är den slutar-metod som vanligtvis används i CMOS-sensorer. Den metod som gav mest fördelaktiga resultat vid utvärdering implementerades i en prototyp baserad på en kommersiell nätverkskamera av Fixed box-typ från Axis Communications. Denna prototyp visade framgångsrikt ett koncept för hur synkronisering av bildsensorn och belysningssystemet kan uppnås.
Registreringsskyltigenkänning (LPR) på en mörk motorväg valdes som utvärderingsscenario och en analys av bilens rörelser gjordes för att skapa en motsvarande testuppställning inomhus. Inomhustesterna gav en kontrollerad miljö medan testerna utomhus utsatte prototypen för verkliga förhållanden. Testresultaten visar att med strobed IR blev bilden från kameran både ljusare och uppvisade mindre artefakter till följd av rullande slutare, jämfört med konstant IR-belysning. Teoretiska beräkningar visade också att dessa förbättringar inte påverkar varken kamerans genomsnittliga effektförbrukning eller ögonsäkerheten för belysningssystemet negativt.
Nyckelord
övervakningskamera, infraröd, strobing, rullande slutare , registreringsskyltigenkänning
-iv-
ACKNOWLEDGMENTS
This thesis project was carried out in Axis Communication AB, so my thanks go to these gurus:
• Jenny Karlsson, for LPR advice and the outdoor test company,
• Andreas Karlsson, who helped us on PLD problems,
• Anders Svensson, who made sure that everything ran smoothly,
• Christian Adielsson, Johan Kjörnsberg, and Ola Synnergren, the triplet which guided every of our baby steps.
On the KTH side, my gratitude goes to Mark T. Smith who reassured us about the administration and academic point-of-view. I would also thank all my life support in Lund: Sofia Collberg, Fei Shenyang, Martin Nilsson, Karolis Poskus, Zhou Zhuo Hang, and the Indonesian communities in Skåne and Stockholm.
Now it comes the most important part. Carlos Tormo is the man whom I successfully persuaded to be part of the project. So, countless thanks go to him for his 24/7 passion in making the project both enjoyable and memorable.
-v-
ABBREVIATIONS
ADC Analog to Digital Converter CCD Charge-coupled Device
CMOS Complementary Metal–Oxide–Semiconductor EIT Extend Integration Time
FPS Frame per Second IP Internet Protocol
IR Infrared
I/O Input/output
LPR License Plate Recognition
LVDS Low-voltage Differential Signaling MCU Micro-controller Unit
PD Powered Device
PLD Programmable Logic Device PoE Power over Ethernet PWM Pulse-width Modulation RPM Rotation per Minute STS Shared-time Strobing WDR Wide Dynamic Range WFS Whole-frame Strobing
-vi-
TABLE OF CONTENT
Abstract ... ii
Sammanfattning ... iii
Acknowledgments ... iv
Abbreviations ... v
Table of Content ... vi
1 Introduction ... 1
1.1 Problem Statement ... 2
1.2 Goal... 2
1.3 Benefits, Ethics, and Sustainability ... 2
1.4 Methodology ... 2
1.5 Scope and Limitation ... 3
2 Strobing Methods ... 4
2.1 Whole-frame Strobing ... 5
2.2 Shared-time Strobing ... 7
2.3 Method comparison ... 8
3 License Plate Recognition ...10
4 Evaluation Method ...17
4.1 Chosen Strobing Method ...17
4.2 Indoor Test ...20
4.3 Outdoor Test ...21
5 System Implementation ...24
5.1 LED Driver ...24
5.2 Strobe Signal Generation ...24
5.2.1 Sync codes ...25
5.2.2 PLD and ARTPEC Modification ...27
5.3 Eye-safety ...30
6 Results and Analysis ...31
6.1 Indoor Test ...31
6.2 Outdoor Test ...32
6.3 Implementation Feasibilities ...36
7 Conclusion and Future Work ...37
7.1 Conclusion ...37
7.2 Future Work ...37
Bibliography ...38
Appendix ...39
-1-
1 INTRODUCTION
Many surveillance cameras are equipped with an adjustable IR-cut filter to improve usability in low-light condition. During daytime, this filter is enabled to avoid color misinterpretation due to sensor sensitivity outside the visible light range. When the scene gets darker than a certain level, the filter is disabled (usually by moving the filter glass off the sensor) to allow the sensor to collect more light from any wavelength within the sensor sensitivity. The infrared light, however, will distort the overall perceived image. Hence, the output image is converted to grayscale when the filter is removed.
However, in some darker conditions, this method is not good enough to produce an acceptable image.
Longer shutter times and higher sensor gains are usually set to get more details. In other cases where there is virtually no light source (or its reflection) in the scene, a near-infrared (IR-A) light emitter is added either built-in or as an external accessory.
Being a pioneer in IP cameras, Axis Communications AB (Axis) is a company which focuses on network- enabled products. The ability to have data and power connections through the same cable (i.e., PoE) is one of the main selling points for their current main business (i.e., video surveillance). The convenience of installation and cabling system, however, has an impact on the power that the camera can use. The most common PoE standard (class 3, type 1, 802.3af) can supply at most 12.95 W to the powered device, restricting the power consumption of different sub-systems in the camera, like the IR illumination.
As an example, one of Axis network cameras has a built-in IR emitter which draws 2A at 1.9V [1]. The power is delivered to the LED through two power stages (PoE 48V to 3.3V and LED driver) which create 10% losses in each stage according to an internal testing data. Therefore, using equation (1), it is logical to assume that the IR illumination costs almost 5W.
𝑃𝑎𝑐𝑡𝑢𝑎𝑙= 𝑃𝑖𝑑𝑒𝑎𝑙× 𝜇1× 𝜇2 (1)
Meanwhile, other component groups also consume large portions of available power, as listed in Table 1.
Axis overcomes this problem by carefully scheduling group operations to avoid unwanted bootup due to power failure. Given the strict power condition, increasing the IR emitter power (to get better image quality) without using additional strategy is unfavorable.
Table 1. Components Power Consumption in One of Axis’ Built-in IR Network Camera
Group Power (W) ratio to PD PoE (%)
PoE PD (class 3, type 1) (available power) 12.95 100
Heater 4.2 32
Iris adjustment 1.8 14
Lens zoom and focus 1 8
IR illumination 5 39
other functions (e.g., MCUs) 7.1 55
Due to the high power-demand, building a camera with built-in IR illumination would be challenging, especially when recording fast moving objects in a low-light environment. Hence, Axis is looking for ways to improve the image quality through strobing the IR light. Strobing is an illumination technique which consists of emitting flash of light in a cyclic manner. This method is expected to improve the level of luminous energy delivered to the object by only emitting IR when the sensor is acquiring the image (integrating). The non-integrating time will save energy and thus, enable designers to choose a more powerful IR emitter.
The reduction of exposure time in strobing can potentially reduce the hazard introduced to the human eye.
According to IEC-62471 (Photobiological Safety of Lamps and Lamp Systems), particularly its application on
-2-
repetitively pulsed devices [2], the system can be evaluated by averaging the pulsed emission. Therefore, image quality improvements through IR power increment can be less harmful using strobed IR.
1.1 Problem Statement
The main question brought from the background is:
Can the image quality of surveillance cameras be improved by utilizing strobed IR?
This high-level question can be broken down into several sub-questions as follows:
1. How to synchronize frame capture (sensor) with strobed IR light?
2. How does strobed IR performs compared to constant IR regarding image quality?
1.2 Goal
The research aims to build a prototype which can answer the questions mentioned in the problem statement.
Furthermore, it also aims to:
• Prove that IR illumination can be well synchronized with sensor timing.
• Present strobing method/s that can provide significant improvements in image quality.
• Evaluate image quality difference between constant light and proposed strobing methods in a related use case.
1.3 Benefits, Ethics, and Sustainability
The thesis presents an early verdict of strobed IR implementation for surveillance camera industries, especially from image quality perspective. Possibilities of reducing motion blur and improving object brightness in very challenging situations will bring more-detailed footages to the user. This means that the camera would provide more information and be more reliable in surveillance activities. On the other hand, the strobing function would not create new ethical questions, as it does not alter how the camera works as a surveillance device. Concerns about eye safety for people who are exposed to the camera’s IR radiation will be addressed in Section 5.3.
1.4 Methodology
The methodology used resembles a conceptual research method with five research stages, as shown in Figure 1. The idea of strobing IR light was built on top of an existing surveillance camera system. Hence, the first stage of the research is understanding how the system works through a literature study. The second stage is closely related with the first one. It consists of shutter timing analysis to produced new strobing methods that suit the rolling shutter. This stage also includes analysis on object movement in License Plate Recognition (LPR) which produces the speed transformation formulas that supports the development of the evaluation method.
The third stage is the evaluation method itself. It consists of two different tests: an indoor test, which simulates the license plate characters’ movement to check the prototype performance before the real LPR outdoor test. The fourth stage consists of prototyping activities, where a commercial product will be modified to produce the intended strobing function for both tests. In this stage, an eye-hazard calculation will also be done with the implemented specification. The prototype would be evaluated with the tests, and later the test results will be analyzed qualitatively and presented.
-3-
Figure 1. Research Methodology
1.5 Scope and Limitation
Strobing light itself can be beneficial for many applications including the well-known flashbulb in still-image photography, and time-of-flight based distance measurement[3]. In video surveillance application, light strobing opens possibilities for image quality and power efficiency improvement.
However, these two benefits could act as opposite perspectives. This thesis focuses on discovering image quality improvements while keeping the power consumption below or at least similar with constant IR.
Meanwhile, work on power efficiency perspective was conducted by another writer [4] during the same timeframe. Although the two works are closely related since the prototype implementation was based on the same platform, many differences can be found. One example is the usage of Whole-frame Strobing (WFS) technique in this work, is limited to the theoretical overview. Comparison between strobing techniques (Section 2.3) concluded that it gives no meaningful benefit on image quality. However, the use of WFS can be found in the other writer’s work as it is beneficial for the power efficiency improvement.
Other research limitations are explained in the following points:
• The prototype is built on a commercialized Axis fixed-box camera which will be addressed as the development platform for the rest of report. This model is using Programmable Logic Device (PLD) to bridge the sensor and main microcontroller. Hence, it will ease the strobing signal generation (access to modification, reliability, and complexity compared to implementing it in microcontroller). For test purposes, the development platform will be equipped with a zoom telephoto lens (f=12.5-50mm).
• Any development related to sensor timing will be based on one capture mode (1920x1080
@25FPS, readout rate = 60FPS). More details are discussed in Section 4.1.
• The scenario chosen to evaluate the prototype is License Plate Recognition (LPR). A Spanish EU license plate is used as a reference (due to its availability).
• The research will focus on image quality improvement, which will be evaluated qualitatively.
Therefore, analysing the LPR algorithm is outside the scope of the project.
Literature Study
Development platform, rolling shutter, LED driver Strobing Method and LPR Analysis
Two strobing methods and speed tranformation formulas Evaluation Method
Indoor and Outdoor test
Implementation (prototyping)
Strobing signal generation and eye-hazard calculation Test and result analysis
Image quality comparison between strobed and constant IR
-4-
2 STROBING METHODS
This part of the report will discuss possible strobing methods suitable for rolling shutter sensors. It should be noted that the analysis provided here is based on the simplified assumption that the recording scene is totally dark (no other source of light except the IR emitter).
Rolling shutter is a sensor timing commonly found in Complementary Metal Oxide Semiconductors (CMOS) sensors. Many modern imaging devices, including Axis cameras, utilize this sensor type due to its faster readout, lower heat, and reduced power consumption compared to Charge Coupled Devices (CCD) sensors [5]. However, CMOS sensors generally adopt rolling shutter, as opposed to global shutter, which is found in CCD. The use of this mechanism introduces the so-called rolling shutter artifact, shown in Figure 2.
The artifact is introduced as the pixels are integrated (collect or exposed to light) in a per-row basis.
Figure 2. Images with (left) and without (right) Rolling Shutter Artifact
note: both images are captured by the same camera, but the right picture is produced by using shared-time strobing to simulate a global shutter.
This behavior is unique to CMOS sensors, as shown in Figure 3. The type of sensor has several Analog to Digital Converters (ADC)s, each serving a column of pixels. The ADC array will digitalize one column at the same time and make the digital output accessible, before proceeding to the next line. This digitalization and output-providing process is called the readout (represented in blue boxes in Figure 4). The delay between readouts of different lines forces the lines to integrate light (collect photons) at a slightly different time (approx. 14.82uS at 60FPS-full HD, more details in Section 4.1) which eventually will create the rolling shutter artifact. This effect will be more pronounced if the captured objects are faster.
While rolling shutter creates a distinct artifact, it also introduces a limitation on how the IR light can be strobed in a synchronized manner. For a given integration time (𝑇𝑖), every line in the sensor will accumulate light for the same duration (e.g., 8 time-units in Figure 4). The time unit will be further called as H-unit, which also represents the smallest time difference that the sensor can work with, depending on the selected drive mode (discussed in Section 4.1).
Figure 3. CCD and CMOS Sensor Basic Schematic
-5-
Figure 4. Rolling Shutter Timing
In constant IR, the emitter will be always active, including the entire integration time. This means that, every line will receive the same amount of light from the emitter, regardless some artifacts which are caused by moving objects. Meanwhile, in strobed IR, the emitter will be turned on and off periodically. Thus, the next question is when the emitter should switch state.
If there was a disparity between the length of the strobe experienced by different lines, two different unwanted effects would happen. The first effect is intensity gradient, as explained in [6]. As shown in Figure 5, the upper lines receive longer strobe compared to the lower ones. As a result, the upper part of the picture will appear brighter. The lowest line will instead be completely dark because of the light absence. The second effect is motion blur variation. If a long strobe length is used to capture a fast moving object, part of the object that is exposed longer by the light will form a longer trail (blurrier).
Figure 5. Unsynchronized Strobing Light Timing
To avoid these effects, the strobe length (𝑇𝑂𝑁) should equally overlap with each line’s integration time. The next section will discuss two different methods to fulfill this requirement: whole-frame strobing (WFS), and shared-time strobing (STS).
2.1 Whole-frame Strobing
The idea of WFS is to turn off the emitter at times when none of the lines are integrating. Since during the integration time the emitter works the same way as constant IR, theoretically this method will not alter the image output. Energy saving will happen during the off cycle, and hence the average power consumption can be reduced. In Figure 6, each frame has 9H frame period (𝑇𝐹𝑃). The H unit is defined as the smallest time step which the sensor can differentiate (more about this are explained in section 4.1). However, the integration only happens at 4 ≤ 𝑡 ≤ 8, so afterwards, the IR light can be turned off until the integration of next frame occurs (𝑡 = 13).
The minimum strobe length for the given scenario in Figure 6 is:
𝑇𝑂𝑁_𝑚𝑖𝑛 = 𝑇𝑖+ (𝑁𝑣𝑎𝑙𝑖𝑑− 1) (2)
= 2 + (4 − 1)
4 Integration time
3 Shutter timing
2 Readout
1
1 2 3 4 5 6 7 8 9 10 11 12 13 shared time (Ts)
integration time line
number
t[H]
IR state
4 Integration time
3 Shutter timing
2 Readout
1
1 2 3 4 5 6 7 8 9 FRAME PERIOD
IR OFF IR ON IR OFF
line number
t[H]
-6-
= 5
Moreover, the average power consumption for WFS in this case is:
𝑃𝐴𝑉𝐺 =𝑇𝑂𝑁
𝑇𝐹𝑃 × 𝑃𝐿𝐸𝐷 (3)
=5 9 𝑃𝐿𝐸𝐷
With WFS activated, 𝑃𝐴𝑉𝐺 is almost half of the original consumption, 𝑃𝐿𝐸𝐷. However, this number will vary much as 𝑇𝑂𝑁 depends heavily on the frame period, the integration time, and the number of valid lines (𝑁𝑣𝑎𝑙𝑖𝑑). Valid lines are lines that contains image data, as opposed to blanking lines which exist just to justify the sensor timing. Discussion about the sensor data sequence will be covered in Section 4.1.
In another case, as shown in Figure 7, a higher 𝑁𝑣𝑎𝑙𝑖𝑑 will force the integration time to extend further towards the previous frame, leaving no time for off state and thus, the system becomes a constant IR again.
Similar condition happens when using a longer integration time. To keep the readout position consistent, the extension of integration time happens backwards (towards the previous frame). Therefore, the free time between adjacent frames is decreasing, so less off time would be available as seen in Figure 8 .
Figure 6. Whole-Frame Strobing
Figure 7. Whole-frame Strobing with More Lines
IR state
4 3 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Integration time Shutter timing Readout
FRAME 1 FRAME 2 FRAME 3
IR ON IR OFF IR ON IR OFF IR ON
IR OFF line number
t[H]
IR state IR OFF
8 7
6 Integration time
5 Shutter timing
4 Readout
3 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
FRAME 1 FRAME 2
IR ON line number
t[H]
-7-
Figure 8. Whole-frame Strobing with Longer Integration Time
2.2 Shared-time Strobing
Another method that can be used to fit the strobing pulse is the Shared-time Strobing (STS). This method relies on the fact that, in some conditions (e.g., Figure 9), the integration time is long enough to provide a common integration time for all lines (shared-time). Similar ideas also appear in earlier research such as [6]
and [7], but with different motivations.
At the positive side, STS will be significantly more efficient compared to WFS. The method will make sure that every line is utilizing the emitted light whenever the IR is on. Moreover, the strobed IR will create a virtual global shutter (removing the rolling shutter artifacts) as every line is considered being exposed only when the emitter is on. Nevertheless, the minimal requirement for integration time could be huge compared to the strobe length itself.
As an example, consider having an initial scene with constant IR as seen in Figure 10. Each frame is having 2H of integration time. To keep the brightness level, extension of integration time (EIT) must be done to provide equivalent shared time. The result will look similar to Figure 9. The minimum required integration time can be calculated using equation (4).
𝑇𝑖_𝑚𝑖𝑛 = 𝑇𝑂𝑁+ (𝑁𝑣𝑎𝑙𝑖𝑑− 1) (4)
= 2 + (4 − 1)
= 5
The formula also shows that the 𝑇𝑖_𝑚𝑖𝑛 is very dependent to the 𝑁𝑣𝑎𝑙𝑖𝑑. If a full-HD sensor with 1080 lines is used, the resulting 𝑇𝑖_𝑚𝑖𝑛 is 1082. For a large 𝑇𝑂𝑁, this method would be unfavorable, as 𝑇𝑖_𝑚𝑖𝑛 would be very large and tend to grow beyond the available frame period. Moreover, a too large 𝑇𝑂𝑁 would make the system act like the constant IR. No significant improvement can be expected from this technique, unless the conditions is restricted to short 𝑇𝑂𝑁.
IR state IR OFF IR OFF
4 3 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Integration time Shutter timing Readout
FRAME 1 FRAME 2 FRAME 3
IR ON IR ON IR ON
line number
t[H]
-8-
Figure 9. Shared-time Strobing
Figure 10. Example Case Requiring EIT for Shared-time Strobing
2.3 Method comparison
Notice that the formula in STS (4) is very similar to (2) in WFS, with the position of 𝑇𝑖_𝑚𝑖𝑛 and 𝑇𝑂𝑁 being swapped. The complete formulas after taking the upper limit (𝑇𝐹𝑃) into account can be found in (5) and (6) for WFS and STS respectively. The upper limit in (6) protects the system from dropping the frame rate, while in (5), the upper limit becomes a boundary between WFS and constant IR.
𝑇𝑖+ (𝑁𝑣𝑎𝑙𝑖𝑑− 1) ≤ 𝑇𝑂𝑁 ≤ 𝑇𝐹𝑃 (5) 𝑇𝑂𝑁+ (𝑁𝑣𝑎𝑙𝑖𝑑− 1) ≤ 𝑇𝑖 ≤ 𝑇𝐹𝑃 (6) Having these equations sorted out, the energy consumption ratio (compared to constant IR) of each strobing method can be analyzed. The general energy consumption ratio is:
Γ =𝑇𝑂𝑁
𝑇𝐹𝑃 (7)
For WFS, the largest energy reduction will happen when the shortest 𝑇𝑖 is used for a given 𝑁𝑣𝑎𝑙𝑖𝑑. Meanwhile, there is no reduction at all when WFS is used for tight timing scenario as in Figure 7 (𝑇𝑂𝑁= 𝑇𝐹𝑃). The energy reduction ratio range for WFS can be written as follows:
𝑇𝑖+ (𝑁𝑣𝑎𝑙𝑖𝑑− 1)
𝑇𝐹𝑃 ≤ Γ𝑊𝐹𝑆≤ 1 (8)
While for STS, the situation is more dynamic since 𝑇𝑂𝑁 is the first thing to set.
0 ≤ Γ𝑆𝑇𝑆≤𝑇𝑂𝑁
𝑇𝐹𝑃 (9)
IR state
4 3 2 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Integration time Shutter timing Readout
FRAME 1 FRAME 2 FRAME 3
IR ON IR OFF IR ON IR OFF IR ON IR OFF
IR OFF line number
t[H]
IR state
4 3 2 1
1 2 3 4 5 6 7 8 9
FRAME PERIOD IR ON line
t[H]
-9-
However, the limitation of STS is not on energy reduction but instead on the maximum achievable 𝑇𝑂𝑁. Condition (10) should be fulfilled to prevent frame rate drop, and substituting this into (4) will yield 𝑇𝑂𝑁_𝑚𝑎𝑥(11). With a long 𝑇𝑂𝑁, the requirement of 𝑇𝑖_𝑚𝑖𝑛 (equation (4)) will be enormous. The tendency of requiring large 𝑇𝑖 will force the sensor to collect more light and hence, making STS sensitive to motion blur if there is any ambient light source.
𝑇𝑖 ≤ 𝑇𝐹𝑃 (10)
𝑇𝑂𝑁_𝑚𝑎𝑥 = 𝑇𝐹𝑃− (𝑁𝑣𝑎𝑙𝑖𝑑− 1) (11)
Choosing between one of the previous methods is merely prioritizing between small 𝑇𝑖 (short integration time) or small 𝑇𝑂𝑁 (short strobing length). However, the priority will vary on the shooting condition.
Table 2 gives some examples of requirements in three different conditions.
Table 2. Suitable Method for Different Shooting Conditions Condition Requirements Suitable method
High-speed object;
low ambient light
𝑇𝑖 does not matter (if smaller than 𝑇𝐹𝑃) as ambient light is not significant compared to IR emitter brightness.
However, 𝑇𝑂𝑁 should be small.
STS.
As a small 𝑇𝑂𝑁 is mandatory, the LED average power can be greatly reduced. Alternatively, if the LED average power is kept the same, the peak power can be increased to improve scene brightness.
High-speed object;
medium ambient light
𝑇𝑖 should be small since ambient light is significant. 𝑇𝑂𝑁 should be long enough to cover the integration time.
WFS.
The IR average power reduction depends heavily on 𝑁𝑣𝑎𝑙𝑖𝑑 and 1H period.
High-speed object;
high ambient light
𝑇𝑖 should be small since ambient light is significant.
WFS or no IR at all.
If the shooting condition is very bright, there is a chance to get good image quality without any IR light.
-10-
3 LICENSE PLATE RECOGNITION
LPR is one of the video surveillance scenarios where the recorded objects (vehicles) are very dynamic. The system works by feeding an image or a sequence of images (e.g., video) to an algorithm, in which a series of image processing stages and optical character recognition are done. The scenario of LPR itself can vary depending on the surveillance objectives, such as toll or parking payment, traffic surveillance, and road-rule enforcement.
The quality of the input image plays an essential role in the success of LPR [8]. Some applications (i.e., traffic surveillance) require the system to work real-time in different conditions (outdoor, day to night). This means that the camera used should be designed to yield blur-free and reasonably bright images, regardless of the availability of ambient light.
This requirement fits perfectly on possible improvements that strobing can achieve. As discussed in Chapter 2, strobing works best when the required integration time is short. On the other hand, the efficiency boost that strobing provides might be utilized to improve the brightness of the license plate, and/or to reduce the motion blur. LPR with strobing becomes even more relevant when designing a built-in IR camera which has a tight power limit.
After choosing LPR as one of the evaluation methods, we need to specify a scenario. Night-time recording of cars, which travels in a dark highway, is one of the hardest scenarios in LPR (also in general photography).
It requires a fast shutter speed, typically between 500 to 2000uS, according to the common practice in Axis.
This is done to freeze the object while keeping acceptable brightness and avoiding excessive noise. By using this extreme case, we could maximize the image quality improvement that strobed IR can offer, exactly where image brightness improvement is restricted in constant IR (due to PoE power limitation).
However, evaluating the performance regarding motion blur for a camera is not straightforward. To extract valid conclusions, a controlled environment is needed, in which the quality of images can be analyzed for different camera configurations in the same condition.
An outdoor test with real vehicles is an option that can be useful, but there are a few external factors that can pose problems and cause difficulties when analyzing specific aspects of the camera performance. For example, some cars’ headlights will be stronger than others, and some plates will be more reflective than others, which will yield different license plate image quality for the same camera settings. Thus, a comparison of two different camera configurations, in these two different situations, might not be fair.
We need to consider that, while it is possible to have different cameras shooting at the same scene with constant IR, it is not when using strobed IR. This is because the cameras must be synchronized with the light emitter, which at this point is only possible with one camera.
Meanwhile, a controlled scenario in a laboratory allows testing different camera configurations in the same conditions. Unfortunately, LPR testing in a laboratory means moving a license plate (or similar object) at high speeds towards the camera, which is a hard task considering the limited dimensions of a laboratory.
For example, to capture the object in 100km/h in a 30m room, the object should accelerate from 0 to 100km/h in 2.16s. Therefore, we need to find a way to simulate the object movement, without moving the object towards the camera.
Since the video is recorded in the sensor plane (2-dimensional), movements in the optical axis (3rd dimension) of the camera will be translated into movements in the sensor plane. The direction of both movements is better described in Figure 11. If we mathematically translate movement in the optical axis into an equivalent movement in the sensor plane, the test setup can be greatly simplified.
In Figure 12, the proposed setup is shown. This setup consists of a reflective spinner with similar letters found in EU license plates. The spinner will be spin using a stepper motor at a certain speed. In a particular angular speed, each character will move in various linear speeds, depending on its position relative to the shaft. The positions of the characters are as listed in Table 3.
-11-
Figure 11. Direction of Object Movement (left) and Car Velocity Components (right)
The next step is to correlate the car speed in a real case with the characters’ speed in the spinner. At first, we need to understand what causes the movement of the projected image (at the sensor). The object movements can be broken down into two different types of movement, shown in Figure 11. The first type is the movement in sensor plane (X and Y direction), while the second type is in the optical axis (Z direction).
In the LPR situation shown in Figure 11, the car movement consists of both movement types. The spinner, on the other hand, is considered to create movements only in the sensor plane (assuming the spinner is aligned with the sensor plane). Since we can only do the comparison in the sensor plane domain, the optical axis movement of the car (𝑉𝑐𝑎𝑟_𝑧) must be transformed first.
Figure 12. Spinner Bar
Table 3. Character Position and Speed in The Spinner Characters
from center distance from center [cm]
First 5
Second 15
Third 25
Fourth 35
Fifth 45
Sixth 55
Both types of movements are affected by the optical properties, such as the relation between magnification, distance, and height. The relationship is described in (12) and can be seen in Figure 13.
𝑀 = 𝑑𝑖 𝑑𝑜 = ℎ𝑖
ℎ𝑜
(12)
The spinner moves in both X and Y directions (∆ho), thus creates a shift in the projected image (∆hi) as shown in Figure 14. Meanwhile, movement due to 𝑉𝑐𝑎𝑟_𝑧 can be perceived as a change in magnification.
With a constant focal length (di = distance between sensor plane and point of convergence), a change in magnification would modify the projected image height as seen in Figure 15..
-12-
Figure 13. The Optical Properties
Figure 14. Object Movement in X or Y Direction (sensor plane)
Figure 15. Object Movement in Z Direction (optical axis)
As explained earlier, we would like to relate the speed of the spinner (XY) with the car speed (XYZ). To generate the similar level of motion blur, both car and spinner should generate an equivalent movement in their projected image. First, we will calculate the equivalent spinner movement (XY) of the car Z component in (13).
∆ℎ𝑖_𝑧= ∆ℎ𝑖_𝑥𝑦 (13)
ℎ𝑖2_𝑧− ℎ𝑖1_𝑧 = ℎ𝑖2_𝑥𝑦− ℎ𝑖1_𝑥𝑦 (14)
We will derive the equation one by one for each side to improve readability (without writing the extra subscript to differentiate direction). We start with the Z movement (left-hand side in (14)). Using (12), all ℎ𝑖 in (14) can be substituted with its ℎ𝑜counterparts to yield (15) and (16).
-13-
(𝑀2∙ ℎ𝑜2) − (𝑀1∙ ℎ𝑜1) (15)
𝑑𝑖2
𝑑𝑜2∙ ℎ𝑜2−𝑑𝑖1
𝑑𝑜1∙ ℎ𝑜1 (16)
Furthermore, ℎ𝑜2 and ℎ𝑜1 can be reduced to ℎ𝑜 since we will consider car speed in Y direction later. Also, 𝑑𝑖2 and 𝑑𝑖1are 𝑑𝑖 as the focal length is constant, so finally (16) can be reduced to (17).
( 1 𝑑𝑜2− 1
𝑑𝑜1) ∙ ℎ𝑜𝑑𝑖 (17)
Merging the fractions will yield (18) and (19).
(𝑑𝑜1− 𝑑𝑜2
𝑑𝑜1𝑑𝑜2 ) ∙ ℎ𝑜𝑑𝑖 (18)
( ∆𝑍
𝑑𝑜1𝑑𝑜2) ∙ ℎ𝑜𝑑𝑖 (19)
At last, we substitute ∆𝑍 with the measurable variables in (20). 𝑉𝑐𝑎𝑟_𝑧 is the car speed in Z direction, while 𝑇𝑂𝑁 is the strobe length. 𝑇𝑂𝑁 can also be the shutter speed if constant IR is used. The final equation for Z direction is shown in 21.
∆𝑍 = 𝑉𝑧∙ 𝑇𝑂𝑁 (20)
(𝑉𝑐𝑎𝑟_𝑧 ∙ 𝑇𝑂𝑁
𝑑𝑜1𝑑𝑜2 ) ∙ ℎ𝑜𝑑𝑖 (21)
Now we are finished with the Z direction (car movement). For the XY direction (spinner movement), the derivation is much more straightforward as the magnification does not change due to a constant object distance (𝑑𝑜1 = 𝑑𝑜2). The final equation for the spinner movement is shown in (22).
(𝑑𝑖
𝑑𝑜) ∙ (𝑉𝑋𝑌. 𝑇𝑂𝑁) (22)
Before we put back (21) and (22) side by side as in (14), the directional subscripts are added to avoid confusion between different directions. Assuming focal length for both directions are the same (𝑑𝑖_𝑍= 𝑑𝑖_𝑋𝑌), (23) can be reduced to (24).
(𝑉𝑐𝑎𝑟_𝑧 ∙ 𝑇𝑂𝑁
𝑑𝑜1_𝑍. 𝑑𝑜2_𝑍) ∙ ℎ𝑜𝑍∙ 𝑑𝑖_𝑍 = (𝑑𝑖_𝑋𝑌
𝑑𝑜_𝑋𝑌) ∙ (𝑉𝑋𝑌. 𝑇𝑂𝑁) (23) (𝑉𝑐𝑎𝑟_𝑧 ∙ 𝑇𝑂𝑁
𝑑𝑜1_𝑍. 𝑑𝑜2_𝑍) ∙ ℎ𝑜𝑍 = ( 1
𝑑𝑜_𝑋𝑌) ∙ (𝑉𝑋𝑌. 𝑇𝑂𝑁) (24)
Knowing that ∆𝑍 is simply 𝑑𝑜1_𝑍− 𝑑𝑜2_𝑍, we can finally derive (24) into two useful equations, (25) and (26), which correlate car movement in the Z direction (𝑉𝑧) with spinner movement in XY direction (𝑉𝑋𝑌).
Furthermore, 𝑉𝑋𝑌 can represent 𝑉𝑍_𝑥 or 𝑉𝑍_𝑦 depending on the height of the object ℎ𝑜_𝑍 given to the equation (whether it is the width or the height of the license plate).
-14- 𝑉𝑋𝑌= ℎ𝑜_𝑍∙ 𝑑𝑜_𝑋𝑌
𝑑𝑜1_𝑍(𝑉𝑐𝑎𝑟_𝑧 . 𝑇𝑂𝑁− 𝑑𝑜1_𝑍)∙ 𝑉𝑐𝑎𝑟_𝑧 (25) 𝑉𝑐𝑎𝑟_𝑧 = 𝑑𝑜1_𝑍2 . 𝑉𝑋𝑌
𝑑𝑜_𝑋𝑌 . ℎ𝑜_𝑍+ 𝑇𝑂𝑁 . 𝑑𝑜1_𝑍. 𝑉𝑋𝑌
(26)
Notice that the analysis is not finished yet since car movement in Y direction should also be considered.
Figure 16 explains where should the 𝑉𝑐𝑎𝑟_𝑦 be included to the conversion.
Figure 16. Vehicle to Equivalent Spinner Speed Conversion
To review the significance of object movement in each direction to motion blur creation, we will do an example case for the conversion theory. Like Figure 11, the camera is mounted on a bridge, looking down to a highway. The car is assumed to move straight, so there is no X component involved in the movement.
The license plate dimensions are taken from EU standards.
Table 4. Parameters of an Example Case
Variable Abbreviation Value
Car speed 𝑉𝑐𝑎𝑟 33.33 m/s (120 km/h)
License plate height ℎ𝑜_𝑍𝑦 0.114 m
License plate width ℎ𝑜_𝑍𝑥 0.52 m
Camera mounting height ℎ𝑐𝑎𝑚 5 m
Camera to license plate
distance 𝑑𝑜1_𝑍 14.62 m
Camera to Spinner distance 𝑑𝑜_𝑋= 𝑑𝑜_𝑦 20 m
Viewing angle α 70 deg
Strobe length 𝑇𝑂𝑁 10-3 s
For the given values in Table 4, the calculations could be done as follows:
1. Calculate the z and y component of the car speed.
𝑉𝑐𝑎𝑟_𝑧= 𝑉𝑐𝑎𝑟∙ 𝑠𝑖𝑛𝛼 = 31.32 𝑚/𝑠 𝑉𝑐𝑎𝑟_𝑦 = 𝑉𝑐𝑎𝑟∙ 𝑐𝑜𝑠𝛼 = 11.39 𝑚/𝑠 2. Calculate 𝑉𝑋 from Z movement.
𝑉𝑋 = ℎ𝑜_𝑍𝑥∙ 𝑑𝑜_𝑋
𝑑𝑜1_𝑍(𝑑𝑜1_𝑍− 𝑉𝑐𝑎𝑟𝑧∙𝑇𝑂𝑁)∙ 𝑉𝑐𝑎𝑟𝑧
-15- = 1.52 m/s
3. Calculate 𝑉𝑌 from Z movement.
𝑉𝑌= ℎ𝑜_𝑍𝑦 ∙ 𝑑𝑜_𝑌
𝑑𝑜1_𝑍(𝑑𝑜1_𝑍− 𝑉𝑐𝑎𝑟𝑧∙𝑇𝑂𝑁)∙ 𝑉𝑐𝑎𝑟𝑧 = 0.334 m/s
4. Calculate total 𝑉𝑋 and 𝑉𝑌 .
𝑇𝑜𝑡𝑎𝑙𝑉𝑌 = 𝑉𝑌+ 𝑉𝑐𝑎𝑟_𝑦 = 11.72𝑚
𝑠 = 42.2 𝑘𝑝ℎ 𝑇𝑜𝑡𝑎𝑙𝑉𝑋= 𝑉𝑋 = 1.52𝑚
𝑠 = 5.47 𝑘𝑝ℎ
From this example, it can be concluded that ‘the movement of the car in the optical axis’ (𝑉𝑐𝑎𝑟_𝑧) does not contribute significantly to the movement projected in the sensor plane (𝑉𝑋 and 𝑉𝑌). Figure 17 shows that even with a bigger viewing angle (which increases 𝑉𝑐𝑎𝑟_𝑧), 𝑉𝑌 ratio to 𝑉𝑐𝑎𝑟_𝑦 is still very small (lower than 0.1). In other words, the blur generated by the optical axis movement of the car, is negligible when compared to the blur generated by the sensor plane movement.
Figure 17. Contribution of Optical Axis Movement in Producing Motion Blur
On the other hand, the speed conversion shows that the dominant movement comes from the car movement in Y axis (sensor plane). The viewing angle affects 𝑉𝑐𝑎𝑟_𝑦 heavily, and so does the 𝑇𝑜𝑡𝑎𝑙𝑉𝑌, as shown in Figure 18. Thus, using a high viewing angle in LPR will help reducing the blur caused by the movements in Y axis.
Figure 18. 𝑇𝑜𝑡𝑎𝑙𝑉𝑌 relation to Viewing Angle
-16-
However, it is important to note that this speed conversion utilizes over-simplification of the optical system and geometry. These are the assumptions that have been made:
• The lens is an ideal thin lens with specific properties, e.g., has no barrel distortion and known point of convergence.
• The angled view of the real case also affects how the LPR algorithm performs to some degree, which cannot be reproduced by the spinner (entirely perpendicular to the camera).
• The equations assume that the optical axis of the camera intersects the license plate at its center, yielding minimum motion blur.
Also, we need to point out that only motion blur is being evaluated, while resolution and LED power are not considered yet. From the previous equations, we could wrongly conclude that higher viewing angles would yield better image quality. While this is true in terms of motion blur, the resolution of the camera will limit the camera’s angle (since a greater angle means longer distance to the license plate). If the resolution is not the limiting factor (e.g., a proper zoom lens is used), then the LED power is the only remaining variable to consider (even more critical when zoom lens are used).
Accuracy test of the equations is somewhat challenging to do and out of the project bound. Therefore, the proposed method serves as a rough approximation which might ease LPR related product development. It also helps to understand what factors are contributing to the motion blur.
-17-
4 EVALUATION METHOD
As the flow of the research has been briefly explained in Section 1.4, this chapter will focus on the evaluation method. In general, there are two kinds of evaluation that will be done to answer the research question:
• An indoor test with the proposed method discussed in the LPR chapter will be done in order to estimate if the prototype is good enough for an outdoor test, which is more difficult to do and more resource consuming.
• An outdoor test which resembles a real-life situation of the LPR application.
The section of each test will discuss in detail the motivation of the test as well as the test setup and test cases. However, it is crucial to specify which strobing method is appropriate for image quality improvements in LPR before determining the test cases for each test.
4.1 Chosen Strobing Method
In this section, we will analyze the timing properties of the sensor used in the development platform. The sensor output in this platform is sent through a low-voltage differential signaling (LVDS) communication protocol and the output data sequence follows the pixel arrangement shown in Figure 19.
The sensor has recording pixels (gradient color) in the center, and extra surrounding pixels for color processing and optical blanking for calibration (white and blue respectively). Meanwhile, the rest of the diagram represents the additional data which is also output to the data sequence (does not represent physical pixels). The device which receives the sensor output (i.e., the PLD in the camera) will go through every line and use the same timing (by detecting sync codes) to move between lines.
The white and colorful lines are categorized as valid lines, while the grey ones as blanking lines. The number of valid lines (𝑁𝑣𝑎𝑙𝑖𝑑) is fixed. The number of blanking lines (𝑁𝑏𝑙𝑎𝑛𝑘𝑖𝑛𝑔), however, can be adjusted to fit a certain timing requirement.
Figure 19. Sensor Pixel Arrangement
In Table 5, several timing combinations that can be achieved by the sensor is listed. Notice that there are two different terms for rate. The first one, readout rate, is the speed of data transfer. The higher the rate, the faster the sensor can move between different lines, and thus it determines the smallest applicable time step (1H period).
