Automatic Detection of Mouse Mating Behavior by Video Surveillance
Lo, Neng-Wen; Chang, Hsuan-Ting; Wang, Ming-Tsung
Proceedings : APSIPA ASC 2009 : Asia-Pacific Signal and Information Processing Association, 2009 Annual Summit and Conference: 668-671
APSIPA ASC 2009: Asia-Pacific Signal and Information Processing Association, 2009 Annual Summit and Conference. 4-7 October 2009. Sapporo, Japan. Poster session: 3D Mapping, Modeling, Tracking, and Applications (6 October 2009).
Instructions for use
Hokkaido University Collection of Scholarly and Academic Papers : HUSCAP
Automatic Detection of Mouse Mating Behavior by Video Surveillance *
Neng-Wen Lo*, Hsuan-Ting Chang†, and Ming-Tsung Wang†
Tunghai University, 181 Taichung Harbor Road, Section 3, Taichung 40704, Taiwan E-mail: [email protected]
Tel: +886-4-2359-0121 #37115 † National Yunlin University of Science and Technology, 123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan E-mail: [email protected]
Tel: +886-5-534-2601 #4263 E-mail: [email protected]
Tel: +886-5-534-2601 #4299 Abstract— The purpose of this study was to detect the behavior of mouse mating using video surveillance. To the best of our knowledge, this is the first study on developing the algorithm for the automatic detection of mouse mating based on video surveillance. First, the mouse movement in a cage was detected by the background subtraction method. When two mice contacted, the corresponding contour can be extracted by edge detection method, by which the distances between the centroid and the edge pixels of contour were determined as a waveform signal. After further sub-sampling and smoothing the waveform, two decision criteria emerged: (1) the total number n of peaks and troughs and (2) the angle θ between two lines connecting the adjacent peak-edges and the centroid of the contour. By using these two criteria, we can successfully detect the mating behavior in the recorded video. Experimental results showed that 87.0% of precision and recall accuracy can be achieved.
constructed from the previous images. The current image is subtracted by the background image. Then the difference is considered as moving object. This method is simpler and thus the computation load is low. However, it is easily influenced by noise. For general surveillance systems, the static background subtraction method is more efficient than the temporal difference method. There are several kinds of background construction methods proposed at present. For example, authors in  utilized the block-based background extraction (BBE) method to construct background image. On the other hand, authors in  uses the method of statistics to construct the background image and taking the method of doublebackground to update and reconstruct the background image. A background registration technique is used to construct a reliable background image from the accumulated frame difference information . Tracking techniques, for example, the model-based tracking of experimental animals, are always an important and hot research topic . Tracking laboratory animals can be achieved based on optical flow and active contours . The algorithm was applied to detect lion face based on Haar-like features and AdaBoost classifiers, and the tracking was implemented by the Kanade–Lucas–Tomasi method . This paper is organized as follows: Section 2 describes the methods of features extraction and Section 3 describes analysis of mouse mating. Experiment results and conclusions are presented in Sections 4 and 5.
Objective measurement of physical activities and behaviors of experimental animals is an important issue in biological research. However, it would inevitably consume a great deal of time and manpower to study animal activities through visual observation. It would be very helpful if the video surveillance techniques can be used to automatically detect unusual animal activities or behaviors [1-3]. In this paper, the proposition of automatic detection of mouse mating behavior was investigated. By placing a camera on top of a transparent container housing two mice (one male and one female), we recorded video sequences of their physical behaviors and used them for further analyses in pattern recognitions. The mice in action were regarded as targeted objects. When two objects moved close to each other, the proposed algorithm would be launched to determine if the mating actually occurred. Many efficient methods used to detect moving object and to analyze its behavior have been proposed. Each part of the whole framework has also been investigated extensively. Practical applications include traffic flow monitoring, pedestrian monitoring, and animal behavior detection, etc. Methods for moving object detection basically can be divided into two categories: (1) temporally differencing : the current image is subtracted by the previous image in the time domain. This method is not easily influenced by light variations. But it is difficult to obtain complete contour and features of the moving object; (2) static background subtraction : this method uses the background image
Background Image Construction In the BBE method for background image reconstruction, a frame is divided into a certain number of n × n blocks and then the block variance is calculated. If the block variance is less than a threshold value, the block is regarded as the background block. On the other hand, if the variance is greater than the threshold value, the threshold value is increased in the following frames to build left background blocks until the background images are completed.
After using the background subtraction (BS) method to subtract background from foreground, some large noise regions still exist as shown in Fig. 5(a). We labeled each object by using connected component analysis method, and then deleted the unreasonable area according to the area size. Fig. 5(b) shows that the real moving objects finally can be obtained.
Fig. 1: Block diagram of pre-processing.
In this paper, the YCbCr color space was adopted to analyze the video frames. First, both the background and foreground images shown in Figs. 2(a) and 2(b) were translated to YCbCr color space by using (1). Figs. 2(c) and 2(d) show the corresponding Cb components of images. Figs. 3(a) and 3(b) show the binarized Cb components of images in Figs. 2(c) and 2(d), respectively. Finally, Fig. 4 shows the noise-removed images. ⎡ Y ⎤ ⎡ 16 ⎤ ⎡ 65.481 128.553 24.966 ⎤ ⎡ R ⎤ ⎢Cb ⎥ = ⎢128⎥ + ⎢− 37.797 − 74.203 112.000 ⎥ ⎢G ⎥. ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎣⎢Cr ⎦⎥ ⎣⎢128⎦⎥ ⎣⎢ 112.000 − 93.786 − 18.214 ⎦⎥ ⎣⎢ B ⎦⎥
(a) (b) Fig. 5: (a) Background subtraction. (b) Connected component analysis.
Edge Detection According to the number of detected moving objects, further processing was utilized. If the number of objects was two, the two mice were considered separated and not in a mating status. If the object number was one, two mice joining together could be assumed in a mating status. Therefore, Sobel edge detection was applied on the detected single object. Figs. 6(a), 6(b), and 6(c) show the original video frame, the detected single object, and the edge detection results, respectively.
(a) (b) (c) Fig. 6: (a) Original image. (b) After pre-processing. (c) Sobel edge detection result of (b).
Contour Feature Extraction To distinguish the two mice in a mating status from the detected single object, the object’s contour requires further calculation for determination . The two-dimensional contour was transformed into one-dimensional waveform signal. The detailed procedures are given as follows:
(c) (d) Fig. 2: (a) Background image in the RGB color space. (b) Foreground image in the RGB color space. (c) The Cb component of the background image. (d) The Cb component of the foreground image.
1. Calculation of centroid As shown in Fig. 7(a), the centroid coordinate (x c , y c ) of a single object contour is calculated as xc =
(a) (b) Fig. 3: (a) Binarized image of Fig. 2(c). (b) Binarized image of Fig. 2(d).
∑x ,y i =1
∑y j =1
where (x i , y i ) denotes the coordinate of boundary pixels and N b denotes the number of boundary pixels. 2. Evaluation of distances Calculate the distance d between the centroid and each boundary point (x i , y j ) in the clockwise direction: di, j =
(a) (b) Fig. 4: (a) Morphological erosion result of Fig. 3(a). (b) Morphological erosion result of Fig. 3(b).
(xi − xc )2 + (y j − yc )2 ,
Note that d i,j can be expressed as a one-dimension waveform
signal, d(k) = d i,j . The first point locates at the north of centroid.
(a) (b) Fig. 9: (a) Original image under mating status. (b) Waveform signal under mating status. (a) (b) Fig. 7: (a) The boundary contour is unwrapped as a distance function with respective to the centroid. (b) The one-dimensional waveform signal representing the distance between the centroid and the edge points in (a).
3. Waveform smoothing As shown in Fig. 7(b), there were some fluctuations in the waveform signal. Actually, only the global features were significant for mating behavior detection. Therefore, these small fluctuations should be removed. To increase the processing speed, the waveform was sub-sampled such that the contour points were reduced in half only. Fig. 8(a) shows the sub-sampled waveform. Next, an average function was used to filter out the waveform fluctuation and the result is shown in Fig. 8(b). The average function is shown below: dˆ (1) = d (1) dˆ (k ) =
∑ d (i ) / 3, i = 2
(a) (b) Fig. 10: (a) Original image of the mice are not in a mating status. (b) Waveform signal obtained from (a).
If the number of peaks and troughs was four, the second criterion would be used for further confirmation. The angle θ between two lines connecting the adjacent peak and the centroid of the contour was measured. Figs. 11(a) and 11(b) show the selected peak points and the corresponding locations in the contour. According to our experiments, the threshold value for the angle θ was set to be 150° under the mated status.
k = i −1
dˆ (k ) =
∑ d (i ) / 5, i ≥ 3.
denotes the smoothed waveform.
(a) (b) Fig. 11: (a) Two peak points in the waveform. (b) The angle θ between the two peak points and the centroid. (a) (b) Fig. 8: (a) Sub-sampled result of waveform. (b) Smoothed result of waveform.
TEMPORAL MOTION ANALYSIS
Since the mating behavior is an active motion, it is necessary to consider the temporal movement information in the video. Even though that the two criteria shown in the above section are satisfactory. Two mice could not be in a mating status. Therefore, the temporal information in the consecutive frames which satisfies the two selection criteria would be subjected to final analyses. The procedures are summarized as follows:
Waveform peaks and troughs After waveform sub-sampling and smoothing stages, two decision criteria were used to determine the mating behavior. Consider the image shown in Fig. 9(a). The corresponding waveform is shown in Fig. 9(b). The total number of peaks and troughs on common mating status is four. On the other hand, the total number of peaks and troughs is more than four when the two mice are not in a mating status. Figs. 10(a) and 10(b) show an example of this case. The detected contour of two mice in contact is more irregular than that in mating. The number of detected peaks and troughs were six.
Auto-selection of the region of interest (ROI) in images As shown in Fig. 12, according to the centroid of the detected object, the area is selected for further processing.
In this paper, we successfully applied the video-based surveillance system on the study of automatic mating behavior detection. The monitored mice were treated as moving objects in the image frames. When two mice moved in contact, the detected contour was analyzed to determine whether a mating behavior occurred. The experimental results show that the proposed method can efficiently detect the mating behavior in various recorded videos. Our future work will focus on the algorithm development for real-time applications. Other specific behavioral patterns of animals will be considered as well.
Fig. 12: Dividing a square area of size 400 x 400 based on centroid.
Calculate mean square error (MSE) values of ROIs The MES values of previous and current ROI image are determined using (5). MSEi =
1 M −1 N −1 ∑∑ [ f i (x, y ) − f i−1 (x, y )]2 , i = 2,3,4,.... (5) M × N x =0 y =0
Calculate average MSE value The calculation of the averaging MSE value is given in (6). MSEk + MSEk +1 + ... + MSEk +n (6) MSE = . n +1
ACKNOWLEDGMENT The authors would like to thank National Science Council of the Republic of China, Taiwan, for financially supporting this research under Contract No. NSC 97-2221-E225-057.
A large MSE value corresponds to obvious movements, which describes the mating behavior. On the other hand, when the mice activity is low, which is evident by the averaged MSE value MSE less than a threshold value, the two mice will not be considered in a mating status.
Y. Nie, et al, “High-speed video analysis of laboratory rats behaviors in forced swim test,” IEEE International Conference on Automation Science and Engineering, pp. 206-211, August, 2008.  H. Ishii, M. Ogura, S. Kurisu, A. Komura, and A. Takanishi, “Development of autonomous experimental setup for behavior analysis of rats,” IEEE International Conference on Intelligent Robots and Systems, pp. 4152-4157, October, 2007.  X. Xinwei and T. C. Henderson, “Video-based animal behavior analysis from multiple cameras,” IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 335-340, September, 2006.  A. J. Lipton, H. Fujiyoshi, and R. S. Patil, “Moving target classification and tracking from real-time video,” Fourth IEEE Workshop on Applications of Computer Vision, pp. 8-14, October, 1998.  D. Gao and J. Zhou, “Adaptive background estimation for real-time traffic monitoring,” IEEE Transactions on Intelligent Transportation Systems, pp. 330-333, August, 2001.  C. C. Hsu, H. T. Chang, and T. C. Chang, “Efficient moving object extraction in compressed low-bit-rate video,” IEEE International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pp. 411-414, December, 2006.  W. Pan, K. Wu, and Z. S. You, “A background reconstruction method based on double-background,” Fourth International Conference on Image and Graphics, pp. 502-507, August, 2007.  S. Y. Chien, S. Y. Ma, and L. G. Chen, “Efficient moving object segmentation algorithm using background registration technique,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 12, pp. 577-586, July, 2002.  Z. Kalafatic, “Model-based tracking of laboratory animals,” The IEEE Region 8 Conference on Computer as a Tool, vol. 2, pp. 175-178, September, 2003.  Z. Kalafatic, S. Ribaric, and V. Stanisavljevic, “A system for tracking laboratory animals based on optical flow and active contours,” International Conference on Image Analysis and Processing, pp. 334339, September, 2001.  T. Burghardt and J. Calic, “Analysing animal behaviour in wildlife videos using face detection and tracking,” IEE Proceedings on Vision, Image and Signal Processing, vol. 153, pp. 305-312, June, 2006.  H. Fujiyoshi and A. J. Lipton, “Real-time human motion analysis by image skeletonization,” Fourth IEEE Workshop on Applications of Computer Vision, pp. 15-21, October, 1998.
IV. EXPERIMENT RESULTS The set up of our experiment is shown below: (1) Camera type: Bravio VIP 800 IP-Camera (2) Frame rate: 3 frames per second, Frame size: 640 x 480 (3) Tested target: ICR mouse After examining 51 recorded video clips, we determined that the total number n of peaks and troughs would be four and the angle θ would be greater than 150 degrees for each video frame tested. Among the tested 51 video clips, 23 clips exhibited mating activities. Table I shows the results of our experiments. The calculation ways of precision and recall are given in (7) and (8). TABLE I EXPERIMENT RESULTS Total number of mating behaviors Total number of detected mating behaviors Number of the clips of miss detection Number of the clips of error detection Number of the clips of correct detection
23 23 3 3 20