Nowadays, the rapid increase in the number of the automobiles on the highway and urban roads have created many challenges regarding the proper management and control of the traffic. Detection and tracking of vehicles using the traffic surveillance system gives more promising way to manage and control the road traffic. Vehicle surveillance represents a challenging task of moving object segmentation in complex environment. The detection ratio of such algorithms depends upon the quality of the generated foreground mask. Therefore, the aim of this paper is to present an efficient method for detection and tracking of vehicles which focuses on the trajectory of motion of the objects. The proposed method preserves the group of pixels in foreground which can be probable vehicles and discards the rest as noise. Therefore, it selectively rejects the objects which cannot be vehicles at the same time consolidate the candidate vehicles. Here, the foreground mask generation process is improved so that the quality of generated foreground mask better consequently increases the detection ratio. The performance of the proposed method is evaluated by comparing it with other standard methods qualitatively as well as quantitatively.