Security has always been a problem for developing countries like Pakistan and street crimes are one of the key factors in security-related challenges. Law enforcement agencies have already been in search of different technological solutions to detect street crimes and other anomalies on immediate notice for prompt action. Therefore, developing an intelligent surveillance system for video anomaly detection is a pressing need. Surveillance systems have become increasingly popular in the globalization process. However, even after the deployment of huge CCTV infrastructures, the full involvement of human operators in traditional surveillance systems has led to shortcomings. For instance, high labor cost, limited capability for multiple-screen monitoring, inconsistency during long-durations, etc. are the key issues. We aim to develop an intelligent surveillance system “IntelliSurv”, for the detection of anomalies in real-time videos. We intend to target street crimes considering them as one of the key anomalies in the CCTV videos. IntelliSurv system will utilize deep neural networks to perform deep video analysis for the detection and tracking of humans from the live videos. Human action detection algorithms will be developed to identify abnormal actions in the videos. The intended algorithms will be deployed on state-of-the-art edge devices to pitch a complete black-box solution for real-time anomaly detection in a realistic environment. As a result, emergency response units can be more effective and efficient to deal with the anomaly. Deployment of IntelliSurv in streets, roads, and campus environments will not only help in the prevention of crimes but will also aid law enforcement agencies to take prompt actions against anomalous events.