PKU-Vidar-DVS dataset is a large-scale multimodal neuromorphic object detection dataset with temporally continuous labels. This dataset is recorded using our hybrid camera system, which includes a Vidar (resolution 400*250) and a DAVIS346. This dataset contains 9 indoor and outdoor challenging scenarios by considering velocity distribution, illumination change, category diversity, and object scale, etc. We use the hybrid camera system to record 490 sequences including Vidar spikes and DVS events. In each sequence, we collect approximately 5 seconds as the raw data pool. Manual annotations in the recordings are provided at a frequency of 50 Hz. As a result, this dataset has 103.3k labeled timestamps and 229.5k labels in total. It is the first work to build a neuromorphic multimodal object detection dataset involving high-speed and low-light scenarios.