MAKEN/clipscore

• What's in here?
• How do I run the code?
• • Command Line
  • MSCOCO dataset in pycocoevalcap
  • Reproducibility notes:
  • Acknowledgment

What's in here?

This repo contains the code for our EMNLP 2021 paper: CLIPScore: A
Reference-free Evaluation Metric for Image
Captioning. CLIPScore is a metric
that you can use to evaluate the quality of an automatic image
captioning system. In our paper, we show that CLIPScore achieves high
correlation with human judgment on literal image captioning
tasks. However, unlike BLEU or CIDEr, CLIPScore doesn't require
reference captions.

If you find the paper or this code useful, please consider citing:

@inproceedings{hessel2021clipscore,
  title={{CLIPScore:} A Reference-free Evaluation Metric for Image Captioning},
  author={Hessel, Jack and Holtzman, Ari and Forbes, Maxwell and Bras, Ronan Le and Choi, Yejin},
  booktitle={EMNLP},
  year={2021}
}

How do I run the code?

Command Line

Example usage

> python clipscore.py example/good_captions.json example/images/
...
CLIPScore: 0.8584

If you include optionally some references, you will see RefCLIPScore, alongside a usual set of
caption generation evaluation metrics. The references are optional.

> python clipscore.py example/good_captions.json example/images/ --references_json example/refs.json
...
BLEU-1: 0.6667
BLEU-2: 0.4899
BLEU-3: 0.3469
BLEU-4: 0.0000
METEOR: 0.3444
ROUGE: 0.4280
CIDER: 0.5637
SPICE: 0.4000
CLIPScore: 0.8584
RefCLIPScore: 0.8450

Worse captions should get lower scores:

> python clipscore.py example/bad_captions.json example/images/ --references_json example/refs.json
...
BLEU-1: 0.4815
BLEU-2: 0.2404
BLEU-3: 0.1359
BLEU-4: 0.0000
METEOR: 0.1861
ROUGE: 0.3121
CIDER: 0.2790
SPICE: 0.1500
CLIPScore: 0.7153
RefCLIPScore: 0.7253

You can treat/report CLIPScore and RefCLIPScore similarly to the other
evaluation metrics. See the paper for more details about CLIPScore and
RefCLIPScore. Full usage options can be given by python clipscore.py -h. An example set of inputs, including a candidate json, image
directory, and references json is given this repo under example/

The input files are formatted as follows:

The candidates json should be a dictionary that maps from
{"string_image_identifier": "candidate"}, e.g.,

{'image1': 'an orange cat and a grey cat are lying together.',
 'image2': 'a black dog looks at the camera.'
 ...}

The image directory should be a directory containing the images that
act as the keys in the candidates json, e.g.,

images/
├── image1.jpg
└── image2.jpg

and, finally, the references json should be a dictionary that maps from
{"string_image_identifier": ["list", "of", "references"]}, e.g.,

{"image1": ["two cats are sleeping next to each other.",
            "a grey cat is cuddling with an orange cat on a blanket.",
	    "the orange cat is happy that the black cat is close to it."],
 "image2": ["a dog is wearing ear muffs as it lies on a carpet.",
            "a black dog and an orange cat are looking at the photographer.",
	    "headphones are placed on a dogs ears."]}

MSCOCO dataset in pycocoevalcap

If you're running on the MSCOCO dataset and using the standard
evaluation toolkit, you can use our version of
pycocoevalcap to evaluate.
You won't even need to download the original MSCOCO images, thanks to
a bit of magic :-)

To use pycocoevalcap on the MSCOCO dataset in the MSCOCO format, you
can simply use:

pip install git+https://github.com/jmhessel/pycocoevalcap.git

there is an example evaluation in that repo under
examples/eval.py. After pip installing, if you clone the
pycocoeval repo and run

python eval.py

after a bit of time, the output should be:

Bleu_1: 0.579
Bleu_2: 0.404
Bleu_3: 0.279
Bleu_4: 0.191
METEOR: 0.195
ROUGE_L: 0.396
CIDEr: 0.600
SPICE: 0.133
CLIPScore: 0.528
RefCLIPScore: 0.605

Reproducibility notes:

• CLIPScore can run either on CPU or GPU. But, there are slight
  differences due to floating point precision. As discussed
  here,
  on CPU, all operations run in float32, but on GPU, some operations
  run in float16. The differences are generally small (e.g., for the
  example run above, with example/good_captions.json captions and
  example/images/ images, on CPU, the output is CLIPScore: 0.8585,
  but on GPU, the output is CLIPScore: 0.8584.) All experiments in the
  paper were run on GPU, and this code will raise a warning if you're not
  using a GPU.

• Because CLIPScore depends on the images to compute, resizing,
  compressing, etc. can all cause slight differences in computing the
  CLIPScore. Even saving a jpg twice can result in different
  compression, because that format is lossy! To this end, we release
  the checksums of the images we used for the paper. see checksums/
  for more info. For the pycocoevalcap repo, we have also included the
  checksums for MSCOCO --- see
  here
  for more info.

• The prompt we used for the text side of CLIP, as mentioned in the
  paper is ``A photo depicts" This is hard-coded into this repo. Other
  prompts will result in slightly different results, and we don't
  recommend them for the sake of reproducibility.

  Acknowledgment

  The authors would like to thank Jungo Kasai for being the first to use
  this repo. Thanks to Jungo, we fixed a few issues, and added some
  information about reproducibility that was missing before.