bash baidu_train.sh运行kmqa代码
bash pkukg.sh KG预训练的代码
bash pt_books.sh 在书籍上预训练的代码


结果参考：
IR baseline 36.4 34.1
Random guess 21.3 22.8
Co-Matching (Wang et al., 2018) 56.1 45.8
BiDAF (Seo et al., 2017) 52.7 43.6
SeaReader (Zhang et al., 2018) 58.2 48.4
Multi-Matching (Tang et al., 2019) 58.4 48.7
BERT-base (Devlin et al., 2019) 64.2 52.2
ERNIE (Sun et al., 2019) 64.7 53.4
RoBERTa-wwm-ext-large (Cui et al., 2019) 70.8 57.9
KMQA (BERT-base) 67.9 57.1
KMQA (RoBERTa-wwm-ext-large) 71.1 61.8


之前的其他初步结果，有点乱
random guess: 0.185; 0.228; 0.2; 0.167; 0.1367
ir baseline: 0.31666666666666665 / 0.3436928702010969
ir baseline (wiki): ACC:  0.26
ir baseline (other training data 2636 examples):  0.2943627450980392
bert_base_no_evidence (use other training data 2636 examples): 0.33454545454545453 
bert_base_top_1_evidence (use other training data 2636 examples): 0.4290909090909091 
biobert_english:0.4036363636363636
biobert_english(no evidence): 0.32
albert_english (no evidence): 0.33
bert_base_top_1_evidence_add_part_of_human_reason: 0.4818 0.5



参考(commonsense数据集的结果)：
BERTBase (single model)	University College London	03/13/2019	53.0	
RoBERTa (single model)	Facebook AI	08/13/2019	72.1	
RoBERTa (ensemble model)	Facebook AI	08/13/2019	72.5
ALBERT (ensemble model)	Zhiyan Technology	12/18/2019	76.5

Python环境（pip freeze）：

absl-py @ file:///tmp/build/80754af9/absl-py_1615411197583/work
addict==2.4.0
aiohttp==3.7.4.post0
alabaster==0.7.12
appdirs==1.4.4
argon2-cffi @ file:///tmp/build/80754af9/argon2-cffi_1613037492802/work
aspy.yaml==1.3.0
astor==0.8.1
astroid==2.3.3
async-generator==1.10
async-timeout==3.0.1
attrs @ file:///tmp/build/80754af9/attrs_1604765588209/work
Babel==2.9.0
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
black==19.3b0
bleach @ file:///tmp/build/80754af9/bleach_1612211392645/work
blis==0.4.1
boto==2.49.0
boto3==1.10.26
botocore==1.13.26
catalogue==2.0.4
certifi==2021.5.30
cffi==1.14.0
cfgv==2.0.1
chardet==3.0.4
cleanlab==0.1.0
click==7.1.2
codecov==2.1.11
colorama==0.4.4
conllu==1.3.1
contextvars==2.4
coverage==5.5
cryptography==3.4.7
cycler==0.10.0
cymem==2.0.3
Cython==0.29.23
dataclasses==0.8
decorator==4.4.2
defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work
docutils==0.15.2
dowhy==0.6
editdistance==0.5.3
en-core-sci-sm==0.2.4
en-core-web-lg==2.2.5
en-core-web-sm==2.2.5
entrypoints==0.3
filelock==3.0.12
flake8==3.7.8
flaky==3.7.0
ftfy==5.9
funcsigs==1.0.2
future==0.18.2
gast @ file:///tmp/build/80754af9/gast_1597433534803/work
gensim==3.8.1
gevent==1.4.0
google-pasta==0.2.0
googletrans==2.4.0
greenlet==0.4.15
grpcio==1.25.0
h5py @ file:///tmp/build/80754af9/h5py_1593454121459/work
identify==1.4.9
idna==2.8
idna-ssl==1.1.0
imageio==2.9.0
imageio-ffmpeg==0.4.2
imagesize==1.2.0
immutables==0.15
importlib-metadata @ file:///tmp/build/80754af9/importlib-metadata_1617877310517/work
importlib-resources==1.0.2
ipykernel @ file:///tmp/build/80754af9/ipykernel_1596206602906/work/dist/ipykernel-5.3.4-py3-none-any.whl
ipython @ file:///tmp/build/80754af9/ipython_1593447367857/work
ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work
ipywidgets @ file:///tmp/build/80754af9/ipywidgets_1610481889018/work
isodate==0.6.0
isort==4.3.21
javabridge==1.0.19
jedi==0.17.0
jeepney==0.6.0
jieba==0.42.1
Jinja2 @ file:///tmp/build/80754af9/jinja2_1612213139570/work
jmespath==0.9.4
joblib==0.14.0
JPype1==0.7.1
jsonnet==0.17.0
jsonpickle==2.0.0
jsonschema @ file:///tmp/build/80754af9/jsonschema_1602607155483/work
jupyter==1.0.0
jupyter-client @ file:///tmp/build/80754af9/jupyter_client_1616770841739/work
jupyter-console @ file:///tmp/build/80754af9/jupyter_console_1616615302928/work
jupyter-core @ file:///tmp/build/80754af9/jupyter_core_1612213308682/work
jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work
jupyterlab-widgets @ file:///tmp/build/80754af9/jupyterlab_widgets_1609884341231/work
Keras==2.3.1
Keras-Applications @ file:///tmp/build/80754af9/keras-applications_1594366238411/work
keras-bert==0.81.0
keras-embed-sim==0.7.0
keras-layer-normalization==0.14.0
keras-multi-head==0.22.0
keras-pos-embd==0.11.0
keras-position-wise-feed-forward==0.6.0
Keras-Preprocessing @ file:///tmp/build/80754af9/keras-preprocessing_1612283640596/work
keras-self-attention==0.41.0
keras-transformer==0.32.0
keyring==23.0.1
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
livereload==2.6.3
lxml==4.4.2
Markdown @ file:///tmp/build/80754af9/markdown_1614363833670/work
MarkupSafe==1.1.1
matplotlib==3.2.2
mccabe==0.6.1
mistune==0.8.4
mkl-fft==1.3.0
mkl-random==1.1.1
mkl-service==2.3.0
mmcv==0.3.1
mock==3.0.5
more-itertools==8.7.0
moviepy==1.0.3
mpmath==1.2.1
multidict==5.1.0
murmurhash==1.0.2
mypy==0.730
mypy-extensions==0.4.3
nbclient @ file:///tmp/build/80754af9/nbclient_1614364831625/work
nbconvert @ file:///tmp/build/80754af9/nbconvert_1601914804165/work
nbformat @ file:///tmp/build/80754af9/nbformat_1617383369282/work
nest-asyncio @ file:///tmp/build/80754af9/nest-asyncio_1613680548246/work
networkx==2.5.1
nltk==3.4.5
nodeenv==1.3.4
notebook @ file:///tmp/build/80754af9/notebook_1616443456543/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1603487797006/work
numpydoc==1.1.0
opencv-python==4.5.1.48
OpenNMT-py==0.9.0
opt-einsum==3.1.0
overrides==2.0
Owlready2==0.22
packaging @ file:///tmp/build/80754af9/packaging_1611952188834/work
pandas==0.25.3
pandocfilters @ file:///tmp/build/80754af9/pandocfilters_1605120937332/work
parsimonious==0.8.1
parso @ file:///tmp/build/80754af9/parso_1617223946239/work
pathlib==1.0.1
pathy==0.4.0
patsy==0.5.1
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pgmpy==0.1.14
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==7.0.0
pke @ git+https://github.com/boudinfl/pke.git@9b687742a7335575ec8c8006541e264fda9c9eee
pkginfo==1.7.0
pkuseg==0.0.22
plac==1.1.3
pluggy==0.13.1
pre-commit==1.18.3
preshed==3.0.2
proglog==0.1.9
prometheus-client @ file:///tmp/build/80754af9/prometheus_client_1618088486455/work
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1616415428029/work
protobuf==3.14.0
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py==1.8.1
pycausal @ git+git://github.com/bd2kccd/py-causal@990dd78114d1cf61c637b69d1b009c93d0d8021e
pycodestyle==2.5.0
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pydantic==1.7.3
pydot==1.4.2
pyemd==0.5.1
pyflakes==2.1.1
Pygments @ file:///tmp/build/80754af9/pygments_1615143339740/work
pygraphviz==1.3
pyknp==0.4.1
pylint==2.4.4
pymongo==3.9.0
pypandoc==1.5
pyparsing @ file:///home/linux1/recipes/ci/pyparsing_1610983426697/work
pyrsistent @ file:///tmp/build/80754af9/pyrsistent_1600141725711/work
pytest==5.3.5
pytest-cov==2.11.1
python-dateutil==2.8.0
pytils==0.3
pytorch-crf==0.7.2
pytorch-pretrained-bert==0.6.2
pytorch-transformers==1.1.0
pytz @ file:///tmp/build/80754af9/pytz_1612215392582/work
PyYAML==5.1.2
pyzmq==20.0.0
qtconsole @ file:///tmp/build/80754af9/qtconsole_1616775094278/work
QtPy==1.9.0
rdflib==4.2.2
readme-renderer==29.0
recordclass==0.12.0.1
regex==2019.11.1
requests==2.22.0
requests-toolbelt==0.9.1
responses==0.13.2
rfc3986==1.4.0
s3transfer==0.2.1
sacremoses==0.0.35
scikit-learn==0.21.3
scipy==1.5.4
SecretStorage==3.3.1
Send2Trash @ file:///tmp/build/80754af9/send2trash_1607525499227/work
sentence-transformers==0.2.5
sentencepiece==0.1.83
six @ file:///tmp/build/80754af9/six_1605205335545/work
sklearn==0.0
skorch==0.6.0
smart-open==3.0.0
snowballstemmer==2.1.0
spacy==3.0.6
spacy-legacy==3.0.5
spacy-pkuseg==0.0.28
SPARQLWrapper==1.8.4
Sphinx==3.5.4
sphinx-autobuild==2021.3.14
sphinx-rtd-theme==0.5.2
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
sqlitedict==1.6.0
sqlparse==0.4.1
srsly==2.4.1
statsmodels==0.12.2
sympy==1.8
tabulate==0.8.6
tagme==0.1.3
tensorboard==1.15.0
tensorboardX==2.2
tensorflow==1.14.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
terminado==0.9.4
testpath @ file:///home/ktietz/src/ci/testpath_1611930608132/work
texar==0.2.4
text2vec==0.1.3
thinc==8.0.3
thulac==0.2.1
toml==0.10.0
tools==0.1.9
torch==1.8.1
torchtext==0.4.0
tornado @ file:///tmp/build/80754af9/tornado_1606942266872/work
tox==3.14.3
tqdm==4.39.0
traitlets==4.3.3
transformers==2.3.0
translate==3.5.0
twine==3.4.1
typed-ast==1.4.0
typer==0.3.2
typing-extensions @ file:///home/ktietz/src/ci_mi/typing_extensions_1612808209620/work
Unidecode==1.2.0
urllib3==1.25.11
virtualenv==16.7.9
wasabi==0.8.2
wcwidth @ file:///tmp/build/80754af9/wcwidth_1593447189090/work
webencodings==0.5.1
Werkzeug @ file:///home/ktietz/src/ci/werkzeug_1611932622770/work
widgetsnbextension==3.5.1
word2number==1.1
wrapt==1.12.1
yarl==1.6.3
zh-core-web-sm @ file:///data/ldf/GCI/data/zh_core_web_sm-3.0.0.tar.gz
zipp @ file:///tmp/build/80754af9/zipp_1615904174917/work