Corpus Open Source Projects

NLTK Source

大规模中文自然语言处理语料 Large Scale Chinese Corpus for NLP

A Deep-Learning-Based Chinese Speech Recognition System 基于深度学习的中文语音识别系统

Software in C and data files for the popular GloVe model for distributed word representations, a.k.a. word vectors or embeddings

The AI search platform

Google AI 2018 BERT pytorch implementation

pycorrector is a toolkit for text error correction. 文本纠错，实现了Kenlm，T5，MacBERT，ChatGLM3，Qwen2.5等模型应用在纠错场景，开箱即用。

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Alphabetical list of free/public domain datasets with text data for use in Natural Language Processing (NLP)

A TensorFlow implementation of DeepMind's WaveNet paper

A collection of small corpuses of interesting data for the creation of bots and similar stuff.

Randomized testing for Go

搜索所有中文NLP数据集，附常用英文NLP数据集

This repo contains a list of the 10,000 most common English words in order of frequency, as determined by n-gram frequency analysis of the Google's Trillion Word Corpus.

中文人名语料库。人名生成器。中文姓名,姓氏,名字,称呼,日本人名,翻译人名,英文人名。可用于中文分词、人名实体识别。

Language, engine, and tooling for expressing, testing, and evaluating composable language rules on input strings.

中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard

中文公开聊天语料库

Speech-to-Text-WaveNet : End-to-end sentence level English speech recognition based on DeepMind's WaveNet and tensorflow

Language-Agnostic SEntence Representations

A simple, extensible Markov chain generator.

Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo

Deep Learning and deep reinforcement learning research papers and some codes

Natural Language Processing Tasks and References

My tensorflow implementation of "A neural conversational model", a Deep learning based chatbot

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

A JavaScript Engine Fuzzer

🔊 A comprehensive list of open-source datasets for voice and sound computing (95+ datasets).

Awesome Chatbot Projects,Corpus,Papers,Tutorials.Chinese Chatbot =>:

用于训练中英文对话系统的语料库 Datasets for Training Chatbot System

A Large-scale Chinese Short-Text Conversation Dataset and Chinese pre-training dialog models

Deep Learning model to analyze a large corpus of clear text passwords.

Single-document unsupervised keyword extraction

GPT2 for Multiple Languages, including pretrained models. GPT2 多语言支持, 15亿参数中文预训练模型

A collection of corpora for named entity recognition (NER) and entity recognition tasks. These annotated datasets cover a variety of languages, domains and entity types.

Dual LSTM Encoder for Dialog Response Generation

Turn Chinese natural language into structured data 中文自然语言理解

非常全的文言文（古文）-现代文平行语料

Annotator for Chinese Text Corpus (UNDER DEVELOPMENT) 中文文本标注工具

A multilingual dialog corpus

Question answering dataset featured in "Teaching Machines to Read and Comprehend

公司名语料库。机构名语料库。公司简称,缩写,品牌词,企业名。可用于中文分词、机构名实体识别。

Toy Python implementation of http://www-nlp.stanford.edu/projects/glove/

AutoPhrase: Automated Phrase Mining from Massive Text Corpora

The most accurate natural language detection library for Rust, suitable for short text and mixed-language text

S2ORC: The Semantic Scholar Open Research Corpus: https://www.aclweb.org/anthology/2020.acl-main.447/

:helicopter: 保险行业语料库，聊天机器人

Data repository for pretrained NLP models and NLP corpora.

PISA: Performant Indexes and Search for Academia

Large-scale Pre-training Corpus for Chinese 100G 中文预训练语料

ChatGPT 中文语料库 对话语料 小说语料 客服语料 用于训练大模型

Collections of Chinese NLP corpus

A list of datasets/corpora for NLP tasks, in reverse chronological order.

Generate text line images for training deep learning OCR models

The Classical Language Toolkit

An R package for the Quantitative Analysis of Textual Data

Crawl BookCorpus

[CBLUE1] 中文医疗信息处理基准CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

Four word embedding models implemented in Python. Supporting arbitrary context features

Chatbot in 200 lines of code using TensorLayer