iLoveElysia/openbilibili
1
0
Fork 0
mirror of https://github.com/iLoveElysia/openbilibili.git synced 2026-09-02 07:38:04 -05:00
Code Issues Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Donny
2019-04-22 20:46:32 +08:00
commit 49ab8aadd1
25441 changed files with 4055000 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
vendor/github.com/go-ego/gse/BUILD.bazel generated vendored Normal file
View File

@@ -0,0 +1,31 @@
load("@io_bazel_rules_go//go:def.bzl", "go_library")
go_library(
name = "go_default_library",
srcs = [
"dictionary.go",
"seg.go",
"segmenter.go",
"test_utils.go",
"token.go",
"utils.go",
],
importmap = "go-common/vendor/github.com/go-ego/gse",
importpath = "github.com/go-ego/gse",
visibility = ["//visibility:public"],
deps = ["//vendor/github.com/go-ego/cedar:go_default_library"],
)
filegroup(
name = "package-srcs",
srcs = glob(["**"]),
tags = ["automanaged"],
visibility = ["//visibility:private"],
)
filegroup(
name = "all-srcs",
srcs = [":package-srcs"],
tags = ["automanaged"],
visibility = ["//visibility:public"],
)

69
vendor/github.com/go-ego/gse/CONTRIBUTING.md generated vendored Normal file
View File

@@ -0,0 +1,69 @@
# Contribution Guidelines
## Introduction
This document explains how to contribute changes to the Ego project. It assumes you have followed the README.md and [API Document](https://github.com/go-ego/gse/blob/master/docs/doc.md). <!--Sensitive security-related issues should be reported to [security@Ego.io](mailto:security@Ego.io.)-->
## Bug reports
Please search the issues on the issue tracker with a variety of keywords to ensure your bug is not already reported.
If unique, [open an issue](https://github.com/go-ego/gse/issues/new) and answer the questions so we can understand and reproduce the problematic behavior.
The burden is on you to convince us that it is actually a bug in Ego. This is easiest to do when you write clear, concise instructions so we can reproduce the behavior (even if it seems obvious). The more detailed and specific you are, the faster we will be able to help you. Check out [How to Report Bugs Effectively](http://www.chiark.greenend.org.uk/~sgtatham/bugs.html).
Please be kind, remember that Ego comes at no cost to you, and you're getting free help.
## Discuss your design
The project welcomes submissions but please let everyone know what you're working on if you want to change or add something to the Ego repositories.
Before starting to write something new for the Ego project, please [file an issue](https://github.com/go-ego/gse/issues/new). Significant changes must go through the [change proposal process](https://github.com/go-ego/proposals) before they can be accepted.
This process gives everyone a chance to validate the design, helps prevent duplication of effort, and ensures that the idea fits inside the goals for the project and tools. It also checks that the design is sound before code is written; the code review tool is not the place for high-level discussions.
## Testing redux
Before sending code out for review, run all the tests for the whole tree to make sure the changes don't break other usage and keep the compatibility on upgrade. You must be test on Mac, Windows, Linux and other. You should install the CLI for Circle CI, as we are using the server for continous testing.
## Code review
In addition to the owner, Changes to Ego must be reviewed before they are accepted, no matter who makes the change even if it is a maintainer. We use GitHub's pull request workflow to do that and we also use [LGTM](http://lgtm.co) to ensure every PR is reviewed by vz or least 2 maintainers.
## Sign your work
The sign-off is a simple line at the end of the explanation for the patch. Your signature certifies that you wrote the patch or otherwise have the right to pass it on as an open-source patch.
## Maintainers
To make sure every PR is checked, we got team maintainers. A maintainer should be a contributor of Ego and contributed at least 4 accepted PRs.
## Owners
Since Ego is a pure community organization without any company support, Copyright 2016 The go-ego Project Developers.
## Versions
Ego has the `master` branch as a tip branch and has version branches such as `v0.30.0`. `v0.40.0` is a release branch and we will tag `v0.40.0` for binary download. If `v0.40.0` has bugs, we will accept pull requests on the `v0.40.0` branch and publish a `v0.40.1` tag, after bringing the bug fix also to the master branch.
Since the `master` branch is a tip version, if you wish to use Ego in production, please download the latest release tag version. All the branches will be protected via GitHub, all the PRs to every branch must be reviewed by two maintainers and must pass the automatic tests.
## Copyright
Code that you contribute should use the standard copyright header:
```
// Copyright 2016 The go-ego Project Developers. See the COPYRIGHT
// file at the top-level directory of this distribution and at
// https://github.com/go-ego/gse/blob/master/LICENSE
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.
```
Files in the repository contain copyright from the year they are added to the year they are last changed. If the copyright author is changed, just paste the header below the old one.

21
vendor/github.com/go-ego/gse/Gopkg.lock generated vendored Normal file
View File

@@ -0,0 +1,21 @@
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
[[projects]]
branch = "master"
name = "github.com/go-ego/cedar"
packages = ["."]
revision = "39a3301a49c052d18c55ade2bd080cda1ccc0446"
[[projects]]
branch = "master"
name = "github.com/vcaesar/tt"
packages = ["."]
revision = "54af4381b253a2380245310c25e1ff66dfc46f62"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
inputs-digest = "e7f95d7aa5572a5d63d68ac0ac960c325f9c29fab95419e0ec24688333fb8c6b"
solver-name = "gps-cdcl"
solver-version = 1

38
vendor/github.com/go-ego/gse/Gopkg.toml generated vendored Normal file
View File

@@ -0,0 +1,38 @@
# Gopkg.toml example
#
# Refer to https://golang.github.io/dep/docs/Gopkg.toml.html
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
#
# [prune]
# non-go = false
# go-tests = true
# unused-packages = true
[[constraint]]
branch = "master"
name = "github.com/go-ego/cedar"
[[constraint]]
branch = "master"
name = "github.com/vcaesar/tt"
[prune]
go-tests = true
unused-packages = true

201
vendor/github.com/go-ego/gse/LICENSE generated vendored Normal file
View File

@@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

112
vendor/github.com/go-ego/gse/README.md generated vendored Normal file
View File

@@ -0,0 +1,112 @@
# gse
Go efficient text segmentation; support english, chinese, japanese and other.
<!--<img align="right" src="https://raw.githubusercontent.com/go-ego/ego/master/logo.jpg">-->
<!--<a href="https://circleci.com/gh/go-ego/ego/tree/dev"><img src="https://img.shields.io/circleci/project/go-ego/ego/dev.svg" alt="Build Status"></a>-->
[![CircleCI Status](https://circleci.com/gh/go-ego/gse.svg?style=shield)](https://circleci.com/gh/go-ego/gse)
[![codecov](https://codecov.io/gh/go-ego/gse/branch/master/graph/badge.svg)](https://codecov.io/gh/go-ego/gse)
[![Build Status](https://travis-ci.org/go-ego/gse.svg)](https://travis-ci.org/go-ego/gse)
[![Go Report Card](https://goreportcard.com/badge/github.com/go-ego/gse)](https://goreportcard.com/report/github.com/go-ego/gse)
[![GoDoc](https://godoc.org/github.com/go-ego/gse?status.svg)](https://godoc.org/github.com/go-ego/gse)
[![Release](https://github-release-version.herokuapp.com/github/go-ego/gse/release.svg?style=flat)](https://github.com/go-ego/gse/releases/latest)
[![Join the chat at https://gitter.im/go-ego/ego](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/go-ego/ego?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
<!--<a href="https://github.com/go-ego/ego/releases"><img src="https://img.shields.io/badge/%20version%20-%206.0.0%20-blue.svg?style=flat-square" alt="Releases"></a>-->
[简体中文](https://github.com/go-ego/gse/blob/master/README_zh.md)
<a href="https://github.com/go-ego/gse/blob/master/dictionary.go">Dictionary </a> with double array trie (Double-Array Trie) to achieve,
<a href="https://github.com/go-ego/gse/blob/master/segmenter.go">Sender </a> algorithm is the shortest path based on word frequency plus dynamic programming.
Support common and search engine two participle mode, support user dictionary, POS tagging, run<a href="https://github.com/go-ego/gse/blob/master/server/server.go"> JSON RPC service</a>.
Text Segmentation speed<a href="https://github.com/go-ego/gse/blob/master/tools/benchmark.go"> single thread</a> 9MB/s，<a href="https://github.com/go-ego/gse/blob/master/tools/goroutines.go">goroutines concurrent</a> 42MB/s (8 nuclear Macbook Pro).
## Install / update
```
go get -u github.com/go-ego/gse
```
## [Build-tools](https://github.com/go-ego/re)
```
go get -u github.com/go-ego/re
```
### re gse
To create a new gse application
```
$ re gse my-gse
```
### re run
To run the application we just created, you can navigate to the application folder and execute:
```
$ cd my-gse && re run
```
## Use
```go
package main
import (
"fmt"
"github.com/go-ego/gse"
)
func main() {
// Load the dictionary
var seg gse.Segmenter
// Loading the default dictionary
seg.LoadDict()
// seg.LoadDict("your gopath"+"/src/github.com/go-ego/gse/data/dict/dictionary.txt")
// Text Segmentation
text := []byte("你好世界, Hello world.")
fmt.Println(segmenter.String(text, true))
segments := segmenter.Segment(text)
// Handle word segmentation results
// Support for normal mode and search mode two participle,
// see the comments in the code ToString function.
// The search mode is mainly used to provide search engines
// with as many keywords as possible
fmt.Println(gse.ToString(segments, true))
}
```
[Look at an custom dictionary example](/examples/dict/main.go)
```Go
package main
import (
"fmt"
"github.com/go-ego/gse"
)
func main() {
var seg gse.Segmenter
seg.LoadDict("zh,testdata/test_dict.txt,testdata/test_dict1.txt")
text1 := []byte("你好世界, Hello world")
segments := seg.Segment(text1)
fmt.Println(gse.ToString(segments))
}
```
[Look at an Chinese example](https://github.com/go-ego/gse/blob/master/examples/example.go)
[Look at an Japanese example](https://github.com/go-ego/gse/blob/master/examples/jp/main.go)
## License
Gse is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), base on [sego](https://github.com/huichen/sego).

115
vendor/github.com/go-ego/gse/README_zh.md generated vendored Normal file
View File

@@ -0,0 +1,115 @@
# [gse](https://github.com/go-ego/gse)
Go 语言高效分词, 支持英文、中文、日文等
<!--<img align="right" src="https://raw.githubusercontent.com/go-ego/ego/master/logo.jpg">-->
<!--<a href="https://circleci.com/gh/go-ego/ego/tree/dev"><img src="https://img.shields.io/circleci/project/go-ego/ego/dev.svg" alt="Build Status"></a>-->
[![CircleCI Status](https://circleci.com/gh/go-ego/gse.svg?style=shield)](https://circleci.com/gh/go-ego/gse)
[![codecov](https://codecov.io/gh/go-ego/gse/branch/master/graph/badge.svg)](https://codecov.io/gh/go-ego/gse)
[![Build Status](https://travis-ci.org/go-ego/gse.svg)](https://travis-ci.org/go-ego/gse)
[![Go Report Card](https://goreportcard.com/badge/github.com/go-ego/gse)](https://goreportcard.com/report/github.com/go-ego/gse)
[![GoDoc](https://godoc.org/github.com/go-ego/gse?status.svg)](https://godoc.org/github.com/go-ego/gse)
[![Release](https://github-release-version.herokuapp.com/github/go-ego/gse/release.svg?style=flat)](https://github.com/go-ego/gse/releases/latest)
[![Join the chat at https://gitter.im/go-ego/ego](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/go-ego/ego?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
<!--<a href="https://github.com/go-ego/ego/releases"><img src="https://img.shields.io/badge/%20version%20-%206.0.0%20-blue.svg?style=flat-square" alt="Releases"></a>-->
<a href="https://github.com/go-ego/gse/blob/master/dictionary.go">词典</a>用双数组 trie（Double-Array Trie）实现，
<a href="https://github.com/go-ego/gse/blob/master/segmenter.go">分词器</a>算法为基于词频的最短路径加动态规划。
支持普通和搜索引擎两种分词模式，支持用户词典、词性标注，可运行<a href="https://github.com/go-ego/gse/blob/master/server/server.go"> JSON RPC 服务</a>。
分词速度<a href="https://github.com/go-ego/gse/blob/master/tools/benchmark.go">单线程</a> 9MB/s，<a href="https://github.com/go-ego/gse/blob/master/tools/goroutines.go">goroutines 并发</a> 42MB/s（8核 Macbook Pro）。
QQ 群: 120563750
## 安装/更新
```
go get -u github.com/go-ego/gse
```
## [Build-tools](https://github.com/go-ego/re)
```
go get -u github.com/go-ego/re
```
### re gse
创建一个新的 gse 程序
```
$ re gse my-gse
```
### re run
运行我们刚刚创建的应用程序, CD 到程序文件夹并执行:
```
$ cd my-gse && re run
```
## 使用
```go
package main
import (
"fmt"
"github.com/go-ego/gse"
)
func main() {
// 载入词典
var seg gse.Segmenter
// 加载默认字典
seg.LoadDict()
// seg.LoadDict("your gopath"+"/src/github.com/go-ego/gse/data/dict/dictionary.txt")
// 分词
text := []byte("中华人民共和国中央人民政府")
fmt.Println(seg.String(text, true))
segments := seg.Segment(text)
// 处理分词结果
// 支持普通模式和搜索模式两种分词，见代码中 ToString 函数的注释。
// 搜索模式主要用于给搜索引擎提供尽可能多的关键字
fmt.Println(gse.ToString(segments))
text1 := []byte("深圳地标建筑, 深圳地王大厦")
segments1 := seg.Segment([]byte(text1))
fmt.Println(gse.ToString(segments1, true))
}
```
[自定义词典分词示例](/examples/dict/main.go)
```Go
package main
import (
"fmt"
"github.com/go-ego/gse"
)
func main() {
var seg gse.Segmenter
seg.LoadDict("zh,testdata/test_dict.txt,testdata/test_dict1.txt")
text1 := []byte("所以, 你好, 再见")
segments := seg.Segment(text1)
fmt.Println(gse.ToString(segments))
}
```
[中文分词示例](/examples/example.go)
[日文分词示例](/examples/jp/main.go)
## License
Gse is primarily distributed under the terms of both the MIT license and the Apache License (Version 2.0), base on [sego](https://github.com/huichen/sego).

25
vendor/github.com/go-ego/gse/circle.yml generated vendored Normal file
View File

@@ -0,0 +1,25 @@
# circle.yml #
# machine:
# go:
# version: 1.9.1
version: 2
jobs:
build:
docker:
# - image: golang:1.10.1
- image: govgo/go:1.10.3
working_directory: /gopath/src/github.com/go-ego/gse
steps:
- checkout
# specify any bash command here prefixed with `run: `
# - run: go get -u github.com/go-ego/cedar
# - run: go get -u github.com/go-ego/gse
- run: go get -v -t -d ./...
- run: go test -v ./...
# codecov.io
- run: go test -v -covermode=count -coverprofile=coverage.out
- run: bash <(curl -s https://codecov.io/bash)

89
vendor/github.com/go-ego/gse/dictionary.go generated vendored Normal file
View File

@@ -0,0 +1,89 @@
// Copyright 2013 Hui Chen
// Copyright 2016 ego authors
//
// Licensed under the Apache License, Version 2.0 (the "License"): you may
// not use this file except in compliance with the License. You may obtain
// a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations
// under the License.
package gse
import (
"github.com/go-ego/cedar"
)
// Dictionary 结构体实现了一个字串前缀树，
// 一个分词可能出现在叶子节点也有可能出现在非叶节点
type Dictionary struct {
trie *cedar.Cedar // Cedar 前缀树
maxTokenLen int // 词典中最长的分词
tokens []Token // 词典中所有的分词，方便遍历
totalFrequency int64 // 词典中所有分词的频率之和
}
// NewDict new dictionary
func NewDict() *Dictionary {
return &Dictionary{trie: cedar.New()}
}
// MaxTokenLen 词典中最长的分词
func (dict *Dictionary) MaxTokenLen() int {
return dict.maxTokenLen
}
// NumTokens 词典中分词数目
func (dict *Dictionary) NumTokens() int {
return len(dict.tokens)
}
// TotalFrequency 词典中所有分词的频率之和
func (dict *Dictionary) TotalFrequency() int64 {
return dict.totalFrequency
}
// addToken 向词典中加入一个分词
func (dict *Dictionary) addToken(token Token) {
bytes := textSliceToBytes(token.text)
_, err := dict.trie.Get(bytes)
if err == nil {
return
}
dict.trie.Insert(bytes, dict.NumTokens())
dict.tokens = append(dict.tokens, token)
dict.totalFrequency += int64(token.frequency)
if len(token.text) > dict.maxTokenLen {
dict.maxTokenLen = len(token.text)
}
}
// lookupTokens 在词典中查找和字元组 words 可以前缀匹配的所有分词
// 返回值为找到的分词数
func (dict *Dictionary) lookupTokens(words []Text,
tokens []*Token) (numOfTokens int) {
var (
id, value int
err error
)
for _, word := range words {
id, err = dict.trie.Jump(word, id)
if err != nil {
break
}
value, err = dict.trie.Value(id)
if err == nil {
tokens[numOfTokens] = &dict.tokens[value]
numOfTokens++
}
}
return
}

28
vendor/github.com/go-ego/gse/seg.go generated vendored Normal file
View File

@@ -0,0 +1,28 @@
package gse
// Segment 文本中的一个分词
type Segment struct {
// 分词在文本中的起始字节位置
start int
// 分词在文本中的结束字节位置（不包括该位置）
end int
// 分词信息
token *Token
}
// Start 返回分词在文本中的起始字节位置
func (s *Segment) Start() int {
return s.start
}
// End 返回分词在文本中的结束字节位置（不包括该位置）
func (s *Segment) End() int {
return s.end
}
// Token 返回分词信息
func (s *Segment) Token() *Token {
return s.token
}

530
vendor/github.com/go-ego/gse/segmenter.go generated vendored Normal file
View File

@@ -0,0 +1,530 @@
// Copyright 2013 Hui Chen
// Copyright 2016 ego authors
//
// Licensed under the Apache License, Version 2.0 (the "License"): you may
// not use this file except in compliance with the License. You may obtain
// a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
// WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
// License for the specific language governing permissions and limitations
// under the License.
/*
package gse Go efficient text segmentation, Go 语言分词
*/
package gse
import (
"bufio"
"fmt"
"io"
"log"
"math"
"os"
"path"
"runtime"
"strconv"
"strings"
"unicode"
"unicode/utf8"
)
const (
version string = "v0.10.0.106, Danube River!"
minTokenFrequency = 2 // 仅从字典文件中读取大于等于此频率的分词
)
// GetVersion get the gse version
func GetVersion() string {
return version
}
// Segmenter 分词器结构体
type Segmenter struct {
dict *Dictionary
}
// jumper 该结构体用于记录 Viterbi 算法中某字元处的向前分词跳转信息
type jumper struct {
minDistance float32
token *Token
}
// Dictionary 返回分词器使用的词典
func (seg *Segmenter) Dictionary() *Dictionary {
return seg.dict
}
// getCurrentFilePath get current file path
func getCurrentFilePath() string {
_, filePath, _, _ := runtime.Caller(1)
return filePath
}
// Read read the dict flie
func (seg *Segmenter) Read(file string) error {
log.Printf("Load the gse dictionary: \"%s\" ", file)
dictFile, err := os.Open(file)
if err != nil {
log.Printf("Could not load dictionaries: \"%s\", %v \n", file, err)
return err
}
defer dictFile.Close()
reader := bufio.NewReader(dictFile)
var (
text string
freqText string
frequency int
pos string
)
// 逐行读入分词
line := 0
for {
line++
size, fsErr := fmt.Fscanln(reader, &text, &freqText, &pos)
if fsErr != nil {
if fsErr == io.EOF {
// End of file
break
}
if size > 0 {
log.Printf("File '%v' line \"%v\" read error: %v, skip",
file, line, fsErr.Error())
} else {
log.Printf("File '%v' line \"%v\" is empty, read error: %v, skip",
file, line, fsErr.Error())
}
}
if size == 0 {
// 文件结束或错误行
// break
continue
} else if size < 2 {
// 无效行
continue
} else if size == 2 {
// 没有词性标注时设为空字符串
pos = ""
}
// 解析词频
var err error
frequency, err = strconv.Atoi(freqText)
if err != nil {
continue
}
// 过滤频率太小的词
if frequency < minTokenFrequency {
continue
}
// 过滤, 降低词频
if len([]rune(text)) < 2 {
// continue
frequency = 2
}
// 将分词添加到字典中
words := splitTextToWords([]byte(text))
token := Token{text: words, frequency: frequency, pos: pos}
seg.dict.addToken(token)
}
return nil
}
// DictPaths get the dict's paths
func DictPaths(dictDir, filePath string) (files []string) {
var dictPath string
if filePath == "en" {
return
}
if filePath == "zh" {
dictPath = path.Join(dictDir, "dict/dictionary.txt")
files = []string{dictPath}
return
}
if filePath == "jp" {
dictPath = path.Join(dictDir, "dict/jp/dict.txt")
files = []string{dictPath}
return
}
// if strings.Contains(filePath, ",") {
fileName := strings.Split(filePath, ",")
for i := 0; i < len(fileName); i++ {
if fileName[i] == "jp" {
dictPath = path.Join(dictDir, "dict/jp/dict.txt")
}
if fileName[i] == "zh" {
dictPath = path.Join(dictDir, "dict/dictionary.txt")
}
// if str[i] == "ti" {
// }
dictName := fileName[i] != "en" && fileName[i] != "zh" &&
fileName[i] != "jp" && fileName[i] != "ti"
if dictName {
dictPath = fileName[i]
}
if dictPath != "" {
files = append(files, dictPath)
}
}
// }
log.Println("Dict files path: ", files)
return
}
// IsJp is jp char return true
func IsJp(segText string) bool {
for _, r := range segText {
jp := unicode.Is(unicode.Scripts["Hiragana"], r) ||
unicode.Is(unicode.Scripts["Katakana"], r)
if jp {
return true
}
}
return false
}
// SegToken add segmenter token
func (seg *Segmenter) SegToken() {
// 计算每个分词的路径值，路径值含义见 Token 结构体的注释
logTotalFrequency := float32(math.Log2(float64(seg.dict.totalFrequency)))
for i := range seg.dict.tokens {
token := &seg.dict.tokens[i]
token.distance = logTotalFrequency - float32(math.Log2(float64(token.frequency)))
}
// 对每个分词进行细致划分，用于搜索引擎模式，
// 该模式用法见 Token 结构体的注释。
for i := range seg.dict.tokens {
token := &seg.dict.tokens[i]
segments := seg.segmentWords(token.text, true)
// 计算需要添加的子分词数目
numTokensToAdd := 0
for iToken := 0; iToken < len(segments); iToken++ {
// if len(segments[iToken].token.text) > 1 {
// 略去字元长度为一的分词
// TODO: 这值得进一步推敲，特别是当字典中有英文复合词的时候
if len(segments[iToken].token.text) > 0 {
hasJp := false
if len(segments[iToken].token.text) == 1 {
segText := string(segments[iToken].token.text[0])
hasJp = IsJp(segText)
}
if !hasJp {
numTokensToAdd++
}
}
}
token.segments = make([]*Segment, numTokensToAdd)
// 添加子分词
iSegmentsToAdd := 0
for iToken := 0; iToken < len(segments); iToken++ {
// if len(segments[iToken].token.text) > 1 {
if len(segments[iToken].token.text) > 0 {
hasJp := false
if len(segments[iToken].token.text) == 1 {
segText := string(segments[iToken].token.text[0])
hasJp = IsJp(segText)
}
if !hasJp {
token.segments[iSegmentsToAdd] = &segments[iToken]
iSegmentsToAdd++
}
}
}
}
}
// LoadDict load the dictionary from the file
//
// The format of the dictionary is (one for each participle):
// participle text, frequency, part of speech
//
// Can load multiple dictionary files, the file name separated by ","
// the front of the dictionary preferentially load the participle,
// such as: "user_dictionary.txt,common_dictionary.txt"
// When a participle appears both in the user dictionary and
// in the `common dictionary`, the `user dictionary` is given priority.
//
// 从文件中载入词典
//
// 可以载入多个词典文件，文件名用 "," 分隔，排在前面的词典优先载入分词，比如:
// "用户词典.txt,通用词典.txt"
// 当一个分词既出现在用户词典也出现在 `通用词典` 中，则优先使用 `用户词典`。
//
// 词典的格式为（每个分词一行）：
// 分词文本 频率 词性
func (seg *Segmenter) LoadDict(files ...string) error {
seg.dict = NewDict()
var (
dictDir = path.Join(path.Dir(getCurrentFilePath()), "data")
dictPath string
// load bool
)
if len(files) > 0 {
dictFiles := DictPaths(dictDir, files[0])
if len(dictFiles) > 0 {
// load = true
// files = dictFiles
for i := 0; i < len(dictFiles); i++ {
err := seg.Read(dictFiles[i])
if err != nil {
return err
}
}
}
}
if len(files) == 0 {
dictPath = path.Join(dictDir, "dict/dictionary.txt")
// files = []string{dictPath}
err := seg.Read(dictPath)
if err != nil {
return err
}
}
// if files[0] != "" && files[0] != "en" && !load {
// for _, file := range strings.Split(files[0], ",") {
// // for _, file := range files {
// err := seg.Read(file)
// if err != nil {
// return err
// }
// }
// }
seg.SegToken()
log.Println("Gse dictionary loaded finished.")
return nil
}
// Segment 对文本分词
//
// 输入参数：
// bytes UTF8 文本的字节数组
//
// 输出：
// []Segment 划分的分词
func (seg *Segmenter) Segment(bytes []byte) []Segment {
return seg.internalSegment(bytes, false)
}
// ModeSegment segment using search mode if searchMode is true
func (seg *Segmenter) ModeSegment(bytes []byte, searchMode ...bool) []Segment {
var mode bool
if len(searchMode) > 0 {
mode = searchMode[0]
}
return seg.internalSegment(bytes, mode)
}
// Slice use modeSegment segment retrun []string
// using search mode if searchMode is true
func (seg *Segmenter) Slice(bytes []byte, searchMode ...bool) []string {
segs := seg.ModeSegment(bytes, searchMode...)
return ToSlice(segs, searchMode...)
}
// Slice use modeSegment segment retrun string
// using search mode if searchMode is true
func (seg *Segmenter) String(bytes []byte, searchMode ...bool) string {
segs := seg.ModeSegment(bytes, searchMode...)
return ToString(segs, searchMode...)
}
func (seg *Segmenter) internalSegment(bytes []byte, searchMode bool) []Segment {
// 处理特殊情况
if len(bytes) == 0 {
// return []Segment{}
return nil
}
// 划分字元
text := splitTextToWords(bytes)
return seg.segmentWords(text, searchMode)
}
func (seg *Segmenter) segmentWords(text []Text, searchMode bool) []Segment {
// 搜索模式下该分词已无继续划分可能的情况
if searchMode && len(text) == 1 {
return nil
}
// jumpers 定义了每个字元处的向前跳转信息，
// 包括这个跳转对应的分词，
// 以及从文本段开始到该字元的最短路径值
jumpers := make([]jumper, len(text))
if seg.dict == nil {
return nil
}
tokens := make([]*Token, seg.dict.maxTokenLen)
for current := 0; current < len(text); current++ {
// 找到前一个字元处的最短路径，以便计算后续路径值
var baseDistance float32
if current == 0 {
// 当本字元在文本首部时，基础距离应该是零
baseDistance = 0
} else {
baseDistance = jumpers[current-1].minDistance
}
// 寻找所有以当前字元开头的分词
numTokens := seg.dict.lookupTokens(
text[current:minInt(current+seg.dict.maxTokenLen, len(text))], tokens)
// 对所有可能的分词，更新分词结束字元处的跳转信息
for iToken := 0; iToken < numTokens; iToken++ {
location := current + len(tokens[iToken].text) - 1
if !searchMode || current != 0 || location != len(text)-1 {
updateJumper(&jumpers[location], baseDistance, tokens[iToken])
}
}
// 当前字元没有对应分词时补加一个伪分词
if numTokens == 0 || len(tokens[0].text) > 1 {
updateJumper(&jumpers[current], baseDistance,
&Token{text: []Text{text[current]}, frequency: 1, distance: 32, pos: "x"})
}
}
// 从后向前扫描第一遍得到需要添加的分词数目
numSeg := 0
for index := len(text) - 1; index >= 0; {
location := index - len(jumpers[index].token.text) + 1
numSeg++
index = location - 1
}
// 从后向前扫描第二遍添加分词到最终结果
outputSegments := make([]Segment, numSeg)
for index := len(text) - 1; index >= 0; {
location := index - len(jumpers[index].token.text) + 1
numSeg--
outputSegments[numSeg].token = jumpers[index].token
index = location - 1
}
// 计算各个分词的字节位置
bytePosition := 0
for iSeg := 0; iSeg < len(outputSegments); iSeg++ {
outputSegments[iSeg].start = bytePosition
bytePosition += textSliceByteLen(outputSegments[iSeg].token.text)
outputSegments[iSeg].end = bytePosition
}
return outputSegments
}
// updateJumper 更新跳转信息:
// 1. 当该位置从未被访问过时 (jumper.minDistance 为零的情况)，或者
// 2. 当该位置的当前最短路径大于新的最短路径时
// 将当前位置的最短路径值更新为 baseDistance 加上新分词的概率
func updateJumper(jumper *jumper, baseDistance float32, token *Token) {
newDistance := baseDistance + token.distance
if jumper.minDistance == 0 || jumper.minDistance > newDistance {
jumper.minDistance = newDistance
jumper.token = token
}
}
// minInt 取两整数较小值
func minInt(a, b int) int {
if a > b {
return b
}
return a
}
// maxInt 取两整数较大值
func maxInt(a, b int) int {
if a > b {
return a
}
return b
}
// splitTextToWords 将文本划分成字元
func splitTextToWords(text Text) []Text {
output := make([]Text, 0, len(text)/3)
current := 0
inAlphanumeric := true
alphanumericStart := 0
for current < len(text) {
r, size := utf8.DecodeRune(text[current:])
if size <= 2 && (unicode.IsLetter(r) || unicode.IsNumber(r)) {
// 当前是拉丁字母或数字（非中日韩文字）
if !inAlphanumeric {
alphanumericStart = current
inAlphanumeric = true
}
} else {
if inAlphanumeric {
inAlphanumeric = false
if current != 0 {
output = append(output, toLower(text[alphanumericStart:current]))
}
}
output = append(output, text[current:current+size])
}
current += size
}
// 处理最后一个字元是英文的情况
if inAlphanumeric {
if current != 0 {
output = append(output, toLower(text[alphanumericStart:current]))
}
}
return output
}
// toLower 将英文词转化为小写
func toLower(text []byte) []byte {
output := make([]byte, len(text))
for i, t := range text {
if t >= 'A' && t <= 'Z' {
output[i] = t - 'A' + 'a'
} else {
output[i] = t
}
}
return output
}

38
vendor/github.com/go-ego/gse/test_utils.go generated vendored Normal file
View File

@@ -0,0 +1,38 @@
package gse
import (
"fmt"
"testing"
)
func printTokens(tokens []*Token, numTokens int) (output string) {
for iToken := 0; iToken < numTokens; iToken++ {
for _, word := range tokens[iToken].text {
output += fmt.Sprint(string(word))
}
output += " "
}
return
}
func toWords(strings ...string) []Text {
words := []Text{}
for _, s := range strings {
words = append(words, []byte(s))
}
return words
}
func bytesToString(bytes []Text) (output string) {
for _, b := range bytes {
output += (string(b) + "/")
}
return
}
func expect(t *testing.T, expect string, actual interface{}) {
actualString := fmt.Sprint(actual)
if expect != actualString {
t.Errorf("期待值=\"%s\", 实际=\"%s\"", expect, actualString)
}
}

75
vendor/github.com/go-ego/gse/token.go generated vendored Normal file
View File

@@ -0,0 +1,75 @@
package gse
// Text 字串类型，可以用来表达
// 1. 一个字元，比如 "中" 又如 "国", 英文的一个字元是一个词
// 2. 一个分词，比如 "中国" 又如 "人口"
// 3. 一段文字，比如 "中国有十三亿人口"
type Text []byte
// Token 一个分词
type Token struct {
// 分词的字串，这实际上是个字元数组
text []Text
// 分词在语料库中的词频
frequency int
// log2(总词频/该分词词频)，这相当于 log2(1/p(分词))，用作动态规划中
// 该分词的路径长度。求解 prod(p(分词)) 的最大值相当于求解
// sum(distance(分词)) 的最小值，这就是“最短路径”的来历。
distance float32
// 词性标注
pos string
// 该分词文本的进一步分词划分，见 Segments 函数注释。
segments []*Segment
}
// Text 返回分词文本
func (token *Token) Text() string {
return textSliceToString(token.text)
}
// Frequency 返回分词在语料库中的词频
func (token *Token) Frequency() int {
return token.frequency
}
// Pos 返回分词词性标注
func (token *Token) Pos() string {
return token.pos
}
// Segments 该分词文本的进一步分词划分，比如 "中华人民共和国中央人民政府" 这个分词
// 有两个子分词 "中华人民共和国 " 和 "中央人民政府"。子分词也可以进一步有子分词
// 形成一个树结构，遍历这个树就可以得到该分词的所有细致分词划分，这主要
// 用于搜索引擎对一段文本进行全文搜索。
func (token *Token) Segments() []*Segment {
return token.segments
}
// Equals compare str split tokens
func (token *Token) Equals(str string) bool {
tokenLen := 0
for _, t := range token.text {
tokenLen += len(t)
}
if tokenLen != len(str) {
return false
}
bytStr := []byte(str)
index := 0
for i := 0; i < len(token.text); i++ {
textArray := []byte(token.text[i])
for j := 0; j < len(textArray); j++ {
if textArray[j] != bytStr[index] {
index = index + 1
return false
}
index = index + 1
}
}
return true
}

168
vendor/github.com/go-ego/gse/utils.go generated vendored Normal file
View File

@@ -0,0 +1,168 @@
package gse
import (
"bytes"
"fmt"
)
// ToString segments to string 输出分词结果为字符串
//
// 有两种输出模式，以 "中华人民共和国" 为例
//
// 普通模式（searchMode=false）输出一个分词 "中华人民共和国/ns "
// 搜索模式（searchMode=true） 输出普通模式的再细致切分：
// "中华/nz 人民/n 共和/nz 国/n 共和国/ns 人民共和国/nt 中华人民共和国/ns "
//
// 默认 searchMode=false
// 搜索模式主要用于给搜索引擎提供尽可能多的关键字，详情请见 Token 结构体的注释。
func ToString(segs []Segment, searchMode ...bool) (output string) {
var mode bool
if len(searchMode) > 0 {
mode = searchMode[0]
}
if mode {
for _, seg := range segs {
output += tokenToString(seg.token)
}
return
}
for _, seg := range segs {
output += fmt.Sprintf("%s/%s ",
textSliceToString(seg.token.text), seg.token.pos)
}
return
}
func tokenToBytes(token *Token) (output []byte) {
for _, s := range token.segments {
output = append(output, tokenToBytes(s.token)...)
}
output = append(output, []byte(fmt.Sprintf("%s/%s ",
textSliceToString(token.text), token.pos))...)
return
}
func tokenToString(token *Token) (output string) {
hasOnlyTerminalToken := true
for _, s := range token.segments {
if len(s.token.segments) > 1 || IsJp(string(s.token.text[0])) {
hasOnlyTerminalToken = false
}
if !hasOnlyTerminalToken {
if s != nil {
output += tokenToString(s.token)
}
}
}
output += fmt.Sprintf("%s/%s ", textSliceToString(token.text), token.pos)
return
}
// ToSlice segments to slice 输出分词结果到一个字符串 slice
//
// 有两种输出模式，以 "中华人民共和国" 为例
//
// 普通模式（searchMode=false）输出一个分词"[中华人民共和国]"
// 搜索模式（searchMode=true） 输出普通模式的再细致切分：
// "[中华 人民 共和 国 共和国 人民共和国 中华人民共和国]"
//
// 默认 searchMode=false
// 搜索模式主要用于给搜索引擎提供尽可能多的关键字，详情请见Token结构体的注释。
func ToSlice(segs []Segment, searchMode ...bool) (output []string) {
var mode bool
if len(searchMode) > 0 {
mode = searchMode[0]
}
if mode {
for _, seg := range segs {
output = append(output, tokenToSlice(seg.token)...)
}
return
}
for _, seg := range segs {
output = append(output, seg.token.Text())
}
return
}
func tokenToSlice(token *Token) (output []string) {
hasOnlyTerminalToken := true
for _, s := range token.segments {
if len(s.token.segments) > 1 || IsJp(string(s.token.text[0])) {
hasOnlyTerminalToken = false
}
if !hasOnlyTerminalToken {
output = append(output, tokenToSlice(s.token)...)
}
}
output = append(output, textSliceToString(token.text))
return output
}
// 将多个字元拼接一个字符串输出
func textToString(text []Text) string {
var output string
for _, word := range text {
output += string(word)
}
return output
}
// 将多个字元拼接一个字符串输出
func textSliceToString(text []Text) string {
return Join(text)
}
// 返回多个字元的字节总长度
func textSliceByteLen(text []Text) (length int) {
for _, word := range text {
length += len(word)
}
return
}
func textSliceToBytes(text []Text) []byte {
var buf bytes.Buffer
for _, word := range text {
buf.Write(word)
}
return buf.Bytes()
}
// Join is better string splicing
func Join(text []Text) string {
switch len(text) {
case 0:
return ""
case 1:
return string(text[0])
case 2:
// Special case for common small values.
// Remove if github.com/golang/go/issues/6714 is fixed
return string(text[0]) + string(text[1])
case 3:
// Special case for common small values.
// Remove if #6714 is fixed
return string(text[0]) + string(text[1]) + string(text[2])
}
n := 0
for i := 0; i < len(text); i++ {
n += len(text[i])
}
b := make([]byte, n)
bp := copy(b, text[0])
for _, str := range text[1:] {
bp += copy(b[bp:], str)
}
return string(b)
}