Java UAX29URLEmailTokenizer类代码示例

OGeek|极客世界-中国程序员成长平台 › 门户 › 编程› Java›Java编程经验

原作者: [db:作者] 来自: [db:来源] 收藏邀请

本文整理汇总了Java中org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer类的典型用法代码示例。如果您正苦于以下问题：Java UAX29URLEmailTokenizer类的具体用法？Java UAX29URLEmailTokenizer怎么用？Java UAX29URLEmailTokenizer使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。

UAX29URLEmailTokenizer类属于org.apache.lucene.analysis.standard包，在下文中一共展示了UAX29URLEmailTokenizer类的20个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: testCombiningMarksBackwards

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
/** @deprecated remove this and sophisticated backwards layer in 5.0 */
@Deprecated
public void testCombiningMarksBackwards() throws Exception {
  Analyzer a = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents
      (String fieldName, Reader reader) {

      Tokenizer tokenizer = new UAX29URLEmailTokenizer(Version.LUCENE_3_1, reader);
      return new TokenStreamComponents(tokenizer);
    }
  };
  checkOneTerm(a, "ざ", "さ"); // hiragana Bug
  checkOneTerm(a, "ザ", "ザ"); // katakana Works
  checkOneTerm(a, "壹゙", "壹"); // ideographic Bug
  checkOneTerm(a, "아゙",  "아゙"); // hangul Works
}

开发者ID:europeana，项目名称:search，代码行数:18，代码来源:TestUAX29URLEmailTokenizer.java

示例2: testCombiningMarksBackwards

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
/** @deprecated remove this and sophisticated backwards layer in 5.0 */
@Deprecated
public void testCombiningMarksBackwards() throws Exception {
  Analyzer a = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents
      (String fieldName, Reader reader) {

      Tokenizer tokenizer = new UAX29URLEmailTokenizer(Version.LUCENE_31, reader);
      return new TokenStreamComponents(tokenizer);
    }
  };
  checkOneTerm(a, "ざ", "さ"); // hiragana Bug
  checkOneTerm(a, "ザ", "ザ"); // katakana Works
  checkOneTerm(a, "壹゙", "壹"); // ideographic Bug
  checkOneTerm(a, "아゙",  "아゙"); // hangul Works
}

开发者ID:pkarmstr，项目名称:NYBC，代码行数:18，代码来源:TestUAX29URLEmailTokenizer.java

示例3: testLongEMAILatomText

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
public void testLongEMAILatomText() throws Exception {
  // EMAILatomText = [A-Za-z0-9!#$%&'*+-/=?\^_`{|}~]
  char[] emailAtomChars
      = "!#$%&'*+,-./0123456789=?ABCDEFGHIJKLMNOPQRSTUVWXYZ^_`abcdefghijklmnopqrstuvwxyz{|}~".toCharArray();
  StringBuilder builder = new StringBuilder();
  int numChars = TestUtil.nextInt(random(), 100 * 1024, 3 * 1024 * 1024);
  for (int i = 0 ; i < numChars ; ++i) {
    builder.append(emailAtomChars[random().nextInt(emailAtomChars.length)]);
  }
  int tokenCount = 0;
  String text = builder.toString();
  UAX29URLEmailTokenizer ts = new UAX29URLEmailTokenizer(new StringReader(text));
  ts.reset();
  while (ts.incrementToken()) {
    tokenCount++;
  }
  ts.end();
  ts.close();
  assertTrue(tokenCount > 0);

  tokenCount = 0;
  int newBufferSize = TestUtil.nextInt(random(), 200, 8192);
  ts.setMaxTokenLength(newBufferSize);
  ts.setReader(new StringReader(text));
  ts.reset();
  while (ts.incrementToken()) {
    tokenCount++;
  }
  ts.end();
  ts.close();
  assertTrue(tokenCount > 0);
}

开发者ID:europeana，项目名称:search，代码行数:33，代码来源:TestUAX29URLEmailTokenizer.java

示例4: testHugeDoc

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
public void testHugeDoc() throws IOException {
  StringBuilder sb = new StringBuilder();
  char whitespace[] = new char[4094];
  Arrays.fill(whitespace, ' ');
  sb.append(whitespace);
  sb.append("testing 1234");
  String input = sb.toString();
  UAX29URLEmailTokenizer tokenizer = new UAX29URLEmailTokenizer(newAttributeFactory(), new StringReader(input));
  BaseTokenStreamTestCase.assertTokenStreamContents(tokenizer, new String[] { "testing", "1234" });
}

开发者ID:europeana，项目名称:search，代码行数:11，代码来源:TestUAX29URLEmailTokenizer.java

示例5: UAX29URLEmailTokenizer

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
@Override
protected TokenStreamComponents createComponents
  (String fieldName, Reader reader) {

  Tokenizer tokenizer = new UAX29URLEmailTokenizer(newAttributeFactory(), reader);
  return new TokenStreamComponents(tokenizer);
}

开发者ID:europeana，项目名称:search，代码行数:8，代码来源:TestUAX29URLEmailTokenizer.java

示例6: incrementToken

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
@Override
public final boolean incrementToken() throws java.io.IOException {
  boolean isTokenAvailable = false;
  while (input.incrementToken()) {
    if (typeAtt.type() == UAX29URLEmailTokenizer.TOKEN_TYPES[UAX29URLEmailTokenizer.URL]) {
      isTokenAvailable = true;
      break;
    }
  }
  return isTokenAvailable;
}

开发者ID:europeana，项目名称:search，代码行数:12，代码来源:TestUAX29URLEmailTokenizer.java

示例7: createComponents

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
@Override
protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
  UAX29URLEmailTokenizer tokenizer = new UAX29URLEmailTokenizer(newAttributeFactory(), reader);
  tokenizer.setMaxTokenLength(Integer.MAX_VALUE);  // Tokenize arbitrary length URLs
  TokenFilter filter = new URLFilter(tokenizer);
  return new TokenStreamComponents(tokenizer, filter);
}

开发者ID:europeana，项目名称:search，代码行数:8，代码来源:TestUAX29URLEmailTokenizer.java

示例8: testMailtoBackwards

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
/** @deprecated remove this and sophisticated backwards layer in 5.0 */
@Deprecated
public void testMailtoBackwards()  throws Exception {
  Analyzer a = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
      Tokenizer tokenizer = new UAX29URLEmailTokenizer(Version.LUCENE_3_4, reader);
      return new TokenStreamComponents(tokenizer);
    }
  };
  assertAnalyzesTo(a, "mailto:[email protected]",
      new String[] { "mailto:test", "example.org" });
}

开发者ID:europeana，项目名称:search，代码行数:14，代码来源:TestUAX29URLEmailTokenizer.java

示例9: testVersion36

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
/** @deprecated uses older unicode (6.0). simple test to make sure its basically working */
@Deprecated
public void testVersion36() throws Exception {
  Analyzer a = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
      Tokenizer tokenizer = new UAX29URLEmailTokenizer(Version.LUCENE_3_6, reader);
      return new TokenStreamComponents(tokenizer);
    }
  };
  assertAnalyzesTo(a, "this is just a t\u08E6st [email protected]", // new combining mark in 6.1
      new String[] { "this", "is", "just", "a", "t", "st", "[email protected]" });
}

开发者ID:europeana，项目名称:search，代码行数:14，代码来源:TestUAX29URLEmailTokenizer.java

示例10: testVersion40

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
/** @deprecated uses older unicode (6.1). simple test to make sure its basically working */
@Deprecated
public void testVersion40() throws Exception {
  Analyzer a = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
      Tokenizer tokenizer = new UAX29URLEmailTokenizer(Version.LUCENE_4_0, reader);
      return new TokenStreamComponents(tokenizer);
    }
  };
  // U+061C is a new combining mark in 6.3, found using "[[\p{WB:Format}\p{WB:Extend}]&[^\p{Age:6.2}]]"
  // on the online UnicodeSet utility: <http://unicode.org/cldr/utility/list-unicodeset.jsp>
  assertAnalyzesTo(a, "this is just a t\u061Cst [email protected]",
      new String[] { "this", "is", "just", "a", "t", "st", "[email protected]" });
}

开发者ID:europeana，项目名称:search，代码行数:16，代码来源:TestUAX29URLEmailTokenizer.java

示例11: testHugeDoc

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
public void testHugeDoc() throws IOException {
  StringBuilder sb = new StringBuilder();
  char whitespace[] = new char[4094];
  Arrays.fill(whitespace, ' ');
  sb.append(whitespace);
  sb.append("testing 1234");
  String input = sb.toString();
  UAX29URLEmailTokenizer tokenizer = new UAX29URLEmailTokenizer(TEST_VERSION_CURRENT, new StringReader(input));
  BaseTokenStreamTestCase.assertTokenStreamContents(tokenizer, new String[] { "testing", "1234" });
}

开发者ID:pkarmstr，项目名称:NYBC，代码行数:11，代码来源:TestUAX29URLEmailTokenizer.java

示例12: UAX29URLEmailTokenizer

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
@Override
protected TokenStreamComponents createComponents
  (String fieldName, Reader reader) {

  Tokenizer tokenizer = new UAX29URLEmailTokenizer(TEST_VERSION_CURRENT, reader);
  return new TokenStreamComponents(tokenizer);
}

开发者ID:pkarmstr，项目名称:NYBC，代码行数:8，代码来源:TestUAX29URLEmailTokenizer.java

示例13: createComponents

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
@Override
protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
  UAX29URLEmailTokenizer tokenizer = new UAX29URLEmailTokenizer(TEST_VERSION_CURRENT, reader);
  tokenizer.setMaxTokenLength(Integer.MAX_VALUE);  // Tokenize arbitrary length URLs
  TokenFilter filter = new URLFilter(tokenizer);
  return new TokenStreamComponents(tokenizer, filter);
}

开发者ID:pkarmstr，项目名称:NYBC，代码行数:8，代码来源:TestUAX29URLEmailTokenizer.java

示例14: testMailtoBackwards

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
/** @deprecated remove this and sophisticated backwards layer in 5.0 */
@Deprecated
public void testMailtoBackwards()  throws Exception {
  Analyzer a = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
      Tokenizer tokenizer = new UAX29URLEmailTokenizer(Version.LUCENE_34, reader);
      return new TokenStreamComponents(tokenizer);
    }
  };
  assertAnalyzesTo(a, "mailto:[email protected]",
      new String[] { "mailto:test", "example.org" });
}

开发者ID:pkarmstr，项目名称:NYBC，代码行数:14，代码来源:TestUAX29URLEmailTokenizer.java

示例15: testVersion36

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
/** @deprecated uses older unicode (6.0). simple test to make sure its basically working */
@Deprecated
public void testVersion36() throws Exception {
  Analyzer a = new Analyzer() {
    @Override
    protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
      Tokenizer tokenizer = new UAX29URLEmailTokenizer(Version.LUCENE_36, reader);
      return new TokenStreamComponents(tokenizer);
    }
  };
  assertAnalyzesTo(a, "this is just a t\u08E6st [email protected]", // new combining mark in 6.1
      new String[] { "this", "is", "just", "a", "t", "st", "[email protected]" });
}

开发者ID:pkarmstr，项目名称:NYBC，代码行数:14，代码来源:TestUAX29URLEmailTokenizer.java

示例16: testLemmagenFilterFactoryWithDefaultLexicon

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
public void testLemmagenFilterFactoryWithDefaultLexicon() throws IOException {
    ESTestCase.TestAnalysis analysis = createAnalysis();

    TokenFilterFactory tokenFilter = analysis.tokenFilter.get("lemmagen_default_filter");
    assertThat(tokenFilter, instanceOf(LemmagenFilterFactory.class));

    String source = "I was late.";
    String[] expected = new String[]{"I", "be", "late"};

    Tokenizer tokenizer = new UAX29URLEmailTokenizer();
    tokenizer.setReader(new StringReader(source));

    assertTokenStreamContents(tokenFilter.create(tokenizer), expected);
}

开发者ID:vhyza，项目名称:elasticsearch-analysis-lemmagen，代码行数:15，代码来源:LemmagenAnalysisTest.java

示例17: testLemmagenFilterFactoryWithCustomLexicon

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
public void testLemmagenFilterFactoryWithCustomLexicon() throws IOException {
    ESTestCase.TestAnalysis analysis = createAnalysis();

    TokenFilterFactory tokenFilter = analysis.tokenFilter.get("lemmagen_cs_filter");
    assertThat(tokenFilter, instanceOf(LemmagenFilterFactory.class));

    String source = "Děkuji, že jsi přišel.";
    String[] expected = {"Děkovat", "že", "být", "přijít"};

    Tokenizer tokenizer = new UAX29URLEmailTokenizer();
    tokenizer.setReader(new StringReader(source));

    assertTokenStreamContents(tokenFilter.create(tokenizer), expected);
}

开发者ID:vhyza，项目名称:elasticsearch-analysis-lemmagen，代码行数:15，代码来源:LemmagenAnalysisTest.java

示例18: testLemmagenFilterFactoryWithShortLexiconCode

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
public void testLemmagenFilterFactoryWithShortLexiconCode() throws IOException {
    ESTestCase.TestAnalysis analysis = createAnalysis();

    TokenFilterFactory tokenFilter = analysis.tokenFilter.get("lemmagen_fr_filter");
    assertThat(tokenFilter, instanceOf(LemmagenFilterFactory.class));

    String source = "Il faut encore ajouter une pincée de sel.";
    String[] expected = new String[]{"Il", "falloir", "encore", "ajouter", "un", "pincer", "de", "sel"};

    Tokenizer tokenizer = new UAX29URLEmailTokenizer();
    tokenizer.setReader(new StringReader(source));

    assertTokenStreamContents(tokenFilter.create(tokenizer), expected);
}

开发者ID:vhyza，项目名称:elasticsearch-analysis-lemmagen，代码行数:15，代码来源:LemmagenAnalysisTest.java

示例19: testLemmagenFilterFactoryWithPath

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
public void testLemmagenFilterFactoryWithPath() throws IOException {
    ESTestCase.TestAnalysis analysis = createAnalysis();

    TokenFilterFactory tokenFilter = analysis.tokenFilter.get("lemmagen_cs_path_filter");
    assertThat(tokenFilter, instanceOf(LemmagenFilterFactory.class));

    String source = "Děkuji, že jsi přišel.";
    String[] expected = {"Děkovat", "že", "být", "přijít"};

    Tokenizer tokenizer = new UAX29URLEmailTokenizer();
    tokenizer.setReader(new StringReader(source));

    assertTokenStreamContents(tokenFilter.create(tokenizer), expected);
}

开发者ID:vhyza，项目名称:elasticsearch-analysis-lemmagen，代码行数:15，代码来源:LemmagenAnalysisTest.java

示例20: create

import org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer; //导入依赖的package包/类
@Override
public Tokenizer create() {
    UAX29URLEmailTokenizer tokenizer = new UAX29URLEmailTokenizer();
    tokenizer.setMaxTokenLength(maxTokenLength);
    return tokenizer;
}

开发者ID:justor，项目名称:elasticsearch_my，代码行数:7，代码来源:UAX29URLEmailTokenizerFactory.java

注：本文中的org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer类示例整理自Github/MSDocs等源码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。

鲜花

握手

雷人

路过

鸡蛋

该文章已有0人参与评论

请发表评论

全部评论

专题导读

More+

10-27 六六分期app的软件客服如何联系？(六六分期

11-06 可心卡盟:win10系统火狐flash插件崩溃怎么

11-06 亲亲特价:怎么删除回收站图标

11-06 济南大学虚拟社区:鲁大师节能降温的具体办

11-06 xlueops.exe:无线网络安装向导

11-06 女斗合众国:win7系统cf与主机连接不稳定怎

11-06 0xc000022-[cf烟雾头]cf怎么调烟雾头

11-06 qizideyouhuo:应用程序无法正常启动0xc0000

11-06 ipz-185:win7系统vcf文件怎么打开

11-06 傻哥蹦迪:win10系统s4怎么打开usb调试

11-06 八神浩树gtaste:回收站清空了怎么恢复

11-06 妖尾之黑色守护:win10系统电脑没有1440x900

11-06 校园至尊魔王小说:win7系统浏览网页时字体

11-06 女斗合众国:win10系统访问共享文件夹提示请

11-06 tokyo hot n0654:恢复win7系统默认字体一招

11-06 雨酷仙境:设置win7系统转移临时文件夹腾出

11-06 阿穆纳伊之杖:win7系统开始菜单在右边还原

11-06 tunespotting:win10系统火狐flash插件总是

11-06 甘尔葛分析师：计谋网站seo关键词暴涨有什

11-06 蔡贵霖: 计谋网站seo关键词暴涨有什么秘密

11-06 博益网首页:ao3网页版进入不了解决方法

11-06 漏斗子专栏: 网站数据分析小白易懂精华篇

11-06 见证双虹怎么做:win7系统开启telnet命令的

11-06 颾狐蝶蜋:系统资源不足无法完成请求的服务

11-06 国光中学校歌:提交网站到alexa查询详细步骤

11-06 西安有情天:静态网页和动态网页的区别

11-06 红木雅尚斋:外部链接构造对网站的好处

11-06 前官礼遇：防止域名劫持–增强域安全性的10

11-06 密传二转答案: 中文分词算法有哪些

11-06 金泉家园邮编:百度快照劫持的表现及应对方

Java PsiKeyword类代码示例发布时间：2022-05-22

Java BshInterpreter类代码示例发布时间：2022-05-22

剪的笔顺,诠释剪的笔画,认识剪的部首

1 六六分期app的软件客服如何联系？(六六分期

六六分期app的软件客服如何联系？不知道吗？加qq群【895510560】即可！标题：六六分期

阅读：18285|2023-10-27

2 可心卡盟:win10系统火狐flash插件崩溃怎么

今天小编告诉大家如何处理win10系统火狐flash插件总是崩溃的问题，可能很多用户都不知

阅读：9682|2022-11-06

3 亲亲特价:怎么删除回收站图标

今天小编告诉大家如何对win10系统删除桌面回收站图标进行设置，可能很多用户都不知道

阅读：8181|2022-11-06

4 济南大学虚拟社区:鲁大师节能降温的具体办

今天小编告诉大家如何对win10系统电脑设置节能降温的设置方法，想必大家都遇到过需要

阅读：8552|2022-11-06

5 xlueops.exe:无线网络安装向导

我们在使用xp系统的过程中,经常需要对xp系统无线网络安装向导设置进行设置，可能很多

阅读：8459|2022-11-06

6 女斗合众国:win7系统cf与主机连接不稳定怎

今天小编告诉大家如何处理win7系统玩cf老是与主机连接不稳定的问题，可能很多用户都不

阅读：9396|2022-11-06

7 0xc000022-[cf烟雾头]cf怎么调烟雾头

电脑对日常生活的重要性小编就不多说了，可是一旦碰到win7系统设置cf烟雾头的问题，很

阅读：8431|2022-11-06

8 qizideyouhuo:应用程序无法正常启动0xc0000

我们在日常使用电脑的时候，有的小伙伴们可能在打开应用的时候会遇见提示应用程序无法

阅读：7866|2022-11-06

9 ipz-185:win7系统vcf文件怎么打开

今天小编告诉大家如何对win7系统打开vcf文件进行设置，可能很多用户都不知道怎么对win

阅读：8416|2022-11-06

10 傻哥蹦迪:win10系统s4怎么打开usb调试

今天小编告诉大家如何对win10系统s4开启USB调试模式进行设置，可能很多用户都不知道怎

阅读：7394|2022-11-06

客服电话

电子邮件

Java UAX29URLEmailTokenizer类代码示例

示例1: testCombiningMarksBackwards

示例2: testCombiningMarksBackwards

示例3: testLongEMAILatomText

示例4: testHugeDoc

示例5: UAX29URLEmailTokenizer

示例6: incrementToken

示例7: createComponents

示例8: testMailtoBackwards

示例9: testVersion36

示例10: testVersion40

示例11: testHugeDoc

示例12: UAX29URLEmailTokenizer

示例13: createComponents

示例14: testMailtoBackwards

示例15: testVersion36

示例16: testLemmagenFilterFactoryWithDefaultLexicon

示例17: testLemmagenFilterFactoryWithCustomLexicon

示例18: testLemmagenFilterFactoryWithShortLexiconCode

示例19: testLemmagenFilterFactoryWithPath

示例20: create

请发表评论

全部评论

上一篇：

下一篇：

librespeed/speedtest: Self-hosted Speedt

avehtari/BDA_m_demos: Bayesian Data Anal

四维彩超怎么看性别？四维看男孩女孩诀窍

TUNER88/iOSSystemSoundsLibrary: List of

medfreeman/markdown-it-toc-and-anchor: m

剪的笔顺,诠释剪的笔画,认识剪的部首

六六分期app的软件客服如何联系？(六六分期

florent37/ViewAnimator: A fluent Android

florent37/Shrine-MaterialDesign2: implem

CVE-2020-36276

SimpleSoftwareIO/simple-sms: Send and re

关于我们

产品与服务

解决方案

139-2527-9053