Skip to content

Fix an issue where input tokens to the WangchanBERTa NER pipeline may be in the incorrect form  #32

@lalital

Description

@lalital

According to the model finetuning pipeline, the input tokens are tokenized with other tokenizer (e.g. PyThaiNLp's newmm for thainer dataset) and then retokenizer with SentencePiece tokenizer. However, the input tokens fed to the finetuned model is the tokens tokenized with SentencePiece only (not newmm first, and then followed by SentencePiece)

Proposed solution:

  1. Pretokenize with PyThaiNLP's newmm tokenizer
  2. Retokenize with the subword tokenizer (SentencePiece)
  3. Map the prediction results of the subword tokens to the tokens tokenized with newmm
  4. Return the prediction results in word-level and chunk-level

Branch name: feature/ner_pipeline

NER Pipeline Demo (via Colab): https://colab.research.google.com/drive/1-54NeM_wsjitaiSXfMBpcnqzbPMR0a9R#scrollTo=VzSGZbwWaiOI

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions