Breakthrough advances in large-scale language models (LLMs) are causing a technological “transition phase” expected to bring major social change. Supported by the high performance and versatility of LLM models, this transformation is spreading across all industries, and it is expected to become an essential foundation for a diverse range of scientific and technological research.
To truly encourage healthy trial and error in this progress and to stimulate innovation, it is necessary to address the following challenges:
Technical issues related to LLMs： The learning principles of mathematical reasoning (how emergence and generality are learned), efficiency (data/model efficiency, sustainability)
Social issues related to LLMs： Explainability/interpretability (black box issue), fairness (bias issue), safety (misinformation/hallucination, personal information, copyright issues, compliance), reliability (how to ensure it)
Multidisciplinary expansion of LLM: Expansion into medical, legal, education, etc. Combination with multimodal information, robot control, etc.
To achieve this, it is necessary to continually build models that are completely open source and commercially available to advance research and development to solve those problems. The clear control of coverage of information in Japanese, the usage rules, and the confidentiality of inputs are essential requirements from an economic security perspective.
Currently, research and development of LLMs is limited to some private organizations overseas, and the lack of an open research and development environment is a major problem. In addition, large-scale investments in GPU clusters are being made globally at universities (e.g. Hong Kong University of Science and Technology) and research institutes in various fields (e.g. NIH), but Japan is far behind…
Futures policies
Although LLM research and development is progressing rapidly every day, the current situation is that the emergence of large models such as GPT-3 (175B) may occur and later smaller models will be released through knowledge distillation. It’s necessary to build models of this scale and establish a framework to work on the elucidation of principles in Japan as well.
Considering the speed of progress in LLM research, the top-down approach of R&D programs might not be necessarily optimal. The most important part is to create the foundation that will prepare the computational infrastructure, to construct the language model architecture (including human resources such as engineers), and to create an environment in which domestic and international researchers can conduct various trials and errors.
