a hidden state is worth a thousand words.

I am Bokai Xu, I received bachelor degree at CUHK Shenzhen. After that I work as an engineer at Modelbest Inc. (OpenBMB).

My passion is understanding the connection between deep neural networks and biological brain.

At Modelbest Inc. (Jan 2024 -), I mainly worked on multimodal language models and information retrieval.

• contributed to MiniCPM-V-2.6, in OCR capability and multi-image understanding capability.

• contributed to multimodal information retrieval VisRAG & VisRAG-Ret, in training and evaluation of VLM-based dense retrieval models, training data synthesis, and data filtering. VisRAG enables VLM to read and comprehend with vision and retrieve multimodal information, in a way like human.

At THUNLP@THU (Jun 2022 - Jan 2024), I worked on various topics with large language models.

• contributed to language model alignment UltraChat: Enhancing Chat Language Models by Scaling High-quality Instructional Conversations, the dataset were widely used by community, including Falcon, Zephyr and others.

• contributed to Tool Learning with Foundation Models & BMTools for LLM to use tools, we thoroughly investigated the LLM’s ability to sequentially use tools for completing various tasks.

At ICBI@SIAT (Apr 2023 - ), I worked on brain reconstruction infrastructures. I work closely with Dr.Chaoyu Yang and Prof.Pengcheng Zhou.

• led foundation model pretraining Masked Autoencoders with large-scale microscopic images then aligned pretrained model for trustful brain reconstruction task: Brain Reconstruction by Self Supervised Semantic Stitching of Non-overlapping 3D Microscopic Image .