LLMs can see and hear without any training

30 Jan 2025 · Kumar Ashutosh, Yossi Gandelsman, Xinlei Chen, Ishan Misra, Rohit Girdhar ·

We present MILS: Multimodal Iterative LLM Solver, a surprisingly simple, training-free approach, to imbue multimodal capabilities into your favorite LLM. Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which are scored and fed back iteratively, eventually generating a solution to the task. This enables various applications that typically require training specialized models on task-specific data. In particular, we establish a new state-of-the-art on emergent zero-shot image, video and audio captioning. MILS seamlessly applies to media generation as well, discovering prompt rewrites to improve text-to-image generation, and even edit prompts for style transfer! Finally, being a gradient-free optimization approach, MILS can invert multimodal embeddings into text, enabling applications like cross-modal arithmetic.

Paper: https://arxiv.org/pdf/2501.18096v1.pdf

Code: https://github.com/facebookresearch/mils
✅
https://www.tg-me.com/deep_learning_proj

Awesome-LLM-as-a-judge Survey

▪ Github

🔸https://www.tg-me.com/deep_learning_proj

Demystifying Long Chain-of-Thought Reasoning in LLMs

🖥 paper
🧠 code


@Machine_learn

با عرض سلام
تخفیف ۵۰٪ دو پکیچ یادگیری ماشین و یادگیری عمیق که شامل ۳۶ پروژه عملی در بحث پردازش تصویر و پردازش متن می باشند رو در نظر گرفتیم. دوستانی که نیاز به این دو پک دارند می تونن به بنده پیام بدن. ۱ ماه مشاوره ریکان راجع به این پروژه ها هم خواهیم داشت.
🟥🟥🟥🟥🟥🟥
@Raminmousa

DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding

13 Dec 2024 · Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, Chong Ruan ·

We present DeepSeek-VL2, an advanced series of large Mixture-of-Experts (MoE) Vision-Language Models that significantly improves upon its predecessor, DeepSeek-VL, through two key major upgrades. For the vision component, we incorporate a dynamic tiling vision encoding strategy designed for processing high-resolution images with different aspect ratios. For the language component, we leverage #DeepSeekMoE models with the Multi-head Latent Attention mechanism, which compresses Key-Value cache into latent vectors, to enable efficient inference and high throughput. Trained on an improved vision-language dataset, DeepSeek-VL2 demonstrates superior capabilities across various tasks, including but not limited to visual question answering, optical character recognition, document/table/chart understanding, and visual grounding. Our model series is composed of three variants: DeepSeek-VL2-Tiny, #DeepSeek-VL2-Small and DeepSeek-VL2, with 1.0B, 2.8B and 4.5B activated parameters respectively. DeepSeek-VL2 achieves competitive or state-of-the-art performance with similar or fewer activated parameters compared to existing open-source dense and MoE-based models. Codes and pre-trained models are publicly accessible at https://github.com/deepseek-ai/DeepSeek-VL2.

Paper: https://arxiv.org/pdf/2412.10302v1.pdf

Code: https://github.com/deepseek-ai/deepseek-vl2

Datasets: RefCOCO TextVQA MMBench
DocVQA
💠
https://www.tg-me.com/deep_learning_proj

CycleGuardian: A Framework for Automatic RespiratorySound classification Based on Improved Deep clustering and Contrastive Learning

🖥 Github: https://github.com/chumingqian/CycleGuardian

📕 Paper: https://arxiv.org/abs/2502.00734v1

🌟 Dataset: https://paperswithcode.com/dataset/icbhi-respiratory-sound-database


@Machine_learn

با عرض سلام بسیاری از دوستان که می خواهند مقاله شروع کنند نیاز به نقش راهی برای شروع دارند. از این رو سعی داریم جهت مشاوره در موضوعات زیر همکاری داشته باشیم. انتخاب موضوع، انتخاب ایده، بررسی ساختار کلی مقاله و انتخاب ژورنال با بنده خواهد بود و هر هفته یک جلسه جهت بررسی کارهای انجام شده خواهیم داشت. هزینه مشاوره در هر موضوع ۵ تومن می باشد.

💠Medical Image

1-alzheimer disease classification
2-Wound image classification
3- skin cancer classification
4- breast cancer segmentation

💠Time series:
1- crypro market price prediction: High dimensional features
2-crypto market illiquidity prediction: High dimensional features
3-Air quality prediction
4-Network traffic prediction
5-Malware detection

💠Text mining
1-Large languge model: systmatic survey
2-multi-domain sentiment analysis
3- Extracting psychiatric stressors for suicide from twitter
جهت مشارکت می تونین با ایدی بنده در ارتباط باشین.
@Raminmousa