Beberapa model memberikan hasil yang lebih baik dengan dataset forum kesehatan dan dengan text based retriever BM25 dan TF-IDF. Model multilingualBERT dan metode fine-tuning layer freezing memberikan hasil yang terbaik dari semua kombinasi. Kenaikan tertinggi terdapat pada kombinasi BM25 dan multilingualBERT dengan layer freezing dengan kenaikan sebesar 0.051 dibandingkan BM25. ......The system of acquiring similar questions is implemented on many Question and Answering sites, including health forums. Implementations of similar question search systems can vary, such as text-based retrievers and neural rankers. The main issue with neural rankers is the lack of research in Indonesian language for neural ranker models, especially those using BERT. This study aims to investigate how far BERT as a neural re-ranker can improve the ranking quality of a text-based retriever when applied with fine-tuning. The model used in this research is BERT, and the test collection used is a health forum dataset compiled by Nurhayati (2019). To answer the research question, experiments were conducted on multiple pre-trained models: multilingual BERT, IndoBERT, stevenWH, and distilBERT to identify the best model for fine-tuning. This study also proposes two new fine-tuning methods: attention mask filter with IDF threshholding and frozen layer by freezing some layers within BERT. These models and methods were then tested under predefined scenarios. The experiment results show that the re-ranker can enhance the quality of the text-based retriever when fine-tuned with specific methods and scenarios. These models perform especially well using the health form dataset aswell as using the text based retrievers BM25 and TF-IDF. Out of all models, multilingulBERT performed the best with freezed layer fine-tuning performing as the best fine-tuning method. The most significant increase of all combinations is the combination of BM25 and multilingualBERT with freezed layer fine-tuning with a 0.051 increase compared to the baseline BM25.