Kamariah Ali, 60, says she should not be tried under Islamic law because she is no longer a Muslim.
She follows the Sky Kingdom sect, known as the teapot cult because it built a giant teapot to symbolise its belief in the healing purity of water.
But judges ruled that only Malaysia's Islamic courts could decide on the case because Ms Kamariah was born a Muslim.
Malaysia has two different judicial systems - one for Muslims and the other for non-Muslims.
The BBC's Jennifer Pak in Kuala Lumpur says Ms Kamariah's case is one of a growing number of legal challenges brought by those caught between the two systems.
Ms Kamariah had asked the civil courts to declare her freedom to worship, as guaranteed by the constitution.
But the judging panel said she had to go through the Islamic courts system in order to renounce her faith - something that is rarely granted, our correspondent says.
The dual-track system has created problems for people who want to convert to another religion from Islam, or in child-custody battles involving Muslims and non-Muslims.
The most high-profile case involved a Muslim father who secretly converted his children to Islam. He gained custody through the Islamic courts while the mother, a Hindu, was granted guardianship under civil law.
The Sky Kingdom sect, based in the strongly Muslim state of Terengganu, is regarded as heretical by the Islamic authorities.
It claims to promote harmony between religious groups, and its leader Ayah Pin believes he is the saviour of the world.
In 2005 many of its members - including Ms Kamariah - were prosecuted by the Islamic authorities. Ayah Pin is believed to have fled to Thailand.