A threaded pathfinding doesn't bring many benefits due to global lock and it's a source of complex code (more than it needs to be).

The computation must be handled with a proper algorithm or we find situations like this one https://forum.minetest.net/viewtopic.php?p=280752&sid=ea546846ac1449335b1ae2ee4d23e67f#p280752 where users with not so new hardware and the computation of a lot of paths can cause a perceptible freeze in the game.

there are a lot of games which doesn't require that computation on its own thread but the require to be fast (normally no more than 2 or 3 ms).

The best time efficient alternatives are algorithms which use search trees from previous searches to speed up the current search. That implies more RAM usage.

RAM is cheap and even more as years go by, I think the better option to prevent blocking is a faster calculation. Here is my proposal for the pathfinding API:

-- no cache
minetest.find_path(pos1, pos2, searchdistance, agent definition)
this function doesn't use any cache from previous computations, it will create data for every call, its speed will be the same as in the first call of the pathfinding using cache. I may write a specialized version optimized for this function in the future.

-- with cache pool
entities won't reserve cache in the creation, mob mods will have to define which will have cache with the following function:
minetest.register_path_cache(agent definition, entity)

2 registrations of the same entity will cause an override of the previous cache.
'agent definition' is a table defining the properties of the entity, here is an example of the default values:

agent_definition = {
	height = 1,
	width = 1,
	fly = false,
	climb_walls = false,
	over_ground = true,
	diving = false,
	swimming_surface = true,
	max_jump_height = 1,
	max_drop_height = 2,
	avoided_nodes = {
    	"default:lava_source",
    	"default:lava_flowing",
    },
    node_cost_list = { -- to define special costs
        default:water_source = 20,
        default:water_flowing = 20,
	},
   jump_cost = 20
}

Entities will free its cache when deleted of unloaded due to the lack of players in their surroundings, that way we prevent an uncontrolled number of growing cache.
I think we could define some kind of load factor or limit to automatically free part of the ram from time to time.

Anyway, mob modders will have a Lua function allowing to free al the cache of a single entity:
minetest.free_path_cache(entity)

I have a design doubt, should we pass an ObjectRef or the entity id. I haven't found a method to get the id of an entity from Lua.

Well, and finally the function to find the path using the cache reserved previously:
minetest.find_path_with_cache(pos1, pos2, entity, searchdistance)

If we try to use that function with an entity without cache it will return nil.